Algorithm Engineer, LLM Evaluation

What you’ll do

1. 01Evaluation systems and benchmarks: design tasks, scoring criteria, and metrics for core model capabilities and real-world applications. Build benchmarks for text generation, multimodal understanding and generation, and agent tool use, and update them as model capabilities evolve.
2. 02Evaluation data production and quality control: combine real-world scenarios, expert annotations, and synthetic data to build automated workflows for producing, augmenting, filtering, and validating evaluation data. Ensure the data is accurate, diverse, and traceable, with difficulty levels that distinguish model capabilities.
3. 03Automated evaluation methods: explore and implement rule-based checks, LLM-as-a-judge, executable verification, and human-in-the-loop evaluation. Improve evaluation efficiency, and assess and improve agreement between automated scores and human judgments.
4. 04Model failure analysis and iteration support: analyze performance across tasks, domains, and difficulty levels. Establish methods to classify failures and identify their causes, pinpoint capability gaps, and develop actionable recommendations for improving data, training, or applications. Track and validate the resulting improvements.
5. 05Evaluation tools and platforms: build reproducible, scalable evaluation workflows that support batch experiments, version control, results analysis, and regression testing, and apply them to model R&D and business delivery.

What we look for

1. 01A master’s degree or higher in computer science, artificial intelligence, mathematics, statistics, or a related field, with at least 1 year of experience in machine learning or related algorithm R&D; experience in LLM evaluation, training optimization, or real-world applications is preferred.
2. 02Solid foundations in machine learning and familiarity with at least one of NLP, CV, multimodal modeling, or reinforcement learning, with an understanding of the fundamentals of LLMs, their training workflows, and common capability limitations.
3. 03Proficiency in Python and solid engineering skills, with the ability to independently process data, develop experiments, and analyze results.
4. 04Solid experimental design and statistical analysis skills, with the ability to break complex, open-ended tasks into measurable evaluation dimensions, assess the reliability of results, and identify data bias and metric limitations.
5. 05Clear, logical thinking and attention to detail, with the ability to identify patterns in failure cases and work with algorithm, data, and product teams to resolve problems.

Nice to have

• Experience leading or making substantial contributions to evaluation benchmarks, evaluation platforms, or automated data production workflows.
• Experience with judge-model calibration, preference evaluation, designing complex scoring criteria, or human-in-the-loop annotation.
• Experience evaluating agents, multi-turn interactions, or tool use.
• Familiarity with data contamination detection, evaluation dataset deduplication, adversarial testing, or model robustness assessment.
• Publications at conferences such as NeurIPS, ICML, ICLR, ACL, EMNLP, or CVPR, or in related journals, or high-quality open-source contributions.