AI Applied Scientist, AI Evaluation Science
Seattle, WA - USA
Job Summary
Our team is building self-service evaluation: tools SDKs and AI-powered capabilities that let engineers across Apple evaluate their AI features on their own quickly and correctly so they can build and ship those features faster. You will develop the methods that make this possible: raising the quality of the evaluations engineers run and helping them understand what their results mean and what to do next. Often the method a problem needs doesnt exist yet. When that happens you will figure it out whether that means extending a recent paper or developing something new and share what you learn through publications and internal research reports. We are looking for individuals across a range of experience research and measurement scientists you will co-develop new methods make sure the tools give engineers valid trustworthy signals and bring back the hard problems that should shape the teams research agenda. With platform and research engineers you will define what each method needs to work reliably without an expert in the loop including its assumptions failure modes and validation tests so it can ship in the evaluation SDKs and APIs used across successful candidate combines scientific depth with practical judgment: you know which methods fit a problem and why can design experiments that show whether they actually work can diagnose why they break down on messy real-world data and can explain clearly what an evaluation result does and does not support.
Develop and adapt methods that raise the quality of the evaluations engineers run on LLMs and agentic and improve the ML capabilities behind the teams developer-facing evaluation tools such as rubric and test-set generation automated judging and failure analysis. Work with measurement scientists to integrate validity frameworks and psychometric methods so the tools measure what they claim to measure and give developers the right ways for engineers to understand what their evaluation results mean and what to do next such as uncertainty likely failure causes and when a result shouldnt be trusted. Partner with measurement scientists on how results are where engineers get stuck when evaluating AI features and turn their recurring questions (is this better why did this fail is it ready to ship) into capabilities the tools can and run large-scale experiments that test whether methods hold up on production data including robustness to distribution shift across domains and languages applying the teams statistical and validity standards for reliability and methods in well-structured Python and partner with research and platform engineers to productionize them into evaluation SDKs APIs and services owning the scientific correctness of what existing methods fall short develop new ones with research and measurement scientists. Turn recurring gaps into open problems for the teams research agenda and share findings through publications at top-tier venues and internal research findings limitations and recommendations clearly to technical and non-technical audiences.
Ph.D. in Computer Science Machine Learning Statistics or a closely related field or a Masters degree in one of these fields with 3 years of applied research expertise in evaluating AI systems gained through research on evaluation methods substantial hands-on experience evaluating LLMs or agentic systems in real products or proficiency in Python and the modern ML ecosystem with demonstrated ability to implement complex methods from recent papers and run large-scale -on experience with large language models whether fine-tuning inference prompting pipelines or building LLM-powered research ability through publications at peer-reviewed venues preprints or equivalent research record of applying research to practical problems: methods prototypes or tools adopted by other teams or used to inform real product software engineering fundamentals for research code: version control testing reproducibility and code others can build written and verbal communication skills including the ability to explain complex results and their limitations to diverse with ambiguity and new problem spaces; you thrive when building something that doesnt yet have a playbook.
First-author publications at top-tier venues (NeurIPS ICML ICLR ACL EMNLP) especially on evaluation methodology: papers about how to evaluate not just papers that use evaluation to demonstrate model experience in one or more of: LLM-as-judge approaches (calibration rubric design bias mitigation); benchmark design and validation (IRT contamination detection); human evaluation methodology (protocol design quality control); experimentation and comparisons between system versions (A/B testing uncertainty quantification); or agentic and multi-agent system evaluation including failure diagnosis and automated with theoretical foundations of evaluation: measurement theory and validity frameworks statistical learning theory (calibration reliability decision theory) causal inference or preference with the practical challenges of evaluating generative AI and agentic systems: non-deterministic outputs token economics multi-turn agent trajectories and tool building methods or tools used directly by non-experts such as developer-facing ML tooling or self-service with modern evaluation tools and frameworks (e.g. DeepEval Ragas TruLens LangSmith or agent tracing and observability tools).nExperience working with distributed computing or cloud-native tooling (e.g. Ray Spark Docker/Kubernetes) to run evaluations at with cross-disciplinary work such as collaboration with social scientists psychometricians or domain in fast-moving early-stage teams where you helped define technical direction from the ground up.
Required Experience:
IC
About Company
Ask Siri to name the most successful company in the world and it might respond: Apple. And it's not just out of familial pride. Apple consistently ranks highly in profit, revenue, market capitalization, and consumer cachet. In 2018, the company became the first reach a trillion dollar ... View more