Evaluation Science Lead
Austin, TX - USA
Job Summary
Language carries culture nuance and context as Evaluation Science Lead you will own the scientific rigor behind measuring how our AI-powered Globalization solutions perform across 50 languages and dozens of markets. As a strategic individual contributor on the Globalization Quality and Operations team you will design statistically grounded evaluation frameworks combining human judgment with scalable automation partnering with Engineering AI Strategy and Production to turn evaluation into a strategic role requires strong analytical and scientific thinking hands-on execution and sound judgment to communicate complex findings clearly. You thrive in ambiguity and know evaluation delivers value only when operationalized if you believe rigorous culturally-informed evaluation is one of the highest-leverage ways to improve AI language quality globally this role was built for you.
Framework u0026 Methodology DesignnDefine the long-term evaluation science strategy and roadmap for Globalization AI solutions across 50 languages including targeted approaches for low-resource robust statistical methodologiesincluding power analysis sampling confidence intervals and significance thresholdsto evaluate models with scientific scalable evaluation workflows blending human annotation (calibration protocols inter-annotator agreement) with automated systems (autograders LLM-as-judge and agent-based first-pass scoring).nExpand evaluation criteria beyond core linguistic quality to incorporate behavioral and user engagement u0026 Continuous ImprovementnTranslate complex evaluation data and loss patterns into clear actionable recommendations and go/no-go evidence for Engineering and continuous performance monitoring and drift-detection systems to distinguish real regressions from measurement feedback loops with Quality Operations and Engineering to rapidly refine evaluation rubrics as models as the resident authority on multilingual AI evaluation communicating methodologies and upleveling best practices across Globalization Apple Services.
5 years of experience in evaluation science data science or ML systems development with demonstrated experience owning evaluation systems at scale end-to-endnExperience applying statistical methodology sampling significance testing and confidence intervalsnPractical understanding of measurement validity principles nHands-on experience measuring annotator agreement diagnosing divergence and improving annotation protocolsnProficiency with statistical tools and languages (R Python SQL) for analysis and reproducibilitynExperience designing evaluation frameworks adopted and scaled by operational teamsnStrong written and verbal communication skills for technical and non-technical audiencesnDirect experience handling sensitive and confidential information with integrity and discretionnAbility to be onsite; this role is an in-person onsite positionnAvailability to work occasional evenings and weekends as business needs requirenUp to 10% travel; both domestic and international
Masters PhD or comparable experience in Statistics Computational Linguistics Computer Science Psychometrics Data Science or related quantitative evaluating generative AI systemsincluding hallucination detection safety and cultural alignment autograders / LLM-as-judge systems benchmark design and synthetic data with A/B testing causal inference or experimental designnExperience in applied linguistics translating cultural and linguistic nuances into quantitative evaluation framework
About Company
Ask Siri to name the most successful company in the world and it might respond: Apple. And it's not just out of familial pride. Apple consistently ranks highly in profit, revenue, market capitalization, and consumer cachet. In 2018, the company became the first reach a trillion dollar ... View more