Software Engineer, Data Quality Tools & Automation — AIML Data Operations
Cupertino, CA - USA
Job Summary
Youll own projects from problem definition through deployment and ongoing operation. A project might start with a question such as Where are annotation errors increasing or Is our audit sample giving us enough coverage Youll work with QA Data Science Engineering and Operations to define the measurement build the workflow or pipeline validate the results and put them in the hands of the people who can act on them.
Build internal QA tools and automation for workflows such as audit sampling gold dataset management calibration adjudication and defect triage. Integrate these tools with existing annotation and review and operate reliable data pipelines that combine annotation human review and automated QA outputs. Build in validation monitoring and support for backfills so reported metrics remain traceable and dashboards and alerts that show quality coverage and emerging risks across projects and workflows. Enable users to trace an aggregate change to the relevant task types error categories and underlying scalable quality measurement for accuracy inter-annotator agreement error rates and audit coverage. Work with QA and Data Science to define denominators minimum sample sizes and alert quality regressions using SQL Python and statistical methods. Identify recurring failure modes quantify their impact and assess whether changes to guidelines tooling or workflows resolve automated QA methods against human-reviewed examples. Measure false positives missed defects and review effort to determine where automation can reliably support human the systems you ship including design testing deployment documentation monitoring and support. Use feedback from QA and Operations teams to prioritize improvements.
Bachelors degree in Computer Science Data Science Engineering or related designing building and maintaining production software internal tools or data systems used by other programming skills in Python and strong SQL skills for data transformation validation and building automated workflows or data pipelines that integrate multiple data sources or presenting data through dashboards analytical applications or reporting interfaces that support operational to investigate data discrepancies and translate ambiguous quality or operational questions into measurable to work across technical and operational teams and explain findings and tradeoffs clearly.
Experience with data annotation human evaluation ML data pipelines or quality engineering for data-intensive with statistical sampling confidence intervals inter-annotator agreement or measurement design for subjective evaluating rule-based or model-assisted QA against human judgments including precision recall and error building systems for global operations or workflows with multiple sites and review teams.
Required Experience:
IC
About Company
Ask Siri to name the most successful company in the world and it might respond: Apple. And it's not just out of familial pride. Apple consistently ranks highly in profit, revenue, market capitalization, and consumer cachet. In 2018, the company became the first reach a trillion dollar ... View more