Engineering Manager, Private Cloud Compute Inference Routing
Seattle, WA - USA
Job Summary
Routing LLM inference at Apples scale takes systems that adapt to changing demand available capacity and failures while keeping latency low and accelerator utilization high. As the fleet grows across regions and cells routing must account for capacity and health across the whole fleet not just a single site. PCC adds a basic architectural requirement: request content stays encrypted outside trusted compute nodes. Youll lead the design of routing and scheduling strategies within this privacy model. Together with inference and security teams youll evaluate techniques such as cache-aware request placement and prefill/decode disaggregation. Your scope includes the inference AI gateway layer where requests are authenticated metered prioritized and streamed back to clients all without exposing request work has broad reach. Every improvement to routing makes Siri AI and Apple Intelligence faster and more reliable and makes better use of Apples inference your first year youll be expected to grow a high-performing organization set a clear technical roadmap and measurably improve routing latency availability and fleet utilization.
Direct the recruitment development and leadership for engineers and engineering managers responsible for PCCs inference routing and request scheduling services fostering an inclusive culturenEstablish technical strategy with senior engineers guiding architectural decisions and complex trade-offs in global traffic management capacity-aware placement load balancing and failovernTranslate organizational objectives into actionable roadmaps execution plans and measurable resultsnOversee delivery of adaptive routing admission control prioritization and overload management to maintain service responsiveness and share capacity fairly across competing workloadsnEvolve inference AI gateway capabilities including including authentication token-aware rate limiting and quotas model and version routing fallback and retries and streaming response deliveryoperating on request metadata to preserve end-to-end encryptionnCollaborate with model serving and infrastructure leaders to assess prefill/decode disaggregation and cache locality optimizations within PCCs privacy architecturenDevelop and scale robust operations through observability service-level objectives (SLOs) testing safe deployment and incident management in partnership with Site Reliability Engineering (SRE)nEnhance efficiency via benchmarking and capacity-aware scheduling leveraging queueing delay latency throughput and utilization to guide investmentnProvide senior leadership with transparent updates on team health technical risk and strategic trade-offs converting complex technical issues into actionable decisionsnPartner with Siri AI Apple Intelligence infrastructure security and SRE teams to deliver well-integrated routing capabilities
Extensive software engineering experience with a demonstrated track record of leading teams that design build and operate large-scale distributed systems in production environmentswhether through direct management indirect oversight or a combination thereofnProven ability to recruit develop and retain high-performing engineers and leadersnExperience delivering API gateways (preferably inference AI gateways) request routing load balancing scheduling or traffic management systems at scalenExpertise of LLM inference serving including request routing KV-cache placement and reuse prefill/decode disaggregation and the tradeoffs between latency throughput and accelerator usenA record of shipping highly reliable systems in high-availability environments including leading incident management on-call and reliability practicesnStrong technical depth: capable of contributing meaningfully to architectural and design discussions with senior engineers with a server development background in Java C/C Go or SwiftnAbility to drive alignment and execution across teams in complex matrixed environmentsnBachelors degree in an engineering discipline or equivalent professional experience
Proven leadership in developing LLM inference serving or routing platforms (e.g. llm-d NVIDIA Dynamo vLLM SGLang or managed inference services) nExperience building or running inference AI gateways and their core behaviors: token-based rate limiting and usage metering multi-model routing with fallback retries and circuit breaking and streamingresponse handlingnTrack record of scaling teams and organizations during rapid growth while maintaining top-tier hiring and engineering standardsnExperience delivering capacity tiers reservations or QoS for high-volume workloads and improving accelerator utilization through cross-region capacity-aware routingnUnderstanding of privacy-preserving system design: TLS X.509 key management Oblivious HTTP remote attestationnExperience designing and maturing observability practices including SLO/SLI frameworks distributed tracing and on-call culturenExperience with container orchestration systems such as KubernetesnDemonstrated ability to build psychologically safe high-trust team environments that attract and retain top talent nExperience formulating multi-year roadmaps in collaboration with product research and senior leadership within large organizations that prioritize quality and reliability
Required Experience:
Manager
About Company
Ask Siri to name the most successful company in the world and it might respond: Apple. And it's not just out of familial pride. Apple consistently ranks highly in profit, revenue, market capitalization, and consumer cachet. In 2018, the company became the first reach a trillion dollar ... View more