Senior Software Engineer, Data Infrastructure Compute
San Mateo, CA - USA
Job Summary
Every day tens of millions of people come to Roblox to explore create play learn and connect with friends in 3D immersive digital experiences all created by our global community of developers and creators.
At Roblox were building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together from anywhere in the world and on any device. Were on a mission to connect a billion people with optimism and civility and looking for amazing talent to help us get there.
A career at Roblox means youll be working to shape the future of human interaction solving unique technical challenges at scale and helping to create safer more civil shared experiences for everyone.
Robloxs data platform processes petabytes of data daily powering analytics ML and product decisions. The Data Platform Compute team owns the compute layer underneath it: the Kubernetes (EKS) clusters where all of Robloxs Spark workloads run and the software that lets engineers analysts and ML practitioners submit and run that work reliably and cost-effectively.
As a Senior Software Engineer on this team you will build and operate the systems that sit between our users and the cluster. These include the Spark job submission service our internal Spark libraries interactive notebook compute and cluster lifecycle automation. Youll have real ownership youll work on problems where the design is not yet settled and youll help decide which technologies Roblox adopts next.
You Will:
- Build and own core compute platform services: Design build and operate the software behind Spark job submission including APIs scheduling integration and the libraries users depend on.
- Launch interactive Spark at scale: Deliver Spark Connect to support notebook workflows including provisioning and managing the EKS compute that backs them with sensible isolation resource limits and cost controls.
- Run a safe automated cluster fleet: Build fleet management automation for EKS such as blue-green rollouts that validate new AMIs and Kubernetes versions before they reach production workloads with minimal disruption to running jobs.
- Make jobs run better without user effort: Develop our auto-tuner which analyzes Spark history and event logs to recommend or apply job configuration changes that improve performance and reduce cost.
- Own batch scheduling on Kubernetes: Operate and upgrade our YuniKorn-based scheduling including queue design quotas fairness across tenants and version upgrades.
- Shape what we adopt next: Evaluate emerging technology such as native Spark acceleration (e.g. Apache Comet). Run rigorous benchmarks on real Roblox workloads and make a clear recommendation backed by data.
- Be an owner of reliability and cost: Define SLOs and metrics build observability and drive down toil incidents and compute spend across a multi-tenant platform.
- Partner across the company: Work with data engineering ML analytics and infrastructure teams to understand their needs communicate changes and migrate users to better paths.
You Have:
- 5 years of experience building and operating production-grade distributed systems or infrastructure platforms.
- Spark on Kubernetes: Hands-on experience running Apache Spark on Kubernetes (EKS GKE or self-managed) as a platform including executor and driver behavior resource tuning and troubleshooting failed or slow jobs. We are looking for platform builders not just Spark users.
- Kubernetes depth: Strong experience operating production Kubernetes clusters at scale such as upgrades node lifecycle autoscaling and operators or controllers.
- Engineering foundations: Strong proficiency in at least one of Go Java Scala or Clojure with a track record of writing clean well-tested maintainable backend code. Python is a plus since much of our tuning and analysis tooling uses it.
- Cloud fluency: Experience with AWS (EKS EC2 IAM S3) or GCP equivalents and infrastructure-as-code.
- Operational ownership: Experience taking a system from design through production operation including on-call incident response and observability.
- Technical leadership: Ability to influence design decisions across teams mentor engineers and drive alignment on cross-cutting changes.
- B.S. in Computer Science or equivalent practical experience.
- Bonus Points:
- Experience with YuniKorn Volcano Kueue or other batch schedulers on Kubernetes.
- Experience with Spark Connect Livy Kyuubi or other interactive and gateway layers for Spark.
- Experience with the Spark Operator Karpenter Helm Terraform or Argo.
- Experience building tooling that uses Spark event logs or metrics for tuning cost attribution or anomaly detection.
- Experience maintaining internal Spark builds or driving Spark version upgrades.
- Familiarity with native execution engines (Comet Gluten Velox) or JVM performance tuning.
- Experience migrating workloads from YARN/Hadoop to Kubernetes.
- Contributions to open-source projects in the data or Kubernetes ecosystem.
- Experience operating at consumer-internet scale (100M users).
Required Experience:
Senior IC
About Company
Roblox is the ultimate virtual universe that lets you create, share experiences with friends, and be anything you can imagine. Join millions of people and discover an infinite variety of immersive experiences created by a global community!