Sr. Backend Software Engineer (Infrastructure ML Ops)
San Francisco, CA - USA
Department:
Job Summary
At Philo were a group of technology and product people who set out to build the future of television marrying the best in modern technology with the most compelling medium ever invented in short were building the TV experience that weve always wanted for practice this means leveraging cloud delivery modern tech stacks machine learning and hand-crafted native app experiences on all of our platforms. We aim to deliver a rock solid experience on the streaming basics while cooking up next generation multi-screen and multi-user playback experiences.
The infrastructure team at Philo builds the foundation for a 24/7 live and on-demand streaming product serving video to our growing subscriber base. It provides a platform that enables groups across the company such as video delivery product support and analytics to do their best work easily and smoothly. Our systems are highly distributed horizontally scalable and run in the cloud on Kubernetes. We continuously test and deploy our code and closely monitor the health and performance of our production environment.
We are seeking an enthusiastic Infrastructure MLOps Software Engineer to help us build and improve the ML infrastructure at Philo. We are building out ML and LLM infrastructure and this role will be pivotal in setting direction adopting best practices and shipping infrastructure that will be used across the company.
As a sample the ML infrastructure supports teams like Discovery in building content recommendation models that will help our users find more favorite shows from the large Philo catalog. The engineering team is designing and developing our own agent harnesses to support coding and empowering everyone at the company to get answers across our code bases analytics logs and communication tools. The video player team is using LLMs to classify ads and label our video catalog so that users have a better experience. Youll build the serving evaluation and platform infrastructure that these workloads depend on.
Responsibilities:
Manage the core infrastructure at Philo with a focus on supporting ML and LLM workloads. This infrastructure is relied on by a large subscriber base of Philos product as well as internal stakeholders across the company.
Design build and maintain cloud infrastructure components (AWS and Kubernetes) using a combination of in-house technology and open source software.
Build and scale model serving and inference infrastructure for both real-time recommendation models and LLM workloads including model servers (e.g. NVIDIA Triton) LLM inference engines (e.g. vLLM TGI TensorRT-LLM) autoscaling and latency/throughput optimization.
Manage GPU and accelerator infrastructure on Kubernetes scheduling capacity and quota management and cost optimization across on-demand and spot capacity.
Design build and maintain monitoring and observability systems spanning both system health and model quality (drift evaluation and online performance) that enable Philo engineers and the support team to gain insight and to discover and debug issues.
Build and operate model evaluation infrastructure: offline and online eval pipelines golden/regression datasets LLM-as-judge harnesses A/B testing and experimentation and feedback/ground-truth loops
Design build and maintain CI/CD systems enabling Philo engineers to create pipelines to test and deploy their code.
Establish tools methods and best practices for other Philo engineers interfacing with the Philo ML infrastructure. Ensure reliability security and scalability of the Philo platform. Promote Infrastructure as Code.
Work closely with other Philo engineers to deploy and instrument software systems.
Drive evaluation selection and integration of third-party vendor systems and work closely with vendors to configure and manage them.
Qualifications:
8 years of software development and infrastructure management experience.
4 years of experience developing and deploying ML systems at scale across training and inference
Experience working with distributed systems and an understanding of microservices architecture principles.
Experience with Linux and containerized (i.e. Docker) environments.
Experience managing cloud computing environments (AWS or GCP) and configuring cloud services e.g. CloudWatch Route 53 RDS ElastiCache SQS ALB/NLB/ELB VPC networking IAM security.
Experience with container orchestration platforms (i.e. Kubernetes) and familiarity with running GPU/accelerated workloads on them (e.g. NVIDIA device plugin MIG timeslicing).
Strong understanding of networking and internet application protocols including but not limited to TCP/IP DNS and HTTP.
Strong understanding of network and application security principles and best practices.
Familiarity or hands-on experience with configuration management systems and Infrastructure as Code (e.g. Terraform CloudFormation).
Familiarity or hands-on experience with Monitoring/Observability systems (e.g. Prometheus Grafana TICK/InfluxDB Fluentd ELK Datadog).
Familiarity or hands-on experience with CI/CD automation systems e.g. Jenkins Gitlab.
Experience with relational and non-relational databases and familiarity with modern data warehousing and querying.
Proficient in writing testing and profiling software in Golang JavaScript/TypeScript C Ruby Python or similar programming languages.
Experience and aptitude for collaborating and communicating with internal and external stakeholders in both business and technical roles.
A strong candidate may also have one or more of these:
Hands-on experience optimizing inference dynamic/continuous batching quantization KV-cache management or GPU memory tuning for low-latency or highthroughput serving.
Experience with ML platform tooling: model registry and experiment tracking (e.g. MLflow Weights & Biases) feature stores (e.g. Feast Tecton) and pipeline/workflow orchestration (e.g. Airflow Dagster Kubeflow Ray Metaflow).
Experience building model evaluation monitoring or experimentation systems including drift detection LLM-as-judge or A/B testing for models.
Experience with LLMOps: inference gateways/routers (e.g. LiteLLM) hosted model APIs (Bedrock OpenAI Anthropic) RAG pipelines vector databases (e.g. pgvector Pinecone Weaviate) and prompt/version management.
Experience building or operating agent harnesses or tool-using LLM applications in production.
Experience with data and model versioning (e.g. DVC LakeFS) and data/labeling pipelines.
Experience with cost management / FinOps for GPU and inference workloads
We are language agnostic but most of our backend code is written in Golang Ruby and TypeScript with some C and Python. Our services run on Kubernetes and we practice continuous deployment across all of our systems.
More about Philo
At Philo were a company that puts people firstboth our subscribers and our team. We empower our colleagues to do their best work while supporting one another in pursuing shared goals. We value pragmatism pride in our work and passion with transparency and openness as fundamental parts of our culture.
Were committed to diversity equity inclusion and accessibility as we grow the Philo team and shape the future of TV. We believe that a diverse range of voices and perspectives enables us to innovate faster and create the best experiences for our subscribers. Philo is proud to be an Equal Opportunity Employer. Were committed to supporting every candidate and employee. If you need an accommodation at any stage of the process please email and well work with you to meet your needs.
Philo offers 70 top-rated networks including AMC BET CMT Comedy Central Discovery Channel Food Network Hallmark Channel HGTV HISTORY Investigation Discovery Lifetime MTV Nickelodeon OWN VH1 We TV and more. It also includes all the groundbreaking originals and blockbuster movies available with AMC and access to HBO Max Basic With Ads and discovery. Our service also includes 100 free channels and premium add-ons like STARZ and MGM.
Our extensive library boasts over 85000 titles and our unlimited DVR allows users to save their favorite shows and movies for up to a year skipping ads for a seamless viewing experience. Stream on up to three devices simultaneously whether on your phone tablet laptop or TV using Roku Apple TV Fire TV Samsung TV Android TV Vizio TV or Chromecast.
Philo is headquartered in San Francisco with offices in New York City and Cambridge MA. Our leadership team includes alums from HBO Tubi and Meraki and is backed by NEA and industry partners like Warner Brothers Discovery Paramount AMC and A&E.
Join us at Philo and be part of a team thats shaping the future of TV!
Status: Full-time
Location: San Francisco CA Brooklyn NY Cambridge MA or remote
Compensation: Includes annual salary company stock options and health benefits. Salary is determined by experience and location:
San Francisco New York City: $195K - $250K
Boston DC Metro Los Angeles Seattle: $185K - $240K
Denver Atlanta Austin Las Vegas Sacramento Chicago: $175K - $230K
Texas Florida: $165K - $220K
We value a diverse and inclusive workplace and we welcome people of different backgrounds experiences skills and perspectives. Philo is an equal opportunity employer. We believe that everyone does their best work when they are supported by each other and the company and we offer a generous set of benefits to make sure the Philo team is happy and healthy. Here is a sampling of the benefits we offer our team:
Full health dental and vision coverage for you and your family
401(k) plan with employer contributions (we match 100% of deferrals up to 3% of pay and 50% of the next 2% of pay)
Flexible working hours
Up to 20 weeks of fully paid parental leave
Unlimited paid time off for vacation and sick leave
$2000 annual vacation bonus (we pay you to take a two week vacation)
$5250 annually for professional development and educational assistance
$1250 annual home office TV stipend during first year of employment ($250 annually thereafter)
$500/month ($6000/year) bonus for employees who commit to working at least 3 days per week in our offices plus generous commuter benefits ($315/month towards transit rideshare bike rental or parking at our HQ office in San Francisco)
Free Gympass subscription an all-in-one corporate benefit that gives employees the largest selection of gyms studios classes training and wellness apps
Dog-friendly office
And much more!
For California Residents: Philos CCPA Notice at Collection Employees Applicants Owners Directors Officers and Contractors
Pursuant to the San Francisco Fair Chance Ordinance we will consider for employment qualified applicants with arrest and conviction records.
Use of Automated Tools in Our Hiring Process
We use software tools including our applicant tracking system (Ashby) to help manage applications for this role. Automated features (i) assess the skills experience and qualifications relevant to the role as described in this posting and (ii) flag applications that appear fraudulent duplicative or not genuinely submitted by the applicant.
A summary of the most recent independent bias audit of the automated employment decision tool used in this process including the audit date and data source is available here: Bias Audit Summary.
You may request that a member of our team review a decision through a process that does not rely on the automated tools or request a reasonable accommodation by emailing . We will not retaliate against you for exercising your rights.
Information about the type and source of the data our tools collect our data retention policy and how these tools were used in evaluating your application is described in our Privacy Notice EmployeesApplicants Officers Directors and Contractors or available by emailing .
Depending on your state of residence you may have additional rights including to an explanation of an adverse decision correction of inaccurate data and meaningful human review. To exercise these or ask whether they apply email
Required Experience:
Senior IC
About Company
Why pay more for TV? Philo offers live TV and on-demand content for just $28/month. Stream your favorite shows, movies, and more across all your devices.