AIML Operations & Assurance Engineer
Chantilly, VA - USA
Job Summary
Peraton is a next-generation national security company that drives missions of consequence spanning the globe and extending to the farthest reaches of the galaxy. As the worlds leading mission capability integrator and transformative enterprise IT provider we deliver trusted highly differentiated solutions and technologies to protect our nation and allies. Peraton operates at the critical nexus between traditional and nontraditional threats across all domains: land sea space air and cyberspace. The company serves as a valued partner to essential government agencies and supports every branch of the U.S. armed forces. Each day our employees do the cant be done by solving the most daunting challenges facing our customers. Visit to learn how were keeping people around the world safe and secure.
We are seeking an AI/ML Operations & Assurance Engineer to monitor evaluate and maintain production AI-enabled applications foundation-model integrations and agentic workflows. This role focuses on operational reliability workflow performance evaluation and regression testing configuration control usage and cost monitoring and the safe introduction of AI-related changes into production environments.
The ideal candidate combines cloud and application operations experience with an understanding of large language model (LLM) applications agent behavior and AI-specific operational risks. The engineer will collaborate with AI/ML developers site reliability engineers cybersecurity teams and platform operations personnel to maintain reliable auditable and secure AI-enabled services.
Monitor availability latency error rates model and token usage service costs tool-call failures and workflow completion metrics for production AI services.
Integrate AI service health and agent-workflow telemetry into observability dashboards monitoring systems and alerting processes.
Investigate failed stalled repetitive or abnormal AI workflows and coordinate remediation with software platform and AI engineering teams.
Maintain operational runbooks escalation procedures troubleshooting guidance and recovery processes for AI-enabled applications.
Distinguish infrastructure and application failures from issues associated with model behavior prompts retrieval sources or agent orchestration.
Develop and maintain repeatable evaluation scenarios for critical AI workflows developer assistants and model-service updates.
Assess task completion output quality error patterns inappropriate tool use and other indicators of operational suitability.
Coordinate regression testing before changes to models prompts tools workflow logic or retrieval sources are deployed to production.
Analyze user feedback and operational results to identify where workflows require refinement additional controls or restrictions on automated execution.
Document evaluation results limitations and recommendations to support evidence-based deployment decisions.
Maintain version control and change records for prompts agent instructions tool definitions model selections evaluation datasets and workflow configurations.
Coordinate model and service updates with AI engineering cybersecurity platform operations and architecture stakeholders.
Support rollback to validated configurations when changes negatively affect reliability performance or workflow quality.
Ensure AI-related changes follow established review testing approval and deployment procedures.
Monitor configuration drift and verify that production services remain aligned with approved configurations.
Coordinate with cybersecurity and identity teams to verify access controls tool permissions logging and auditability for production AI services.
Support investigations involving anomalous AI behavior unauthorized actions or potential data-handling concerns.
Collect and maintain operational evidence supporting security assessments continuous monitoring and service reviews.
Verify that production workflows use approved models endpoints tools and data sources in accordance with established policies.
Support the implementation and validation of operational safeguards for AI-enabled functions.
Identify repetitive operational tasks and support activities suitable for controlled AI-assisted automation.
Measure the impact of deployed workflows on processing time support workload service quality and operational efficiency.
Collaborate with service delivery Tier 2 support SRE and platform engineering teams to improve workflows based on operational demand.
Report trends in service health model usage operating costs evaluation results and unresolved issues.
Recommend improvements to monitoring testing configuration control and operational procedures.
Location: This position is fully onsite in Chantilly VA.
Active TS/SCI clearance with CI Polygraph.
- BS/BA and 5 years of experience an additional 4 years of experience may be considered in lieu of a degree.
Approximately 58 years of experience in platform operations site reliability engineering cloud engineering software operations applied AI/ML operations or production application support.
Experience operating and troubleshooting production software applications or cloud services.
Experience with monitoring logging metrics incident response and configuration or change management.
Experience evaluating software behavior through repeatable test cases regression testing quality measures or operational performance analysis.
Working knowledge of foundation-model or LLM-based applications including prompts model endpoints tool calls and retrieval-augmented applications.
Experience with AWS or comparable cloud platforms.
Strong analytical troubleshooting documentation and communication skills.
Experience with AWS Bedrock or comparable managed foundation-model services.
Familiarity with LLM or agent evaluation frameworks automated evaluation harnesses or AI application testing.
Experience with OpenTelemetry Amazon CloudWatch OpenSearch Prometheus Grafana or comparable observability technologies.
Familiarity with ServiceNow IT service management and incident-management processes.
Experience with GitLab CI/CD pipelines and controlled configuration promotion.
Familiarity with Kubernetes and containerized application environments.
Experience monitoring cloud utilization model consumption and service costs.
Knowledge of risk management frameworks continuous monitoring or security evidence collection.
Experience supporting classified restricted or highly regulated environments.
Peraton offers eligible employees a variety of benefits including medical dental vision life health savings account short/long term disability EAP parental leave 401(k) paid time off (PTO) for vacation and company paid holidays. A full listing of available benefits can be viewed at Statements
Required Experience:
IC
About Company
Peraton provides innovative solutions for the most sensitive and critical programs in government today, developed and executed by scientists, engineers, and other experts.