Lead Observability Engineer
Posted:
19 November 2026 (10 hours ago)
Application Deadline:
16 February 2027
Vacancies:
1 Vacancy
Job Summary
OmniOps is a Riyadh-based technology solutions provider serving organizations across Saudi Arabia and beyond. With offices in Jordan Egypt and Morocco we specialize in empowering businesses to migrate and scale their AI technology infrastructure confidently achieving a high level of maturity in their digital landscapes. Our comprehensive services and products guarantee seamless cloud migration scalability and management ensuring optimal performance and consistent reliability for our valued partners.
**Role Summary**
You set the technical direction for how we deliver observability to every client across Grafana Alloy the LGTM stack (Loki Grafana Tempo Mimir) and OpenTelemetry. At Intermediate level the job is to deploy and run the stack well. At Senior/Lead level the job is to design the reference architecture define the standards others deploy to mentor the engineers doing the work and be the senior technical voice in front of clients.
You stay hands-on: you will still write Alloy pipelines debug cardinality blow-ups and lead incident response on the platform. But your main output is a platform and a team that scale across many tenants without you in every change.
**Architecture and platform strategy**
* Own the reference architecture for client observability across all deployment models: on-prem LGTM Grafana Cloud and hybrid setups that mix the two.
* Decide which model fits each client based on data volume retention compliance data residency cost and the client's in-house skills.
* Design for scale and multi-tenancy: Mimir and Loki tenant isolation shardingreplication object storage backends high availability and disaster recovery.
* Set the OpenTelemetry strategy: when to use OTLP end to end when to use Alloy's
native components and how semantic conventions and resource attributes are applied across clients.
* Track the Grafana and OpenTelemetry roadmaps run proofs of concept (for example Grafana Beyla/eBPF Pyroscope profiling Faro frontend observability) and decide what we adopt.
**Standards automation and pipelines**
* Define and maintain the standard Alloy configurations for Kubernetes Linux and Windows as versioned reusable modules rather than per-client copies.
* Own the label and attribute taxonomy and the cardinality budget; set guardrails (relabelling drop rules limits) that stop one client or service from degrading a shared stack.
* Drive infrastructure-as-code and GitOps for the whole platform: stacks agents dashboards alert rules and Grafana Cloud resources.
* Build the release and upgrade process for agents and backends including staged rollouts canaries and rollback across client estates.
**Reliability cost and operations**
* Set SLOs for the observability platform itself (ingest availability query latency data freshness) and report against them.
* Own capacity planning and cost: storage growth retention tiers Grafana Cloud usage and plan limits and the levers to reduce spend (adaptive metrics log sampling trace tail sampling).
* Lead or act as escalation point for platform incidents run blameless postmortems and make sure the fixes land.
* Mature self-monitoring into a meta-monitoring layer that detects silent data loss agent drift and backend saturation before clients notice.
* Embed security in the platform: authentication and SSO RBAC and team/folder permissions secrets handling encryption in transit and at rest and PII scrubbing/redaction in log and trace pipelines.
**Client engagement and advisory**
* Act as the senior technical lead on client engagements: discovery solution design effort estimates and technical sign-off.
* Run observability maturity assessments and produce roadmaps that move clients from basic monitoring to SLO-driven operations.
* Lead migrations onto the platform from tools such as Prometheus/Thanos ELK/OpenSearch Datadog New Relic Splunk or legacy Grafana Agent/Promtail.
* Coach client app teams on OpenTelemetry instrumentation and SLO design. The boundary from the Intermediate role still holds: clients own what they emit and their target numbers; we own the standards the pipelines and the alerting machinery. Atthis level you set and negotiate that boundary not just follow it.
**Leadership and mentoring**
* Mentor Intermediate and Junior engineers through pairing design reviews and code reviews.
* Write and own the runbooks architecture decision records and onboarding material that let the team deliver consistently without you.
* Set the team's engineering quality bar: review standards testing of configurations and definition of done for a client rollout.
* Work with delivery managers sales/pre-sales and account teams to shape scope pricing inputs and service offerings.
**Required Skills And Experience**
* 6 years in DevOps SRE or platform engineering with at least 3 years focused on observability in production.
* Deep production-scale experience with the Grafana stack: Grafana Loki Mimir (or Prometheus/Thanos/Cortex) Tempo and Alertmanager including tuning scaling and troubleshooting their internals (ingesters distributors compactors query frontends object storage).
* Expert with Grafana Alloy and/or the OpenTelemetry Collector: designing pipelines writing reusable modules processors and sampling strategies and running them across Kubernetes Linux and Windows.
* Strong OpenTelemetry knowledge: OTLP SDKs and auto-instrumentation in at least two languages semantic conventions context propagation and the logs/metrics/traces correlation model.
* Advanced Kubernetes: operators Helm chart authoring RBAC resource tuning and multi-cluster setups.
* Advanced query and data-modelling skills: PromQL LogQL and TraceQL; recording rules; cardinality analysis and control.
* Proven experience designing SLOs error budgets and burn-rate alerting and turning them into working Grafana or Alertmanager rules.
* Infrastructure-as-code and GitOps in practice (Terraform Helm Argo CD or Flux Ansible) and CI/CD for configuration.
* Grafana Cloud at an administrative level: stacks and tenants access policies usage and cost management private connectivity.
* A track record of technical leadership: owning architecture decisions mentoring engineers and leading incidents.
* Clear client-facing communication: you can explain trade-offs to engineers and to non technical stakeholders and hold scope boundaries politely and firmly.
**Nice To Have**
* Experience leading observability delivery in an MSP or multi-tenant managed-service setting across many clients at once.
* Hands-on with AWS Azure and GCP native telemetry (CloudWatch Azure Monitor Cloud Logging) and integrating it into a central backend.
* Windows telemetry at depth: Windows Exporter Event Log collection ETW and IIS logs via Alloy.
* eBPF-based observability (Grafana Beyla Cilium/Hubble) continuous profiling (Pyroscope) or frontend observability (Grafana Faro).
* Grafana k6 synthetic monitoring Grafana OnCall/IRM or incident tooling such as PagerDuty or Opsgenie.
* Contributions to Grafana OpenTelemetry or related open-source projects or public talks and writing on observability.
* Certifications such as Grafana Certified Associate CKA/CKAD or a cloud architect certification.
**Role Summary**
You set the technical direction for how we deliver observability to every client across Grafana Alloy the LGTM stack (Loki Grafana Tempo Mimir) and OpenTelemetry. At Intermediate level the job is to deploy and run the stack well. At Senior/Lead level the job is to design the reference architecture define the standards others deploy to mentor the engineers doing the work and be the senior technical voice in front of clients.
You stay hands-on: you will still write Alloy pipelines debug cardinality blow-ups and lead incident response on the platform. But your main output is a platform and a team that scale across many tenants without you in every change.
**Architecture and platform strategy**
* Own the reference architecture for client observability across all deployment models: on-prem LGTM Grafana Cloud and hybrid setups that mix the two.
* Decide which model fits each client based on data volume retention compliance data residency cost and the client's in-house skills.
* Design for scale and multi-tenancy: Mimir and Loki tenant isolation shardingreplication object storage backends high availability and disaster recovery.
* Set the OpenTelemetry strategy: when to use OTLP end to end when to use Alloy's
native components and how semantic conventions and resource attributes are applied across clients.
* Track the Grafana and OpenTelemetry roadmaps run proofs of concept (for example Grafana Beyla/eBPF Pyroscope profiling Faro frontend observability) and decide what we adopt.
**Standards automation and pipelines**
* Define and maintain the standard Alloy configurations for Kubernetes Linux and Windows as versioned reusable modules rather than per-client copies.
* Own the label and attribute taxonomy and the cardinality budget; set guardrails (relabelling drop rules limits) that stop one client or service from degrading a shared stack.
* Drive infrastructure-as-code and GitOps for the whole platform: stacks agents dashboards alert rules and Grafana Cloud resources.
* Build the release and upgrade process for agents and backends including staged rollouts canaries and rollback across client estates.
**Reliability cost and operations**
* Set SLOs for the observability platform itself (ingest availability query latency data freshness) and report against them.
* Own capacity planning and cost: storage growth retention tiers Grafana Cloud usage and plan limits and the levers to reduce spend (adaptive metrics log sampling trace tail sampling).
* Lead or act as escalation point for platform incidents run blameless postmortems and make sure the fixes land.
* Mature self-monitoring into a meta-monitoring layer that detects silent data loss agent drift and backend saturation before clients notice.
* Embed security in the platform: authentication and SSO RBAC and team/folder permissions secrets handling encryption in transit and at rest and PII scrubbing/redaction in log and trace pipelines.
**Client engagement and advisory**
* Act as the senior technical lead on client engagements: discovery solution design effort estimates and technical sign-off.
* Run observability maturity assessments and produce roadmaps that move clients from basic monitoring to SLO-driven operations.
* Lead migrations onto the platform from tools such as Prometheus/Thanos ELK/OpenSearch Datadog New Relic Splunk or legacy Grafana Agent/Promtail.
* Coach client app teams on OpenTelemetry instrumentation and SLO design. The boundary from the Intermediate role still holds: clients own what they emit and their target numbers; we own the standards the pipelines and the alerting machinery. Atthis level you set and negotiate that boundary not just follow it.
**Leadership and mentoring**
* Mentor Intermediate and Junior engineers through pairing design reviews and code reviews.
* Write and own the runbooks architecture decision records and onboarding material that let the team deliver consistently without you.
* Set the team's engineering quality bar: review standards testing of configurations and definition of done for a client rollout.
* Work with delivery managers sales/pre-sales and account teams to shape scope pricing inputs and service offerings.
**Required Skills And Experience**
* 6 years in DevOps SRE or platform engineering with at least 3 years focused on observability in production.
* Deep production-scale experience with the Grafana stack: Grafana Loki Mimir (or Prometheus/Thanos/Cortex) Tempo and Alertmanager including tuning scaling and troubleshooting their internals (ingesters distributors compactors query frontends object storage).
* Expert with Grafana Alloy and/or the OpenTelemetry Collector: designing pipelines writing reusable modules processors and sampling strategies and running them across Kubernetes Linux and Windows.
* Strong OpenTelemetry knowledge: OTLP SDKs and auto-instrumentation in at least two languages semantic conventions context propagation and the logs/metrics/traces correlation model.
* Advanced Kubernetes: operators Helm chart authoring RBAC resource tuning and multi-cluster setups.
* Advanced query and data-modelling skills: PromQL LogQL and TraceQL; recording rules; cardinality analysis and control.
* Proven experience designing SLOs error budgets and burn-rate alerting and turning them into working Grafana or Alertmanager rules.
* Infrastructure-as-code and GitOps in practice (Terraform Helm Argo CD or Flux Ansible) and CI/CD for configuration.
* Grafana Cloud at an administrative level: stacks and tenants access policies usage and cost management private connectivity.
* A track record of technical leadership: owning architecture decisions mentoring engineers and leading incidents.
* Clear client-facing communication: you can explain trade-offs to engineers and to non technical stakeholders and hold scope boundaries politely and firmly.
**Nice To Have**
* Experience leading observability delivery in an MSP or multi-tenant managed-service setting across many clients at once.
* Hands-on with AWS Azure and GCP native telemetry (CloudWatch Azure Monitor Cloud Logging) and integrating it into a central backend.
* Windows telemetry at depth: Windows Exporter Event Log collection ETW and IIS logs via Alloy.
* eBPF-based observability (Grafana Beyla Cilium/Hubble) continuous profiling (Pyroscope) or frontend observability (Grafana Faro).
* Grafana k6 synthetic monitoring Grafana OnCall/IRM or incident tooling such as PagerDuty or Opsgenie.
* Contributions to Grafana OpenTelemetry or related open-source projects or public talks and writing on observability.
* Certifications such as Grafana Certified Associate CKA/CKAD or a cloud architect certification.