Reliability & Observability Baseline

One instrumentation and alerting contract every service inherits, so on-call does not require service-specific tribal knowledge.

Each service had its own dashboards, its own alert thresholds and its own definition of healthy.

Incidents were slow to diagnose because responders had to learn the service before they could read its signals.

Cardinality and retention cost had to stay bounded, and instrumentation could not require rewriting existing services.

OpenTelemetry instrumentation feeds a shared metrics and logs stack. SLOs are declared next to the service and rendered into dashboards and alerts automatically.

Alerted on symptoms defined by SLO burn rate rather than on resource-level thresholds.

Started with the three services that generated the most pages, then generalised the pattern.

Placeholder — measured outcomes will be published with the real case study.

Deleting alerts improved reliability more than adding them.