Observability & Monitoring

An automated notification triggered when a condition is met. Good alerts are actionable and tied to customer impact.

The desensitization that happens when people receive too many alerts, especially noisy or non-actionable ones, so they start ignoring the pager, including real incidents.

Alerting on what users experience (errors, latency) rather than on causes (CPU, memory). It keeps pages meaningful because every one maps to real impact.

An alert that fires only when both a short and a long window show a high error-budget burn rate, catching fast outages while ignoring brief blips.

Routing severity by urgency: a fast burn pages a human immediately, while a slow burn opens a ticket for the next working day.

The rotation of engineers responsible for responding to alerts for a service during a shift.

The average time to restore service after an incident begins. A common reliability metric alongside MTBF (mean time between failures).

The four signals Google recommends watching for any user-facing system: latency, traffic, errors, and saturation.

A dashboard pattern for request-driven services: Rate, Errors, and Duration.

A pattern for resources: Utilization, Saturation, and Errors, useful for diagnosing infrastructure like CPU, disk, and network.

Latency reported at a rank rather than an average, like p99 (the value 99% of requests come in under). Percentiles reveal the slow tail that averages hide.

A curated set of charts giving an at-a-glance view of a service's health, usually built around the golden signals or an SLO.

The investigation, often after an incident, that traces a symptom back to its underlying cause, typically using logs and traces together.

Following a single request across service boundaries by propagating a trace context, so you can see which hop was slow or failed.

The ability to understand a system's internal state from the data it emits, so you can answer questions you did not anticipate in advance.

Collecting and alerting on predefined signals to tell you whether known conditions are healthy. Monitoring answers "what" and "when"; observability helps with "why".

Numeric measurements sampled over time, such as request rate, error count, or latency. Cheap to store and query, ideal for dashboards and alerts.

Timestamped, often text records of discrete events. Rich in detail and good for root-cause analysis, but more expensive to store and search at scale.

Records of a single request as it moves across services, showing the path and timing of each step. Essential for debugging distributed systems.

One unit of work within a trace (for example, a single service call or database query). Spans nest to form the full trace.

The number of unique label combinations on a metric. High cardinality (like a label per user ID) explodes storage and cost, so it is a common source of monitoring bills getting out of control.

A vendor-neutral standard and set of SDKs for generating and exporting metrics, logs, and traces, so you are not locked into one backend.

A widely used open-source metrics system that scrapes time-series data and queries it with PromQL. The de facto standard for Kubernetes metrics.

Prometheus's query language for selecting and aggregating time series, used to build dashboards, recording rules, and alerts.

An open-source visualization tool that builds dashboards over metrics, logs, and traces from many data sources.

A small process that exposes a system's metrics in a format a collector (like Prometheus) can scrape, for software that does not emit them natively.

How often a metrics collector pulls fresh samples from a target. Shorter intervals mean more resolution but more storage.

Emitting logs as machine-parseable key/value data (usually JSON) instead of free-form text, so they can be filtered and aggregated reliably.

Keeping only a fraction of traces (or logs) to control cost and volume. Head sampling decides up front; tail sampling decides after seeing the whole trace.

A measured signal of service quality, usually a ratio of good events to total events, such as the fraction of requests served under 300ms.

A target for an SLI over a window, like 99.9% of requests succeeding over 30 days. It is the internal reliability goal a team commits to.

A contract with customers that includes consequences (like credits) if reliability drops below an agreed level. An SLA is usually looser than the internal SLO.

The allowed amount of unreliability, equal to 100% minus the SLO. It is how much failure you can spend before you must stop shipping features and focus on reliability.

How fast you are consuming the error budget relative to the SLO window. A burn rate above 1 means you will exhaust the budget before the window ends.

Manual, repetitive operational work that scales with the service and has no lasting value. Reducing toil is a core goal of SRE.