Google's SRE origin, the SRE vs DevOps distinction, and the core principles of reliability engineering.

On-call engineering, toil reduction, capacity planning, and the balance between feature velocity and reliability.

Why reliability is a product requirement, not just an operational concern.

Practical guidance and case studies that complement the original SRE book.

ps, top, htop, kill, nice, systemd units, and process states.

inodes, mount points, df/du, lsof, and diagnosing disk I/O bottlenecks with iostat.

vmstat, free, /proc/meminfo, perf, and understanding OOM killer behaviour.

strace, ltrace, dmesg, and reading kernel logs for low-level system debugging.

How packets travel from source to destination, TCP handshake, and connection states.

Resolution chain, TTL, dig/nslookup, CNAME vs A records, and DNS propagation issues.

Request lifecycle, status codes, headers, TLS handshake, and certificate chains.

curl, traceroute, mtr, tcpdump, netstat/ss, and Wireshark for packet analysis.

Variables, conditionals, loops, functions, error handling (set -euo pipefail), and exit codes.

grep, awk, sed, cut, sort, uniq, and jq for structured log and JSON processing.

crontab syntax, systemd timers, and avoiding overlapping job execution.

Scripting operational tasks in Python requests, subprocess, boto3, and argparse.

Branching strategies, pull requests, tagging releases, and git blame for audit trails.

Git as the single source of truth declarative config, automated reconciliation, and self-healing.

Store operational runbooks in Git alongside the systems they describe for version control and discoverability.

Container lifecycle, resource limits (--cpus, --memory), log drivers, and health checks.

Pod lifecycle, rolling deployments, resource requests/limits, liveness and readiness probes.

Managed node groups, Fargate profiles, cluster upgrades, and EKS add-ons.

Scale workloads based on CPU, memory, and custom metrics from SQS or Prometheus.

SLIs are quantitative measures of service behaviour availability, latency, throughput, and error rate.

User-centric SLIs vs system metrics measure what the user experiences, not what is easy to collect.

Latency, traffic, errors, and saturation Google's framework for monitoring any system.

Rate/Errors/Duration for request-driven services; Utilisation/Saturation/Errors for resources.

SLO = SLI over a time window with a target threshold how to set achievable, meaningful targets.

Error budget = 1 - SLO target. Use it to balance reliability work vs feature velocity.

Define what happens when the budget is consumed feature freeze, reliability sprints, or escalation.

Regularly review and adjust SLOs as user expectations and system capabilities evolve.

The hierarchy: SLI measures it, SLO targets it internally, SLA commits to it externally.

SLAs should be set tighter than SLOs to provide a safety buffer before breaching customer commitments.

Financial penalties, service credits, and reputational impact of breached SLAs.

Scrape model, targets, exporters, the Prometheus data model, and labels.

Instant vectors, range vectors, rate(), irate(), histogram_quantile(), and recording rules.

Write Prometheus alerting rules with for clauses, severity labels, and runbook annotations.

Serverless Prometheus-compatible monitoring at scale on AWS with AMP.

Panels, variables, annotations, alerts, and dashboard-as-code with Grafonnet/Jsonnet.

Fully managed Grafana on AWS with native CloudWatch, AMP, and X-Ray data sources.

Design dashboards for on-call use clear signal hierarchy, avoid noise, and link to runbooks.

Build operational dashboards with CloudWatch metrics, alarms, and Logs Insights widgets.

JSON log format, log levels, correlation IDs, and consistent field naming conventions.

Log groups, log streams, metric filters, subscription filters, and Logs Insights queries.

Ship logs from Lambda, EC2, and ECS to Amazon OpenSearch for full-text search and dashboards.

Collect and route logs from containers and EC2 instances to CloudWatch or OpenSearch.

Traces, spans, context propagation, sampling, and the OpenTelemetry data model.

Instrument Lambda, ECS, and API Gateway; read service maps and analyse trace segments.

Collect traces and metrics with the AWS-supported OpenTelemetry distribution.

Use trace IDs in logs, link metrics to traces, and build a unified observability workflow.

Alert on symptoms not causes, avoid alert fatigue, and ensure every alert has a runbook.

Composite alarms, anomaly detection alarms, metric math alarms, and SNS integration.

Route, deduplicate, group, and silence Prometheus alerts; integrate with PagerDuty and Slack.

Design fair rotations, primary/secondary escalation, and override schedules in PagerDuty or OpsGenie.

Detection, triage, response, mitigation, resolution, and follow-up the five phases of an incident.

Coordinate responders, manage communication, and make decisions under pressure.

Define SEV1–SEV4 (or P1–P4) with clear criteria for customer impact and response SLAs.

Internal Slack war rooms, external status pages (Statuspage.io), and stakeholder updates.

Centralise operational issues, correlate findings from GuardDuty, Config, and CloudWatch.

Focus on systemic failures, not individual mistakes psychological safety is critical.

Impact summary, timeline, root cause, contributing factors, action items, and lessons learned.

5 Whys, fishbone diagrams, and fault tree analysis for structured root cause investigation.

Assign owners, set deadlines, and track PIR action items to completion in Jira or Linear.

Hypothesis-driven experimentation, blast radius control, and GameDay practices.

Run controlled fault injection experiments on EC2, ECS, EKS, RDS, and more with AWS FIS.

CNCF chaos engineering framework for Kubernetes pod kill, network latency, and disk fill experiments.

Structured team exercises that simulate real outage scenarios to validate runbooks and incident response.

Manual, repetitive, automatable work that scales with service growth and provides no enduring value.

Track toil as a percentage of engineer time the SRE goal is to keep toil below 50%.

Automate tickets, self-service provisioning, runbook automation, and event-driven remediation.

Create SSM Automation runbooks to execute common operational tasks without manual intervention.

Write, plan, and apply infrastructure changes safely remote state, workspaces, and drift detection.

Manage AWS resources declaratively with CloudFormation stacks or CDK constructs.

Idempotent server configuration, patching automation, and drift remediation with Ansible.

Lint and test infrastructure code with tflint, Checkov, cfn-lint, and Terratest.

Rolling, blue/green, canary, and feature flag deployments risk profiles and rollback mechanisms.

Automate EC2, Lambda, and ECS deployments with health checks and automatic rollback.

Automate operational workflows AMI baking, certificate rotation, and compliance checks.

Track deployment frequency, lead time, change failure rate, and MTTR (DORA metrics).

Analyse traffic trends, seasonality, and growth projections to model future resource needs.

Simulate production load with k6, Locust, or Artillery to validate capacity assumptions.

ML-based right-sizing recommendations for EC2, Lambda, ECS, and EBS volumes.

Configure dynamic and predictive scaling policies to handle demand spikes automatically.

Prevent cascading failures by temporarily blocking requests to a failing downstream service.

Retry transient failures with jitter and backoff to avoid thundering herd problems.

Isolate failures by partitioning resources separate thread pools, queues, or service instances.

Protect services from overload using token bucket or leaky bucket rate limiting.

Serve reduced functionality rather than full failure when dependencies are unavailable.

Apply least privilege to automation scripts, Lambda functions, and EC2 instance profiles.

Retrieve credentials at runtime from AWS Secrets Manager or Parameter Store no hardcoded secrets.

Integrate GuardDuty findings into incident response workflows and PagerDuty escalation.

Automate patching with AWS Systems Manager Patch Manager and track CVEs with Amazon Inspector.

Use AWS Cost Anomaly Detection to alert on unexpected spend spikes automatically.

Tag all resources consistently to enable service-level cost attribution and showback.

Build fault-tolerant workloads on Spot with interruption handlers and mixed instance policies.

Set cost and usage budgets with alert thresholds to catch runaway workloads early.

Clear, step-by-step procedures with expected outcomes, rollback steps, and escalation paths.

Execute runbook steps remotely across EC2 fleets without SSH using SSM Run Command.

Document why systems are designed the way they are to aid future incident diagnosis.

Gate new services going on-call with a checklist: SLOs defined, runbooks written, dashboards built.

Deploy the OTel Collector as a sidecar or DaemonSet to pipeline traces, metrics, and logs.

Profile CPU and memory in production with Pyroscope or Amazon CodeGuru Profiler.

Capture frontend performance from real user sessions with Amazon CloudWatch RUM.

Simulate user journeys 24/7 with CloudWatch Synthetics canaries to detect outages proactively.

Embedded vs centralised vs consulting SRE choosing the right model for your organisation.

Deployment frequency, lead time, change failure rate, and MTTR as engineering performance indicators.

A structured checklist every new service must pass before receiving SRE support and going on-call.

Assess and grow your SRE practice across five dimensions: monitoring, incident response, toil, SLOs, and culture.