--- title: "Site Reliability Engineer Beginner to Expert" description: "A comprehensive roadmap to master Site Reliability Engineering from Linux and networking fundamentals to advanced SLOs, observability, incident management, and automation on AWS." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/roadmaps/post/site-reliability-engineer-roadmap --- # Site Reliability Engineer Beginner to Expert This roadmap takes you from the fundamentals of Linux and systems thinking through to advanced observability, chaos engineering, and SRE organisational culture. Each stage builds on the last master reliability principles and AWS tooling together, treating every production system as an opportunity to learn, automate, and improve. The goal is not zero failures, but fast recovery and continuous reduction of their impact. ## Roadmap ### Stage 1: SRE Foundations Understand what Site Reliability Engineering is, where it came from, and how it differs from traditional ops. #### What is SRE? Google's SRE origin, the SRE vs DevOps distinction, and the core principles of reliability engineering. - [Google SRE Book](https://sre.google/sre-book/introduction/) #### The SRE Role & Responsibilities On-call engineering, toil reduction, capacity planning, and the balance between feature velocity and reliability. #### Reliability as a Feature Why reliability is a product requirement, not just an operational concern. #### The SRE Workbook _(recommended)_ Practical guidance and case studies that complement the original SRE book. - [The SRE Workbook](https://sre.google/workbook/table-of-contents/) ### Stage 2: Linux & Systems Fundamentals Master the operating system skills every SRE depends on daily. #### Linux Process Management ps, top, htop, kill, nice, systemd units, and process states. - [Linux man pages](https://man7.org/linux/man-pages/) #### File Systems & I/O inodes, mount points, df/du, lsof, and diagnosing disk I/O bottlenecks with iostat. #### Memory & CPU Analysis vmstat, free, /proc/meminfo, perf, and understanding OOM killer behaviour. #### Kernel & System Calls _(recommended)_ strace, ltrace, dmesg, and reading kernel logs for low-level system debugging. ### Stage 3: Networking Fundamentals for SREs Understand the network layers that underpin distributed systems and cloud infrastructure. #### TCP/IP & the OSI Model How packets travel from source to destination, TCP handshake, and connection states. - [Cloudflare Learning: OSI Model](https://www.cloudflare.com/learning/ddos/glossary/open-systems-interconnection-model-osi/) #### DNS Deep Dive Resolution chain, TTL, dig/nslookup, CNAME vs A records, and DNS propagation issues. #### HTTP/HTTPS & TLS Request lifecycle, status codes, headers, TLS handshake, and certificate chains. #### Network Troubleshooting Tools curl, traceroute, mtr, tcpdump, netstat/ss, and Wireshark for packet analysis. ### Stage 4: Shell Scripting & Automation Basics Write Bash scripts to automate operational tasks and reduce manual toil. #### Bash Scripting Variables, conditionals, loops, functions, error handling (set -euo pipefail), and exit codes. - [Bash Guide](https://mywiki.wooledge.org/BashGuide) #### Text Processing grep, awk, sed, cut, sort, uniq, and jq for structured log and JSON processing. #### Cron & Scheduling crontab syntax, systemd timers, and avoiding overlapping job execution. #### Python for SRE _(recommended)_ Scripting operational tasks in Python requests, subprocess, boto3, and argparse. ### Stage 5: Version Control & GitOps Practices Use Git to version infrastructure, runbooks, and automation code. #### Git for Operations Branching strategies, pull requests, tagging releases, and git blame for audit trails. - [Pro Git Book](https://git-scm.com/book/en/v2) #### GitOps Principles Git as the single source of truth declarative config, automated reconciliation, and self-healing. #### Runbooks as Code _(recommended)_ Store operational runbooks in Git alongside the systems they describe for version control and discoverability. ### Stage 6: Containerisation & Orchestration Operate containerised workloads and understand the platforms SREs are responsible for. #### Docker Operations Container lifecycle, resource limits (--cpus, --memory), log drivers, and health checks. - [Docker Docs](https://docs.docker.com/) #### Kubernetes for SREs Pod lifecycle, rolling deployments, resource requests/limits, liveness and readiness probes. - [Kubernetes Docs](https://kubernetes.io/docs/home/) #### Amazon EKS Operations Managed node groups, Fargate profiles, cluster upgrades, and EKS add-ons. #### Horizontal Pod Autoscaler & KEDA _(recommended)_ Scale workloads based on CPU, memory, and custom metrics from SQS or Prometheus. ### Stage 7: Service Level Indicators (SLIs) Identify and measure the metrics that best represent user experience for your services. #### What are SLIs? SLIs are quantitative measures of service behaviour availability, latency, throughput, and error rate. - [Google SRE: SLIs](https://sre.google/sre-book/service-level-objectives/) #### Choosing the Right SLIs User-centric SLIs vs system metrics measure what the user experiences, not what is easy to collect. #### The Four Golden Signals Latency, traffic, errors, and saturation Google's framework for monitoring any system. #### RED & USE Methods Rate/Errors/Duration for request-driven services; Utilisation/Saturation/Errors for resources. ### Stage 8: Service Level Objectives (SLOs) Set meaningful reliability targets and use them to make data-driven engineering decisions. #### Defining SLOs SLO = SLI over a time window with a target threshold how to set achievable, meaningful targets. - [Google SRE: SLOs](https://sre.google/workbook/implementing-slos/) #### Error Budgets Error budget = 1 - SLO target. Use it to balance reliability work vs feature velocity. #### Error Budget Policies Define what happens when the budget is consumed feature freeze, reliability sprints, or escalation. #### SLO Review Cadence _(recommended)_ Regularly review and adjust SLOs as user expectations and system capabilities evolve. ### Stage 9: Service Level Agreements (SLAs) Understand the contractual reliability commitments and how SLOs feed into them. #### SLA vs SLO vs SLI The hierarchy: SLI measures it, SLO targets it internally, SLA commits to it externally. - [Google SRE: Terminology](https://sre.google/sre-book/service-level-objectives/#indicators-in-practice) #### Setting Conservative SLAs SLAs should be set tighter than SLOs to provide a safety buffer before breaching customer commitments. #### SLA Breach Consequences _(recommended)_ Financial penalties, service credits, and reputational impact of breached SLAs. ### Stage 10: Metrics & Prometheus Collect and query time-series metrics to understand system health and trends. #### Prometheus Architecture Scrape model, targets, exporters, the Prometheus data model, and labels. - [Prometheus Docs](https://prometheus.io/docs/introduction/overview/) #### PromQL Instant vectors, range vectors, rate(), irate(), histogram_quantile(), and recording rules. #### Alerting Rules Write Prometheus alerting rules with for clauses, severity labels, and runbook annotations. #### Amazon Managed Service for Prometheus _(recommended)_ Serverless Prometheus-compatible monitoring at scale on AWS with AMP. - [AMP Docs](https://docs.aws.amazon.com/prometheus/latest/userguide/what-is-Amazon-Managed-Service-Prometheus.html) ### Stage 11: Dashboards & Visualisation Build meaningful dashboards that surface actionable information for on-call engineers. #### Grafana Fundamentals Panels, variables, annotations, alerts, and dashboard-as-code with Grafonnet/Jsonnet. - [Grafana Docs](https://grafana.com/docs/grafana/latest/) #### Amazon Managed Grafana _(recommended)_ Fully managed Grafana on AWS with native CloudWatch, AMP, and X-Ray data sources. - [Amazon Managed Grafana Docs](https://docs.aws.amazon.com/grafana/latest/userguide/what-is-Amazon-Managed-Grafana.html) #### Dashboard Design Principles Design dashboards for on-call use clear signal hierarchy, avoid noise, and link to runbooks. #### Amazon CloudWatch Dashboards Build operational dashboards with CloudWatch metrics, alarms, and Logs Insights widgets. ### Stage 12: Logging & Log Management Collect, centralise, and query logs to diagnose incidents and understand system behaviour. #### Structured Logging JSON log format, log levels, correlation IDs, and consistent field naming conventions. - [AWS Logging Best Practices](https://docs.aws.amazon.com/prescriptive-guidance/latest/logging-monitoring-for-application-owners/logging-best-practices.html) #### Amazon CloudWatch Logs Log groups, log streams, metric filters, subscription filters, and Logs Insights queries. #### Centralised Logging with OpenSearch _(recommended)_ Ship logs from Lambda, EC2, and ECS to Amazon OpenSearch for full-text search and dashboards. #### AWS Fluent Bit & Fluentd _(recommended)_ Collect and route logs from containers and EC2 instances to CloudWatch or OpenSearch. ### Stage 13: Distributed Tracing Track requests across microservices to diagnose latency and pinpoint failure sources. #### Distributed Tracing Concepts Traces, spans, context propagation, sampling, and the OpenTelemetry data model. - [OpenTelemetry Docs](https://opentelemetry.io/docs/) #### AWS X-Ray Instrument Lambda, ECS, and API Gateway; read service maps and analyse trace segments. - [AWS X-Ray Docs](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html) #### AWS Distro for OpenTelemetry (ADOT) _(recommended)_ Collect traces and metrics with the AWS-supported OpenTelemetry distribution. #### Correlating Logs, Metrics & Traces Use trace IDs in logs, link metrics to traces, and build a unified observability workflow. ### Stage 14: Alerting & On-Call Management Design actionable alerting and sustainable on-call rotations that minimise engineer burnout. #### Alerting Best Practices Alert on symptoms not causes, avoid alert fatigue, and ensure every alert has a runbook. - [Google SRE: Alerting](https://sre.google/sre-book/practical-alerting/) #### Amazon CloudWatch Alarms Composite alarms, anomaly detection alarms, metric math alarms, and SNS integration. #### Alertmanager Route, deduplicate, group, and silence Prometheus alerts; integrate with PagerDuty and Slack. #### On-Call Rotations & Escalation Policies Design fair rotations, primary/secondary escalation, and override schedules in PagerDuty or OpsGenie. ### Stage 15: Incident Management Detect, respond to, and resolve incidents quickly while communicating clearly with stakeholders. #### Incident Lifecycle Detection, triage, response, mitigation, resolution, and follow-up the five phases of an incident. - [PagerDuty Incident Response Guide](https://response.pagerduty.com/) #### Incident Commander Role Coordinate responders, manage communication, and make decisions under pressure. #### Severity Levels Define SEV1–SEV4 (or P1–P4) with clear criteria for customer impact and response SLAs. #### Incident Communication Internal Slack war rooms, external status pages (Statuspage.io), and stakeholder updates. #### AWS Systems Manager OpsCenter _(recommended)_ Centralise operational issues, correlate findings from GuardDuty, Config, and CloudWatch. ### Stage 16: Post-Incident Reviews (PIRs) Learn from incidents systematically to prevent recurrence and build institutional knowledge. #### Blameless Post-Mortems Focus on systemic failures, not individual mistakes psychological safety is critical. - [Google SRE: Post-Mortems](https://sre.google/sre-book/postmortem-culture/) #### Post-Mortem Template Impact summary, timeline, root cause, contributing factors, action items, and lessons learned. #### Root Cause Analysis (RCA) Techniques 5 Whys, fishbone diagrams, and fault tree analysis for structured root cause investigation. #### Action Item Tracking Assign owners, set deadlines, and track PIR action items to completion in Jira or Linear. ### Stage 17: Chaos Engineering Proactively inject failures into production systems to discover weaknesses before they cause incidents. #### Chaos Engineering Principles Hypothesis-driven experimentation, blast radius control, and GameDay practices. - [Principles of Chaos Engineering](https://principlesofchaos.org/) #### AWS Fault Injection Service (FIS) Run controlled fault injection experiments on EC2, ECS, EKS, RDS, and more with AWS FIS. - [AWS FIS Docs](https://docs.aws.amazon.com/fis/latest/userguide/what-is.html) #### LitmusChaos on EKS _(recommended)_ CNCF chaos engineering framework for Kubernetes pod kill, network latency, and disk fill experiments. #### GameDays _(recommended)_ Structured team exercises that simulate real outage scenarios to validate runbooks and incident response. ### Stage 18: Toil Identification & Reduction Identify, measure, and systematically eliminate operational toil to free up engineering capacity. #### What is Toil? Manual, repetitive, automatable work that scales with service growth and provides no enduring value. - [Google SRE: Eliminating Toil](https://sre.google/sre-book/eliminating-toil/) #### Measuring Toil Track toil as a percentage of engineer time the SRE goal is to keep toil below 50%. #### Toil Reduction Strategies Automate tickets, self-service provisioning, runbook automation, and event-driven remediation. #### AWS Systems Manager Automation Create SSM Automation runbooks to execute common operational tasks without manual intervention. - [SSM Automation Docs](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-automation.html) ### Stage 19: Infrastructure as Code for SREs Manage infrastructure reliably and repeatably using code the foundation of operational consistency. #### Terraform for SREs Write, plan, and apply infrastructure changes safely remote state, workspaces, and drift detection. - [Terraform Docs](https://developer.hashicorp.com/terraform/docs) #### AWS CloudFormation & CDK Manage AWS resources declaratively with CloudFormation stacks or CDK constructs. #### Ansible for Configuration Management _(recommended)_ Idempotent server configuration, patching automation, and drift remediation with Ansible. #### IaC Testing & Validation _(recommended)_ Lint and test infrastructure code with tflint, Checkov, cfn-lint, and Terratest. ### Stage 20: CI/CD Reliability Make deployment pipelines reliable, observable, and safe to run at high frequency. #### Deployment Strategies Rolling, blue/green, canary, and feature flag deployments risk profiles and rollback mechanisms. - [AWS Deployment Strategies](https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/deployment-strategies.html) #### AWS CodeDeploy Automate EC2, Lambda, and ECS deployments with health checks and automatic rollback. #### GitHub Actions for SRE _(recommended)_ Automate operational workflows AMI baking, certificate rotation, and compliance checks. #### Deployment Observability Track deployment frequency, lead time, change failure rate, and MTTR (DORA metrics). ### Stage 21: Capacity Planning Forecast resource needs and provision ahead of demand to prevent capacity-driven incidents. #### Demand Forecasting Analyse traffic trends, seasonality, and growth projections to model future resource needs. - [Google SRE: Forecasting](https://sre.google/sre-book/forecasting/) #### Load Testing Simulate production load with k6, Locust, or Artillery to validate capacity assumptions. #### AWS Compute Optimizer ML-based right-sizing recommendations for EC2, Lambda, ECS, and EBS volumes. - [AWS Compute Optimizer Docs](https://docs.aws.amazon.com/compute-optimizer/latest/ug/what-is-compute-optimizer.html) #### Auto Scaling & Predictive Scaling Configure dynamic and predictive scaling policies to handle demand spikes automatically. ### Stage 22: Reliability Patterns Apply proven engineering patterns to make distributed systems resilient to partial failures. #### Circuit Breaker Pattern Prevent cascading failures by temporarily blocking requests to a failing downstream service. - [AWS: Circuit Breaker](https://aws.amazon.com/blogs/compute/using-the-circuit-breaker-pattern-with-aws-step-functions-and-amazon-dynamodb/) #### Retry & Exponential Backoff Retry transient failures with jitter and backoff to avoid thundering herd problems. #### Bulkhead Pattern _(recommended)_ Isolate failures by partitioning resources separate thread pools, queues, or service instances. #### Rate Limiting & Throttling Protect services from overload using token bucket or leaky bucket rate limiting. #### Graceful Degradation Serve reduced functionality rather than full failure when dependencies are unavailable. ### Stage 23: Security for SREs Integrate security into reliability practices secure systems fail less and recover faster. #### Least Privilege & IAM Roles Apply least privilege to automation scripts, Lambda functions, and EC2 instance profiles. - [AWS IAM Best Practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) #### Secrets Management Retrieve credentials at runtime from AWS Secrets Manager or Parameter Store no hardcoded secrets. #### AWS GuardDuty for SREs _(recommended)_ Integrate GuardDuty findings into incident response workflows and PagerDuty escalation. #### Vulnerability Management Automate patching with AWS Systems Manager Patch Manager and track CVEs with Amazon Inspector. ### Stage 24: Cost Reliability Prevent cost incidents and ensure financial reliability alongside technical reliability. #### Cost Anomaly Detection Use AWS Cost Anomaly Detection to alert on unexpected spend spikes automatically. - [AWS Cost Anomaly Detection](https://docs.aws.amazon.com/cost-management/latest/userguide/getting-started-ad.html) #### Tagging & Cost Attribution Tag all resources consistently to enable service-level cost attribution and showback. #### Spot Instance Reliability _(recommended)_ Build fault-tolerant workloads on Spot with interruption handlers and mixed instance policies. #### AWS Budgets & Alerts Set cost and usage budgets with alert thresholds to catch runaway workloads early. ### Stage 25: Runbook & Documentation Culture Build a culture of operational documentation that empowers any engineer to respond to incidents. #### Writing Effective Runbooks Clear, step-by-step procedures with expected outcomes, rollback steps, and escalation paths. - [PagerDuty: Runbook Guide](https://www.pagerduty.com/resources/learn/what-is-a-runbook/) #### AWS Systems Manager Run Command Execute runbook steps remotely across EC2 fleets without SSH using SSM Run Command. #### Architecture Decision Records (ADRs) _(recommended)_ Document why systems are designed the way they are to aid future incident diagnosis. #### Operational Readiness Reviews (ORRs) Gate new services going on-call with a checklist: SLOs defined, runbooks written, dashboards built. ### Stage 26: Advanced Observability Go beyond basic monitoring to build a deep, unified view of complex distributed systems. #### OpenTelemetry Collector Deploy the OTel Collector as a sidecar or DaemonSet to pipeline traces, metrics, and logs. - [OTel Collector Docs](https://opentelemetry.io/docs/collector/) #### Continuous Profiling _(recommended)_ Profile CPU and memory in production with Pyroscope or Amazon CodeGuru Profiler. #### Real User Monitoring (RUM) _(recommended)_ Capture frontend performance from real user sessions with Amazon CloudWatch RUM. - [CloudWatch RUM Docs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-RUM.html) #### Synthetic Monitoring Simulate user journeys 24/7 with CloudWatch Synthetics canaries to detect outages proactively. ### Stage 27: SRE Maturity & Org Culture Grow SRE practices across the organisation and measure the maturity of your reliability programme. #### SRE Team Models Embedded vs centralised vs consulting SRE choosing the right model for your organisation. - [Google SRE: Organisational Change](https://sre.google/workbook/organizational-change/) #### DORA Metrics Deployment frequency, lead time, change failure rate, and MTTR as engineering performance indicators. #### Production Readiness Reviews A structured checklist every new service must pass before receiving SRE support and going on-call. #### SRE Maturity Model _(recommended)_ Assess and grow your SRE practice across five dimensions: monitoring, incident response, toil, SLOs, and culture.