---
title: "Production Backend: Scaling, Monitoring & Reliability"
description: "Run production backend systems: scaling patterns, monitoring, error handling, logging, resilience, capacity planning, and incident response."
author: "Mohammad Abu Mattar"
canonical: https://mkabumattar.com/quizzes/post/production-backend-scaling-quiz
---

# Production Backend: Scaling, Monitoring & Reliability

Welcome to the "Production Backend: Scaling, Monitoring & Reliability" quiz! This quiz tests your knowledge of key concepts and best practices for running production backend systems at scale. You'll be challenged on topics like scaling patterns, monitoring strategies, error handling, resilience techniques, capacity planning, and incident response. Each question includes detailed explanations to help you learn from any mistakes. Good luck!

## Questions

### 1. What is horizontal scaling?

- **Add more server instances to distribute incoming load** ✅
  - Horizontal: load balancer needed. Better availability. More complex.
- Upgrade existing hardware with more CPU and RAM resources
  - This describes vertical scaling (scaling up), not horizontal (scaling out).
- Optimize application code to improve per-request efficiency
  - This is performance tuning, not a scaling strategy.
- Migrate local storage to a centralized cloud-based system
  - This is infrastructure migration, not a scaling pattern.

**Hint:** Think about many servers.

### 2. What is connection pooling?

- **Maintain a cache of open database connections for reuse** ✅
  - Connection pooling: reduces latency, prevents connection exhaustion. PgBouncer, HikariCP.
- Create a new database connection for every incoming user
  - This is the opposite of pooling and causes high overhead.
- Synchronize multiple database instances for high availability
  - This describes database replication or clustering.
- Encrypt all traffic between the application and database
  - This describes Transport Layer Security (TLS).

**Hint:** Think about reusing database connections.

### 3. What is a message queue?

- **Buffer tasks between services for asynchronous processing** ✅
  - Message queue: RabbitMQ, Kafka. Improves resilience, handles spikes.
- Sort database records based on their arrival timestamp
  - This describes a database sorting operation or index.
- Manage the priority of incoming HTTP requests at the edge
  - This describes a Load Balancer or API Gateway function.
- Verify the identity of users before they access a service
  - This describes an authentication or authorization service.

**Hint:** Think about asynchronous processing.

### 4. What is rate limiting in production?

- **Restrict request volume per user to prevent system abuse** ✅
  - Rate limiting: graceful degradation, fairness. Token bucket algorithm common.
- Maximize the data throughput of a specific network interface
  - This is throughput optimization, not a restrictive limit.
- Calculate the average response time for all API endpoints
  - This describes latency monitoring or SLI measurement.
- Allocate specific CPU time to different background processes
  - This is OS scheduling or resource contention management.

**Hint:** Think about protecting system.

### 5. What is the circuit breaker pattern?

- **Prevent calls to a failing service to allow it to recover** ✅
  - Circuit breaker: prevents cascading failures. States: Closed, Open, Half-open.
- Retry every failed request immediately until it succeeds
  - This is a simple retry policy, which can cause "retry storms."
- Distribute requests across multiple healthy server nodes
  - This describes the core function of a Load Balancer.
- Log every service error into a centralized storage system
  - This describes centralized error logging or aggregation.

**Hint:** Think about handling failing services.

### 6. What is an SLA (Service Level Agreement)?

- **External contract defining uptime goals like 99.9% availability** ✅
  - SLA: business commitment. Backed by compensation. Affects infrastructure cost.
- Internal metric used by developers to track system performance
  - This describes a Service Level Indicator (SLI).
- Software license agreement governing the use of open-source tools
  - SLA is about performance reliability, not code licensing.
- Security protocol used to authenticate third-party API providers
  - SLA is a business agreement, not a technical security protocol.

**Hint:** Think about availability commitment.

### 7. What is an SLO (Service Level Objective)?

- **Internal performance target used to guide system reliability** ✅
  - SLO: measurable goal. Drives architecture. Error budget based on SLO.
- The actual measured availability of a service over 30 days
  - This is the SLI (Indicator) result, not the target (Objective).
- Legally binding document that specifies financial penalties
  - This describes the SLA (Agreement).
- Maximum number of concurrent users a single instance supports
  - This describes the saturation point or capacity limit.

**Hint:** Think about internal performance target.

### 8. What is an error budget?

- **Calculated allowable downtime used to prioritize new changes** ✅
  - Error budget: if 99.9% SLO, ~43 min/month allowed downtime. Guides risk-taking.
- Financial allocation for repairing production infrastructure
  - This is a capital/operational expenditure budget.
- Total number of bugs allowed in a release during QA testing
  - Error budgets relate to uptime/reliability, not bug counts.
- Threshold of failed requests before an alert is triggered
  - This describes an alert threshold or monitor.

**Hint:** Think about acceptable downtime.

### 9. What is incident response?

- **Structured process for detecting and resolving service outages** ✅
  - Incident response: on-call, runbooks, communication. Minimizes MTTR.
- Automated scaling of resources during a sudden traffic spike
  - This is auto-scaling, which is a proactive/reactive system response.
- Customer support workflow for handling user feature requests
  - Incident response handles system failures, not feature requests.
- The initial planning phase of a software development lifecycle
  - This is requirements gathering or design, not operations.

**Hint:** Think about handling outages.

### 10. What is MTTR (Mean Time To Recover)?

- **Average time required to fix a service after an issue starts** ✅
  - MTTR: lower is better. Improved by runbooks, monitoring, team practices.
- Average time between two consecutive system failure events
  - This describes MTBF (Mean Time Between Failures).
- Total time spent on developing new reliability features
  - This is a measure of engineering effort, not recovery speed.
- Minimum time a server can run before a required reboot
  - This is related to uptime or stability, not recovery.

**Hint:** Think about recovery speed.

### 11. What is chaos engineering?

- **Inject intentional system failures to test resilience** ✅
  - Chaos engineering: Netflix Chaos Monkey. Discovers weaknesses before prod outages.
- Run production systems without any monitoring or logs
  - This is simply poor operational practice, not an engineering discipline.
- Randomly update dependencies to find potential bugs
  - This is exploratory testing, not chaos engineering.
- Allowing developers to push code without a review process
  - This describes a lack of quality control or deployment guardrails.

**Hint:** Think about intentional failures.

### 12. What is graceful degradation?

- **Turn off non-core features to keep the primary service live** ✅
  - Graceful degradation: serve stale cache, reduce computation, limit features. Improves UX.
- Permanently remove old features to simplify the codebase
  - This is technical debt reduction or feature deprecation.
- Automatically restart servers when they reach high latency
  - This is a self-healing mechanism or aggressive restart policy.
- Reduce the security requirements for internal service traffic
  - This is a security risk and is not related to availability patterns.

**Hint:** Think about reduced functionality under load.

### 13. What is bulkhead isolation?

- **Partition system resources to prevent total failure spread** ✅
  - Bulkhead: if one fails, others unaffected. Prevents total system failure.
- Replicate data across different physical disk drives
  - This describes RAID or physical storage redundancy.
- Encrypt data stored at rest to prevent unauthorized access
  - This is a data security measure, not failure isolation.
- Use a single large server to handle all service components
  - This is a monolithic approach and creates a single point of failure.

**Hint:** Think about fault isolation.

### 14. What is a canary deployment?

- **Release new code to a small group of users before everyone** ✅
  - Canary: 5% → 25% → 100%. Catches issues early. Low risk.
- Deploy the new version to all production servers at once
  - This describes an "all-at-once" or "recreate" deployment.
- Run the old and new versions in a simulated environment
  - This is staging or shadow testing, not a live deployment.
- Switch all traffic from the old cluster to the new cluster
  - This describes a Blue-Green deployment strategy.

**Hint:** Think about risky rollouts.

### 15. What is structured logging?

- **Use JSON or key-value pairs to make logs machine-parsable** ✅
  - Structured logging: Winston, Pino, Bunyan. Enables filtering, aggregation. Better than free-form strings.
- Write logs in chronological order within a text file
  - This is standard sequential logging, not necessarily structured.
- Categorize logs into simple levels like INFO, WARN, and ERROR
  - This is log leveling, which can exist in unstructured logs too.
- Manually review every log entry at the end of the day
  - This is manual auditing, not a technical format for logs.

**Hint:** Think about machine-readable logs.

### 16. What is log aggregation?

- **Collect logs from multiple sources into a central search tool** ✅
  - Log aggregation: single source of truth. Enables dashboards, alerts, debugging. Critical for distributed systems.
- Delete old logs automatically to save on storage costs
  - This describes log rotation or retention policies.
- Convert raw logs into a summarized PDF for management
  - This is reporting, not the operational process of aggregation.
- Compress log files before they are sent to long-term storage
  - This is log compression, a sub-step of data management.

**Hint:** Think about collecting logs from many services.

### 17. What is distributed tracing?

- **Track a request as it flows through various microservices** ✅
  - Distributed tracing: Jaeger, Datadog, New Relic. Identifies bottlenecks, service dependencies.
- Map the physical location of servers across the globe
  - This is geographic mapping of infrastructure.
- Monitor the bandwidth usage of all internal load balancers
  - This describes network performance monitoring.
- Follow the career path of developers across different teams
  - This is unrelated to technical system observability.

**Hint:** Think about tracking requests across services.

### 18. What is a liveness vs readiness probe (Kubernetes)?

- **Liveness ensures the app runs; Readiness ensures it can take traffic** ✅
  - Liveness: /healthz. Readiness: /ready. Both critical. Different failure modes.
- Liveness tracks CPU usage; Readiness tracks memory usage
  - These are resource metrics, not health probes.
- Liveness is for internal users; Readiness is for external users
  - Probes manage container state, not user categories.
- Liveness measures latency; Readiness measures throughput
  - These are performance indicators, not state probes.

**Hint:** Think about health checks.

### 19. What is a load balancer algorithm?

- **Method like Round-robin used to distribute traffic to nodes** ✅
  - Algorithms: Round-robin (simple), Least-conn (fair), IP-hash (sticky), Weighted. Choice affects UX.
- Encryption process that protects data between the LB and user
  - This describes SSL/TLS termination, not a distribution algorithm.
- Decision tree used to determine if a request should be blocked
  - This describes a Web Application Firewall (WAF) rule.
- Compression routine used to shrink the size of HTTP headers
  - This is header compression (like HPACK), not traffic distribution.

**Hint:** Think about distributing requests.

### 20. What is a database read replica?

- **Copy of a database used to handle query traffic and offload primary** ✅
  - Read replicas: improves read throughput. Eventual consistency risk. PostgreSQL, MySQL, RDS support.
- Complete backup of the database stored in a different region
  - This describes a disaster recovery backup or snapshot.
- Temporary table used to store intermediate calculation results
  - This describes a materialized view or temporary table.
- Write-only instance used to record application audit logs
  - Replicas are for reading; writes still go to the primary node.

**Hint:** Think about scaling read traffic.

### 21. What is an auto-scaling policy?

- **Set of rules to add or remove instances based on live metrics** ✅
  - Auto-scaling: target tracking, step scaling. Reduces cost, handles spikes. Prevents over-provisioning.
- Manual schedule used to reboot servers every weekend
  - Auto-scaling is automated and metric-driven, not a manual schedule.
- Strategy for increasing the pricing tier of a cloud account
  - This is account management, not infrastructure scaling.
- Policy that limits the maximum bandwidth a user can consume
  - This describes bandwidth throttling or QoS.

**Hint:** Think about automatic capacity adjustment.

### 22. What is backpressure handling?

- **Signal the data source to slow down when the receiver is full** ✅
  - Backpressure: prevent memory explosion, data loss. RxJS, streams support. Better than dropping requests.
- Increase the frequency of requests to bypass slow network nodes
  - This would worsen the problem by increasing congestion.
- Switch to a different server when the current one becomes slow
  - This is failover or load balancing, not backpressure.
- Store all incoming data in a permanent database indefinitely
  - This is persistent storage and does not manage flow control.

**Hint:** Think about when downstreams slow down.

### 23. What impacts cost in scaling?

- **Usage of compute, data transfer, storage, and redundant nodes** ✅
  - Cost factors: underutilized instances waste money. Reserved instances, spot instances, rightsizing reduce cost.
- Total number of lines of code written by the engineering team
  - Engineering time is a cost, but it is not a direct scaling cost factor.
- Frequency of code commits made to the main production branch
  - This affects CI/CD costs but not the cost of scaling the running system.
- Complexity of the CSS used in the frontend user interface
  - Frontend complexity has negligible impact on backend scaling costs.

**Hint:** Think about infrastructure expenses.

### 24. What is blue-green deployment?

- **Maintain two identical environments and switch traffic between them** ✅
  - Blue-green: instant rollback. Requires double resources. Less risky than canary.
- Color-code different versions of the API for developer testing
  - This is a visual organization technique, not a deployment strategy.
- Deploying code updates on Tuesdays and Thursdays only
  - This is a release schedule, not a technical deployment pattern.
- Splitting the database into two halves based on the user ID
  - This describes database sharding or partitioning.

**Hint:** Think about zero-downtime releases.

### 25. What is a feature flag?

- **Remote toggle to enable or disable code without a new deploy** ✅
  - Feature flags: LaunchDarkly, Unleash. Enable gradual rollout without redeployment. A/B testing.
- Special icon used in the UI to highlight new application updates
  - This is a UI design element, not a backend deployment tool.
- Code comment that marks a section for future development
  - This describes a "TODO" comment or placeholder.
- Strict rule that prevents any new features from being added
  - This describes a "feature freeze" during a release cycle.

**Hint:** Think about gradual rollout control.

### 26. What is observability vs monitoring?

- **Monitoring alerts on symptoms; Observability enables deep debugging** ✅
  - Observability: logs, metrics, traces (three pillars). Answers "what happened?" not just if threshold crossed.
- Monitoring is for hardware; Observability is for software
  - Both concepts apply to the entire infrastructure and application stack.
- Monitoring is automatic; Observability requires manual data entry
  - Both are technical processes that should be automated.
- Monitoring uses logs; Observability only uses metrics and traces
  - Observability requires all three: logs, metrics, and traces.

**Hint:** Think about understanding systems.

### 27. What is an alerting strategy?

- **Notify on actionable symptoms instead of noisy low-level events** ✅
  - Alerting: target user impact, not low-level metrics. Reduces alert fatigue. On-call rotation.
- Send an email to the whole company for every server warning
  - This leads to extreme alert fatigue and ignored notifications.
- Only trigger alerts during standard business hours (9am to 5pm)
  - Production systems require 24/7 monitoring and alerting.
- Log every alert to a local text file without sending notifications
  - This makes alerts useless for real-time incident response.

**Hint:** Think about notifying about problems.

### 28. What is eventual consistency?

- **Distributed data model where replicas will eventually align** ✅
  - Eventual consistency: improves availability, reduces latency. DynamoDB, Cassandra. Trade strict consistency for resilience.
- Requirement that all database writes must succeed or fail together
  - This describes ACID atomicity or strong consistency.
- System state where data is deleted after it becomes one year old
  - This describes a data expiration or retention policy.
- Configuration that ensures every server has the exact same hardware
  - Consistency in scaling refers to data state, not hardware specs.

**Hint:** Think about data across replicas.

### 29. What is a cache invalidation strategy?

- **Logic used to decide when to remove or refresh cached data** ✅
  - Invalidation challenge: TTL (Redis), event-driven (message queue), manual (admin). Depends on use case.
- Process of adding more RAM to the caching server cluster
  - This is scaling the cache, not an invalidation strategy.
- Technique for encrypting cache keys to prevent data leakage
  - This is a security measure for the caching layer.
- Method for ensuring that every user gets a unique cache key
  - This relates to cache key design and partitioning.

**Hint:** Think about stale data.

### 30. What is security hardening in production?

- **Apply multiple layers of protection to minimize the attack surface** ✅
  - Hardening: HashiCorp Vault for secrets, security groups (AWS), mTLS, encryption in transit/rest. Defense in depth.
- Physically lock the server racks in a secure data center
  - This is physical security, only one small part of hardening.
- Require all developers to change their passwords every 24 hours
  - This is an extreme password policy, not comprehensive hardening.
- Remove all external internet access from the entire company network
  - This is not practical for modern backend services.

**Hint:** Think about protecting systems.