--- title: "Observability Stack: Monitoring, Logging & Tracing" description: "Master observability fundamentals: monitoring with metrics, structured logging, distributed tracing, and observability practices. Learn to build observable systems and debug production issues." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/quizzes/post/observability-stack-quiz --- # Observability Stack: Monitoring, Logging & Tracing Welcome to the Observability Stack Quiz! This quiz will test your knowledge of monitoring, logging, and tracing concepts in modern software systems. Each question is designed to challenge your understanding of how observability works and how it can be applied to build reliable, maintainable applications. Good luck! ## Questions ### 1. What are the three pillars of observability? - **Metrics, logs, and distributed traces** ✅ - These represent the core telemetry data types: quantitative measurements, event records, and request paths. - Uptime, availability, and error rates - These are service level indicators (SLIs), not the primary telemetry pillars. - Dashboards, alerts, and report logs - These are methods of data visualization and notification, not the data types themselves. - CPU, memory, and disk utilization - These are specific infrastructure metrics, which are a subset of one pillar. **Hint:** Think about the main data types collected from systems. ### 2. What are metrics in observability? - **Timestamped numerical measurements of system behavior** ✅ - Metrics are quantitative data points like CPU usage or requests per second, typically stored as time-series. - Textual records of specific application events - This describes logs, which contain unstructured or structured text rather than pure numbers. - Contextual paths of requests across services - This describes distributed tracing, which follows a request through multiple components. - Automated notifications triggered by thresholds - This describes alerting, which uses metrics but is not the metric itself. **Hint:** Think about numerical measurements over time. ### 3. What is structured logging? - **Logs using fixed key-value formats like JSON** ✅ - Structured logs use parseable formats to make searching and automated analysis more efficient. - Logs organized strictly by severity levels - Severity (Info, Error) is a feature of most logs but does not define the format structure. - Logs stored in a relational SQL database - Storage location is independent of whether the log data itself is structured or plaintext. - Logs restricted to system-level kernel events - Structured logging refers to the format of the data, not the source of the events. **Hint:** Think about logs with consistent, parseable format. ### 4. What is a distributed trace in observability? - **A map of a request as it moves through services** ✅ - Traces visualize the end-to-end journey of a request, identifying bottlenecks across microservices. - A collection of all error logs from one host - This is log aggregation, which lacks the request-level correlation found in tracing. - A snapshot of memory usage during a crash - This describes a memory dump or profile, which is used for debugging local execution. - A history of configuration changes in a cluster - This describes an audit trail, not the execution path of a specific user request. **Hint:** Think about following a request through multiple services. ### 5. What is Prometheus in the observability stack? - **A time-series database for metric collection** ✅ - Prometheus pulls (scrapes) numerical data from targets and stores it for querying via PromQL. - A centralized engine for log search and index - This describes tools like Elasticsearch or Loki, which handle text-based logs. - A dashboarding UI for multi-source visualization - This describes Grafana, which displays data that Prometheus collects. - A proxy for load balancing network requests - Prometheus is a monitoring tool, not a networking component like Nginx or HAProxy. **Hint:** Think about a tool for collecting and scraping metrics. ### 6. What is Grafana used for? - **Visualizing data via dashboards and graphs** ✅ - Grafana is the visualization layer that connects to various data sources like Prometheus or InfluxDB. - Scraping and storing time-series metric data - This is the role of the data source (like Prometheus), not the visualization tool. - Generating and propagating trace context IDs - This is handled by instrumentation libraries (like OpenTelemetry), not by Grafana. - Parsing and transforming raw log aggregates - Parsing is typically done by collectors like Logstash or Fluentd before reaching the UI. **Hint:** Think about visualizing metric data with dashboards. ### 7. What is the ELK Stack in observability? - **A platform for log search, ingestion, and UI** ✅ - ELK (Elasticsearch, Logstash, Kibana) provides a complete pipeline for managing log data. - A suite for infrastructure performance metrics - While it can ingest metrics, the ELK stack is primarily optimized for text-based logs. - A protocol for standardized trace propagation - W3C Trace Context or B3 are protocols; ELK is a software stack. - A framework for automated incident response - ELK provides the data for analysis, but response is usually handled by separate alerting tools. **Hint:** Think about three tools for log aggregation and analysis. ### 8. What is Jaeger used for? - **Monitoring and troubleshooting distributed traces** ✅ - Jaeger is an open-source tool used for capturing and visualizing the flow of requests. - Aggregating and indexing application log files - Jaeger is specific to tracing; log aggregation is handled by tools like Elasticsearch. - Proving uptime through external health checks - This is synthetic monitoring; Jaeger monitors internal request spans. - Managing alert notifications and on-call rotations - This is the role of incident management platforms like PagerDuty. **Hint:** Think about visualizing request flows across services. ### 9. What is a span in distributed tracing? - **The record of a single unit of work or operation** ✅ - A span is the building block of a trace, representing a specific start and end time for an operation. - The total time a request takes from start to finish - The total time of the entire request is represented by the Trace, not a single span. - A unique identifier assigned to a specific user - A User ID might be a tag within a span, but it is not the definition of a span. - A threshold limit for triggering a system alert - Thresholds are part of alerting rules, not distributed tracing structures. **Hint:** Think about a single operation within a trace. ### 10. What is a trace context, and why is it important? - **Metadata passed between services to link spans** ✅ - Trace context ensures that separate operations in different services are recognized as part of the same trace. - The configuration file for a tracing backend - Backends use configuration, but "context" refers to the request-specific metadata. - A summary report of system-wide performance - This describes a dashboard or report, not the technical mechanism of request correlation. - The specific code library used for instrumentation - The library (SDK) facilitates the context, but the context itself is the metadata (IDs). **Hint:** Think about correlating requests across services. ### 11. What is sampling in observability? - **Recording a subset of events to reduce overhead** ✅ - Sampling helps manage the volume and cost of data by only storing a percentage of traces or logs. - Testing a system under heavy simulated load - This is load testing or stress testing, not telemetry sampling. - Selecting random servers for manual log inspection - Sampling is an automated process for data collection, not a manual inspection method. - The process of converting raw logs into metrics - This is log-to-metric extraction, which is distinct from sampling data volume. **Hint:** Think about not collecting every single event. ### 12. What is an SLO (Service Level Objective)? - **A target value or range for service reliability** ✅ - SLOs define the desired state of a service, such as "99.9% of requests must succeed." - The actual measured uptime of a service - Actual measurements are Service Level Indicators (SLIs), not the objectives (targets). - A legal contract specifying penalties for downtime - This is a Service Level Agreement (SLA), which is a business contract based on SLOs. - A technical tool used for synthetic monitoring - An SLO is a goal or policy, not a specific piece of software. **Hint:** Think about target reliability or performance. ### 13. What is an SLI (Service Level Indicator)? - **A specific metric used to measure service levels** ✅ - SLIs are the actual quantitative measures (like error rate or latency) used to track performance against SLOs. - The target percentage for service availability - This is an SLO; the SLI is the data point used to see if that target is met. - A notification sent when a service is failing - This is an alert; SLIs are the underlying data that might trigger an alert. - The documentation for an application API - An SLI is a performance metric, not code documentation. **Hint:** Think about measuring actual system performance. ### 14. What is an error budget in observability? - **The total allowed unreliability before missing an SLO** ✅ - Calculated as 100% minus the SLO, it represents the room for failure or maintenance. - The financial cost allocated for cloud monitoring - This describes an operational budget, not the technical concept of an error budget. - The maximum number of errors a developer can make - An error budget applies to service performance, not individual developer activity. - A buffer for network latency in a distributed system - Latency might be part of an SLO, but the "budget" refers to the reliability margin. **Hint:** Think about allowed downtime within an SLO. ### 15. What is cardinality in metrics? - **The number of unique sets of label-value pairs** ✅ - High cardinality occurs when labels (like user_id) have many unique values, increasing storage costs. - The frequency at which metrics are collected - This is the "resolution" or "scrape interval" of the metric collection. - The total number of servers being monitored - While servers contribute to cardinality, cardinality specifically refers to label combinations. - The mathematical accuracy of a numerical metric - This refers to precision or fidelity, not the cardinality of the data set. **Hint:** Think about the number of unique label combinations. ### 16. What is APM (Application Performance Monitoring)? - **A suite for monitoring code-level execution and health** ✅ - APM tools provide deep insights into application code, database queries, and transaction traces. - A protocol for basic hardware-level status checks - APM focuses on the application layer, whereas protocols like SNMP focus on hardware. - A method for managing project development lifecycles - This is Application Lifecycle Management (ALM), not performance monitoring. - A dashboard showing only network bandwidth usage - APM is much broader, covering internal application logic and transaction paths. **Hint:** Think about end-to-end application monitoring. ### 17. What is a RED metric (Rate, Errors, Duration)? - **A framework focusing on request-based service health** ✅ - RED stands for Rate (requests/sec), Errors (failed requests), and Duration (latency). - A color-coded system for prioritizing incident response - While "red" implies urgency, the RED acronym refers to specific metric types. - A dashboard for tracking resource-level utilization - Resource utilization is typically handled by the USE framework, not RED. - A validation check for data integrity in logs - RED is a service monitoring methodology, not a data validation technique. **Hint:** Think about key metrics for service health. ### 18. What is a USE metric (Utilization, Saturation, Errors)? - **A framework for monitoring hardware resource health** ✅ - USE stands for Utilization (busy time), Saturation (waiting work), and Errors (resource failures). - A method for calculating user engagement rates - This would be a business metric; USE focuses on system infrastructure resources. - A strategy for optimizing cloud computing costs - While monitoring helps optimize costs, the USE framework is for health and performance. - A protocol for encrypting telemetry data in transit - USE is a monitoring methodology, not a security or encryption protocol. **Hint:** Think about infrastructure health metrics. ### 19. What is alerting fatigue, and how do you prevent it? - **Operational burnout from high volumes of noise** ✅ - Prevented by setting actionable thresholds and only alerting on symptoms that impact customers. - System degradation caused by frequent health checks - This is "observer effect" or overhead, not alerting fatigue of the human team. - The loss of data when an alerting tool fails - Fatigue refers to the human response to excessive notifications, not data loss. - A delay in notification delivery during outages - This is alert latency; fatigue is the result of too many notifications, not late ones. **Hint:** Think about too many false alarms from alerts. ### 20. What is a runbook in observability? - **A guide containing procedures for resolving incidents** ✅ - Runbooks provide step-by-step instructions for on-call engineers to follow when an alert triggers. - A chronological log of all past system failures - This is an incident history or post-mortem archive, not a proactive response guide. - A configuration script for automated cloud deployments - This is Infrastructure as Code (IaC) or a deployment script, not a runbook. - A database index of all available system metrics - This is a metric catalog or metadata registry, not an operational procedure guide. **Hint:** Think about documentation for responding to alerts. ### 21. What is MTTR (Mean Time To Recover)? - **The average time taken to restore service after failure** ✅ - MTTR measures the speed of the incident response process from detection to resolution. - The typical duration between two separate outages - This is Mean Time Between Failures (MTBF), which measures reliability, not recovery speed. - The time required to detect that a problem exists - This is Mean Time To Detect (MTTD), only one component of the total recovery time. - The total uptime divided by the number of users - This is not a standard DevOps metric; MTTR specifically focuses on recovery duration. **Hint:** Think about how fast you can fix an incident. ### 22. What is a correlation ID in observability? - **A unique value used to tag related telemetry data** ✅ - Correlation IDs allow engineers to search for a single ID and see all logs and traces for one request. - A statistical score measuring metric relationships - While "correlation" is a statistical term, in observability, it refers to a tracking identifier. - A password used to access restricted monitoring tools - IDs are for data organization, not for authentication or access control. - A version number for application deployment tracking - This is a Commit ID or Version Tag, used for code management rather than request tracking. **Hint:** Think about linking logs across requests. ### 23. What is log aggregation? - **Centralizing logs from various sources for analysis** ✅ - Aggregation moves logs from individual servers to a single searchable location like Elasticsearch. - Compressing old log files to save storage space - This is log rotation or archival, which deals with storage management rather than centralizing. - The process of deleting duplicate error messages - This is de-duplication, which may happen during aggregation but is not its primary definition. - Converting unstructured text into numerical metrics - This is metric extraction or parsing, whereas aggregation is about the movement and storage of data. **Hint:** Think about collecting logs from multiple sources. ### 24. What is flame graph visualization? - **A chart visualizing code execution and CPU usage** ✅ - Flame graphs show which functions are consuming the most resources by stacking call frames. - A real-time map of global network traffic spikes - Flame graphs are used for application profiling, not for geographic network mapping. - A heat map showing server room temperature data - Despite the name, flame graphs are performance tools, not physical environment monitors. - A trend line showing the growth of log storage - This would be a capacity planning graph, whereas flame graphs focus on execution timing. **Hint:** Think about seeing where time is spent in a profile. ### 25. What is the relationship between observability and monitoring? - **Monitoring tracks knowns; observability probes unknowns** ✅ - Monitoring tells you when a system is broken (the what); observability helps you find out why it broke. - Observability is the tool; monitoring is the data - Monitoring is a practice, while observability is a property of the system architecture. - They are identical terms used by different vendors - While related, they have distinct technical meanings regarding "known-unknowns" vs "unknown-unknowns." - Monitoring is for apps; observability is for hardware - Both concepts apply to all layers of the technology stack, from hardware to application code. **Hint:** Think about monitoring vs. the deeper investigation. ### 26. What is OpenTelemetry (OTel)? - **A vendor-neutral standard for telemetry collection** ✅ - OpenTelemetry provides a unified set of APIs and SDKs to collect metrics, logs, and traces. - A commercial SaaS platform for metric storage - OTel is an open-source framework, not a paid storage service like Datadog or New Relic. - A specialized database for high-cardinality logs - OTel is for data collection and instrumentation, not for the long-term storage of that data. - A security protocol for monitoring user behavior - OTel is for system observability, not for security auditing or user behavior analytics. **Hint:** Think about standardized instrumentation across languages. ### 27. What is continuous profiling in observability? - **Always-on analysis of application resource usage** ✅ - Continuous profiling captures stack traces across the fleet to identify performance bottlenecks in production. - A one-time test performed before a software release - One-time tests are "benchmark" or "snapshot" profiles, not "continuous" profiling. - The process of constantly updating user permissions - This refers to Identity Access Management (IAM), not performance observability. - A method for real-time monitoring of SQL queries - While it can profile DB drivers, continuous profiling generally refers to the whole application runtime. **Hint:** Think about always-on performance monitoring. ### 28. What is observability-driven development (ODD)? - **Designing systems with instrumentation as a priority** ✅ - ODD encourages developers to consider how a feature will be observed and debugged during the coding phase. - A practice of debugging only using visual dashboards - ODD is about the development process and instrumentation, not just the choice of UI. - The requirement to use only open-source monitoring - ODD is a methodology that can be applied using any tools, whether open-source or commercial. - Automated code generation based on system metrics - ODD involves human design of telemetry, not the automated creation of application code. **Hint:** Think about building systems with observability from the start. ### 29. What is data retention policy in observability? - **Rules defining the lifespan of stored telemetry data** ✅ - Retention policies balance the need for historical data against the costs of storage and compliance requirements. - The encryption standard used for data at rest - Retention defines the duration of storage, while encryption defines the security of that storage. - A legal requirement to never delete application logs - Retention policies often mandate when data *should* be deleted to comply with privacy laws like GDPR. - The speed at which data is written to a database - This describes "ingestion rate" or "write throughput," not a retention policy. **Hint:** Think about how long to keep observability data. ### 30. What is alert routing and escalation policy? - **Rules that direct alerts to the correct personnel** ✅ - Routing ensures the right team is notified, while escalation ensures a backup is notified if the first person is unavailable. - A method for automatically fixing system errors - This is "self-healing" or "auto-remediation," whereas routing is about human notification. - The physical network path taken by alert packets - Routing in this context refers to organizational workflow, not network layer-3 routing. - A limit on the number of alerts sent per hour - This is "alert throttling" or "rate limiting," not routing or escalation. **Hint:** Think about directing alerts to right people. ### 31. What is observability for databases? - **Monitoring internal database health and query execution** ✅ - Database observability tracks metrics like lock contention, execution plans, and buffer pool usage. - Performing regular backups of relational tables - Backups are for disaster recovery; observability is for real-time performance and health insight. - Restricting access to the database using firewall rules - This is network security, not the observability of the database performance. - The process of normalizing data to reduce redundancy - Normalization is a database design technique, not a monitoring or observability practice. **Hint:** Think about monitoring query performance and health. ### 32. What is observability cost optimization? - **Managing telemetry volume to control expenses** ✅ - Optimization involves using sampling, filtering, and aggregation to keep monitoring costs sustainable. - Selecting the cheapest monitoring vendor available - Cost optimization is about data management efficiency, not just the unit price of a vendor. - Removing all logs to eliminate storage overhead - Removing all data eliminates observability; optimization seeks to maintain insight while reducing cost. - Moving all monitoring data to local on-prem servers - On-prem storage still has costs (hardware/labor); optimization is independent of the host location. **Hint:** Think about reducing observability tool spending.