As software systems have grown from single applications into sprawling networks of services communicating over a network, understanding what they are actually doing has become dramatically harder. A request from a user might pass through a dozen separate services, touch several databases, call external APIs, and traverse multiple queues before a response comes back. When something goes wrong in that chain, the old approach of logging into a single server and reading a log file is hopelessly inadequate. This is the problem observability exists to solve, and it has become one of the most important disciplines in operating modern software.
Monitoring Versus Observability
It helps to start by distinguishing observability from the older practice of monitoring, because the two are often conflated. Monitoring answers questions you knew to ask in advance. You decide that you care about CPU usage, error rates, and response times, you set up dashboards and alerts for those metrics, and the system tells you when they cross a threshold. Monitoring is essential, but it is fundamentally about known unknowns, the failure modes you anticipated.
Observability is about the unknown unknowns. A truly observable system lets you ask new questions about its behavior that you never anticipated, without shipping new code to gather the data. When a customer reports that requests are mysteriously slow but only for a particular account in a particular region during a particular hour, no pre-built dashboard will have anticipated that exact scenario. Observability is the property that lets you investigate it anyway, drilling into the system’s actual behavior to find an explanation. The difference is the difference between checking the gauges you installed and being able to interrogate the machine about anything.
The Three Pillars
Observability is commonly described as resting on three kinds of telemetry data, each answering a different question about the system. Understanding what each one is good for, and what it is not good for, is the foundation of building observable systems.
- Logs are discrete, timestamped records of events. They are rich in detail and excellent for understanding exactly what happened at a specific moment, but they can be voluminous and expensive to store and search at scale.
- Metrics are numerical measurements aggregated over time, such as the number of requests per second or the ninety-fifth percentile latency. They are cheap to store and ideal for spotting trends and triggering alerts, but they lack the detail to explain why something happened.
- Traces follow a single request as it moves through every service in a distributed system, recording how long each step took. They are uniquely powerful for understanding where time goes in a complex request and for pinpointing which service in a long chain is the bottleneck.
No single pillar is sufficient on its own. Metrics tell you something is wrong, traces tell you where, and logs tell you why. The discipline of observability is about weaving these three together so that an engineer can move fluidly from a high-level alert down to the specific line of detail that explains a failure.
The Importance of Context
Raw telemetry is only useful if you can connect the pieces. A log line that says an error occurred is far less valuable than one that also records which user triggered it, which request it belonged to, which version of the code was running, and which downstream service had just been called. This connective tissue is often called context, and propagating it consistently across service boundaries is one of the central technical challenges of observability.
The key mechanism is a shared identifier, often called a trace or correlation identifier, that is attached to a request when it enters the system and carried with it through every service it touches. With this identifier threaded through every log line, metric, and trace span, an engineer can reconstruct the complete story of a single request across the entire distributed system. Without it, they are left trying to correlate timestamps across disconnected logs from different machines, which is slow, error-prone, and frequently impossible.
Cardinality and Cost
Observability is not free, and the tension between richness and cost is a constant practical concern. The more detailed and high-cardinality your telemetry, the more questions you can answer, but the more it costs to collect, store, and query. Cardinality, the number of distinct values a piece of data can take, is particularly important. Attaching a unique user identifier to every metric, for example, can explode the storage requirements astronomically, because each unique value multiplies the data the system must track.
Mature teams treat observability as something to be designed deliberately rather than collected indiscriminately. They sample traces intelligently, keeping enough to understand normal behavior while preserving the rare slow or failed requests that matter most. They are thoughtful about which dimensions deserve high cardinality and which can be aggregated. The goal is to capture enough signal to answer the questions that arise in practice, without drowning in data or in the bills that come with storing it.
A Culture, Not Just a Toolset
Ultimately, observability is less about any particular tool and more about a way of building and operating software. It means writing code that emits useful telemetry as a matter of habit, instrumenting new features with the same care given to the features themselves, and treating the ability to understand production as a first-class requirement rather than an afterthought. Teams that take this seriously resolve incidents in minutes rather than hours, because they can ask the system what is happening and get an answer. Teams that neglect it spend their incidents guessing. In a world where systems are only growing more distributed and complex, the quiet discipline of being able to see clearly into your own software has become one of the defining capabilities of a high-functioning engineering organization.