Observability is the ability to understand what happens inside a system just by looking at what it emits. Monitoring answers “is it broken?”; observability answers “why is it broken?”, including for problems nobody anticipated.
The three signals
| Signal | What it is | Answers |
|---|---|---|
| Logs | Event records with text and context | What happened in this request? |
| Metrics | Numbers aggregated over time (counters, histograms) | How many errors per minute? What is the p99 latency? |
| Traces | A request’s path through every service, with the time of each step | Where did this request spend 2 seconds? |
The trace above shows right away that the problem is in payments, and more specifically in the external call. With logs alone, you would have to cross-reference records from four services.
What ties it together: context
Each request gets a trace id, propagated between services (in the traceparent header, a W3C standard) and written to every log. So from a spike in a metric you get to the trace, and from the trace to the exact logs.
In practice
- Structured logs (JSON), with the trace id, instead of loose text.
- RED metrics for each service: Rate (requests per second), Errors and Duration.
- OpenTelemetry: the CNCF’s open standard for generating and exporting all three signals, vendor-neutral. Instrument once and send to Grafana, Datadog, Jaeger or whatever you use.
- Alerts based on what the user feels (SLOs, see the glossary), not on high CPU.
See also: System design and factor XI of The Twelve-Factor App.
- To dig deeper: opentelemetry.io