What Do We Observe When Doing Observability?

“We need better observability” often turns into a list of instruments: metrics, logs, traces, profiles, dashboards, and alerts. These instruments are useful, but listing them avoids the harder question.

What are we trying to observe?

The answer is not the structure of the application. Architecture diagrams, deployment manifests, and source code already tell us how the system was designed. At runtime we need to observe something less stable: its behavior.

The distinction matters because a system can preserve its structure while failing to provide its service. Every process can be running. Every component can answer its local health check. The user can still be unable to complete the operation that brought them to the system.

A service is defined by a perspective

SRE is concerned with the reliability of a site or service. Before measuring reliability, we need to define the service.

A service is a system that provides a function to users. That definition depends on perspective. A customer may interact with one video service. The engineers who operate it see catalogue services, identity systems, playback services, content delivery networks, databases, queues, and clients.

Neither view is wrong, but they answer different questions.

The customer asks whether video playback works. The catalogue team asks whether its API is serving requests. The database team asks whether replicas are available. The infrastructure team asks whether machines have capacity. Each perspective selects a different system boundary and therefore a different set of useful observations.

The mistake is to use a measurement from one boundary as proof about another. A healthy database does not prove playback works. A running process does not prove its responses are correct. A successful HTTP request does not prove that the user journey completed.

Structure explains where behavior can emerge

Applications are designed as components. Some components are logical, some are physical, and many boundaries exist because different teams need to build and operate the system independently.

Those boundaries help us reason about implementation, but the service emerges from components working together. Communication between components matters as much as each component's local condition.

This gives us two related but different forms of knowledge:

• Structural knowledge describes what the system contains and how parts are intended to connect.
• Behavioral evidence describes what the system actually does under current conditions.

We need both. Structural knowledge helps us decide where to instrument and how to interpret evidence. Behavioral evidence tells us whether the current system is producing the outcome we need.

Observability is not a replacement for architecture. Architecture is not a replacement for observing runtime behavior.

Healthy components do not add up to a healthy service

Naive monitoring starts with the serviceability of machines: CPU, memory, disk, load, and process status. More mature monitoring adds component-level request rates, error rates, queues, and health endpoints.

These signals are valuable. They make local failures visible and support diagnosis. But they cannot establish the health of the whole service by addition.

Imagine a user request that crosses four components. Each component can satisfy its local check while the request fails because:

• The caller and callee disagree about a protocol or schema.
• A routing rule sends traffic to the wrong place.
• Authentication succeeds locally but the propagated identity is incomplete.
• A queue accepts work that no consumer can finish before the deadline.
• A retry policy turns a partial slowdown into sustained overload.
• The final response is technically valid but semantically wrong.

Nothing in these cases requires a component to be “down.” The failure exists in the interaction or in the outcome produced by the complete path.

The system is a team. Each member must perform, but the hand-offs also need to work.

Observe behavior at boundaries

One useful way to choose measurements is to begin at system boundaries.

At the outer boundary, observe what the user sends and what the user receives:

• Can the user begin the operation?
• Does the operation complete?
• Is the result correct and timely?
• Does the service fail in a way the user can recover from?

At internal boundaries, observe the contracts between components:

• Was the request accepted?
• Was identity and context propagated?
• Did the dependency respond within the caller's useful deadline?
• Was work acknowledged before it was durable?
• Did retries, fallbacks, or circuit breakers change the path?

Inside a component, observe the state needed to explain those interactions:

• Saturation and queueing.
• Resource exhaustion.
• Error classification.
• Cache behavior.
• Concurrency and lock contention.

This ordering keeps internal telemetry connected to the service behavior it is supposed to explain.

Passive telemetry and active measurement

Runtime telemetry observes work that the system is already doing. It is essential, but it inherits the traffic and failure modes that happen to occur.

Active measurement deliberately exercises behavior. A synthetic probe can ask whether DNS resolves from a particular network, whether an authenticated API workflow succeeds, or whether a browser can complete a critical journey. It can perform the same scenario repeatedly, even when real traffic is sparse.

Continuous verification is especially useful for distributed systems because location changes the answer. A service can be reachable from a public probe and unreachable from a factory network. A regional dependency can work for one worker and fail for another. A browser interaction can break while the HTTP endpoint continues returning successful status codes.

Active checks do not replace passive evidence. They create known inputs and make the resulting behavior easier to compare. Passive telemetry then helps explain why the behavior changed.

“Understand the system” is too vague

Observability is often described as the ability to understand internal state from external outputs. That definition is useful, but “understand” can become an unlimited goal. A team can always collect more data and build another dashboard in the name of understanding.

SRE needs a more operational test:

Which uncertainty prevents us from deciding what to do?

The answer might require a new measurement. It might also require a clearer service objective, a better system model, or a safer control action.

A useful observation reduces uncertainty about one of three things:

1. Whether the service is producing the desired outcome.
2. Why current behavior differs from that outcome.
3. Whether an intervention moved the system in the intended direction.

Data that does none of these may still be interesting, but it should not automatically become an alert or a permanent dashboard.

Design observations from the decision backwards

Instead of beginning with available telemetry, begin with a decision the team must make.

For each important decision, write down:

1. Outcome — What user- or service-level behavior are we protecting?
2. Boundary — Which system must produce that behavior?
3. Disturbance — What can move it away from the desired outcome?
4. Evidence — What would distinguish normal variation from harmful change?
5. Action — What can an operator or automation safely do?
6. Verification — How will we know that the action helped?

This process will still lead to metrics, logs, traces, profiles, and synthetic tests. The difference is that each instrument has a job.

Observe the service that users depend on

The purpose of observability is not to reproduce the architecture diagram in a dashboard. It is to make runtime behavior available to the feedback loops that keep a service reliable.

Start with the function users depend on. Define the system boundary that produces it. Measure the behavior at that boundary, then instrument internal interactions and components so that changes in the outcome can be explained.

When all components report healthy but the user cannot complete the journey, believe the user-facing behavior. It is the service—not the inventory of parts—that SRE is trying to make reliable.