SRE as Applied Control Theory

An operations team can collect every metric it can name and still fail to make a service reliable. The missing piece is not always another signal. Often it is the connection between the signal, the outcome the team wants, and the action the team is prepared to take.

That connection is a feedback loop.

Control theory gives us a useful way to reason about this loop. It asks us to identify a system we care about, define its desired behavior, measure its actual behavior, compare the two, and apply an action when they differ. This is also a good description of Site Reliability Engineering at work.

The analogy is not perfect. Software systems contain people, organizations, delays, partial knowledge, and nonlinear failure modes. But it is precise enough to expose a problem in how we often approach reliability: we invest in measurement without designing the control loop that gives measurement a purpose.

Start with the outcome, not the telemetry

A control system starts with a reference: the state or behavior it is trying to maintain. For a service, the reference is not “CPU below 70 percent.” It is an outcome such as:

• Requests that satisfy a latency and correctness objective.
• Jobs that complete before their deadlines.
• Users who can complete a critical journey.
• Data that remains available and consistent within defined bounds.
• A system that degrades safely when a dependency fails.

Infrastructure conditions can influence those outcomes, but they are not the outcomes themselves.

This distinction changes the first question we ask. Instead of asking, “What metrics can this platform expose?”, we ask, “What behavior are we trying to preserve?” Only then can we decide which measurements might tell us whether the system is moving toward or away from that behavior.

The service is the plant

In control theory, the plant is the system being controlled. In SRE, the plant is the service we want to make reliable.

Defining that service is less obvious than it sounds. A user may experience one service while engineers operate a constellation of applications, queues, databases, networks, and third-party dependencies. The plant boundary should therefore follow the outcome we care about, not merely an ownership chart or a deployment boundary.

If the desired outcome is successful video playback, the relevant plant is not just the playback process. It includes enough of the catalogue, authorization, delivery, network, and client interaction to explain whether playback works for the user. Internal components still matter, but they matter as parts of the system that produces the outcome.

This is why a collection of healthy components does not prove that the service is healthy. The behavior emerges from components working together and from the connections between them.

Observation is the sensor system

The controller cannot act on a system it cannot measure. Telemetry, service-level indicators, synthetic checks, traces, logs, and user-reported outcomes form the sensor system of the SRE control loop.

This framing gives observability a more demanding purpose than “help us understand the system.” Understanding matters, but the operational question is:

What decision can this evidence support?

A measurement might support a fast automated action, such as removing an instance from service. It might support a slower human decision, such as reducing deployment velocity while an error budget is exhausted. It might support a design change after an incident. The time scale differs, but the measurement still participates in a loop.

Passive telemetry is only one way to measure. Synthetic monitoring and continuous verification actively exercise a behavior and inspect the result. They are particularly useful when internal state looks normal but the user-visible interaction is broken.

An effective sensor system usually includes several levels:

1. Outcome measurements tell us whether users receive the intended service.
2. Interaction measurements show whether important paths between components are working.
3. Component measurements help explain why the outcome has changed.
4. Resource measurements expose constraints that may cause or amplify a failure.

The order matters. Resource and component telemetry are excellent diagnostic signals, but poor substitutes for measuring the service outcome directly.

SRE provides the controller and control actions

The controller compares actual behavior with desired behavior and decides what should happen next. In a software organization, control is distributed across several participants:

• Automation reacts to known conditions.
• Operators diagnose ambiguous behavior and choose interventions.
• SRE practices define thresholds, policies, error budgets, and response patterns.
• Product and engineering teams change the plant through design and code.
• AI agents can propose, test, or apply bounded changes when their evidence and authority are explicit.

The resulting control action is not always a restart. It can be:

• Removing a failing instance from service.
• Shifting traffic between regions.
• Reducing work admitted to an overloaded subsystem.
• Rolling back a deployment.
• Disabling an optional feature to protect a critical path.
• Changing a retry policy that amplifies a failure.
• Redesigning a dependency boundary.
• Adding a new measurement where the system was previously silent.

Some actions restore operations. Others prevent the next disturbance from causing the same failure. This is why SRE belongs in design conversations as well as incident response. Changing the plant can be a more effective control action than repeatedly correcting its output.

Delays and side effects matter

Feedback loops become unstable when the controller reacts too aggressively, reacts too slowly, or misunderstands the effect of its own action.

Software systems are full of these risks:

• A health check fails, so the platform restarts a slow process. The restart increases load and makes more processes fail.
• A client retries a timed-out request. The retry adds work to an already overloaded dependency.
• An autoscaler observes a delayed metric and continues adding capacity after the demand spike has passed.
• An alert fires on a transient condition, and a manual intervention interrupts a recovery already in progress.

Control theory encourages us to examine not just whether an action is possible, but also its delay, gain, scope, and side effects. An automated action needs a clear model of what it controls and how the system is expected to respond.

This is also why every alert should imply a decision. If no safe or useful action follows from an alert, it is not participating in a control loop. It is only producing attention demand.

Postmortems close a slower loop

The operational loop responds while the system is running. Postmortems operate at a slower time scale. They examine why the loop failed to preserve the desired outcome.

A useful postmortem can reveal that:

• The desired outcome was never defined clearly.
• The service boundary excluded a critical dependency.
• The measurement arrived too late.
• The available measurement described components but not the user journey.
• The controller selected an action that made the disturbance worse.
• The system had no safe degraded mode.
• Ownership and authority delayed the necessary action.

The corrective work can then improve the sensors, controller, actions, or plant itself. The postmortem closes a learning loop around the operational loop.

This view also changes the role of an “observability gap.” It is not merely a missing graph. It is missing evidence that prevented a useful decision or hid the effect of an intervention.

Where the analogy stops

A distributed software service is not a simple mechanical plant. Desired outcomes conflict. Measurements are incomplete. Human operators have limited attention. Organizations change more slowly than code. Users adapt their own behavior. A control action can alter the conditions under which future measurements are interpreted.

The control-theory model should therefore guide questions, not produce false precision. It does not tell us that every system can be stabilized by a clever controller. It tells us to make the loop explicit:

• What outcome are we trying to preserve?
• Where is the system boundary for that outcome?
• What disturbances move the system away from it?
• What evidence arrives soon enough to matter?
• Who or what can act on that evidence?
• What does the action change, and what can it make worse?
• How do we verify that the action worked?
• How does incident learning change the next response?

Design the loop, not only the dashboard

The practical consequence is simple: reliability work should produce a coherent loop.

Begin with the user-relevant outcome. Define the service that produces it. Choose measurements that reveal meaningful changes in behavior. Connect each measurement to a decision. Define safe actions and verify their effects. When the loop fails, use the postmortem to improve the system that observes and controls the service.

Observability is essential, but it is not the destination. The purpose of measurement is to help the system and its operators choose what happens next.