Your Service Health Check Is Not Enough

A health endpoint returns 200 OK, every pod is ready, and the load balancer is sending traffic. The service still does not work.

This situation is common because we ask health checks to prove more than they can. A check can answer a narrow question for a particular caller. It cannot compress the behavior of a distributed service into one universal bit.

The right question is not “Is this service healthy?” It is:

Which decision must this caller make, and what evidence does it need?

That shift turns health checks from ritual endpoints into explicit parts of a control loop.

A health check is part of a contract

A health check is an interface that one component calls to decide how it should treat another component.

The caller might be:

• A process supervisor deciding whether to restart a process.
• Kubernetes deciding whether a container has started.
• Kubernetes deciding whether a pod should receive traffic.
• A load balancer deciding whether to route a request to an instance.
• A deployment controller deciding whether a rollout can continue.
• An operator deciding whether to investigate.

These callers have different responsibilities and different available actions. They therefore need different evidence.

A process can be alive but not ready. It can be ready for one class of request but unable to perform another. It can serve useful degraded responses while a dependency is unavailable. It can answer its own health endpoint while the network path used by real clients is broken.

One endpoint named /health rarely expresses all of these distinctions.

Classify the question before implementing the check

Health checks become clearer when classified by what they establish.

Process liveness

Question: Is the process running, or is it irrecoverably stuck?

A liveness check supports a severe action: restart the process. It should therefore be conservative. Temporary dependency failure is usually not proof that the process itself must be restarted.

Useful liveness evidence can include:

• The process can execute its event loop.
• A critical internal worker has not deadlocked.
• An invariant required for any further progress is intact.

Liveness is not an end-to-end service check.

Startup completion

Question: Has the process completed initialization?

Some applications need time to load data, compile state, recover a log, or make a first connection. A startup check prevents the platform from applying liveness policy before initialization has had a fair chance to complete.

Without this distinction, a slow but progressing startup can be trapped in a restart loop.

Instance readiness

Question: Should this instance receive new work now?

Readiness supports traffic routing or work assignment. It can depend on local capacity, required configuration, and the ability to accept the next request.

Readiness can change without implying that the process should restart. An overloaded instance may need a short period without new work while it drains an existing queue.

Protocol response

Question: Can the process accept a connection and speak the expected protocol?

Opening a TCP port or returning an HTTP response establishes more than process existence, but less than useful service behavior. It proves that part of the request path is functioning.

Dependency-aware readiness

Question: Can this instance perform work that requires a particular dependency?

This check can be useful, but it is dangerous when applied mechanically. If every instance declares itself unready because one shared dependency fails, the load balancer can remove the entire service even when a degraded response would have been useful.

A dependency check also creates new traffic. If many instances probe the same struggling dependency at high frequency, the check can contribute to the failure it is trying to detect.

End-to-end verification

Question: Can a user-relevant operation complete from a relevant vantage point?

This is usually a synthetic test rather than an instance health endpoint. It can cross routing, authentication, dependencies, storage, and response validation. It measures the service behavior that local health checks cannot prove.

End-to-end verification is excellent for detection, but often too broad and too slow to drive an instance restart directly.

The actuator determines the risk

In a control loop, a measurement matters because it can trigger an action. The action attached to a health check is the actuator.

Common actuators include:

• Restart this process.
• Stop sending new traffic to this instance.
• Stop the deployment.
• Replace an instance.
• Alert an operator.

The same failed measurement can be safe for one actuator and destructive for another.

Consider a database dependency outage. Removing one instance from service may be reasonable if other instances still have a working path. Restarting every instance is unlikely to repair the database and can destroy warm caches, increase connection churn, and delay recovery. Marking every instance unready can turn a partial outage into a complete one.

Before adding a condition to a check, ask:

1. What action follows when this condition fails?
2. Can that action correct the condition?
3. What happens if every instance takes the same action at once?
4. Does the action remove a useful degraded capability?
5. How quickly can the check and action reverse after recovery?

A check that cannot answer these questions is not ready to control the platform.

Common failure modes

The check only proves that its own endpoint works

A health handler can be isolated from the code paths that serve real work. It may respond while request workers are deadlocked or critical queues are full.

The solution is not to execute every business workflow inside /health. It is to be explicit about the narrow fact the endpoint proves and complement it with behavioral measurements.

Every dependency becomes mandatory

Deep checks often treat the failure of any downstream dependency as complete instance failure. This ignores optional features, cached responses, read-only modes, and other forms of graceful degradation.

Represent capabilities separately where possible. An instance can be ready for one operation and unable to perform another.

The check overloads the dependency

Frequent checks from many replicas can create a synchronized workload. The effect becomes worse during recovery, when every process reconnects at once.

Use bounded timeouts, jitter, caching, and rate limits. Reuse existing evidence when it has the right freshness instead of creating a new request for every probe.

Timeouts do not match useful work

A check that waits longer than the caller can wait provides late evidence. A check with an unrealistically short timeout creates false failures during normal variation.

Align timeouts with the decision the caller must make, not with a generic platform default.

The check reports a binary answer for a multidimensional service

A service may have independent read, write, search, ingest, and administrative capabilities. One Boolean result hides which capability failed and which work can continue.

Use separate checks, capability status, or richer operational signals when the caller can make a more selective decision.

The check is measured from the wrong place

An in-process check cannot see a broken load-balancer rule. A cluster-local check cannot prove reachability from a private customer network. A public probe cannot see an internal DNS failure.

Place checks at the boundaries whose behavior you need to verify.

Build a check matrix

Instead of beginning with endpoint names, map callers to decisions and actions.

Caller	Question	Evidence	Action
Process supervisor	Can the process make progress?	Internal liveness invariant	Restart process
Startup controller	Has initialization completed?	Startup state	Continue waiting or fail startup
Load balancer	Should this instance receive new work?	Readiness and local capacity	Add or remove route
Deployment controller	Is the new version serving safely?	Readiness plus rollout measurements	Continue or stop rollout
SRE automation	Is a service objective at risk?	Outcome and path measurements	Apply bounded remediation
Operator	Is the user journey failing?	Synthetic and production evidence	Investigate or intervene

For each row, document:

• The exact semantics of success and failure.
• The timeout and freshness requirements.
• The action taken after failure.
• The expected mechanism by which that action helps.
• The behavior when many instances fail together.
• The signal used to verify recovery.

This matrix is more valuable than a single generic /health contract.

Health checks are not observability

A health check produces one observation for one decision. Observability is the broader measurement capability that helps us detect, explain, and verify system behavior.

You still need evidence about:

• User-visible success and failure.
• Latency distributions and deadlines.
• Queueing and work completion.
• Communication between components.
• Degraded modes and fallback behavior.
• The effect of restarts, routing changes, and other interventions.

Health checks can participate in this system, but they cannot replace it.

Make every check answer to its caller

A good health check is deliberately incomplete. It answers a specific question quickly enough for a specific caller to make a specific decision.

Separate liveness from readiness. Separate startup from steady-state behavior. Avoid attaching destructive actions to broad dependency checks. Verify the complete service with synthetic and user-facing measurements from relevant vantage points.

Most importantly, review the action behind the check. A measurement that causes the wrong intervention can reduce reliability while reporting that it is protecting it.