Most health checks answer the easiest question: is the process still running?
That matters, but a service can return 200, keep its workers busy, and still produce stale data, drop work, or respond too late to be useful. It is available. It is not healthy.
A system is healthy when it keeps its promise to users: the right result, for the right scope, within the required freshness and time bounds. That gives each signal a job. Metrics show whether the promise holds at scale, traces show where work spent its time, and logs preserve why it did not.
Why uptime is not health
An uptime check proves only that a process can accept a request. It says nothing about whether the request reached a current dependency or produced a useful answer.
Consider a market-data system that promises a current, complete price stream for every configured pair. The API can be alive while one upstream connection has stopped advancing, a queue is growing faster than consumers can drain it, or the published watermark is stalled behind a broken source. Each component can pass its local check while the product promise has failed.
The same distinction applies to payments, email, and search. The process is a dependency of the outcome, not evidence of the outcome.
Start with the promise
Write the promise as an observable outcome. “Keep the service up” is not one. “Publish finalized one-minute bars for every configured pair within two minutes of the end of the minute” is. “Process an order once and return its durable outcome within five seconds” is.
The sentence will usually reveal several dimensions of health:
- Correctness: did the system produce the right result?
- Coverage: did it produce that result for the full population it promised to serve?
- Freshness: how far behind the relevant present is the result?
- Timeliness: how long did the customer wait for it?
- Durability: will the result survive the failure modes the system claims to handle?
Not every system needs every dimension. A batch system may care about a deadline rather than per-item latency. What matters is that the dimensions come from the promise, not a monitoring product's default dashboard.
This avoids both familiar failures: monitoring every implementation detail, or building a museum of metrics that only explain past incidents. Measure the promise directly; add diagnostic detail where it can break.
What belongs in a metric
A metric answers a question that must be asked continuously across a population: is the promise holding, and is it getting worse? Rates, distributions, counts, proportions, and gauges are cheap to aggregate, so they work for dashboards and alerts.
For the market-data promise, useful health metrics might be the age of the latest event per source, the lag between real time and the finalized watermark, the percentage of configured pairs advancing within their expected interval, and queue arrival versus drain rate. Those measurements answer meaningful questions:
- Is the watermark advancing uniformly across all configured pairs?
- Are queues draining at the rate work is arriving?
- What proportion of the promised population is fresh enough to trust?
- How much of the result is provisional rather than final?
These describe current and complete output, not the existence of a worker. A restart count may help diagnose the failure, but it is not a health signal until it threatens the promise.
Use metric dimensions only for bounded populations an operator needs to compare: region, operation, source, queue, status class, or configured pair set. Request IDs, user IDs, payloads, and error messages belong in traces or logs.
What belongs in a trace
When a metric shows pressure, a trace answers where that unit of work spent its time. It follows one request, job, event, or workflow through queueing, downstream calls, retries, and joins. That causal path is more useful than a total duration.
Suppose end-to-end freshness is worsening. A metric can show that the watermark is twenty minutes behind. A trace can show whether an event waited fifteen minutes for a saturated consumer, spent four minutes retrying one upstream API, or was processed quickly but blocked behind an ordered finalization step. Those are different failures with different fixes, even though the health metric is the same.
Trace boundaries where time or authority changes: enqueue and claim, network calls, database transactions, retries, fan-out, fan-in, and publication. Record the stable identifiers that connect the work to the promise, not every internal line.
What belongs in a log
Logs preserve the discrete facts that explain the decisive moment. A useful log records what the system knew and decided: pair, source, event range, expected and observed watermark, retry attempt, error class, recovery action, and the identifiers that connect it to a trace. It should explain a skipped range, rejected result, or fallback without reconstructing old state from a message.
Rare or high-detail evidence also belongs here: a provider response, validation failure, unusual state transition, or agent decision. Preserve it as structured events with retention and access controls suited to their contents.
The placement rule is simple: “how often, how much, or is the promise holding?” is a metric. “Where did this work spend its time?” is a trace. “What happened and why?” is a log.
Health needs a failure budget
Promises become operational at a boundary. “Fresh” needs a maximum age. “Fast” needs a budget. “Complete” needs an allowed missing fraction. Set the boundary from the consumer's need, not from the implementation's current performance.
Alert on a sustained breach or a trend that makes one likely, then retain the trace and log context needed after the system has moved on. “Queue depth is 30,000” is an implementation fact. “4% of configured pairs have been stale for ten minutes, primarily behind source X” describes product risk.
Name the promise, find where it is kept, and measure what its failure looks like to the person depending on it.
For a deeper look at why the monitoring path itself changes runtime behavior, read Observability Is Runtime Architecture. For the alert side of this model, read Why Streaming Alerts Need Explanations.
The one-line version
System health is whether the user-facing promise holds; metrics detect its breach, traces show where time went, and logs preserve why it happened.