Observability on Brian Faust

Observability Must Answer the Next Operator Question

Mon, 01 Sep 2025 00:00:00 +0000

A catalogue quote request returns 503 after its price dependency times out. One telemetry bundle produces five uncorrelated log lines, a metric labelled with request ID and raw URL, and three pages. None of those pages names an owner, the affected scope, the hold duration, or even the first query to run. There is plenty of telemetry and very little help.

The other bundle records less and tells the operator more. One page identifies the affected route, region, and release. Its first query measures the failure users can see. From there, an exemplar selects a failed trace, the trace locates the dependency wait, and a correlated event names the application failure class. The runbook then asks the question which actually matters: is it safe to bypass that dependency?

Audit History Is Different from Application Logging

Wed, 01 Nov 2017 00:00:00 +0000

When a cart appears to expire, support may ask:

Which cart changed?
What was its prior state?
What state followed?
Who or what caused the transition?
Which policy allowed it?

An engineer investigating the same report may ask:

Which request handled it?
Which worker retried?
How long did the database call take?
Which exception and stack occurred?
What else happened on that host?

Both groups need evidence, but making one record answer both sets of questions usually gives you a bloated audit trail and strangely incomplete logs. Audit history explains durable business change. Application logs explain software execution.

A Correlation Identifier Is a Boundary Contract

Thu, 01 Sep 2016 00:00:00 +0000

We had logs everywhere and still could not follow one shipment through the system. It had failed somewhere after payment and before its documents were ready, but the application, queue worker, and remote adapter did not share an identifier.

The investigation started with timestamps, account details, and values which looked related. Payment was plausible, as were several remote calls nearby. Some trails took hours to disprove because they belonged to different requests which happened at roughly the same time. We did not need more log lines. We needed a chain of custody.

Reproduce the Failure Before Repairing It

Fri, 01 Jan 2016 00:00:00 +0000

Submit two different browser error reports. The application accepts both requests, sends both reports, and records the same tracing identifier for each.

Nothing crashed during submission. No assertion failed. The defect appears later, when someone searches the logs and finds two executions presented as one trace.

The tempting response is to replace the identifier generator immediately. The collision already points in that direction. But a plausible diagnosis is not a reproduction. Before changing the generator, we need a check which states what was observed, what should differ, and which surrounding machinery does not matter.