Reliability on Brian Faust

Recover the First Useful Service, Not Every Server

Thu, 01 Oct 2026 00:00:00 +0000

Six components sit on the recovery checklist: a database, queue, cache, carrier adapter, identity service, and API. You can follow that checklist exactly, restore the first four without error, and still leave every customer unable to retrieve a shipment.

The OPS-05 fixture gives every recovery action a deterministic duration. Inventory order reaches an accepted customer journey at 540 virtual milliseconds. Restoring identity, shipment data, and the API first reaches a bounded read-only journey at 240. Both schedules finish all six components at 540.

Replication Copies Corruption Too

Tue, 01 Sep 2026 00:00:00 +0000

The primary answers SELECT 1. So does its streaming standby. Their WAL positions show that the standby has caught up, and PostgreSQL promotes it without error. shipment-100 is missing from both.

Every infrastructure signal in that sequence can be true at the same time. The OPS-04 fixture makes it happen with PostgreSQL 18.1: a valid DELETE commits on the primary, reaches the physical standby, and survives promotion. Failover restores a writable endpoint, but the deleted shipment is still deleted. Replication worked perfectly. That is the problem.

A Backup Is Only an Input to Recovery

Sat, 01 Aug 2026 00:00:00 +0000

The backup job is green. The archive exists, its checksum matches, and pg_restore exits successfully. A shipment lookup still returns nothing.

That is not a hypothetical combination. The OPS-03 fixture contains a schema-only archive with a stable SHA-256 identity. PostgreSQL restores it without error. The tables and constraints exist. The application probe then asks for shipment-100, receives no row, and rejects the recovery.

The checksum answered which bytes we selected. The restore command answered whether PostgreSQL could apply them. Neither answered whether the recovered database could do useful work.

Rollback Cannot Restore a Contract You Already Removed

Wed, 01 Oct 2025 00:00:00 +0000

Release A stores a catalogue quote in amount_cents. Release B renames the column to amount_minor and deploys successfully. Every new B process is healthy, so the deployment dashboard looks reassuring.

One A process is still serving traffic during the rolling update. Its next query asks PostgreSQL for amount_cents and fails because the column no longer exists. Restoring the A image produces the same error. The old code returned; the contract it needs did not.

Observability Must Answer the Next Operator Question

Mon, 01 Sep 2025 00:00:00 +0000

A catalogue quote request returns 503 after its price dependency times out. One telemetry bundle produces five uncorrelated log lines, a metric labelled with request ID and raw URL, and three pages. None of those pages names an owner, the affected scope, the hold duration, or even the first query to run. There is plenty of telemetry and very little help.

The other bundle records less and tells the operator more. One page identifies the affected route, region, and release. Its first query measures the failure users can see. From there, an exemplar selects a failed trace, the trace locates the dependency wait, and a correlated event names the application failure class. The runbook then asks the question which actually matters: is it safe to bypass that dependency?

A Code Review Should Test the Decision, Not the Author

Thu, 01 Aug 2019 00:00:00 +0000

One booking flow could trigger the same automatic wallet refill twice. The first refill happened while booking. A later debit event ran the ordinary refill check again.

The correction wrote a short-lived marker per account containing the pending amount. When a matching shipment debit arrived, it consumed the marker and skipped one refill. Focused tests covered the expected sequence.

That is enough to begin a review, not finish one. A tidy helper and a green happy-path test can make the change look settled. The decision is not “is this cache helper elegant?” It is:

A Cache Must Distinguish Empty from Unavailable

Mon, 01 Oct 2018 00:00:00 +0000

A cached search returns two pickup locations. One location is deleted locally, but the search keeps returning both.

The tempting repair is a shorter time to live. That trades a known correctness bug for a shorter known correctness bug. The system already knows the cached answer is false; waiting five minutes instead of an hour does not make it less false. A TTL is not invalidation with worse timing.

The opposite failure looks similar. A remote lookup returns nothing during an outage. Replacing the cache with an empty list would remove locations which may still exist.

Partial Work Needs a Child Ledger

Sat, 01 Sep 2018 00:00:00 +0000

A queue job receives 1,000 events in one remote response. It dispatches 600 child jobs, then times out.

The parent row says processing failed. Some children have already finished. Others are waiting in the queue. The remaining 400 were never dispatched. A counter can report 573 completed events, but it cannot say which 573 they were.

Running the parent again may duplicate the first 600. Leaving it alone loses the last 400. Neither choice is defensible because the system retained a total and discarded the membership behind it.

An Idempotency Key Is Not an Identity

Fri, 01 Jun 2018 00:00:00 +0000

A booking test sends the same request twice with the same idempotency key. The remote booking service is called once, and both callers receive the same successful response. Then one field changes while the key stays the same. The remote service is still called once, but the second request receives a conflict instead of the first result.

The key did not establish sameness. It only gave the application somewhere to look.

A Third-Party API Is an Unreliable Collaborator

Thu, 01 Feb 2018 00:00:00 +0000

A scheduled import begins by listing remote orders. The list request returns a 502 HTML page, so there are no trustworthy item identifiers to process.

On another run the list succeeds. One order then returns a 504 while its details are being loaded. The remaining orders are still independent and can continue.

These failures share a provider, but not a recovery boundary. Calling both “the API is down” is convenient and useless: it either discards work which could continue or pretends the response established more than it did.

A Release Gate Must Fail Before Incompatible Code Goes Live

Tue, 01 Nov 2016 00:00:00 +0000

We had a deployment step everybody knew about and people still forgot it. The code was pulled manually, then someone had to run a shell script which applied pending migrations. The instructions existed, but the habit remained concentrated in one operator.

Eventually a deployment completed without the script. Traffic reached code which expected columns or tables that did not exist, and database queries began failing. Running the migrations restored the application. There was no obscure database bug here; the release process had declared success before one of its prerequisites was true.

Retries Can Multiply the Original Failure

Mon, 01 Aug 2016 00:00:00 +0000

We once had roughly 50,000 failed jobs sitting in queues and trying again every 60 seconds. Some represented a provider outage. Others contained malformed input which could wait for three months and still be malformed. The queue made no distinction and had no terminal condition.

During a long outage, workers spent their time reproducing failures we already understood. Redis filled with delayed work, fresh operations had to compete with old ones, and a dependency problem became sustained load on our own application. The retry loop was meant to make work reliable. What it actually made durable was failure.

A Timeout Is Part of the Contract

Fri, 01 Jul 2016 00:00:00 +0000

One accounting integration went down often enough that we stopped being surprised by it. Sometimes it came back with an expired TLS certificate. A different provider scheduled maintenance during our peak operating hours. Neither system became more reliable just because our code was willing to wait.

Waiting longer was not resilience. It turned someone else’s outage into our capacity problem while leaving the caller with the same unanswered question:

How long may this attempt wait before its result is no longer useful?

That answer belongs to the operation, not the HTTP library. If we leave it to a library default, a transport setting decides how much of the customer’s time and our worker capacity one dependency may consume.