Disaster-Recovery on Brian Faust

Recover the First Useful Service, Not Every Server

Thu, 01 Oct 2026 00:00:00 +0000

Six components sit on the recovery checklist: a database, queue, cache, carrier adapter, identity service, and API. You can follow that checklist exactly, restore the first four without error, and still leave every customer unable to retrieve a shipment.

The OPS-05 fixture gives every recovery action a deterministic duration. Inventory order reaches an accepted customer journey at 540 virtual milliseconds. Restoring identity, shipment data, and the API first reaches a bounded read-only journey at 240. Both schedules finish all six components at 540.

Replication Copies Corruption Too

Tue, 01 Sep 2026 00:00:00 +0000

The primary answers SELECT 1. So does its streaming standby. Their WAL positions show that the standby has caught up, and PostgreSQL promotes it without error. shipment-100 is missing from both.

Every infrastructure signal in that sequence can be true at the same time. The OPS-04 fixture makes it happen with PostgreSQL 18.1: a valid DELETE commits on the primary, reaches the physical standby, and survives promotion. Failover restores a writable endpoint, but the deleted shipment is still deleted. Replication worked perfectly. That is the problem.

A Backup Is Only an Input to Recovery

Sat, 01 Aug 2026 00:00:00 +0000

The backup job is green. The archive exists, its checksum matches, and pg_restore exits successfully. A shipment lookup still returns nothing.

That is not a hypothetical combination. The OPS-03 fixture contains a schema-only archive with a stable SHA-256 identity. PostgreSQL restores it without error. The tables and constraints exist. The application probe then asks for shipment-100, receives no row, and rejects the recovery.

The checksum answered which bytes we selected. The restore command answered whether PostgreSQL could apply them. Neither answered whether the recovered database could do useful work.