Operations on Brian Faust

Rollback Cannot Restore a Contract You Already Removed

Wed, 01 Oct 2025 00:00:00 +0000

Release A stores a catalogue quote in amount_cents. Release B renames the column to amount_minor and deploys successfully. Every new B process is healthy, so the deployment dashboard looks reassuring.

One A process is still serving traffic during the rolling update. Its next query asks PostgreSQL for amount_cents and fails because the column no longer exists. Restoring the A image produces the same error. The old code returned; the contract it needs did not.

Observability Must Answer the Next Operator Question

Mon, 01 Sep 2025 00:00:00 +0000

A catalogue quote request returns 503 after its price dependency times out. One telemetry bundle produces five uncorrelated log lines, a metric labelled with request ID and raw URL, and three pages. None of those pages names an owner, the affected scope, the hold duration, or even the first query to run. There is plenty of telemetry and very little help.

The other bundle records less and tells the operator more. One page identifies the affected route, region, and release. Its first query measures the failure users can see. From there, an exemplar selects a failed trace, the trace locates the dependency wait, and a correlated event names the application failure class. The runbook then asks the question which actually matters: is it safe to bypass that dependency?

A Migration Is a Production Change

Wed, 01 Mar 2017 00:00:00 +0000

We deployed an additive schema change which the database could execute without rewriting the table. On paper it was cheap. In production it waited indefinitely.

The statement needed an exclusive table lock. A long-running session already held a conflicting lock, so the migration queued. New application traffic then queued behind the migration. The column addition was cheap after lock acquisition; waiting to begin was the incident.

Another schema change looked cheap for the opposite reason: remove fields and tables which the application no longer read. The SQL was small, but the decision would erase financial history from data sets measured in tens of gigabytes. Neither change could be judged by reading the migration method alone.

Logging Is a Workload, Not a Side Effect

Tue, 01 Mar 2016 00:00:00 +0000

One system was writing roughly a thousand log lines per second to local disk. Debug records were part of that stream. The individual call sites were small; the combined stream was not.

It was not cheap in aggregate. Every record still had to be constructed, serialized, copied through handlers, written, rotated, and eventually removed or archived. Whether anyone queried it later did not change that work. Calling the record debug did not exempt it from arithmetic.