Rollback Cannot Restore a Contract You Already Removed

Release A stores a catalogue quote in amount_cents. Release B renames the column to amount_minor and deploys successfully. Every new B process is healthy, so the deployment dashboard looks reassuring.

One A process is still serving traffic during the rolling update. Its next query asks PostgreSQL for amount_cents and fails because the column no longer exists. Restoring the A image produces the same error. The old code returned; the contract it needs did not.

The same failure can survive in less visible places. An A queue worker can receive a B-only payload after the web rollout, and an A process can read a B-only cache value after the application image is restored. Neither queued bytes nor cached bytes care that a deployment controller selected an older image.

A safe deployment is a compatibility window. Every version which can still run must be able to read the state every version can still write. Rollback is one tested transition inside that window, not a universal undo button.

Compatibility belongs to readers and writers

“Backward compatible” is too vague to be useful unless it names four things:

• the reader;
• the writer;
• the state carrier; and
• the direction of the change.

The OPS-02 fixture uses three synthetic releases. A reads and writes amount_cents. B reads either field and temporarily writes both. C reads and writes only amount_minor after the old contract retires.

Those releases encounter four database phases:

B’s reader compatibility does not make its writer compatible with every phase. Before expansion, the new column is absent. After contraction, the old dual-write target is absent. The fixture keeps 27 reader cells across the database, queue, and cache, plus 12 separate database-writer cells, because one friendly green “compatible” label would hide all of those directions.

The releases and quote are controlled examples, not claims about Brian’s systems. The fixture has no production traffic, state size, downtime budget, rollback objective, broker retention, cache behavior, or staffing model.

Expand before a new writer needs the field

The safe database sequence begins by adding amount_minor while retaining amount_cents. A still has the schema it expects. B can now be deployed as a reader which prefers the new value and falls back to the old one.

Only after that expansion does B become a dual writer. Its update changes both amount fields in one database statement. The duplication is temporary and a little ugly, but it buys a period in which A and B observe the same semantic quote while processes are replaced. That is the job, not architectural purity during a rollout.

The destructive control does the opposite and renames the column first. A’s read fails with PostgreSQL error 42703, and its post-rollback write fails with the same code. The controller did its job: it restored the old image. The release design failed because no controller can infer and recreate the schema and values that image expects.

Kubernetes documents that a rolling update gradually replaces old Pods . Its rollout status reports progress through process replacement. That establishes why old and new code may overlap. It does not prove that either version understands the other’s database rows, messages, or cache entries.

The PostgreSQL expansion in the fixture acquires an AccessExclusiveLock for the exercised ALTER TABLE. PostgreSQL’s ALTER TABLE documentation explains that lock level and distinguishes operations which scan or rewrite data from those which can update metadata. The local fixture has three rows. Its elapsed time says nothing useful about a production migration; the DDL form, table size, traffic, lock wait, and version must be evaluated in the real system.

A backfill is a restartable data process

Adding the column makes B executable. Existing rows are still incomplete.

The fixture backfills three quotes in bounded batches. It persists the last completed quote ID. A simulated failure interrupts the run after quote-40; the next run resumes after that checkpoint and finishes at quote-42.

A checkpoint only answers where to resume. It does not make a stale write safe. A can update a quote between the backfill’s read and write: the fixture captures revision 1 of quote-42, then A changes the amount and advances the revision. A guarded update expects the old revision and affects zero rows. Retrying from current state writes the new amount instead of quietly restoring a stale snapshot.

Another control creates a disagreement deliberately: amount_cents is 1100 and amount_minor is 999. B does not silently prefer one. The migration records the row and applies the declared authority rule before continuing.

That leaves a concrete order:

expand schema
deploy compatible reader
enable dual-write
backfill with checkpoint and revision guard
reconcile disagreements
validate the new constraint

The new constraint begins as NOT VALID and becomes validated only after the rows satisfy it. PostgreSQL documents this split as a way to enforce new rows without performing the initial existing-row scan in the same operation. It is a database capability, not permission to skip the later validation gate.

The web rollout does not drain the queue

The fixture retains a v1 message produced by A before B deploys. Its delayed retry becomes available after the web release has advanced. B must still read amount_cents or translate that retained payload.

B also publishes an additive message containing both fields. A and B can read it during the overlap. C eventually publishes a v2-only message. A rejects that message with the named failure missing-amount_cents; C accepts it.

That rejection is terminal for the A consumer. Repeating the same incompatible attempt is just retry theatre. The retained repair action translates with B, preserves the message identity, and retries under a reader which can actually understand it.

Laravel’s queue documentation describes workers as long-lived processes which do not notice code changes until restarted. Its deployment guidance separately requires queue workers, Octane, Reverb, and other long-running services to reload after deployment. Reloading the web fleet is not evidence that the old queue worker stopped, and stopping the worker is not evidence that its delayed payloads disappeared.

Before contraction, the fixture checks an owned process inventory for both web and queue workers, then queries the durable queue for pending v1 payload dependencies. Both A process counts and the pending v1 dependency count must be zero. A healthy replica count says the new processes are alive; it says nothing about old workers or delayed messages.

Cached values are deployed contracts too

A shared cache can keep the old shape alive after every process has restarted, or expose a new shape to old code after an image rollback. Deployments replace processes. They do not travel through Redis rewriting values out of courtesy.

The Redis fixture uses explicit envelopes:

{
  "schema": "additive",
  "quote_id": "quote-42",
  "quote_revision": 3,
  "amount_cents": 1234,
  "amount_minor": 1234,
  "writer_release": "B",
  "expires_at": 400
}

B reads an A-written v1 envelope. A and B read the additive envelope. A rejects a C-written v2-only envelope, while C reads it. An unknown v99 envelope and malformed JSON are classified as incompatible rather than disguised as cache misses.

There are two credible migration shapes. An additive envelope keeps one key while both readers understand the transition. Versioned keys keep v1 and v2 values separate while selection moves between them. Both still need a retirement rule. A version suffix is metadata, not evidence that no old reader can receive traffic.

The fixture removes only the owned v1 and additive quote keys and proves that an unrelated Redis sentinel remains. Laravel’s cache documentation warns that flush ignores the configured prefix. A global flush is therefore not a safe default for retiring one application’s value shape from a shared store.

Restoring A after B or C has written v2 does not delete that envelope. The recovery needs targeted invalidation, expiry, a compatible reader, or a rewrite from the authoritative source.

The rollback target changes during the deployment

“Roll back if it fails” sounds like a plan until someone asks which artifact and which state phase should come back.

The last safe A boundary in the fixture is before any release publishes a new-only artifact. B remains the rollback target through migration because it can read both shapes and operate the expanded schema. Once the old column is removed, dual-write B also fails with 42703; C succeeds because it writes only the new field. The rollback target moved while the deployment progressed.

If A must return after that point, the work is no longer an image selection. A repair release must restore the old schema and reconstruct its values, or adapt A to the state which now exists. Messages and cache values may still need translation or invalidation. Externally observed business effects may require compensation rather than either kind of rollback.

The contract gate therefore asks for evidence from each state carrier:

• no A web or queue process remains;
• no old-only write was observed for the declared window;
• the backfill checkpoint is complete and disagreement count is zero;
• the new constraint is validated;
• no pending v1 payload dependency remains;
• v1 cache values have expired or been removed from the owned key family; and
• every failure action, rollback target, and forward repair has an owner.

A single deployment-safety score cannot replace these facts. Compatibility, reversibility, duration, downtime, data volume, and operating cost can disagree with each other, and averaging them does not make the missing gate disappear.

Sometimes downtime is the cleaner answer

Expand-migrate-contract adds columns, fallbacks, dual writes, checkpoints, queries, translators, deployment stages, and cleanup. It is real work, and it is justified when uninterrupted mixed-version operation matters.

A stop-the-world deployment remains a credible alternative. For a small service with an accepted downtime window, bounded state, one owned worker set, and a tested restoration plan, one coordinated transition may be safer than temporary compatibility logic. This fixture cannot choose between them because it contains none of those production facts.

The reproduction retains 27 reader cells, 12 writer cells, three migrated PostgreSQL rows, five queue outcomes, seven cache outcomes, a 14-stage gate ledger, four score-free state-carrier decisions, 23 rejected counterfactuals, 11 hashed primary sources, and a clean export of commit f7f1b04035039aff012e8d160c3010059381c94b.

Before the next deployment, write the compatibility matrix rather than only the rollback command. Name every live reader and writer, the state each can leave behind, and the evidence which retires the old contract. If the final gate is still “the old code probably won’t run again,” the contract is not ready to be removed.