A Release Gate Must Fail Before Incompatible Code Goes Live

We had a deployment step everybody knew about and people still forgot it. The code was pulled manually, then someone had to run a shell script which applied pending migrations. The instructions existed, but the habit remained concentrated in one operator.

Eventually a deployment completed without the script. Traffic reached code which expected columns or tables that did not exist, and database queries began failing. Running the migrations restored the application. There was no obscure database bug here; the release process had declared success before one of its prerequisites was true.

Of course a remembered step was eventually missed. People are dealing with the release, an incident, a deadline, or three other things at once. The durable fix is to make incompatible code impossible to activate until the required schema and data state can be observed.

A checklist records intent; a gate controls state

Written instructions can explain why a step exists, how to inspect it, and what to do when it fails. They cannot stop a deployment which never reads them.

The legacy deployment effectively said:

pull new code
restart application
remember to run post-deploy script

The system reached “deployed” after the second line. The third was a request to a person, not a condition of activation.

A release gate reverses the relationship:

prepare release
run required transitions
verify required state
activate release only if verification passes

If the transition fails, the release remains inactive. If verification cannot run, the state is unknown rather than successful. That may be inconvenient during a release; serving incompatible code is considerably more inconvenient.

The distinction survives changes in tooling. A managed deployment service, shell script, container rollout, and hand-built process can all enforce the same state machine. Automation helps because it executes consistently, but the invariant matters more than the logo on the deployment screen.

Name the dependency between code and state

“Run migrations” is too broad to describe the actual release risk. A useful plan says which code requires which state:

new reader requires column shipment.status_reason
new writer requires table booking_attempts
background operation backfills existing rows
old reader does not understand the new enum value

The dependency determines safe ordering.

If new code reads a new nullable column but old code ignores it, the schema can often arrive before activation. If a migration removes a column still read by the old release, it cannot safely run while old processes remain. If a backfill touches millions of rows, coupling it to the web deployment may exceed the release window even though the final schema is required later.

The gate should therefore ask a concrete question:

Is every prerequisite for the code being activated complete and compatible
with every code version which may still be running?

That is stricter than asking whether a command exited zero. A successful command may have applied only part of a multi-stage transition, and the release may depend on a data operation which deliberately is not a schema migration.

Zero downtime creates an overlap contract

A zero-downtime deployment normally means old and new processes overlap. The release directory may switch atomically, but workers, in-flight requests, and several application nodes can continue executing the previous code.

Schema changes must tolerate that overlap:

expand:
    add state understood safely by old code

migrate:
    write or backfill both representations as needed

switch:
    activate code which uses the new representation

contract:
    remove old state only after old code can no longer execute

Not every change needs four releases. The model exists to expose when one change cannot honestly be compressed into one activation.

For the original incident, the invariant was simpler: the required schema had to exist before new code served traffic. Moving to a managed zero-downtime process made the migration part of the release and stopped the omission. That worked, but it does not mean every migration belongs inside every deployment forever.

Automatic is a policy, not a permanent answer

A retained deployment history shows the migration step moving between enabled and disabled states. Data operations were also enabled, paused for releases which needed separate verification, and enabled again. The current path leaves schema migrations separate while processing approved application operations synchronously.

That history is not indecision. It is the normal consequence of the release risk changing over time:

automatic transition
  advantage: omission blocks the release instead of reaching production
  risk: a slow, unsafe, or irreversible transition sits on the activation path

separate transition
  advantage: dangerous work gets its own timing, observation, and approval
  risk: activation can outrun it unless an explicit gate checks completion

Automatic is not always safe and manual is not always irresponsible. The bad state is separation without a machine-observable dependency.

If migrations are intentionally disabled in the deploy script, the release needs another source of truth:

required transition set
completed transition set
compatibility state
approval or ownership
activation decision

A comment beside a disabled command explains why it was disabled. It proves nothing about whether another process completed the work.

Fail fast before side effects accumulate

The successor deployment script fails fast. When a required command exits unsuccessfully, later steps do not continue as if the release were still going according to plan.

That control matters because deployment steps have side effects:

install dependencies
run transitions
change active release
reload processes
restart workers
clear or warm caches

Continuing after an early failure can combine old dependencies, new code, partial schema, and restarted workers into a state nobody designed. Finishing the script is not a useful achievement if the result is nonsense.

Fail-fast behaviour is necessary but incomplete. Commands can return zero while leaving the wrong state, and some tools call warnings or skipped work a success. Consequential steps need state verification:

command: apply required schema transitions
verify:  required transition identifiers are recorded

command: run approved data operation
verify:  operation reached terminal success

command: activate release
verify:  health check exercises a dependency on the new state

The check should observe the contract rather than trusting the command which was supposed to create it.

Concurrency needs one transition owner

When several application nodes deploy together, all of them may try to apply the same migration. Some database operations are naturally idempotent; many are not safe to race.

Current Laravel documentation describes migrate --isolated as using an atomic lock so only one server runs migrations, provided the servers share a supported cache backend.

That solves concurrent ownership for the command. It does not make a migration compatible, fast, reversible, or safe under load. It also does not prove that every node waits for the winner before activating dependent code.

The release plan still needs:

one transition owner
all activations wait for the result
failure is visible to every participant
lock loss and timeout have defined outcomes
the resulting state is verified

An isolated command is one mechanism inside that plan, not a plan by itself.

A release needs more states than success and failure

The DBG-11 fixture models three plans:

automatic:
    transition -> verify -> activate

separated:
    transition completed elsewhere
    verify recorded evidence -> activate

unsafe:
    transition omitted -> activate

The automatic plan activates because its prerequisite becomes observable. The separated plan activates only after fixture evidence names the completed transition and compatible overlap. The unsafe plan is rejected before activation.

This small model uses release states which operational tools often blur:

prepared
blocked
transitioning
ready
active
failed

“Blocked” does not mean the release failed. It means the release has not earned activation yet, which is far safer than treating a missing decision as success.

The fixture does not run a database, web server, Laravel, or deployment platform. It proves the declared state transitions and rejection of missing evidence, not migration duration, locking, rollback, or production compatibility.

Recovery should not depend on the original operator

Recovery in the original incident was direct: run the omitted migrations and the application could query the expected schema. That was possible because we recognized the mechanism.

A useful recovery record should let another operator reach the same decision:

This is where a runbook earns its place. It starts from an observable symptom and leads to a bounded decision instead of asking the reader which private script one person usually remembers to run.

The original instructions were not worthless. We had simply assigned them too much authority. Documentation should explain and support a release control, not impersonate one.

Make exceptions expire

There are good reasons to remove a transition from the ordinary deployment path:

• it cannot complete within the release window;
• it carries destructive or irreversible risk;
• it needs a backup, replica check, or maintenance window;
• it must be observed before application activation;
• it changes data semantics which require business approval; or
• it needs staged backfill and compatibility releases.

Each exception should name the affected release, owner, prerequisite evidence, activation rule, recovery plan, and removal condition. Without those fields, “temporarily disabled” becomes a second deployment process with no end date.

The retained history prevents a simplistic lesson. Automatic migrations fixed a repeated human omission. Later, separating some transitions became rational because their risk needed independent verification. The invariant stayed the same: incompatible code must not become active merely because the deployment command reached its final line.

A release is ready when its dependencies are true, not when somebody has finished typing. Put recurring work in the path, stop on failure, verify the result, and require explicit evidence whenever risky work moves outside that path.