Test Cannot Prove Production When It Runs Different Rules

A label-layout change passed in test and disrupted document distribution in production. The same revision was deployed to both environments and the input looked equivalent. Our test still could not predict what customers received.

We had built that disagreement on purpose. Test loaded one default and production loaded another because some users had never configured the layout they wanted. The option existed; our environment branch kept compensating for the omission. A new recommendation passed against the test rule while production quietly ran a different product decision.

Nothing mysterious happened during deployment. We had called two different behaviours the same test.

Credentials, endpoints, capacity, and representative data will differ between environments. The business decision a test claims to prove should not change just because the deployment has another name.

The same revision is not the same program

Source control tells us which instructions were deployed. Runtime inputs decide which program those instructions become.

Consider a small rule:

if environment is test:
    default layout = compact
else:
    default layout = standard

The revision is identical in both places. The effective program is not:

test input       -> test branch       -> compact
production input -> production branch -> standard

A successful test proves only the first path. It says nothing about the second unless another test deliberately executes the production branch. This is how “works in test” and “the same commit is deployed” can both be true while the result is still wrong. Neither statement establishes behavioural parity.

Separate deployment configuration from business choice

Environments genuinely need different values:

database address
credentials
queue capacity
log destination
remote sandbox endpoint
feature rollout percentage

Those values connect and operate the application. They should not casually change the meaning of a booking, invoice, or label request.

Business choices look different:

which label layout wins
which tax rule applies
whether an order is eligible
how a shipment is routed
when a payment is considered settled

Put the second group behind an environment check and we have done more than configure a deployment. We have created an environment-specific product.

Sometimes that is intentional. A destructive payment call should reach a sandbox from test, a staged rollout may expose new behaviour to a bounded audience, and a simulator may need failure modes which production does not. Those differences should be named, observable, and tested as differences.

The dangerous form is an unnamed accommodation:

production needs this odd default for old users
test needs the new default so current work can proceed

At that point test is no longer a detached representation of production behaviour. It is a second policy implementation, and pretending otherwise only makes the eventual surprise more expensive.

Defaults are decisions with invisible callers

An explicit setting has an owner. A default collects callers without recording them.

Suppose layout selection appears to work like this:

explicit request
stored user preference
system default

If a user sends neither an override nor a stored preference, the system default becomes part of that user’s effective contract. Changing it can therefore change output for every silent caller.

Defaults are not inherently bad. A system needs a final answer when omission is valid. The mistake is pretending a widely observed default remains internal just because nobody selected it explicitly.

The label case had an obvious warning sign: users could configure the required layout and some simply had not. Environment-specific defaults absorbed that ambiguity for a while, but each accommodation made the result harder to predict:

same request
same stored data
same revision
different environment
different document

Keeping that state forever would force every later recommendation, downstream change, and regression test to negotiate two implicit products. That is a lot of permanent complexity to avoid asking someone to choose a setting once.

One resolver should own precedence

The cleaner design is one decision function whose inputs are business facts:

resolve(
    requested layout,
    stored preference,
    allowed layouts,
    driver default
)

Its precedence can be explicit:

1. use the requested layout when it is allowed;
2. otherwise use the stored preference when it is allowed;
3. otherwise use the driver’s declared default.

Environment is not an input.

A retained implementation follows that shape. Its tests cover a request override, stored preference, disallowed values, and a driver fallback. The resolver reads the allowed layouts for the selected service; it never asks whether it is running in test or production.

That does not prove the surrounding systems are identical. Databases, provider sandboxes, fonts, printers, and customer data can still differ. It proves the narrow invariant we actually need here: the same resolver inputs produce the same layout regardless of the deployment name.

The DBG-10 fixture runs one synthetic decision matrix under development, test, and production labels. A mutation adds a production-only default and the matrix immediately rejects it.

Allowed values are policy, not an escape hatch

A user preference should not necessarily beat every other input. A selected layout may be invalid for the chosen service or unsupported by the downstream system.

That gives the resolver two independent jobs:

precedence: which source should win?
validity:   may this value be used here?

Conflating them produces another kind of accidental default. If the explicit request is invalid, blindly accepting it can fail downstream. If the resolver silently replaces it, the user may receive a document they did not ask for.

The retained resolver gives an allowed explicit request highest priority, then an allowed stored preference, then the driver fallback. A nullable form can also report that neither explicit nor stored input is usable before the caller applies a fallback.

The precise failure behaviour is a product choice. An API may reject an unsupported layout, a user interface may prevent the selection, and a legacy import may need a compatibility mapping. The parity rule does not choose among them. It requires every environment to execute the chosen rule for the same facts.

Representative data still matters

Removing environment branches does not make test equivalent to production. Production may still contain preferences which test never seeds, old rows with values the current interface cannot create, or inputs missing from ordinary fixtures. This series has already covered failures which live in data rather than code.

The two problems need different repairs:

Calling all four cases “environment drift” hides the mechanism. We can only fix the mismatch efficiently once we name which kind it is.

Compatibility has a cost and an owner

We could have kept the old production default, identified every user relying on it, provided a migration window, recorded explicit preferences with approval, and removed the branch after adoption. That was the credible alternative to removing the accommodation immediately.

That approach reduces surprise. It also preserves two behaviors while the migration runs, requires a reliable way to identify silent reliance, and keeps new work coupled to the old exception.

We removed the accommodation without a compatibility window and required users to configure the layout they actually wanted. That created short-term support friction, but it left one explicit rule instead of two permanent implicit ones.

I would not apply that migration policy blindly. Blast radius, contractual promises, the cost of incorrect output, ability to identify affected users, and available support capacity all matter. A high-impact financial rule deserves a different rollout from an already self-service preference.

What should not survive is an unowned environment branch with no removal condition. If compatibility is necessary, give it:

affected population
old and new behavior
selection mechanism
observation
owner
deadline
removal test

Then it is a migration rather than folklore someone has to rediscover later.

Verify parity as a matrix

The smallest useful parity check feeds the same cases to the same resolver under every supported deployment label:

The environment label is deliberately absent from the expected behaviour.

This test will not catch a database loaded with different preferences or a sandbox which renders documents differently. It catches the regression at issue: making a business result conditional on the deployment name.

Architecture checks can help locate direct environment reads in domain code, but source shape is only a warning. A feature-assignment service may read deployment configuration legitimately while exposing an explicit variant to the domain. Conversely, a value loaded indirectly from separate environment files can change behavior without any obvious if production branch.

The behavioural matrix is the proof. Static checks merely narrow the search.

A pre-production environment should be allowed to disagree honestly

Perfect replicas are usually impractical. Test may use smaller infrastructure, sanitised data, fake credentials, and provider sandboxes. Those differences are useful when they are visible and their evidence limits are explicit.

For each material difference, record:

what differs
why it differs
which claim test can no longer make
how production behavior is verified safely
who owns removal or continued acceptance

If the remote sandbox generates a placeholder document, test can still prove request mapping and error handling. It cannot prove the real document’s dimensions. If test has less traffic, it can prove functional selection. It cannot prove production capacity.

That record is more honest than calling an environment “production-like” and letting one adjective absorb every unknown.

The label failure did not make test environments useless. We had asked one product rule to answer differently depending on where it ran, then treated one answer as proof of the other.

Keep deployment concerns configurable and business inputs explicit. Run the same decision function everywhere. When an environment has to differ, name the mechanism and stop claiming that environment proves the behaviour it no longer executes.