Testing on Brian Faust

Testing the Go Port Against PHP

Fri, 23 Oct 2026 00:00:00 +0000

A Go function can return the same failure code as the PHP function and still behave differently in the application. Laravel might replace the exception before the PHP resolver sees it. A database reader might fail while loading a relationship that a direct unit test supplied as an already populated struct. Neither difference appears in a test which compares only the final price.

The rating engine’s migration checks therefore needed more than translated assertions. They needed executable observations from PHP, a clear point at which to compare them with Go, and enough detail to explain a disagreement. Otherwise, “the implementations match” could mean little more than “both tests contain the same assumption.”

Retire the Browser, Keep the Test Runner

Thu, 15 Oct 2026 00:00:00 +0000

The Go port deliberately dropped the browser UI. The API tester was the awkward exception: not its pages, but the program behind them. It could create a test configuration, publish it, make real API calls, extract identifiers, wait for a result and clean up afterward. Deleting its screen was straightforward. Deleting all of that would have thrown away a useful part of the migration.

Replacing it with a CLI also exposed distinctions the screen could make easy to overlook. An HTTP 200 response could mean a failed test step. A failed run could still be a successfully produced command result. A saved step record didn’t mean its API call would never execute again. Those weren’t reasons to redesign the tester during the port; they were behavior to understand before moving its entry point.

A Rewrite Can Change Data Without Changing the Price

Thu, 01 Oct 2026 00:00:00 +0000

The Go history append preserved the existing JSON. That looked reasonable: read a stored transition list, add the new transition, leave everything else alone. Unfortunately, leaving everything else alone wasn’t what the PHP implementation did.

The shared idempotency updater in my rating engine read history through an Eloquent array cast, appended a transition, and assigned the resulting value back to the model. That round trip could turn an old nested {} into []. The Go translation preserved {} instead. Both could complete the operation and return the same price while storing different histories.

A PHP Test File Isn't One Migration Case

Tue, 01 Sep 2026 00:00:00 +0000

One PHP file contained two test bodies and eleven migration cases. Counting it as one file would hide ten of them. Counting its two it(...) declarations would still miss most of the work. Copying the first assertion into a Go test would look plausible and preserve almost nothing of the original matrix.

The file covered customer lifecycle states in the pricing application. One body tested eight transitions; another tested three sets of state predicates. It was small enough to follow completely, which makes it a useful way into the much larger problem: using an existing PHP suite to constrain a Go port without pretending that translating test syntax proves equivalent behavior.

Is Your Benchmark Doing New Work?

Wed, 01 Jul 2026 00:00:00 +0000

One of the rating engine’s early benchmark fixes made the results slower. That was the improvement. The runner had reused request identities across invocations, so later runs could return stored results instead of evaluating new quotes. The endpoint was doing its job. The benchmark wasn’t measuring the job I thought it was.

This is an awkward mistake because the output can look entirely healthy: successful HTTP responses, small latencies, no obvious application errors. It is also worth dealing with before celebrating a faster index or cache read . A replay can skip the calculation whose improvement you’re trying to measure.

The Order of Charges Changes the Price

Wed, 15 Apr 2026 00:00:00 +0000

Two row orders can produce the same €35 quote while disagreeing about the fuel charge. In one, fuel is €2.70; in the other, it’s €2.40. A minimum-charge top-up hides the difference in the total. Assert only that total and the test passes either way.

The difference comes from when the fuel row sees a handling charge. The percentage reads the components already built, not a final basket reconstructed after every row has run. Scope also limits which of those components it can use. A shipment percentage doesn’t automatically include parcel charges just because they contribute to the same quote.

A Design Is Good for a Change, Not in Isolation

Wed, 01 May 2024 00:00:00 +0000

Two delivery-option implementations return the same quotes.

One has a DeliveryService interface, a class for each service, and a tagged Laravel registry. The other has a validated rule table and one evaluator. The registry looks extensible. The table looks compact.

Which design is better?

You cannot answer yet. “Extensible” names a possibility, not a requirement. “Compact” counts structure, not the decisions a future change will disturb.

Apply a locker-delivery requirement and the table absorbs the change in one policy definition. Reset to the original baseline, add capacity-aware same-day delivery, and the table has to grow a behavior interface, a collaborator, and new failure semantics. Its old advantage belonged to one kind of change.

A Characterization Test Is a Boundary, Not a Recording

Mon, 01 Apr 2024 00:00:00 +0000

An undocumented postage calculator needs to be split apart. Before touching it, its current behavior goes under test.

One suite stores the returned quote, every intermediate calculation, the order of the diagnostic fields, and the sequence in which four pricing rules run. Another asserts the caller’s result across weight, insurance, destination, and discount boundaries.

Both suites are green. Both characterize the current implementation. They do not buy the same freedom to change it.

A Surviving Mutant Is a Question, Not a Score

Fri, 01 Mar 2024 00:00:00 +0000

Two PHPUnit suites are green. The first checks a known product. The second also checks that a missing product throws ProductNotFound.

Run two mutations against them. The extra assertion kills the missing-exception defect, exactly as intended. Yet a 100% mutation-score gate rejects both suites. The stronger suite finishes at 50% because a second mutation changes no behavior for any input the method accepts.

The gate has calculated its number correctly. It has not interpreted the survivor. That distinction is the whole problem: the stronger suite is better, but the dashboard makes it look half finished.

An Interface Does Not Make a Fake Substitutable

Thu, 01 Feb 2024 00:00:00 +0000

An HTTP product catalogue and an in-memory fake both implement this interface:

interface ProductCatalogue
{
 /** @throws ProductNotFound */
 public function priceCents(string $sku): int;
}

Both return 10000 for CHAIR-BLACK. Both make the checkout test green. Both pass PHP’s signature checks.

Now seed one divergence. When the product is absent, the HTTP adapter throws ProductNotFound while the fake returns zero. The interface remains satisfied, and the familiar checkout test still passes because it never asks for a missing SKU.

A Mock Fortress Tests a Call Script

Mon, 01 Jan 2024 00:00:00 +0000

Two tests produce the same checkout quote. One builds five Mockery doubles, specifies every argument, fixes their global call order, and makes a serializer mock return the expected JSON. The other resolves the use case through Laravel’s container, replaces one HTTP catalogue, and asserts the JSON emitted by the real policies and serializer.

Both are green. That tells us they agree with today’s code. It does not tell us what either test will reject tomorrow. Five mocks can look impressively precise while carefully excluding the code most likely to be wrong.

Choose the Test Boundary from the Defect

Fri, 01 Dec 2023 00:00:00 +0000

Four test boundaries protect the same reservation flow. Then four defects arrive:

a quantity of zero becomes valid;
a database migration loses a uniqueness constraint;
an inventory provider renames available to stock; and
browser JavaScript submits an empty string instead of the entered quantity.

Calling all four tests “feature tests” would tell us almost nothing. Putting them into a pyramid would tell us how many tests we hope to own, but not which one can catch each defect.

Coverage Tells You What Ran, Not What Your Tests Would Catch

Wed, 01 Nov 2023 00:00:00 +0000

Here are two PHPUnit suites for the same shipping-fee rule. Both report 100% line coverage. Both also report 100% branch and path coverage. One suite lets an off-by-one defect escape; the other catches it.

Nothing is wrong with the coverage report. It did its job. The mistake is turning a comforting number into an answer to a question it cannot answer.

Coverage can tell us which executable structure a test run visited. It cannot tell us whether the chosen inputs and assertions would notice a defect that matters. That sounds like a small distinction until a green suite and a perfect score ship the wrong price.

Construction Should Not Hide Durable Writes

Tue, 01 Sep 2020 00:00:00 +0000

A test called a model factory’s make() method. The model itself was never saved. Two related records were.

The factory computed the model’s default address and account fields by creating real database rows. A test which only needed an in-memory user therefore changed shared state before it reached its first assertion.

A surrounding test transaction usually hid the damage by rolling it back. Tests without that transaction left the rows behind, while another test used TRUNCATE for cleanup, committed the MySQL transaction, and made earlier writes survive the rollback. The method returned an unpersisted object, but its name still gave the wrong account of what had happened.

Global State Makes Local Reasoning Impossible

Sat, 01 Aug 2020 00:00:00 +0000

A repository test created a shipment two days before now, then counted shipments between the start and end of the current month. It passed most days. During the first two days of a month, “two days ago” belonged to the previous month and the count became zero.

Nothing in the test name mentioned a calendar boundary, and nothing in the method signature mentioned time. The same test logic produced a different result because its generated data moved with one more input: the instant at which the test ran. Freezing the clock in the middle of the month stabilized the example. It did not make the dependency local.

Dependencies Should Be Visible at Construction

Mon, 01 Jun 2020 00:00:00 +0000

This line created a database record:

$user = UserFactory::new()->create([
 'external_id' => 123,
]);

It also performed an HTTP request.

The factory did not mention a client. The model did not accept one. A synchronous observer noticed the external identifier, prepared a profile export, and resolved a connector from global framework state. The connector then looked up a remote account.

In tests without a fake, that lookup reached DNS or the network and retried three times with exponential backoff. Tests which had run in under five seconds took more than thirty. The slow test was annoying, but it was only the visible symptom. A local-looking construction operation had acquired an undeclared remote dependency.

A Rewrite Needs a Behavior Ledger

Wed, 01 May 2019 00:00:00 +0000

A desktop rewrite had screens for shipments, reports, settings, and logs. If we had tracked the work as a feature checklist, all four could have been marked complete and everyone could have gone home feeling productive.

The old application also had behavior no screen list would reveal:

a failed temporary-file cleanup must not turn a submitted print into a failure, or the next run may print the label again;
the first run treats older unprinted records differently from later runs;
setup unlocks when a credential setting exists, not when its value is non-empty; and
a visible report search box does nothing.

Some of those rules protect operators. One is an awkward contract. One is probably a dead feature. Calling all of them “parity” hides the decision we actually need to make. A rewrite needs a behavior ledger: a finite record of what must survive, what will change deliberately, what may be dropped, and what evidence closes each decision.

A Characterization Test Is Not a Product Decision

Fri, 01 Mar 2019 00:00:00 +0000

A label setting behaved differently in test and production. Some customers had come to depend on the difference even though the product already let them choose the label size they wanted.

A characterization test could record both defaults. It could tell us which callers would notice if we made them equal. It could not answer the important question: should the difference survive?

Keeping it would protect customers who had never configured the setting. Removing it would give every environment one rule and make the customer’s choice explicit. The same observation supported arguments for either response. Only one response matched the product we wanted to operate.

A Characterization Suite Is Scoped to a Change

Fri, 01 Feb 2019 00:00:00 +0000

A booking endpoint had more than thirty characterization cases. They covered successful bookings, validation failures, wallet rules, locks, pending work, email routing, and several provider response shapes.

The next refactor removed one ambient HTTP dependency from the booking services. The suite looked broad enough. It was not broad in the direction of the change.

One service read the global request to persist the original payload. Replacing that read with the existing booking data object would keep ordinary responses green while silently storing normalized values where operators expected the raw input. The suite already protected a test-mode short circuit. It needed a new case for the two representations.

A Capability Interface Is Smaller Than a Vendor Client

Sat, 01 Dec 2018 00:00:00 +0000

A controller needs to register an order webhook. Instead, it constructs a vendor SDK connection, accesses a dynamic resource property, calls its creation method, names the remote event, chooses response fields, and catches the SDK’s base exception. That is a lot of vendor vocabulary for one product decision.

The test is no better. It has to intercept construction of that concrete connection and mock a dynamic property chain. Two useful behavior tests ended up skipped with the same explanation: the code needed an interface and dependency injection.

Test Cannot Prove Production When It Runs Different Rules

Sat, 01 Oct 2016 00:00:00 +0000

A label-layout change passed in test and disrupted document distribution in production. The same revision was deployed to both environments and the input looked equivalent. Our test still could not predict what customers received.

We had built that disagreement on purpose. Test loaded one default and production loaded another because some users had never configured the layout they wanted. The option existed; our environment branch kept compensating for the omission. A new recommendation passed against the test rule while production quietly ran a different product decision.

Production Bugs Can Live in the Data, Not the Code

Mon, 01 Feb 2016 00:00:00 +0000

A delivery integration returned a label, but the application could not save it. The same persistence path had passed with the fixtures used elsewhere.

The failed label was not malformed. It was simply larger than the database column could hold after Base64 encoding.

That distinction changes the investigation. Comparing deployed revisions would find nothing because the revision was not the variable. Repeating a familiar fixture would also find nothing because the fixture did not cross the same storage boundary. The investigation had treated “same code” as “same experiment” while leaving the input that decided the result uncontrolled.

Reproduce the Failure Before Repairing It

Fri, 01 Jan 2016 00:00:00 +0000

Submit two different browser error reports. The application accepts both requests, sends both reports, and records the same tracing identifier for each.

Nothing crashed during submission. No assertion failed. The defect appears later, when someone searches the logs and finds two executions presented as one trace.

The tempting response is to replace the identifier generator immediately. The collision already points in that direction. But a plausible diagnosis is not a reproduction. Before changing the generator, we need a check which states what was observed, what should differ, and which surrounding machinery does not matter.