Debugging Real Systems on Brian Faust

Debugging Ends with an Explanation, Not a Green Build

Thu, 01 Dec 2016 00:00:00 +0000

A cryptography-heavy application kept its CPUs near 100 percent. We found several cryptographic paths worth improving, so the diagnosis felt obvious: expensive mathematics was consuming the machine. How could logging possibly be more expensive than the work the application existed to do?

Then we measured the log stream. It was writing roughly a thousand lines per second to local disk. We reduced the emitted logging without changing the cryptographic work, and CPU use settled around 10–20 percent.

A Release Gate Must Fail Before Incompatible Code Goes Live

Tue, 01 Nov 2016 00:00:00 +0000

We had a deployment step everybody knew about and people still forgot it. The code was pulled manually, then someone had to run a shell script which applied pending migrations. The instructions existed, but the habit remained concentrated in one operator.

Eventually a deployment completed without the script. Traffic reached code which expected columns or tables that did not exist, and database queries began failing. Running the migrations restored the application. There was no obscure database bug here; the release process had declared success before one of its prerequisites was true.

Test Cannot Prove Production When It Runs Different Rules

Sat, 01 Oct 2016 00:00:00 +0000

A label-layout change passed in test and disrupted document distribution in production. The same revision was deployed to both environments and the input looked equivalent. Our test still could not predict what customers received.

We had built that disagreement on purpose. Test loaded one default and production loaded another because some users had never configured the layout they wanted. The option existed; our environment branch kept compensating for the omission. A new recommendation passed against the test rule while production quietly ran a different product decision.

A Correlation Identifier Is a Boundary Contract

Thu, 01 Sep 2016 00:00:00 +0000

We had logs everywhere and still could not follow one shipment through the system. It had failed somewhere after payment and before its documents were ready, but the application, queue worker, and remote adapter did not share an identifier.

The investigation started with timestamps, account details, and values which looked related. Payment was plausible, as were several remote calls nearby. Some trails took hours to disprove because they belonged to different requests which happened at roughly the same time. We did not need more log lines. We needed a chain of custody.

Retries Can Multiply the Original Failure

Mon, 01 Aug 2016 00:00:00 +0000

We once had roughly 50,000 failed jobs sitting in queues and trying again every 60 seconds. Some represented a provider outage. Others contained malformed input which could wait for three months and still be malformed. The queue made no distinction and had no terminal condition.

During a long outage, workers spent their time reproducing failures we already understood. Redis filled with delayed work, fresh operations had to compete with old ones, and a dependency problem became sustained load on our own application. The retry loop was meant to make work reliable. What it actually made durable was failure.

A Timeout Is Part of the Contract

Fri, 01 Jul 2016 00:00:00 +0000

One accounting integration went down often enough that we stopped being surprised by it. Sometimes it came back with an expired TLS certificate. A different provider scheduled maintenance during our peak operating hours. Neither system became more reliable just because our code was willing to wait.

Waiting longer was not resilience. It turned someone else’s outage into our capacity problem while leaving the caller with the same unanswered question:

How long may this attempt wait before its result is no longer useful?

That answer belongs to the operation, not the HTTP library. If we leave it to a library default, a transport setting decides how much of the customer’s time and our worker capacity one dependency may consume.

Silencing an Error Does Not Protect the Customer

Wed, 01 Jun 2016 00:00:00 +0000

One of our shipment-booking paths returned success after it had already failed. The remote booking existed and so did a local shipment, but the shipment had no label and no tracking number. From the customer’s point of view, the one thing we had promised to do was not done.

The failure happened while persisting the carrier response. A broad catch swallowed the exception, reported nothing, and let the request continue. This was meant to serve a reasonable operational goal: do not bother customers with errors if we can avoid it. Instead, we gave them a broken shipment and called it a success.

Runtime Configuration Is Process State

Sun, 01 May 2016 00:00:00 +0000

A queued job selected the settings for one business context by overwriting Laravel’s global configuration. The values were correct for that job. The problem was what happened afterward.

The worker stayed alive.

Only a small fraction of the queued job classes selected their context this way. A later job could read a key without first replacing it and inherit whatever the previous job had left behind. Nothing was wrong with the configuration file, environment variables, or deployment. The process contained valid values for the wrong work.

A Failure Timeline Can Eliminate the Best-Looking Suspect

Fri, 01 Apr 2016 00:00:00 +0000

A shipment appeared in the local database without the tracking number or label that made it useful. Payment was the first suspect because it was known to be brittle, and the visible result looked like an operation which had stopped halfway through.

But payment had completed. The remote booking had completed too. The failure happened while persisting the response, and a broad catch continued without reporting it.

Payment was guilty by reputation, not by the state in front of us. Ordering the effects made that mismatch visible before another log line was added.

Logging Is a Workload, Not a Side Effect

Tue, 01 Mar 2016 00:00:00 +0000

One system was writing roughly a thousand log lines per second to local disk. Debug records were part of that stream. The individual call sites were small; the combined stream was not.

It was not cheap in aggregate. Every record still had to be constructed, serialized, copied through handlers, written, rotated, and eventually removed or archived. Whether anyone queried it later did not change that work. Calling the record debug did not exempt it from arithmetic.

Production Bugs Can Live in the Data, Not the Code

Mon, 01 Feb 2016 00:00:00 +0000

A delivery integration returned a label, but the application could not save it. The same persistence path had passed with the fixtures used elsewhere.

The failed label was not malformed. It was simply larger than the database column could hold after Base64 encoding.

That distinction changes the investigation. Comparing deployed revisions would find nothing because the revision was not the variable. Repeating a familiar fixture would also find nothing because the fixture did not cross the same storage boundary. The investigation had treated “same code” as “same experiment” while leaving the input that decided the result uncontrolled.

Reproduce the Failure Before Repairing It

Fri, 01 Jan 2016 00:00:00 +0000

Submit two different browser error reports. The application accepts both requests, sends both reports, and records the same tracing identifier for each.

Nothing crashed during submission. No assertion failed. The defect appears later, when someone searches the logs and finds two executions presented as one trace.

The tempting response is to replace the identifier generator immediately. The collision already points in that direction. But a plausible diagnosis is not a reproduction. Before changing the generator, we need a check which states what was observed, what should differ, and which surrounding machinery does not matter.