Debugging on Brian Faust

Debugging Ends with an Explanation, Not a Green Build

Thu, 01 Dec 2016 00:00:00 +0000

A cryptography-heavy application kept its CPUs near 100 percent. We found several cryptographic paths worth improving, so the diagnosis felt obvious: expensive mathematics was consuming the machine. How could logging possibly be more expensive than the work the application existed to do?

Then we measured the log stream. It was writing roughly a thousand lines per second to local disk. We reduced the emitted logging without changing the cryptographic work, and CPU use settled around 10–20 percent.

Test Cannot Prove Production When It Runs Different Rules

Sat, 01 Oct 2016 00:00:00 +0000

A label-layout change passed in test and disrupted document distribution in production. The same revision was deployed to both environments and the input looked equivalent. Our test still could not predict what customers received.

We had built that disagreement on purpose. Test loaded one default and production loaded another because some users had never configured the layout they wanted. The option existed; our environment branch kept compensating for the omission. A new recommendation passed against the test rule while production quietly ran a different product decision.

A Correlation Identifier Is a Boundary Contract

Thu, 01 Sep 2016 00:00:00 +0000

We had logs everywhere and still could not follow one shipment through the system. It had failed somewhere after payment and before its documents were ready, but the application, queue worker, and remote adapter did not share an identifier.

The investigation started with timestamps, account details, and values which looked related. Payment was plausible, as were several remote calls nearby. Some trails took hours to disprove because they belonged to different requests which happened at roughly the same time. We did not need more log lines. We needed a chain of custody.

A Failure Timeline Can Eliminate the Best-Looking Suspect

Fri, 01 Apr 2016 00:00:00 +0000

A shipment appeared in the local database without the tracking number or label that made it useful. Payment was the first suspect because it was known to be brittle, and the visible result looked like an operation which had stopped halfway through.

But payment had completed. The remote booking had completed too. The failure happened while persisting the response, and a broad catch continued without reporting it.

Payment was guilty by reputation, not by the state in front of us. Ordering the effects made that mismatch visible before another log line was added.

Production Bugs Can Live in the Data, Not the Code

Mon, 01 Feb 2016 00:00:00 +0000

A delivery integration returned a label, but the application could not save it. The same persistence path had passed with the fixtures used elsewhere.

The failed label was not malformed. It was simply larger than the database column could hold after Base64 encoding.

That distinction changes the investigation. Comparing deployed revisions would find nothing because the revision was not the variable. Repeating a familiar fixture would also find nothing because the fixture did not cross the same storage boundary. The investigation had treated “same code” as “same experiment” while leaving the input that decided the result uncontrolled.

Reproduce the Failure Before Repairing It

Fri, 01 Jan 2016 00:00:00 +0000

Submit two different browser error reports. The application accepts both requests, sends both reports, and records the same tracing identifier for each.

Nothing crashed during submission. No assertion failed. The defect appears later, when someone searches the logs and finds two executions presented as one trace.

The tempting response is to replace the identifier generator immediately. The collision already points in that direction. But a plausible diagnosis is not a reproduction. Before changing the generator, we need a check which states what was observed, what should differ, and which surrounding machinery does not matter.