Debugging Ends with an Explanation, Not a Green Build

Thu, 01 Dec 2016 00:00:00 +0000

A cryptography-heavy application kept its CPUs near 100 percent. We found several cryptographic paths worth improving, so the diagnosis felt obvious: expensive mathematics was consuming the machine. How could logging possibly be more expensive than the work the application existed to do?

Then we measured the log stream. It was writing roughly a thousand lines per second to local disk. We reduced the emitted logging without changing the cryptographic work, and CPU use settled around 10–20 percent.

A Failure Timeline Can Eliminate the Best-Looking Suspect

Fri, 01 Apr 2016 00:00:00 +0000

A shipment appeared in the local database without the tracking number or label that made it useful. Payment was the first suspect because it was known to be brittle, and the visible result looked like an operation which had stopped halfway through.

But payment had completed. The remote booking had completed too. The failure happened while persisting the response, and a broad catch continued without reporting it.

Payment was guilty by reputation, not by the state in front of us. Ordering the effects made that mismatch visible before another log line was added.

Causality on Brian Faust

Debugging Ends with an Explanation, Not a Green Build

A Failure Timeline Can Eliminate the Best-Looking Suspect