Integrations on Brian Faust

A Third-Party API Is an Unreliable Collaborator

Thu, 01 Feb 2018 00:00:00 +0000

A scheduled import begins by listing remote orders. The list request returns a 502 HTML page, so there are no trustworthy item identifiers to process.

On another run the list succeeds. One order then returns a 504 while its details are being loaded. The remaining orders are still independent and can continue.

These failures share a provider, but not a recovery boundary. Calling both “the API is down” is convenient and useless: it either discards work which could continue or pretends the response established more than it did.

Silencing an Error Does Not Protect the Customer

Wed, 01 Jun 2016 00:00:00 +0000

One of our shipment-booking paths returned success after it had already failed. The remote booking existed and so did a local shipment, but the shipment had no label and no tracking number. From the customer’s point of view, the one thing we had promised to do was not done.

The failure happened while persisting the carrier response. A broad catch swallowed the exception, reported nothing, and let the request continue. This was meant to serve a reasonable operational goal: do not bother customers with errors if we can avoid it. Instead, we gave them a broken shipment and called it a success.

A Failure Timeline Can Eliminate the Best-Looking Suspect

Fri, 01 Apr 2016 00:00:00 +0000

A shipment appeared in the local database without the tracking number or label that made it useful. Payment was the first suspect because it was known to be brittle, and the visible result looked like an operation which had stopped halfway through.

But payment had completed. The remote booking had completed too. The failure happened while persisting the response, and a broad catch continued without reporting it.

Payment was guilty by reputation, not by the state in front of us. Ordering the effects made that mismatch visible before another log line was added.