One of our shipment-booking paths returned success after it had already failed. The remote booking existed and so did a local shipment, but the shipment had no label and no tracking number. From the customer’s point of view, the one thing we had promised to do was not done.
The failure happened while persisting the carrier response. A broad catch swallowed the exception, reported nothing, and let the request continue. This was meant to serve a reasonable operational goal: do not bother customers with errors if we can avoid it. Instead, we gave them a broken shipment and called it a success.
That is worse than returning an error. At least an error tells the customer to stop, retry, or contact support. A half-finished shipment looks usable until someone tries to print a label or waits for a tracking number which will never arrive.
Success must name its required effects
“The request did not throw” tells us something about control flow. It says nothing about whether the customer got what they asked for. In this case, the useful definition of success was quite small:
remote booking accepted
local shipment complete
tracking number available
label available
result returned to the customerA remote reference on its own was not enough, and neither was a row in our database. The customer needed the label and tracking number to continue. Once we write the contract that way, the old behaviour becomes hard to defend:
usable success =
remote booking
AND complete local state
AND tracking number
AND labelThe old path returned success when only the first line was true. The response and the stored state disagreed about what had happened, while the customer was left to discover the disagreement.
The preceding article used completed and missing effects to locate a failure. Here we already know where the failure happened. The question is what the system may truthfully leave behind after it happens.
An error can protect the next decision
A clear error is not a failure of customer service. It is useful information:
booking could not be completed
no usable local shipment was created
retry or contact support with this referenceThat message prevents the customer from continuing with a shipment which does not exist in a usable form. It also gives support a declared outcome to investigate instead of an object which looks valid until someone inspects its missing fields.
We changed the path to roll back the failed database work, remove the incomplete local shipment, report the exception, and return a clear failure with a support path. Those are four parts of one outcome, not four independent improvements. The customer sees a failure, the local catalogue contains no pretend shipment, and the error reaches an operational surface.
The DBG-06 fixture makes that comparison explicit. Its silent path retains an incomplete row and reports success. Its corrected path removes the row, records the failure, and returns an error. The fixture is a synthetic state machine rather than a reproduction of the private application; it checks that state, response, and evidence agree.
Catch only what you can finish handling
Broad catches are not inherently wrong. An application boundary often needs to turn many internal failures into one stable API response, and cleanup may need to run for database exceptions, client failures, type errors, and other throwables. The problem starts when intercepting an exception is mistaken for handling it.
A catch has handled an error only when it has decided:
- which local writes remain;
- which transaction commits or rolls back;
- what remote effects already exist;
- whether cleanup succeeded;
- what the caller is told;
- what evidence operators receive; and
- whether retry is safe.
If the catch cannot answer those questions, continuing is usually the least honest branch. The original code did not merely simplify an error message. It changed failed persistence into apparent success without repairing the missing state.
“Never show customers an error” is an understandable request and an awful software contract. The useful version is: avoid failures where we can, and make unavoidable failures clear enough that the customer knows what to do next. The technical design owns that translation. Nobody asked us to return a stack trace, but that did not oblige us to invent success either.
A database rollback stops at the database
There was still an awkward part. The remote booking had already succeeded, and rolling back our database transaction could not undo it. After the correction, the outcome was:
customer response: failure
local shipment: absent
reported error: present
remote booking: presentThat last line is easy to gloss over. “Rolled back” sounds reassuringly atomic, but a database controls only its own writes. The remote booking remains an orphan unless another operation cancels, imports, or reconciles it.
In this case, the relevant carriers charged only when a label was scanned. The orphan had no scanned label, so removing the incomplete local record did not create a booking charge. That commercial rule is what made deletion and a clean retry reasonable:
remote booking exists
label was not delivered or scanned
charge is not triggered
customer can retryI would not copy that policy into every integration. A remote reservation may consume scarce capacity, a payment may already be captured, or a booking may become billable when it is created. Even a harmless-looking identifier may need to remain available for audit. Cleanup has to follow the real remote consequence, not the comforting fact that our own transaction rolled back.
Delete, preserve, or reconcile deliberately
Deleting partial local state is one option, not a universal rule. I use the remote consequence to decide what the local system must preserve:
| Remote consequence | Credible local response |
|---|---|
| no charge or scarce resource until later use | remove the incomplete local record and allow a clean retry |
| remote operation can be cancelled reliably | cancel it, record the outcome, then remove or close the local attempt |
| remote operation is billable or irreversible | preserve an explicit failed or reconciling attempt with the remote identity |
| remote outcome is unknown | retain enough evidence to query or reconcile before retrying |
A failed-attempt record is often better than a half-valid business object. It can retain the request identity, remote reference, failure stage, cleanup attempts, and final decision without pretending that the shipment is usable. That keeps us away from both bad extremes: deleting the only path to a billable remote effect, or putting an incomplete shipment into the normal catalogue and making every reader understand its accidental shape.
The correct state model might be:
booking attempt: failed, remote reference known, reconciliation pending
shipment: not createdOr it might be:
booking attempt: failed, remote effect non-billable, cleanup complete
shipment: not createdThe business rules choose between those states. A silent catch chooses neither; it simply leaves the decision for whoever finds the mess later.
Cleanup can fail too
The retained code contains a later example of the same boundary. A local shipment row could survive when an unexpected throwable occurred after initial creation but before tracking and metadata persistence, so the cleanup path was widened to cover that failure class. The change also stopped pretending that deletion itself could not fail. A failed delete emitted a critical report with enough identity to find the partial row.
That second failure deserves its own outcome:
original operation: failed
cleanup: failed
local state: potentially partial
operator signal: required
customer response: still failureCleanup failure must not overwrite the original error or somehow turn the request into success. It makes the situation more urgent because the response and local state may now contradict each other. The report should preserve both failures and the attempt identity.
finally is useful for releasing resources; it does not make a compensating
business action reliable. Deletes, cancellations, refunds, and rollbacks have
their own failure modes, timeouts, and retry rules. Treating cleanup as magic is
just another way to hide the hard part.
Report the boundary, not only the exception
An exception class and stack trace help developers. They do not tell an operator which business transition failed or what remains true. For this workflow, a useful report includes:
attempt identifier
failure stage: response persistence
remote booking reference received: yes
local transaction outcome: rolled back
local shipment cleanup: succeeded or failed
label available: no
tracking number available: no
customer response: failureThe exact carrier payload and database exception from the historic case are no longer retained, so I am not going to reconstruct them. The fields above are the boundary a current implementation can record.
This is also why the support reference matters. “Something went wrong” is honest, but expensive, when neither the customer nor support can connect the response to the reported failure. The public reference does not need to expose an internal identifier. It only needs to resolve to the same attempt in the operational record.
Test the contradiction you are removing
An error-path test needs to inspect more than the response code. Inject the failure after the remote effect and before required local persistence, then assert the whole outcome:
response says failure
no usable local shipment exists
exception is reported
cleanup outcome is recorded
remote reference is retained when reconciliation requires it
retry cannot duplicate a chargeable effectThe final two assertions depend on the remote contract. The fixture uses the supplied scan-triggered charging rule and verifies that the orphan is not charged. It also runs a bill-on-booking alternative to show why deleting the local row would be insufficient there. This cannot prove any provider’s commercial terms; it proves only that changing the charge trigger changes the safe cleanup decision.
The lesson is not that customers should see more errors. It is that every response should tell the truth about what the customer can do next. If required output is missing, return a failure they can act on. If a remote effect already happened, record or compensate it according to its real cost. Silence will not reconcile two systems which already disagree.