Retry Budgets Multiply Across Layers

Wed, 01 Aug 2018 00:00:00 +0000

An HTTP connector allows three attempts. The queue job calling it also allows three attempts.

When the remote service keeps failing, the job does not make three calls. It makes nine:

job attempt 1 -> connector calls 1, 2, 3
job attempt 2 -> connector calls 4, 5, 6
job attempt 3 -> connector calls 7, 8, 9

Both settings look modest in isolation. The remote system still receives their product. Three attempts multiplied by three attempts is nine calls, regardless of how harmless each configuration file looks.

Retries Can Multiply the Original Failure

Mon, 01 Aug 2016 00:00:00 +0000

We once had roughly 50,000 failed jobs sitting in queues and trying again every 60 seconds. Some represented a provider outage. Others contained malformed input which could wait for three months and still be malformed. The queue made no distinction and had no terminal condition.

During a long outage, workers spent their time reproducing failures we already understood. Redis filled with delayed work, fresh operations had to compete with old ones, and a dependency problem became sustained load on our own application. The retry loop was meant to make work reliable. What it actually made durable was failure.

Retries on Brian Faust

Retry Budgets Multiply Across Layers

Retries Can Multiply the Original Failure