Retries Can Multiply the Original Failure

We once had roughly 50,000 failed jobs sitting in queues and trying again every 60 seconds. Some represented a provider outage. Others contained malformed input which could wait for three months and still be malformed. The queue made no distinction and had no terminal condition.

During a long outage, workers spent their time reproducing failures we already understood. Redis filled with delayed work, fresh operations had to compete with old ones, and a dependency problem became sustained load on our own application. The retry loop was meant to make work reliable. What it actually made durable was failure.

A retry is another production request

Retry configuration looks harmless when written as a little sequence:

try
wait 60 seconds
try again

The waiting is cheap when the queue stores a delayed message. The next attempt is not. It reserves a worker, loads state, queries databases, deserializes the payload, acquires locks, writes logs, and calls the dependency again. A retry is a new production request with an old reason.

At the scale from the supplied incident, the first-order projection is uncomfortable:

50,000 jobs
× 1 attempt per minute
× 1,440 minutes per day
= 72,000,000 scheduled attempts per day

That is a theoretical upper bound, not a retained production measurement. The workers could not execute every scheduled attempt; execution time, congestion, and queue ordering reduced that number while making the backlog older. The DBG-08 fixture deliberately calls them 72 million scheduled opportunities rather than observed remote requests.

The exact executed count is not the important part. Every attempt which does run competes with new work and with retries which have a genuine chance of succeeding. Retry policy is load policy whether we acknowledge it or not.

Classify before scheduling another attempt

The old loop asked only one question:

Did this attempt fail?

A useful policy needs at least two more:

Can the same operation succeed later without changing its input?
Is repeating it safe?

Those answers separate common failure classes:

“Transient” is not a synonym for “exception.” A parse error does not become valid after a minute, and a missing field does not heal overnight. Expired credentials need configuration work, not a queue worker with admirable stamina.

Unknown outcomes need their own state. If a remote write may have succeeded before its response was lost, retrying can duplicate the effect. The previous article established that a timeout stops our wait, not the remote work. A retry policy has to consume that result instead of flattening it into failed.

Backoff changes pressure, not correctness

Increasing the delay gives a dependency time to recover and reduces how often we repeat the failure. It does not make bad input valid or a non-idempotent write safe. Backoff controls pressure; it does not supply judgment.

The DBG-08 fixture uses this present-day teaching schedule:

attempt 1: immediately
attempt 2: after       60 seconds
attempt 3: after      300 seconds
attempt 4: after    1,500 seconds
attempt 5: after    7,500 seconds
attempt 6: after   37,500 seconds
then: terminal failure

The delays grow by a factor of five. This schedule is executable and finite, but it is a teaching input rather than the historic repair schedule or a production recommendation.

Laravel supports a backoff array and successive values for later retries. It also supports maximum attempts or an expiry time . That makes a policy easy to express, but the framework cannot choose the failure classes, factor, total lifetime, or safe side effects for us.

Backoff usually needs jitter when many jobs fail together. Otherwise 50,000 jobs delayed for the same duration become 50,000 eligible jobs at the same moment. The fixture omits jitter to remain deterministic; a production policy should state the jitter rule and how far it may move the deadline.

A valid retry window from a rate-limit response is better evidence than our local curve and can override it. It still needs a local maximum. A provider does not get unlimited use of our queue merely because it returned a header.

Finite attempts create a decision point

“Keep trying until it works” avoids one uncomfortable decision by creating several worse ones. The job can outlive the customer need, source data, credentials, code version, and business deadline which justified it.

A finite policy forces an answer:

after the last attempt:
    mark the operation failed
    preserve the last useful error
    record the final attempt time
    notify the right owner when action is possible
    release queue capacity

Laravel passes the final exception to a job’s failed method and represents attempt exhaustion explicitly. That hook is useful only when terminal state survives outside the worker.

The old system’s queue messages were effectively fire-and-forget. We had no durable intent record from which work could be selected again later, so deleting a job made safe reconstruction difficult. Keeping it alive forever became the path of least resistance. That explains the design, but it does not make it good: queue retention was standing in for application state.

The queue is transport, not the record of intent

A durable operation record can be small:

operation identity
operation kind
input reference or immutable payload
status
attempt count
next eligible time
last failure class
last error summary
remote identity, when known
created, updated, and terminal times

The queue message then means:

attempt operation 42 now

It does not mean:

operation 42 exists only while this message survives

This separation lets a long outage end the current retry series without forgetting the business request. A scheduler can select eligible deferred work later, after a configuration repair or an operator decision. The next message carries the same operation identity and first checks whether the work is still needed. Forty-eight hours is one example, not a supplied schedule; the real date should follow the business deadline and dependency evidence.

Durable intent also makes backlog cleanup less frightening. In this incident, we eventually removed the accumulated jobs and replaced the loop with finite, increasing delays. With an application record, clearing transport backlog does not erase the operations which still need a decision.

I would not build a workflow engine for every email. Some work is cheap, reconstructible, and harmless to lose. A durable operation record earns its cost when the effect matters, outages can be long, input may change, or someone must choose between retry, repair, and abandonment.

Stop failures which input cannot repair

A malformed request should normally fail on its first classified attempt. Repeating it only hides the useful error behind queue noise.

The terminal record should retain why it cannot succeed:

class: invalid-input
field: destination postal code
decision: permanent failure
next action: correct the source data and submit a new operation

Silently mutating the payload until validation passes is not a retry. It asks for a different effect and makes the supposed repair impossible to audit.

There are legitimate exceptions. Validation can fail temporarily when it depends on reference data which is known to be incomplete and will be refreshed. The classification then belongs to that named dependency:

unsupported country in immutable product rules -> terminal
catalogue generation not active yet           -> deferred

The exception class rarely contains that whole distinction. Classification belongs at the boundary which understands both the failure and the operation.

Retry only the smallest safe boundary

If a job performs five steps and the last remote call fails, retrying the entire job may repeat the first four.

load source
create local row
reserve funds
submit remote write
persist response

A safe retry needs at least one of three things:

• earlier steps are pure or idempotent under the same operation identity;
• durable state lets the job resume from the first incomplete transition; or
• compensation returns the operation to a state where the whole sequence is safe to run again.

Without that proof, “the queue retries the job” is an implementation detail which quietly changes business behaviour.

This article is about one queued operation. A later article looks at an HTTP client, queue job, and workflow each applying their own attempt count, where three attempts can become nine or 27 remote calls. That is a different failure from the 50,000-job backlog here and does not need to be smuggled into the same story.

Enforcement can require a decision, not prove it

A retained codebase later added a static-analysis rule for queued work. A recognized job must declare:

• a positive execution timeout;
• a positive attempt limit or retry expiry; and
• positive backoff when the job contains recognized external I/O.

The rule prevents a new job from silently inheriting unbounded worker defaults. It cannot know whether the backoff is appropriate, the failure is transient, the intent is durable, or the effect is safe to repeat.

I use enforcement to make omissions visible and behavioural tests to prove the policy:

malformed input -> one attempt, terminal
transient outage -> increasing delay, then success
long outage -> finite attempts, terminal
unknown remote result -> reconciliation, no blind retry
terminal operation -> later queue delivery performs no effect

The fixture executes the first four shapes as deterministic state transitions. It does not run Laravel or a real queue, and it proves nothing about worker timing, Redis capacity, provider recovery, or production throughput.

Operate retries as accumulated risk

Queue depth cannot distinguish valuable new work from repeated failure. Useful retry observations include:

pending operations by failure class
attempts per completed effect
oldest next-eligible operation
operations at terminal failure
unknown remote outcomes awaiting reconciliation
attempt rate by dependency
backlog age by original creation time

The original creation time matters because every attempt gets a fresh execution timestamp. A job running today may represent a customer request which stopped being useful three days ago.

An emergency response also needs a way to pause one failure class, quarantine malformed work, or postpone one dependency without stopping unrelated queues. Deleting 50,000 jobs can be necessary containment. The next question is which business operations disappeared with those messages and which still exist in a durable record.

The incident did not show that retries are bad. It showed that an unclassified, unbounded retry is not recovery. It is a request generator whose stopping condition is somebody else’s return to health.

Retry only when unchanged input can plausibly succeed later, repetition is safe, and the next attempt still serves a live operation. Increase the delay, stop permanently, and keep the reason for the work somewhere more durable than the message carrying it today.