Queues on Brian Faust

A Worker Count Is Not a Throughput Plan

Wed, 01 Jan 2025 00:00:00 +0000

Four Laravel queue workers complete 114 waiting jobs per second where one worker completes 37. Keep the worker count at four, make every job update the same database row, and the rate falls to 32. Make each job fail twice before succeeding, and those four workers perform 540 attempts to produce 180 effects.

Nothing is wrong with the worker setting. It is answering three different workloads.

A worker count describes available executors. It says nothing about whether a job can run in parallel, which dependency must absorb the concurrency, or how many attempts are needed for one business effect. A throughput plan needs those parts too.

A Queued Command Changes the Contract

Sat, 01 Apr 2023 00:00:00 +0000

Send the same payment-capture request through two Laravel routes. The synchronous route returns 204 after the provider has created charge ch_001 and the order records that reference. When the provider returns 503, the HTTP request returns 500 and the order remains unpaid.

The queued route returns 202 while the order is still unpaid and the provider ledger is empty. Its response contains an operation URL. Cancel the order before a worker starts and that operation later becomes refused; the provider is never called.

Framework Convenience Has a Boundary

Wed, 01 Jul 2020 00:00:00 +0000

A model was saved inside a database transaction. Its observer dispatched a queue job. The worker started quickly, looked for the record which described its work, and found nothing.

The record existed in the transaction which dispatched the job, but it was not yet visible to the worker’s database connection. Retained history connects that race to successful wallet payments without a corresponding wallet transaction. The job had been published; the state it needed had not. The framework did exactly what the code asked. The shortcut failed because its lifecycle was smaller than the business operation.

Deprecation Is a Migration, Not a Warning

Tue, 01 Oct 2019 00:00:00 +0000

A billing rewrite introduced durable export records before removing several legacy queue jobs. Each new worker carried a record identifier. The export type, payload snapshot, attempts, failure details, and retry state lived in the database. A registry selected the current handler.

That order matters. A queued object is not merely code waiting to run. Its serialized class name and properties are data, even if nobody thinks of them as a schema. Delete the class while an old payload remains, and PHP can deserialize it as an incomplete object which no longer has the method the worker expects.

Partial Work Needs a Child Ledger

Sat, 01 Sep 2018 00:00:00 +0000

A queue job receives 1,000 events in one remote response. It dispatches 600 child jobs, then times out.

The parent row says processing failed. Some children have already finished. Others are waiting in the queue. The remaining 400 were never dispatched. A counter can report 573 completed events, but it cannot say which 573 they were.

Running the parent again may duplicate the first 600. Leaving it alone loses the last 400. Neither choice is defensible because the system retained a total and discarded the membership behind it.

Retry Budgets Multiply Across Layers

Wed, 01 Aug 2018 00:00:00 +0000

An HTTP connector allows three attempts. The queue job calling it also allows three attempts.

When the remote service keeps failing, the job does not make three calls. It makes nine:

job attempt 1 -> connector calls 1, 2, 3
job attempt 2 -> connector calls 4, 5, 6
job attempt 3 -> connector calls 7, 8, 9

Both settings look modest in isolation. The remote system still receives their product. Three attempts multiplied by three attempts is nine calls, regardless of how harmless each configuration file looks.

A Rate Limit Belongs to All the Workers Sharing It

Sun, 01 Jul 2018 00:00:00 +0000

One polling job receives a 429 response saying to try again in fifteen seconds, so it releases itself for fifteen seconds. During that pause, twenty other workers poll the same provider. They have not seen the response, and each sends another request into the closed window.

The first job obeyed the limit. The application did not.

Find the scope before choosing the delay

A rate limit applies to some shared identity: an API credential, account, tenant, endpoint family, source address, or a combination of them. Every worker using that identity spends the same remote capacity.

Data Outlives the Code That Created It

Fri, 01 Dec 2017 00:00:00 +0000

A failed queue record kept returning. Its payload named a mail class the running application could not resolve correctly, so retrying it could only repeat the same failure.

The workaround was blunt: once a minute, delete failed records whose payload matched that job. It stopped an unusable delivery attempt from disturbing the queue, but it obviously did not make the payload readable.

That small incident exposes an awkward storage contract. Serializing an executable object does not merely store its arguments. It preserves facts about the code which created it: class name, namespace, property layout, and sometimes framework behavior. The payload can live much longer than those facts.

Retries Can Multiply the Original Failure

Mon, 01 Aug 2016 00:00:00 +0000

We once had roughly 50,000 failed jobs sitting in queues and trying again every 60 seconds. Some represented a provider outage. Others contained malformed input which could wait for three months and still be malformed. The queue made no distinction and had no terminal condition.

During a long outage, workers spent their time reproducing failures we already understood. Redis filled with delayed work, fresh operations had to compete with old ones, and a dependency problem became sustained load on our own application. The retry loop was meant to make work reliable. What it actually made durable was failure.

Runtime Configuration Is Process State

Sun, 01 May 2016 00:00:00 +0000

A queued job selected the settings for one business context by overwriting Laravel’s global configuration. The values were correct for that job. The problem was what happened afterward.

The worker stayed alive.

Only a small fraction of the queued job classes selected their context this way. A later job could read a key without first replacing it and inherit whatever the previous job had left behind. Nothing was wrong with the configuration file, environment variables, or deployment. The process contained valid values for the wrong work.