A Rate Limit Belongs to All the Workers Sharing It

One polling job receives a 429 response saying to try again in fifteen seconds, so it releases itself for fifteen seconds. During that pause, twenty other workers poll the same provider. They have not seen the response, and each sends another request into the closed window.

The first job obeyed the limit. The application did not.

Find the scope before choosing the delay

A rate limit applies to some shared identity: an API credential, account, tenant, endpoint family, source address, or a combination of them. Every worker using that identity spends the same remote capacity.

The polling boundary had many independent jobs for one provider. A job was unique per tracking number, which prevented two workers from polling the same number at once. That was useful and still irrelevant to different numbers sharing the same provider quota.

Those are different scopes:

job uniqueness:
    one worker per tracking number

remote quota:
    all tracking numbers under one provider identity

Putting the pause on the job answers only when that job may try again. The provider’s refusal answers when any caller in the quota scope may try again.

Turn the refusal into admission state

The correction introduced one shared pause deadline for that provider.

worker A sends request
    -> provider returns 429 with 15-second hint
    -> store provider paused-until deadline
    -> release worker A

worker B starts 1 second later
    -> read provider paused-until deadline
    -> release for remaining 14 seconds
    -> do not resolve a client or send a request

The preflight check matters. Deferring after client construction or request signing may still consume local work; deferring after the remote call is just recording that the gate failed.

The test sets nine seconds of shared pause state, runs another job, and proves the provider resolver is never called. That is stronger evidence than checking only that the job was released.

Store a deadline, not a copied delay

If the first worker stores 15, a second worker reading it three seconds later cannot tell whether twelve or fifteen seconds remain.

Store an absolute deadline:

paused_until = observed_at + retry_delay
remaining = max(0, paused_until - now)

The store writes a Unix timestamp with cache expiry at the same instant. Readers accept an integer timestamp, calculate the remaining seconds, and never return a negative delay.

An expired or malformed cache value means no active pause. That keeps a broken coordination record from blocking polling forever, but it deliberately favors availability over strict quota protection. There is no neutral fallback here.

Prefer the remote window, then fall back deliberately

The observed provider placed its retry hint inside a text field in a JSON response. The job extracts a positive number of seconds from that field and uses it for both the shared deadline and its own release.

When parsing fails, the policy falls back to exponential delay:

attempt 1   15 minutes
attempt 2   30 minutes
attempt 3   60 minutes
attempt 4    2 hours
...
cap         12 hours

That fallback is intentionally conservative. A malformed hint is not permission to retry immediately.

A standard Retry-After header would be preferable when the provider supplies one. A documented machine-readable field is better than prose. Text parsing is sometimes the only available contract, but then the regular expression has become part of the integration protocol whether anybody likes it or not. Keep it isolated, test the exact accepted shapes, and let failure enter a safe fallback.

Clamping the parsed value to at least one second prevents a zero-second busy loop. A maximum may also be necessary if an untrusted or broken response can request an absurd delay.

Do not let one provider freeze every integration

The pause key in the boundary is provider-specific. Jobs for other providers continue normally.

That isolation is part of the capacity model:

pause key = provider identity + credential scope

A single global rate-limited-until key would be easy to implement and costly to operate: one constrained integration could stall unrelated work. A key which is too narrow has the opposite failure, with several credentials or workers continuing to spend the same quota independently.

The correct key follows the remote system’s accounting boundary. If each credential has its own quota, include credential identity without storing the secret. If the provider limits an entire account across credentials, key by that account instead.

Concurrent refusals need a monotonic deadline

Shared cache alone does not complete the algorithm.

Imagine two workers receive refusals close together:

worker A observes: retry in 30 seconds
worker B observes: retry in 5 seconds

If B writes last, a simple cache assignment shortens the pause to five seconds. The safe update is:

paused_until = max(existing_paused_until, proposed_paused_until)

That comparison must be atomic across workers. A lock, compare-and-set operation, or small cache-side script can own it. The implementation uses a normal cache write, so it proves shared deferral in the tested sequential cases but not monotonic behavior under concurrent 429 responses. Last write wins is convenient cache behavior and the wrong deadline policy.

A gate and a throughput limiter solve different problems

The shared deadline reacts to a refusal and prevents peers from continuing during a known closed window. It does not pace requests before the first 429. If the provider grants one hundred requests per minute, a proactive limiter can allocate that budget across workers and avoid the refusal in the first place.

Both mechanisms may be useful:

throughput limiter:
    may this request spend capacity now?

shared refusal gate:
    has the provider told this scope to stop until a deadline?

The first models a known budget; the second carries fresh remote feedback. Calling both “rate limiting” does not make them interchangeable. Neither replaces bounded job attempts or an explicit retry owner; INT-08 takes that problem separately.

Verify that peers stay quiet

The useful tests exercise a small timeline:

1. A provider-specific 429 with a fifteen-second hint records a pause and releases the refused job for fifteen seconds.
2. A peer arriving during a nine-second pause is released without resolving the provider client.
3. An unrelated provider ignores that pause.
4. A malformed hint uses the bounded fallback rather than retrying immediately.
5. Concurrent longer and shorter hints retain the later deadline once atomic maximum updates are implemented.

The public fixture executes the first four cases and shows the extra calls made when workers keep only local delay state. It does not claim concurrent cache correctness. The practical decision is simple: when a remote refusal applies to a shared quota, store it at that scope before another worker spends capacity. One polite job is not enough when the rest of the application keeps hammering.