Is Your Benchmark Doing New Work?

One of the rating engine’s early benchmark fixes made the results slower. That was the improvement. The runner had reused request identities across invocations, so later runs could return stored results instead of evaluating new quotes. The endpoint was doing its job. The benchmark wasn’t measuring the job I thought it was.

This is an awkward mistake because the output can look entirely healthy: successful HTTP responses, small latencies, no obvious application errors. It is also worth dealing with before celebrating a faster index or cache read . A replay can skip the calculation whose improvement you’re trying to measure.

The harness eventually grew beyond generating requests and timing replies. It had to say which work it intended to create, which work the API said it created, and which completed records actually existed. Here’s how those checks fitted together, and what they still couldn’t prove.

Warm configuration, not the identities being measured

The quote API had two relevant identifiers: a request reference and an idempotency key. Within the tenant and operation scope, the existing-record lookup could find either. The request-reference match took precedence; a matching key could also find a stored outcome when the request reference was different. The request hash then distinguished the same business request from conflicting use of an identity.

That is normal protection against repeated work. It is terrible protection against a benchmark accidentally measuring repeated work, because there is no application error to report when a legitimate replay succeeds.

Consider these illustrative inputs, with the business shipment held constant:

The measured request has a new request reference, but not a new operation key. It can retrieve the first outcome. Changing a tracing identifier alone doesn’t solve that. Keep the key but change the business payload and the result can be a conflict instead: another quick response that isn’t a new successful calculation.

The historical correction added a unique run token. In the later runner, identity construction was explicit about three dimensions:

def build_identity(run_id, phase, ordinal):
    return {
        "request_id":
            f"rating-request-{run_id}-{phase}-{ordinal}",
        "idempotency_key":
            f"rating-idempotency-{run_id}-{phase}-{ordinal}",
    }

This is the actual naming scheme, with type annotations and the length guard omitted for readability. The run token came from twelve random bytes encoded as hexadecimal. phase separated warmup from measurement; ordinal separated attempts in each phase. Another invocation got another token, so stored results from yesterday didn’t quietly become today’s workload.

The runner replaced the transport metadata, not the shipment:

cloned = payload.copy()
cloned["meta"] = build_identity(run_id, phase, ordinal)
body = json.dumps(cloned, separators=(",", ":")).encode("utf-8")

That excerpt is the canonical-body branch without optional tracing. The full runner also normalized an older body shape. It could cycle through a fixed set of representative shipments while giving each attempt a fresh operation identity. Warmup still exercised the same pricing configuration and cache paths; it didn’t pre-complete the measured requests.

There is nothing wrong with a replay benchmark. Replays matter to clients recovering from uncertain responses. I’d give that workload its own expected outcome instead of mixing it into new-evaluation throughput. Disabling idempotency would be a different experiment again: useful for an isolated calculator, but no longer the complete endpoint.

The response had to say more than 200

Unique generated keys express the client’s intention. They don’t prove the server followed it. The runner recorded each submitted identity and the corresponding response separately, retaining run, phase, and ordinal so that concurrent completion order didn’t become request identity.

The dataset’s expected response included:

{
  "status_code": 200,
  "jsonapi_type": "rating-evaluation",
  "idempotency_status": "created",
  "replayed": false
}

The Python inspector parsed the body, extracted data.type, data.id, and the idempotency metadata, and kept transport failures distinct from valid HTTP responses. It then compared those observations with the expectation. The significant checks included:

if response["status_code"] != expected["status_code"]:
    add_failure(summary, "Unexpected status")

if response["idempotency_status"] != expected["idempotency_status"]:
    add_failure(summary, "Unexpected idempotency outcome")

if response["replayed"] is not expected["replayed"]:
    add_failure(summary, "Unexpected replay state")

Failure messages are shortened here; the source includes phase and ordinal. The last comparison is intentionally stricter than a truthiness check. false is an observation. A missing field isn’t evidence that replay didn’t happen. The full checks also rejected transport errors, malformed JSON, an unexpected resource type, and a missing decision ID.

This didn’t establish that the returned price was correct. The inspector recorded the number of shipping options, but these response expectations weren’t an independent price oracle. They answered whether a reply had the shape and classification of the intended operation. Tests of computation and the later PHP-to-Go comparisons had a different job. A benchmark which accepts newly created but wrongly priced decisions is still wrong; checking identity doesn’t excuse skipping the pricing tests.

Two small validation runs show why the checks arrived in stages. The first established unique input identities and completed records, but its response data didn’t yet record new-versus-replayed classification. The second added that observation and required agreement with stored state. The first was useful progress, not proof of a fact it had never collected.

Join each reply to its completed records

Five successful replies and five extra database rows aren’t enough. Another process could create those rows while one measured request failed to persist. The comparison needs the identities of this run, not a reassuring change in a global count.

The PHP command ran the Python executor, then passed its result and request records to a certifier. The ordinary count-run path checked the expected warmup and measured counts, validated both per-request files, and matched responses to submitted phase/ordinal positions. Missing responses, duplicate positions, and unexpected positions failed the run. The recorded result also bound its inputs through manifest/provenance and checksum checks; a result file wasn’t free to borrow a different run’s request list.

For each response, the persistence verifier received the expected request ID, key, and returned decision ID. It used two narrowly selected queries. These are the actual Eloquent query shapes, with variable names shortened:

$decisions = Decision::query()
    ->select(['request_id', 'decision_id', 'status'])
    ->where('division_id', $tenantId)
    ->where('operation_code', $operationCode)
    ->whereIn('request_id', $requestIds)
    ->get();

$records = IdempotencyRecord::query()
    ->select([
        'request_id', 'idempotency_key', 'decision_id', 'status',
    ])
    ->where('division_id', $tenantId)
    ->where('operation_code', $operationCode)
    ->whereIn('request_id', $requestIds)
    ->get();

The verifier processed request IDs in batches of at most 500. Each decision had to exist, have completed status, and contain the decision ID returned by the API. Each idempotency record had to exist, have the expected key and completed status, and point at that same decision. It rejected duplicate expected request IDs, keys, and returned decision IDs within its input batch. The selected counts also had to match the expected count.

The relationship being checked was:

submitted request + key
    -> response decision ID
    -> completed decision for that request
    -> completed key record linked to that decision

Notice what isn’t in that chain: the engine’s entire heavy snapshot graph. Earlier development had moved heavier evidence persistence off the critical path while retaining a minimal completed decision and replayable outcome. The verifier checked that minimal pair. It didn’t prove that every deferred snapshot, diagnostic child, or background operation had finished. Requiring all of those would change the operation’s completion boundary rather than make this particular benchmark more accurate.

The queries ran after the timed HTTP phase. Putting them inside every client request would mix reconciliation overhead into endpoint latency. They’re still real work for the harness, and the request/response recording still costs time in the driver. Neither should disappear from the explanation of the measurement.

The small public model demonstrates why the join adds something: it removes a simulated stored result after successful new-result responses and rejects the run. That’s a deliberately broken example, not an incident in the engine. The actual verifier uses database records and enum casts, not the model’s in-memory maps.

The driver can hide load from you too

The ordinary count runner used a bounded thread pool. It submitted work up to the concurrency limit, waited for a completed future when full, recorded that response, and then submitted the next attempt. It drained the remaining futures before stopping the phase timer. This avoided queuing every generated body in advance, but it also defined the workload.

With this arrangement, a slowing server makes the client submit fewer new requests per second: occupied slots have to become free first. A count run at concurrency ten does not prove behavior under an independent, fixed arrival rate. It is a useful workload; it’s just not every workload. The later harness had separate paced and saturation/recovery paths for different questions. Their existence doesn’t turn an earlier count run into a saturation test.

There was a simpler arithmetic error as well. The older result divided request count by the sum of individual latencies. Two overlapping requests can each take 50 milliseconds while both finish within the same 50 milliseconds of wall time:

request A: 0 ms ----------------> 50 ms
request B: 0 ms ----------------> 50 ms

elapsed wall time:  50 ms
sum of latencies: 100 ms

The batch completed two requests in 0.05 seconds: 40 requests per second. Dividing by the latency sum gives 20. Those are two descriptions of the same supplied intervals, not two different server performances.

The corrected builder retained individual latency statistics but gave the throughput denominator its own field:

{
    "requestCount": request_count,
    "wallTimeSeconds": wall_time_seconds,
    "throughputRequestsPerSecond": (
        request_count / wall_time_seconds
        if wall_time_seconds > 0 else 0.0
    ),
    "latencySeconds": {
        "sum": sum(latencies),
        # Percentiles, average, fastest and slowest also live here.
    },
}

This is a shortened version of the actual result structure. The count is HTTP attempts; the field isn’t magically useful-quote throughput when some attempts are conflicts or replays. The new-work correctness checks must pass before those attempts can stand for newly completed evaluations. Failure and status distributions remain part of the result rather than being thrown away while printing a nice rate.

What the reported numbers actually describe

The performance notes contain a larger historical direct-application baseline: 500 services and 15,000 tariff rows, a mixed profile, a dense scenario, fifty measured requests, concurrency one, and ten warmup requests. That workload was repeated ten times. Its reported spread was as follows, with every value in milliseconds:

These are the report’s historical measurements, not a fresh rerun. In particular, the average of ten per-run p99 values is not a p99 calculated from all five hundred observations pooled together. Concurrency one describes the request path without substantial overlapping client work; it doesn’t tell us how much traffic the deployment can sustain.

The early identity correction preceded that baseline. Later request/response reconciliation and wall-time changes added stronger checks still. I wouldn’t retroactively label every older run certified under those newer checks, nor discard all earlier timings simply because the harness subsequently improved. The honest distinction is what the original record retained and which conclusions that record supports.

The later correctness run used a much smaller dataset: two services, four tariff rows, two warmup requests, five measured requests, and concurrency two. All seven responses had unique decision IDs, successful new-result classification, and matching completed decision/idempotency pairs. Octane was disabled in that local runtime.

Its recorded measured p50 was about 217 milliseconds. That isn’t a regression from the table above: the runtime and workload weren’t a matched comparison. It was a test of the runner and its reconciliation. Using that number to claim persistent-worker capacity would be just as misleading as using a replay-heavy run to claim fast calculation.

The older result format expressed its percentiles in seconds. A recorded 0.217 means about 217 milliseconds, not 0.217 milliseconds. The newer format put the unit into latencySeconds. Small sample sizes also deserve some restraint: the count runner’s percentile function linearly interpolated between sorted observations. With five samples, a p99 is near the largest sample, not a measured one-in-a-hundred tail under sustained traffic.

Transport was another controlled distinction. Direct app HTTP removed local TLS/ingress behavior from code-level investigations. Ingress-path tests were still needed to understand that path, and remote edge failures or unsupported request contracts weren’t application-performance samples. A fast rejection at the edge doesn’t tell me how quickly the engine can price a shipment.

Slower, but finally measuring the calculation

The first identity fix was valuable precisely because it spoiled a flattering result. The later checks made more failures visible: a missing observation, a duplicate returned ID, an incomplete stored pair, a misleading throughput denominator. None improved the pricing algorithm. They made it harder to attribute an irrelevant number to it.

I want both kinds of confidence before treating a rewrite as faster: tests that the computation is right, and a workload which actually reaches that computation under the stated conditions. A successful replay can be excellent API behavior. It just doesn’t get to win a new-calculation benchmark.