Index

Building a Rating Engine

Index Tariffs Without Losing Charges

Scanning ten thousand tariff rows for each of a hundred candidate services means a million candidate-row visits before doing much interesting pricing. Most visits may end with the same answer: this row belongs to another service. Putting that loop behind a cache doesn’t remove it.

I changed this part of my private logistics engine by preparing an in-memory index of rows by carrier and service match. This wasn’t a database index. The quote already had a restored runtime projection; the repeated work was inside that projection. The index moved some predicates out of the per-candidate loop, which made their preservation a correctness requirement rather than merely a performance detail.

The preparation chapter explains where this representation survived. Here the question is what its projector, selector, and consumer actually did. The PHP excerpts follow those implementations, with imports, tracing, and unrelated branches omitted where stated. Names and sample rows are anonymized. The opening multiplication is illustrative work, not a historical throughput result.

A service-specific map loses rows that still belong in the price

Take a candidate with carrier amber, service ID s17, service code parcel, and canonical service key parcel-domestic. These are distinct fields. The shared candidate matcher accepted any of the service’s usable identities against the row’s allowed service IDs or codes.

A tariff row could restrict neither dimension, only the carrier, only the service, or both. Alternatives within an allow-list were different from conjunction between the carrier and service restrictions. This row required carrier amber and any of the three service identities:

{
  "row_id": "joint-alias",
  "sequence": 6,
  "row_priority": 20,
  "service_match": {
    "carrier_ids": ["amber"],
    "service_ids": ["s17"],
    "service_codes": ["parcel", "parcel-domestic"]
  },
  "amount": "30.000000",
  "currency_code": "EUR"
}

This is a reduced runtime row, not a complete authoring request. A map containing only service_id → rows would miss general rows and carrier-wide rows. A carrier map would retain unrelated services. The projector instead created four bucket families:

wildcard
carrier[carrier]
service[service key]
carrier_service[carrier + service key]

The wildcard bucket meant unrestricted by these two dimensions. It didn’t mean unconditional. A seasonal or zone-specific charge with no service restriction still needed to reach the remaining matcher.

The implementation combined the row’s service-ID and service-code lists before classification. This is the bucket-building part of the projector; $rowId, $carrierIds, and $serviceKeys have already been normalized:

if ($carrierIds === [] && $serviceKeys === []) {
    $rowsByServiceMatch['wildcard'][] = $rowId;
    continue;
}

if ($carrierIds !== [] && $serviceKeys === []) {
    foreach ($carrierIds as $carrierId) {
        $rowsByServiceMatch['carrier'][$carrierId] ??= [];
        $rowsByServiceMatch['carrier'][$carrierId][] = $rowId;
    }
    continue;
}

if ($carrierIds === []) {
    foreach ($serviceKeys as $serviceKey) {
        $rowsByServiceMatch['service'][$serviceKey] ??= [];
        $rowsByServiceMatch['service'][$serviceKey][] = $rowId;
    }
    continue;
}

foreach ($carrierIds as $carrierId) {
    foreach ($serviceKeys as $serviceKey) {
        $compoundKey = $carrierId.'|'.$serviceKey;
        $rowsByServiceMatch['carrier_service'][$compoundKey] ??= [];
        $rowsByServiceMatch['carrier_service'][$compoundKey][] = $rowId;
    }
}

The combined case generated memberships for each allowed carrier/service pair: a row allowing a carriers and b service keys produced a × b memberships. That was an explicit space and preparation cost in exchange for less repeated filtering. It wasn’t enough to call the lookups constant-time and stop counting what the index contained. A wildcard row still had to be visited for every candidate. The opportunity came from selective buckets reused across candidates, not from making all row work disappear.

The projector removed the predicates it had indexed

The prepared row body lived once in rows_by_id; buckets contained its references. More importantly, the projected body no longer carried the carrier/service restrictions moved into those buckets. The projector retained only requested-option requirements in service_match:

$projectedRow = $row;
$residualServiceMatch = $this->compactProjectionArray([
    'option_codes' => $serviceMatch['option_codes'] ?? [],
]);

if ($residualServiceMatch === []) {
    unset($projectedRow['service_match']);
} else {
    $projectedRow['service_match'] = $residualServiceMatch;
}

$rowsById[$rowId] = $projectedRow;

For the earlier row, the prepared body could keep its amount, currency, sequence, priority, and other criteria while dropping service_match entirely. Its restrictions now lived here:

{
  "carrier_service": {
    "amber|s17": ["joint-alias"],
    "amber|parcel": ["joint-alias"],
    "amber|parcel-domestic": ["joint-alias"]
  }
}

This made the index part of the evaluator’s meaning. If it retrieved a different carrier’s row, the residual matcher couldn’t necessarily rescue the mistake: the restriction was gone from the prepared body. A demonstration that kept all original predicates and reapplied them afterward could make an overly broad index look correct.

Required options were deliberately different. A signature-only handling row still had to check that the request asked for the signature option. That condition wasn’t answered by identifying the candidate’s service. Geography, lane, season, zone, metric axes, offer requirements, and goods classification also retained their own matching stages.

The projector rejected a missing row ID or a non-array service-match payload. It didn’t create a usable prepared artifact by quietly inventing an identity for malformed source data. The prepared schema was a contract between the writer and reader, not just a convenient cache shape.

One candidate can retrieve the same row through several aliases

The collector gathered wildcard and carrier buckets, then service and combined buckets for service_id, service_code, and canonical_service_key. Its result contained eight bucket lists. The candidate in this example could retrieve joint-alias three times.

Those were three references to one charge. The candidate-row resolver handled reference hydration and identity deduplication separately. String references were resolved from rows_by_id; direct row arrays were also accepted by this helper. For string references, the hydration cache was local to the resolution call:

if (array_key_exists($rowReference, $hydratedRowsByReference)) {
    return $hydratedRowsByReference[$rowReference];
}

$hydratedRowsByReference[$rowReference] =
    $this->compactRuntimeRowHydrator->hydrateCompactRuntimeRowReference(
        is_array($rowsById[$rowReference] ?? null)
            ? $rowsById[$rowReference]
            : null,
        $rowReference,
        $compactRowEncoding,
        $compactRowFields,
        $omitEmptyArrays,
    );

return $hydratedRowsByReference[$rowReference];

The cache included failed hydration results too: array_key_exists could find a stored null. Repeated references didn’t have to repeat the same attempt. That was a per-candidate-resolution cache, not a promise that a row was expanded only once across every candidate or request.

After obtaining a row, the resolver chose its deduplication identity:

$rowId = $this->stringValue($row, 'row_id')
    ?? $this->stringValue($row, 'row_code')
    ?? '';

if ($rowId === '') {
    continue;
}

if (array_key_exists($rowId, $rows)) {
    continue;
}

$rows[$rowId] = $row;

The fallback to a row code mattered for accepted direct rows; it wasn’t a reason for the normal projector to stop requiring IDs. The database’s row identity and code constraints were also relevant. Deduplication can only preserve meaning if two distinct legitimate rows aren’t accidentally presented as the same identity.

Missing references were skipped. Malformed compact shapes could instead raise an exception, depending on their encoding and field descriptors. These weren’t all one “invalid cache” case. The selector itself required both rows_by_id and rows_by_service_match:

if ($rowsById === null || $indexedRows === null) {
    throw IndexedTariffGridRowsUnavailableException::create();
}

It didn’t silently switch to scanning an unindexed rows list. The published/runtime format had to provide the expected structures. A full scan was useful as a test reference, not an automatic production fallback in this selector.

Retrieval order cannot decide which tariff wins

The collector naturally encountered wildcard rows before some specific rows. Service ID, code, and canonical key also arrived in a fixed collection order. None of that meant a general rate outranked a negotiated or more specific one.

The resolver sorted the deduplicated rows by descending priority and then ascending sequence:

usort($rows, function (array $left, array $right): int {
    $priorityComparison =
        $this->intValue($right, 'row_priority')
        <=> $this->intValue($left, 'row_priority');

    if ($priorityComparison !== 0) {
        return $priorityComparison;
    }

    return $this->intValue($left, 'sequence')
        <=> $this->intValue($right, 'sequence');
});

The comparison didn’t add an ID tie-breaker. Adding one during a performance-only change would need justification from the existing ordering contract, not merely a preference for tidy deterministic code. The source’s numeric accessor treated nonnumeric priority or sequence as zero; a correct compiler and authoring boundary mattered there too.

Even a correctly ordered candidate list wasn’t the final component list. The tariff materializer passed rows through request and scope matching and component construction. A high-priority row for a northern zone could produce nothing for a southern shipment. Under the highest-priority strategy, the stopping condition came after component production:

$matchedComponents = $this->componentList($components['components'] ?? null);

foreach ($matchedComponents as $matchedComponent) {
    $priceComponents[] = $matchedComponent;
}

if ($matchedComponents !== []
    && $rowResolutionStrategy->isHighestPriorityMatch()) {
    break;
}

The surrounding loop first propagated materialization failures. It didn’t interpret an error as an empty match. Stopping when the first indexed row was retrieved, or when its service matched, would suppress a lower-priority row that could actually price the shipment.

That is why a set comparison was insufficient for the optimization. The same rows in a different order can select a different winner. The useful reference result was an ordered list after the remaining eligibility checks, with component and strategy tests covering the later stopping behavior.

A compound string key has an identifier assumption

The source’s combined bucket key was a delimiter join:

$compoundKey = $carrierId.'|'.$serviceKey;

That isn’t an unambiguous representation of arbitrary string pairs. These two pairs produce the same key:

$first = 'a|b'.'|'.'c';
$second = 'a'.'|'.'b|c';

var_dump($first, $second);
// string(5) "a|b|c"
// string(5) "a|b|c"

In a direct check of the actual projector, collector, selector, and candidate matcher, a row for carrier a|b and service c could therefore be retrieved for carrier a and service b|c. The full source predicate rejected that candidate; the projected body had lost the carrier/service restrictions and passed the remaining checks used in that case.

This is a source-helper counterexample over those input strings, not a claim that a complete admitted request produced an incorrect historical quote. It identifies the assumption an end-to-end identifier contract must establish. If those strings are forbidden, the refusal needs to be real and consistently enforced. If they are allowed, the representation needs an unambiguous pair encoding, escaping contract, or nested maps.

For a new implementation I’d avoid introducing that ambiguity. For a faithful language port, I’d also keep the correction separate from the claim of unchanged behavior. Changing the key scheme can be the right fix while still being a behavior change that deserves its own test.

This is one reason the public comparison uses array-encoded combined keys rather than pretending its simplified index is a literal copy of the engine. The model can explain the intended partitioning. The real implementation still has to meet it.

Compare the actual paths before timing them

For the ordinary example candidate, with no fragile option, southern zone, and four-kilogram physical weight, the ordered eligible IDs were:

joint-alias
global
carrier
service-id
service-code
canonical

Higher-priority north-only, fragile-option, and heavy rows didn’t survive their remaining conditions. The other-carrier row didn’t belong in the selected buckets. joint-alias appeared once despite three alias lookups. These facts establish different parts of the optimization’s contract; counting six results would establish much less.

I compared the real PHP projector and selector against the real shared candidate matcher, tariff criteria matcher, and metric-axis matcher over combinations of carrier, service identity form, options, zone, and weight boundaries. Ordered eligibility agreed for that bounded corpus. Removing wildcard retrieval deliberately produced a disagreement. So did the delimiter case outside the ordinary identifier examples.

These checks stop before the complete monetary calculation, offer/goods requirements, and every materialization failure. Those have their own contracts. The dependency-free public model provides the same kind of ordered comparison for synthetic data, but it isn’t a substitute for checking the actual producer and consumer.

The source’s timing boundaries needed inspection as well. The selector’s compact-hydration stage wrapped the resolver call, which also deduplicated and sorted rows. A separate final-ordering stage was recorded afterward, with its start obtained at the recording call. That named timer didn’t isolate the sort already performed inside the resolver. Reading a timing field’s label as an exact account of its work would misattribute the cost.

The historical performance work identified repeated candidate scans as work worth removing. This index made that removal concrete, but its benefit still depended on wildcard prevalence, selectivity, alias fan-out, preparation reuse, and how many rows had to be restored and matched. If nearly every row applied to every service, it could add structure while eliminating little filtering. For a small or unselective tariff I’d keep the direct scan until measurement justified the extra machinery.

The reference scan remained useful precisely because it was boring. It gave a separate account of which original rows should survive. When the indexed path disagreed, the first job was to find the lost condition, duplicated identity, or changed ordering—not to celebrate how quickly the shorter loop returned an incomplete answer.