Data That Must Survive on Brian Faust

Data Outlives the Code That Created It

Fri, 01 Dec 2017 00:00:00 +0000

A failed queue record kept returning. Its payload named a mail class the running application could not resolve correctly, so retrying it could only repeat the same failure.

The workaround was blunt: once a minute, delete failed records whose payload matched that job. It stopped an unusable delivery attempt from disturbing the queue, but it obviously did not make the payload readable.

That small incident exposes an awkward storage contract. Serializing an executable object does not merely store its arguments. It preserves facts about the code which created it: class name, namespace, property layout, and sometimes framework behavior. The payload can live much longer than those facts.

Audit History Is Different from Application Logging

Wed, 01 Nov 2017 00:00:00 +0000

When a cart appears to expire, support may ask:

Which cart changed?
What was its prior state?
What state followed?
Who or what caused the transition?
Which policy allowed it?

An engineer investigating the same report may ask:

Which request handled it?
Which worker retried?
How long did the database call take?
Which exception and stack occurred?
What else happened on that host?

Both groups need evidence, but making one record answer both sets of questions usually gives you a bloated audit trail and strangely incomplete logs. Audit history explains durable business change. Application logs explain software execution.

An Index Must Serve a Real Query

Sun, 01 Oct 2017 00:00:00 +0000

A production query needed a small page of recent records requiring operational attention. The table already had an index which looked relevant. The measured scan still discarded 84,003 historical index entries and recorded 122,936 shared-buffer reads before finding 83 eligible rows.

The table did not need another vaguely relevant index. It needed an index for this query:

one tenant
active records with zero price
created during the last three months
newest creation time first
stable identifier as the tie-breaker
first page only

That is the actual starting point: the access path, not a list of columns which sound important in isolation.

Pagination Needs a Stable Order

Fri, 01 Sep 2017 00:00:00 +0000

A financial report encoded two sort rules as one string:

lowercased customer name + "-" + customer identifier

Sharded tests with concurrently created identifiers exposed unreliable output. The correction stopped asking string concatenation to double as a sorting contract. It compared names directly and consulted the stable customer identifier only when the names matched.

The diff does not prove that equal names alone caused the failing order. It does expose the rule every paginator depends on: the primary comparison, tie-breaker, and comparison semantics must be explicit, and each row needs exactly one reproducible position.

Soft Deletes Preserve Ambiguity Too

Tue, 01 Aug 2017 00:00:00 +0000

Adding deleted_at to a table answers one question: when was this row marked as deleted? It does not answer whether the row was safe to mark, who should still see it, whether its credentials remain valid, whether it still occupies a unique business slot, how it can be restored, or when its data should be erased.

A cart cleanup could not safely reduce those questions to one nullable timestamp. Old carts might still have a shipment, a payment session, a booking attempt, or item work which had crossed a meaningful boundary. The eventual policy archives a deliberately narrow set and preserves the aggregate as evidence. Soft deletion preserves rows, but it also preserves every ambiguity which the lifecycle never defined.

An Identifier Should Outlive Its Storage Engine

Sat, 01 Jul 2017 00:00:00 +0000

A billing export once had enough information to reconstruct a missing account, but not enough to name it. The account still existed in an older system and its stable identifier reached the export action. What did not exist was a matching row in the newer application’s user table.

The action worked around that absence by building a temporary object with safe fallback details. It even gave the object a synthetic local identifier. The field used as the export identity remained null, though, and a queued data object later rejected the payload because that field had to be a string. The fallback had solved the missing attributes while losing the one fact the export actually needed: which account it was about.

Time Zones Are Part of the Data Model

Thu, 01 Jun 2017 00:00:00 +0000

One metering path assigns an event to a billing month after converting its occurrence time into the account’s timezone. The same path derives its daily usage key directly from the occurrence timestamp without that conversion.

Near midnight, those two fields can disagree:

event instant: May 31, 21:30 UTC
local business time: June 1, 00:30
billing period: June
stored usage date: May 31

Each expression looks like an ordinary date calculation. They model different things, and the defect stays invisible until the difference crosses a day or month boundary.

Money Is Not a Floating-Point Number

Mon, 01 May 2017 00:00:00 +0000

A booking-time price arrived as four values:

net amount: 407
tax rate: 25.5%
tax amount: 104
gross amount: 511

The amounts are integers in the currency’s minor unit. The rate is a decimal ratio. The contract checks two things: net plus tax equals gross, and the declared rounding rule produces the supplied tax amount from net and rate.

None of those details is bookkeeping trivia. Remove the unit, mix the rate’s two representations, or let two services round at different stages, and the same price can stop balancing while every individual calculation still looks reasonable.

Null Is a Domain Decision

Sat, 01 Apr 2017 00:00:00 +0000

We added four nullable fields to preserve a pricing snapshot: net amount, tax rate, tax amount, and gross amount. Each field could now be absent. That did not make each field independently optional.

For older records, all four nulls meant one thing: this record predates the snapshot and does not contain captured pricing evidence. For new records, all four values had to arrive together. A row containing gross and net but no tax rate was not “partly known.” It was invalid.

A Migration Is a Production Change

Wed, 01 Mar 2017 00:00:00 +0000

We deployed an additive schema change which the database could execute without rewriting the table. On paper it was cheap. In production it waited indefinitely.

The statement needed an exclusive table lock. A long-running session already held a conflicting lock, so the migration queued. New application traffic then queued behind the migration. The column addition was cheap after lock acquisition; waiting to begin was the incident.

Another schema change looked cheap for the opposite reason: remove fields and tables which the application no longer read. The SQL was small, but the decision would erase financial history from data sets measured in tens of gigabytes. Neither change could be judged by reading the migration method alone.

A Transaction Defines What Must Succeed Together

Wed, 01 Feb 2017 00:00:00 +0000

A booking flow reserved funds, completed the remote booking, and captured the funds. Only then did synchronous response processing fail. The remote booking existed, the funds were captured, and the caller received a failure.

The capture was individually valid, and its own database transaction committed cleanly. It was still too early. We had drawn the success boundary around “remote booking returned” while the operation promised something larger: a complete booking result which the caller could use.

Let the Database Reject Impossible State

Sun, 01 Jan 2017 00:00:00 +0000

Two workers asked the same reasonable question: does this owner already have an active record? Both received the same honest answer: no. Each worker then created one.

Nothing in either request was obviously careless. The application checked first. The insert happened inside a transaction. Tests which ran one request at a time stayed green. The defect lived in the interval between the check and the write, where both transactions were allowed to be correct about a state which had already become stale.