pnclENGINEGet API access

CAPACITY NOTES

Change your odds schema without breaking the poller

Your price table will need a new column one day. Additive-only changes, a view for consumers, and a backfill policy that keeps the poller running.

The day comes when your price table needs one more field: a source age, a fair price computed at ingest, a flag for specials. A schema change on a live pipeline is where weekend projects go to die, unless you follow three rules: changes are additive only, consumers read through a view, and backfills are a separate job with a separate budget.

Add, never change

The only safe migration on a running table is addition. Add the column nullable, or with a default that means unknown, and the poller's inserts keep working because they never named the column. The storage guide's table design, one row per changed price with the full identity in the key, is friendly to exactly this: the key never changes, only the payload beside it, checked 2026-10-16.

Renames, type changes and semantic changes are for the next table, not this one. A column whose meaning shifts is a new column, and the old one retires through the view.

Consumers read through a view

Your dashboard and your exports should never touch the physical table. A view presents the current shape: the columns consumers expect, in the order they expect, with the new column appearing when it is ready. When the next migration lands, the table grows and the view absorbs it, and nothing downstream notices.

This is also where the retired column disappears: it stays in the table for as long as your retention keeps it, and the view simply stops selecting it.

Backfills are their own job

A new column starts empty, and the temptation is to backfill it inline, inside the poller. Resist it. The backfill is a separate job that walks the table in key order, computes the new field for old rows, and writes in batches, on its own schedule and its own rate budget. The rate limits guide's headroom rule applies here too if the backfill needs source data: it runs inside the same allowance as your polling, not beside it.

A backfill that can stop and resume, keyed on the last row it touched, is a backfill that survives the deploy that started it.

Test the migration like the parser

The offline-testing habit applies to schemas too: replay a week of recorded responses through the new code, and confirm the new column fills exactly as expected before the table ever sees it. A migration tested on fixtures is a migration that runs once, in daylight, with a log you can read.

Add only, view in front, backfill as a job. Then the schema grows like the data does: one careful row at a time.