recovering a format nobody documented

The archive had to be pulled incrementally — a day at a time, resumable, verifiable — and the only description of the wire encoding was that it existed. Published notes covered the container and the field order. They did not cover how the values were packed.

The encoding turned out to be a delta scheme: most values are not transmitted at all, only their difference from a prior reference, and those differences are themselves packed in a way that had to be inferred rather than read. Getting it wrong does not crash. It produces plausible numbers, off by a little, and the error would surface months later inside a result nobody could explain.

So the archiver was built checks first. Sharded day files. A content manifest per shard, so any shard can be re-verified without trusting the pull that produced it. Resume state kept separate from the data. And a decode path compared against known-good full-value records before the incremental path is allowed to feed anything downstream.

The wire also carried a depth field that the producers had stopped maintaining, which meant the boundary of usable history had to be measured rather than assumed — a paced probe searching for the first period that still returns data, instead of reading a stated start date that was only aspirational.

It proves the work does not stop at the documented surface. Where the truth sits in an undocumented layer, the team reconstructs it, then proves the reconstruction before building on it.