field notes LATTICE

delete the dumps

Computational exploration has a storage shape that nobody plans for. Every lane that searches writes intermediate tables, and those tables are not results; they are the exhaust of arriving at one. Over a long run the exhaust came to dominate the repository by orders of magnitude, until routine operations were mostly moving byproducts from place to place.

Several gigabytes of generated material had accumulated: tables of coefficients, a stray generated page, output belonging to lanes that finished long before. At that size the cost is no longer disk. It is that every copy and backup carries material nobody will read, and that material is indistinguishable from the work unless you happen to know which lane wrote it.

The tension is genuine. Reproducibility argues for keeping intermediates, and every practical operation argues for destroying them. The line drawn was not large against small but result against byproduct. The generating procedure stays in the history, which makes the tables reconstructible in principle; the tables themselves go.

The move was two commits rather than one: a deletion, and a rule that stops the same shapes from being committed again. The second is the one that lasts. A purge without a rule is a purge you will perform again next month, and the operation had clearly reached the point where the same classes of file kept coming back.

The cost is irreversibility. Anything someone later wants from those tables is gone, and the only defence is an argument about what the procedure can regenerate. What it proves is that the operation can distinguish what it produced from what it must keep, and will destroy the first while the second is still being written.

drawn from: commit cluster · a purge and an ignore rule

Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.