perf(snapshot): 2.18 GB → 1.33 GB archive — long-window zstd + cross-doc raw dedup
The weekly snapshot had silently outgrown its ceiling: a CI-equivalent
build measured 2,183 MB (2.03 GiB) — OVER the 1.9 GiB guard, so the next
scheduled release would have failed at the size gate (the corpus gained
~35k pages when index-diff discovery closed the reference-crawl gap).
Three levers, all measured on the real corpus:
1. zstd --long=27 (128 MB match window): the dominant ratio lever on this
corpus — DocC JSON and SVG redundancy far exceeds the default level-9
window. Alone: 2,183 → 1,904 MB AND 65 s → 24 s to compress. The old
'no --long' constraint predates Bun's window support: verified
decodable via both Bun.zstdDecompressSync and
DecompressionStream('zstd'), and via the real consumer extract path
(streaming, bounded memory, 96 s for the full archive).
2. Raw payloads ship as PLAIN TEXT inside the snapshot DB instead of
per-blob zstd frames. Per-blob encoding hid the massive cross-document
JSON redundancy from the archiver; the long-window outer pass beats it
~4× (measured 1.3 MB vs 5.5 MB on a 40 MB sample) — ~1.1 GB of
near-incompressible archive bytes become ~0.3 GB. decodeSectionContent
is type-directed, so consumers read either representation; setup
re-encodes per-blob after install (id-cursor batch, VACUUM) to restore
the compact at-rest footprint. Correctness never depends on the pass.
3. Dead weight out of the staged tree: matrix-*.admx (an orphan
converted-weights cache no code references — 129 MB of incompressible
fp32 duplicating model.onnx) and the legacy pages content columns
(~246 MB of duplicate text the persist path no longer writes and
nothing reads).
Result: 1,326 MB (1.23 GiB) — 39% smaller, back under the guard with
~35% headroom, and faster to build. Completeness audited against the live
corpus (documents/pages/sections/relationships/body-FTS/symbols/fonts
counts all identical; 266,436 SVGs; 155 fonts; model files pinned) and
the CI determinism gate verified bit-identical across two builds.
Not taken: stripping rebuildable FTS tables (~90 MB compressed) — it
would leave snapshots unsearchable until a rebuild, breaking the
extract-and-go property; documented as a future lever if the ceiling
looms again. G
Gigi committed
929910ecd70a946d26ecb7b9b8931b2c93ba2b9b
Parent: 21a9031