Storage that quietly returns wrong bytes is worse than storage that fails loudly. A corrupted read that looks successful can poison downstream systems for days before anyone notices. This arc turned momo’s content-addressing property into explicit, enforced at-rest integrity.
The Real Problem
Because momo names every blob by the SHA-256 hash of its contents (004), the stored key is the checksum. In principle that makes read-path integrity free: re-hash the bytes, compare to the key, and you know whether they are intact. But “in principle” was doing a lot of work. Verification was scattered across handlers, some paths skipped it, and there was no consistent answer to what happens when a mismatch is found.
We also had a garbage-collection gap. Deleting an object had to remove its bytes without deleting a blob another object still referenced ā and the deletion path had no single, auditable owner.
What Landed
- Central integrity verification (#911): one validate-then-verify path shared by every reader, instead of ad-hoc checks spread through handlers. A metadata accessor was added so a metadata query no longer has to open the content stream just to read a size or a timestamp.
- Verify-on-read with corruption surfacing (#925): when a re-hash does not match the key, the replica is marked suspect rather than silently served. That suspicion is the input to the degraded-read and self-heal arc (020).
- Checksums on the wire: S3 clients can ask for a checksum and momo echoes it back, so a caller can verify the transfer end to end (012).
- Garbage collection: a tombstone ā a deletion marker recorded in metadata ā drives a reference-count sweep. Applying a tombstone deletes the blob content, failures to delete are surfaced instead of swallowed, and the sweep entry point is guarded against being started twice.
š” Sentinel lens
At-rest integrity is a security property, not just a performance nicety: silent bitrot becomes a loud, auditable mismatch that the system can act on. The same sweep that added verification also fixed CRLF injection and path traversal in the blob and metadata layers (015). The principle we traced through it: integrity checks must be compiled into the core and can never be skipped by swapping a seam. If a backend could opt out of verification, the guarantee would be only as strong as the weakest backend.
ā” Bolt lens
- A single combined metadata read replaced three separate key/value views, reducing write-path transactions and CPU.
- Verify-on-read reuses the zero-escape SHA-256 hashing path from 024, so verification does not allocate per object.
See docs/CORE/STANDARDS.md for the ā” Bolt / š” Sentinel mindsets.