/categories/storage/Recent content in Storage on Momo Engineering Journal/images/og-image.png/images/og-image.pngHugoen-usWed, 28 Oct 2026 17:47:15 -0600/posts/058-s3-lifecycle-on-cas/Wed, 21 Oct 2026 03:35:13 +0000/posts/058-s3-lifecycle-on-cas/Lifecycle rules are the first feature that makes momo delete data on a schedule rather than on request — which turns age, storage class, and delete markers into correctness problems. This post walks through the expiration and transition engine, why a background worker is the only safe place to run it, and how expiration composes with versioning&rsquo;s delete markers./posts/056-s3-versioning-on-cas/Sun, 11 Oct 2026 08:56:13 +0000/posts/056-s3-versioning-on-cas/Why S3 versioning and content-addressed storage seem to conflict — mutation identity vs content identity — and how momo keeps both: UUIDv4 version records pointing at deduplicated blobs, delete markers that shadow without deleting, and pagination that can only be computed after sorting. Teaches: the real semantics backup tools depend on, and where the version chain deliberately stops being distributed./posts/055-adaptive-volume-storage/Tue, 06 Oct 2026 02:44:44 +0000/posts/055-adaptive-volume-storage/Why millions of small files per directory collapse a filesystem, and how a SeaweedFS-style volume layout fixes it: append-only superblocks, 8-byte-aligned needles with CRC32C frames, hole-punching deletion, and crash recovery by truncation. Teaches: the real cost of per-blob files, why alignment beats variable-length framing, and how to make crash recovery O(1)./posts/043-reduce-read-verify-hashing/Wed, 26 Aug 2026 19:00:59 +0000/posts/043-reduce-read-verify-hashing/Verify-on-read re-hashes the full object on every local read (~8x slower than write). A ReadVerifier seam skips re-verification for blobs verified this process, while scrub still catches bitrot./posts/054-secrets-management-key-rotation/Tue, 25 Aug 2026 03:25:35 +0000/posts/054-secrets-management-key-rotation/How momo manages encryption keys without a central secret server: a pluggable provider chain (env, then config file), a per-node versioned key registry in local storage, and rotation that promotes a new key without any downtime. Teaches: why a provider chain beats a single source, how key versioning enables online rotation, and when rotation is and is not worth the complexity./posts/053-multitenancy-authorization-audit/Tue, 25 Aug 2026 03:25:29 +0000/posts/053-multitenancy-authorization-audit/How momo isolates tenants without a central secret server: HKDF-derived per-tenant keys, default-deny ACLs evaluated per request, and a SHA-256 hash-chained audit log that makes tampering detectable. Teaches: key derivation vs. key storage, why ACL evaluation order matters, and how hash chaining turns a log into evidence./posts/007-at-rest-integrity-and-gc/Mon, 24 Aug 2026 19:36:43 +0000/posts/007-at-rest-integrity-and-gc/CAS keys make verify-on-read free: central verification, x-amz-checksum echo, tombstone GC, and corruption surfacing./posts/031-core-integrity-verification/Mon, 24 Aug 2026 18:03:31 +0000/posts/031-core-integrity-verification/S3 integrity checksums moved from a surface-level adapter into the storage/ingest core via a protocol-agnostic ChecksumProvider seam — every write path verified, replicas re-verify on receive./posts/006-pluggable-storage-backends/Sun, 26 Jul 2026 14:33:59 +0000/posts/006-pluggable-storage-backends/The <code>[storage] backend</code> seam swaps blob storage (local, nfs, s3, raw) behind one Store interface, keeping CAS metadata local./posts/005-crush-placement/Tue, 30 Jun 2026 13:09:01 +0000/posts/005-crush-placement/Weighted rendezvous hashing picks replicas deterministically and peer-equally without a central allocator — later hardened for failure domains. Teaches: content-hash-based placement, weighted rendezvous hashing, deterministic stable sort, failure domain filtering./posts/004-cas-content-addressable-store/Wed, 11 Mar 2026 14:34:18 +0000/posts/004-cas-content-addressable-store/Identify every object by its SHA-256 content hash — server-side dedup, verify-on-read, and a path that later mounted a filesystem. Teaches: content addressing principles, single write chokepoint, Bolt/Sentinel patterns.