Lifecycle rules are the first feature that makes momo delete data on a schedule rather than on request — which turns age, storage class, and delete markers into correctness problems. This post walks through the expiration and transition engine, why a background worker is the only safe place to run it, and how expiration composes with versioning’s delete markers.
Why S3 versioning and content-addressed storage seem to conflict — mutation identity vs content identity — and how momo keeps both: UUIDv4 version records pointing at deduplicated blobs, delete markers that shadow without deleting, and pagination that can only be computed after sorting. Teaches: the real semantics backup tools depend on, and where the version chain deliberately stops being distributed.
Why millions of small files per directory collapse a filesystem, and how a SeaweedFS-style volume layout fixes it: append-only superblocks, 8-byte-aligned needles with CRC32C frames, hole-punching deletion, and crash recovery by truncation. Teaches: the real cost of per-blob files, why alignment beats variable-length framing, and how to make crash recovery O(1).
Verify-on-read re-hashes the full object on every local read (~8x slower than write). A ReadVerifier seam skips re-verification for blobs verified this process, while scrub still catches bitrot.
How momo manages encryption keys without a central secret server: a pluggable provider chain (env, then config file), a per-node versioned key registry in local storage, and rotation that promotes a new key without any downtime. Teaches: why a provider chain beats a single source, how key versioning enables online rotation, and when rotation is and is not worth the complexity.
How momo isolates tenants without a central secret server: HKDF-derived per-tenant keys, default-deny ACLs evaluated per request, and a SHA-256 hash-chained audit log that makes tampering detectable. Teaches: key derivation vs. key storage, why ACL evaluation order matters, and how hash chaining turns a log into evidence.
CAS keys make verify-on-read free: central verification, x-amz-checksum echo, tombstone GC, and corruption surfacing.
S3 integrity checksums moved from a surface-level adapter into the storage/ingest core via a protocol-agnostic ChecksumProvider seam — every write path verified, replicas re-verify on receive.
The [storage] backend seam swaps blob storage (local, nfs, s3, raw) behind one Store interface, keeping CAS metadata local.
Weighted rendezvous hashing picks replicas deterministically and peer-equally without a central allocator — later hardened for failure domains. Teaches: content-hash-based placement, weighted rendezvous hashing, deterministic stable sort, failure domain filtering.