Architectural decisions, design records (ADRs), system specifications, and structural evolution behind momo’s distributed object store.
x-amz-checksum-CRC32/SHA256 echo on PUT; integrity surfaced to clients so dirty bytes never masquerade as clean. Teaches: client-verifiable integrity, Bolt zero-allocation on read path, Sentinel honesty guarantee.
Three adaptive feedback loops: gossip fanout scales with cluster size, streaming chunk size adapts to cipher/memory, peer quality feeds quorum decisions.
Threshold-OPRF lets E2EE cas blobs dedup without revealing content hashes — plus adaptive failed-auth backoff and lockout.
Scatter-gather lists across shard owners; leases with correct majority quorum — and the two nasty off-by-one/partition bugs found by audit.
Multipart upload support plus a rigorous 501 strategy for unsupported S3 subresources — honest breadths over silent misbehavior.
The Bolt mindset codified: no heap escapes in hashing/encoding, deadline amortization cuts SetDeadline ~98%, allocation hotspots hunted by profile. Teaches: zero-escape patterns, measurement contracts, trade-off posture.
The S3 gateway turned HTTPS-or-explicit-insecure into the default: no silent plaintext for the gateway or inbound tcp/quic.
Query-string SigV4 for presigned URLs, mandatory X-Amz-Date freshness, and decoupling gateway access/secret keys from the native auth token.
The S3 gateway: SigV4, S3-compliant XML errors, Head/List/Copy, pagination, ranges, and metadata — a compatible API surface for real clients.
Envelope encryption with a client-held key: the server stores ciphertext and can’t read it — across S3 and native transports.