/Recent content on Momo Engineering Journal/images/og-image.png/images/og-image.pngHugoen-usWed, 28 Oct 2026 17:47:15 -0600/posts/060-r10-object-lock/Wed, 28 Oct 2026 12:00:00 +0000/posts/060-r10-object-lock/Object Lock is the S3 feature regulated workloads ask for first: write-once-read-many retention that the storage server itself enforces, not a policy the client promises to honor. This phase gives momo bucket-level lock enablement at creation, per-version retention and legal holds, and a delete path that answers 403 instead of quietly removing bytes. Teaches: how COMPLIANCE and GOVERNANCE differ in who can override them, why lock state must live beside the version record rather than inside object metadata, and which parts of AWS&rsquo;s surface momo deliberately does not pretend to implement./posts/059-r10-notification/Wed, 21 Oct 2026 07:14:25 +0000/posts/059-r10-notification/Implemented S3 bucket notification configurations with real webhook delivery, prefix/suffix filtering, exponential-backoff retries, and all six S3 event types — so backup tools and pipelines can react to object changes instead of polling. Teaches: why a notification dispatcher that returns nil without sending is the trap to watch for, and how events stay honest across plain PUT, multipart, and delete paths./posts/058-s3-lifecycle-on-cas/Wed, 21 Oct 2026 03:35:13 +0000/posts/058-s3-lifecycle-on-cas/Lifecycle rules are the first feature that makes momo delete data on a schedule rather than on request — which turns age, storage class, and delete markers into correctness problems. This post walks through the expiration and transition engine, why a background worker is the only safe place to run it, and how expiration composes with versioning&rsquo;s delete markers./posts/057-bolt-sigv4-zero-alloc-headers/Thu, 15 Oct 2026 11:43:50 +0000/posts/057-bolt-sigv4-zero-alloc-headers/Every authenticated S3 request rebuilds a SigV4 canonical-headers block — and the two obvious ways to write it (strings.Split and Header.Get) each allocate on the hottest authorization path. This post walks through replacing both with an IndexByte scan and an EqualFold lookup, and the golden-equivalence test that proves the rewrite is byte-identical for every request AWS clients actually send./posts/056-s3-versioning-on-cas/Sun, 11 Oct 2026 08:56:13 +0000/posts/056-s3-versioning-on-cas/Why S3 versioning and content-addressed storage seem to conflict — mutation identity vs content identity — and how momo keeps both: UUIDv4 version records pointing at deduplicated blobs, delete markers that shadow without deleting, and pagination that can only be computed after sorting. Teaches: the real semantics backup tools depend on, and where the version chain deliberately stops being distributed./posts/052-bolt-etag-matches-zero-alloc/Tue, 06 Oct 2026 11:16:37 +0000/posts/052-bolt-etag-matches-zero-alloc/How we removed the heap allocation from ETag list parsing on the S3 conditional-request path by replacing strings.Split with a strings.IndexByte scan. Teaches: what If-Match/If-None-Match actually carry, why Split allocates per request, and how a microbenchmark proves the win./posts/052-adaptive-systems-foundations/Tue, 06 Oct 2026 05:19:24 +0000/posts/052-adaptive-systems-foundations/How Momo replaces static concurrency limits and rigid replica selection with bio-inspired adaptive mechanisms: Ant Colony pheromone-based replica exploration and Epigenetic host-capacity admission./posts/055-adaptive-volume-storage/Tue, 06 Oct 2026 02:44:44 +0000/posts/055-adaptive-volume-storage/Why millions of small files per directory collapse a filesystem, and how a SeaweedFS-style volume layout fixes it: append-only superblocks, 8-byte-aligned needles with CRC32C frames, hole-punching deletion, and crash recovery by truncation. Teaches: the real cost of per-blob files, why alignment beats variable-length framing, and how to make crash recovery O(1)./posts/051-bolt-aws-chunked-zero-alloc/Thu, 01 Oct 2026 11:32:41 +0000/posts/051-bolt-aws-chunked-zero-alloc/How we removed the last heap allocation from the aws-chunked ingest path by replacing strings.Split and strings.TrimPrefix with a strings.IndexByte scan. Teaches: what a streaming S3 upload body looks like, why Split allocates per chunk, and how a microbenchmark proves the win./posts/049-eliminate-redundant-time-format-allocations/Sat, 12 Sep 2026 00:00:00 +0000/posts/049-eliminate-redundant-time-format-allocations/By replacing a second <code>time.Format</code> call with a simple string slice (<code>amzDate[:8]</code>), we eliminated redundant heap allocations and CPU overhead when generating timestamps for AWS SigV4 requests./posts/048-bolt-listparts-appendformat/Tue, 08 Sep 2026 11:29:58 +0000/posts/048-bolt-listparts-appendformat/How we eliminated the final per-response heap allocation in S3 XML rendering by swapping time.Format for time.AppendFormat in the ListParts handler. Teaches: the stack-buffer time formatting pattern, why Format allocates, and how microbenchmarks prove the win./posts/029-fuse-go-fuse-v2-migration/Tue, 01 Sep 2026 18:40:07 +0000/posts/029-fuse-go-fuse-v2-migration/The momofs FUSE mount traded its hand-rolled bazil.org/fuse adapter for the go-fuse/v2 high-level fs API, mapped all 22 node callbacks to go-fuse interfaces, and deprecated consistency=cached./posts/045-bolt-lastmodified-header/Tue, 01 Sep 2026 11:19:08 +0000/posts/045-bolt-lastmodified-header/time.Format allocates a fresh string on every S3 GET/HEAD response. Swapping to time.AppendFormat drops that heap escape to zero — measured 32 B/op saved per request on the hottest read path./posts/047-bolt-s3-copyresult-time-alloc/Sun, 30 Aug 2026 11:20:58 +0000/posts/047-bolt-s3-copyresult-time-alloc/How we optimized S3 XML responses by replacing time.Format with time.AppendFormat to eliminate heap allocations. Teaches: stack-buffer time formatting pattern, hot path identification via pprof./posts/023-momofs-fuse-transport/Fri, 28 Aug 2026 15:57:34 +0000/posts/023-momofs-fuse-transport/The FUSE transport (<code>momo -imp fs</code>): bazil.org/fuse binding, per-handle buffered writes, /dev/fuse-gated e2e, and a clean unmount lifecycle./posts/025-benchmark-benchstat-gate/Fri, 28 Aug 2026 00:32:50 +0000/posts/025-benchmark-benchstat-gate/Base-vs-branch benchmark comparison gates every PR — plus the runner-noise saga (S3PutSpool, LocalWrite, allowlist, apostrophe) that made it honest./posts/022-momofs-posix-core/Thu, 27 Aug 2026 22:40:31 +0000/posts/022-momofs-posix-core/The R4 POSIX core: inodes as content-addressed manifests, files as blobs, atomic rename, hardlinks, permission enforcement — a filesystem on momo./posts/021-r3-write-durability-quorum/Thu, 27 Aug 2026 19:24:05 +0000/posts/021-r3-write-durability-quorum/Ack only what survived: fsync/group-commit durability barrier, survivor quorum, and read-your-writes — the write-side of the P0 stack./posts/020-r2-degraded-read-self-heal/Thu, 27 Aug 2026 10:14:18 +0000/posts/020-r2-degraded-read-self-heal/When a local replica is corrupted or missing, degraded read fetches from healthy survivors transparently, while a background rebuild loop restores cluster redundancy./posts/019-r1-failure-domain-placement/Wed, 26 Aug 2026 23:52:57 +0000/posts/019-r1-failure-domain-placement/Replicas must not ride in the same rack: CRUSH weights become rack/zone/DC-aware so a single domain failure can&rsquo;t lose the data./posts/043-reduce-read-verify-hashing/Wed, 26 Aug 2026 19:00:59 +0000/posts/043-reduce-read-verify-hashing/Verify-on-read re-hashes the full object on every local read (~8x slower than write). A ReadVerifier seam skips re-verification for blobs verified this process, while scrub still catches bitrot./posts/042-perf-profiling-baseline/Wed, 26 Aug 2026 17:46:56 +0000/posts/042-perf-profiling-baseline/Phase-0 profiling harness via go test -cpuprofile/-memprofile; Rule 75 forbids networked pprof on unauthenticated listeners. The baseline proved stdlib SHA-256 already uses AVX2./posts/044-plugin-seam-architecture/Wed, 26 Aug 2026 16:45:18 +0000/posts/044-plugin-seam-architecture/Adaptive behaviors use compile-time Go interface seams injected at decision points — not dynamic plugins. The fast path stays concrete; the trust core stays auditable; policy is declarative./posts/026-metrics-observability/Tue, 25 Aug 2026 06:12:59 +0000/posts/026-metrics-observability/Prometheus metrics export with a per-node metrics_host/metrics_port bind — observability that scales past the &lsquo;just scrape node 0&rsquo; era./posts/054-secrets-management-key-rotation/Tue, 25 Aug 2026 03:25:35 +0000/posts/054-secrets-management-key-rotation/How momo manages encryption keys without a central secret server: a pluggable provider chain (env, then config file), a per-node versioned key registry in local storage, and rotation that promotes a new key without any downtime. Teaches: why a provider chain beats a single source, how key versioning enables online rotation, and when rotation is and is not worth the complexity./posts/053-multitenancy-authorization-audit/Tue, 25 Aug 2026 03:25:29 +0000/posts/053-multitenancy-authorization-audit/How momo isolates tenants without a central secret server: HKDF-derived per-tenant keys, default-deny ACLs evaluated per request, and a SHA-256 hash-chained audit log that makes tampering detectable. Teaches: key derivation vs. key storage, why ACL evaluation order matters, and how hash chaining turns a log into evidence./posts/032-r5-metrics-phases-2-4/Tue, 25 Aug 2026 03:25:22 +0000/posts/032-r5-metrics-phases-2-4/R5 Phases 2-4 shipped: storage metrics (disk, CAS, GC), P2P metrics (SWIM, leases, scatter-gather), and opt-in latency histograms — all with &lt;1% overhead via sync/atomic counters./posts/028-roadmap-and-research/Tue, 25 Aug 2026 03:24:12 +0000/posts/028-roadmap-and-research/After R1–R4: the P0 hardening done, the P1/P2 tracks (metrics, HA metadata, auth, secrets, S3 breadth) and the research reading guide that seeded it all./posts/035-s3-501-discipline-remaining-ops/Mon, 24 Aug 2026 21:44:12 +0000/posts/035-s3-501-discipline-remaining-ops/Final 501 sweep: SelectObjectContent, UploadPartCopy, analytics, inventory, metrics, intelligent-tiering now return honest 501 NotImplemented./posts/034-s3-501-discipline-object-subresources/Mon, 24 Aug 2026 21:13:07 +0000/posts/034-s3-501-discipline-object-subresources/5 object-level query parameters now return honest 501 NotImplemented instead of silently falling into GetObject/PutObject/DeleteObject./posts/033-s3-501-discipline-bucket-config/Mon, 24 Aug 2026 20:00:50 +0000/posts/033-s3-501-discipline-bucket-config/16 bucket config subresources (versioning, ACL, policy, CORS, lifecycle, etc.) now return honest 501 NotImplemented instead of silently misrouting to ListObjects/GetObject./posts/007-at-rest-integrity-and-gc/Mon, 24 Aug 2026 19:36:43 +0000/posts/007-at-rest-integrity-and-gc/CAS keys make verify-on-read free: central verification, x-amz-checksum echo, tombstone GC, and corruption surfacing./posts/031-core-integrity-verification/Mon, 24 Aug 2026 18:03:31 +0000/posts/031-core-integrity-verification/S3 integrity checksums moved from a surface-level adapter into the storage/ingest core via a protocol-agnostic ChecksumProvider seam — every write path verified, replicas re-verify on receive./posts/012-s3-integrity-checksums/Mon, 24 Aug 2026 16:36:34 +0000/posts/012-s3-integrity-checksums/x-amz-checksum-CRC32/SHA256 echo on PUT; integrity surfaced to clients so dirty bytes never masquerade as clean. Teaches: client-verifiable integrity, Bolt zero-allocation on read path, Sentinel honesty guarantee./posts/036-s3-listxml-appendformat-optimization/Mon, 24 Aug 2026 11:24:01 +0000/posts/036-s3-listxml-appendformat-optimization/S3 ListObjectsV2 XML serialization dropped from ~1000 allocations to 16 per op via inlined time formatting and a pre-allocated escape buffer./posts/018-adaptive-scaling-peer-quality/Fri, 14 Aug 2026 19:43:24 +0000/posts/018-adaptive-scaling-peer-quality/Three adaptive feedback loops: gossip fanout scales with cluster size, streaming chunk size adapts to cipher/memory, peer quality feeds quorum decisions./posts/014-confidential-dedup-oprf/Fri, 14 Aug 2026 00:26:56 +0000/posts/014-confidential-dedup-oprf/Threshold-OPRF lets E2EE cas blobs dedup without revealing content hashes — plus adaptive failed-auth backoff and lockout./posts/017-scatter-gather-lease-quorum/Thu, 13 Aug 2026 07:58:21 +0000/posts/017-scatter-gather-lease-quorum/Scatter-gather lists across shard owners; leases with correct majority quorum — and the two nasty off-by-one/partition bugs found by audit./posts/009-s3-multipart-and-breadth/Thu, 13 Aug 2026 06:02:26 +0000/posts/009-s3-multipart-and-breadth/Multipart upload support plus a rigorous 501 strategy for unsupported S3 subresources — honest breadths over silent misbehavior./posts/024-bolt-performance-engineering/Wed, 12 Aug 2026 11:40:41 +0000/posts/024-bolt-performance-engineering/The Bolt mindset codified: no heap escapes in hashing/encoding, deadline amortization cuts SetDeadline ~98%, allocation hotspots hunted by profile. Teaches: zero-escape patterns, measurement contracts, trade-off posture./posts/011-s3-https-tls-enforcement/Wed, 12 Aug 2026 07:23:29 +0000/posts/011-s3-https-tls-enforcement/The S3 gateway turned HTTPS-or-explicit-insecure into the default: no silent plaintext for the gateway or inbound tcp/quic./posts/010-s3-auth-presigned-sigv4/Tue, 11 Aug 2026 23:11:16 +0000/posts/010-s3-auth-presigned-sigv4/Query-string SigV4 for presigned URLs, mandatory X-Amz-Date freshness, and decoupling gateway access/secret keys from the native auth token./posts/008-s3-gateway-core/Tue, 11 Aug 2026 18:25:16 +0000/posts/008-s3-gateway-core/The S3 gateway: SigV4, S3-compliant XML errors, Head/List/Copy, pagination, ranges, and metadata — a compatible API surface for real clients./posts/013-e2ee-envelope-encryption/Tue, 11 Aug 2026 16:36:53 +0000/posts/013-e2ee-envelope-encryption/Envelope encryption with a client-held key: the server stores ciphertext and can&rsquo;t read it — across S3 and native transports./posts/039-signed-payload-sse-s3/Tue, 11 Aug 2026 05:14:24 +0000/posts/039-signed-payload-sse-s3/SigV4 signed payload verification for S3 PUT/POST and Server-Sent Events for async operation notifications./posts/040-aws-chunked-streaming/Tue, 11 Aug 2026 04:52:58 +0000/posts/040-aws-chunked-streaming/aws-chunked payload encoding streams large S3 uploads in signed chunks — no full-body buffering, per-chunk SigV4 integrity./posts/003-transport-tcp-to-quic/Tue, 11 Aug 2026 04:02:50 +0000/posts/003-transport-tcp-to-quic/Momo&rsquo;s transport layer grew from TCP to a QUIC/TLS 1.3 fan-out — fixing handshake, ACK framing, deadlines, and TLS identity along the way./posts/015-sentinel-security-audit/Tue, 04 Aug 2026 00:47:24 +0000/posts/015-sentinel-security-audit/A 76-deep audit (authed via Sentinel) surfaced CRLF injection, smuggling, traversal and path leaks — the pattern that forged momo&rsquo;s security mindset./posts/006-pluggable-storage-backends/Sun, 26 Jul 2026 14:33:59 +0000/posts/006-pluggable-storage-backends/The <code>[storage] backend</code> seam swaps blob storage (local, nfs, s3, raw) behind one Store interface, keeping CAS metadata local./posts/030-external-s3-client-replication-downgrade/Wed, 01 Jul 2026 23:36:39 +0000/posts/030-external-s3-client-replication-downgrade/External S3 clients like aws-cli don&rsquo;t send momo headers — the server detects this and downgrades client-side replication modes to server-side alternatives, ensuring replication never silently drops./posts/005-crush-placement/Tue, 30 Jun 2026 13:09:01 +0000/posts/005-crush-placement/Weighted rendezvous hashing picks replicas deterministically and peer-equally without a central allocator — later hardened for failure domains. Teaches: content-hash-based placement, weighted rendezvous hashing, deterministic stable sort, failure domain filtering./posts/037-zero-crash-hardening-patterns/Wed, 03 Jun 2026 06:03:39 +0000/posts/037-zero-crash-hardening-patterns/Systematic defensive coding: nil-safety, numeric overflow guards, resource lifecycle discipline, panic recovery at every boundary, and concurrency safety under -race./posts/002-replication-strategies-polymorphic/Mon, 04 May 2026 18:54:37 +0000/posts/002-replication-strategies-polymorphic/Chain, Splay, and Primary-Splay replication — and the metrics-driven controller that switches between them at runtime./posts/004-cas-content-addressable-store/Wed, 11 Mar 2026 14:34:18 +0000/posts/004-cas-content-addressable-store/Identify every object by its SHA-256 content hash — server-side dedup, verify-on-read, and a path that later mounted a filesystem. Teaches: content addressing principles, single write chokepoint, Bolt/Sentinel patterns./posts/016-p2p-gossip-swim/Wed, 11 Mar 2026 14:34:18 +0000/posts/016-p2p-gossip-swim/A scaleless-gossip ring: SWIM-style failure detection, discovered-peer dialing, and ALIVE/OFFLINE restoration — momo&rsquo;s nervous system./posts/001-origin-and-genesis/Tue, 09 Sep 2025 20:10:05 +0000/posts/001-origin-and-genesis/How momo started as a file-replication playground in Go and grew into a distributed content-addressed object store.