[{"content":"The Real Problem Every storage system eventually meets a workload that cannot be allowed to delete its own data. Financial records must survive a compliance window. Backups must survive a compromised application account. Legal teams issue holds that outlive any bucket policy.\nThe naive answer is \u0026ldquo;the client promises not to delete.\u0026rdquo; That promise is worth exactly nothing the moment the same credentials are used by a bug, a script, or an attacker. The point of S3 Object Lock is that the server enforces the promise: a locked version cannot be removed until its retention expires, no matter who asks.\nTwo independent mechanisms exist in the S3 model:\nRetention — a deadline attached to a version, in one of two modes. GOVERNANCE mode may be bypassed by a caller holding a special permission (x-amz-bypass-governance-retention: true). COMPLIANCE mode may not be bypassed by anyone, including the bucket owner, until the deadline passes. Legal hold — a boolean flag with no expiry. It is set and cleared by an authorized caller, independent of retention, and it overrides everything. Why Lock State Belongs to the Version An object name is not a stable identity. Overwrite report.csv five times and you have five different byte streams behind one key. If lock state were stored as object metadata, the sixth write would silently discard the previous version\u0026rsquo;s retention — and with it the entire guarantee.\nSo the lock record is keyed by the triple bucket, key, version. It lives in its own storage bucket beside the version records that R10\u0026rsquo;s versioning phase introduced. The consequences are worth stating plainly:\nOverwriting a locked key is allowed. It creates a new version. The locked version is untouched, so WORM is preserved without blocking ordinary writes. Deleting the name is allowed. That creates a delete marker, which hides the object from plain reads while every locked version remains recoverable. Deleting a specific version is where enforcement happens: if that version carries an active retention deadline or a legal hold, the delete is refused. That third rule is the one that has to be implemented inside the storage layer, not only in the HTTP handler. A handler-level check protects one protocol; the native protocol, a maintenance job, or a future API would walk straight past it. Momo\u0026rsquo;s version store therefore returns a permission error for a protected delete, and each protocol maps that error into its own vocabulary — HTTP 403 AccessDenied for the S3 surface.\nEnabling Lock: Creation Only AWS lets you turn Object Lock on only when the bucket is created. Momo keeps that rule for the same reason AWS does: a lock that can be switched off after the fact protects nobody. The gateway honors the creation-time header x-amz-bucket-object-lock-enabled: true, and enables versioning implicitly, because retention without versions is meaningless.\nA later PUT ?object-lock is refused with 409 InvalidBucketState, whether it tries to enable or disable. GET ?object-lock reports the enabled document, or 404 NoSuchObjectLockConfiguration for an ordinary bucket. A storage backend without lock support answers an honest 501 NotImplemented rather than pretending — the same discipline momo applies to every unsupported corner of the S3 surface.\nTwo smaller decisions follow from the same logic. Enabling lock on a bucket whose versioning was suspended is refused, because WORM would silently stop capturing versions. And COMPLIANCE retention may be extended but never shortened or downgraded to GOVERNANCE while it is active — a rule enforced by comparing the new state against the stored one, using the clock supplied by the request rather than a storage-side timer.\nThe Capture Ceiling, Stated Out Loud A single-part PUT in momo\u0026rsquo;s gateway streams the request body to storage while the response is decided; the version identifier is only known after the write. That ordering makes it impossible to attach lock headers to that version during the first phase. Rather than fake success, the phase documents the ceiling: lock headers are captured on multipart completion, where the version id exists before the response is sent. Clients that need retention on single-part writes use multipart uploads, which backup and archival tools already do.\nThis is the same honesty rule the gateway applies elsewhere: a feature either works, or it says clearly that it does not.\nWhat an Operator Sees Reading a specific version now echoes the lock state back:\nx-amz-object-lock-mode — COMPLIANCE or GOVERNANCE x-amz-object-lock-retain-until-date — the deadline, in S3 timestamp form x-amz-object-lock-legal-hold-status — ON when a hold is set A delete attempt against a protected version returns 403 AccessDenied and leaves the record intact. A GOVERNANCE-locked version yields to a request carrying the bypass header; a COMPLIANCE-locked or held version never does.\nFailure Modes Worth Knowing Clock trust. Retention compares against the request clock, which is the same assumption S3 makes. A node with a badly skewed clock will evaluate deadlines incorrectly; keep NTP healthy, as you would for TLS validity. Versioning suspension. Once lock is enabled, momo refuses to suspend versioning, because doing so would break the WORM contract silently. The native protocol. Lock state is enforced by the version store, so deletes that go through S3 are covered. The native protocol\u0026rsquo;s plain delete operates on names and produces a tombstone without consulting lock records — an intentional boundary, documented rather than hidden. When NOT to Use This Object Lock is not a backup strategy. It prevents deletion of versions that exist; it does not create copies, survive disk failure, or replicate across failure domains. Use it together with replication and scrubbing, not instead of them. It is also a poor fit for high-churn working data: every retained version keeps its bytes referenced, and storage grows monotonically until retention expires.\nReferences / Dig deeper OpenSpec change: openspec/changes/r10-s3-breadth ADR: docs/adr/0058-r10-s3-breadth.md Tracking issue: https://github.com/alsotoes/momo/issues/938 Related phases: S3 versioning, S3 lifecycle, S3 notifications ","permalink":"/posts/060-r10-object-lock/","summary":"Object Lock is the S3 feature regulated workloads ask for first: write-once-read-many retention that the storage server itself enforces, not a policy the client promises to honor. This phase gives momo bucket-level lock enablement at creation, per-version retention and legal holds, and a delete path that answers 403 instead of quietly removing bytes. Teaches: how COMPLIANCE and GOVERNANCE differ in who can override them, why lock state must live beside the version record rather than inside object metadata, and which parts of AWS\u0026rsquo;s surface momo deliberately does not pretend to implement.","title":"R10 Phase 4: S3 Object Lock — WORM That Actually Refuses Deletes"},{"content":"The Real Problem Production S3 workflows aren\u0026rsquo;t just PUT/GET/DELETE. Backup tools (Veeam, restic), CI/CD pipelines, and data pipelines need to react to object changes — not poll for them. S3\u0026rsquo;s ?notification API lets a client register webhooks that fire on object events (s3:ObjectCreated:Put, s3:ObjectRemoved:Delete, …) with prefix/suffix filtering.\nWithout it, momo couldn\u0026rsquo;t integrate with:\nBackup tools (Veeam, restic) that want post-backup verification hooks Data pipelines that trigger downstream processing on new data Audit/logging systems that want a push-based event stream Custom workflows that chain S3 operations together Why Obvious Solutions Failed 1. Polling \u0026ldquo;Just poll ListObjectsV2\u0026rdquo; — polling wastes API calls, adds latency proportional to the polling interval, and cannot detect deletes reliably: a list shows what is there, never what was there.\n2. Database triggers \u0026ldquo;Put triggers in the DB\u0026rdquo; — momo\u0026rsquo;s storage is content-addressable. The object namespace is a thin metadata layer over content hashes, so there is no stable \u0026ldquo;row\u0026rdquo; per object to hang a trigger on. Identical content is one physical blob shared by many logical keys; the thing you want to notify about is the logical write, which the CAS layer does not model.\n3. An external message queue \u0026ldquo;Just use Kafka/RabbitMQ\u0026rdquo; — that adds an operational dependency and a second source of truth. The goal is a self-contained S3 gateway that works out of the box; an operator who wants a queue can point a webhook at their own gateway.\nSolution: S3 Notification on CAS Architecture Notification configuration is storage state; the S3 gateway is only an adapter over it. That split mirrors momo\u0026rsquo;s protocol/cluster separation (ADR 0053): the storage layer owns the rules, the transport layer owns the XML, and the server layer decides when a write counts as an event.\n1 2 3 4 5 6 7 8 S3 client ──PUT ?notification──▶ s3_communicator ──▶ NotificationStore (XML parse/format) (storage rules) │ object write (PUT / DELETE / multipart complete) │ │ │ ▼ ▼ server.afterS3ObjectWriteBucket ──────▶ DispatchNotification ─▶ HTTP POST (version + event) (rule match) (webhook) Key design decisions 1. Storage owns the configuration, not the gateway NotificationConfig lives in the storage layer (a dedicated notification bucket keyed by bucket name). The gateway holds only a NotificationStore interface. This keeps the S3 gateway protocol-agnostic — the same storage backend serves native TCP/QUIC and S3, and a store that does not implement the capability simply answers 501 NotImplemented (the honest-rejection posture used across momo\u0026rsquo;s S3 surface).\n2. Rules are evaluated at dispatch time DispatchNotification(bucket, key, versionID, eventType, now, retryMax) walks the bucket\u0026rsquo;s enabled rules and matches on:\nEvent type — six S3 events: s3:ObjectCreated:Put|Post|Copy|CompleteMultipartUpload and s3:ObjectRemoved:Delete|DeleteMarkerCreated Prefix filter (uploads/) Suffix filter (.log) Every matching rule gets its own delivery attempt, and one rule failing does not stop the others: the outcome reports both Dispatched and Failed counts.\n3. Real webhook delivery with exponential backoff This is where a naive implementation goes wrong. It is easy to write a dispatchToTarget that builds a payload and returns nil without ever sending it — the config CRUD tests pass and the feature looks done, but no bytes leave the process. momo\u0026rsquo;s spec (R10-N1) requires the POST to actually happen. So delivery is a real http.Client call:\nThe body is the S3 event document — a Records array carrying the bucket name, the object key, and the version id. Network errors and 5xx responses are retried with exponential backoff, up to [s3] notification_retry_max attempts (default 3). A 4xx response returns immediately: the server has already made a decision about the request, and retrying a rejected request only amplifies load. The event time comes from an injected now, not time.Now() inside the dispatcher, so tests stay deterministic. The payload is built from typed structs, not map[string]interface{}: the bucket name and object key go where they belong, which sounds obvious until a first draft puts the rule\u0026rsquo;s ARNs in those fields and ships a payload that is structurally valid and semantically wrong.\n4. Events fire from every write path, not just the happy one A notification that only fires on the simplest PUT is a trap. Objects reach storage through several paths:\nplain PUT and dedup-hit PUTs — recorded by the server\u0026rsquo;s ingest hook (afterS3ObjectWriteBucket), which also records the version on versioned buckets CompleteMultipartUpload — assembled inside the gateway, so it dispatches its own ObjectCreated:CompleteMultipartUpload single and batch DELETE — dispatch ObjectRemoved:Delete Native (non-S3) writes never notify: notifications are an S3-surface concept, gated on IsExternalClient().\n5. CAS dedup is untouched Versioning and notification do not interfere with content addressing. Identical content still shares one physical blob; notifications fire on the logical key, not the content hash.\nTesting The interesting tests are the ones that prove delivery, not just matching:\nStorage: delivery to a real httptest server with payload assertions; prefix/suffix/event filtering; retry-then-succeed with attempt counting; exhausted retries counted as Failed; 4xx not retried; unsupported target reported as a failure rather than a silent success Transport: XML parse/format round-trip, all six event mappings, invalid rules, every handler branch including the 501/404/400/500 paths, and the gateway dispatch helper\u0026rsquo;s retry-budget default Server: the ingest hook end-to-end against a real CASStore — version recorded and webhook delivered — plus the native-write no-op Shortcuts and ceilings Shortcut: HTTP webhooks only. SQS/SNS/Lambda destinations parse and are rejected with an explicit ENOSYS at dispatch rather than silently succeeding. Upgrade: implement one delivery adapter per destination type. Shortcut: synchronous dispatch. Delivery runs inline in the write path, bounded by the retry budget, so a slow webhook adds latency to the write. An async worker with a persisted retry queue is the upgrade path when webhook latency must not touch the request path. Ceiling: no retry persistence. Retries are in-memory; a crash between attempts loses the pending delivery. References / Dig deeper Spec: openspec/changes/r10-s3-breadth/specs/r10-s3-breadth/spec.md (R10-N1) ADR: docs/adr/0058-r10-s3-breadth.md Issue: #938 (R10 S3 breadth) Related: R10 Phase 1 (Versioning, 056), Phase 2 (Lifecycle, 058) Source: src/storage/notification.go, src/transport/s3_notification.go, src/transport/s3_communicator.go, src/server/server.go ","permalink":"/posts/059-r10-notification/","summary":"Implemented S3 bucket notification configurations with real webhook delivery, prefix/suffix filtering, exponential-backoff retries, and all six S3 event types — so backup tools and pipelines can react to object changes instead of polling. Teaches: why a notification dispatcher that returns nil without sending is the trap to watch for, and how events stay honest across plain PUT, multipart, and delete paths.","title":"R10 Phase 3: S3 Notification — Event-Driven Webhooks on CAS"},{"content":"Every storage system eventually grows a feature that deletes data without anyone asking. Until now, momo only removed an object when a client sent a DELETE: every byte on disk was there because a request put it there. Lifecycle rules change that. A bucket policy like \u0026ldquo;expire anything under logs/ after 30 days\u0026rdquo; means the server now removes data on its own, on a clock — and that single change turns time into a correctness dimension.\nThis post covers momo\u0026rsquo;s lifecycle implementation: expiration, transitions, and the reconciliation loop that drives them.\nThe Real Problem: deletion without a request Two S3 lifecycle actions matter in practice:\nExpiration — delete an object once it is older than N days. Transition — move an object to a colder storage class once it is older than N days. Both depend on age, which no client tells the server. Age is now − last-modified. So the server must (a) remember when each object was written, (b) periodically compare that to the current time, and (c) act. That is a reconciliation loop — the same shape as garbage collection and scrubbing, but with a user-supplied policy.\nThe tempting shortcut is to compute expiration lazily: only when an object is read, check whether it should have expired. That is wrong for several reasons. An object nobody reads never expires, so a \u0026ldquo;delete after 30 days\u0026rdquo; rule silently keeps data forever. Billing, compliance, and data-minimization guarantees all break. And a read-path check would put a policy evaluation in the hot path of every GET — the opposite of what we want.\nWhy \u0026ldquo;just run a cron\u0026rdquo; is not enough The other naive approach is a one-shot sweep triggered by a timer that iterates and deletes. The problems are concurrency and ordering:\nA sweep must not race with in-flight writes. If an object is being re-uploaded as the sweep decides to expire it, you can delete the new version. On a versioned bucket, \u0026ldquo;expire\u0026rdquo; does not mean \u0026ldquo;delete the bytes\u0026rdquo;. S3 semantics say expiration of the current version creates a delete marker — a tombstone that hides the object but preserves the previous version. Deleting the bytes would destroy history the user explicitly asked to keep. Transitions must not rewrite content. Moving an object to a colder class is a metadata change, not a data copy. Each of these is a decision the storage layer — not the timer — has to make, because only the storage layer knows about versions, metadata, and locks.\nThe Solution: a policy engine in storage, a clock in the server The split follows the project\u0026rsquo;s standing boundary: storage owns correctness, the server owns scheduling.\nThe policy lives in storage. A LifecycleConfig is an ordered list of rules, stored per bucket in its own BoltDB bucket. Each rule carries a prefix, an optional ExpirationDays, and an optional TransitionDays + StorageClass. The engine walks the bucket\u0026rsquo;s objects, finds the first enabled rule whose prefix matches, and applies one action:\n1 2 3 4 5 6 switch { case rule.ExpirationDays \u0026gt; 0 \u0026amp;\u0026amp; ageDays \u0026gt;= rule.ExpirationDays: // expire: delete marker if versioned, permanent delete otherwise case rule.TransitionDays \u0026gt; 0 \u0026amp;\u0026amp; ageDays \u0026gt;= rule.TransitionDays \u0026amp;\u0026amp; rule.StorageClass != \u0026#34;\u0026#34;: // transition: update x-amz-storage-class in the object\u0026#39;s S3 metadata } Objects whose modification time is unknown are skipped rather than guessed at — a wrong age would delete the wrong thing.\nThe clock lives in the server. A background worker ticks on [s3] lifecycle_worker_interval and calls EvaluateLifecycle(bucket, now):\n1 2 3 case now := \u0026lt;-ticker.C: out, err := engine.EvaluateLifecycle(bucket, now) // log expired/transitioned counts Keeping the clock in the server means the storage engine stays deterministic and trivially testable — the tests pass an arbitrary now and assert the outcome, with no sleeping and no timers. The worker itself is panic-safe (Rule 43) so a policy bug can never take down the daemon.\nPrinciple: separate what should happen (a pure policy evaluation, given a clock) from when to check (a scheduler). The deterministic half is where the correctness lives; the scheduler should be dumb enough to have no bugs.\nExpiration meets versioning The most interesting interaction is with versioning, shipped in the previous phase. On an unversioned bucket, expiration is a plain delete. On a versioned bucket, the engine writes a delete marker first:\n1 2 3 4 5 6 7 8 func (s *CASStore) expireObject(bucket, name string, versioned bool) error { if versioned { if _, err := s.RecordDeleteMarker(bucket, name); err != nil { return err } } return s.Delete(name) } The delete marker becomes the new \u0026ldquo;latest\u0026rdquo; version, so a normal GET returns 404 — the object looks expired. But the previous version survives and is still reachable by versionId, which is exactly what S3 promises and what point-in-time recovery tools depend on. The two features compose without either knowing the other\u0026rsquo;s internals: versioning provides RecordDeleteMarker, and lifecycle just calls it.\nFailure Modes Lazy expiration. Checking age on read means unread objects never expire. A background sweep is the only way \u0026ldquo;delete after 30 days\u0026rdquo; is actually true. Expiring bytes on a versioned bucket. Deleting content instead of writing a marker destroys version history. The versioned branch must be taken. Racing an in-flight write. The engine runs under the store\u0026rsquo;s lock discipline, so a concurrent re-upload cannot be clobbered mid-expiry. Unknown modification time. Treating a missing timestamp as \u0026ldquo;very old\u0026rdquo; would delete fresh data. Unknown age is skipped. When NOT to Use This A background sweep is the right shape when the policy is time-based and the object set is bounded enough to walk periodically. It is a poor fit for per-object TTLs at massive scale (a full scan becomes expensive — that wants an indexed time wheel or a priority queue), or for sub-second expiry precision. For momo\u0026rsquo;s single-bucket gateway and day-granularity rules, a periodic reconciliation pass is the simplest thing that is correct.\nReferences / Dig deeper The standard: docs/CORE/STANDARDS.md Related posts: 056 S3 versioning on a CAS store, 004 content-addressed storage, 031 core integrity verification. Tracking issue: #938. ","permalink":"/posts/058-s3-lifecycle-on-cas/","summary":"Lifecycle rules are the first feature that makes momo delete data on a schedule rather than on request — which turns age, storage class, and delete markers into correctness problems. This post walks through the expiration and transition engine, why a background worker is the only safe place to run it, and how expiration composes with versioning\u0026rsquo;s delete markers.","title":"S3 Lifecycle Rules on a Content-Addressed Store: Time as a First-Class Dimension"},{"content":"Every request to the S3-compatible gateway that carries a signature goes through AWS Signature Version 4 verification. Part of that dance is building the canonical request — a normalized string that both the client and the server must produce identically, byte for byte, or the signature will not match. One of its components is the canonical headers block: a sorted, newline-separated list of name:value pairs for the headers the client chose to sign.\nThis post is about removing two heap allocations from that construction — and, more importantly, about how to prove an optimization on authentication code is safe.\nThe Real Problem: allocations on the authorization path The canonical-headers builder did two innocuous-looking things:\nIt split the SignedHeaders list (host;x-amz-content-sha256;x-amz-date) with strings.Split. For each name, it fetched the value with req.Header.Get(name). Both allocate. strings.Split allocates a slice header plus a backing array to hold the parts. And Header.Get is not a plain map lookup: it first canonicalizes the key through net/textproto.CanonicalMIMEHeaderKey, which allocates a copy of the name string on every call.\nThis runs on every authenticated request — the busiest authorization path in the gateway. Two allocations per request is not catastrophic, but it is pure, avoidable GC pressure exactly where latency matters, and it is the kind of thing the project\u0026rsquo;s performance standard forbids on hot paths.\nWhy the Obvious Fix Is Not Enough The tempting move is \u0026ldquo;just cache it\u0026rdquo; or \u0026ldquo;only optimize if a benchmark says so\u0026rdquo;. Neither is right here.\nCaching a per-request canonical string buys nothing — each request has different headers, so there is no reuse to exploit. And optimizing blind is dangerous: this function sits on the authentication path, so a subtle behavior change does not show up as a slow request, it shows up as rejected valid signatures or, worse, accepted invalid ones. Performance work on auth code must be provably behavior-preserving, not just faster.\nThe Solution: two allocation-free primitives Replace each allocating call with a scan over the data you already have.\nParsing the signed-header list — walk the string with strings.IndexByte, slicing the original rather than building new strings:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 func parseSignedHeaders(s string, buf []string) ([]string, error) { headerNames := buf[:0] // reuse a caller-provided stack buffer for { idx := strings.IndexByte(s, \u0026#39;;\u0026#39;) if idx == -1 { if len(s) \u0026gt; 0 { if len(headerNames) \u0026gt;= cap(buf) { return nil, syscall.EBADMSG // bounded: fail closed } headerNames = append(headerNames, s) } break } if part := s[:idx]; len(part) \u0026gt; 0 { if len(headerNames) \u0026gt;= cap(buf) { return nil, syscall.EBADMSG } headerNames = append(headerNames, part) } s = s[idx+1:] } return headerNames, nil } The caller passes a stack-allocated [16]string buffer, so the common case never touches the heap. The capacity check is a defensive bound: a malicious client cannot force an unbounded list.\nLooking up a header value — iterate the header map with a case-insensitive comparison instead of asking Header.Get to canonicalize the key:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 func getCanonicalHeaderValue(req *http.Request, h string) (string, bool) { if h == \u0026#34;host\u0026#34; { return req.Host, true // Go keeps Host out of the header map } for k, v := range req.Header { if strings.EqualFold(k, h) { if len(v) \u0026gt; 0 { return v[0], true } return \u0026#34;\u0026#34;, true } } return \u0026#34;\u0026#34;, false } EqualFold compares without allocating, and the first-value semantics match what Header.Get returned.\nPrinciple: on a hot path, prefer scanning the bytes you already own over calling a helper that builds a new string. IndexByte and EqualFold read; Split and CanonicalMIMEHeaderKey write.\nVerification: the golden test A faster function that produces a different string is a security bug, not an optimization. So the rewrite ships with a golden-equivalence oracle: a verbatim copy of the old implementation kept in the test file, and a table of conformant inputs asserting the new implementation\u0026rsquo;s output is byte-identical.\n1 2 3 4 5 want := legacyBuildCanonicalHeaders(req, tc.signedHeaders) got, err := buildCanonicalHeaders(req, tc.signedHeaders) if got != want { t.Fatalf(\u0026#34;canonical output diverged:\\n legacy: %q\\n optimized: %q\u0026#34;, want, got) } The oracle immediately caught a real divergence. For the input SignedHeaders: \u0026quot;Host;X-Amz-Date\u0026quot; (note the capital H), the old code emitted Host:\\n — an empty host value, because its host special-case only matched the exact lowercase \u0026quot;host\u0026quot;. That is a broken canonical request. The new code fails closed with an error instead. Since AWS mandates lowercase signed-header names, that input never occurs in practice — but the test now documents the difference explicitly rather than hiding it.\nTwo intentional, fail-closed divergences are pinned by dedicated tests:\na signed header absent from the request now returns an error instead of silently contributing an empty value; empty segments in a malformed list (host;;x-amz-date) are skipped instead of emitting a stray :\\n. Failure Modes Over-optimizing the auth path. If the golden test had been skipped, the capital-H divergence could have shipped unnoticed. On auth code, equivalence proof is not optional. Stack-buffer overflow. A fixed [16]string buffer means an oversized list must be rejected, not truncated — hence the explicit capacity check returning an error. Assuming Host lives in the header map. Go promotes the request\u0026rsquo;s host into a separate field; the lookup has to special-case it, exactly as before. When NOT to Use This This pattern pays off on per-request hot paths with bounded input sizes. It is not worth it for code that runs once at startup, for lists whose size is genuinely unbounded (you would need a heap buffer anyway), or where the input is already a slice and Split would not allocate a new backing array. Reach for IndexByte/EqualFold when the data is already a string you can slice and the call count is high.\nReferences / Dig deeper The performance standard: docs/CORE/STANDARDS.md Related Bolt posts: 024 the performance mindset, 049 SigV4 datestamp formatting, 051 aws-chunked parsing, 052 ETag parsing. Tracking issue: #1195. ","permalink":"/posts/057-bolt-sigv4-zero-alloc-headers/","summary":"Every authenticated S3 request rebuilds a SigV4 canonical-headers block — and the two obvious ways to write it (strings.Split and Header.Get) each allocate on the hottest authorization path. This post walks through replacing both with an IndexByte scan and an EqualFold lookup, and the golden-equivalence test that proves the rewrite is byte-identical for every request AWS clients actually send.","title":"⚡ Bolt: Zero-Allocation SigV4 Canonical Headers"},{"content":"The Real Problem Backup tools — Veeam, restic, rclone — do not just upload files. They upload the same file many times and rely on S3 versioning to keep every upload addressable: \u0026ldquo;give me the object as it existed on Tuesday.\u0026rdquo; Delete a file and the tool expects a delete marker, not a missing object — so it can restore Tuesday even though you deleted Wednesday.\nmomo\u0026rsquo;s storage layer is content-addressed (CAS): a blob\u0026rsquo;s identity is its SHA-256 hash. Put the same file twice, get one physical blob. This is the property that makes deduplication free and integrity verification trivial.\nAt first glance the two models fight each other:\nCAS (momo\u0026rsquo;s core) S3 versioning (what clients expect) Identity of \u0026ldquo;the same file put twice\u0026rdquo; One blob — same hash Two versions — two version ids Delete Remove the mapping Create a delete marker — nothing is removed If we derived version ids from content hashes, two uploads of identical content would collapse into one version — and point-in-time recovery would silently lose \u0026ldquo;Tuesday\u0026rdquo; whenever Tuesday\u0026rsquo;s file was unchanged. That is exactly the bug a backup customer cannot see until the day they need the restore.\nThe Design: Version Records Are Pointers, Not Copies The resolution is to keep the two identity systems in separate layers and never let either borrow from the other:\nContent identity stays CAS. The blob layer does not change at all. A version record only points at a hash. Mutation identity is a new metadata layer. Every PUT to a versioned bucket appends a VersionRecord: an opaque UUIDv4 id, the content hash, the size, a timestamp, and an IsLatest flag. Because records only point at hashes, the space cost of uploading the same file a hundred times is zero — one physical blob, one hundred version records of a few dozen bytes. CAS dedup is not compromised; it is composed with versioning.\n1 2 3 4 5 6 7 8 type VersionRecord struct { VersionID string // opaque UUIDv4 — never derived from content Hash string // CAS blob reference (\u0026#34;\u0026#34; for delete markers) Size int64 Timestamp int64 // unix nano — the ordering key IsLatest bool // exactly one record per key IsDeleteMarker bool // shadows the object without deleting anything } Why UUIDv4 and not the hash? Because the version id must distinguish events (two uploads) while the hash distinguishes bytes. Conflating them is the design mistake this whole layer exists to avoid.\nDelete Markers: Shadowing, Not Deleting In S3, DELETE on a versioned bucket does not delete anything. It appends a record whose only job is to say \u0026ldquo;newer than everything below me, and this key is gone.\u0026rdquo; Plain reads see the marker and return 404 — but the underlying data versions are untouched.\nThe revival rule is the subtle part: delete the marker itself (by version id) and the object comes back — the previous data version becomes latest again. momo implements this transactionally in one bbolt update: remove the record, re-point IsLatest to the next-newest record, and re-point the flat namespace mapping so even non-version-aware reads follow the revival.\n1 2 3 4 5 6 7 8 if deleted.IsLatest { // Next-newest wins — even if it is another marker. next, ok, _ := latestVersionTx(tx, bucket, key) if ok \u0026amp;\u0026amp; !next.IsLatest { next.IsLatest = true b.Put(versionKey(bucket, key, next.VersionID), marshal(next)) } } A versioned delete of the latest data version works the same way: the marker that shadows it (if any) becomes latest again. Every path funnels through one primitive — \u0026ldquo;re-point IsLatest\u0026rdquo; — instead of each caller implementing revival differently.\nPagination You Can Only Compute After Sorting S3 lists versions in key order ascending, and within a key, newest first. momo\u0026rsquo;s version ids are random UUIDs, which means the on-disk byte order of version records carries no relationship to that listing order. The first implementation tried to skip ahead with a cursor seek to the marker key — and lost records, because a seek lands in UUID byte order, which is the wrong order for everything the marker key still holds.\nThe fix is to stop seeking and just sort:\nScan all version records of the bucket (they are tiny metadata rows). Sort: key ascending, then timestamp descending, then version id descending as the tie-break. Find the (keyMarker, versionIdMarker) pair in the sorted order and resume after it. 1 2 3 4 5 6 7 8 9 sort.SliceStable(entries, func(i, j int) bool { if entries[i].key != entries[j].key { return entries[i].key \u0026lt; entries[j].key } if entries[i].rec.Timestamp != entries[j].rec.Timestamp { return entries[i].rec.Timestamp \u0026gt; entries[j].rec.Timestamp // newest first } return entries[i].rec.VersionID \u0026gt; entries[j].rec.VersionID // stable tie-break }) The lesson generalizes: pagination markers must be computed in the same ordering the page will be served in. A marker saved from one ordering and applied in another silently drops or duplicates rows — the kind of bug that only shows up when a customer pages through more than one screen of history.\nWhere It Deliberately Stops (The Ceiling) The version chain is node-local in this phase. Every S3 write records its version on the node that ingested it; replication of the version chain across nodes through the R6 metadata RPC is the tracked follow-up. The ceiling is real: if you lose the ingest node, you lose the version history (the blobs themselves are already replicated).\nLikewise, plain PUT responses do not echo x-amz-version-id yet — the gateway\u0026rsquo;s write response is composed before the body streams, so the header is not reachable on that path. Multipart completions (which are fully gateway-handled) do echo it, and any client can discover version ids via GET ?versions.\nBoth ceilings are written into the spec\u0026rsquo;s task list, not hidden — a shortcut without a recorded upgrade path is just a future outage. The defensive patterns used throughout — bounded recovery on every public method, length-limited error bodies — are the repo\u0026rsquo;s standing rules from docs/CORE/STANDARDS.md.\nWhen NOT to Use This If your storage layer is not content-addressed, do not build versioning this way — a plain version table keyed by object name is simpler and has no dedup interplay to reason about. The layered-identity design earns its complexity only when the content layer already owns an identity system that versioning must not disturb. And if your clients never use delete markers or point-in-time reads, S3-style versioning is ceremony: object locking and lifecycle rules build on it, but a store without those requirements can stay single-version.\nReferences / Dig deeper Spec: openspec/changes/r10-s3-breadth/ (R10-V1 — versioning requirements and scenarios) Tracking issue: alsotoes/momo#938 (R10 breadth; phases 2–4 = lifecycle, notification, object lock) Storage implementation: src/storage/versioning.go — version records, markers, pagination S3 surface: src/transport/s3_versioning.go — versioning config, ListVersionsResult XML, versioned deletes The CAS core the records point into: post 055 (adaptive volume storage), post 031 (core integrity verification) S3 versioning semantics: the AWS S3 API VersioningConfiguration, ListVersionsResult, and delete-marker behavior this implementation mirrors ","permalink":"/posts/056-s3-versioning-on-cas/","summary":"Why S3 versioning and content-addressed storage seem to conflict — mutation identity vs content identity — and how momo keeps both: UUIDv4 version records pointing at deduplicated blobs, delete markers that shadow without deleting, and pagination that can only be computed after sorting. Teaches: the real semantics backup tools depend on, and where the version chain deliberately stops being distributed.","title":"S3 Versioning on a Content-Addressed Store: Two Kinds of Identity"},{"content":"HTTP lets a client ask a server a precise question: \u0026ldquo;only send me this object if it has changed.\u0026rdquo; The client answers with the last entity tag (ETag) it saw, and the server compares it against the current one. Done well, this turns a full object transfer into a few bytes. Done carelessly, the comparison itself allocates memory on every request — which is what this change fixed.\nThe Problem: An Allocation on Every Conditional Request An ETag is an opaque identifier for a specific version of an object, usually a hash. A client sends it back in one of two headers:\nIf-None-Match — \u0026ldquo;send the body only if its tag is not in this list\u0026rdquo; (the basis of 304 Not Modified and caching). If-Match — \u0026ldquo;apply this write only if the tag is in this list\u0026rdquo; (the basis of safe conditional writes). The value is not always one tag. HTTP allows a comma-separated list:\n1 If-None-Match: \u0026#34;aaa111\u0026#34;, \u0026#34;bbb222\u0026#34;, W/\u0026#34;ccc333\u0026#34; A tag may be quoted, and a W/ prefix marks a weak validator — one that matches when the content is semantically equivalent, not byte-identical. There is also the wildcard *, meaning \u0026ldquo;any version.\u0026rdquo; So the server has to walk the list, normalise each entry, and compare it against the object\u0026rsquo;s current hash.\nThe old implementation walked the list with strings.Split:\n1 2 3 4 5 6 7 8 for _, item := range strings.Split(header, \u0026#34;,\u0026#34;) { item = strings.TrimSpace(item) if item == \u0026#34;*\u0026#34; { return true } item = strings.TrimPrefix(item, \u0026#34;W/\u0026#34;) item = strings.TrimSpace(item) item = strings.Trim(item, `\u0026#34;`) if item == etag { return true } } strings.Split allocates a slice header and a backing array to hold the parts — even though the caller only ever looks at one entry at a time and then discards it. On a busy gateway serving cached reads, that is one heap allocation per conditional request, feeding the garbage collector for no reason: the bytes are already in memory as a single string.\nThe Fix: Scan and Slice, Don\u0026rsquo;t Copy Go strings are immutable and cheap to slice: s[i:j] produces a new string header that shares the same bytes — no copy, no heap allocation. strings.IndexByte finds a byte without allocating. Together they replace Split with a single pass:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 for len(header) \u0026gt; 0 { var item string if idx := strings.IndexByte(header, \u0026#39;,\u0026#39;); idx == -1 { item = header // last field header = \u0026#34;\u0026#34; } else { item = header[:idx] // slice, no copy header = header[idx+1:] } item = strings.TrimSpace(item) if item == \u0026#34;*\u0026#34; { return true } item = strings.TrimPrefix(item, \u0026#34;W/\u0026#34;) item = strings.TrimSpace(item) item = strings.Trim(item, `\u0026#34;`) if item == etag { return true } } return false The normalisation steps are untouched — the same TrimSpace, *, W/, and quote-stripping logic runs on each entry, in the same order. Only the way the entries are produced changed: from an allocated slice to a cursor over the original string.\nPrinciple — prefer slicing to splitting. When you only need offsets into a string, reach for IndexByte + slicing. Reserve Split for cases where you genuinely need independent substrings, and treat every Split on a request path as a small tax the GC collects.\nVerifying It Two things must hold after a parser rewrite: nothing broke, and the cost fell.\nThe first is a table test. We converted the ETag-match test into a table covering the cases the old and new code must agree on: quoted and unquoted tags, weak tags (W/\u0026quot;…\u0026quot;, and W/ \u0026quot;…\u0026quot; with a space), the wildcard alone and inside a list, leading and trailing spaces, a trailing comma, a tag absent from the list, an empty header, and an empty object tag. Every case returns the same result as before.\nThe second is a microbenchmark — a benchmark that isolates one function. BenchmarkEtagMatches reports allocations per operation, with BenchmarkEtagMatchesSplit kept as the \u0026ldquo;before\u0026rdquo; baseline:\nns/op B/op allocs/op strings.Split (before) ~257–296 80 1 IndexByte + slice (after) ~127–181 0 0 The allocation count is the number that matters. Driving it to zero means the GC never sees these calls — regardless of how many conditional requests arrive.\nFailure Modes Risk Guard A subtle parse difference changes matching Table test asserts old and new agree on every input A weak or wildcard tag stops matching Dedicated cases for W/ and * (alone and in a list) A sliced entry aliases and outlives the header Entries are consumed immediately; the header string is request-scoped The win is only a microbenchmark artifact The split baseline runs the same header, so the delta is the allocation, not the input When NOT to Use This When the parsed pieces must outlive the source or be modified. A slice aliases the parent string and keeps it alive; Split returns owned copies. If an entry is stored beyond the request, or written to, copy it. Here each entry is normalised and compared in place, so slicing is safe. And as always: profile first. This pattern earns its place only after a measurement points at the parser.\nReferences / Dig deeper The standard: docs/CORE/STANDARDS.md Related Bolt posts: 024 performance mindset, 048 the last XML allocation, 051 streaming-header parsing. Tracking issue: #1133. ","permalink":"/posts/052-bolt-etag-matches-zero-alloc/","summary":"How we removed the heap allocation from ETag list parsing on the S3 conditional-request path by replacing strings.Split with a strings.IndexByte scan. Teaches: what If-Match/If-None-Match actually carry, why Split allocates per request, and how a microbenchmark proves the win.","title":"⚡ Bolt: Zero-Allocation Parsing of ETag Lists"},{"content":"Static configurations in distributed systems are ticking time bombs. In traditional client-server designs, constants like fixed concurrency pools and single-node fail-fast routing look harmless during benchmarks on uniform development machines. Yet when deployed into real-world networks with heterogeneous hardware, fluctuating latencies, and resource-constrained edge devices, those rigid numbers quickly become points of failure or artificial bottlenecks.\nMomo\u0026rsquo;s biological vision establishes that the system must self-tune without human intervention. This post explains the first concrete implementation of those biological foundations: ant-colony pheromone routing for resilient client downloads and epigenetic host sensing for dynamic server concurrency.\nThe Real Problem: Fragility Under Uniform Assumptions Prior to this work, Momo had two critical rigidities:\nFirst, client downloads targeted a single daemon identifier. When an object was replicated across multiple storage nodes via CRUSH placement, the client attempted to retrieve the data from only one designated server. If that server was congested, rebooting, or partitioned, the download immediately failed, even though identical, perfectly healthy copies existed on sibling replicas within the failure domain.\nSecond, peer selection and routing suffered from deterministic hot-spotting. When picking the best node, sorting peers strictly by round-trip latency channeled all traffic to a single node until it collapsed. Concurrently, new nodes or recovering replicas had zero traffic routed to them, starving them of the chance to prove their health.\nThird, server connection admission relied on a hardcoded constant: a semaphore capped at exactly 1,000 concurrent connections. On a constrained edge device or container with only 256 MB of RAM, 1,000 concurrent streaming uploads would trigger the kernel out-of-memory killer. On a high-throughput 64-core storage server with 256 GB of RAM, 1,000 connections left 90% of the hardware idle.\nBoth mechanisms violated the core engineering principles documented in docs/CORE/STANDARDS.md: systems must maintain predictable performance under stress and gracefully absorb perturbations.\nAnt Colony Routing: Balancing Exploitation and Exploration In nature, an ant searching for food deposits a chemical marker called a pheromone along its path. Shorter, faster trails accumulate pheromone more quickly because ants traverse them more frequently. At the same time, pheromones evaporate continuously over time. Ants choose paths probabilistically based on pheromone concentration. This natural mechanism creates a powerful self-organizing feedback loop: fast paths attract the majority of traffic, but weaker paths are still explored, preventing lock-in and quickly discovering alternate routes when an obstacle appears.\nMomo translates this biological process directly into replica routing:\nPheromone Trails: Each replica node begins with a neutral baseline pheromone weight of 1.0. When a download succeeds, the client reinforces that node\u0026rsquo;s trail inversely proportional to latency: faster responses yield stronger reinforcements. Failure Penalties: If a node times out, drops connections, or encounters network faults, its pheromone trail is sharply docked by an immediate evaporation penalty, driving traffic away from the failing node. Continuous Evaporation: An exponential decay interval slowly reduces pheromone levels back toward baseline, allowing previously degraded nodes to recover and re-enter the candidate rotation. Roulette-Wheel Selection: Rather than always picking the single highest-rated node, candidate replicas are sampled with probability proportional to their pheromone weight. The fastest node receives the majority of downloads, but sibling replicas are sampled periodically. If the primary replica fails mid-flight, the client automatically falls back to sibling candidates in priority order. Crucially, this selection fast-path complies with Bolt performance standards: the entire weighted sampling path runs with zero heap allocations (0 B/op, 0 allocs/op) in under 200 nanoseconds.\nEpigenetics: Adapting Server Concurrency to the Host Biological epigenetics refers to gene expression changes that adapt an organism to its environment without altering its underlying DNA sequence. In software, the equivalent principle dictates that a single compiled binary must sense its operational environment on boot and configure its own operational boundaries.\nInstead of a static 1,000-connection limit, Momo introduces homeostatic admission control:\nHost Probing on Boot: During initialization, the daemon reads Linux cgroups memory limits (supporting both cgroups v1 and v2), falls back to system memory metrics, queries available CPU cores, and inspects operating system file descriptor limits (RLIMIT_NOFILE). Memory-Bounded Capacity: The server budgets concurrency based on streaming buffer allocations: assuming approximately 2 MB of working memory per active streaming connection, the server calculates a safe capacity ceiling that preserves a 20% safety margin for OS buffers and background housekeeping. Homeostatic Throttling: As host CPU utilization exceeds 85% or available system memory drops below safe thresholds, dynamic admission limits contract in real time, shedding load before the Linux kernel invokes the out-of-memory killer. When load subsides, the admission ceiling gracefully expands back to full capacity. Performance and Verification Microbenchmarks and unit tests verify that these adaptive layers introduce negligible overhead while preventing catastrophic failures:\nMetric Baseline (Static) Adaptive Foundations Impact Single-Node Failure Tolerance 0% (Immediate error) 100% (Transparent sibling fallback) Continuous availability Route Selection Allocations 0 allocs/op 0 allocs/op Zero GC impact Low-Memory Container Safety OOM risk under surge Self-throttled admission Zero host crashes By combining ant-colony probabilistic routing with epigenetic host awareness, Momo transforms static fragile code into resilient, living infrastructure that thrives across diverse hardware and fluctuating network conditions.\nReferences / Dig deeper Standards and Bolt philosophy: docs/CORE/STANDARDS.md Adaptive Systems Vision: see the Adaptive Systems architectural specification in the architecture documentation. Related blog posts: 018 Adaptive Scaling and Peer Quality 024 Bolt Performance Engineering Tracking Issue: #1129 OpenSpec Change Proposal: openspec/changes/adaptive-systems-foundations ","permalink":"/posts/052-adaptive-systems-foundations/","summary":"How Momo replaces static concurrency limits and rigid replica selection with bio-inspired adaptive mechanisms: Ant Colony pheromone-based replica exploration and Epigenetic host-capacity admission.","title":"🐜 Adaptive Systems: Ant-Colony Routing and Epigenetic Concurrency"},{"content":"Storing a hundred thousand small objects as individual files is deceptively expensive. Each file costs an inode, a directory entry, and at least one disk block regardless of payload size. Under a content-addressed store that keeps every version of every object, those costs compound until directory lookups — not disk bandwidth — dominate every read. This post explains the volume superblock layout momo adopted for small objects and why each design choice earns its bytes.\nThe Real Problem A content-addressed store names files by their hash: blobs/ab/cd/ef0123…. That layout spreads entries across a directory tree, which keeps directories short — but the per-file floor remains. On ext4 a 1 KiB object consumes 4 KiB of blocks plus an inode plus directory metadata: an overhead of 5× or more. Worse, a cluster writing many small objects pays it in system calls: one open, one write, one fsync, one rename, one close per object.\nThe workload that exposes this is telemetry-like: many small, immutable objects, written once, read many times, deleted when expired. The goal is a layout where reading any given object costs one seek, and where deletion actually returns space to the filesystem.\nWhy the Obvious Solutions Failed More directory sharding. Splitting the hash across more levels shortens each directory but multiplies the number of directories and does nothing about the per-file block/inode floor.\nOne big log file, compacted later. Appending everything to a single file solves the per-file overhead but makes individual deletion impossible without rewriting the file, and makes crash recovery a full-file scan.\nKeeping an index in the metadata DB and hoping the OS page cache hides the rest. The index is necessary either way; it does nothing for the four extra system calls per write or the space amplification per read.\nThe Solution with Annotated Code An 8-byte-aligned needle Each object is a needle inside an append-only volume file: a fixed header, the payload padded to 8 bytes, and a CRC32C trailer.\n1 2 | magic(4) | len(4) | sha256(32) | flags(2) | payload | pad | crc32c(4) | | | | | length padded up to 8 | The padding is what buys O(1) reads: with every needle\u0026rsquo;s size known at the offset, reading object N is a single pread(offset, headerLen) — no scan, no variable-length parsing to discover where the next needle starts.\n1 2 3 // Alignment keeps every needle a whole number of 8-byte units, so the next // needle\u0026#39;s offset is always computable from the current one. func alignTo8(n int) int { return (n + NeedleAlignment - 1) \u0026amp;^ (NeedleAlignment - 1) } Both integrity checks ride along in the frame: the SHA-256 in the header confirms the payload is the object it claims to be, and the CRC32C trailer detects torn or corrupted frames. A needle that fails either check fails the read — closed, never returned to the caller.\nHole-punching deletion Deleting a needle punches a hole in the file, returning its blocks to the filesystem without rewriting anything:\n1 2 3 unix.Fallocate(int(f.Fd()), unix.FALLOC_FL_PUNCH_HOLE|unix.FALLOC_FL_KEEP_SIZE, offset, int64(NeedleOverhead)+int64(alignTo8(size))) On filesystems without hole-punch support the needle is tombstoned instead, and the read path treats a tombstone as a miss.\nCrash recovery by truncation The write path commits a needle to the metadata index only after its bytes are synced. On startup the active volume is truncated to the highest committed offset, discarding any partial needle a crash left in the tail. Recovery cost is one stat call, not a scan.\nPrinciple: Recover from a crash by trusting committed state and cutting off everything after it. Scanning to discover what survived is what makes recovery slow; an explicit commit point makes it instant.\nVerification Frame round-trip writes a needle and reads it back byte-for-byte. Volume rolling fills a superblock and proves the store continues into a fresh one without losing any object. Hole-punching deletes a blob and checks the file\u0026rsquo;s block count via fstat drops. Crash recovery kills the store mid-life and reopens it, asserting every committed object is intact. Routing proves the adaptive seam sends small objects to the volume store and large ones to the per-blob fallback. On the reference machine, a 64 KiB needle reads at ~1.7 GB/s single-threaded — the O(1) seek the layout exists for.\nFailure Modes A torn final needle is discarded by truncation; a torn middle needle is caught by the CRC/SHA checks and surfaces as a read error, where the self-heal layer can repair it from a replica. Filesystems without hole-punch support fall back to tombstones, which reclaim no space until the volume is rewritten. Vacuum is a separate, deferred task. Very large objects never belong in the layout; the adaptive seam routes them to per-blob storage so a single object cannot dominate a superblock. When NOT to Use If objects are large and few, a volume layout adds indirection for nothing — the per-blob local layout is simpler and just as fast. The layout pays off specifically where objects are small and numerous relative to the block size.\nReferences / Dig deeper Adaptive volume storage specification: openspec/changes/adaptive-volume-storage Needle framing: src/storage/volume_needle.go Volume store, deletion, crash recovery: src/storage/volume_blobstore.go Adaptive seam: src/storage/adaptive_blobstore.go Tracking issue: https://github.com/alsotoes/momo/issues/1128 ","permalink":"/posts/055-adaptive-volume-storage/","summary":"Why millions of small files per directory collapse a filesystem, and how a SeaweedFS-style volume layout fixes it: append-only superblocks, 8-byte-aligned needles with CRC32C frames, hole-punching deletion, and crash recovery by truncation. Teaches: the real cost of per-blob files, why alignment beats variable-length framing, and how to make crash recovery O(1).","title":"Packing Small Objects into Superblocks: O(1) Reads Without Per-Blob Inodes"},{"content":"An allocation is memory claimed from the heap, the pool the garbage collector manages. Every allocation on a hot path — code that runs on nearly every request — adds GC pressure and latency. We had been removing these one at a time from the S3 gateway; this change removed two more, from a place that runs many times per object: the parser for streaming upload headers.\nThe Problem: One Allocation Per Chunk An S3 client can upload a body as a streaming, signed payload. Instead of sending the whole object and a signature up front, it sends the object in chunks — typically 64 KiB each — so the server can verify data as it arrives. Each chunk is preceded by a small text header line:\n1 10000;chunk-signature=0123abcd... The first field is the chunk size in hexadecimal; the rest are semicolon- separated fields, of which chunk-signature= carries the per-chunk signature.\nParsing that line is not a one-time cost. A 100 MB object split into 64 KiB chunks arrives as roughly 1,600 chunk headers, each parsed once on the ingest path. The old implementation parsed each line with:\n1 2 3 parts := strings.Split(line, \u0026#34;;\u0026#34;) // allocates a slice + backing array ... sig = strings.TrimPrefix(part, \u0026#34;chunk-signature=\u0026#34;) // allocates a new string strings.Split allocates a slice header and a backing array to hold the parts, even when the caller only needs the first two fields. strings.TrimPrefix allocates again whenever it matches. Two heap allocations per chunk, multiplied by the chunk count, is pure waste — the data is already in memory as a string, and all we need are slices of it.\nThe Fix: Scan and Slice, Don\u0026rsquo;t Copy Go strings are immutable and cheap to slice: s[i:j] produces a new string header that shares the same underlying bytes, with no copy and no heap allocation. strings.IndexByte finds a byte without allocating. Together they replace Split and TrimPrefix with a single pass and zero allocations:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 var sizeStr, rest string if idx := strings.IndexByte(line, \u0026#39;;\u0026#39;); idx != -1 { sizeStr = line[:idx] // slice, no copy rest = line[idx+1:] } else { sizeStr = line } size, err := strconv.ParseInt(strings.TrimSpace(sizeStr), 16, 64) // ... sig := \u0026#34;\u0026#34; for len(rest) \u0026gt; 0 { var part string if idx := strings.IndexByte(rest, \u0026#39;;\u0026#39;); idx != -1 { part = rest[:idx] rest = rest[idx+1:] } else { part = rest rest = \u0026#34;\u0026#34; } if strings.HasPrefix(part, awsChunkSigField) { sig = part[len(awsChunkSigField):] // slice off the prefix, no copy // ... length bound unchanged } } The loop walks the same fields in the same order as the old range over parts[1:], so the semantics are unchanged — including the subtle rule that if a line somehow carries two chunk-signature= fields, the last one wins.\nPrinciple — prefer slicing to splitting. When you only need offsets into a string, use IndexByte + slicing. Reserve Split (and TrimPrefix) for the cases where you genuinely need independent substrings, and remember they never land on a hot path by accident.\nVerifying It A refactor of a parser must prove two things: nothing broke, and the cost went down. The first is a table test. We extended TestParseAWSChunkHeader with the edge cases the old code and the new code must agree on: a line with no ;, a trailing ;, an empty field (;;), extension fields before and after the signature, whitespace around the size, multiple signatures, an empty signature, an empty size, a non-hex size, and a negative size. Both implementations return the same result for every one.\nThe second is a microbenchmark — a small benchmark that isolates one function. BenchmarkParseAWSChunkHeader reports allocations per operation:\nns/op B/op allocs/op strings.Split + TrimPrefix (before) ~280–520 32 1 IndexByte + slice (after) ~58–69 0 0 The allocation count is the number that matters. Driving it to zero means the GC never sees these calls at all, no matter how many chunks an upload contains.\nFailure Modes Risk Guard A subtle parse difference changes behavior Edge-case table test asserts old and new agree on every input Signature length bound is lost in the rewrite The 64-byte limit is preserved and covered A path inside a rebuilt string leaks out Fields are slices of the original line, bounded by the existing header-line limit The win is only a microbenchmark artifact The de-framing benchmarks (BenchmarkAWSChunkedReaderSigned/Unsigned) exercise the full path When NOT to Use This When you need real substrings. Split returns owned, independent strings; a slice aliases the parent and keeps it alive. If the parsed pieces must outlive the source line — or be modified — copy them. Here the fields are consumed immediately and the source line is bounded to 1 KiB, so slicing is both safe and correct. As always, measure first: reach for this pattern only after a profile points at the parser.\nReferences / Dig deeper The standard: docs/CORE/STANDARDS.md Related Bolt posts: 024 performance mindset, 045 header formatting, 047 XML timestamps, 048 the last XML allocation, 049 SigV4 datestamps. The aws-chunked framing spec: 040 and 039 in the S3 series. Tracking issue: #1113. ","permalink":"/posts/051-bolt-aws-chunked-zero-alloc/","summary":"How we removed the last heap allocation from the aws-chunked ingest path by replacing strings.Split and strings.TrimPrefix with a strings.IndexByte scan. Teaches: what a streaming S3 upload body looks like, why Split allocates per chunk, and how a microbenchmark proves the win.","title":"⚡ Bolt: Zero-Allocation Parsing of AWS Chunked Headers"},{"content":"Signing an AWS request is not a place you expect to find waste. The signing code is small, runs on every request, and looks harmless. But during a performance audit we noticed it was formatting the same instant twice — and paying for two heap allocations where one would do.\nTo talk about the fix, a little context. AWS SigV4 is the signing scheme that authenticates requests to AWS-compatible services. It needs two pieces of temporal data: a full timestamp used in the X-Amz-Date header, formatted as 20060102T150405Z, and a shorter datestamp used in the credential scope, formatted as 20060102. Both come from the same time.Time.\nAn allocation is memory claimed from the heap, the pool the garbage collector manages; every allocation adds GC pressure on a hot path. Formatting a time in Go produces a string, and producing that string means allocating a new slice on the heap.\nWhy the Obvious Approach Is Slow The original code called time.Format twice:\n1 2 3 4 // First attempt: format twice now := time.Now().UTC() amzDate := now.Format(\u0026#34;20060102T150405Z\u0026#34;) dateStamp := now.Format(\u0026#34;20060102\u0026#34;) time.Format is not free: it walks the layout string and allocates a new string for the result. Calling it twice does double the work and allocates two strings per signed request when we only ever needed one distinct piece of data.\nThe Solution: Slice, Don\u0026rsquo;t Re-Format The full amzDate string already begins with exactly the YYYYMMDD sequence the datestamp needs, so we can take it as a prefix:\n1 2 3 4 5 6 now := time.Now().UTC() amzDate := now.Format(\u0026#34;20060102T150405Z\u0026#34;) // Derive the datestamp directly from amzDate to avoid a redundant // format pass and its string allocation. dateStamp := amzDate[:8] In Go, slicing a string (amzDate[:8]) is essentially free. It does not copy the bytes; it creates a new string header — a pointer and a length — that points into the same underlying memory as the original. One format call, one allocation.\nPattern: Deriving Sub-Strings by Slicing When a substring is a strict prefix or suffix of a string you just produced, slice it instead of regenerating or re-formatting the data.\nApplies when: you need multiple formatted views of the same value and one is a subset of the other — timestamps and datestamps being the classic case. Doesn\u0026rsquo;t apply: when the substring must outlive the original by a long margin, because the slice keeps the whole backing string alive. Here both strings are tiny and share the same short lifetime, so that risk does not arise.\nApplied across the S3 storage and transport layers, this removed one redundant heap allocation on every signed request, reducing GC pressure and CPU cycles for free.\nWhen NOT to Use This When the derived substring would pin a large original string in memory long after the rest of it is unneeded — slice then copy, or re-format. When the offset or length is not a fixed, obvious prefix/suffix; a magic index is a bug waiting to happen. When readability suffers more than the allocation is worth. A single format call on a cold path is fine. References / Dig deeper Mindset standards: docs/CORE/STANDARDS.md. Bolt engineering overview: 024. The change spec: openspec/changes/bolt-optimize-datestamp. The signing code lives in src/storage/s3_blobstore.go. ","permalink":"/posts/049-eliminate-redundant-time-format-allocations/","summary":"By replacing a second \u003ccode\u003etime.Format\u003c/code\u003e call with a simple string slice (\u003ccode\u003eamzDate[:8]\u003c/code\u003e), we eliminated redundant heap allocations and CPU overhead when generating timestamps for AWS SigV4 requests.","title":"Eliminating Redundant Time Formats: Slicing for Speed"},{"content":"For a while, every S3 protocol response we rendered carried a hidden cost: a fresh heap allocation just to format a timestamp. We had been removing them one by one — the CopyObjectResult body, then the HTTP Last-Modified header — and this change removed the last one in the XML path.\nAn allocation is memory claimed from the heap, the pool the garbage collector manages. Under sustained object-store traffic every allocation adds GC pressure and latency to a hot path, the code that runs on nearly every request. The S3 gateway builds its protocol responses as hand-written XML, so every timestamp rendered into that XML is a candidate for this fix.\nWhy time.Format Allocates time.Format returns a string. In Go, producing that string allocates a new slice on the heap — Format has no way to write into caller-owned memory, because the caller has not provided any.\nSo in the ListParts handler:\n1 tstr := time.Now().UTC().Format(time.RFC3339) every ListParts request paid one 24-byte heap allocation just to render the timestamp inside each \u0026lt;LastModified\u0026gt; element. Polling multipart-upload progress is a common repeated SDK and CLI pattern, so this allocation landed squarely on a hot path.\nThe Fix: AppendFormat Into a Stack Buffer time.AppendFormat is the zero-allocation sibling: it appends the formatted time into a caller-provided []byte and returns the grown slice. Give it a stack-allocated [32]byte array and there is no heap involvement at all:\n1 2 var timeBuf [32]byte tstr := time.Now().UTC().AppendFormat(timeBuf[:0], time.RFC3339) One subtlety: AppendFormat returns a []byte, not a string, so the XML write site switches from buf.WriteString(tstr) to buf.Write(tstr):\n1 2 3 buf.WriteString(`\u0026lt;LastModified\u0026gt;`) buf.Write(tstr) buf.WriteString(`\u0026lt;/LastModified\u0026gt;`) The emitted bytes are identical — same time.RFC3339 layout — so the AWS CLI and every SDK parse the response exactly as before. There is zero protocol change.\nProof, Not Claims Following the house rule that performance changes ship with evidence, we added a paired microbenchmark:\n1 2 BenchmarkListPartsTime_AppendFormat-8 10762976 120.7 ns/op 0 B/op 0 allocs/op BenchmarkListPartsTime_Format-8 6846415 163.3 ns/op 24 B/op 1 allocs/op 0 allocations per operation versus 1 (24 bytes), and roughly 26% faster formatting. The full 289-test transport suite passes under the race detector.\nThe Pattern The general rule we codified: scan loops that generate XML or JSON arrays for time.Now() or strconv.Itoa calls. Hoist constant strings out of the loop, and convert time.Format to time.AppendFormat with a stack buffer. This change applied that rule to the timestamp in the ListParts handler.\nPattern: Stack-Buffer Time Formatting When a hot path formats a timestamp, write it with time.AppendFormat into a stack-allocated byte array instead of calling time.Format.\nApplies when: high-frequency rendering — protocol responses, HTTP headers, log lines. Doesn\u0026rsquo;t apply: one-off or user-facing formatting, where the allocation is negligible.\nWhen NOT to Use This Low-frequency admin or startup paths. User-facing display dates, which are clearer as strings and not hot. Anywhere the surrounding API contract requires a string; the allocation is unavoidable at that boundary. References / Dig deeper Mindset standards: docs/CORE/STANDARDS.md — ⚡ Bolt / 🛡 Sentinel. Bolt engineering overview: 024. The prior XML and header changes: 047, 045. The change spec: openspec/changes/bolt-listparts-appendformat. The paired microbenchmarks: BenchmarkListPartsTime_AppendFormat vs BenchmarkListPartsTime_Format, in the transport test suite. ","permalink":"/posts/048-bolt-listparts-appendformat/","summary":"How we eliminated the final per-response heap allocation in S3 XML rendering by swapping time.Format for time.AppendFormat in the ListParts handler. Teaches: the stack-buffer time formatting pattern, why Format allocates, and how microbenchmarks prove the win.","title":"⚡ Bolt: Eliminating the Last Remaining S3 XML Time Allocation"},{"content":"The first version of our FUSE mount worked. Directories were content-addressed JSON manifests, files were content-addressed blobs, and a hand-rolled adapter reconciled the kernel\u0026rsquo;s byte-range writes with our write-whole-file model. It also re-implemented a surprising amount of FUSE wire protocol — mount negotiation, lookup reference counting, kernel coordination — that we had no business owning. The adapter was fine as a prototype and wrong as a foundation. This is the story of trading it for a library that already knows the protocol.\nThe Real Problem FUSE is the kernel bridge that lets a userspace program be a filesystem: the kernel forwards file operations to your process over /dev/fuse. The bridge has a precise wire protocol with sharp edges. Get mount negotiation subtly wrong and the mount hangs or dies at a bad moment. Mishandle lookup reference counting and the kernel forgets an inode your process still believes in, or never forgets one you released. These are not bugs you find in testing; they are bugs you find in production, intermittently.\nOur adapter had implemented all of this by hand against the raw protocol. It was correct enough to mount and pass round-trips, but every edge was ours to maintain, and the effort bought us nothing a mature library could not provide. Meanwhile the filesystem needed to grow: 22 node callbacks, each one a place to get the protocol details wrong.\nWhy the Obvious Solutions Failed Keep hardening the hand-rolled adapter. This is the sunk-cost answer, and it is sometimes right. It was wrong here because the protocol work was pure duplication — the same negotiation and refcounting every FUSE filesystem needs — so every hour spent on it was an hour not spent on storing bytes correctly.\nAdopt a low-level binding. There are FUSE libraries that expose the raw protocol as opaque byte operations: you get the messages, but you still own inodes, lookups, and error semantics yourself. That would have traded one hand-written protocol layer for another with fancier types, and left the hard bookkeeping intact.\nSwitch transports entirely (e.g. VirtioFS on every platform). On macOS, Docker Desktop already serves files through the kernel\u0026rsquo;s VirtioFS, and no momofs FUSE process is needed at all. But on bare-metal Linux we still need a userspace mount, so \u0026ldquo;use VirtioFS everywhere\u0026rdquo; is only half a plan. We kept both: VirtioFS where the platform provides it, and a real FUSE transport on Linux.\nThe Solution: Move to go-fuse/v2\u0026rsquo;s High-Level API hanwen/go-fuse/v2 is a mature Go FUSE implementation. Its fs package provides a high-level, inode-based API: you describe a tree of nodes and implement methods on them, and the library owns the protocol underneath — mount negotiation, kernel-managed node lifetimes, and translating returned errors into the codes the kernel expects. That is exactly the division of labor we wanted.\nIssue #980, with the ratified add-fuse-implementation spec, chose this path. Pull request #984 is what shipped.\nThe migration was mostly a re-expression of what we already had:\nThe node model was replaced. Our previous node and handle types, and their attribute structs, were rewritten as the go-fuse fs interfaces. The full prior callback set — lookup, getattr, setattr, mkdir, create, unlink, rmdir, rename, link, opendir, readdir, open, read, write, flush, release, statfs, forget — now maps one-to-one onto go-fuse methods. Crucially, the public entry points the CLI and end-to-end test call kept their signatures, so nothing above the adapter changed: the mount command and its tests were untouched. Unmount now tracks live servers by mount point, so cleanup is per-mount rather than global. Error mapping got explicit. go-fuse wants methods to return a system error code. The adapter preserves the precise code the core already chooses — \u0026ldquo;not found,\u0026rdquo; \u0026ldquo;is a directory,\u0026rdquo; \u0026ldquo;invalid argument\u0026rdquo; — and maps anything unknown to a generic I/O error. We also added a panic guard: a two-line recover converts a stray panic into an I/O error, so one bad operation cannot tear down the whole mount. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 // go-fuse wants a syscall.Errno back. Preserve the specific code the core picked // (so \u0026#34;no such file\u0026#34; stays ENOENT, not a vague failure), and fail closed to EIO // for anything unexpected. The deferred recover means a panic in a callback // returns EIO for that one call instead of killing the mount for every process. func toErrno(err error) syscall.Errno { if err == nil { return 0 } var errno syscall.Errno if errors.As(err, \u0026amp;errno) { return errno // ENOENT, EISDIR, EINVAL, ... — as the core intended } return syscall.EIO } A new [momofs] config section. The old consistency=cached setting is deprecated. It used to let the mount cache aggressively, but kernel-level consistency mechanisms (DAX, and VirtioFS on macOS) make that redundant — the kernel already guarantees what the flag was emulating. Setting it now logs an audit message and is ignored; any other value is rejected when the config loads, so a typo cannot silently change caching behavior. The dependency was swapped. The previous binding was removed from both the module and the vendored tree, and go-fuse/v2 was added as a direct dependency. The Read Path and splice There is a zero-copy read path in FUSE called splice, where the kernel copies data directly from a file descriptor into a socket without bouncing through userspace. It only helps when there is a real file descriptor to splice from. Our core is a hash-addressed blob reader: reads flow into a memory view, so there is no backing descriptor to hand the kernel. The natural return is therefore \u0026ldquo;here is the data,\u0026rdquo; not \u0026ldquo;here is the descriptor.\u0026rdquo; The zero-copy splice path is documented as a follow-up for blob backends that can expose raw descriptors — we chose the honest implementation over a nominal optimization.\nHow We Verified The full test target (vet, race detector, coverage) passed across all modules. The end-to-end mount test ran against a real /dev/fuse mount and passed: writes made natively surfaced inside the mount, and writes made through the mount surfaced natively. go vet and formatting checks were clean, the vendored tree stayed consistent, and the old binding was confirmed gone from the module. Remaining follow-ups are tracked on issue #980: a cross-platform fallback (the macOS VirtioFS guidance) and load measurements for the new path.\nImplemented vs. Planned (Honesty Note) The go-fuse/v2 migration and the consistency=cached deprecation are shipped. Still planned, per the spec\u0026rsquo;s task list: a cross-platform FUSE fallback, splice support for descriptor-backed blob stores, and the Phase-5 load and memory measurements. Treat those as open work, not delivered behavior.\n⚡ Bolt + 🛡 Sentinel Lens Per docs/CORE/STANDARDS.md, the transport follows two mindsets:\n⚡ Bolt: go-fuse owns the syscall-heavy protocol work, and the core stays protocol-agnostic, so no momofs wire format is locked to the transport — we can improve the byte path without renegotiating FUSE. 🛡 Sentinel: failing closed is explicit. Unknown errors become a generic I/O error rather than optimism, the panic guard isolates a bad callback, and a deprecated config value is rejected instead of silently reinterpreted. Failure Modes Risk Guard A panic in one callback kills the mount recover converts it to an I/O error for that call Unknown error leaks as success Unmapped errors fail closed to EIO consistency=cached silently changes behavior Deprecated: logged, ignored, other values rejected at load Inode never forgotten → leak go-fuse owns node lifetimes and forget handling Public entry points changed under callers Serve/Unmount signatures preserved; CLI e2e untouched When NOT to Use This When you need kernel-level zero-copy today. Our blob reader has no descriptor to splice from, so splice is not a win until a backing store exposes one. Non-Linux hosts. This transport targets Linux FUSE; macOS uses VirtioFS through Docker Desktop and needs no momofs FUSE process. References / Dig deeper Spec: openspec/changes/add-fuse-implementation. Issue #980; pull request #984. The core it mounts: 004: Content-Addressable Storage. Where the migration sits: 028: Roadmap and Research. The transport it replaced: 023: momofs FUSE Transport. ","permalink":"/posts/029-fuse-go-fuse-v2-migration/","summary":"The momofs FUSE mount traded its hand-rolled bazil.org/fuse adapter for the go-fuse/v2 high-level fs API, mapped all 22 node callbacks to go-fuse interfaces, and deprecated consistency=cached.","title":"FUSE Transport: Migrating to go-fuse/v2"},{"content":"One allocation per request is invisible until it is not. Every S3 GET, HEAD, range, and 304 Not Modified response carries a Last-Modified HTTP header: the timestamp of when the object was last written, which clients and caches use to decide whether their copy is stale. Rendering that header used to call Go\u0026rsquo;s time.Format, which returns a brand-new string on the heap every time.\nAn allocation is memory taken from the heap, the pool the garbage collector manages; a heap escape is when a value that could have lived on the stack is forced onto the heap because it outlives the call. On the hot path — the code that runs on essentially every request — that is one 32-byte allocation per response, all of it handed to the garbage collector. On an object store, reads are the hot path.\nThe fix is the same standard-library idiom we had already shipped for XML timestamp rendering: stop asking time.Format for a string, and write the formatted bytes directly into the response buffer instead.\nBefore / After 1 2 3 4 5 // before: heap string, then copied into the response b = append(b, formatHTTPLastModified(meta.ModTime)...) // after: zero-allocation append directly into the growing slice b = time.Unix(0, meta.ModTime).UTC().AppendFormat(b, http.TimeFormat) AppendFormat takes the destination slice as its first argument and returns the extended slice, so the formatted bytes land straight in the response buffer with no intermediate string. The header output is byte-for-byte identical — http.TimeFormat, the IMF-fixdate format the HTTP specification mandates — so AWS SDKs and the AWS CLI parse the response exactly as before. There is no protocol change to reason about.\nMeasured, Not Guessed The micro-benchmark isolates just the header rendering:\n1 2 BenchmarkLastModifiedHeader_AppendFormat 462.6 ns/op 0 B/op 0 allocs/op BenchmarkLastModifiedHeader_Format 451.3 ns/op 32 B/op 1 allocs/op 32 bytes and one allocation eliminated per response on the GET/HEAD/304 path — zero GC pressure from timestamp formatting, at no measurable throughput cost. The tiny ns/op difference is well inside run-to-run noise; the allocation count is the signal that matters.\n⚡↔🛡 Trade-off Posture Bolt optimizes inside hot paths, never at the expense of invariants. Here the invariant is byte-identical header output, verified by the existing S3 test suite — a pure allocation removal with no behavioral surface. The now-dead formatHTTPLastModified helper was deleted along with it, keeping the codebase to a single idiom for rendering HTTP times.\nWhen NOT to Use This One-off formatting in startup, shutdown, or admin paths, where the allocation is irrelevant. When you genuinely need a string return value for an API boundary — the allocation is unavoidable there, so converting is pointless churn. Human-readable display dates, which are not on a hot path and are clearer as a string. References / Dig deeper Mindset standards: docs/CORE/STANDARDS.md. Bolt engineering overview: 024. The same pattern applied to XML responses: 047 and 048. Earlier XML work: 036. Integrity and checksums: 012. The change spec: openspec/changes/bolt-http-lastmodified-appendformat; PR #976; issue #977. ","permalink":"/posts/045-bolt-lastmodified-header/","summary":"time.Format allocates a fresh string on every S3 GET/HEAD response. Swapping to time.AppendFormat drops that heap escape to zero — measured 32 B/op saved per request on the hottest read path.","title":"⚡ Bolt: Zero-Allocation Last-Modified Headers"},{"content":"A single 32-byte allocation per request sounds harmless until you multiply it by a hundred thousand requests a second. That is the story of this change: a timestamp that was quietly generating megabytes of garbage per second inside the S3 CopyObject response path.\nAn allocation is memory claimed from the heap, the shared pool the garbage collector manages. A hot path is code executed on nearly every request. Go\u0026rsquo;s profiler, pprof, is what let us see the allocation in the first place: it samples a running program and attributes time and memory to the functions that consume them.\nThe Real Problem During a sustained S3 load test — 100k requests per second of CopyObject — the garbage-collector trace revealed a surprising hotspot: time.Format was allocating 32 bytes per request just to render the LastModified timestamp in the XML response. At peak throughput that is roughly 3 MB/s of garbage only for timestamps, and the GC pause spiked to 12 ms — well past our 5 ms p99 latency target.\nWe did not catch this in unit tests. It only appeared under sustained load, when the allocation rate overwhelmed the generational collector\u0026rsquo;s ability to keep up. The \u0026ldquo;small allocation\u0026rdquo; was not small at scale.\nWhy the Obvious Solutions Failed First attempt: pool pre-formatted buffers.\n1 2 3 4 5 6 7 // REJECTED: pool mutex + GC pressure from pooled objects \u0026gt; allocation cost var timePool = sync.Pool{ New: func() any { s := make([]byte, 32) return \u0026amp;s }, } The pool\u0026rsquo;s mutex contention at 100k operations per second added more latency than the allocation itself. And pooled objects still pressure the collector — they just delay it, and risk promotion to the old generation where pauses are longer.\nSecond attempt: cache formatted timestamps. Timestamps are unique per request at nanosecond precision, so a cache would have a near-zero hit rate and only waste memory.\nThird attempt: pre-allocate a buffer and use fmt.Sprintf.\n1 2 3 // REJECTED: fmt uses reflection/interface{} internally, so it still allocates buf := make([]byte, 32) n := fmt.Sprintf(buf, \u0026#34;%s\u0026#34;, t.Format(time.RFC3339Nano)) Formatting through fmt goes through reflection and interface conversions, which allocate regardless of how the destination buffer was prepared.\nThe Solution — Stack-Buffer Time Formatting The standard library provides time.AppendFormat specifically for this. It writes directly into a byte slice and returns the extended slice — no intermediate string, no heap allocation.\n1 2 3 4 5 6 7 8 // Stack buffer: 32 bytes fits RFC3339Nano (20 chars) + timezone + padding. // Zero heap allocation — it lives on the stack and dies with the function. var timeBuf [32]byte // AppendFormat writes directly into the buffer and returns the extended slice. // timeBuf[:0] is an empty slice backed by the array (cap=32, len=0). t := time.Unix(0, modTime).UTC() buf.Write(t.AppendFormat(timeBuf[:0], \u0026#34;2006-01-02T15:04:05.000Z\u0026#34;)) Key insight: timeBuf[:0] creates a slice with length 0 but capacity 32, backed by the stack array. AppendFormat appends into that capacity, so the result points into the stack array and never escapes to the heap.\nPrinciple Callout Pattern: Stack-Buffer Time Formatting Use time.AppendFormat with a stack-allocated byte array in hot paths. It eliminates the heap allocation per call.\nApplies when: high-frequency timestamp rendering — HTTP headers, XML/JSON fields, log lines. Doesn\u0026rsquo;t apply: one-off formatting or human-readable output, where the allocation is negligible.\n1 2 3 4 5 6 7 8 9 10 11 // ✅ GOOD: Hot path — HTTP header, XML field, log line func writeLastModified(buf *bytes.Buffer, modTime int64) { var timeBuf [32]byte t := time.Unix(0, modTime).UTC() buf.Write(t.AppendFormat(timeBuf[:0], time.RFC3339Nano)) } // ❌ AVOID: Low-frequency path — allocation negligible func formatForDisplay(modTime int64) string { return time.Unix(0, modTime).Format(\u0026#34;Jan 2, 2006\u0026#34;) } Verification Metric Before After Delta Allocs/op 1 0 -100% Bytes/op 32 B 0 B -100% ns/op 451 463 +2.6% (within noise) GC pause (p99) 12 ms \u0026lt;1 ms -92% The paired microbenchmarks isolate the two implementations, and under 100k requests per second of production-like load the GC cycles dropped from 45 per minute to 3, with p99 latency settling at 3.2 ms.\nFailure Modes Risk Mitigation Buffer too small 32 bytes covers RFC3339Nano (20) + .000Z (5) + padding; it is a compile-time constant Timezone edge cases .UTC() normalizes before formatting, so output is consistent Thread safety The stack buffer is per-call — no sharing, no races When NOT to Use This Request IDs, trace IDs, and user-facing dates — these are not hot paths. One-off formatting in startup or shutdown paths. When you need a string return value for API compatibility; the allocation is unavoidable at that boundary. References / Dig deeper Mindset standards: docs/CORE/STANDARDS.md — ⚡ Bolt / 🛡 Sentinel. Bolt engineering overview: 024. The same pattern for HTTP Last-Modified headers: 045. Applied to ListParts XML: 048. The change spec: openspec/changes/bolt-s3-copyresult-time-alloc. The paired microbenchmarks live in src/s3/response_test.go (BenchmarkCopyObjectResponse_AppendFormat vs BenchmarkCopyObjectResponse_Format). ","permalink":"/posts/047-bolt-s3-copyresult-time-alloc/","summary":"How we optimized S3 XML responses by replacing time.Format with time.AppendFormat to eliminate heap allocations. Teaches: stack-buffer time formatting pattern, hot path identification via pprof.","title":"⚡ Bolt: Eliminating Time Formatting Allocations in S3 HTTP Responses"},{"content":"We had a filesystem core — directories as content-addressed manifests, files as blobs, atomic rename, hardlinks. We had no way for a normal program to use it. You cannot open() a manifest. The gap between \u0026ldquo;we can model a POSIX tree\u0026rdquo; and \u0026ldquo;/mnt/momo/notes.txt behaves like a file\u0026rdquo; is the kernel, and closing that gap is what the FUSE transport does.\nThe Real Problem Linux exposes filesystems through the kernel. A program calls open, read, write, and readdir, and the kernel decides which filesystem answers. Most filesystems are kernel code — you write a kernel module, and a bug in it can take down the machine. That is an unreasonable bar for a storage daemon whose whole point is to be a userspace program.\nFUSE — Filesystem in Userspace — removes that bar. The kernel provides a generic bridge: it forwards filesystem calls to a userspace process over a device file called /dev/fuse, and that process implements the filesystem in whatever language it likes. If the daemon crashes, the kernel marks the mount dead instead of panicking. The trade is a small cost per call (a round trip to userspace) for the enormous benefit of not writing kernel code.\nSo the problem narrowed to: how does a userspace process translate the kernel\u0026rsquo;s file operations into operations on content-addressed manifests and blobs, and how do we keep the mount from leaking when the process dies?\nWhy the Obvious Solutions Failed An S3 gateway is not enough. The native API and the S3 gateway already let programs read and write objects, but they speak HTTP, not the read/write syscalls a normal program expects. Unmodified tools — cat, cp -r, an editor — cannot use an HTTP object API without a client. A filesystem is the universal interface.\nA custom kernel module. Writing the filesystem as kernel code would remove the userspace hop, but it means a memory-safety bug in our storage logic becomes a kernel panic, and every kernel version becomes a compatibility surface. We chose the safe boundary.\nBuffering the whole file in memory. The simplest way to reconcile the kernel\u0026rsquo;s byte-range write model with our write-whole-file blob model is to hold the entire file in RAM until close. That is correct but unbounded: a large write would pin its full size in the daemon\u0026rsquo;s memory, and a few concurrent writers could exhaust it.\nCaching aggressively for speed. The mount could answer getattr from a cache for longer, but then a file written through the native API would stay invisible in the mount for an arbitrary window. We wanted read-your-own-writes to hold.\nThe Solution: A FUSE Transport Over the Core The R4 FUSE transport exposes the POSIX core to the kernel through FUSE, using the bazil.org/fuse binding (the Go library that speaks the FUSE protocol). Tracking issue #962 led to pull request #963. The daemon gains a new impersonation mode: instead of acting as a server, it mounts a store as a filesystem and serves kernel calls until told to stop.\nFigure 1: Architectural layers bridging POSIX filesystem calls to the immutable CAS store The pieces:\nMount entrypoint. Running the daemon in filesystem mode (momo -imp fs) with a mount point and an optional data directory starts a FUSE server. The server is context-cancellable, and on SIGINT or SIGTERM the process unmounts cleanly before exiting. The FUSE adapter. A set of node types model the tree to the kernel: a root, directories, files, and per-open handles for files and directories. Every error the core can return is mapped to the matching POSIX error code (the errno the kernel expects) so that a missing file returns \u0026ldquo;not found\u0026rdquo; rather than a generic failure. Writes buffer per handle, not per file. The kernel writes files in arbitrary byte ranges; our CAS layer wants a whole blob. The adapter keeps a bounded buffer per open handle and materializes a single whole blob on flush or release. Reads go the other way and stream from the core. Testing that respects the environment. Node and handle behavior is covered by unit tests, and a kernel round-trip end-to-end test runs against a real /dev/fuse mount — but self-skips on machines (such as most CI runners) where /dev/fuse is absent. Goroutine-leak and race-detector checks accompany it. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 // The kernel hands us byte-range writes; the CAS layer stores whole immutable // blobs. We reconcile the two by buffering per open handle and materializing one // blob at flush/release. The buffer is bounded, so a large write costs a bounded // amount of daemon memory rather than the whole file. func (h *fuseFileHandle) Write(ctx context.Context, req *fuse.WriteRequest, resp *fuse.WriteResponse) error { // Grow within a cap; refuse rather than pin unbounded memory. if len(h.buf)+len(req.Data) \u0026gt; maxHandleBuffer { return syscall.ENOSPC } h.buf = append(h.buf, req.Data...) resp.Size = len(req.Data) // the kernel believes the range landed return nil } Implemented vs. Planned (Honesty Note) The POSIX write model and metadata semantics described here are implemented — the core plus the transport plus the mount user guide all agree. What is still planned is finer-grained write behavior: full mmap byte-range correctness and POSIX locks. Today, writes are buffered per handle and materialized whole on flush; truncation size hints are accepted but their semantics land with the byte-range follow-up.\n⚡ Bolt + 🛡 Sentinel Lens Per docs/CORE/STANDARDS.md:\n⚡ Bolt: per-handle write buffers are bounded, so memory does not grow without limit during large in-flight writes; reads stream rather than copy the blob into daemon memory. 🛡 Sentinel: a clean unmount is a safety property, not housekeeping. A stale FUSE mount left behind by a crash presents a partial tree, so shutdown unmounts deliberately; scatter operations stay lease-validating so rename and refcount survive across the transport. Failure Modes Risk Guard Daemon crash leaves a dead mount Kernel marks the mount dead; shutdown path unmounts on signal Large write overwhelms daemon memory Per-handle buffer is bounded and refuses beyond the cap Native write invisible in mount Manifests are read fresh, so reads are read-your-own-writes Core error surfaces as generic failure Errors are mapped to precise POSIX codes CI lacks /dev/fuse Kernel e2e self-skips; unit tests still run When NOT to Use This Non-Linux hosts. FUSE here targets Linux /dev/fuse; macOS and Windows use different transports (see the migration post and the roadmap). Pure object workloads. If no program needs the mount, the FUSE hop is pure overhead — use the native or S3 interface instead. Write-heavy large appends today. Whole-blob materialization makes large append workloads pay whole-file cost until the byte-range follow-up lands. References / Dig deeper Spec: openspec/changes/r4-momofs. Pull request #963; tracking issue #962. The core underneath: 022: momofs POSIX Core. Byte layer: 004: Content-Addressable Storage. Where R4 sits: 028: Roadmap and Research. Later migration to go-fuse/v2: 029: Migrating to go-fuse/v2. Operational docs: docs/momofs/MOUNT_USER_GUIDE.md, docs/momofs/IMPLEMENTATION.md. ","permalink":"/posts/023-momofs-fuse-transport/","summary":"The FUSE transport (\u003ccode\u003emomo -imp fs\u003c/code\u003e): bazil.org/fuse binding, per-handle buffered writes, /dev/fuse-gated e2e, and a clean unmount lifecycle.","title":"R4: momofs FUSE Transport — Mounting Momo as a Filesystem"},{"content":"A performance rule with no teeth is a suggestion. We had a pile of zero-allocation patterns we believed in, and no way to stop the next change from quietly undoing one. So we built a gate: on every pull request, run the benchmarks against the base branch and the proposed branch, compare them, and block the merge if the new code is measurably slower or allocates more.\nThe comparison tool is benchstat, the standard Go program that separates real performance differences from run-to-run noise. A regression is any statistically significant slowdown or extra allocation introduced by a change. A benchmark gate is the CI rule that turns a regression into a failed build. An allowlist is the short list of benchmarks we tell the gate to ignore, because we know they measure the environment rather than the code.\nThat last idea — that some benchmark failures are not about code at all — is the whole story of August 2026.\nThe Real Problem The gate kept failing for reasons that had nothing to do with the change under review. This is classic benchmark runner noise: the shared CI machine is noisy, so a benchmark can look 8% slower on one run and 3% faster on the next with no code difference. A gate that cries wolf is worse than no gate, because people learn to override it.\nEach false alarm forced a fix that made the signal more trustworthy:\nA quoting crash. benchstat exited with a crash code because an apostrophe inside a comment in the shell allowlist broke the surrounding shell quoting. The benchmark data was fine; the script parsing it was not. The fix was as small as it was embarrassing: no apostrophes in those comments. A disk-spool artifact. The S3PutSpool/64MiB benchmark regressed seemingly at random. It was not a code change: the CI virtual machine writes its spool to a slow temporary disk, so the benchmark was measuring disk speed, not our code. We allowlisted it as an I/O-artifact benchmark. A local-write artifact. LocalWrite behaved the same way — the runner\u0026rsquo;s own filesystem noise. Allowlisted for the same reason. Why the Obvious Solutions Failed \u0026ldquo;Just raise the threshold.\u0026rdquo; Loosening the allowed percentage hides the environment noise, but it also hides the real regressions we built the gate to catch. We would have traded a noisy gate for a useless one.\n\u0026ldquo;Retry until it passes.\u0026rdquo; Re-running a flaky benchmark until it looks green is not measurement; it is a coin flip with extra compute. It also trains reviewers to ignore the gate.\n\u0026ldquo;Delete the noisy benchmarks.\u0026rdquo; Tempting, but spool and local-write paths are exactly the storage behavior we care about. The answer was to keep measuring them and label the noise, not to stop looking.\nThe Solution: An Honest, Auditable Gate Three decisions made the gate trustworthy:\nCompare against the base with a three-dot diff. A three-dot diff compares the branch tip against the common ancestor it forked from, so merging the main branch into your branch cannot make the comparison look artificially clean. The gate measures your change, not the merge. Allowlist environment-sensitive benchmarks explicitly. Rather than silently ignoring failures, the allowlist names the benchmarks that measure the runner, and that list is reviewed like code. Keep an authoritative history. A generated performance document and a checked-in benchmark history file record every run, so a disputed regression can be traced over time instead of argued about. The result distinguishes real regressions (blocked) from environment jitter (allowlisted and recorded). The distinction is the entire point: a gate you trust is a gate people respect.\n⚡ Bolt Lens A benchmark gate is Bolt practiced as process: measure, compare, gate. The three-dot and allowlist story is the reminder that measurement infrastructure itself needs hardening — an honest gate is an auditable trust invariant. The ⚡ Bolt mindset supplies the performance bar; the 🛡 Sentinel mindset insists the bar be enforced by something auditable rather than by good intentions.\nSee docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Mindset standards: docs/CORE/STANDARDS.md. The Bolt engineering post this gate protects: 024. Profiling baseline: 042. The performance document: docs/REFERENCE/PERFORMANCE.md; benchmark history: .github/data/benchmark_history.csv; the gate workflow: .github/workflows/benchmark_compare.yml. The three-dot diff governance spec: openspec/changes/steering-three-dot-diff-gate. Tracking issues #958, #960 and the fix PR #961. ","permalink":"/posts/025-benchmark-benchstat-gate/","summary":"Base-vs-branch benchmark comparison gates every PR — plus the runner-noise saga (S3PutSpool, LocalWrite, allowlist, apostrophe) that made it honest.","title":"The Benchstat Gauntlet: Performance Regressions as CI"},{"content":"For three releases momo was an object store: you wrote bytes under a key and read them back by that key. Then we asked a question that turned out to be the biggest architecture bet yet — can it also behave like a filesystem? Not a filesystem bolted on top, with its own separate storage and its own separate consistency rules, but a filesystem whose substrate is the object store we already had. R4 is the answer: a POSIX core, tracked by the R4 issue and shipped in the R4 pull request.\nThe Real Problem An object store and a filesystem disagree about what an object is. A filesystem organizes bytes into a tree of named files and directories with metadata — permissions, ownership, timestamps — and expects operations like rename to be atomic. An object store knows only immutable blobs addressed by a key. If we tried to run both by keeping two separate stores, we\u0026rsquo;d immediately own two sources of truth: a file written one way would have to be copied into the other, and the copies would eventually disagree.\nSo the real requirement was harsher than \u0026ldquo;add a filesystem.\u0026rdquo; It was: one store, visible through two interfaces. An object written through the native API must appear in the filesystem tree, and a file written through the mount must be a real object — with no syncing, no second database, and no window where the two views disagree.\nWe had one asset that made this plausible. The object store is content-addressed (see the CAS post): every blob is named by the SHA-256 hash of its own bytes, so identical content collides on purpose and deduplicates for free. If the filesystem layer could be built entirely out of content-addressed objects, consistency would not be a synchronization problem at all — both interfaces would simply be reading and writing the same named bytes.\nWhy the Obvious Solutions Failed A separate metadata database for the tree. The conventional filesystem design keeps an inode table and a directory index in their own database. We rejected it because it recreates exactly the two-sources-of-truth problem we were trying to avoid: the tree and the object store could drift, and reconciling them would be someone\u0026rsquo;s full-time job.\nReusing the existing key/value namespace as the tree. We could have mapped /a/b.txt onto a single object key a/b.txt. That gives you a flat map, not a filesystem: there is no directory object to readdir, no place to store POSIX metadata per entry, and rename becomes a multi-key rewrite that cannot be atomic.\nTreating directories as empty marker objects. A zero-byte \u0026ldquo;directory\u0026rdquo; object has nowhere to record its children, so listing a directory means scanning every key for a prefix — correct but O(n) per readdir, and it still cannot express hardlinks, where two names must point at one underlying file.\nEach of these solved the visible problem (expose files) while quietly deferring the hard part (one consistent substrate). We wanted the opposite trade.\nThe Solution: The Tree Is Made of Objects The R4 core reuses the content-addressed store as the byte layer and invents a metadata layer on top of it. The whole design rests on one mapping:\nDirectories are content-addressed manifests. A directory is itself a stored object whose contents are a list of entries — each entry pairs a name with the address of a child (its manifest or blob) and carries POSIX metadata such as mode, owner, and group. Because a manifest is an object, it is immutable and versioned by its content hash: changing a directory means producing a new manifest, and the old one remains addressable. Files are the existing blobs themselves. File content is hash-addressed exactly like any other object, so deduplication applies to file data with no extra machinery. Inode metadata rides alongside each manifest or blob. POSIX attributes are not a separate table; they travel with the object that represents the entry. The payoff is the property we were after. A directory listing such as mkdir x followed by writing a file inside it is, underneath, one manifest object and one blob object: a single store that is simultaneously an S3-style key space and a POSIX tree. Consistency across the two interfaces falls out for free, because there is only ever one manifest to read.\nFigure 1: The POSIX core maps the kernel\u0026rsquo;s file operations onto content-addressed manifest and blob objects 1 2 3 4 5 6 7 8 9 10 11 // A directory is an object whose payload is its entry list. Each entry names a // child object by content hash and carries that child\u0026#39;s POSIX attributes — so a // single manifest read answers both \u0026#34;what is here?\u0026#34; and \u0026#34;what are its modes?\u0026#34;. type ManifestEntry struct { Name string Child BlobID // address of the child\u0026#39;s manifest (dir) or blob (file) Mode os.FileMode UID int GID int Nlink int // hardlink count, kept aligned with the GC refcount sweep } What Came With It The core implements the operations a filesystem must answer — lookup, getattr, setattr, readdir, open, create, read, and write — against the content-addressed layer rather than a local inode table.\nThe subtler work was atomic rename and hardlinks. A rename must be atomic from a reader\u0026rsquo;s perspective: either the old name or the new name, never a half-moved entry. Because a directory is a manifest and manifests are immutable, a rename rewrites one manifest and swaps it in a single step — a reader sees the old manifest or the new one, never both. Hardlinks are the reverse pressure: two names point at one content, so the filesystem has to keep a reference count telling the garbage collector how many live names still need the bytes. We aligned that count with the tombstone-driven GC sweep, so a hardlinked blob is not reaped while a name still points at it. That work connected the filesystem directly to the at-rest integrity and GC arc.\nImplemented vs. Planned (Honesty Note) The momofs documentation suite is largely a design and plan corpus. In this post the factual, shipped layer is the POSIX core implemented in the R4 pull request, plus the operational docs we keep in sync with it. Design features that are not yet implemented — full POSIX ACLs, quotas, and snapshots — remain planned, not shipped, and should not be read as available.\n⚡ Bolt + 🛡 Sentinel Lens Per docs/CORE/STANDARDS.md:\n⚡ Bolt: manifests are small, hot, and read constantly, so metadata handling is written to be dense and allocation-light; the byte path rides the existing zero-allocation hashing of the CAS layer. 🛡 Sentinel: reference counts are aligned with the garbage-collection sweep, so a hardlink cannot permit the collector to reap a still-referenced blob. The delete path is a single auditable owner rather than scattered logic. Failure Modes Risk Guard Rename observed half-applied Manifest swap is a single immutable object replacement Blob reaped while a hardlink still points at it Refcount tracks live names and gates the tombstone sweep Tree and object view disagree Both views read the same manifests — there is no second database Planned feature assumed shipped This post and the design suite label them explicitly When NOT to Use This Mutable, in-place files. Content addressing rewrites a whole new blob on change; this design suits files that are rewritten wholesale, not edited in place with fine-grained range updates (a follow-up area). When you only need an object API. If no POSIX consumer needs the tree, the metadata manifests are pure overhead — use the native key space directly. References / Dig deeper Spec: openspec/changes/r4-momofs. Pull request #957 (POSIX core); tracking issues #932 (R4) and #962 (FUSE). Byte layer: 004: Content-Addressable Storage. Placement: 005: CRUSH-lite. Mount it with FUSE: 023: momofs FUSE Transport. GC and at-rest integrity: 007: At-Rest Integrity. Operational docs: docs/momofs/MOUNT_USER_GUIDE.md, docs/momofs/IMPLEMENTATION.md. ","permalink":"/posts/022-momofs-posix-core/","summary":"The R4 POSIX core: inodes as content-addressed manifests, files as blobs, atomic rename, hardlinks, permission enforcement — a filesystem on momo.","title":"R4: momofs — POSIX Core Over the CAS Store"},{"content":"Two earlier milestones answered \u0026ldquo;what happens when data dies\u0026rdquo; — where copies live, and how we recover a corrupted one. But they left the most uncomfortable question untouched: when we tell a client a write succeeded, what exactly have we promised?\nFor a long time the honest answer was \u0026ldquo;not much.\u0026rdquo; We returned success as soon as the bytes were handed to the operating system. If the machine lost power a millisecond later, those bytes might still be sitting in volatile page cache and simply vanish. The client believed its data was safe. It was not.\nThe Real Problem A write acknowledgement is a promise about the future: this data will survive. If we acknowledge before the data has actually reached durable storage, every restart window becomes a silent data-loss window. \u0026ldquo;ACK then restart = lost object\u0026rdquo; is exactly the kind of guarantee failure that erodes trust — not loud, not crashy, just quietly missing data.\nTwo notions of \u0026ldquo;written\u0026rdquo; were being conflated:\nthe bytes are in memory somewhere in the pipeline, and the bytes are safely on disk on enough machines to survive a failure. R3 makes the acknowledgement mean the second one.\nWhy the Obvious Solutions Failed \u0026ldquo;Acknowledge immediately, persist in the background.\u0026rdquo; Fast, and wrong for durability. It trades a latency win for invisible data loss on crash.\n\u0026ldquo;Always fsync on every write.\u0026rdquo; Correct, and painfully slow. Forcing a disk flush on every small write serializes throughput and turns a fast engine into a sluggish one.\n\u0026ldquo;Acknowledge after one node writes.\u0026rdquo; This ignores the whole point of replication: if that one node dies, the \u0026ldquo;successful\u0026rdquo; write is gone. Success has to reflect what survived, not what one machine happened to accept.\nThe Solution: Define Success Explicitly R3 ships a durability barrier with three movable parts, so operators can pick the trade-off deliberately instead of inheriting whatever the implementation happened to do.\nR3 Write durability barrier fsync-before-ack. A write is acknowledged only after the blob has durably hit disk — not just page cache — on the surviving replicas. fsync is the system call that forces the operating system to flush buffered writes all the way to physical media; before it returns, the data is genuinely persistent. This is the change that kills the \u0026ldquo;ACK then restart = lost object\u0026rdquo; window. Group commit and a none mode. A configurable barrier with three settings: fsync (flush every write, strictest), group-commit (batch several writes into one flush, trading a little latency for a lot of throughput), and none (no flush, fastest, least durable). Operators choose their strictness explicitly rather than discovering it by accident. Survivor quorum. Success requires the required number of surviving replicas to confirm the flush. A quorum is simply a threshold of confirmations — you only report success when enough independent copies have agreed the data is safe, following the same lease and quorum logic we use elsewhere for consistency. Read-your-writes. After an acknowledged write, a subsequent read of that object returns the acknowledged data — never an older version. This closes the confusing gap where a client writes something, reads it back, and finds stale bytes. Group commit, plainly The interesting design choice is group commit. Instead of one flush per write, we let a small batch of writes accumulate and share a single flush. Ten writes that would have caused ten disk round trips now cause one. The writers all wait for the same flush and are acknowledged together. Latency per write rises slightly; throughput rises sharply. For workloads with many small writes, that is a good trade — and crucially, it is an explicit one.\nThe Durability Floor in Practice The durability work also closed a set of silent correctness gaps around replication forwarding. A later audit-era batch bounded the whole layer:\nno silent acknowledgement when forwarding to a replica failed; no blocking on a semaphore without a context deadline (so a stuck replica could not hang a writer forever); and correct behavior when a delete was only partially propagated. Each of these was the same class of bug: the system reporting success for work it had not actually completed. R3\u0026rsquo;s principle — ack only what survived — is what let us find and fix them.\nThe Sentinel Lens fsync-before-ack is the Sentinel reading of the word \u0026ldquo;success\u0026rdquo;: a client must never observe a lie about durability. The three R-milestones fit together:\nR1 — where replicas live. R2 — what survives corruption. R3 (this) — what \u0026ldquo;written\u0026rdquo; actually means. Together they discharge the P0 durability and consistency mandate.\nWhat Could Go Wrong none mode is genuinely unsafe. It exists so the trade-off is explicit, not so it is free. A crash can silently lose acknowledged writes. A quorum that cannot be reached. If too few survivors are healthy, the write fails rather than lying. That is correct, but it means availability depends on having enough replicas in distinct failure domains. Group commit under light load. With only one write in flight there is nothing to batch, so latency looks like plain fsync. The win only appears under concurrency. When NOT to Use This Scratch or regenerable data. If losing a write after a crash is genuinely acceptable, none mode is a legitimate choice — just make it a conscious one. When the bottleneck is the network, not the disk. fsync-before-ack adds latency; if disks are not the durability risk you care about, measure before paying for strictness. As a substitute for replication. The barrier tells you what survived. It does not create copies — R1 and R2 do that. References / Dig deeper Failure-domain placement: 019. Degraded read and self-heal: 020. Scatter-gather lease quorum: 017. Replication strategies: 002. Production readiness roadmap: 028. Specs: openspec/changes/r3-durability-consistency, openspec/changes/add-replication-durability-floor; PR #954; tracking issue #928. Audit-era durability batch: 015. ","permalink":"/posts/021-r3-write-durability-quorum/","summary":"Ack only what survived: fsync/group-commit durability barrier, survivor quorum, and read-your-writes — the write-side of the P0 stack.","title":"R3: Write Durability — fsync-Before-Ack, Survivor Quorum"},{"content":"A user reported that a video file played fine one minute and then, without any warning, returned a server error. Nothing had changed. No node was down, no deploy had shipped. One sector on one disk had quietly rotted, and the request happened to land on that disk.\nDisks do not merely fail catastrophically by dropping offline — they also rot incrementally. A single bad sector can corrupt a 4 KB block in the middle of a 50 MB video while the rest of the file looks perfectly healthy. We call this bitrot: silent, localized corruption that a disk reports only when you happen to read the affected bytes.\nThe Real Problem Before milestone R2, if a client\u0026rsquo;s read landed on a node whose local disk block was corrupted or missing, the storage engine surfaced a raw system error and gave up. The client saw a hard failure, even though two perfectly healthy copies of the object existed on peer nodes elsewhere in the cluster. We had redundancy we were not using at read time.\nThe fix had to do two things: serve the read anyway, from a healthy peer, and then repair the damaged node so the cluster returned to full redundancy without a human in the loop. That second half is what we call self-heal — the cluster detecting its own under-replication and autonomously converging back to the target number of copies.\nWhy the Obvious Solutions Failed \u0026ldquo;Return the error.\u0026rdquo; This preserves correctness at the cost of availability. We had the data; refusing to serve it makes our redundancy worthless exactly when it matters.\n\u0026ldquo;Repair synchronously, then serve.\u0026rdquo; We considered blocking the client while we re-wrote the corrupt block locally. But the repair might need to fetch a large object across the network, turning a read into a long stall on the user\u0026rsquo;s thread.\n\u0026ldquo;Ignore corruption and keep serving local bytes.\u0026rdquo; This is the worst option. Serving rotted data silently corrupts the objects the cluster is supposed to protect.\nWe chose a third path: serve from a healthy survivor immediately, explain the corruption to no one, and repair in the background.\nThe Solution: A Seam for Survivors The storage engine must not be tightly coupled to cluster transports, network sockets, or placement logic — otherwise we could never test the recovery logic without a live cluster. Following our seam discipline (a compile-time interface that marks the one place a subsystem is allowed to plug into another), we defined a RebuildSource interface as the boundary between the read path and the cluster:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 type Peer struct { ID int Domain string // R1 failure domain } type RebuildSource interface { // Survivors returns peer daemons holding verified copies of the blob. Survivors(hash string) ([]Peer, error) // Fetch opens a verify-before-use stream from a healthy survivor. Fetch(hash string, survivor Peer) (io.ReadCloser, error) // Restore pushes verified bytes to repair under-replicated nodes. Restore(hash string, content io.Reader) error } Production wires this seam to placement and peer RPCs. Tests inject deterministic in-memory mock survivors, so the recovery logic is exercised without a network.\nTransparent degraded read Inside the read path, when a local read fails, we intercept the error and try the cluster instead:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 stream, meta, err := s.blobs.GetBlob(hash) if err != nil || corruptOnRead { // Local read failed (ENOENT or EBADMSG bitrot) if s.rebuildConfig.DegradedRead \u0026amp;\u0026amp; s.rebuildSource != nil { log.Printf(\u0026#34;R2: Local blob %s corrupt or missing; attempting degraded read\u0026#34;, hash) // 1. Find surviving peers holding valid copies survivors, sErr := s.rebuildSource.Survivors(hash) if sErr == nil \u0026amp;\u0026amp; len(survivors) \u0026gt; 0 { // 2. Fetch verified stream from the first healthy survivor remoteStream, fErr := s.rebuildSource.Fetch(hash, survivors[0]) if fErr == nil { // 3. Quarantine corrupt local block to prevent re-serving _ = s.quarantineBlob(hash) // 4. Return remote stream directly to client return remoteStream, meta, nil } } } return nil, meta, fmt.Errorf(\u0026#34;read failed and no survivors available: %w\u0026#34;, err) } The client receives the complete, verified stream with no protocol error. The only observable artifact of the underlying disk failure is a slightly higher Time-to-First-Byte (TTFB) — the delay before the first byte of the response arrives. We quarantine the bad local block — mark it unusable so it is never served again — rather than deleting it, so operators can still inspect the damage.\nThe background self-heal loop Surviving a degraded read is only half the battle. The cluster is now running at one replica fewer than its target, which leaves it vulnerable to a second failure. A background reconciliation loop repairs the gap:\nInventory. The worker walks the object index for active blobs. Audit. For each blob it asks the cluster how many healthy copies currently exist. Restore. If that count is below the configured target, it fetches a verified stream from a survivor and pushes it to a node that needs the copy. Domain awareness. The new copy\u0026rsquo;s placement prefers unoccupied failure domains, from 019, so a restored copy does not land in the same rack as every survivor. The Trade-off The diagram below traces what happens on a corrupt local read, with and without R2:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ======================================================================================== TRANSPARENT DEGRADED READ \u0026amp; REBUILD ======================================================================================== Client GET /data/file.bin | v [ Node 0 (Local Primary) ] | v Opens blobs/e3/b0/c4/... -\u0026gt; DISK CORRUPTION DETECTED (syscall.EBADMSG) | +-------------------------------------------------------+ | | (Without R2) (With R2) | | v v Fail client with 500 / EBADMSG 1. Intercept EBADMSG (Availability Lost!) 2. Mark local blob in bucketQuarantine 3. Query RebuildSource.Survivors(hash) | v [ Dial Node 1 (Survivor) ] | v Stream verified payload to Client (Client experiences 0 errors!) | v 4. Queue Async Self-Heal Rebuild: Restore healthy blob to Node 0 disk Strategy Client Impact Network Overhead Recovery Speed Fail-Stop (No R2) Immediate client 500 error Zero Manual operator intervention Immediate Synchronous Repair High latency spike (client waits for local re-write) Spikes on user thread Immediate Degraded Read + Async Self-Heal (momo R2) Zero errors; minor TTFB increase on first read Smooth background stream Autonomous background convergence We accepted degraded-read complexity and background network traffic in exchange for an availability guarantee. The cost is real: a corrupted node now generates repair traffic, and a degraded read is slower than a healthy one. But a user watching a video never sees the failure.\nHow We Verified Per docs/CORE/STANDARDS.md:\n🛡 Sentinel (verify-before-use): Fetch() enforces strict streaming verification. A survivor\u0026rsquo;s bytes are never accepted or written to local disk without re-deriving the object\u0026rsquo;s SHA-256 hash. Bad data on one node can never propagate to heal another — we verify before we trust, every time. ⚡ Bolt (bounded concurrency): the rebuild loop runs a small bounded worker pool (default two workers) so background repair traffic cannot starve customer read and write IOPS. What Could Go Wrong Every survivor is corrupt. If no healthy copy remains, the degraded read fails; R2 cannot invent data. This is why background scrub and repair must run promptly after the first failure. A rebuild loop under load. Repair traffic competes with user traffic. The bounded worker pool caps that competition, but a very degraded cluster may take a while to converge. Quarantine growth. Quarantined blocks accumulate until repaired; a persistently failing disk needs operator attention even though the cluster keeps serving. When NOT to Use This A single-node deployment. With no peers there are no survivors to fall back to; degraded read cannot help. When read latency is sacrosanct above all. The remote fallback adds a round trip. Systems with no latency budget may prefer to fail fast and repair out-of-band. As a replacement for scrubbing. Degraded read only triggers when a client happens to touch a bad block. Background scrub is what catches rot no one has read yet. References / Dig deeper Integrity and background scrubbing: 007. Domain-aware placement: 019. Write durability and quorum: 021. Seams architecture: 044. Scatter-gather lease quorum: 017. Implementation: src/storage/rebuild.go, src/storage/storage.go. Spec: openspec/changes/r2-self-heal-rebuild; PR #953; tracking issue #928. ","permalink":"/posts/020-r2-degraded-read-self-heal/","summary":"When a local replica is corrupted or missing, degraded read fetches from healthy survivors transparently, while a background rebuild loop restores cluster redundancy.","title":"R2: Degraded-Read Survivor Fallback and Self-Heal"},{"content":"We had three copies of every object, and we thought that meant we were safe. Then someone on the team drew the actual racks on a whiteboard and the room went quiet.\nIn distributed storage there is a dangerous difference between independent failures and correlated failures. Independent failures hit one machine at a time: a disk dies, a process crashes, a node reboots. Correlated failures hit a whole group at once because the group shares something — a rack, a switch, a power feed, a data center. Three copies only protect you against the first kind. If all three copies live behind the same Top-of-Rack switch, a single firmware update takes out every copy simultaneously. The redundancy was real on paper and imaginary in the building.\nThe Real Problem Our placement algorithm scored each server in isolation. It asked \u0026ldquo;which nodes are the best fit for this object?\u0026rdquo; and never asked \u0026ldquo;do those nodes share a point of failure?\u0026rdquo; In a cluster of nine servers spread across three physical racks, an object needing three copies could easily land on the three highest-scoring nodes — all three of which happened to sit in the same rack.\nThe guarantee we advertised was three replicas. The guarantee we actually delivered was one rack\u0026rsquo;s worth of durability. A single tripped breaker or switch update caused complete, unrecoverable data loss, and no amount of node health-checking would have warned us.\nWe needed to convert the promise from a simple replica count into physical blast-radius separation: copies should be spread so that no single physical failure can take out more than one of them.\nWhy the Obvious Solutions Failed \u0026ldquo;Just raise the replication factor.\u0026rdquo; More copies in the same rack do not help against a rack failure. Going from three copies to five inside one rack buys nothing against the failure that actually kills you.\n\u0026ldquo;Hand-assign replicas to racks.\u0026rdquo; Static pinning works until hardware changes. Nodes get added, replaced, and rebalanced; a hand-maintained mapping rots the first time someone swaps a disk without telling the placement layer. We wanted the placement to stay automatic and deterministic.\n\u0026ldquo;Refuse to place when domains are short.\u0026rdquo; This is safe but brittle. A small cluster with two domains and a replication factor of three would reject every write, turning a durability concern into an availability outage. Falling back gracefully is better than failing closed.\nThe Solution: Dispersion Before Rank We upgraded the placement algorithm to be failure-domain aware. A failure domain is simply a label for a group of machines that share a common point of failure — a rack, an availability zone, a data center. We taught each node to carry one.\n1 2 3 4 5 type Node struct { ID int Weight uint32 Domain string // e.g. \u0026#34;us-east-1a\u0026#34;, \u0026#34;rack-04\u0026#34;, \u0026#34;dc-2\u0026#34; } Placement then runs in two phases instead of one:\nScore every node. The algorithm computes its deterministic placement score for every eligible node, exactly as before. Select with a domain constraint. The highest-scoring node becomes the primary, and its domain is marked occupied. For each remaining copy, the algorithm walks down the score ranking and takes the highest-scoring node whose domain is not yet occupied. A copy is only allowed to share a domain with an existing copy after every distinct domain has been used at least once. That last clause is the graceful fallback: if the requested replication factor exceeds the number of available domains, we still fill the requested number of copies rather than refusing the write.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 // src/common/crush.go if domainsConfigured { selected := make([]*Node, 0, replicationFactor) usedDomains := make(map[string]bool) // Pass 1: Select highest-scoring node per unique domain for _, s := range scores { if len(selected) == replicationFactor { break } if !usedDomains[s.node.Domain] { selected = append(selected, s.node) usedDomains[s.node.Domain] = true } } // Pass 2: If R \u0026gt; num_domains, fill remaining slots by raw score rank if len(selected) \u0026lt; replicationFactor { for _, s := range scores { if len(selected) == replicationFactor { break } if !containsNode(selected, s.node) { selected = append(selected, s.node) } } } return selected, nil } The diagram below contrasts the old unconstrained result with the new domain-constrained one:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 ======================================================================================== FAILURE-DOMAIN AWARE PLACEMENT ======================================================================================== CRUSH-Lite Naive Placement (UNSAFE): +-----------------------------------------------------------------------------------+ | Rack A (Domain: rack-1) | Rack B (Domain: rack-2) | Rack C (Domain: rack-3) | | [Node 0]* [Node 1]* [Node 2]* | [Node 3] [Node 4] | [Node 5] [Node 6] | +-----------------------------------------------------------------------------------+ * All 3 replicas land in Rack A! If Rack A\u0026#39;s PDU trips -\u0026gt; Complete Data Loss! R1 Domain-Constrained Placement (SAFE): +-----------------------------------------------------------------------------------+ | Rack A (Domain: rack-1) | Rack B (Domain: rack-2) | Rack C (Domain: rack-3) | | [Node 1]* (Score: 0.91) | [Node 4]* (Score: 0.85) | [Node 5]* (Score: 0.79) | +-----------------------------------------------------------------------------------+ * Replicas forced into distinct failure domains: Cluster survives any single rack loss! The Trade-off: Capacity Utilization vs. Domain Isolation Domain isolation is not free. Forcing copies into distinct domains means the cluster\u0026rsquo;s usable capacity is capped by its smallest domain, because you can never fill one rack past the point where the others can still host their share. The placement computation also does a little more work per call, and rebalancing can move data across domains. We judged all three costs clearly worth paying: a slightly less efficient cluster that survives a rack failure beats a perfectly packed one that does not.\nMetric / Dimension Unconstrained CRUSH Failure-Domain Aware CRUSH (R1) Correlated Failure Resilience Poor: High probability of multiple replicas in 1 rack Guaranteed: Replicas strictly isolated across distinct domains Disk Capacity Utilization Purely proportional to node weights Constrained by the smallest domain\u0026rsquo;s capacity ceiling Placement Compute Latency A single sort over the nodes A single sort plus a domain filter pass Data Movement on Rebalance Minimal (a small fraction of data) Minimal within domains; bounded inter-domain migration How We Verified In accordance with docs/CORE/STANDARDS.md:\n🛡 Sentinel (trust invariant): reliability claims that ignore failure domains are dishonest. A cluster configured with three copies across three racks can now lose an entire rack without losing data availability. ⚡ Bolt (benchmark guard): a microbenchmark drives a 100-node cluster map through the domain-aware placement path and confirms it completes in under 1.8 microseconds, with zero heap allocations during the filtering loop. What Could Go Wrong Misconfigured domains. If an operator labels every node with the same domain, the algorithm silently degrades to unconstrained placement. The protection is only as good as the domain labels. Too few domains. With two domains and three copies, one domain necessarily holds two copies; a single failure still costs one copy. The fallback keeps the write available, but it cannot manufacture physical separation that does not exist. Weight skew. A very small domain can cap effective capacity, so domain sizing deserves the same planning attention as raw node weights. When NOT to Use This Single-rack deployments. If everything already shares one point of failure, domain constraints add complexity without adding protection. When capacity matters more than blast radius. A cluster that is already replicated off-site and can tolerate a full-rack rebuild may prefer maximum packing. As a substitute for replication. Domain separation spreads copies; it does not create them. It only helps once you already keep more than one. References / Dig deeper The CRUSH-lite base algorithm: 005. Degraded-read and self-heal recovery: 020. Write durability and quorum: 021. Production readiness roadmap: 028. Placement implementation: src/common/crush.go; benchmark: src/storage/bench_test.go (BenchmarkPlacementDomainSpread). Spec: openspec/changes/r1-failure-domains; PR #952; tracking issue #928. ","permalink":"/posts/019-r1-failure-domain-placement/","summary":"Replicas must not ride in the same rack: CRUSH weights become rack/zone/DC-aware so a single domain failure can\u0026rsquo;t lose the data.","title":"R1: Failure-Domain-Aware Placement"},{"content":"We ran our first proper performance profile on the storage engine and got a number that made no sense. Writing an object was fast. Reading it back was miserably slow. On a machine with a fast NVMe drive, the same bytes that went in at roughly 2400 MB/s came out at about 313 MB/s — reading was nearly eight times slower than writing.\nA read is supposed to be the easy direction. Something was badly wrong.\nThe profiler pointed at one function: the SHA-256 hash. Nearly all CPU cycles were going into recomputing a cryptographic digest of files we had already verified minutes earlier. To protect against silent disk corruption, our verify-on-read logic re-hashed every object on every single read, even when nothing had changed. The storage server was CPU-bound, spending its time re-proving something it already knew.\nThe Real Problem Verify-on-read exists for a good reason. Bitrot — silent, localized disk corruption — can flip bits in a file without the disk reporting an error. The defense is to hash the object as you read it and compare against the known-good digest; a mismatch means the bytes are bad, and you return an error instead of serving corruption. Hashing a large object is expensive, but it is the difference between integrity and silent data loss.\nThe trouble is that we had applied the defense uniformly, with no memory. Every read paid the full cost, even the hundredth read of the same immutable file in the same process. In a read-heavy workload, nearly every request re-derived a digest whose answer could not have changed since the last time we computed it.\nWhy the Obvious Solutions Failed \u0026ldquo;Add a toggle to disable verify-on-read.\u0026rdquo; This is the tempting fix and exactly the wrong one. Turning verification off converts silent bitrot into silent data corruption — the worst outcome in a storage system. Any solution had to keep integrity intact.\n\u0026ldquo;Cache the digest on disk.\u0026rdquo; Persisting trust markers across restarts is worse than not having them: a marker written before the disk rotted would decline to re-check after the rot, and a restart would carry stale trust forward. Trust must not survive a process restart.\n\u0026ldquo;Only verify small objects.\u0026rdquo; Size is not the risk. A corrupted 4 KB block can hide in a huge file; skipping verification for large objects is backwards.\nSo we set three non-negotiable invariants:\nZero compromise on cold reads. The first time a blob is read in a server\u0026rsquo;s life, it is fully verified. Zero persisted trust. Trust markers live only in memory and vanish on restart, forcing a fresh verification pass. Continuous background detection. Independent scrubbers keep auditing disk blocks regardless of read traffic. The Solution: Dynamically Earned Trust We introduced a seam — a compile-time interface marking the one place the storage engine is allowed to hook into verification — called ReadVerifier:\n1 2 3 4 5 6 7 8 type ReadVerifier interface { // Verify returns an io.ReadCloser that verifies hash at EOF, // or returns the underlying reader directly if already trusted. Verify(underlying io.ReadCloser, hash string) io.ReadCloser // MarkTrusted records that this blob has passed a full SHA-256 verification. MarkTrusted(hash string) } Two implementations ship compiled in:\neveryReadVerifier (default). The historical behavior: every read is wrapped in a verifying reader that hashes to end-of-file. verifiedCache (opt-in). Remembers, per process, which blobs have already passed a full verification, and skips re-hashing them: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 type verifiedCache struct { mu sync.RWMutex trusted map[string]struct{} } func (v *verifiedCache) Verify(underlying io.ReadCloser, hash string) io.ReadCloser { v.mu.RLock() _, ok := v.trusted[hash] v.mu.RUnlock() if ok { // Blob was already verified during this process lifecycle. // Return raw stream directly — 0 hashing overhead! return underlying } // Wrap in verifyingReader with callback to record trust on clean EOF vr := newVerifyingReader(underlying, hash) vr.onVerified = func() { v.MarkTrusted(hash) } return \u0026amp;verifyingReadCloser{verifyingReader: vr, underlying: underlying} } func (v *verifiedCache) MarkTrusted(hash string) { v.mu.Lock() v.trusted[hash] = struct{}{} v.mu.Unlock() } The key word is earned. A blob is trusted only after it has actually been verified end to end in this process. The first read pays the full hash cost; every subsequent warm read streams bytes straight to the socket with no hashing at all.\nThe read path branches on that trust cache:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 ======================================================================================== DYNAMICALLY EARNED TRUST READ PATH ======================================================================================== Client GET /file.bin | v [ CASStore.Get() ] | v Check Trust Cache: IsTrusted(hash)? | +---------------------------------------+ | | [ NO: First Read / Cold ] [ YES: Previously Verified ] | | v v [ verifyingReader ] [ Raw LocalBlobStore Stream ] - Stream bytes to client - Stream bytes directly to socket - Stream bytes to SHA-256 - Zero hashing CPU overhead | | v v At io.EOF: Digest Matches? Max NVMe Read Throughput (~2400 MB/s) | +---\u0026gt; [ YES ] -\u0026gt; MarkTrusted(hash) | `---\u0026gt; [ NO ] -\u0026gt; Return syscall.EBADMSG The Trade-off: Speed vs. the Bitrot Window Dimension everyReadVerifier verifiedCache Cold Read Throughput ~313 MB/s (CPU-bound) ~313 MB/s (CPU-bound) Warm Read Throughput ~313 MB/s (CPU-bound) ~2400 MB/s (disk-bound, 7.6x faster) Memory Footprint zero bytes of state roughly 64 bytes per unique active blob in RAM Bitrot Window Instant detection on every read Detected on cold read; later rot caught by background scrub Allocations per Op Scales with chunk count Constant 11 allocations/op (hasher omitted on warm reads) The honest cost is the bitrot window. Here is exactly what it means: if a disk sector rots after the first read has marked the blob trusted, verifiedCache will serve the bad bytes until the next background scrub pass notices. That window is bounded — the scrubber periodically scans all stored blobs, bypassing the read cache, and on a mismatch it purges the blob from the trust cache, quarantines it, and triggers a self-heal rebuild from a healthy replica.\nWe accepted a bounded detection delay on the rare post-first-read corruption in exchange for removing a hashing tax from every warm read. The alternative — a permanent 8x read penalty for a defense that mostly re-proves unchanged files — was worse.\nHow We Verified Benchmarks confirmed the improvement:\n1 2 BenchmarkReadVerify/1MiB-4 1000 3,189,451 ns/op 313.52 MB/s 28 allocs/op BenchmarkTrustedBlobRead/1MiB-4 10000 416,210 ns/op 2402.63 MB/s 11 allocs/op Throughput: warm reads went from ~313 MB/s to over 2400 MB/s. Allocation invariant: the trusted read path holds a constant 11 allocations per operation regardless of payload size — 1 MiB, 64 MiB, or 256 MiB — because the crypto engine is simply absent from the warm path. See docs/CORE/STANDARDS.md for the Bolt and Sentinel engineering principles.\nWhat Could Go Wrong Trust is per-process, by design. A restart wipes the cache and every blob is re-verified on its first read after boot. That is desirable, and it means a restart temporarily loses the warm-read speedup. A hash collision in the trusted set. The set is keyed by object digest, so two objects with the same hash would share trust. SHA-256 makes this a theoretical non-issue, but the design depends on the digest being the identity. Concurrency. The trust map is guarded by a read-write mutex; writes are rare (once per blob per process), reads are frequent. A scrubbing gap. If the background scrubber stops running, the bitrot window stops being bounded. Scrub is part of the guarantee, not decoration. When NOT to Use This Short-lived or one-shot processes. If a process reads each blob once and exits, there are no warm reads to speed up and the cache only adds bookkeeping. When instant detection is mandatory. Systems that must detect corruption on the very read where it occurs should keep the always-verify behavior. As a replacement for scrubbing. This optimization assumes an independent scrubber bounds the window; remove the scrubber and you have re-introduced silent corruption. References / Dig deeper Storage integrity and GC: 007. Profiling baseline: 042. Central integrity verification: 031. Seams architecture: 044. Implementation: src/storage/read_verifier.go; benchmarks: src/storage/bench_test.go. Spec: openspec/changes/reduce-read-verify-hashing; PR #951; issue #950. ","permalink":"/posts/043-reduce-read-verify-hashing/","summary":"Verify-on-read re-hashes the full object on every local read (~8x slower than write). A ReadVerifier seam skips re-verification for blobs verified this process, while scrub still catches bitrot.","title":"Reducing Read-Verify Hashing: Trust Earned by Real Verification"},{"content":"You cannot optimize what you cannot measure, and you cannot tell a real win from a lucky benchmark run without a reference point. So before we changed a single line of performance code, we built the measurement harness — and in the process settled a security question that will outlive every optimization.\nA profile is a sampled record of where a program spends its time or memory. Go ships a profiler called pprof that can capture a CPU profile (which functions consumed processor time) and a memory profile (where allocations happened). A baseline is the profile you take before optimizing, so every later change can be compared against it.\nThe Real Problem We had opinions about what was slow. We did not have evidence. Worse, the obvious way to get evidence — expose the profiler over HTTP so you can pull a live profile from a running server — is exactly the wrong move for us.\nmomo\u0026rsquo;s data-path server has no authentication and no TLS. A live profiler endpoint on such a server is a remote-code-execution-class surface: goroutine dumps leak internal state, an attacker who can steer what gets profiled can steer memory behavior, and the trace endpoints can crash the process. Convenience was not worth that.\nThe Solution: Profile to a File, Never to the Wire The harness uses Go\u0026rsquo;s built-in, file-based profilers — no custom tooling, no daemons, no listening socket:\n1 go test -cpuprofile cpu.pprof -memprofile mem.pprof -bench . ./src/storage/... The benchmark segments cover the real hot paths: content hashing, local writes, verify-on-read, and S3 spooling. The important property is that the profile is a file, produced by the test process, not a listener on a server.\nThe security rule that came out of this is simple and absolute: profile to disk, never to the wire.\ngo test -cpuprofile X -memprofile X -blockprofile X writes profile files. No HTTP debug listener on the data path, ever. Any future admin endpoint would be loopback- or Unix-socket-only, enabled at boot, and TLS-protected the moment it leaves the loopback interface. What the Baseline Found The first real profile was a surprise that saved real work. Every cycle of hashing CPU was already inside sha256.blockAVX2 — the standard library\u0026rsquo;s SHA-256 implementation already uses AVX2, the SIMD (single-instruction, multiple-data) vector instructions modern x86-64 CPUs provide. Our naive plan had been to \u0026ldquo;swap in a fast SIMD SHA-256.\u0026rdquo; The baseline retired that idea before it became a pull request, because there was almost nothing left to win on this hardware.\nThat is the whole case for measuring first: the baseline did not just guide an optimization, it cancelled one.\n⚡ Bolt / 🛡 Sentinel Lens ⚡ Bolt: measure, then optimize. The baseline is the reference every later performance change is judged against by the benchstat gate. 🛡 Sentinel: no profiler on the wire — fail closed rather than expose a remote-code-execution-class surface for convenience.\nSee docs/CORE/STANDARDS.md.\nReferences / Dig deeper Mindset standards: docs/CORE/STANDARDS.md. The benchstat gate that consumes this baseline: 025. Bolt engineering overview: 024. A Bolt optimization that shipped: 036. Follow-on work: 043, 044. The profile-baseline spec: openspec/changes/perf-profiling-baseline; tracking issue #948. The \u0026ldquo;no networked profiler\u0026rdquo; rule is recorded as Rule 75 in the steering rules. ","permalink":"/posts/042-perf-profiling-baseline/","summary":"Phase-0 profiling harness via go test -cpuprofile/-memprofile; Rule 75 forbids networked pprof on unauthenticated listeners. The baseline proved stdlib SHA-256 already uses AVX2.","title":"Perf Profiling Baseline: Measure Before You Optimize — and Keep pprof Off the Wire"},{"content":"\u0026ldquo;Should everything be a plugin?\u0026rdquo; The question arrives whenever a codebase wants adaptive, mutating behavior — the ability to swap in a new read-verification policy, a new replication strategy, or a new filesystem policy without a rebuild. For momo the answer was no, and the distinction matters.\nThe vocabulary first. A plugin in the dynamic sense is code loaded at runtime — a shared object, a separate process, or a remote procedure call — that the host did not compile and cannot fully audit. A seam is the opposite: a small Go interface compiled into the binary at a known decision point, where the concrete implementation is chosen by configuration. The fast path is the common, happy case that must stay lean; the trust core is the set of invariants (content addressing, placement, integrity) that must never be swappable. A policy is declarative data that selects behavior; fail closed means an unknown or absent policy falls back to the safe default.\nThe Real Problem We wanted adaptability without paying for it in latency or trust. Two forces pushed in opposite directions. The data path is performance-critical: an RPC hop per decision would break the byte flow and add allocation to every transfer. And the storage node is security-critical: loading unreviewed code onto a node that holds other people\u0026rsquo;s data is a Trojan surface, not a feature.\nWhy the Obvious Solutions Failed External dynamic plugins In-process seams (chosen) Loading shared objects, subprocesses, cross-process RPC compiled-in Go interface Fast path serialization and allocation tax concrete, zero-indirection Trust executing unreviewed code compile-time auditable Versioning pinning pain across processes a single binary Dynamic plugins are the right answer when the extension authors are unknown and the host must isolate them — think a browser extension or an editor. They are the wrong answer for a storage node whose entire value proposition is predictable performance and auditable behavior.\nFigure 1: Core storage invariants remain concrete while seams provide clean decision points The constraint that decided it: seam over the changeable, keep the fast path concrete. Seams dispatch only at decision points, never inside the byte stream.\nFigure 2: In-process interface dispatch vs out-of-process dynamic RPC plugins The Rule We codified the architecture as a steering rule with five commitments:\nAdaptive, mutating behaviors are seams — Go interfaces such as a read verifier, a rebuild converger, a filesystem policy, a replication strategy, and a durability barrier. Fast paths stay concrete — seams dispatch at decision points only, never per byte. Trust-core invariants are pinned — content addressing, content hashing, placement, verify-on-read, and the single validate-then-write chokepoint stay compiled into the auditable core. Policy is declarative — behavior is selected by swapping a policy struct through an atomic pointer at runtime, never by mutating code. Fail closed — an unknown or absent strategy falls back to the safe default rather than guessing. Figure 3: Comprehensive tradeoff comparison between seams and dynamic plugins Examples Already in the Tree The read verifier seam — trust-earned read verification, where a blob that has proven itself is verified less often. The durability barrier seam — fsync, group commit, or no barrier behind one interface. The checksum provider seam — protocol-agnostic integrity verification, so every transport reports expectations the same way. ⚡ Bolt / 🛡 Sentinel lens ⚡ Bolt: concrete fast path, zero indirection, no RPC in the data plane. 🛡 Sentinel: fail-closed policy, no dynamic code loading, the trust core pinned and auditable. Out-of-process code is allowed only as a read-only policy or control-plane feed, never in the compute plane.\nSee docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Spec: openspec/changes/plugin-seam-architecture. Design doc: docs/momofs/PLUGIN_ARCHITECTURE.md. Steering rule: openspec/config.yaml (Rule 74). Sibling posts: adaptive loops 018, integrity seam 031, read-verify seam 043, performance discipline 042, degraded-read/self-heal 020. ","permalink":"/posts/044-plugin-seam-architecture/","summary":"Adaptive behaviors use compile-time Go interface seams injected at decision points — not dynamic plugins. The fast path stays concrete; the trust core stays auditable; policy is declarative.","title":"Seams, Not Plugins: A Fast Path That Stays Concrete"},{"content":"Momo was designed as a \u0026ldquo;metrics-driven controller\u0026rdquo; — a cluster that changes its own replication topology in response to live telemetry. That made it embarrassing when the cluster could not tell us how it was doing. We had log lines, but no numbers: no way to see at a glance how many blobs we stored, how much disk was left, or whether the peer mesh was healthy. This is the story of making observability a first-class part of the daemon instead of an afterthought.\nA short vocabulary check first, because the rest of the post leans on it. Prometheus is a monitoring system that periodically scrapes a plain-text /metrics endpoint on each server and stores the values as time series. A gauge is a metric that can rise and fall (free disk bytes, active peers); a counter only ever climbs (total bytes replicated); a histogram sorts observations into buckets so we can compute percentiles such as p99. Grafana is the dashboard tool we point at Prometheus to draw the charts.\nThe Real Problem Our first exporter answered \u0026ldquo;is the process up?\u0026rdquo; and little else. In a multi-node ring, every operator scraped node 0, because that was the only address wired into the config. Node 0\u0026rsquo;s counters were a summary, not the truth: they could not tell you which peer was sick, which disk was filling, or whether replication was falling behind on a specific node. When something went wrong we were reduced to grepping logs and guessing.\nWhy the Obvious Solutions Failed \u0026ldquo;Scrape node 0 and be done.\u0026rdquo; Centralizing the scrape was simple, but it made node 0 a single point of both failure and confusion. A metric that aggregates across nodes hides the one node that is actually broken — exactly the node you need to see.\n\u0026ldquo;Expose the standard Go runtime debug endpoint.\u0026rdquo; Go ships an endpoint that dumps goroutine stacks and memory internals. It is wonderful for local debugging and dangerous on a network: it leaks runtime internals to anyone who can reach the port. We wanted metrics, not a live introspection surface.\n\u0026ldquo;Compute the expensive numbers on every request.\u0026rdquo; Reading the object index and calling the filesystem for disk usage are cheap once, but ruinous if done per request. Observability must not tax the data path it observes.\nThe Solution Two moves, one architectural and one about where work happens.\nPer-node binding. Each daemon now gets its own metrics address — a metrics_host and metrics_port in its configuration — so an operator (or Prometheus itself, via service discovery) can scrape every server individually. The ring becomes observable node by node instead of through one aggregate window. The listener is a dedicated, config-explicit port: it is never the runtime debug endpoint, so it cannot be used to dump goroutines or process internals. If we ever ship admin profiling, it will be loopback-only or a Unix socket, gated behind mutual TLS.\nScrape-time collection. Heavy reads happen only when /metrics is fetched, not on the hot path. Storage numbers come from a single pass over the embedded key/value store plus one filesystem statfs call. The host name, which is constant for the process lifetime, is cached once instead of looked up on every scrape.\nThe metrics themselves fall into three families:\nStorage / content-addressed store — blob count, stored bytes, disk used/free, and garbage-collection runs and evicted bytes. Eviction on a zero-reference delete is immediate, so a deleted blob cannot linger as an orphan. Replication and peer mesh — replicated bytes, replication failures, peer counts split into alive / suspect / offline, ping round-trip latency (an exponentially weighted moving average), active leases, and scatter-gather query timeouts. SWIM is the gossip protocol the cluster uses to detect failed peers; the alive/suspect/offline gauges are its view of the world. Latency histograms (opt-in) — request and replication latency buckets, armed only when enable_latency_histograms is set to true. When the flag is off, the timing code is never reached: no clock read, no bucket increment, no allocation. Pattern: Scrape-Time Aggregation for Expensive Metrics Compute metrics that require heavy reads (index scans, filesystem stats) only when the metrics endpoint is scraped, not on every request. Cache process-constant values at startup.\nApplies when: a metric\u0026rsquo;s source is expensive to read but changes slowly relative to traffic. Doesn\u0026rsquo;t apply: counters that must capture every event in real time — those belong on the hot path as atomic increments.\nHow We Verified The design goal was \u0026ldquo;observable without being felt.\u0026rdquo; The latency histograms use plain atomic integer increments rather than a floating-point metrics library, so the disabled path costs nothing and the enabled path is a handful of nanoseconds. Storage gauges are bounded by a single store read and one statfs. We watched scrape latency under concurrent upload load to confirm the endpoint stays responsive, and kept the numbers alongside the rest of the performance history.\nFailure Modes Risk Guard Metrics endpoint becomes an attack surface Dedicated listener; no runtime/goroutine dump; profiling would be loopback + mTLS Scraping every node floods the network Scrape interval is the operator\u0026rsquo;s choice; each response is a single bounded read Histograms silently tax throughput Default off; when on, atomic increments only A node\u0026rsquo;s metrics go stale Per-node binding means each node reports for itself — no proxy to lie When NOT to Use This Single-node deployments do not need per-node binding; one address is fine. Debugging a live process is not a metrics job — use a profiler on a loopback or Unix-socket endpoint, not the metrics port. Sub-millisecond, per-event latency tracing is better served by distributed tracing than by histogram buckets. See docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Specs: openspec/changes/add-metrics-exporter, openspec/changes/metrics-per-node-binding, openspec/changes/r5-metrics-p2. Performance history: docs/REFERENCE/PERFORMANCE.md. Sibling posts: controller origin 002, performance arc 024, security audit 015, R5 phases 2–4 032. ","permalink":"/posts/026-metrics-observability/","summary":"Prometheus metrics export with a per-node metrics_host/metrics_port bind — observability that scales past the \u0026lsquo;just scrape node 0\u0026rsquo; era.","title":"Metrics and Observability: Per-Node Bind, Prometheus Export"},{"content":"Every system that encrypts data eventually faces the same operational question: where does the encryption key live, and how do you change it without taking the service down? The naive answers — a hard-coded key, or a single central vault every node must reach — either cannot rotate or reintroduce the single point of failure the rest of the system was built to avoid. This post explains how momo resolves that question.\nThe Real Problem Three operational requirements collide:\nThe key must be reachable by every node, always. If node B cannot reach the key store, it cannot serve encrypted data — even though the data is right there on its disk. The key must be rotatable without downtime. Rotation is a security hygiene task: a key may be rotated because of policy, or because it is suspected compromised. Either way, in-flight and at-rest data must keep working through the change. The key must not become the new single point of failure. A central secret server is the obvious place to put the key, and the obvious thing to go down at 3 a.m. Why the Obvious Solutions Failed A key in the config file, forever. This is the simplest option and it works — until you need to rotate. There is no versioning, so the moment you change the key, every object encrypted under the old key becomes unreadable. Rotation is impossible without a migration that re-encrypts everything.\nA single central vault. This solves \u0026ldquo;where is the key\u0026rdquo; but not \u0026ldquo;what if the vault is down\u0026rdquo;. Every read now depends on a network round trip to one service. That is a single point of failure by construction, and it turns a local read into a distributed one.\nRotate by re-encrypting everything. Re-encrypting the whole dataset on every rotation is correct but ruinous at scale, and it needs a maintenance window. The right design makes rotation a metadata change, not a data change.\nThe Solution with Annotated Code A provider chain instead of a single source Secret lookup is a chain of providers tried in order. The common case (a key in the local config file, or an environment variable injected by the orchestrator) is answered locally; a remote provider can be added later without changing any caller.\n1 2 3 // BuildProviderChain tries providers left to right and returns the first hit. // The default chain is \u0026#34;env, momo\u0026#34; (environment, then the config file). chain, _ := BuildProviderChain(SecretsConfig{Sources: []SecretsSource{SecretsSourceEnv, SecretsSourceFile}}) The chain is a seam, not a plugin system: each provider is a compile-time implementation of one small interface, so there is no dynamic loading and no new attack surface in the data path.\nA per-node versioned key registry Rather than a single key, each node keeps a registry of key versions in its own local storage. A key has a status — active, pending_rotation, retired, or compromised — and an identifier. Encrypted data records which key version it used, so decryption is always possible as long as the version is retained.\n1 2 3 4 // Rotation stores the new version, then activates it. The previous active key // is demoted to retired — but NOT deleted, so old ciphertext still decrypts. newID, _ := registry.StoreKey(ctx, \u0026#34;encryption\u0026#34;, material, \u0026#34;AES-256-GCM\u0026#34;, \u0026#34;\u0026#34;, KeyStatusPendingRotation, 0) registry.SetActive(ctx, newID) Because the registry lives on each node, any node can resolve the active key locally. There is no fetch step to fail, and no central server to restate the whole system\u0026rsquo;s vulnerability.\nRotation as a metadata change Rotation generates new material, stores it as a new version, and promotes it. No data is re-encrypted. Dependent components are told to re-read the registry via a reload signal (an HTTP endpoint and a SIGHUP handler).\n1 2 3 // The old key stays retrievable; new writes use the new active key. The grace // period is the retention window during which both are valid. rm.RotateAll(ctx) // rotates every purpose: encryption, auth, e2ee, oprf Principle: A secret you can derive or reconstruct is worth more than a secret you must fetch. Put the derivable material on every node and keep the single source of truth out of the request path.\nVerification Provider chain proves a miss in the first provider falls through to the next, and that an all-miss returns \u0026ldquo;not found\u0026rdquo; rather than an error. Rotation audit proves every rotation writes a chained audit entry with the old and new key IDs. Old-key retention proves that after rotation the previous key is still retrievable (for decryption) while the active key has changed. Failure Modes A provider that is configured but unreachable fails the chain: if the first provider errors rather than reporting \u0026ldquo;not found\u0026rdquo;, the lookup stops. Order the chain so the always-local source is first. Retaining retired keys forever preserves decryptability but enlarges the key registry. A retention policy must eventually drop keys whose ciphertext is gone — which means those objects can never be decrypted again. Rotation without reload leaves running components using the old key until restarted. The reload signal exists precisely to avoid that. When NOT to Use If the dataset is small and a maintenance window is acceptable, rotating by re-encrypting under one new key is simpler to reason about than a versioned registry. And if a managed KMS is a hard requirement, this design is the wrong shape — it deliberately trades external-KMS integration for zero-SPOF local resolution.\nReferences / Dig deeper Secrets management and key rotation specification: openspec/changes/r9-secrets Provider chain and providers: src/common/secrets.go Versioned key registry: src/common/keyregistry.go Rotation manager and audit: src/common/rotation.go Tracking issue: https://github.com/alsotoes/momo/issues/937 Related: R10 Phase 3 (Notification, 055) ","permalink":"/posts/054-secrets-management-key-rotation/","summary":"How momo manages encryption keys without a central secret server: a pluggable provider chain (env, then config file), a per-node versioned key registry in local storage, and rotation that promotes a new key without any downtime. Teaches: why a provider chain beats a single source, how key versioning enables online rotation, and when rotation is and is not worth the complexity.","title":"Rotating Secrets Without a Secret Server: Deriving, Versioning, and Hot-Reloading Keys"},{"content":"A shared object store has a trust problem the moment two organizations put data in it: organization A must never read, overwrite, or even enumerate organization B\u0026rsquo;s objects, and when something goes wrong, both parties must be able to prove which operations actually happened. That is the multi-tenancy problem, and it is easier to get subtly wrong than it looks. This post explains how momo isolates tenants, why the obvious shortcuts fail, and what makes its audit trail trustworthy.\nThe Real Problem Three requirements have to hold at the same time:\nConfidentiality and integrity across tenants. Tenant A\u0026rsquo;s key must not open tenant B\u0026rsquo;s data, and compressing or deduplicating identical content across tenants must not leak the fact that the content is identical. An authorization decision for every request. Reads, writes, deletes, and listings all need to be checked, not just the \u0026ldquo;dangerous\u0026rdquo; ones. A trustworthy record. When a dispute arises, the operator must be able to show what happened, and prove that the record was not edited after the fact. Doing all three centrally — one key server, one policy database, one log server — reintroduces exactly the single point of failure the rest of the system was built to avoid.\nWhy the Obvious Solutions Failed One shared encryption key with a tenant column in the metadata. This is the most common shortcut. It fails the confidentiality requirement the moment the metadata database is read by the wrong process, and it makes per-tenant key rotation impossible without re-encrypting everyone.\nDeriving keys by storing them. The second shortcut is to generate a random key per tenant and store it (encrypted) next to the tenant record. Now the tenant record is a single high-value secret, and every node that mints a tenant key needs a round trip to fetch it. That is a dependency on a reachable secret store — a single point of contention, if not failure.\nChecking authorization only on writes. Reads leak as much as writes. Listing all objects is a read, and if it is unchecked, a tenant learns the shape of the whole cluster. A partial policy is a broken policy.\nAn audit log that is just an append. An append-only file is easy to write and easy to edit. Without a mechanism that binds each entry to the previous one, an attacker with storage access can delete or rewrite entries and the log looks perfectly normal.\nThe Solution with Annotated Code Per-tenant keys by derivation, not storage Instead of storing a key per tenant, momo derives it from the root key, using the tenant ID as domain-separation input. The root key is the only secret that must exist; every tenant key is a pure function of it.\n1 2 3 4 // Each tenant gets its own KEK, OPRF share, and auth token, all derived from // the root material with the tenant ID as the HKDF \u0026#34;info\u0026#34; (domain separation). kek := HKDF(rootKEK, info=\u0026#34;tenant/\u0026#34;+tenantID+\u0026#34;/kek\u0026#34;) share := HKDF(rootShare, info=\u0026#34;tenant/\u0026#34;+tenantID+\u0026#34;/oprf\u0026#34;) Because derivation is deterministic, any node can reconstruct a tenant\u0026rsquo;s key locally from the root material it already holds. There is no tenant-key fetch step, so there is no tenant-key round trip to fail. The tenant\u0026rsquo;s KEK is then wrapped under the root key for storage:\n1 2 3 // WrapTenantKEK seals the tenant KEK with AES-256-GCM using the tenant ID as // additional authenticated data, binding the ciphertext to its tenant. wrapped, _ := WrapTenantKEK(rootKEK, tenantKEK, tenantID) Binding the tenant ID as associated data matters: a wrapped key cannot be \u0026ldquo;moved\u0026rdquo; to another tenant without the unwrap failing, because the associated data will not match.\nDefault-deny ACL evaluation Authorization is evaluated against explicit entries, and the default when nothing matches is deny. Deny always wins over allow.\n1 2 3 4 5 6 7 8 9 func checkACL(entry *ACLEntry, tenantID string, perm Permission) bool { if entry.TenantID != tenantID { return false // not this tenant\u0026#39;s rule } if !entry.Permission.Has(perm) { return false // rule does not cover this permission } return entry.Effect == \u0026#34;allow\u0026#34; } The order of evaluation is the security-critical part. A request is allowed only if an explicit allow matches; an explicit deny short-circuits to rejection. There is no \u0026ldquo;no rule, so allow\u0026rdquo; branch — the absence of a rule is a denial.\nMaking the audit log tamper-evident Each entry stores the hash of the entry before it. Editing any entry breaks the chain from that point forward, so verification is a single walk.\n1 2 3 4 // entry.PrevHash is the EntryHash of the previous entry. The first entry // chains from the empty hash, giving the chain a fixed genesis. entry.PrevHash = lastEntryHash entry.EntryHash = SHA256(canonical(entry)) Verification recomputes every hash and checks that each entry\u0026rsquo;s PrevHash equals its predecessor\u0026rsquo;s EntryHash. A single altered byte anywhere makes the chain diverge.\nPrinciple: Derive secrets you can reconstruct; store secrets you cannot. Every piece of state a node must fetch to do its job is a piece of state that can be unavailable at the worst moment.\nVerification The design is exercised by three tests that correspond to the three requirements:\nKey derivation proves two tenants never produce the same key, and that wrap/unwrap round-trips and rotation preserve decryptability. Authorization proves an allow entry admits the matching tenant while a missing entry, a different tenant, or an explicit deny rejects. Audit chaining proves a valid chain verifies, and that mutating an entry causes verification to fail. Failure Modes Root key compromise defeats tenant isolation. Derivation is only as strong as the root. Rotating the root key re-derives every tenant key, which is the intended recovery path. ACL misconfiguration is fail-closed but can lock out a tenant. A missing allow denies, which is safe but visible as access denials rather than silent access. Audit chaining detects tampering but does not prevent it. A party with write access can truncate the log from the end. Detecting truncation requires an external witness (publishing the latest hash periodically). When NOT to Use If every client is fully trusted and there is only one tenant, the per-tenant key derivation and ACL machinery add overhead and configuration for no security benefit. Run single-tenant with encryption-at-rest and skip the tenant layer. The audit log is still worth enabling for its operational value.\nReferences / Dig deeper Multi-tenancy, authorization, and audit specification: openspec/changes/r8-multitenancy-authz Per-tenant key hierarchy: src/crypto/crypto.go ACL storage and evaluation: src/storage/storage.go Audit chaining: src/storage/storage.go (WriteAuditLog, VerifyAuditLog) Tracking issue: https://github.com/alsotoes/momo/issues/936 ","permalink":"/posts/053-multitenancy-authorization-audit/","summary":"How momo isolates tenants without a central secret server: HKDF-derived per-tenant keys, default-deny ACLs evaluated per request, and a SHA-256 hash-chained audit log that makes tampering detectable. Teaches: key derivation vs. key storage, why ACL evaluation order matters, and how hash chaining turns a log into evidence.","title":"Tenant Isolation: Deriving Per-Tenant Keys, Enforcing ACLs, and Proving What Happened"},{"content":"A monitoring endpoint is only useful if people trust it. The first phase of our metrics work wired a Prometheus exporter into the daemon; this is the follow-on that made the numbers worth looking at: what the storage engine is holding, whether the peer mesh is healthy, and — optionally — how slow our requests actually are. The constraint on all of it was the same: observability must not cost measurable throughput.\nA quick recap of the metric types, since they drive every design choice below. A gauge is a value that can go up or down (disk free, active peers). A counter only increases (total bytes replicated). A histogram groups observations into buckets so you can compute percentiles. Prometheus scrapes these from a /metrics endpoint on an interval. bbolt is the embedded key/value store that holds our object index; statfs is the operating-system call that reports filesystem space. Deduplication means storing identical content once and pointing many names at it; CAS is the content-addressed store that makes that possible.\nThe Real Problem Phase 1 told us the process was alive. It did not tell us the things an operator actually asks at 3 a.m.: is the disk filling? Are peers dropping out of the mesh? Is replication stalling? Are our reads getting slower? Answering those required either expensive per-request bookkeeping (which would slow the very requests we were measuring) or nothing at all.\nWhy the Obvious Solutions Failed \u0026ldquo;Instrument everything on the hot path.\u0026rdquo; A clock read and a bucket increment on every request sounds harmless until you multiply it by every request. The latency we added to measure latency would corrupt the measurement.\n\u0026ldquo;Pull in the standard Prometheus client library.\u0026rdquo; It is convenient, but its floating-point counters and per-metric locking cost far more than the atomic integer increments our access patterns need. We also did not want an external dependency in the trust core.\n\u0026ldquo;Only measure storage; ignore the cluster.\u0026rdquo; A healthy disk on a node that has been isolated from its peers is not a healthy node. Storage numbers without peer-health numbers tell half a story.\nThe Solution Three phases, each with the same discipline: do the expensive work at scrape time, and use lock-free atomics everywhere else.\nPhase 2 — Storage metrics, computed only when scraped These read the object index and the filesystem at scrape time and never touch the request path:\nMetric Type Source momo_blob_count Gauge object index momo_stored_bytes_total Counter sum of object sizes momo_disk_used_bytes Gauge statfs momo_disk_free_bytes Gauge statfs momo_gc_runs_total Counter GC loop iterations momo_gc_evicted_total Counter evicted objects momo_dedup_hits_total Counter CAS deduplication hits Phase 3 — Peer mesh and replication These are read from live subsystem state at scrape time, with no locks and no allocations:\nMetric Type Source momo_cluster_peers Gauge alive peer count momo_swim_alive_count Gauge peers seen as alive momo_swim_suspect_count Gauge peers under suspicion momo_swim_offline_count Gauge peers declared offline momo_swim_ping_latency_seconds Histogram EWMA round-trip samples momo_leases_active Gauge lease manager momo_scatter_queries_total Counter scatter-gather dispatches momo_scatter_timeouts_total Counter query timeouts momo_replication_total Counter replication transfers momo_replication_bytes_total Counter bytes replicated SWIM is the gossip protocol peers use to detect failure; EWMA is an exponentially weighted moving average, which smooths round-trip samples so a single slow ping does not distort the picture. Scatter-gather is a query broadcast to many peers whose partial answers are merged.\nPhase 4 — Latency histograms, off by default Latency tracking is gated behind a single configuration flag:\n1 2 [metrics] enable_latency_histograms = true When it is off — the default — the timing code is never reached: no clock read, no bucket increment, no allocation. When it is on, requests and replication record into fixed-bucket histograms labeled by operation, using atomic increments of roughly a few tens of nanoseconds per observation.\nOverhead Guarantees Guarantee Mechanism Under 1% throughput regression atomic increments on plain integers; no locks, no allocations Heavy work never on the hot path memory stats, statfs, and index reads only at scrape No interference with request serving metrics run on their own goroutine and port No heavyweight dependency pure atomics instead of a floating-point metrics library How We Verified We validated the overhead claim three ways: a benchmark suite comparing nanoseconds-per-op before and after; a load test ramping virtual users and watching throughput; and scrape-latency measurement under a thousand concurrent uploads, where the endpoint stayed responsive. Memory growth stayed in the low single-digit megabytes.\nFailure Modes Risk Guard Histogram overhead silently enabled Default off; the disabled path is dead code at runtime Scrape blocks request handling Separate listener and goroutine A metric reads a subsystem mid-mutation Values are read from atomic state at scrape, not locked mid-write Dashboards drift from the metrics Panels and alerts are maintained alongside the metric definitions Follow-ups Grafana dashboards gained storage, peer, and replication panels, and Prometheus alert rules cover disk pressure, peer loss, and replication stalls. The next phase looks at custom bucket boundaries, exemplars, and a bridge to OpenTelemetry for distributed tracing.\nStandards See docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Spec: openspec/changes/r5-metrics-p2. Sibling posts: 026, peer gossip 016, scatter-gather and leases 017, performance engineering 024. Performance history: docs/REFERENCE/PERFORMANCE.md. ","permalink":"/posts/032-r5-metrics-phases-2-4/","summary":"R5 Phases 2-4 shipped: storage metrics (disk, CAS, GC), P2P metrics (SWIM, leases, scatter-gather), and opt-in latency histograms — all with \u0026lt;1% overhead via sync/atomic counters.","title":"R5 Metrics Phases 2-4: Storage, P2P, and Latency Histograms"},{"content":"Every system that wants to be taken seriously has to answer a boring question: what is left before someone can run this in production? For a long time our answer was a mental list. This post is the moment we wrote the list down, ordered it, and started checking items off — and the reading list that fed the ideas in the first place.\nWhere the arc stands Momo roadmap R1-R6 The P0 track — correctness and durability — is done. We worked through it in order, each rung depending on the last:\nR1 — failure domains: place replicas so that one rack or machine going dark does not take every copy of an object with it. R2 — degraded reads and self-heal: when a replica is missing, serve from what remains and repair the gap in the background instead of failing the read. R3 — write durability and quorum: a write is acknowledged only once enough replicas hold it, so an acknowledged write survives a node loss. R4 — mountable POSIX filesystem: expose the object store as a normal filesystem mount, first in userspace, then over a faster transport. With the correctness floor in place, the remaining work splits into two tracks: P1 (operability, multi-tenancy, security) and P2 (S3 API breadth).\nID Item R5 Metrics 2–4 + dashboards/alerts (seed: 026) R6 Metadata catalog HA + backup/recovery R7 Error model \u0026amp; ops (out-of-space surfacing, exit codes) R8 Multi-tenancy + authorization + audit R9 Secrets management + key rotation R10 S3 lifecycle/versioning/notification/lock breadth R11 Auto-rebalance on membership change Some of these deserve a gloss. Metadata catalog HA means the index that maps object names to locations must survive a node loss and be restorable from backup. Out-of-space surfacing means turning a raw disk-full error into a clear, actionable signal instead of a mysterious failure. Auto-rebalance means that when a node joins or leaves, the cluster moves data to restore the placement guarantees without an operator running a script by hand.\nThe research guide None of this was invented in a vacuum. The design leans on published work: quorum theory (when is a write durable?), SWIM/gossip (how do peers detect failure without a central monitor?), CRUSH/RADOS (how do you place data deterministically without a directory?), private set membership using an oblivious pseudorandom function (how do you check membership without revealing the query?), and filesystem semantics for the mount layer. We keep a curated reading list so the \u0026ldquo;why\u0026rdquo; behind each mechanism is one hop away.\nOur durable lens, stated once: code is the implementation; research is the why; specs are the contract; posts are the narrative. Each artifact has a job, and this journal is the one that explains rather than specifies.\nReferences / Dig deeper Roadmap: docs/GUIDES/ROADMAP.md. Research reading list: docs/RESEARCH_PAPERS.md. Spec: openspec/changes/prod-ready-roadmap. Completed P0 stack: 019, 020, 021, 022, 023. ","permalink":"/posts/028-roadmap-and-research/","summary":"After R1–R4: the P0 hardening done, the P1/P2 tracks (metrics, HA metadata, auth, secrets, S3 breadth) and the research reading guide that seeded it all.","title":"Forward: Production Roadmap (R5–R11) and the Research Guide"},{"content":"Two earlier sweeps taught momo\u0026rsquo;s S3 gateway to say \u0026ldquo;not implemented\u0026rdquo; for unsupported bucket and object subresources. This is the last sweep: the handful of operations that were not subresources at all but still slipped through to the wrong handler. With these six fixed, every unsupported S3 operation we know of now fails honestly.\nThe Real Problem Three kinds of operation were still misrouting:\nSelectObjectContent — a SQL-like query that filters an object\u0026rsquo;s contents and streams the matching rows back. Unsupported, it fell through to a plain GetObject and returned the whole object. UploadPartCopy — a server-side copy that uses an existing object as one part of a multipart upload. Unsupported, it fell through to UploadPart and treated the copy request as an ordinary data upload. Four bucket configuration subresources — analytics, inventory, metrics, and intelligent-tiering — that fell through to object listing. Each returned either the wrong data or a 200 over an operation that never happened.\nWhy the Obvious Fixes Failed There was no single mechanism to extend. SelectObjectContent is a query action on an object; UploadPartCopy is signalled by a request header, not a query parameter; the four configuration subresources are bucket-level. A fix had to cover all three shapes rather than lean on one router hook.\nThe tempting shortcut was to reject anything that looked unusual. That would have broken supported features that share the same URLs and headers — multipart uploads in particular.\nThe Solution: Complete 501 Coverage We extended both rejection lists and added one targeted intercept:\nOperation How it is signalled Where it is rejected SelectObjectContent Query on an object Object-level rejection UploadPartCopy X-Amz-Copy-Source header on a multipart PUT Dedicated check in the PUT dispatch, before the part-upload handler analytics Bucket configuration subresource Bucket-level rejection inventory Bucket configuration subresource Bucket-level rejection metrics Bucket configuration subresource Bucket-level rejection intelligent-tiering Bucket configuration subresource Bucket-level rejection The four configuration subresources joined the existing bucket-level list. SelectObjectContent is a query action, not a subresource, so it needed its own branch rather than a list entry. UploadPartCopy is distinguished by a header, so it is checked in the PUT dispatch before the ordinary multipart part handler can claim it — the one ordering-sensitive part of the change.\nComplete 501 Coverage Phase Target Count P1 SSE-KMS / SSE-C 2 P3 Bucket configuration subresources 16 P4 Object-level subresources 5 P5 Remaining operations 6 Total All known unsupported 29 How We Verified All six operations return 501 Not Implemented. The UploadPartCopy check runs before the part-upload handler, so there is no collision. Existing multipart operations (uploadId, partNumber) are untouched. Bucket configuration subresources such as ?versioning still return 501 from the earlier phase. Failure Modes Risk Guard UploadPartCopy intercepted after the wrong handler claims it The check is placed before the part-upload branch in PUT dispatch Over-rejecting normal multipart uploads The copy check requires the copy-source header; a normal part upload does not carry it A new unsupported operation slipping through Both rejection lists and the dispatch intercept are the single place to add coverage When NOT to Use This For supported operations. The lists contain only operations we do not implement. Never add a supported one. As a substitute for implementation. This is honest failure, not a feature. When we implement one of these, it leaves the list. Engineering Standards (🛡 Sentinel) Per docs/CORE/STANDARDS.md: 🛡 Sentinel — complete, honest 501 coverage with no silent misrouting.\nReferences / Dig deeper Spec: openspec/changes/s3-p5-remaining-subresource-501/. Issue: #920. Related posts: 008: S3 Gateway Core, 033: Bucket Configuration Subresources, 034: Object-Level Subresources. ","permalink":"/posts/035-s3-501-discipline-remaining-ops/","summary":"Final 501 sweep: SelectObjectContent, UploadPartCopy, analytics, inventory, metrics, intelligent-tiering now return honest 501 NotImplemented.","title":"S3 501 Discipline: Remaining Ops — SelectObjectContent, UploadPartCopy, Analytics, Inventory, Metrics, Intelligent-Tiering"},{"content":"The previous post in this series gave bucket configuration operations an honest \u0026ldquo;not implemented\u0026rdquo; answer. The same misrouting bug, however, lived one level deeper: on the object itself. Five object-level query parameters were being ignored, and a client that asked to read an object\u0026rsquo;s tags got the object\u0026rsquo;s bytes instead. This post closes that gap.\nThe Real Problem In S3, an operation can be selected by a query parameter appended to an object\u0026rsquo;s URL rather than by the URL path. These are subresources — GET /bucket/key reads an object, while GET /bucket/key?tagging reads that object\u0026rsquo;s tags.\nFive object-level subresources were unimplemented:\n?tagging — the object\u0026rsquo;s tag set ?acl — the object\u0026rsquo;s access control list ?versionId — a specific historical version ?retention — object retention settings ?legal-hold — a legal hold None of them had a handler. The router fell through to the ordinary object handlers, so a GET with ?tagging ran the normal \u0026ldquo;read the object\u0026rdquo; path and returned the object body. The client\u0026rsquo;s tag-parsing code then choked on binary data, or worse, silently treated it as an empty tag set.\nWhy the Obvious Fixes Failed The trade-offs here are the same as for the bucket-level case, and so is the tempting wrong answer: return 404. But the object exists; it is the operation that is unsupported. A 404 would tell a client the object is missing and send it down the wrong retry path.\nImplementing the five operations was also out of scope: retention and legal hold carry compliance semantics we did not want to half-build.\nThe Solution: Object-Level 501 Intercept We added a sibling rejection branch to the one used for bucket configuration. It runs after the bucket and key have been parsed and only fires when an object key is present. A request that names one of the five subresources gets 501 Not Implemented, the same honest answer as at the bucket level.\nSubresource Meaning tagging Object tag set acl Object access control list versionId A specific object version retention Object retention legal-hold Legal hold The split between the two rejection lists is deliberate. Bucket-level subresources are checked when the key is empty; object-level ones when the key is present. That is what lets ?tagging mean \u0026ldquo;bucket tags\u0026rdquo; at the root and \u0026ldquo;object tags\u0026rdquo; on a key without the two checks colliding.\nGuarding What We Do Support The intercept must not swallow the multipart operations that legitimately use query parameters on an object URL:\nSubresource Reason it must pass through uploadId / partNumber Multipart upload parts uploads List multipart uploads delete Batch delete How We Verified Ten request cases across the five subresources all return 501 Not Implemented. Multipart routing on uploadId and partNumber is intact. ?tagging on a GET now returns 501, where it previously returned 200 with the object body. ?versionId on a GET now returns 501, where it previously served the current version. A bucket-root ?versioning still returns 501 from the earlier fix. Failure Modes Risk Guard Catching multipart requests The intercept checks the subresource against a fixed list; multipart parameters are not in it Colliding with the bucket-level check The two branches are guarded by key-presence (empty key vs present key) Stale documentation on the shared parser The orphaned doc comment on the bucket/key parser left by the previous change was corrected here When NOT to Use This When the operation is supported. Never add a supported subresource to the rejection list just because it is uncommon. When the request is a normal object read. The intercept only triggers on the named subresources; an unadorned GET still reads the object. Engineering Standards (🛡 Sentinel) Per docs/CORE/STANDARDS.md: 🛡 Sentinel — honest error semantics, no silent misrouting. An unsupported operation says so.\nReferences / Dig deeper Spec: openspec/changes/s3-p4-object-subresource-501/. Issue: #914. Related posts: 008: S3 Gateway Core, 033: Bucket Configuration Subresources, 035: Remaining Operations. ","permalink":"/posts/034-s3-501-discipline-object-subresources/","summary":"5 object-level query parameters now return honest 501 NotImplemented instead of silently falling into GetObject/PutObject/DeleteObject.","title":"S3 501 Discipline: Object-Level Subresources (tagging, acl, versionId, retention, legal-hold)"},{"content":"S3 is a large API, and momo implements the part of it a storage backend actually needs. The rest has to fail in a way a client can understand. For a long time, \u0026ldquo;the rest\u0026rdquo; failed in a way no client could: a request for a feature we do not support was quietly answered as if it were a different request. This post is about the first batch of those we fixed — sixteen bucket configuration operations that now return an honest \u0026ldquo;not implemented\u0026rdquo; instead of the wrong data.\nThe Real Problem Some S3 operations are not addressed by the URL path alone. A GET on a bucket is normally \u0026ldquo;list the objects\u0026rdquo;; add a query parameter such as ?versioning and it becomes \u0026ldquo;read the bucket\u0026rsquo;s versioning configuration\u0026rdquo;. A query parameter that selects an operation like this is called a subresource.\nmomo\u0026rsquo;s gateway did not implement sixteen of these bucket-level configuration subresources — versioning, access control lists, bucket policies, CORS, lifecycle rules, and so on. The trouble was not that they were unsupported. It was that the router, seeing no code to handle them, fell through to the default bucket handlers: it answered a versioning request with an object listing, or ignored the parameter and returned the current object. A client asked one question and got an answer to a different one — the most dangerous kind of bug, because nothing looks broken.\nWhy the Obvious Fixes Failed \u0026ldquo;Implement them.\u0026rdquo; Versioning, policies, and object lock are large features with real semantics. We will implement some of them eventually, but we needed them to fail honestly now, not silently misbehave until then.\n\u0026ldquo;Return 404 Not Found.\u0026rdquo; That is a lie in the other direction: the route exists, and the server does understand what was asked. It just does not support it. A wrong status sends clients down the wrong error-handling path.\n\u0026ldquo;Let the fall-through stand.\u0026rdquo; This is what we had, and it is how a client ends up treating a bucket listing as a versioning configuration.\nThe Solution: Explicit, Centralized Rejection We added one place where unsupported bucket configuration subresources are recognized and rejected, checked immediately after the request\u0026rsquo;s bucket name and key have been parsed. When the request targets the bucket root (no object key) and names one of the sixteen subresources, every HTTP method gets the same answer: 501 Not Implemented.\n501 is the honest status here — the server recognizes the operation and does not support it, which is distinct from 404 (no such thing) and from 200 (here is your data).\nThe sixteen subresources now rejected:\nSubresource Meaning versioning Bucket versioning configuration versions List all object versions acl Access control list policy Bucket policy cors Cross-origin resource sharing rules website Static website hosting lifecycle Lifecycle/expiry rules tagging Bucket tags encryption Default encryption configuration publicAccessBlock Public access block settings accelerate Transfer acceleration replication Cross-region replication requestPayment Request payer setting logging Access logging objectLock Object lock configuration notification Event notification configuration Guarding What We Do Support The rejection has to be precise. Several bucket-level subresources are real features here, and they must keep working:\nSubresource Status location Supported — returns the bucket\u0026rsquo;s region list-type Supported — object listing uploads Supported — list multipart uploads uploadId / partNumber Supported — multipart operations delete Supported — batch delete How We Verified Nine unsupported subresources all return 501 Not Implemented. location still returns 200 OK with the region. list-type=2 still returns 200 OK with the object listing. An object-level tag request sent to the bucket root correctly returns 404, because tagging is only valid on objects. Multipart routing (uploads, uploadId) is untouched. Failure Modes Risk Guard Rejecting a subresource we actually support The rejection list is an explicit set; supported ones are excluded and covered by tests A future subresource silently falling through again New unsupported operations must be added to the same centralized list An over-large error response The 501 body is written with a bounded writer When NOT to Use This When you intend to implement the feature. The rejection list is for genuinely unsupported operations, not a parking lot for half-built ones. On paths where the subresource is meaningful but different. Object-level tagging is valid on an object key, so the bucket-level rejection must not catch it — which is why the check is scoped to the bucket root. Engineering Standards (🛡 Sentinel) Per docs/CORE/STANDARDS.md: 🛡 Sentinel — honest error semantics and fail-closed behavior. Unsupported means 501, never a wrong 200 or a misleading 404.\nReferences / Dig deeper Spec: openspec/changes/s3-p3-bucket-subresource-501/. Issue: #912. Related posts: 008: S3 Gateway Core, 009: S3 Multipart and Breadth, 011: S3 HTTPS/TLS Enforcement, 034: Object-Level Subresources, 035: Remaining Operations. ","permalink":"/posts/033-s3-501-discipline-bucket-config/","summary":"16 bucket config subresources (versioning, ACL, policy, CORS, lifecycle, etc.) now return honest 501 NotImplemented instead of silently misrouting to ListObjects/GetObject.","title":"S3 501 Discipline: Honest 'Not Implemented' for 16 Bucket Config Subresources"},{"content":"Storage that quietly returns wrong bytes is worse than storage that fails loudly. A corrupted read that looks successful can poison downstream systems for days before anyone notices. This arc turned momo\u0026rsquo;s content-addressing property into explicit, enforced at-rest integrity.\nThe Real Problem Because momo names every blob by the SHA-256 hash of its contents (004), the stored key is the checksum. In principle that makes read-path integrity free: re-hash the bytes, compare to the key, and you know whether they are intact. But \u0026ldquo;in principle\u0026rdquo; was doing a lot of work. Verification was scattered across handlers, some paths skipped it, and there was no consistent answer to what happens when a mismatch is found.\nWe also had a garbage-collection gap. Deleting an object had to remove its bytes without deleting a blob another object still referenced — and the deletion path had no single, auditable owner.\nWhat Landed At-rest integrity pipeline Central integrity verification (#911): one validate-then-verify path shared by every reader, instead of ad-hoc checks spread through handlers. A metadata accessor was added so a metadata query no longer has to open the content stream just to read a size or a timestamp. Verify-on-read with corruption surfacing (#925): when a re-hash does not match the key, the replica is marked suspect rather than silently served. That suspicion is the input to the degraded-read and self-heal arc (020). Checksums on the wire: S3 clients can ask for a checksum and momo echoes it back, so a caller can verify the transfer end to end (012). Garbage collection: a tombstone — a deletion marker recorded in metadata — drives a reference-count sweep. Applying a tombstone deletes the blob content, failures to delete are surfaced instead of swallowed, and the sweep entry point is guarded against being started twice. 🛡 Sentinel lens At-rest integrity is a security property, not just a performance nicety: silent bitrot becomes a loud, auditable mismatch that the system can act on. The same sweep that added verification also fixed CRLF injection and path traversal in the blob and metadata layers (015). The principle we traced through it: integrity checks must be compiled into the core and can never be skipped by swapping a seam. If a backend could opt out of verification, the guarantee would be only as strong as the weakest backend.\n⚡ Bolt lens A single combined metadata read replaced three separate key/value views, reducing write-path transactions and CPU. Verify-on-read reuses the zero-escape SHA-256 hashing path from 024, so verification does not allocate per object. See docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Specs: openspec/changes/core-integrity-verification, openspec/changes/storage-at-rest-integrity. Pull requests: #911 (central verification), #925 (verify-on-read). Related: 004, 012, 020, 015. ","permalink":"/posts/007-at-rest-integrity-and-gc/","summary":"CAS keys make verify-on-read free: central verification, x-amz-checksum echo, tombstone GC, and corruption surfacing.","title":"At-Rest Integrity: Verify-on-Read, Checksums, and GC"},{"content":"A checksum is a short fingerprint of a stream of bytes: recompute it on the other side and you know whether the data survived the trip. When we first supported AWS S3\u0026rsquo;s additive integrity checksums, we implemented the verification inside the S3 transport adapter — the code that speaks the S3 HTTP protocol. It worked, for S3 clients. It was also a trap.\nThe trap: integrity was treated as a transport feature rather than a storage invariant. If an upload arrived over momo\u0026rsquo;s native binary protocol, or was forwarded between nodes during replication, it bypassed the S3 layer entirely — and with it, every integrity check. Worse, a corrupted primary node would happily replicate its corrupted bytes to secondary replicas, spreading the poison across the cluster. Verification that only runs on one of several front doors is not verification; it is a coincidence.\nThe Real Problem Storage systems have many doors. S3 clients come in through HTTP; native clients come in over a binary protocol; other nodes come in during replication. If each door decides for itself whether to check integrity, then the guarantee depends on which door you used. We wanted a single, unavoidable checkpoint: no matter how bytes enter, they are verified before they are committed.\nWhy the Obvious Solutions Failed \u0026ldquo;Add the check to each transport.\u0026rdquo; This is duplication with a drift problem. Each adapter re-implements the same logic against different header names and status codes, and the next protocol someone adds silently misses the check.\n\u0026ldquo;Trust the primary to verify before replicating.\u0026rdquo; This makes every replica trust the primary\u0026rsquo;s memory and network card. A single faulty node then becomes a cluster-wide data-corruption event.\n\u0026ldquo;Buffer the upload, verify, then store.\u0026rdquo; Correct, but it doubles memory and latency: you hold the whole object before committing it.\nThe Solution: Verification at the Core, in One Pass We adopted a principle we call the Commodity Protocol Principle: transport adapters are disposable commodities, and all integrity verification belongs in the centralized storage ingest core. The core defines a protocol-agnostic seam — a small interface that any transport can implement to describe what checksums it expects and how to report a mismatch.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 ======================================================================================== CENTRALIZED CORE INGEST ARCHITECTURE ======================================================================================== [ S3 Client (aws-cli) ] [ Native TCP Client ] [ Peer Node (Replication) ] | | | (x-amz-checksum-sha256) | (X-Momo-S3-Meta header) | | | v v v [ S3Communicator ] [ MomoTCPComm ] [ Peer Ingest Comm ] | | | +----------------------------+-----------------------------+ | v Implements transport.ChecksumProvider +-------------------------------------+ | server/file.go: getFile() | | Central Storage Ingest Seam | +-------------------------------------+ | +------------------+------------------+ | | v v [ Stream to Hasher ] [ Stream to CASStore ] CRC32, CRC32C, SHA256 blobs/e3/b0/... | | +------------------+------------------+ | v Check: Matches Digest? / \\ [ YES ] [ NO ] | | v v Commit Metadata 1. Abort stream Return HTTP 200 OK 2. CASStore.Delete(hash) 3. Return 400 BadDigest / EBADMSG Concretely, the server\u0026rsquo;s ingest routine asks the active connection for a checksum provider — the seam. If the connection offers expectations, the core verifies the incoming stream against them. The stream flows through a tee: one branch goes to the physical blob store, the other to streaming hash engines that compute CRC32, CRC32C, SHA-1, and SHA-256 as the bytes arrive. There is no intermediate buffer and no temporary spool file — the digest is computed in flight.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 type ChecksumAlgorithm string const ( AlgoCRC32 ChecksumAlgorithm = \u0026#34;CRC32\u0026#34; AlgoCRC32C ChecksumAlgorithm = \u0026#34;CRC32C\u0026#34; AlgoSHA1 ChecksumAlgorithm = \u0026#34;SHA1\u0026#34; AlgoSHA256 ChecksumAlgorithm = \u0026#34;SHA256\u0026#34; ) type ChecksumRef struct { Algorithm ChecksumAlgorithm Value []byte } type ChecksumSet []ChecksumRef // The seam every transport implements so the core can verify without // knowing which protocol delivered the bytes. type ChecksumProvider interface { ChecksumExpectations() common.ChecksumSet OnChecksumMismatch(err error) error } 1 2 3 4 5 6 7 8 9 10 11 // server/file.go — ingest verifies, then commits or rolls back. if provider, ok := comm.(transport.ChecksumProvider); ok { expectations := provider.ChecksumExpectations() if len(expectations) \u0026gt; 0 { if err := verifier.Verify(expectations); err != nil { // Rollback: immediately purge the partially written blob. _ = s.store.Delete(meta.Hash) return provider.OnChecksumMismatch(err) } } } On a mismatch the core aborts the stream, deletes the partially written blob, and returns a transport-appropriate error: the S3 adapter translates it into an AWS-compliant 400 BadDigest, while native connections return the operating system\u0026rsquo;s \u0026ldquo;bad message\u0026rdquo; code. The rollback is part of the invariant, not an afterthought.\nReplicas re-verify. The other half of the design is zero trust between nodes. When a node forwards a replicated stream, it encodes the expected checksum manifest into a small metadata header. The receiving node runs the same core ingest pipeline, extracts the manifest, and independently verifies the bytes against the original client\u0026rsquo;s fingerprint. A faulty sender is caught at the receiver before the bad data is indexed.\nTradeoff Analysis: Surface vs. Centralized Core Verification Dimension Surface-Level Verification (Old) Centralized Core Ingest (New) Architectural Purity Coupled: transport layers dictate storage logic Decoupled: core enforces invariants; transports are adapters Multi-Protocol Safety Only S3 uploads were protected All protocols (S3, TCP, QUIC, replicas) are protected Cluster Blast Radius Corrupt primary poisons all secondary replicas Zero-trust: replicas independently verify every byte Rollback Reliability Ad-hoc cleanup left orphaned temp files Guaranteed atomicity: immediate delete on mismatch CPU Overhead Single pass on the primary only Single pass executed concurrently across replicas Engineering Standards In accordance with docs/CORE/STANDARDS.md:\n⚡ Bolt (Zero Extra Copies): the tee computes CRC32/SHA-256 digests in flight as network buffers pass directly into disk storage. No intermediate memory buffer or temporary spool file is required. 🛡 Sentinel (Fail-Closed Core): core ingest never trusts transport guarantees. Any digest mismatch triggers an immediate rollback and purges the blob address before it can be indexed. References / Dig deeper Spec: openspec/changes/core-integrity-verification. Source: src/common/checksum.go, src/server/file.go, src/transport/s3_communicator.go. Sibling posts: S3 checksum specification 012, storage verify-on-read \u0026amp; GC 007, content-addressable foundation 004, read verification optimization 043. ","permalink":"/posts/031-core-integrity-verification/","summary":"S3 integrity checksums moved from a surface-level adapter into the storage/ingest core via a protocol-agnostic ChecksumProvider seam — every write path verified, replicas re-verify on receive.","title":"Centralized Integrity Verification: Checksums Move to the Storage Core"},{"content":"A checksum is a short digest computed from a block of data: change one byte and the digest changes, so anyone can recompute it and compare. The S3 protocol carries checksums as x-amz-checksum-* response headers. This post is about moving integrity from something the server does privately to something the client can verify for itself.\nThe Real Problem We already had strong integrity inside the store. Objects are content-addressed — an object\u0026rsquo;s identity is the hash of its bytes — and every read re-hashes the data and compares it against that identity before returning it. If bits rotted on disk, the store noticed and failed loudly.\nBut clients could not see any of that. They had to trust us. A security reviewer asked the uncomfortable question: \u0026ldquo;If your server is compromised, how does the client know the bytes it received are the bytes it asked for?\u0026rdquo; The honest answer was: it doesn\u0026rsquo;t. The checksum was an internal implementation detail, not a client contract.\nThat is a real gap. Integrity only the server can check is integrity the client must take on faith. We needed to surface it on the wire so the client could verify independently.\nWhy the Obvious Solutions Failed A custom header with a proprietary format. We could invent X-Momo-Checksum: sha256:..., but no SDK understands it. Only our own client would verify it, which defeats the point of an S3-compatible gateway. Rejected: not interoperable.\nReusing the ETag. The ETag is an opaque identifier S3 uses for conditional requests and caching; its meaning is not a promise about content integrity. Overloading it would confuse clients that already depend on it. Rejected: wrong semantics.\nA separate verification API. We could expose an endpoint that returns the hash for a key. But that adds a round-trip, and clients will not call it — the point is to verify the bytes already received, not to ask a second request about them. Rejected: extra latency, breaks streaming.\nThe Solution: Native S3 Checksum Headers S3 integrity checksums flow S3 already defines the right mechanism: x-amz-checksum-* headers. We implement them natively. The lifecycle is:\nOn upload, the server hashes the incoming bytes as they stream to the store — the hasher runs alongside the write, so the whole object never sits in memory. If the client supplied a checksum, the server compares it against the computed digest and rejects a mismatch instead of storing bad data. The computed checksum is cached with the object\u0026rsquo;s metadata, so it never needs to be recomputed. On download, the server echoes the cached checksum in the response headers, and the client compares it against its own hash of the body it received. Here is the shape of that path, illustratively:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 // Illustrative pseudo-code — the real handler is wired through the store. // On PUT, hash while streaming so the object never sits in memory. hasher := sha256.New() tee := io.TeeReader(r.Body, hasher) blobID, err := store.Write(tee) // content-addressed: blobID is the hash // If the client declared a checksum, reject a mismatch now. if declared != \u0026#34;\u0026#34; \u0026amp;\u0026amp; declared != hex(hasher.Sum(nil)) { return checksumMismatch } meta.ChecksumSHA256 = hex(hasher.Sum(nil)) // cached with the object // On GET, the cached digest is echoed — no re-hashing on the read path. w.Header().Set(\u0026#34;x-amz-checksum-sha256\u0026#34;, meta.ChecksumSHA256) Because the checksum is captured at write time and stored beside the object, the read path stays allocation-free: echoing a cached string costs nothing.\nPrinciple Callout Pattern: Client-Verifiable Integrity via Native Protocol Headers Use the protocol\u0026rsquo;s own integrity mechanism (S3: x-amz-checksum-*). Compute once on ingest, cache with metadata, echo on read. The client verifies; the server does not ask to be trusted.\nApplies when: the protocol has native checksum support (S3, HTTP digests, TLS records). Doesn\u0026rsquo;t apply: protocols without checksum semantics — do not invent a custom header and call it integrity.\n⚡ Bolt \u0026amp; 🛡 Sentinel 🛡 Sentinel: the honesty guarantee. An echoed checksum is only as honest as the machinery behind it. Verify-on-read means every read re-hashes the data and compares it against the object\u0026rsquo;s content-addressed identity. If bits flipped on disk, the read fails loudly before the checksum is ever echoed. The header is backed by a check, not by a stored claim.\n⚡ Bolt: zero allocation on the read path. Because the checksum is cached with metadata, a GET sets the header from an existing string — no re-hashing, no allocation. The ingest path hashes once, streaming, so it never buffers the whole object.\nSee docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nHow We Verified Test Expected result Client provides the correct checksum 200 OK, checksum echoed Client provides the wrong checksum 400 Bad Request, nothing stored Server corruption (a flipped bit) Verify-on-read fails before the echo AWS SDK compatibility aws s3 cp validates automatically Allocation on GET Zero allocation for the checksum (cached) Failure Modes Risk Mitigation Client does not verify SDK default behaviour; documented explicitly A proxy strips the checksum header TLS is enforced, so the header cannot be rewritten in transit (011) Multipart upload checksum Parts are aggregated into one checksum at completion (009) When NOT to Use This Internal-only APIs where the network itself is the trust boundary — mutual TLS is the stronger tool there. Protocols without checksum semantics — do not invent a custom header and pretend it is a standard. When compute cost outweighs the value — a fast checksum such as CRC32 is enough for large objects, while a cryptographic hash such as SHA-256 is worth it for small, security-sensitive ones. References / Dig deeper Integrity core: 007. Gateway: 008. Multipart: 009. Performance: 024. Transport security: 011. Core integrity verification: 031. The Last-Modified header optimization: 045. Checksum spec and pull request: openspec/changes/s3-integrity-checksums (PR #902). ","permalink":"/posts/012-s3-integrity-checksums/","summary":"x-amz-checksum-CRC32/SHA256 echo on PUT; integrity surfaced to clients so dirty bytes never masquerade as clean. Teaches: client-verifiable integrity, Bolt zero-allocation on read path, Sentinel honesty guarantee.","title":"S3 Integrity Checksums: x-amz-checksum-*"},{"content":"We run a microbenchmark over the XML that the S3 \u0026ldquo;list objects\u0026rdquo; API returns. One day it reported roughly a thousand heap allocations for a single listing. That number is not a bug in the response — it was a bug in how we wrote one field of it. This post is about finding the thousand allocations and turning them into sixteen.\nThe Real Problem ListObjectsV2 is the S3 operation that lists the objects in a bucket. Its response is XML: a \u0026lt;Contents\u0026gt; block per object, each carrying a LastModified timestamp. To build the response, we serialize every block into the output buffer, formatting each timestamp as it goes.\nEvery one of those timestamp formats was going through a helper that returned a fresh string. In Go, a string is immutable and heap-allocated, so each call reserved new memory that the garbage collector would later have to reclaim. Roughly a thousand objects meant roughly a thousand allocations per response. Under sustained listing load that is a steady stream of garbage and a measurable drag on CPU as the collector runs.\nWhy the Obvious Fixes Failed \u0026ldquo;Format the timestamp once and reuse it.\u0026rdquo; Timestamps differ per object, so there is nothing to cache.\n\u0026ldquo;Pool the formatted strings.\u0026rdquo; We considered a sync.Pool, but the pool\u0026rsquo;s own bookkeeping and the retained strings cost more than the allocation we were trying to remove.\n\u0026ldquo;Change the output format to something cheaper.\u0026rdquo; The XML must stay byte-compatible with what S3 clients expect. The format was never the problem.\nThe Solution: AppendFormat into a Stack Buffer The insight is that Go\u0026rsquo;s time package has two formatting paths. Format returns a new string — the allocation. AppendFormat appends the formatted timestamp to a byte slice you provide. If that slice is backed by an array on the stack, the whole format operation stays off the heap.\n1 2 3 4 5 6 7 // A 32-byte scratch array lives on the stack; nothing is heap-allocated. var buf [32]byte b := buf[:0] // AppendFormat writes the timestamp into the stack-backed slice and // returns the extended slice. No intermediate string is created. b = t.AppendFormat(b, time.RFC3339Nano, e.LastModified) We inlined this into the per-entry serialization so there is no helper call and no intermediate string. Two more details mattered:\nA shared escape buffer. XML special characters (\u0026lt;, \u0026gt;, \u0026amp;) must be escaped. The escaper now reuses a pre-allocated buffer instead of allocating per entry. Panic recovery. The format path is wrapped so that any panic is converted into a normal I/O error rather than crashing the server. How We Verified Metric Before After Allocations per operation ~1000 16 CPU baseline ~60% less Output bytes — byte-identical A microbenchmark over the XML serializer reports 1000 → 16 allocations per operation and roughly 60% less CPU. The output is byte-identical: timestamps stay in UTC as 2006-01-02T15:04:05.000Z. The full transport test suite passes. Failure Modes Risk Guard Stack buffer too small for an unusual timestamp 32 bytes covers the fixed RFC3339Nano layout; the panic-recovery wrapper turns an overflow into an error rather than a crash A non-UTC timestamp producing different bytes The value is converted to UTC before formatting, matching the previous output exactly Output drift going unnoticed Byte-identical output is asserted in the tests When NOT to Use This Low-frequency formatting. If a timestamp is formatted once per request, the allocation is noise; time.Format is clearer. When the output format is allowed to change. AppendFormat is worth it because it preserves the exact bytes. If you can change the format, measure first — the win may not be the allocation. Outside hot paths. This is a hot-path optimization; apply it where a profile shows the allocation actually matters. Engineering Standards (⚡ Bolt) Per docs/CORE/STANDARDS.md: ⚡ Bolt — zero-allocation hot paths, stack buffers, and pre-allocated reuse.\nReferences / Dig deeper Spec: openspec/changes/s3-listxml-appendformat/. Issue: #900. Related posts: 008: S3 Gateway Core, 024: Bolt Performance Engineering, 042: Performance Profiling Baseline, 045: Bolt Last-Modified Header. ","permalink":"/posts/036-s3-listxml-appendformat-optimization/","summary":"S3 ListObjectsV2 XML serialization dropped from ~1000 allocations to 16 per op via inlined time formatting and a pre-allocated escape buffer.","title":"⚡ Bolt: 1000 → 16 Allocations in S3 ListObjectsV2 XML"},{"content":"Momo does not tune itself with a single knob. It grows three independent feedback loops, each watching one signal and adjusting one decision at runtime, so the cluster adapts to its own size, its own memory budget, and its own slowest members. None of them change behavior for requests already in flight; they change what the next request does.\nGossip fanout scales with cluster size Gossip fanout is how many peers a node contacts per gossip round. A fixed number is wrong at both ends: pick it high enough for a large ring and a tiny lab cluster drowns in redundant traffic; pick it low enough for the small cluster and news takes too long to reach every node in a big one. The fix scales the fanout target with a bounded function of node count — enough to spread information in a hundred-node ring, capped so a three-node cluster stays quiet.\nStreaming cipher chunk size adapts When momo streams encrypted data, the cipher has to frame the byte stream into chunks, the units it encrypts and authenticates. Small chunks mean tighter memory use and finer granularity, but more per-chunk framing overhead; large chunks mean less overhead but bigger buffers. The chunk size is chosen at runtime from the stream format version and the node\u0026rsquo;s memory profile, keeping allocation bounded and the on-the-wire semantics unchanged. The choice is a seam: a single policy point that selects a compiled-in strategy rather than one that alters the byte format itself.\nPeer quality feeds quorum selection Lease voting, confidential deduplication (built on an oblivious pseudorandom function, or OPRF), and scatter-gather all have to choose which peers to ask. Instead of a one-rule policy like \u0026ldquo;fastest first\u0026rdquo;, the system ranks candidates by a quality signal, so slow or flaky peers lose quorum weight and reliable replicas win. That ranking also feeds the survivor choice for degraded reads in 020, so the node that answers a degraded read is the one most likely to have good data.\nKeeping decisions off the byte path Each adaptive loop is a decision point, deliberately separated from the flow of bytes: measurement happens off the hot path, and the selection updates a compile-time predicate. There is no runtime reflection and no dynamic plugin loading — a declarative policy feeds a registry that was compiled in. This keeps the adaptive machinery cheap enough to run on every node, and keeps its cost from touching the request path. See docs/CORE/STANDARDS.md for the ⚡ Bolt and 🛡 Sentinel mindsets behind that split.\nReferences / Dig deeper Adaptive gossip spec: openspec/changes/add-adaptive-gossip-scale; PR #835. Adaptive streaming chunk spec: openspec/changes/add-adaptive-streaming-chunk-size; PR #834. Peer-quality quorum spec: openspec/changes/add-peer-quality-quorum-selection; PR #833. Sibling posts: 016: Gossip, SWIM, and Membership, 017: Scatter-Gather and Lease Consensus, 020: Degraded Read Self-Heal. ","permalink":"/posts/018-adaptive-scaling-peer-quality/","summary":"Three adaptive feedback loops: gossip fanout scales with cluster size, streaming chunk size adapts to cipher/memory, peer quality feeds quorum decisions.","title":"Adaptive Scaling: Gossip Fanout, Stream Chunks, Peer Quality"},{"content":"Client-held end-to-end encryption bought us zero trust at the server, but it created a paradox. Deduplication — storing one physical copy of a file that many users upload — normally works by hashing the content: identical files produce identical hashes, and the store keeps one copy. But if only the client can see content, and every object is encrypted under a fresh random key, how do we dedup without leaking which files users have?\nThe paradox and the fix The naive approach, unidirectional encryption of content, has a quiet flaw: it leaves a stable hash of the content that the server can compute. That stable hash is a fingerprint. An attacker holding a candidate file can hash it and ask \u0026ldquo;does this blob match the target file?\u0026rdquo; — a content-confirmation attack — turning the dedup index into an oracle that confirms file possession. Encryption was supposed to prevent exactly that.\nWe wanted the opposite property: the server should be able to recognize that two uploads are identical without ever learning what they are. The tool for that is a threshold OPRF.\nAn OPRF (oblivious pseudo-random function) is a two-party protocol: the server holds a secret key and can evaluate a keyed function on the client\u0026rsquo;s input, but it learns nothing about the input; the client gets the function\u0026rsquo;s output but cannot compute it alone, because it does not have the server\u0026rsquo;s key. Blinding is the trick that hides the input: the client transforms its value with a random mask before sending it, so the server sees an unrecognizable number, evaluates on that, and returns a result the client can unblind. Threshold means the server is not one machine but a quorum of peers that must cooperate — no single node holds the whole key, so no single compromised node can evaluate the function on its own.\nThe result is a dedup signal that is not a deterministic function of content alone. Two clients uploading the same file agree on the OPRF output and dedup correctly, but the server sees only blinded values and can never confirm a candidate file. We get confidential dedup without content-hash disclosure, and without forcing every upload to break encryption.\nThe protocol shape is:\n1 client → OPRF(blind(password[content])) → server (threshold quorum) → PRF output Supporting security hardening The same hardening wave taught authentication to fail adaptively rather than either accepting everything or locking the world out. Two behaviors:\nFailed-auth backoff and temporary lockout. Repeated wrong credentials trigger a growing delay and then a short lockout, which throttles credential-stuffing without permanently denying the legitimate holder. The lockout is temporary by design, so a burst of guesses cannot permanently disable an account. Debounced, adaptive thresholds. The backoff parameters respond to observed pressure, modeled on the same adaptive patterns we use for gossip and peer quality elsewhere in the system, rather than a fixed constant that is either too loose under attack or too tight under normal load. 🛡 Sentinel lens An OPRF is a private-set-membership protocol, and those are subtle to get right: a single implementation mistake can turn the oracle back on. That is why it lives in the auditable core, the small set of trust-critical code that is reviewed as a unit and never bypassed by a seam or a plugin. We pair it with an honest server-side-encryption and key posture, and with replay protection on the request path, so the protocol is not the only line of defense.\nSee docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Specs: openspec/changes/secure-e2ee-confidential-dedup, openspec/changes/add-adaptive-auth-backoff. Pull requests: #819, #826. Sibling posts: 013: Client-Held E2EE, 015: The Sentinel Sweep, 017: Scatter-Gather Lease Quorum, 018: Adaptive Peer Quality, 010: S3 Auth, Presigned URLs, and SigV4. ","permalink":"/posts/014-confidential-dedup-oprf/","summary":"Threshold-OPRF lets E2EE cas blobs dedup without revealing content hashes — plus adaptive failed-auth backoff and lockout.","title":"Confidential Dedup via Threshold OPRF — and Auth Lockout"},{"content":"Metadata in momo does not live in one database. Each node owns a slice — a shard — of the key space, so no single machine has the whole picture. That shapes two operations every S3-style cluster needs: listing objects, and protecting mutable shared state. They are two halves of the same problem.\nScatter-gather is the listing half. To answer \u0026ldquo;what objects exist under this prefix?\u0026rdquo;, a node fans the question out to the shard owners, collects their partial answers, and merges them into one sorted, deduplicated list — scatter the request, gather the replies. A lease is the mutable-state half: a time-limited grant that lets exactly one node modify something like a rename or a reference count at a time. Granting a lease requires a quorum, a majority of the voting peers to agree, so two network partitions cannot both hand out the same lease and corrupt state — the failure mode known as split-brain.\nScatter-gather listing Scatter-gather and lease quorum A list request fans out to shard owners, merges their metadata lists, dedups by content hash (so the same bytes stored under two names appears once), and paginates the result. The filesystem layer later narrowed the fan-out to only the shard owners that can actually hold the requested prefix, turning a cluster-wide scatter into a handful of targeted requests.\nLeases and the majority-quorum bug class A lease protects mutable shared state, and the correctness of the whole scheme rests on one arithmetic rule: a lease is granted only when the number of votes is a strict majority of the peer count. For three peers the majority is two; for five it is three. Get that comparison wrong and the guarantee evaporates. Two bugs found by audit show how.\nThe off-by-one majority. The quorum test was written in a way that could resolve too low for odd peer counts, accepting fewer votes than a true majority. The fix is to require strictly more votes than half the peers, computed on the correct integer basis — for three peers, more than one means at least two. The zero-quorum lease. Worse, lease acquisition could succeed with no quorum at all while the cluster was partitioned — the split-brain enabler. The fix is to fail closed: no quorum, no lease, even if that means the operation is refused until the partition heals. Keeping the hot paths allocation-free Both operations mutate shared state, so both follow a merge discipline that avoids allocations on the request path: merge by content hash, in a stable order, with no intermediate copies. The metadata-list encoder also gained length limits, so a maliciously oversized entry cannot blow up the message buffer.\nWhy this is a security posture, not just a correctness fix A partition that grants a lease is how a split ring silently corrupts reference counts, which is why the zero-quorum fix is treated as security-adjacent rather than as a routine bug. It is the same fail-loud stance we take elsewhere: when the system cannot prove it is safe to proceed, it refuses instead of guessing.\nReferences / Dig deeper Lease/quorum fixes: PR #806, PR #810. Spec: openspec/changes/gossip-scatter-lease. Filesystem listing design: docs/momofs/IMPLEMENTATION.md; implementation in src/momofs. Sibling posts: 016: Gossip, SWIM, and Membership, 021: R3 Write Durability and Quorum, 015: Sentinel Security Audit, 018: Adaptive Scaling. ","permalink":"/posts/017-scatter-gather-lease-quorum/","summary":"Scatter-gather lists across shard owners; leases with correct majority quorum — and the two nasty off-by-one/partition bugs found by audit.","title":"Scatter-Gather and Lease Consensus: Quorum Math"},{"content":"S3\u0026rsquo;s surface is enormous, and we had a choice about how to grow into it: pretend to support everything, or support a precise subset and say so loudly. We chose the second, and it shaped both halves of this post — multipart upload, which we added for real, and a disciplined 501 Not Implemented response for everything else.\nThe Real Problem Two different gaps kept showing up. The first was concrete. Large-file tooling does not upload a multi-gigabyte object in one request; it uses multipart upload, a three-step dance where the client opens an upload, sends the object as numbered parts, then asks the server to assemble the parts into one object. Without it the biggest and most common S3 workload — copying a big file — simply did not work.\nThe second gap was philosophical. Clients probe capabilities we never intended to implement: bucket lifecycle rules, versioning, CORS, per-object access control lists, tagging, and dozens of other \u0026ldquo;subresources\u0026rdquo; — the sub-paths that hang off a bucket or object URL. The tempting answer is to return 200 OK and do nothing. That is a trap. A client that believes its lifecycle rule was accepted will behave as if it were, and corrupted assumptions are far worse than an error.\nMultipart Upload S3 multipart upload flow The flow is CreateMultipartUpload → UploadPart (repeated) → CompleteMultipartUpload, with AbortMultipartUpload to discard a half-finished upload. Each part is stored as it arrives; at completion the parts are concatenated into a single content-addressed blob, so the finished object is indistinguishable from one written in a single request. This is what large-object tooling actually calls, and it closed the biggest \u0026ldquo;S3-shaped\u0026rdquo; hole in the gateway core (008).\n501 Discipline 501 Not Implemented is the HTTP status meaning \u0026ldquo;this server does not support the functionality required to fulfil the request\u0026rdquo;. We made it the honest default for subresources momo deliberately does not do:\nBucket-configuration subresources — lifecycle, versioning, CORS, and friends — return 501. Object-level subresources — ACLs, tagging, and friends — return 501. Everything remaining returns 501 from a documented, synchronized catalog, so no operation quietly pretends to succeed. Why not a best-effort 200? Because a silent partial implementation corrupts client assumptions far worse than an explicit rejection. The Sentinel mindset is fail closed, loudly: an operation we cannot honour must not look like it succeeded.\nPattern: Fail Closed, Loudly When a protocol exposes operations you do not implement, return that protocol\u0026rsquo;s explicit \u0026ldquo;not implemented\u0026rdquo; signal rather than a success code. A success code is a promise; if you cannot keep it, do not make it.\nApplies when: a partial protocol implementation faces clients that probe for capabilities you have no plans to build. Doesn\u0026rsquo;t apply: operations you intend to implement soon — there, a 501 is a temporary honest placeholder, not a permanent answer.\n⚡ Bolt \u0026amp; 🛡 Sentinel ⚡ Bolt. ListParts (the XML listing of an in-progress multipart upload) was a standout allocation hotspot. It was profiled and de-allocated in the Bolt arc (024), and part listings reuse the same pagination discipline as ListObjectsV2.\n🛡 Sentinel. The 501 catalog is a safety property: the set of supported operations is explicit, synchronized, and testable, so \u0026ldquo;unsupported\u0026rdquo; can never masquerade as \u0026ldquo;done\u0026rdquo;.\nSee docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Core gateway: 008. Integrity echo: 012. 501 discipline in depth: 033. Compatibility matrix and the 501 catalog: docs/REFERENCE/COMPATIBILITY.md, openspec/changes/s3-p5-remaining-subresource-501/. Multipart spec and pull requests: openspec/changes/s3-multipart-upload (PRs #801, #913, #915, #921). ","permalink":"/posts/009-s3-multipart-and-breadth/","summary":"Multipart upload support plus a rigorous 501 strategy for unsupported S3 subresources — honest breadths over silent misbehavior.","title":"S3 Multipart and Protocol Breadth: 501 Discipline"},{"content":"Every \u0026ldquo;optimization\u0026rdquo; we shipped seemed to win in micro-benchmarks and then do nothing in production. We were tuning the wrong things — and worse, we had no way to tell which things were wrong. The fix was not another clever trick; it was a discipline, and the discipline earned its name the hard way.\nThe moment it clicked was a 10 TB data migration. The content-addressed write path was allocating 2.4 KB per object just to hash the object\u0026rsquo;s contents and render that hash as hexadecimal text. At 50k objects per second that is 120 MB/s of garbage, and the garbage collector\u0026rsquo;s pauses reached 40 ms — an eternity next to a latency target in the low milliseconds. The \u0026ldquo;fast\u0026rdquo; hashing was not fast once the collector got involved.\nThe mental model is one sentence: in a hot path, allocation is latency. A garbage-collection pause is part of your p99 whether or not you wrote it down.\nA few terms, because the rest of the post leans on them. An allocation is memory taken from the heap, the shared pool the runtime manages. A heap escape is when a value that could have stayed on the function\u0026rsquo;s stack is pushed onto the heap because something outlives the call. Garbage collection (GC) is the runtime reclaiming that heap memory, and its pauses are time your request spends waiting. A hot path is code that runs on nearly every request. A syscall is a call from the program into the operating-system kernel; each one pays a context-switch cost.\ndocs/CORE/STANDARDS.md codifies the two engineering mindsets; this post is the ⚡ Bolt side — measured, profiled, allocation-light work on hot paths.\nWhy the Obvious Solutions Failed Pool the hash buffers. Reusing a buffer through a pool sounds free, but the pool\u0026rsquo;s mutex becomes a contention point at 50k operations per second — more expensive than the allocation it removes. Worse, pooled objects survive the young generation of the collector and get promoted to the old generation, which produces longer pauses, not shorter ones.\n1 2 3 // REJECTED: pool mutex contention at 50k ops/s costs more than the allocation, // and promoted pooled objects make pauses longer. var hashPool = sync.Pool{New: func() any { return make([]byte, 32) }} Cache the hex strings. Content-addressed storage means every object\u0026rsquo;s hash is effectively unique, so a cache would have a near-zero hit rate while growing without bound.\n1 2 // REJECTED: content-addressed hashes are unique → near-zero hit rate, unbounded memory. var hexCache = sync.Map{} Call the standard encoder and accept its destination buffer. The standard hex encoder writes into a destination slice you have to allocate first, so the allocation just moves rather than disappearing.\n1 2 3 // REJECTED: hex.Encode writes into dst, but dst itself still has to be allocated. dst := make([]byte, hex.EncodedLen(len(src))) hex.Encode(dst, src) The Solution — Three Patterns With Annotated Code Figure 1: The three core pillars of Bolt performance engineering in momo 1. Zero-Escape SHA-256 and Hex Encoding Figure 2: Stack arrays vs heap pooling — eliminating garbage collector pauses A stack array lives and dies with the function call. If we hand the hashing and encoding routines a slice backed by that array, their output never touches the heap.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 // Stack-allocated hash buffer (SHA-256 = 32 bytes) var hashBuf [sha256.Size]byte // Stack-allocated hex buffer (2 chars per byte = 64 bytes) var hexBuf [sha256.Size * 2]byte // Write the hash directly into the stack buffer h := sha256.New() h.Write(data) hash := h.Sum(hashBuf[:0]) // hashBuf[:0] = empty slice, cap=32 // Encode directly into the stack hex buffer — zero allocation hex.Encode(hexBuf[:0], hash) // hexBuf now holds the hex text, backed by the stack array hashBuf[:0] and hexBuf[:0] are the trick: each creates a slice with the array\u0026rsquo;s capacity but length zero, backed by stack memory. Sum and Encode append into that capacity, and the resulting slices point into the stack array — so the compiler can prove there is no heap escape.\n2. Deadline Amortization — Cut SetDeadline ~98% Figure 3: Phased deadlines cut kernel syscalls from 200/sec to 3/sec A deadline is the point at which the network layer gives up on a connection. Resetting it before every read and write is the safe, obvious thing to do — and it costs two kernel syscalls per chunk. We amortized the cost by setting one absolute deadline per phase of the connection instead.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 // BEFORE: SetDeadline on every read/write (2 syscalls per chunk) conn.SetDeadline(time.Now().Add(5 * time.Second)) n, err := conn.Read(buf) conn.SetDeadline(time.Now().Add(5 * time.Second)) conn.Write(buf[:n]) // AFTER: phased absolute deadlines, one SetDeadline per phase // Handshake: 10s absolute conn.SetDeadline(time.Now().Add(10 * time.Second)) // ... handshake ... // Metadata: 60s absolute (reuses the same deadline) conn.SetDeadline(time.Now().Add(60 * time.Second)) // ... metadata ... // Data transfer: dynamic, based on observed throughput deadline = calculateTransferDeadline(bytesRemaining, throughput) conn.SetDeadline(deadline) // ... streaming ... Result: from 200 SetDeadline calls per second per connection to 3. The syscall overhead all but vanishes, without weakening the Slowloris defenses — the phases still bound how long a stalled client can hold a slot.\n3. Combined Metadata Reads Figure 4: Collapsing three separate Bbolt transactions into a single atomic read view Our embedded key-value store exposes a view: a read-only, snapshot-consistent transaction. Each view takes a lock. We were opening three separate views per write to read three buckets — three locks, three snapshots — when one view can read all three.\n1 2 3 4 5 6 7 8 9 10 11 12 13 // BEFORE: 3 separate store views per write view1, _ := db.View(fn1) // bucket A view2, _ := db.View(fn2) // bucket B view3, _ := db.View(fn3) // bucket C // AFTER: 1 view, multiple buckets err := db.View(func(tx *bbolt.Tx) error { a := tx.Bucket(bucketA).Get(key) b := tx.Bucket(bucketB).Get(key) c := tx.Bucket(bucketC).Get(key) // process a, b, c return nil }) Result: three times fewer transactions and lock acquisitions, and a third of the contention.\nThe Measurement Contract (Non-Negotiable) No \u0026ldquo;it\u0026rsquo;s probably faster\u0026rdquo; claims. Ever.\n1 2 3 # Every PR must pass the benchstat gate go test -bench=. -benchmem -count=10 ./... benchstat old.txt new.txt benchstat is the standard Go tool that compares two sets of benchmark runs and tells you whether a difference is statistically real or just noise. A regression gate turns that comparison into a build condition: if a branch allocates more or runs measurably slower than the base, CI fails and the merge stops. A profile baseline is a saved CPU or memory profile that every later change is judged against.\nRequirement Tool Gate No allocation regression benchstat CI fails if allocation delta \u0026gt; 0% No latency regression benchstat CI fails if ns/op delta \u0026gt; 5% Profile baseline go test -cpuprofile file-based profiles only History tracking checked-in benchmark history per-benchmark deltas over time The \u0026ldquo;file-based profiles only\u0026rdquo; rule is a security decision, not a convenience one: a live profiling endpoint on a server with no authentication is a remote-code-execution-class surface, so we never expose one on the data path. The exact rule number and the paths live in the References section.\n⚡↔🛡 Trade-off Posture Bolt optimizes inside hot paths but never at the expense of invariants:\nInvariant Never Compromised Verify-on-read stays ON (see 007) Bounds checking all inputs validated, no unsafe casts Integrity / crypto / placement core compiled-in, auditable Principle: speed comes from removing waste, not skipping checks.\nWhen NOT to Apply Bolt Patterns Pattern Don\u0026rsquo;t Use When Stack-buffer formatting low-frequency paths (startup, config, admin APIs) Deadline amortization per-request deadlines are required (multi-tenant isolation) Combined reads the buckets live on different storage backends Zero-escape hashing non-hot paths, or when the hash must escape (e.g. returned to a caller) References / Dig deeper The two mindsets: docs/CORE/STANDARDS.md — ⚡ Bolt and 🛡 Sentinel. Measurement gate: 025: The Benchstat Gauntlet. Profiling baseline: 042: Perf Profiling Baseline. Integrity and verify-on-read: 007: At-Rest Integrity and GC. Transport: 003: TCP → QUIC. Applied patterns: 047, 048, 049, 045, 036. Benchmark harness and history: the CI benchmark-compare workflow, docs/REFERENCE/PERFORMANCE.md, and the checked-in benchmark history data file. The networked-profiler prohibition is recorded as Rule 75 in the steering rules; the profile-baseline spec is openspec/changes/perf-profiling-baseline. ","permalink":"/posts/024-bolt-performance-engineering/","summary":"The Bolt mindset codified: no heap escapes in hashing/encoding, deadline amortization cuts SetDeadline ~98%, allocation hotspots hunted by profile. Teaches: zero-escape patterns, measurement contracts, trade-off posture.","title":"⚡ Bolt: Zero-Allocation and Hot-Path Engineering"},{"content":"TLS — Transport Layer Security — is the protocol that encrypts and authenticates a network connection; HTTPS is simply HTTP carried over TLS. \u0026ldquo;Plaintext\u0026rdquo; means a connection with no encryption at all, where anyone on the path can read or alter the bytes. This post is about a small policy change with a large blast radius: making TLS the default for the S3 gateway and for inbound daemon traffic, so that silence no longer means \u0026ldquo;insecure\u0026rdquo;.\nThe Real Problem The gateway shipped with an HTTP listener that just worked. That convenience was the danger: a client could send object data and credentials in the clear and never be told. Worse, because plaintext was the default, an attacker on the network could downgrade a connection — strip or block the TLS attempt and watch the client fall back to HTTP without complaint. A protocol that silently permits weakness will be silently weakened.\nWe also had to be honest about scope. momo did not (yet) have an authentication story of its own, which meant the wire was the last line of defence. If the store goes to great lengths to protect bytes at rest, letting those same bytes cross the network unencrypted is a contradiction.\nThe Solution: No Silent Plaintext Two changes enforced the rule:\nThe S3 gateway requires HTTPS. A plain HTTP request is rejected unless the operator has explicitly opted into an insecure listener. A client that \u0026ldquo;just works\u0026rdquo; over HTTP must now opt into weakness loudly. Inbound daemon transport requires TLS. Before the daemon accepts replication or transport connections, the peer must complete a TLS handshake — or the operator must set an explicit insecure flag. The key word is explicit. Insecure mode did not disappear; it became something you have to ask for by name. That turns an accident into a decision.\nThe Trade-off Cost. The daemon has no certificate-management flow — no automated certificate issuance or renewal — so TLS defaults to a self-signed certificate or one the operator supplies. Clients must be told to trust it, which is real operational friction. Win. Silent downgrade attacks and on-wire sniffing are structurally removed. The insecure flag still exists, but it can only be set deliberately; it can no longer be reached by omission. We chose the friction. A self-signed certificate that a client explicitly trusts is still far better than an encryption-free connection nobody chose.\n⚡ Bolt \u0026amp; 🛡 Sentinel 🛡 Sentinel. This is a pure Sentinel call: the wire must not leak what the store protects elsewhere. Combined with SigV4 freshness (010), the gateway is both replay-safe and confidentiality-safe.\n⚡ Bolt. The TLS decision sits on the control plane, not the data path. Once a connection is established, the same allocation-light streaming rules apply as before; enforcing TLS does not add per-object overhead.\nSee docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Gateway core: 008 → 010: SigV4 → this post → 015: Security Audit. Daemon transport TLS/QUIC: 003. Gateway and inbound TLS specs: openspec/changes/s3-https-enforcement, openspec/changes/s3-inbound-tls-enforcement (PRs #792, #793). Configuration keys and pentest guidance: docs/GUIDES/CONFIGURATION.md, docs/GUIDES/PENTESTING.md. ","permalink":"/posts/011-s3-https-tls-enforcement/","summary":"The S3 gateway turned HTTPS-or-explicit-insecure into the default: no silent plaintext for the gateway or inbound tcp/quic.","title":"S3 and Inbound TLS: HTTPS-by-Default Enforcement"},{"content":"SigV4 — AWS Signature Version 4 — is the request-signing scheme every S3 client speaks. It is more than \u0026ldquo;an auth scheme\u0026rdquo;: the client derives a signing key from its secret, canonicalizes the request (method, path, headers, payload hash), and signs that canonical form. Done wrong, the scheme becomes a replay machine — an attacker who captures one valid request can send it again and again. This post is about making signing correct, honest, and separate from momo-native auth.\nThe Real Problem Authentication is the one place where a small implementation mistake is a full compromise rather than a bug. Three specific things worried us.\nFirst, presigned URLs. A presigned URL is a normal request whose signature travels in the query string instead of an Authorization header, so the URL itself grants time-limited, scoped access without ever exposing the secret key. Real tools depend on this: SDKs generate them, curl fetches them, and rclone uses them for expiring links. If the gateway cannot verify a query-string signature, half the ecosystem cannot use it.\nSecond, freshness. A signature is only as safe as the request it is bound to. If the timestamp can be omitted or ignored, a captured signature stays valid forever. Freshness — rejecting requests whose signed time is too far from the server\u0026rsquo;s clock — is what turns a signature into a short-lived credential.\nThird, key separation. The gateway\u0026rsquo;s S3 access key and the momo-native auth token were originally the same secret. That meant leaking one surface exposed the other, and it made two very different lifetimes impossible to reason about.\nWhat We Changed Query-string SigV4 for presigned URLs. The gateway verifies signatures carried as X-Amz-Credential, X-Amz-Signature, and X-Amz-Date in the query string, so temporary links work exactly as SDKs expect. X-Amz-Date is required. We removed a dead fallback path that accepted a signature with no freshness check at all — a replay hole in the making. Freshness applies to native auth too. The plaintext challenge-response path is now checked against clock skew, not just the S3 path. Encoding bounds by runes, not bytes. Percent-escaping a signing key must count characters, not bytes, or keys longer than about a kilobyte containing multi-byte UTF-8 sign incorrectly. Bounding by runes fixed that. Gateway keys are decoupled from the native token. Two keys, two lifetimes: the S3 access/secret pair and the momo-native token no longer double as each other. The Trade-off Requiring freshness means a client with a badly skewed clock is rejected. That is the correct failure — a loud RequestTimeTooSkewed beats silently accepting a replayable request — but it does mean operators must keep their clocks synchronized. Decoupling the keys also means two credentials to provision and rotate instead of one; we accepted that cost because shared secrets are exactly the kind of coupling a security review flags.\n⚡ Bolt \u0026amp; 🛡 Sentinel 🛡 Sentinel. Auth is where Sentinel thinking concentrated hardest: replay protection, freshness, least-privilege key separation, and honest signing. The signed payload posture — signing the payload hash rather than trusting a declared value — is part of the same story.\n⚡ Bolt. Signing is compute-regular: stable, allocation-light hashing, with deadlines amortized across header reads rather than re-derived per field.\nSee docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Gateway core: 008. Transport security: 011. Audit deep-dive: 015. Signed-payload and SSE: 039, 040. Auth spec and pull requests: openspec/changes/signed-payload-and-sse (PRs #789, #791, #885). Pentest guidance: docs/GUIDES/PENTESTING.md. ","permalink":"/posts/010-s3-auth-presigned-sigv4/","summary":"Query-string SigV4 for presigned URLs, mandatory X-Amz-Date freshness, and decoupling gateway access/secret keys from the native auth token.","title":"S3 Auth: SigV4, Presigned URLs, and Key Decoupling"},{"content":"S3 is the de-facto object-storage API: a handful of HTTP verbs (PUT, GET, HEAD, DELETE) plus XML documents that describe buckets, objects, and errors. \u0026ldquo;Speaking S3\u0026rdquo; means a stock AWS client — the aws CLI, any vendor SDK, rclone — can point at momo with no custom driver. That compatibility bet is what made the gateway useful, and it is why the gateway is deliberately boring.\nThe core design decision: the gateway is a stateless adapter, not a second storage server. It keeps no index, no cache, and no separate object model. Each S3 request maps onto one call against the native store, so the same bytes are visible through the native protocol, replication, and the filesystem mount. One content-addressed store, many doors.\nThe Real Problem We wanted real clients to work on day one, but a gateway that only understands PUT and GET is a toy. Real tools immediately exercise the long tail: they probe metadata with HEAD, enumerate buckets and objects, resume partial downloads with byte ranges, send conditional requests, attach content types and custom metadata, and copy or bulk-delete objects. A missing verb does not fail loudly — it fails as a confusing parse error deep inside a client.\nThe second problem was errors. Go\u0026rsquo;s default is to write a text/plain body and an HTTP status. S3 clients do not read text/plain; they parse an XML error document with a machine-readable code such as NoSuchKey or BucketAlreadyExists. Without that XML a client cannot tell \u0026ldquo;not found\u0026rdquo; from \u0026ldquo;permission denied\u0026rdquo; from \u0026ldquo;malformed request\u0026rdquo;, so it retries the wrong things or gives up silently.\nWhat We Built The first wave covered the verbs clients actually call:\nS3-shaped XML errors. Every failure returns the error XML an SDK expects, with a code and a message, instead of an opaque plaintext body. HeadObject and HeadBucket. The HEAD verb returns the same headers as GET — size, content type, ETag, custom metadata — with an empty body, so a client can inspect an object without downloading it. Bucket management. CreateBucket, DeleteBucket, and ListBuckets. ListObjectsV2 with pagination. Large buckets are enumerated in pages using a continuation token, the opaque cursor the client passes back to fetch the next page. One subtle bug here is worth remembering: a page\u0026rsquo;s key count must not include common prefixes (the \u0026ldquo;folders\u0026rdquo; in a delimiter listing), or a client\u0026rsquo;s progress arithmetic drifts. Range and conditional requests. A byte range such as bytes=0-1023 lets a download resume; conditional headers such as If-None-Match let caches and sync tools avoid transferring unchanged objects. Object metadata. Content-Type and custom x-amz-meta-* headers round-trip through the store. CopyObject and batch DeleteObjects. Server-side copy and the bulk delete document tools use to clean up many keys at once. Why an Adapter, Not a Server Because the gateway owns no state, compatibility work stays contained. Adding an S3 verb is a translation, not a new data path; durability, replication, and integrity guarantees are inherited from the store. It also means we never have to reconcile \u0026ldquo;the S3 copy\u0026rdquo; with \u0026ldquo;the native copy\u0026rdquo; — there is only one copy.\nThe trade-off is honesty about scope. Anything S3 specifies that the native store does not model is rejected rather than emulated, which is the subject of the 501-discipline work (009). And because S3 is the widest untrusted surface in the system, every new verb is also new attack surface.\n⚡ Bolt \u0026amp; 🛡 Sentinel 🛡 Sentinel. S3 was the widest untrusted surface we had. The same month produced a security sweep (015) that found HTTP request smuggling, header-read limit leaks, missing body closes, and CRLF injection in and around the gateway; all were fixed as part of the hardening arc. Inbound TLS enforcement followed (011).\n⚡ Bolt. The list handlers were the most allocation-heavy code in the gateway, so a series of allocation optimizations attacked exactly there (024).\nSee docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Sibling posts: 006: Pluggable Storage Backends, 004: Content-Addressable Store, 009: Multipart and Protocol Breadth, 010: SigV4 and Presigned URLs, 011: HTTPS/TLS Enforcement, 012: Integrity Checksums. 501 discipline: 033, 034, 035. Performance follow-ups: 036, 040. The gateway spec and pull requests: openspec/changes/add-s3-protocol (PRs #782, #783, #785). ","permalink":"/posts/008-s3-gateway-core/","summary":"The S3 gateway: SigV4, S3-compliant XML errors, Head/List/Copy, pagination, ranges, and metadata — a compatible API surface for real clients.","title":"S3 Gateway Core: XML, Buckets, Objects, Lists"},{"content":"The strongest security guarantee a cloud storage system can offer is simple: even if an attacker gains full root access to the storage nodes, the physical disks, and the metadata databases, they cannot read a single byte of customer data.\nFor a long time we did not have that guarantee. Encryption-at-rest protected the disks, but the servers still held the keys, so a rogue operator or a compromised daemon could read anything. We wanted a design where the server is structurally incapable of reading the data it stores — not merely trusted not to. That is end-to-end encryption (E2EE): the data is encrypted on the client and stays ciphertext everywhere else. momo implements it as client-held envelope encryption, across both its S3 gateway and its native binary transport.\nFigure 1: High-level overview of client-held envelope encryption 1. The Key Hierarchy: KEK, CEK, and Envelopes The obvious first idea is to encrypt every file directly with one long-lived key. We rejected it for two concrete reasons:\nKey wear-out. AES-GCM requires that the same key never be reused with the same nonce. A single key encrypting terabytes of data will eventually repeat a nonce, and a nonce repeat is catastrophic — it can leak the XOR of two plaintexts. One key cannot safely cover an unbounded amount of data. Rotation nightmares. If that one key is ever compromised, re-securing petabytes means reading, decrypting, and rewriting every physical block on disk. A rotation that should be a metadata operation becomes a full data migration. Envelope encryption solves both by splitting one key into two. The data itself is encrypted with a short-lived key, and that short-lived key is encrypted with the long-lived key. The long-lived key never touches the data directly, so it can be rotated cheaply.\nThe three pieces:\nContent Encryption Key (CEK) — also called a data key. A fresh, high-entropy 256-bit AES key generated randomly in client RAM, used for exactly one object. Key Encryption Key (KEK) — the client\u0026rsquo;s long-lived master key. The KEK never leaves the client device. It is never transmitted across the network and never stored in server configuration. Wrapped CEK — the client encrypts the 32-byte CEK under its KEK (AES-256-GCM.Seal(KEK, CEK)). The resulting small encrypted blob, the \u0026ldquo;envelope\u0026rdquo;, travels alongside the ciphertext. Figure 2: Cryptographic separation between Master KEK, Ephemeral CEK, and Wrapped CEK Because the KEK only ever wraps a 32-byte key, rotating it means re-wrapping one small envelope per object — constant work per object, regardless of how large the object is. That is the whole point of the split.\n2. The Ingest (PUT) Pipeline During an upload, encryption happens entirely before any byte crosses the network boundary. The client generates the CEK, wraps it, and streams the payload through an encrypting reader:\nFigure 3: Upload flow — client encrypts payload and wraps key before network transit The encryption helper in the crypto core looks like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 type Envelope struct { KeyID string `json:\u0026#34;key_id\u0026#34;` WrappedCEK []byte `json:\u0026#34;wrapped_cek\u0026#34;` Nonce []byte `json:\u0026#34;nonce\u0026#34;` } // ClientEncrypt streams plaintext into AES-256-GCM ciphertext func (c *Client) EncryptStream(plaintext io.Reader, masterKEK []byte) (io.Reader, *Envelope, error) { // 1. Generate random 32-byte CEK — unique per object, never reused cek := make([]byte, 32) if _, err := rand.Read(cek); err != nil { return nil, nil, err } // 2. Wrap CEK with Master KEK — the KEK encrypts the data key, not the data wrappedCEK, nonce, err := sealKey(masterKEK, cek) if err != nil { return nil, nil, err } // 3. Construct streaming AES-GCM cipher over payload — constant memory // regardless of object size cipherStream, err := newGCMEncryptReader(plaintext, cek) if err != nil { return nil, nil, err } env := \u0026amp;Envelope{KeyID: \u0026#34;default\u0026#34;, WrappedCEK: wrappedCEK, Nonce: nonce} return cipherStream, env, nil } The server receives only the raw AES-GCM ciphertext stream and the JSON envelope. It indexes the envelope in its metadata store and writes the ciphertext directly into the content-addressed blob store. At no point does a plaintext byte or the KEK reach the server.\n3. The Retrieval (GET) Pipeline When downloading, the process is inverted. The server can only ever hand back ciphertext; the client does all the decryption:\nFigure 4: Download flow — server returns ciphertext, client decrypts with private KEK The server streams the raw ciphertext blob from disk and includes the WrappedCEK envelope in the response headers. The client receives the envelope, extracts the wrapped key, and unwraps it locally using its private KEK (CEK = KEK.Open(WrappedCEK)). The client streams the ciphertext through an AES-GCM decrypting reader, verifying the authentication tag in real time as bytes arrive. A tampered or truncated payload fails authentication before the client ever consumes it, so the client never acts on unverified bytes.\n4. Trade-off Analysis Envelope encryption is not free. Here is what it buys and what it costs against the alternative of server-side encryption, where the server holds the keys:\nMetric / Dimension Server-Side Encryption (SSE-S3) Client-Held Envelope Encryption (E2EE) Server trust assumption Server holds keys; a rogue admin can read data Zero trust: server holds only ciphertext CAS deduplication Preserved (plaintext hashes match) Neutralized (a random CEK produces unique ciphertext) Key rotation overhead Requires re-encrypting all stored blobs Constant time per object: re-wrap only the 32-byte CEK Client CPU overhead Zero client CPU Client performs AES-NI encryption/decryption Data loss risk on key loss Recoverable via server/cloud KMS Permanent: a lost KEK means the data is irrecoverable The deduplication row is the one that hurts most. Because every object gets a fresh random CEK, two identical files encrypt to two different ciphertexts, so the store can no longer collapse them. We treat that as an acceptable price for true zero trust, and we claw some of it back with a separate confidential-deduplication protocol rather than by weakening the encryption.\n5. Security Invariants (🛡 Sentinel Mindset) In accordance with docs/CORE/STANDARDS.md:\n🛡 Sentinel (truncation defense). Streaming AES-GCM carries an authenticated integrity footer. If an attacker truncates a file in transit or deletes trailing disk blocks, the client\u0026rsquo;s GCM tag verification fails with syscall.EBADMSG, which prevents partial-read vulnerabilities where a client might act on a silently shortened object. 🛡 Fail-closed key semantics. If a client attempts to decrypt an object with an invalid KEK, decryption halts immediately. The server never attempts \u0026ldquo;fallback\u0026rdquo; plaintexts or heuristic recoveries — a failure to authenticate is always a hard stop, never a guess. References / Dig deeper Encryption implementation: src/crypto/envelope.go. Spec: openspec/changes/add-e2e-encryption. Pull requests: #779, #781. Sibling posts: 006: Pluggable Storage Backends, 014: Confidential Dedup via OPRF, 015: The Sentinel Sweep. ","permalink":"/posts/013-e2ee-envelope-encryption/","summary":"Envelope encryption with a client-held key: the server stores ciphertext and can\u0026rsquo;t read it — across S3 and native transports.","title":"Client-Held E2EE: Envelope Encryption Real"},{"content":"S3 requests are signed, and the signature is supposed to cover the request body. For a while ours covered the claim about the body but not the body itself. In parallel, we added a way for long-running operations to talk back to the client without the client having to poll. Two complementary pieces: closing an integrity gap, and opening a notification channel.\nThe Real Problem Signing. AWS Signature Version 4 (SigV4) is the scheme S3 clients use to prove a request is authentic and unmodified. Part of the signature is a hash of the request body, sent in the x-amz-content-sha256 header. A server that verifies the signature but never recomputes that hash is trusting a claim it never checks: an attacker — or a buggy proxy — can swap the body while leaving the signed header intact.\nThat was our gap. We accepted the signed header and did not verify the payload.\nNotifications. Some operations take a long time: completing a large multipart upload, deleting thousands of objects. Polling for status wastes requests and adds latency. We wanted a way for the server to push progress to a client over a single long-lived connection.\nWhy the Obvious Fixes Failed \u0026ldquo;Verify the body by reading it all first.\u0026rdquo; That means buffering the entire upload in memory before checking the hash — exactly the bounded-memory violation we refuse elsewhere. Verification has to happen while streaming.\n\u0026ldquo;Skip the hash and rely on TLS.\u0026rdquo; TLS protects the connection, not the application-layer contract. A client that signs a body is entitled to have that body checked, even against an honest-but-buggy intermediary.\n\u0026ldquo;Poll for status.\u0026rdquo; It works, but it is the wrong shape for a server that already holds the operation\u0026rsquo;s state. A push channel is cheaper for both sides.\nThe Solution Signed payload verification When a request arrives, the server captures the declared body hash — either the plain x-amz-content-sha256 value or the streaming variant STREAMING-AWS4-HMAC-SHA256-PAYLOAD, which signals that the body is framed in signed chunks (covered in the companion post). The body is then streamed through a hasher as it is read. When the body ends, the computed hash is compared with the declared one:\nMatch → the request proceeds. Mismatch → 400 with the XAmzContentSHA256Mismatch error code. Because the hash is computed during streaming, a multi-gigabyte body never has to be held in memory. The integrity check and the memory bound are not in conflict.\nServer-Sent Events Server-Sent Events (SSE) is a simple HTTP mechanism for one-way server-to-client streaming: the server responds with Content-Type: text/event-stream and writes a sequence of events until it closes the connection. The client reads them as they arrive; there is no polling.\nWe use it to report progress for asynchronous operations. Each event carries the operation\u0026rsquo;s identifier, its status, and its progress, so a client can follow a long-running job on one connection.\nHow We Verified A signed request whose body does not match its declared hash returns 400 XAmzContentSHA256Mismatch. A validly signed payload is accepted. The signed-chunk path verifies each chunk, preserving per-chunk integrity. An SSE client connects, receives events, and sees the stream close cleanly. Failure Modes Risk Guard A large body forcing a buffer The hash is computed during streaming; only one chunk is held A streaming request bypassing verification The streaming content-hash value selects the chunked verifier, not a skip path A client unable to tell when a job is done The server closes the stream when the operation completes When NOT to Use This Non-S3 traffic. These are S3 compatibility features; the SSE channel is for operations that actually run asynchronously. Requests with no body. There is no payload hash to verify on a GET. Engineering Standards (🛡 Sentinel) Per docs/CORE/STANDARDS.md: 🛡 Sentinel — verify what you sign, with no trust gaps and an honest mismatch error.\nReferences / Dig deeper Spec: openspec/changes/signed-payload-and-sse/. Issue: #776. Related posts: 008: S3 Gateway Core, 010: S3 Auth, Presigned URLs, and SigV4, 040: aws-chunked Streaming. ","permalink":"/posts/039-signed-payload-sse-s3/","summary":"SigV4 signed payload verification for S3 PUT/POST and Server-Sent Events for async operation notifications.","title":"Signed Payloads and Server-Sent Events for S3"},{"content":"A signed upload has a chicken-and-egg problem: the signature covers the body, but you cannot know the body\u0026rsquo;s hash until you have read it all, and reading it all means buffering it. For a multi-gigabyte object that is not an option. The S3 protocol\u0026rsquo;s answer is aws-chunked encoding, and this is how we implemented it.\nThe Real Problem SigV4 signs a hash of the request body. To verify that hash, a server would normally need the complete body. For a small object that is fine; for a multi-gigabyte upload it means holding the whole thing in memory before accepting a single byte — which breaks momo\u0026rsquo;s rule that memory stays bounded no matter how large the object.\naws-chunked is a payload encoding that solves this. Instead of one opaque body, the request carries a sequence of chunks, each preceded by its own size and its own signature. The server can verify and accept one chunk at a time, never holding more than a single chunk.\nWhy the Obvious Fixes Failed \u0026ldquo;Buffer it.\u0026rdquo; The whole point of a streaming store is not to.\n\u0026ldquo;Skip verification for large uploads.\u0026rdquo; That reintroduces exactly the trust gap we closed elsewhere: a signed request whose body is never checked.\n\u0026ldquo;Verify only at the end.\u0026rdquo; By then the bytes have been written. Detecting corruption after the fact means undoing work, and it means trusting the client for the duration of the upload.\nThe Solution: STREAMING-AWS4-HMAC-SHA256-PAYLOAD A request opts into chunked signing by declaring a special content hash:\n1 2 header: x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD body: [chunk-size][chunk-signature][chunk-data] ... The server decodes that framing and verifies each chunk before accepting it:\nEach chunk carries its own SigV4 signature. The signatures are chained: the signature of chunk N covers the hash of chunk N-1. Dropping, reordering, or truncating chunks breaks the chain, so the server notices at the first affected chunk. A mismatch returns 400 with XAmzContentSHA256Mismatch, at the exact chunk where the problem appears, and the connection is torn down. Memory stays bounded to one chunk. How We Verified An aws-chunked PUT streams a large body without running out of memory; the server holds a single chunk at a time. A corrupted chunk returns 400 at the offending chunk and the connection is closed. A standard, non-chunked PUT is unaffected. The behavior was cross-checked against aws-cli and the AWS4-HMAC-SHA256-PAYLOAD scheme. Failure Modes Risk Guard A corrupted chunk being accepted Each chunk is verified before its data is accepted; a mismatch returns 400 Reordered or dropped chunks Signatures are chained, so a broken chain is detected at the next chunk The whole body being buffered Only one chunk is held at a time Non-chunked uploads being affected The streaming content-hash value gates the chunked path When NOT to Use This Small uploads. If the whole body fits comfortably in memory, ordinary signing is simpler. Clients that do not send chunked framing. The server detects the streaming content-hash value; without it, the standard path applies. Engineering Standards (⚡ Bolt \u0026amp; 🛡 Sentinel) Per docs/CORE/STANDARDS.md:\n⚡ Bolt: bounded memory. The server never buffers more than one chunk. 🛡 Sentinel: per-chunk integrity with honest, precise mismatch errors. References / Dig deeper Spec: openspec/changes/aws-chunked-streaming/. Issue: #773. Related posts: 008: S3 Gateway Core, 010: S3 Auth, Presigned URLs, and SigV4, 039: Signed Payloads and SSE. ","permalink":"/posts/040-aws-chunked-streaming/","summary":"aws-chunked payload encoding streams large S3 uploads in signed chunks — no full-body buffering, per-chunk SigV4 integrity.","title":"AWS Chunked Streaming: Signed Payloads Without Buffering"},{"content":"momo ships two transports over one logical wire protocol: legacy TCP and QUIC over TLS 1.3. TCP is the mature, universally available path; QUIC is the newer one. QUIC runs over UDP and multiplexes many independent streams, so a lost packet on one stream no longer stalls the others — the problem TCP suffers from known as head-of-line blocking, where one delayed segment blocks every byte behind it. QUIC also offers 0-RTT: a returning client can send data in its very first flight, skipping a round trip of connection setup.\nThe split was an explicitly measured bet: TCP for high-bandwidth LAN chains, QUIC for lossy wide-area fan-out where its stream independence and fast reconnection pay off.\nWhat the transport had to get right Transport evolution: TCP vs QUIC A decade of fixes concentrated on correctness at the socket boundary:\nHandshake. Challenge-response authentication with a pre-padded token, so the reply always occupies a fixed size and leaks no length information, plus enforced freshness (later reduced to a plaintext timestamp check with revocation). ACK framing. A fragile acknowledgement scheme that relied on a 5 ms timeout was replaced with a fixed-length 4-byte acknowledgement. The old design could desync the stream when a read returned a partial frame; a fixed-size frame cannot. QUIC listener identity. The daemon previously ignored its configured TLS certificate and CA pool, so a QUIC listener presented a self-signed default instead of the operator\u0026rsquo;s certificate. The QUIC server now binds the configured certificate, matching the TCP/TLS path. Idempotent close. QUIC Close() once spawned a fire-and-forget goroutine with a magic delay; it is now guarded by a sync.Once, so repeated closes are safe and immediate. Read deadlines. An IdleTimeoutConn wrapper rolls a read deadline so idle sockets cannot stall the daemon forever. This was a direct seed of the Bolt deadline-amortization work described below. ⚡ Bolt lens The single most Bolt-visible artifact here is bitwise deadline amortization: SetDeadline syscalls were cut by roughly 98% in hot paths by computing deadline windows in bulk instead of per-operation. The same pooling-and-buffer principles apply to the transfer payload ring. The full story is in 024.\nSee docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Wire protocol doc: docs/REFERENCE/PROTOCOL.md. Spec: openspec/changes/add-quic-protocol. Pull requests: #763 (QUIC listener identity), #818 (idempotent close). Chained reads: 002 → 011 → 024. ","permalink":"/posts/003-transport-tcp-to-quic/","summary":"Momo\u0026rsquo;s transport layer grew from TCP to a QUIC/TLS 1.3 fan-out — fixing handshake, ACK framing, deadlines, and TLS identity along the way.","title":"Transport Evolution: TCP, QUIC, and the Wire Protocol"},{"content":"On 2026-08-04 we pointed a mechanized audit at an early momo and let it run deep: 76 findings, filed as a chain of issues from #593 onward. It was uncomfortable. It was also the moment the 🛡 Sentinel mindset was born — the idea that a security flaw is not a special category of work to schedule later, but a first-class engineering item with a reproducible failure, an auditable fix, and a regression test that keeps it fixed.\nWhat the sweep found The findings clustered into a handful of classes, each a textbook way for untrusted input to cross a boundary it should never cross.\nCRLF injection. CRLF stands for carriage-return / line-feed, the two bytes that terminate a line in HTTP and many text protocols. If attacker-controlled bytes containing \\r\\n reach a header or a log line, they can forge a new line — splitting one response into two, or smuggling fake fields into a log. We found untrusted bytes flowing into metadata and metrics paths. HTTP request smuggling. This is the classic ambiguity attack: when a request carries both a Transfer-Encoding header and a Content-Length header, two servers in the chain can disagree about where the request ends. An attacker crafts a request that the front gateway reads one way and the backend reads another, letting them \u0026ldquo;smuggle\u0026rdquo; a second request past the gateway\u0026rsquo;s checks. We found this in the S3 gateway, and it was the most severe issue of the sweep. Path traversal. When a filename is used to build a filesystem path, a value containing .. can climb out of the intended directory and read or write somewhere it should not. Our first guard flagged false positives — filenames that merely contained .. as part of a longer name were rejected even though they were harmless. The fix was to flag only a complete path component that is exactly .., not any occurrence of the characters. Resource leaks. The long tail: request bodies never closed, a wait-group counter incremented after the wait had already begun, and a lease that could reach zero quorum during a network partition. Individually minor; collectively a slow bleed of file descriptors and goroutines that would eventually take a node down. Process: issuance → regression → hardening The value was not the list; it was the discipline applied to each entry. Every finding got a reproducible acceptance case first — a test that fails against the vulnerable code — then a fix, then, where it made sense, a regression contract test so the exact class of bug could never return unnoticed. The offensive toolkit we used (fuzzing for path traversal plus hand-written exploits) was kept alongside the code and documented, so the next audit starts from where this one ended rather than from scratch.\nThe mindset distilled Three rules fell out of the sweep and became permanent:\nFail closed, loudly. When input is suspicious, refuse it with an honest error rather than papering over it with a fake success. A server that returns \u0026ldquo;not implemented\u0026rdquo; for something it cannot safely do is more trustworthy than one that returns a 200 and hopes. We map internal errors to POSIX syscall constants so callers get a truthful, machine-readable reason. Trust invariants are never bypassed. The checks that matter — CRLF sanitization, crypto, integrity verification — live in the auditable core and cannot be routed around by a plugin or an alternate code path. If a fast path skips a trust check, it is not a fast path, it is a vulnerability. Zero-crash handling. Panics in request handling become recovered errors, not process death; unbounded readers get bounds; every goroutine is verified not to leak. A storage node that crashes on malformed input is a denial-of-service waiting to happen. See docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Pentest toolkit and write-ups: docs/GUIDES/PENTESTING.md, pentest/. Tracking issues: #593, #811, #859, #889. The trust-critical core is governed by the Sentinel constraints in openspec/config.yaml. Sibling posts: 007: At-Rest Integrity and GC, 010: S3 Auth, Presigned URLs, and SigV4, 011: S3 HTTPS/TLS Enforcement, 013: Client-Held E2EE, 014: Confidential Dedup via OPRF, 017: Scatter-Gather Lease Quorum, 026: Metrics and Observability, 037: Zero-Crash Hardening. ","permalink":"/posts/015-sentinel-security-audit/","summary":"A 76-deep audit (authed via Sentinel) surfaced CRLF injection, smuggling, traversal and path leaks — the pattern that forged momo\u0026rsquo;s security mindset.","title":"The Sentinel Sweep: An Audit-Driven Security Journey"},{"content":"momo originally assumed that blob bytes and blob metadata lived on the same machine, in the same local directory. That assumption was fine until the project grew an S3-compatible gateway and remote replication. At that point the placement of bytes had to diverge from the node holding the metadata — and we did not want that divergence to require rewriting the storage core.\nThe storage layer therefore exposes one interface and a single configuration switch. A backend is simply the thing that stores and retrieves opaque blob bytes. A seam is a narrow, compile-time interface we can substitute without a plugin runtime: no shared library, no RPC hop, no serialization boundary.\nThe Store interface stays stable, and a [storage] backend config value chooses the implementation:\nBackend Notes local filesystem path, default nfs shared filesystem volume s3 remote object store over SigV4, TLS-gated (see 011) raw block-device direct I/O Local metadata stays per-node; only the blob bytes route through the chosen backend.\nPluggable storage backends architecture The seam, not the plugin Backends are compiled in and selected by declarative config — the embryo of the project\u0026rsquo;s seam-over-plugins principle. There is no external .so to load and no RPC plugin protocol to negotiate. The blob interface is small and stable (PutBlob/GetBlob/DeleteBlob/stat/path), and each backend implements it with its own guarantees. That keeps the trust boundary explicit and the binary self-contained.\nThe trap this avoids A single hard-coded local blob store was the original design. Once remote replication arrived, \u0026ldquo;metadata here, bytes there\u0026rdquo; had to become a configuration change rather than a rewrite. Pluggable backends are what made that possible: the CAS core keeps naming blobs by hash and recording metadata locally, while the bytes can live on NFS, in an S3 bucket, or on a raw block device.\n⚡ Bolt lens The GetBlobPath guard rejects non-local backends, because \u0026ldquo;a filesystem path\u0026rdquo; is a concept only the local backend can honor. Combined metadata reads collapse three separate key/value views into one, cutting write-path transactions and CPU (perf work referenced in 024). The encryption layer later wrapped the store so end-to-end encryption could sit under the backend seam transparently (013) — a seam earning its keep. See docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Configuration reference (the [storage] section): docs/GUIDES/CONFIGURATION.md. Spec: openspec/changes/add-pluggable-storage; tracking issue #820. Related: 004 → this post → 008, 011, 013. ","permalink":"/posts/006-pluggable-storage-backends/","summary":"The \u003ccode\u003e[storage] backend\u003c/code\u003e seam swaps blob storage (local, nfs, s3, raw) behind one Store interface, keeping CAS metadata local.","title":"Pluggable Storage Backends: local, nfs, s3, raw"},{"content":"An aws-cli upload and a momo-client upload look almost identical on the wire: both are S3 PUT requests. They are not, however, equally capable of keeping your data safe. momo can replicate a write several ways, and two of those ways put the fan-out work on the client. A tool like aws-cli, rclone, or boto3 speaks only the standard S3 API, so it cannot do that fan-out. When one of these \u0026ldquo;external\u0026rdquo; clients hit a node whose active replication mode assumed a momo-aware client, the server did the worst possible thing: it stored a single copy and reported success.\nThe Real Problem Replication is keeping multiple copies of each object on different machines; the replication factor is how many copies. In momo, a write can be replicated in one of four shapes. Two of them are client-side: the client opens the parallel connections to every replica itself, so the server relays no bytes. The other two are server-side: the receiving node fans the write out to its peers.\nAn external S3 client can only ever send one stream. It has no idea which nodes hold replicas and no protocol to reach them. So a client-side mode is simply not available to it.\nThe failure was in how the server told the two client kinds apart. momo clients add a private header naming the mode they intend, plus a timestamp header. The server used those to decide whether a request came from a momo client or from a peer forwarding a write. An external client sends neither, but it does send the standard X-Amz-Date timestamp. Parsing that succeeded, and because the value was not the sentinel \u0026ldquo;dummy\u0026rdquo; timestamp used for forwarded traffic, the server concluded it was a peer forward and applied no replication at all. A single copy landed, with a 200 OK on top.\nWhy the Obvious Fixes Failed \u0026ldquo;Treat a missing header as an error.\u0026rdquo; That would reject every legitimate external client — the exact compatibility we were trying to keep.\n\u0026ldquo;Make external clients speak the momo handshake.\u0026rdquo; We do not control aws-cli, rclone, or the thousands of S3 SDKs. Compatibility means meeting them where they are.\n\u0026ldquo;Just force server-side replication globally.\u0026rdquo; That would throw away client-side replication for momo clients, giving up the zero-relay-bandwidth win for the traffic that can actually use it. The mode must depend on who is asking, not on the cluster\u0026rsquo;s global setting.\nThe Solution: Per-Request Downgrade We taught the server to recognize a request that has no momo mode header and treat it, correctly, as an external client. Then, instead of refusing the write, it steps the mode down to the next server-side alternative.\nA new config key lists which modes are client-side and therefore unusable by an external client:\n1 2 3 4 # momo.conf # Modes that require the client to fan out to replicas. An external # S3 client cannot do this, so the server downgrades past them. client_side_replication_modes = 3 When an external request arrives:\nDetect — no momo mode header means external client; the timestamp is forced to the sentinel value so the peer-forward path is never taken. Downgrade — walk the configured replication order forward to the first server-side mode (for example, client-side primary-splay to server-side splay). Keep it local — the downgrade is per transaction. The cluster\u0026rsquo;s global mode is untouched, so momo clients on the same node still get client-side replication. Stay configurable — if we ever add another client-side mode, we add it to the config list; no code change. The key design choice is per-transaction. A global mutation would have been simpler to write but would have flipped the mode for concurrent momo clients mid-flight. Because the decision is computed per request, two clients with different capabilities can share one node and each get the right behavior.\nHow We Verified An aws-cli PUT to a node running client-side primary-splay downgrades to server-side splay and the object is replicated. A momo client using the sentinel timestamp keeps client-side primary-splay unchanged. Concurrent momo and external clients each get their correct mode while the global mode stays at primary-splay. Config validation defaults the client-side mode list to primary-splay. What Could Go Wrong Risk Guard A future client-side mode added without updating config The downgrade walk is driven entirely by the config list, so an unlisted mode would be treated as server-side; the list is validated at startup Misidentifying a peer forward as an external client The sentinel-timestamp rule is only applied when the momo mode header is absent, preserving the peer path Global mode accidentally mutated The downgrade returns a per-request value; the stored cluster mode is never written on this path When NOT to Use This momo-aware clients. They should keep client-side replication; the downgrade only triggers when the mode header is missing. Clusters with a single node. With no replicas there is nothing to downgrade to, and the mode is irrelevant. Engineering Standards (⚡ Bolt \u0026amp; 🛡 Sentinel) Per docs/CORE/STANDARDS.md:\n⚡ Bolt: the config list is parsed with the same zero-allocation CSV parser used elsewhere, and the downgrade is a single integer walk — no allocation on the request path. 🛡 Sentinel: fail-closed. A missing mode header is treated as the least capable client, never as permission to skip replication, so the silent single-copy outcome cannot recur. References / Dig deeper Spec: openspec/changes/add-external-client-replication/. Issue: #258. Related posts: 016: P2P Gossip and SWIM, 018: Adaptive Scaling and Peer Quality. Replication strategies: 002. ","permalink":"/posts/030-external-s3-client-replication-downgrade/","summary":"External S3 clients like aws-cli don\u0026rsquo;t send momo headers — the server detects this and downgrades client-side replication modes to server-side alternatives, ensuring replication never silently drops.","title":"External S3 Client Replication Downgrade: When aws-cli Can't Fan Out"},{"content":"We started with a traditional approach: a central coordinator assigning replicas to nodes. It worked for a three-node cluster — until it didn\u0026rsquo;t.\nThe Real Problem The breaking point came during a network partition. The coordinator became a single point of failure. When it went down, no new objects could be placed — the entire cluster stopped accepting writes. Worse, when the coordinator recovered, its view of the cluster was stale. It assigned replicas to nodes that had already failed, quietly sending data into a hole.\nWe asked: what if every node could compute placement independently, with zero coordination?\nThe insight came from the storage design itself. Because momo is content-addressed (004), every object already has a SHA-256 content hash. What if that hash is the placement key? That would make placement a pure function — same hash and same cluster map, same replica set, on every node, with no network call.\nHere, placement means deciding which nodes hold which replicas of an object, and a failure domain is a group of machines likely to fail together (a rack, a zone, a power feed). Keeping replicas in distinct failure domains is what prevents a single rack outage from taking out every copy.\nWhy Obvious Solutions Failed Central coordinator with heartbeat\n1 2 3 4 5 6 // REJECTED: Single point of failure type Coordinator struct { nodes map[string]*Node mu sync.Mutex } func (c *Coordinator) Assign(key string, n int) []Node { ... } The coordinator becomes a bottleneck and a single point of failure. Heartbeats add latency and still do not resolve split-brain.\nConsistent hashing (ketama/ring)\n1 2 // REJECTED: Doesn\u0026#39;t support weighted nodes properly // Adding/removing nodes causes massive reshuffling Standard consistent hashing treats every node as equal. Our nodes have different capacities — disk, network, zone — so we needed weighted placement.\nFull RADOS CRUSH\n1 2 // REJECTED: Overkill for our scale // Hierarchical buckets, straws, recursive descent — too complex Ceph\u0026rsquo;s CRUSH models enormous hierarchical failure domains. We needed a lite version: a flat topology, deterministic output, constant CPU per lookup.\nThe Solution — Weighted Rendezvous Hashing Core insight: every node computes the same replica list from the object hash — no coordinator, no RPC, no central state. The technique is weighted rendezvous hashing: score each candidate node with a hash of the object key combined with the node\u0026rsquo;s identity, multiply by the node\u0026rsquo;s weight, and take the highest scores.\nOverview CRUSH-lite placement algorithm overview Detail: Scoring Algorithm 🔍 Expand for scoring detail\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 // placementKey is the object\u0026#39;s SHA-256 content hash (already computed on ingest) func scoreCandidates(key BlobID, candidates []Node, n int) []Node { // 1. Score each candidate: hash(key ⊗ node.id) × weight type scored struct { node Node score uint64 } var ranked []scored for _, node := range candidates { if node.Weight \u0026lt;= 0 { continue } // Skip zero-weight nodes h := hash(key ⊗ node.ID) // Deterministic hash s := fold52(h) * uint64(node.Weight) ranked = append(ranked, scored{node: node, score: s}) } // 2. Stable sort by score (descending) sort.SliceStable(ranked, func(i, j int) bool { return ranked[i].score \u0026gt; ranked[j].score }) // 3. Top-N become replicas out := make([]Node, 0, n) for i := 0; i \u0026lt; n \u0026amp;\u0026amp; i \u0026lt; len(ranked); i++ { out = append(out, ranked[i].node) } return out } Key annotations:\nhash(key ⊗ node.ID): combining the content hash with the node ID makes the score deterministic and uniformly distributed. Every node computes the same number. fold52(): folds the 64-bit hash down to a 52-bit value so it fits exactly in a float64 mantissa, avoiding the precision loss that flipped replica order before this fix (#873). sort.SliceStable: when two nodes score exactly equal, a stable sort keeps their original order so different nodes do not disagree about the ranking (#872). node.Weight: capacity-aware scoring — a node with twice the weight tends to receive proportionally more replicas. Failure Domain Filter After selecting the top N by score, we enforce failure-domain diversity so two replicas do not land in the same rack or zone:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 func filterFailureDomains(replicas []Node) []Node { used := make(map[string]bool) for i, r := range replicas { if !used[r.FailureDomain] { used[r.FailureDomain] = true continue } // Find next-best from different domain for j := i + 1; j \u0026lt; len(candidates); j++ { if !used[candidates[j].FailureDomain] { replicas[i] = candidates[j] used[candidates[j].FailureDomain] = true break } } } return replicas } Principle Callout Pattern: Content-Hash-Based Placement Use the object\u0026rsquo;s content hash as the placement key. Score each node with hash(key ⊗ node.id) × weight, sort, take the top N. Zero coordination.\nApplies when: content-addressed storage, deterministic placement is required, node capacities are heterogeneous. Doesn\u0026rsquo;t apply: mutable objects, or when placement must be controlled by an external policy.\n1 2 3 4 5 // ✅ GOOD: Content-addressed, deterministic func place(key BlobID) []Node { return score(key, nodes) } // ❌ AVOID: Mutable objects, external placement policy func place(key string, policy PlacementPolicy) []Node { ... } Verification Property Test Result Determinism Same input → same output across nodes ✅ Weight proportionality 2× weight → ~2× replicas ✅ Stability Tied scores never flip order ✅ (fixed #872) Float precision 52-bit mantissa, no drift ✅ (fixed #873) Failure domains Replicas in different racks/zones ✅ Production: under 100k req/s, placement adds under 50 µs of latency. Zero coordinator overhead.\nFailure Modes Risk Mitigation Hash collision (SHA-256) 2^256 space — practically impossible; monitor for research breaks Zero-weight nodes Explicit skip in scoring loop Tied scores sort.SliceStable preserves insertion order Float drift fold52() limits to a 52-bit mantissa Split-brain Failure domain filter ensures diversity When NOT to Use This Mutable objects — a content change means a new hash and therefore a new placement; use versioning instead. External placement policy — when placement must be controlled by an operator or scheduler outside the storage layer. Non-content-addressed storage — there is no content hash to key on. See docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Design doc: docs/REFERENCE/CRUSH.md. Spec: openspec/changes/r1-failure-domains. Pull requests: #872 (stable sort), #873 (52-bit fold). Upstream: 004. Downstream: 019, 024. ","permalink":"/posts/005-crush-placement/","summary":"Weighted rendezvous hashing picks replicas deterministically and peer-equally without a central allocator — later hardened for failure domains. Teaches: content-hash-based placement, weighted rendezvous hashing, deterministic stable sort, failure domain filtering.","title":"CRUSH-lite: Deterministic Weighted Placement"},{"content":"A networked object store lives at the mercy of its inputs. Every byte that arrives over a socket is attacker-controlled until proven otherwise, and a single malformed packet should never be able to take a node offline. We learned this the hard way: early on, a panic on an unexpected nil value would unwind the whole process, and a crashed storage node means unavailable data for everyone pointed at it. Availability is the feature.\nThis post documents the defensive patterns we applied across the codebase to make crashes on bad input, silent corruption, and resource leaks structurally difficult rather than merely unlikely.\nThe problem Production services face a recurring set of failure modes, and they compound:\nA panic on a nil or unexpected value takes down the whole process, not just the request that triggered it. An unchecked numeric conversion silently wraps or truncates, turning a large count into a small one and corrupting data without any error. A goroutine or connection that is not released on an error path leaks a little each time, until the process runs out of descriptors. Two goroutines touching shared state without synchronization race, and the result depends on timing — the hardest class of bug to reproduce. None of these are exotic. They are the ordinary ways networked Go services fail.\nThe patterns 1. Nil-safety We check explicitly for nil before dereferencing anything that came from outside the process or from a decoder. We also defensively copy slices and maps at ownership boundaries — when a buffer is handed from one subsystem to another, the receiver gets its own copy rather than a view that the sender might later mutate. An aliased slice that the owner reuses is a data race waiting to happen.\n2. Numeric safety Every narrowing conversion (a larger integer type into a smaller one, or a byte into a rune) is checked for overflow before it happens. Float comparisons use an epsilon tolerance rather than ==, because floating-point equality is almost never what the author means. And allocations that take a length from untrusted input are validated first — make([]T, n) with an attacker-supplied n is an out-of-memory bomb, so n is bounded before it is used.\n3. Resource lifecycle Cleanup is deferred so it runs on every return path, including error returns, and we are careful that a defer inside a loop does not accumulate until the function exits. Every connection and socket is released even when a panic unwinds the stack, which is where leaks most often hide: the happy path is well tested, the panic path is not.\n4. Panic recovery at boundaries Every goroutine and every transport boundary installs a small recovery shim. A panic is converted into an error at the edge, so a malformed request returns a failure to the caller instead of killing the daemon:\n1 2 3 4 5 6 defer func() { if r := recover(); r != nil { log.Printf(\u0026#34;CRITICAL: recovered from panic: %v\u0026#34;, r) *errno = syscall.EIO } }() The rule is simple: panics become errors, never process death. We put the recovery at the boundary — the point where untrusted input enters and where a goroutine is spawned — rather than scattering it everywhere, so normal code stays readable and still fails fast during development.\n5. Concurrency safety Shared mutable state is guarded by a mutex; hot counters use atomic operations instead, to avoid lock contention on the paths that run constantly. The whole test suite runs under Go\u0026rsquo;s race detector, and \u0026ldquo;clean under -race\u0026rdquo; is a release condition, not a nice-to-have.\nWhere it applies The patterns are not specific to one package — they are applied everywhere input crosses a boundary:\nLayer Focus Transport Wire parsing, handshake, connection lifecycle Storage Metadata access, blob I/O, garbage collection, scrubbing Peer-to-peer Gossip, failure detection, peer map, scatter-gather Server Daemon loop, replication, query handlers Common Config parsing, placement, hashing How we verified Defensive code is easy to write and hard to trust, so we measured it:\ngo test -race ./... reports zero data races. Panic-injection tests deliberately trigger panics at transport and storage boundaries and assert the process survives. A goroutine-leak check verifies that no test leaves background goroutines running. Chaos runs kill nodes, partition the network, and inject disk errors, and assert no crashes and no data corruption. When NOT to use this Panic recovery at a boundary is a safety net, not a license to ignore errors. Swallowing a panic deep inside logic hides bugs; recovering at the edge and returning a truthful error is the point. Likewise, not every allocation needs a pre-check — bound the ones whose size comes from untrusted input, and leave internal, small, fixed allocations alone. Defensive copying has a cost; apply it at ownership boundaries, not on every value.\nStandards Per docs/CORE/STANDARDS.md: 🛡 Sentinel (fail-closed, panic-to-error, no silent corruption), ⚡ Bolt (bounded allocations, zero-copy defensive patterns).\nReferences / Dig deeper Spec: openspec/changes/zero-crash-hardening. Tracking issue: #135. Sibling posts: 004: The Content-Addressable Store, 015: The Sentinel Sweep. ","permalink":"/posts/037-zero-crash-hardening-patterns/","summary":"Systematic defensive coding: nil-safety, numeric overflow guards, resource lifecycle discipline, panic recovery at every boundary, and concurrency safety under -race.","title":"🛡 Zero-Crash Hardening: Defensive Patterns for a Networked Object Store"},{"content":"Most distributed storage architectures lock their replication topology into their core design: Ceph peers a primary OSD with replicas, Cassandra has clients fan out to coordinators, and HDFS pipelines a daisy-chain of DataNodes. Each topology optimizes for one constraint at the expense of the others — low latency, low network egress, or even bandwidth distribution.\nmomo was designed around an alternative premise: why hardcode a single replication topology when network conditions, hardware heterogeneity, and client types shift over time? The Polymorphic Engine lets a running momo cluster change its active replication strategy without restarting daemons or dropping in-flight connections.\nHere, replication simply means keeping multiple copies of each object across different machines, and replication factor (written R below) is how many copies we keep. The interesting question is the shape of the write path that creates those copies.\nFigure 1: High-level overview of momo\u0026rsquo;s four replication strategies 1. The Core Topologies: Splay vs. Chain momo formalizes four replication modes:\n1 2 3 4 5 6 const ( ReplicationNone = 0 // Local storage only (no network replication) ReplicationSplay = 1 // Primary server fans out concurrently to replicas ReplicationChain = 2 // Pipelined daisy-chain (N0 -\u0026gt; N1 -\u0026gt; N2) ReplicationPrimarySplay = 3 // Client streams to all replicas concurrently ) Figure 2: Concurrent server fan-out (Splay) vs sequential pipelining (Chain) Splay Replication (Mode 1) The client sends one stream to the primary node, call it the first node. That node writes the payload to its local blob store while simultaneously fanning out concurrent streams to the secondary replicas. The shape is a fan or \u0026ldquo;splay\u0026rdquo;: one inbound stream becomes several outbound streams.\nLatency: one network hop — the fastest write latency. Cost: the primary node carries (R - 1) × payload of network egress bandwidth, creating a bottleneck on the primary\u0026rsquo;s network card. Chain Replication (Mode 2) The client streams to the first node, which streams directly to the second, which streams to the third. Each node forwards to exactly one successor.\nLatency: R sequential hops — higher write-completion latency, because the write is not durable until it reaches the tail. Cost: every node spends exactly one unit of ingress and one unit of egress bandwidth. No single node\u0026rsquo;s egress saturates, which makes chain the right choice for large sequential transfers across bandwidth-constrained links. 2. Primary-Splay: Client-Side Fan-Out In Primary-Splay (Mode 3), the momo client computes the replica set locally (via CRUSH-lite, 005) and connects directly to all R nodes in parallel.\nFigure 3: Client-side concurrent fan-out eliminates server relay bandwidth Zero server relay bandwidth. Daemons spend no egress copying bytes to peers; the client\u0026rsquo;s network carries the fan-out. Dynamic downgrade invariant. When a non-momo client (say aws-cli or curl) connects, it sends only a single stream. The receiving server detects that the peer is external and automatically steps down from Primary-Splay to Splay, so the write still gets replicated instead of silently landing as one copy. 3. The Polymorphic Controller Replication modes are not static configuration flags; they are dynamic cluster states. One node — node 0 — is the single authoritative controller. It is the only node allowed to decide and broadcast a topology change, which prevents two nodes from racing to reconfigure the cluster.\nFigure 4: Real-time telemetry drives dynamic topology transitions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 func (s *Server) ChangeReplicationMode(newMode int) error { if s.id != 0 { return fmt.Errorf(\u0026#34;only node 0 can orchestrate replication mode changes: %w\u0026#34;, syscall.EPERM) } // 1. Broadcast ModeChange RPC to all cluster daemons for _, peer := range s.cluster.Peers() { if err := s.sendModeChangeRPC(peer, newMode); err != nil { return fmt.Errorf(\u0026#34;failed to shift peer %d: %w\u0026#34;, peer.ID, err) } } // 2. Atomically update local state s.currentMode.Store(int32(newMode)) log.Printf(\u0026#34;Polymorphic Engine: Successfully shifted cluster replication mode to %d\u0026#34;, newMode) return nil } When a node receives a mode change, it updates an atomic integer holding the current mode. Ongoing in-flight streams keep using the mode they started with and run to completion; newly accepted connections immediately adopt the new topology. The result is a topology shift with zero downtime and no dropped transfers.\n4. Tradeoff Analysis Mode Write Latency Primary Node Egress Inter-Node Traffic Client Requirements None (0) Lowest (no hops) Zero Zero Works with any client Splay (1) Low (one hop) High ((R-1) × payload) High Works with any client Chain (2) High (R hops) Low (1 × payload) Distributed evenly Works with any client Primary-Splay (3) Low (one hop) Zero Zero Requires momo-aware client The trade-off is genuine: Primary-Splay is the cheapest and fastest, but it requires clients that speak momo\u0026rsquo;s protocol. Splay is universal but makes the primary\u0026rsquo;s NIC the bottleneck. Chain is the most bandwidth-fair but the slowest to acknowledge. The Polymorphic Engine exists so the cluster does not have to pick one forever.\n5. Engineering Standards (⚡ Bolt \u0026amp; 🛡 Sentinel) In accordance with docs/CORE/STANDARDS.md:\n⚡ Bolt: strategy selection is a lightweight control-plane decision. The data plane uses pooled stack buffers and bounded semaphores so replication forwarding adds no heap allocations. 🛡 Sentinel: in-flight streams are never forcibly interrupted during a topology shift, and single-source authority on node 0 prevents split-brain mode oscillations. References / Dig deeper Replication design doc: docs/REFERENCE/REPLICATION_STRATEGIES.md. Spec: openspec/changes/add-replication-durability-floor. Sibling posts: 001: Origin and Genesis, 003: TCP → QUIC Transport, 021: R3 Write Durability and Quorum. CRUSH-lite placement: 005. ","permalink":"/posts/002-replication-strategies-polymorphic/","summary":"Chain, Splay, and Primary-Splay replication — and the metrics-driven controller that switches between them at runtime.","title":"Replication Strategies and the Polymorphic Engine"},{"content":"Our first object store named files by random UUID, kept metadata in a database, and treated deduplication as a cleanup job bolted on afterward. That design held up until it didn\u0026rsquo;t, and the failure taught us the most important lesson in the storage core: the name of an object should be derived from its content, not assigned arbitrarily.\nThe Real Problem A customer uploaded the same 50 GB dataset three times under different names. We stored 150 GB. The nightly dedup job took six hours and still missed duplicates that landed on different nodes. Worse, when a bit flipped on disk and silently corrupted a file, we had no way to detect it without a separate checksum index that could itself drift out of sync.\nWe asked: what if the name was the checksum?\nThat idea is content-addressable storage (CAS): instead of inventing an identifier, hash the object\u0026rsquo;s bytes and use the hash as its address. SHA-256 is the hash momo uses. Two identical byte streams necessarily produce the same hash, so identical uploads collide on purpose and deduplicate for free.\nWhy Obvious Solutions Failed Application-level dedup (separate index)\n1 2 3 // REJECTED: index grows unbounded, eventual consistency = stale dedup // Also: race between upload and dedup scan = double storage temporarily type DedupIndex map[string]string // hash -\u0026gt; blobID Content hash as metadata only\n1 2 3 4 5 6 // REJECTED: still need separate integrity check on read // \u0026#34;Verify-on-read\u0026#34; becomes a separate code path = bugs type ObjectMetadata struct { Name string Hash string // just metadata, not the key } File-system level dedup (ZFS/Btrfs)\n1 2 // REJECTED: not portable, no cross-node dedup, opaque to application // Can\u0026#39;t enforce \u0026#34;validate → write\u0026#34; at application level Each of these keeps the identifier and the integrity check as two separate things that can disagree. Content addressing fuses them.\nThe Solution — Content Addressing Pivotal decision: store objects by their SHA-256 content hash. Names become metadata pointing at a blob address.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 // The write chokepoint — ALL paths funnel through this func (s *Store) Write(data io.Reader) (BlobID, error) { // 1. Hash on ingest (streaming, zero-copy) h := sha256.New() tee := io.TeeReader(data, h) // 2. Write to blob store (backend-agnostic) blobID := BlobID(h.Sum(nil)) // hash IS the key if err := s.backend.Put(blobID, tee); err != nil { return \u0026#34;\u0026#34;, err } // 3. Metadata points to blob (dedup by construction) return blobID, nil } Key insight: the hash is the key. Two identical uploads yield the same hash, therefore the same blob ID, therefore the same stored bytes. Dedup happens by construction — no background job, no window where duplicates coexist.\nPrinciple Callout Pattern: Content-Addressable Write Chokepoint Hash on ingest → hash is the key → validate → write. Every write path (native PUT, S3 gateway, replication, FUSE mount) funnels through ONE function.\nApplies when: you need dedup, integrity, and a single trust boundary. Doesn\u0026rsquo;t apply: mutable objects, append-only logs, or any case where the content is not immutable.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ┌─────────────────────────────────────────────────────────────┐ │ WRITE CHOKEPOINT │ ├─────────────────────────────────────────────────────────────┤ │ Native PUT │ S3-Compatible Gateway │ Replication │ FUSE Mount │ └──────────────┴────────────────────────┴───────────────┴──────────────┘ │ ▼ ┌───────────────────────┐ │ SHA-256 Streaming │ ← Zero-escape (see 024) │ Hash = Blob Key │ │ Validate → Write │ ← Sentinel: path traversal, └───────────────────────┘ CRLF injection checks │ ▼ ┌───────────────────────┐ │ Pluggable Backend │ ← local / nfs / s3 / raw │ CRUSH-lite Placement │ ← 005, 006 └───────────────────────┘ ⚡ Bolt + 🛡 Sentinel Lens Bolt: Zero-Escape SHA-256 Hashing 1 2 3 4 5 6 7 8 9 10 // Stack-allocated hash buffer — zero allocation per object var hashBuf [sha256.Size]byte func hashStream(r io.Reader) (BlobID, error) { h := sha256.New() if _, err := io.Copy(h, r); err != nil { return \u0026#34;\u0026#34;, err } return BlobID(h.Sum(hashBuf[:0])), nil // hashBuf[:0] = stack slice } h.Sum normally allocates a fresh slice for the digest. Passing a stack-array slice with length zero lets the hash write into memory that never escapes to the heap — zero bytes allocated per object. See 024 for the full zero-escape pattern.\nSentinel: Path Traversal \u0026amp; Injection Prevention The chokepoint is the natural place for security checks, because every write passes through it:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 func (s *Store) Write(data io.Reader) (BlobID, error) { // 1. Hash first (content determines identity) blobID, err := hashStream(data) if err != nil { return \u0026#34;\u0026#34;, err } // 2. Validate blob ID format (no path traversal possible) if !blobID.Valid() { return \u0026#34;\u0026#34;, ErrInvalidBlobID } // 3. Write — backend sees only hash, never user input return blobID, s.backend.Put(blobID, data) } Because the backend is handed a validated hash and never a user-supplied name, there is no filename for an attacker to traverse with ../ or to smuggle CRLF into. See 015 for the CRLF injection, request smuggling, and raw-store traversal findings that hardened this path.\nVerification Property Test Result Dedup correctness Upload same content 100x → 1 blob ✅ Integrity Flip 1 bit on disk → read fails ✅ Write throughput 50k obj/s, 2.4 KB/alloc → 0 B/alloc ✅ Cross-backend local, nfs, s3, raw all dedup ✅ Failure Modes Risk Mitigation Hash collision (SHA-256) 2^256 space — practically impossible; monitor for research breaks Backend corruption Verify-on-read hashes every block on GET Metadata/Blob drift GC reconciles metadata with blob files (007) When NOT to Use Content Addressing Mutable objects — a content change means a new hash and a new blob; use versioning instead. Append-only logs — these are written sequentially, not looked up by content. When content isn\u0026rsquo;t immutable — e.g. user-editable documents that change in place. See docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.\nReferences / Dig deeper Spec: openspec/changes/add-cas-storage. Pull request #838; tracking issue #820. Parent: 001: Origin and Genesis. Edges: 005, 006, 007, 022. Further reading: 015, 024. ","permalink":"/posts/004-cas-content-addressable-store/","summary":"Identify every object by its SHA-256 content hash — server-side dedup, verify-on-read, and a path that later mounted a filesystem. Teaches: content addressing principles, single write chokepoint, Bolt/Sentinel patterns.","title":"CAS: Content-Addressable Storage and Deduplication"},{"content":"Momo\u0026rsquo;s cluster has no leader. Nothing decides which machines exist, which are healthy, or where data belongs. Every node runs the same membership logic and converges on the same view of the ring — we call it a masterless ring. That view has to survive nodes joining, dying, and returning, and keeping it correct is the entire job of the gossip layer in this post.\nTwo ideas do the work. Gossip is a rumor mill: each node periodically swaps small summaries with a random handful of peers, and any fact one node learns spreads through the cluster without a central broadcaster. SWIM (Scalable Weakly-consistent Infection-style Membership) is the failure-detection protocol layered on top: a node pings a peer to check it is alive, and if the ping goes unanswered it asks several other peers to ping on its behalf before declaring the target dead. That indirect step is the trick — one congested link can no longer produce a false death sentence.\nP2P Gossip and SWIM Failure detection, and the bugs that broke it Nodes exchange heartbeats, pings, and acknowledgements, and mark each other ALIVE or OFFLINE. The design is simple; the interesting part was everything the first implementation got wrong.\nDiscovery that never dialed. An early audit caught that the discovery step produced peers nobody actually connected to. Dynamic membership simply did not work: a new node could be \u0026ldquo;known\u0026rdquo; and still be unreachable. The fix wired the three steps together — discover, dial, then record the peer in the map. Peers stuck dead forever. A peer that had been marked OFFLINE stayed OFFLINE even after it recovered. The fix lets a ping, acknowledgement, or heartbeat revive a peer, so the ring self-heals without a restart. Leaks on disconnect. Closed connections and detached peers were never removed from the in-memory peer and connection maps, leaking memory every time a node left. The fix prunes both on disconnect. Why gossip instead of a coordinator A coordinator would have to be authoritative, which means a leader election, a single point of failure, and a node whose view of \u0026ldquo;the config\u0026rdquo; everyone else trusts. Momo avoids that: every node computes data placement itself from the same deterministic placement function plus the current membership list. The only thing that has to converge is membership, and gossip is how it converges. Higher-level operations — listing metadata, acquiring leases — ride on top of that converged view rather than on a central server.\nKeeping it cheap Membership runs on every node forever, so it is held to a tight allocation budget. The ping counter became a 64-bit atomic, avoiding a 32-bit wraparound; the hostname is cached once at startup instead of recomputed for every metrics scrape; and the random peer selection is seeded from a secure source so the shuffle is unbiased. Later, gossip fanout itself became adaptive to cluster size, covered in 018, so a hundred-node ring does not flood a three-node lab cluster with traffic.\nReferences / Dig deeper Cluster design doc: docs/REFERENCE/P2P.md. Spec: openspec/changes/add-p2p-transport. Membership fixes: PR #808, PR #809; discovery audit issue #598. Sibling posts: 017: Scatter-Gather and Lease Consensus, 018: Adaptive Scaling. ","permalink":"/posts/016-p2p-gossip-swim/","summary":"A scaleless-gossip ring: SWIM-style failure detection, discovered-peer dialing, and ALIVE/OFFLINE restoration — momo\u0026rsquo;s nervous system.","title":"P2P: Gossip, SWIM, and Membership"},{"content":"momo did not begin its life as an enterprise-grade distributed storage system. It started as an architectural research project: an experimental Go sandbox built to explore how distributed storage protocols behave when their internal replication algorithms are forced to change dynamically under real-time network pressure.\nThe name itself — momo — hints at this polymorphic origin. As the experimental codebase grew, though, the hard realities of distributed state machines, network partitions, and storage durability forced a series of architectural shifts. This post is the story of that arc.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 =================================================================================== THE EVOLUTIONARY ARC =================================================================================== Phase 0: File Replication Playground - In-memory node map, file copies routed across ad-hoc TCP listeners. - Fragile name-based addressing; race conditions during concurrent overwrites. | v Phase 1: The Polymorphic Engine - Formalized replication topologies: None, Splay, Chain, Primary-Splay. - Dynamic runtime mode shifting via single-authority controller. | v Phase 2: Content-Addressable Storage (CAS) - Replaced filename addressing with cryptographic SHA-256 digests. - Embedded Bbolt metadata engine; immutable tiered blob layout. | v Phase 3: Production Distributed Storage - Deterministic CRUSH-lite placement across failure domains. - S3-compatible gateway with SigV4 and streaming multipart upload. - POSIX FUSE filesystem interface (momofs) over the CAS core. 1. The Trap of Name-Addressed Storage In the early prototypes, data was addressed by filename. When a client uploaded document.pdf to one daemon, that daemon wrote a local file named document.pdf and dialed two peer daemons to stream the same bytes to identical paths.\nThis simplistic approach collapsed under basic concurrency:\nWrite collisions. If two clients uploaded different contents under the same filename to different cluster nodes, the nodes entered an unresolvable split-brain state. Corrupted sync. Resolving those conflicts required heavyweight two-phase locking (2PC) or a distributed lock manager (DLM), dragging request latency down by orders of magnitude. Silent corruption. If a bit flipped on a replica\u0026rsquo;s hard drive, the system could not determine which replica held the genuine data without reading all copies and taking a majority vote. The epiphany was to stop naming data by what a user calls it and start naming it by what it is. Content-addressable storage (CAS) names every blob after a cryptographic hash of its bytes — in momo\u0026rsquo;s case, SHA-256. Two identical uploads produce the same hash, so writes become naturally idempotent and deduplicated. A user-facing filename becomes transient metadata — a pointer stored in an embedded key/value database — rather than the storage address itself. That single decision, covered in 004, unlocked the rest of the design.\n2. The Invariant Core: Simplicity and Zero Dependencies As momo grew from a playground into an object store, it resisted the bloat common to modern distributed filesystems. Three durable rules were codified:\nZero external runtime dependencies. momo requires no external consensus clusters (no ZooKeeper, etcd, or Consul). Everything — from the embedded metadata database to SWIM gossip membership — is compiled directly into a single static Go binary. Deterministic placement over central coordination. Instead of maintaining a central lookup master that records where every block lives (the way HDFS\u0026rsquo;s NameNode does), momo computes placement locally. Given an object hash and a cluster map, every node and client deterministically calculates the exact replica set with zero network round-trips. We call this CRUSH-lite, after Ceph\u0026rsquo;s CRUSH algorithm but with a flat, simpler topology (005). Seams over plugins. Extensibility points — storage backends, verifiers, durability barriers — are designed as compile-time interfaces rather than dynamically loaded RPC plugins. A seam is a deliberately narrow interface we can substitute without paying the IPC and security costs of a plugin runtime. The principle is captured in our engineering standards. 3. Tradeoffs: The Evolution from Sandbox to Engine Architectural Aspect Playground Prototype Distributed Object Store (momo) Addressing Model User-defined paths / filenames SHA-256 content-addressed storage Storage Backend Flat temp-directory files Tiered fan-out (blobs/ab/cd/ef/...) + embedded index Placement Logic Hardcoded node lists Deterministic CRUSH-lite with failure-domain awareness Durability Semantics Unchecked OS buffer flush Strict flush-before-ack, group commit, survivor quorum Interface Surface Custom CLI commands only Full AWS S3 HTTP API + POSIX FUSE mount (momofs) Each shift traded simplicity for durability, and each one only happened because the previous prototype failed under a real workload. momo proved that a storage system can provide production-level durability and S3 compatibility while preserving the agile, hackable essence of a minimalist Go playground — provided you are willing to give up centralized coordination.\nReferences / Dig deeper Origin commit: a8114af4. Sibling posts: 002: Replication Strategies, 004: Content-Addressable Storage, 005: CRUSH Placement, 028: Roadmap and Research. The \u0026ldquo;seam over plugins\u0026rdquo; mindset and the project\u0026rsquo;s engineering standards: docs/CORE/STANDARDS.md. ","permalink":"/posts/001-origin-and-genesis/","summary":"How momo started as a file-replication playground in Go and grew into a distributed content-addressed object store.","title":"Origin: From Replication Playground to Object Store"}]