On 2026-08-04 we pointed a mechanized audit at an early momo and let it run deep: 76 findings, filed as a chain of issues from #593 onward. It was uncomfortable. It was also the moment the 🛡 Sentinel mindset was born — the idea that a security flaw is not a special category of work to schedule later, but a first-class engineering item with a reproducible failure, an auditable fix, and a regression test that keeps it fixed.

What the sweep found

The findings clustered into a handful of classes, each a textbook way for untrusted input to cross a boundary it should never cross.

  • CRLF injection. CRLF stands for carriage-return / line-feed, the two bytes that terminate a line in HTTP and many text protocols. If attacker-controlled bytes containing \r\n reach a header or a log line, they can forge a new line — splitting one response into two, or smuggling fake fields into a log. We found untrusted bytes flowing into metadata and metrics paths.
  • HTTP request smuggling. This is the classic ambiguity attack: when a request carries both a Transfer-Encoding header and a Content-Length header, two servers in the chain can disagree about where the request ends. An attacker crafts a request that the front gateway reads one way and the backend reads another, letting them “smuggle” a second request past the gateway’s checks. We found this in the S3 gateway, and it was the most severe issue of the sweep.
  • Path traversal. When a filename is used to build a filesystem path, a value containing .. can climb out of the intended directory and read or write somewhere it should not. Our first guard flagged false positives — filenames that merely contained .. as part of a longer name were rejected even though they were harmless. The fix was to flag only a complete path component that is exactly .., not any occurrence of the characters.
  • Resource leaks. The long tail: request bodies never closed, a wait-group counter incremented after the wait had already begun, and a lease that could reach zero quorum during a network partition. Individually minor; collectively a slow bleed of file descriptors and goroutines that would eventually take a node down.

Process: issuance → regression → hardening

The value was not the list; it was the discipline applied to each entry. Every finding got a reproducible acceptance case first — a test that fails against the vulnerable code — then a fix, then, where it made sense, a regression contract test so the exact class of bug could never return unnoticed. The offensive toolkit we used (fuzzing for path traversal plus hand-written exploits) was kept alongside the code and documented, so the next audit starts from where this one ended rather than from scratch.

The mindset distilled

Three rules fell out of the sweep and became permanent:

  • Fail closed, loudly. When input is suspicious, refuse it with an honest error rather than papering over it with a fake success. A server that returns “not implemented” for something it cannot safely do is more trustworthy than one that returns a 200 and hopes. We map internal errors to POSIX syscall constants so callers get a truthful, machine-readable reason.
  • Trust invariants are never bypassed. The checks that matter — CRLF sanitization, crypto, integrity verification — live in the auditable core and cannot be routed around by a plugin or an alternate code path. If a fast path skips a trust check, it is not a fast path, it is a vulnerability.
  • Zero-crash handling. Panics in request handling become recovered errors, not process death; unbounded readers get bounds; every goroutine is verified not to leak. A storage node that crashes on malformed input is a denial-of-service waiting to happen.

See docs/CORE/STANDARDS.md for the ⚡ Bolt / 🛡 Sentinel mindsets.

References / Dig deeper