Architecture Decisions

R1: Failure-Domain-Aware Placement

Replicas must not ride in the same rack: CRUSH weights become rack/zone/DC-aware so a single domain failure can’t lose the data.

Perf Profiling Baseline: Measure Before You Optimize — and Keep pprof Off the Wire

Phase-0 profiling harness via go test -cpuprofile/-memprofile; Rule 75 forbids networked pprof on unauthenticated listeners. The baseline proved stdlib SHA-256 already uses AVX2.

Seams, Not Plugins: A Fast Path That Stays Concrete

Adaptive behaviors use compile-time Go interface seams injected at decision points — not dynamic plugins. The fast path stays concrete; the trust core stays auditable; policy is declarative.

Metrics and Observability: Per-Node Bind, Prometheus Export

Prometheus metrics export with a per-node metrics_host/metrics_port bind — observability that scales past the ‘just scrape node 0’ era.

Rotating Secrets Without a Secret Server: Deriving, Versioning, and Hot-Reloading Keys

How momo manages encryption keys without a central secret server: a pluggable provider chain (env, then config file), a per-node versioned key registry in local storage, and rotation that promotes a new key without any downtime. Teaches: why a provider chain beats a single source, how key versioning enables online rotation, and when rotation is and is not worth the complexity.

Tenant Isolation: Deriving Per-Tenant Keys, Enforcing ACLs, and Proving What Happened

How momo isolates tenants without a central secret server: HKDF-derived per-tenant keys, default-deny ACLs evaluated per request, and a SHA-256 hash-chained audit log that makes tampering detectable. Teaches: key derivation vs. key storage, why ACL evaluation order matters, and how hash chaining turns a log into evidence.

R5 Metrics Phases 2-4: Storage, P2P, and Latency Histograms

R5 Phases 2-4 shipped: storage metrics (disk, CAS, GC), P2P metrics (SWIM, leases, scatter-gather), and opt-in latency histograms — all with <1% overhead via sync/atomic counters.

Forward: Production Roadmap (R5–R11) and the Research Guide

After R1–R4: the P0 hardening done, the P1/P2 tracks (metrics, HA metadata, auth, secrets, S3 breadth) and the research reading guide that seeded it all.

At-Rest Integrity: Verify-on-Read, Checksums, and GC

CAS keys make verify-on-read free: central verification, x-amz-checksum echo, tombstone GC, and corruption surfacing.

Centralized Integrity Verification: Checksums Move to the Storage Core

S3 integrity checksums moved from a surface-level adapter into the storage/ingest core via a protocol-agnostic ChecksumProvider seam — every write path verified, replicas re-verify on receive.