Metadata in momo does not live in one database. Each node owns a slice — a shard — of the key space, so no single machine has the whole picture. That shapes two operations every S3-style cluster needs: listing objects, and protecting mutable shared state. They are two halves of the same problem.

Scatter-gather is the listing half. To answer “what objects exist under this prefix?”, a node fans the question out to the shard owners, collects their partial answers, and merges them into one sorted, deduplicated list — scatter the request, gather the replies. A lease is the mutable-state half: a time-limited grant that lets exactly one node modify something like a rename or a reference count at a time. Granting a lease requires a quorum, a majority of the voting peers to agree, so two network partitions cannot both hand out the same lease and corrupt state — the failure mode known as split-brain.

Scatter-gather listing

Scatter-gather and lease quorum
Scatter-gather and lease quorum

A list request fans out to shard owners, merges their metadata lists, dedups by content hash (so the same bytes stored under two names appears once), and paginates the result. The filesystem layer later narrowed the fan-out to only the shard owners that can actually hold the requested prefix, turning a cluster-wide scatter into a handful of targeted requests.

Leases and the majority-quorum bug class

A lease protects mutable shared state, and the correctness of the whole scheme rests on one arithmetic rule: a lease is granted only when the number of votes is a strict majority of the peer count. For three peers the majority is two; for five it is three. Get that comparison wrong and the guarantee evaporates. Two bugs found by audit show how.

  • The off-by-one majority. The quorum test was written in a way that could resolve too low for odd peer counts, accepting fewer votes than a true majority. The fix is to require strictly more votes than half the peers, computed on the correct integer basis — for three peers, more than one means at least two.
  • The zero-quorum lease. Worse, lease acquisition could succeed with no quorum at all while the cluster was partitioned — the split-brain enabler. The fix is to fail closed: no quorum, no lease, even if that means the operation is refused until the partition heals.

Keeping the hot paths allocation-free

Both operations mutate shared state, so both follow a merge discipline that avoids allocations on the request path: merge by content hash, in a stable order, with no intermediate copies. The metadata-list encoder also gained length limits, so a maliciously oversized entry cannot blow up the message buffer.

Why this is a security posture, not just a correctness fix

A partition that grants a lease is how a split ring silently corrupts reference counts, which is why the zero-quorum fix is treated as security-adjacent rather than as a routine bug. It is the same fail-loud stance we take elsewhere: when the system cannot prove it is safe to proceed, it refuses instead of guessing.

References / Dig deeper