⚡ Bolt: Eliminating Time Formatting Allocations in S3 HTTP Responses

A single 32-byte allocation per request sounds harmless until you multiply it by a hundred thousand requests a second. That is the story of this change: a timestamp that was quietly generating megabytes of garbage per second inside the S3 CopyObject response path.

An allocation is memory claimed from the heap, the shared pool the garbage collector manages. A hot path is code executed on nearly every request. Go’s profiler, pprof, is what let us see the allocation in the first place: it samples a running program and attributes time and memory to the functions that consume them.

The Real Problem

During a sustained S3 load test — 100k requests per second of CopyObject — the garbage-collector trace revealed a surprising hotspot: time.Format was allocating 32 bytes per request just to render the LastModified timestamp in the XML response. At peak throughput that is roughly 3 MB/s of garbage only for timestamps, and the GC pause spiked to 12 ms — well past our 5 ms p99 latency target.

We did not catch this in unit tests. It only appeared under sustained load, when the allocation rate overwhelmed the generational collector’s ability to keep up. The “small allocation” was not small at scale.

Why the Obvious Solutions Failed

First attempt: pool pre-formatted buffers.

1
2
3
4
5
6
7

// REJECTED: pool mutex + GC pressure from pooled objects > allocation cost
var timePool = sync.Pool{
    New: func() any {
        s := make([]byte, 32)
        return &s
    },
}

The pool’s mutex contention at 100k operations per second added more latency than the allocation itself. And pooled objects still pressure the collector — they just delay it, and risk promotion to the old generation where pauses are longer.

Second attempt: cache formatted timestamps. Timestamps are unique per request at nanosecond precision, so a cache would have a near-zero hit rate and only waste memory.

Third attempt: pre-allocate a buffer and use fmt.Sprintf.

1
2
3

// REJECTED: fmt uses reflection/interface{} internally, so it still allocates
buf := make([]byte, 32)
n := fmt.Sprintf(buf, "%s", t.Format(time.RFC3339Nano))

Formatting through fmt goes through reflection and interface conversions, which allocate regardless of how the destination buffer was prepared.

The Solution — Stack-Buffer Time Formatting

The standard library provides time.AppendFormat specifically for this. It writes directly into a byte slice and returns the extended slice — no intermediate string, no heap allocation.

1
2
3
4
5
6
7
8

// Stack buffer: 32 bytes fits RFC3339Nano (20 chars) + timezone + padding.
// Zero heap allocation — it lives on the stack and dies with the function.
var timeBuf [32]byte

// AppendFormat writes directly into the buffer and returns the extended slice.
// timeBuf[:0] is an empty slice backed by the array (cap=32, len=0).
t := time.Unix(0, modTime).UTC()
buf.Write(t.AppendFormat(timeBuf[:0], "2006-01-02T15:04:05.000Z"))

Key insight: timeBuf[:0] creates a slice with length 0 but capacity 32, backed by the stack array. AppendFormat appends into that capacity, so the result points into the stack array and never escapes to the heap.

Principle Callout

Pattern: Stack-Buffer Time Formatting Use time.AppendFormat with a stack-allocated byte array in hot paths. It eliminates the heap allocation per call.

Applies when: high-frequency timestamp rendering — HTTP headers, XML/JSON fields, log lines. Doesn’t apply: one-off formatting or human-readable output, where the allocation is negligible.

1 2 3 4 5 6 7 8 9 10 11

// ✅ GOOD: Hot path — HTTP header, XML field, log line func writeLastModified(buf *bytes.Buffer, modTime int64) { var timeBuf [32]byte t := time.Unix(0, modTime).UTC() buf.Write(t.AppendFormat(timeBuf[:0], time.RFC3339Nano)) } // ❌ AVOID: Low-frequency path — allocation negligible func formatForDisplay(modTime int64) string { return time.Unix(0, modTime).Format("Jan 2, 2006") }

Verification

The paired microbenchmarks isolate the two implementations, and under 100k requests per second of production-like load the GC cycles dropped from 45 per minute to 3, with p99 latency settling at 3.2 ms.

Failure Modes

When NOT to Use This

• Request IDs, trace IDs, and user-facing dates — these are not hot paths.
• One-off formatting in startup or shutdown paths.
• When you need a string return value for API compatibility; the allocation is unavoidable at that boundary.

References / Dig deeper

• Mindset standards: docs/CORE/STANDARDS.md — ⚡ Bolt / 🛡 Sentinel.
• Bolt engineering overview: 024.
• The same pattern for HTTP Last-Modified headers: 045.
• Applied to ListParts XML: 048.
• The change spec: openspec/changes/bolt-s3-copyresult-time-alloc.
• The paired microbenchmarks live in src/s3/response_test.go (BenchmarkCopyObjectResponse_AppendFormat vs BenchmarkCopyObjectResponse_Format).