Storage optimization evaluation (#19)
Decision
Keep Go-owned slices/maps, 256-row default chunks and standard-library zlib.
Do not add pooling, mmap/decommit, SIMD or physical page alignment. Accept one
measured allocation fix: bound initial mutable-tail capacity by the actual
width × ChunkRows, in addition to the existing byte/preallocation ceilings.
This is an evaluation, not a claim that unimplemented techniques were benchmarked.
No assembly, unsafe code, OS-specific allocator, dependency or background worker
was introduced.
Measurements
September 4, 2026; Linux amd64, AMD Ryzen 5 7535HS, Go 1.27.1. Chunk-size figures
are two one-iteration samples, so small timing differences are not significant.
All variants use 10,000 rows × 120 columns with repeated plain text.
| Chunk rows |
Before: ms/op |
Before: allocated MB/op |
After capacity fix: ms/op |
After: allocated MB/op |
| 32 |
184–197 |
1,089 |
120–127 |
148.0 |
| 64 |
161–163 |
555.9 |
126–130 |
146.9 |
| 128 |
137–139 |
290.0 |
125–129 |
146.9 |
| 256 |
125 |
157.0 |
124–128 |
148.5 |
The old fixed initial capacity overallocated each short tail. A history configured
for four three-cell rows retained 3,408,840 bytes after one append. The corrected
capacity retains 2,248 bytes in the same benchmark. A regression test pins the
initial capacity to at most twelve cells for that case.
For the default 256-row configuration after the fix:
- Uncompressed plain-ASCII history retains 22.8 MB, versus the original
95.3 MB baseline: the required 4× reduction still holds.
- Two cold-maintenance passes retain 4.34 MB, with 38 cold pages and 173,190
compressed bytes. The newest page and mutable tail remain hot.
- This is synthetic retained Go heap, not a real-daemon RSS measurement.
A three-iteration CPU/allocation profile before the capacity fix attributed about
86.8% of allocated bytes to reserveTailRow. CPU samples were concentrated in
compact cell writes, canonical dictionaries and chunk construction. The short
profile supports identifying the allocation source, not predicting production
latency or claiming a precise speedup.
Options considered
| Option |
Result |
Reason |
| Geometry-bounded tail preallocation |
Accepted |
Large reproducible allocation reduction for small chunks; no ownership changes. |
| Change default logical chunk rows |
Rejected for now |
Corrected throughput samples overlap. 256 rows uses fewer allocations/dictionaries than smaller chunks. |
| Pool tail/page memory |
Not justified yet |
The immediate cause was excessive requested capacity. Sealed pages may outlive History through borrowed views, making recycling ownership-sensitive; pools can also retain cold memory. |
| Recycle the private mutable tail across seals |
Accepted |
Only History reads its tail: sealed chunks and views hold compact copies. Reusing that storage removed about 85% of the bytes allocated by BenchmarkHistoryBuild10Kx120. Retention is bounded: at most 128K cells (a default chunk up to 512 columns), and cleared cells, so payload strings are not pinned. Style scratch maps are dropped above 1,024 entries. Sealed pages are never pooled. |
| mmap/decommit |
Not justified |
Go GC already releases dropped page references; no measured syscall/RSS benefit supports platform-specific backing, finalization and failure paths. |
| SIMD |
Not justified |
The measured work is semantic cell writes and dictionary lookup, not a demonstrated bulk transform suited to SIMD. Native copy/clear remains the baseline. |
| Fixed physical page alignment |
Rejected |
No measured benefit. Chunk rows are logical grouping, not an OS page-size commitment. |
Cold random-access restore is cached until later idle visits. Streaming history
search and persistence use transient restores, so scanning does not make every
cold page resident. Restoring the 16×80 test page, including cache release, measured
about 123 microseconds and 92.8 KB allocated; use the benchmark below for current
results rather than treating that sample as a latency guarantee.
Reproduce
go test . -run '^$' -bench '^(BenchmarkHistoryChunkSize|BenchmarkSmallHistoryRetained)$' -benchtime=1x -count=2 -benchmem
go test . -run '^$' -bench '^(BenchmarkHistoryRetained10Kx120|BenchmarkColdHistoryRetained10Kx120)/plain-ascii$' -benchtime=1x -benchmem
go test . -run '^$' -bench '^BenchmarkColdHistoryRestore$' -benchtime=100ms -benchmem
go test . -o /tmp/vev-vt-profile.test -run '^$' -bench '^BenchmarkHistoryBuild10Kx120/plain-ascii$' -benchtime=3x -cpuprofile /tmp/vev-vt-storage.cpu -memprofile /tmp/vev-vt-storage.mem
go tool pprof -top /tmp/vev-vt-profile.test /tmp/vev-vt-storage.cpu
go tool pprof -top -alloc_space /tmp/vev-vt-profile.test /tmp/vev-vt-storage.mem
Keep the 4× original retained-memory gate, ownership/identity tests, strict codec
validation, race tests and 386 portability checks for future experiments. A
production workload/RSS study remains necessary before selecting any rejected
low-level technique.
source
1# Storage optimization evaluation (#19)
2
3## Decision
4
5Keep Go-owned slices/maps, 256-row default chunks and standard-library zlib.
6Do not add pooling, mmap/decommit, SIMD or physical page alignment. Accept one
7measured allocation fix: bound initial mutable-tail capacity by the actual
8`width × ChunkRows`, in addition to the existing byte/preallocation ceilings.
9
10This is an evaluation, not a claim that unimplemented techniques were benchmarked.
11No assembly, unsafe code, OS-specific allocator, dependency or background worker
12was introduced.
13
14## Measurements
15
16September 4, 2026; Linux amd64, AMD Ryzen 5 7535HS, Go 1.27.1. Chunk-size figures
17are two one-iteration samples, so small timing differences are not significant.
18All variants use 10,000 rows × 120 columns with repeated plain text.
19
20| Chunk rows | Before: ms/op | Before: allocated MB/op | After capacity fix: ms/op | After: allocated MB/op |
21| --- | ---: | ---: | ---: | ---: |
22| 32 | 184–197 | 1,089 | 120–127 | 148.0 |
23| 64 | 161–163 | 555.9 | 126–130 | 146.9 |
24| 128 | 137–139 | 290.0 | 125–129 | 146.9 |
25| 256 | 125 | 157.0 | 124–128 | 148.5 |
26
27The old fixed initial capacity overallocated each short tail. A history configured
28for four three-cell rows retained 3,408,840 bytes after one append. The corrected
29capacity retains 2,248 bytes in the same benchmark. A regression test pins the
30initial capacity to at most twelve cells for that case.
31
32For the default 256-row configuration after the fix:
33
34- Uncompressed plain-ASCII history retains **22.8 MB**, versus the original
35 **95.3 MB** baseline: the required 4× reduction still holds.
36- Two cold-maintenance passes retain **4.34 MB**, with 38 cold pages and 173,190
37 compressed bytes. The newest page and mutable tail remain hot.
38- This is synthetic retained Go heap, not a real-daemon RSS measurement.
39
40A three-iteration CPU/allocation profile before the capacity fix attributed about
4186.8% of allocated bytes to `reserveTailRow`. CPU samples were concentrated in
42compact cell writes, canonical dictionaries and chunk construction. The short
43profile supports identifying the allocation source, not predicting production
44latency or claiming a precise speedup.
45
46## Options considered
47
48| Option | Result | Reason |
49| --- | --- | --- |
50| Geometry-bounded tail preallocation | Accepted | Large reproducible allocation reduction for small chunks; no ownership changes. |
51| Change default logical chunk rows | Rejected for now | Corrected throughput samples overlap. 256 rows uses fewer allocations/dictionaries than smaller chunks. |
52| Pool tail/page memory | Not justified yet | The immediate cause was excessive requested capacity. Sealed pages may outlive History through borrowed views, making recycling ownership-sensitive; pools can also retain cold memory. |
53| Recycle the private mutable tail across seals | Accepted | Only History reads its tail: sealed chunks and views hold compact copies. Reusing that storage removed about 85% of the bytes allocated by `BenchmarkHistoryBuild10Kx120`. Retention is bounded: at most 128K cells (a default chunk up to 512 columns), and cleared cells, so payload strings are not pinned. Style scratch maps are dropped above 1,024 entries. Sealed pages are never pooled. |
54| mmap/decommit | Not justified | Go GC already releases dropped page references; no measured syscall/RSS benefit supports platform-specific backing, finalization and failure paths. |
55| SIMD | Not justified | The measured work is semantic cell writes and dictionary lookup, not a demonstrated bulk transform suited to SIMD. Native copy/clear remains the baseline. |
56| Fixed physical page alignment | Rejected | No measured benefit. Chunk rows are logical grouping, not an OS page-size commitment. |
57
58Cold random-access restore is cached until later idle visits. Streaming history
59search and persistence use transient restores, so scanning does not make every
60cold page resident. Restoring the 16×80 test page, including cache release, measured
61about 123 microseconds and 92.8 KB allocated; use the benchmark below for current
62results rather than treating that sample as a latency guarantee.
63
64## Reproduce
65
66```sh
67go test . -run '^$' -bench '^(BenchmarkHistoryChunkSize|BenchmarkSmallHistoryRetained)$' -benchtime=1x -count=2 -benchmem
68go test . -run '^$' -bench '^(BenchmarkHistoryRetained10Kx120|BenchmarkColdHistoryRetained10Kx120)/plain-ascii$' -benchtime=1x -benchmem
69go test . -run '^$' -bench '^BenchmarkColdHistoryRestore$' -benchtime=100ms -benchmem
70go test . -o /tmp/vev-vt-profile.test -run '^$' -bench '^BenchmarkHistoryBuild10Kx120/plain-ascii$' -benchtime=3x -cpuprofile /tmp/vev-vt-storage.cpu -memprofile /tmp/vev-vt-storage.mem
71go tool pprof -top /tmp/vev-vt-profile.test /tmp/vev-vt-storage.cpu
72go tool pprof -top -alloc_space /tmp/vev-vt-profile.test /tmp/vev-vt-storage.mem
73```
74
75Keep the 4× original retained-memory gate, ownership/identity tests, strict codec
76validation, race tests and 386 portability checks for future experiments. A
77production workload/RSS study remains necessary before selecting any rejected
78low-level technique.