#1 Log grows ~15x faster than live data under low traffic · ermite-kv

Observation

On a small, low-traffic deployment (one user, mostly idle overnight), the write-ahead log grows much faster than the live data it holds.

Measured on disk, about 10 hours after the last compaction:

File	Size
snapshot (live data after the last compaction)	~3 MB
current log	~51 MB

The log received ~48 MB in ~10 h, about 5 MB per hour, for ~3 MB of live data.
With the default CompactAfter (64 MB), this means a compaction about every 12 to 13 hours.
Each commit is appended and synced to disk, so the write volume translates into I/O and disk wear, not only space.

What is bounded

Disk usage stays bounded by snapshot + CompactAfter: the log is compacted back to the live data, so this is not a space leak.

Open questions

Which keys or prefixes produce most of the appended bytes? (Count of records and bytes per key prefix over one log.)
Are some writers rewriting values that did not change, or updating frequently (periodic status, timestamps, heartbeats)?
Would a read-only tool that summarizes a log file (records, bytes and overwrites per prefix) help diagnose this in general?

Comments1

bnema 2d ago

Findings from decoding the production log (52.6 MB, 2,889 records, 12,364 ops): - job/ (CI jobs): 51.1 MB, 97% of the appended bytes - runsrc/: 0.5 MB; everything else: about 1 MB - No write rewrote an unchanged value. Cause: write amplification in ermite's CI storage, not in ermite-kv. - Each CIJob value is about 25 KB, and 24.6 KB of it is contexts_json. Most of that is github.event, the full event payload (about 24 KB). - Every step update (UpdateJob -> saveJob) rewrites the whole record, although only the steps change (about 350 bytes). - One job was rewritten about 58 times in a run, so about 1.4 MB per job and about 2 MB per run. The log covers about 25 CI runs, so "low traffic" means a few CI runs per hour. The kv engine behaves as intended: space is bounded by snapshot + CompactAfter. The fix belongs in ermite: store the planned contexts under their own key, written once per job, so step updates rewrite a record of about 1 KB. A read-only log summary tool (records, bytes and overwrites per prefix) would still be useful: a small decoder found this in seconds.