docs(perf): record S5a measurement schema and baseline procedure
- commit
- 78d0fd04e96db66fc3556bbe6d05b3974fee4349
- author
- brice · · 2026-07-12 22:34 UTC
- parent
- 7a49239
- signature
- GPG Verified
CD88F9E564E0102646FB6077A1A07C7C7C3E349D· arch-gpg Verified for the committer email
2 files · +97 -106
M docs/performance.md +54 -91
| @@ -1,107 +1,70 @@ | |||
| 1 | - | # Performance methodology (M5) | |
| 2 | - | ||
| 3 | - | This baseline measures in-process daemon work only. It includes coordinator-enabled | |
| 4 | - | fixture scheduling metrics, but deliberately does **not** measure chunk | |
| 5 | - | encoding, compression, GC tuning, or real network latency. Output metrics count frames and encoded output data produced | |
| 6 | - | by the daemon; snapshot writes and serialized bytes are local snapshot-store | |
| 7 | - | proxies. These metrics approximate local rendering, IPC payload, and persistence | |
| 8 | - | work, not a Unix socket, SSH, or UDP transport. Real transport impairment and | |
| 9 | - | latency benchmarks are deferred. | |
| 10 | - | ||
| 11 | - | ## Large-history daemon baseline | |
| 12 | - | ||
| 13 | - | Use a single iteration only as a smoke check; it is not suitable for drawing | |
| 14 | - | performance conclusions: | |
| 15 | - | ||
| 16 | - | ```sh | |
| 17 | - | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1x -benchmem | |
| 18 | - | ``` | |
| 19 | - | ||
| 20 | - | For before/after comparison, collect repeated samples for the **full matrix** | |
| 21 | - | (all workloads and topologies), pinning CPU/governor and recording the host. | |
| 22 | - | The benchmark harness must be committed and present in **both** measured | |
| 23 | - | revisions: collect the baseline after the harness is introduced but before the | |
| 24 | - | optimization, then collect the optimized revision with that same harness. The | |
| 25 | - | fixed duration and count make the output suitable for `benchstat`: | |
| 26 | - | ||
| 27 | - | ```sh | |
| 28 | - | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1s -count=5 -benchmem > /tmp/vev-history-after.txt | |
| 29 | - | benchstat /tmp/vev-history-before.txt /tmp/vev-history-after.txt | |
| 1 | + | # Performance measurement | |
| 2 | + | ||
| 3 | + | ## S5a transport observability | |
| 4 | + | ||
| 5 | + | S5a is measurement-only. It records process-local runtime marks and harness-owned | |
| 6 | + | end-to-end boundaries without changing rendering, pacing, transport policy, wire | |
| 7 | + | bytes, or `ProtocolVersion`. The canonical matrix and workload/topology/ | |
| 8 | + | transport coverage are in [`testdata/perf/manifest.json`](../testdata/perf/manifest.json): | |
| 9 | + | `1x4`, `4x1`, `4x4`, and `8x1`, all at `120x40` with 10,000 rows per pane; | |
| 10 | + | idle, active/all/inactive output, interactive flood, copy search, resize sweep, | |
| 11 | + | snapshot/output/resize, and attach/restore/tab switch; local, SSH stdio, UDP | |
| 12 | + | baseline, UDP 25/100 ms, and UDP 1% loss. It has 216 scenarios (4×9×6), with | |
| 13 | + | inapplicable cases required to be explicit. | |
| 14 | + | ||
| 15 | + | ### Trace schema and clock domains | |
| 16 | + | ||
| 17 | + | Every process JSONL record has exactly these fields: | |
| 18 | + | `schema` (uint), `process_id` (string), `component` (string), `scenario` | |
| 19 | + | (string), `run`, `sequence`, `request_id`, `epoch` (uint), `kind` (string), | |
| 20 | + | `tick` (int64), `bytes`, `fragments`, `retransmits`, `pending` (uint), | |
| 21 | + | `ack_rtt_nanos` (int64), and `valid` (bool). Schema 1 is represented by: | |
| 22 | + | ||
| 23 | + | ```json | |
| 24 | + | {"schema":1,"process_id":"p","component":"daemon","scenario":"s","run":1,"sequence":1,"request_id":1,"epoch":1,"kind":"diff_start","tick":0,"bytes":0,"fragments":0,"retransmits":0,"pending":0,"ack_rtt_nanos":0,"valid":true} | |
| 30 | 25 | ``` | |
| 31 | 26 | ||
| 32 | - | The matrix exercises live paint, snapshot capture, copy-mode entry, copy search, | |
| 33 | - | and resize at `120x40` client geometry. Each pane is populated with 10,000 | |
| 34 | - | full-width deterministic history rows containing a stable `needle-<tab>-<row>` | |
| 35 | - | prefix and patterned text. The layouts are: | |
| 27 | + | `tick` is stamped only by the concrete JSONL observer using its process-local | |
| 28 | + | injected monotonic clock; producers supply no timestamp. Components in one OS | |
| 29 | + | process share that observer. The harness clock is a separate monotonic domain | |
| 30 | + | and is the sole clock for input-injection → terminal-bytes-successfully-flushed | |
| 31 | + | end-to-end samples. Cross-process ticks are never ordered or subtracted; | |
| 32 | + | records correlate only by scenario/run/sequence/request/epoch IDs. | |
| 36 | 33 | ||
| 37 | - | - 1 tab x 1 pane (control) | |
| 38 | - | - 1 tab x 4 panes | |
| 39 | - | - 4 tabs x 1 pane | |
| 40 | - | - 4 tabs x 4 panes | |
| 41 | - | - 8 tabs x 1 pane | |
| 34 | + | The named process-local spans are capture, compose, diff, queue wait, | |
| 35 | + | ACK-blocked interval, emit, adapter send, and adapter receive. Their start/end | |
| 36 | + | marks are paired within one process and summarized by component/adapter with | |
| 37 | + | raw records, counts, p50/p95/p99, and max. Diagnostic fields (bytes, fragments, | |
| 38 | + | retransmits, pending, ACK RTT) remain separate from duration calculations. | |
| 42 | 39 | ||
| 43 | - | The fixture starts no PTY readers or transports. It primes the renderer shadow | |
| 44 | - | before timing; live paint then alternates fixed writes through the coordinator. | |
| 45 | - | The reported `outputframes/op` value counts output messages. `outputbytes/op` | |
| 46 | - | counts their encoded terminal data and excludes the 24-byte output header; | |
| 47 | - | `framepayloadbytes/op` counts the complete output payload including that header. | |
| 48 | - | `snapshotwrites/op` and `snapshotbytes/op` are local snapshot-store writes and | |
| 49 | - | serialized bytes. `coordinatorinvalidations/op`, `coordinatorwakes/op`, and | |
| 50 | - | `coordinatorcoalesced/op` respectively count scheduling requests, delivered | |
| 51 | - | wakes, and requests coalesced into those wakes; `coordinatorcoalescingratio` | |
| 52 | - | is the average number of coalesced invalidations delivered per wake. They must | |
| 53 | - | not be presented as network | |
| 54 | - | throughput or latency. | |
| 40 | + | ### Baseline procedure and validity | |
| 55 | 41 | ||
| 56 | - | Capture a pre-change result outside the repository, including Go and kernel | |
| 57 | - | context: | |
| 42 | + | Use the public CLI harness (not internal packages): | |
| 58 | 43 | ||
| 59 | 44 | ```sh | |
| 60 | - | { | |
| 61 | - | date -Is | |
| 62 | - | go version | |
| 63 | - | uname -a | |
| 64 | - | git rev-parse HEAD | |
| 65 | - | # Smoke only; do not use this result for comparisons. | |
| 66 | - | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1x -benchmem | |
| 67 | - | # Repeated full matrix for every workload/topology used in conclusions. | |
| 68 | - | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1s -count=5 -benchmem | |
| 69 | - | } > /tmp/vev-history-before.txt 2>&1 | |
| 45 | + | go build -o /tmp/vev-perf ./ | |
| 46 | + | rm -rf testdata/perf/results | |
| 47 | + | go run ./cmd/vev-perf-harness --vev-bin /tmp/vev-perf \ | |
| 48 | + | --manifest testdata/perf/manifest.json --out testdata/perf/results \ | |
| 49 | + | --warmup 10s --duration 30s --repetitions 10 | |
| 70 | 50 | ``` | |
| 71 | 51 | ||
| 72 | - | ## Immutable document allocation gate | |
| 52 | + | The minimums are 30 seconds measured and 10 independent repetitions; warmup is | |
| 53 | + | excluded. Each run must contain `manifest.json`, `raw-harness.jsonl`, per-process | |
| 54 | + | JSONL, `runs.json`, and `summary.json`. Reject results with missing input/flush | |
| 55 | + | boundaries, unmatched IDs, bad correlation, nonmonotonic same-process sequence, | |
| 56 | + | negative same-domain spans, insufficient measured events, invalid denominators, | |
| 57 | + | any cross-process tick arithmetic, or unmet duration/repetition minima. Raw | |
| 58 | + | JSONL and run manifests are the evidence files; no result authorizes a policy | |
| 59 | + | or optimization change. | |
| 73 | 60 | ||
| 74 | - | The copy-mode document retains an immutable view of existing scrollback rows and | |
| 75 | - | clones only the visible frame. Visual-search models share that document; cloning | |
| 76 | - | a model copies only its mutable input and match state. Check those two paths at | |
| 77 | - | 10,000 history rows with: | |
| 61 | + | ## Existing in-process smoke checks | |
| 78 | 62 | ||
| 79 | - | ```sh | |
| 80 | - | go test ./internal/usecase/copy ./internal/usecase/visualsearch -run '^$' -bench '^(BenchmarkNewSnapshot10KRows|BenchmarkVisualSearchClone10KRows)$' -benchtime=1s -count=5 -benchmem | |
| 81 | - | ``` | |
| 82 | - | ||
| 83 | - | Treat `allocs/op` and `B/op` as allocation gates: compare the complete repeated | |
| 84 | - | output with the accepted baseline before merging, and investigate any increase. | |
| 85 | - | The automated scaling gates also compare equal row/match counts at narrow and | |
| 86 | - | wide row widths; `B/op` may grow by at most 2x, a conservative relative | |
| 87 | - | allowance that catches full cell-row copies without depending on an absolute | |
| 88 | - | Go-version-specific budget. This scope does not yet include an | |
| 89 | - | incremental/full-text search index: queries still scan the immutable document | |
| 90 | - | and produce a fresh match list. Chunk encoding, compression, GC tuning, and real Unix-socket, SSH, or UDP | |
| 91 | - | impairment/latency remain out of scope. | |
| 92 | - | ||
| 93 | - | ## Lower-level smoke baseline | |
| 94 | - | ||
| 95 | - | Use this command to check renderer, VT, IPC, and daemon microbenchmarks without | |
| 96 | - | claiming an end-to-end transport result: | |
| 63 | + | These commands measure local daemon work only and are not end-to-end transport | |
| 64 | + | results: | |
| 97 | 65 | ||
| 98 | 66 | ```sh | |
| 67 | + | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1x -benchmem | |
| 99 | 68 | go test ./pkg/renderer ./pkg/vt ./internal/adapters/ipc -run '^$' -bench=. -benchmem | |
| 100 | 69 | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkComposeFloatingFrameCached$' -benchmem | |
| 101 | 70 | ``` | |
| 102 | - | ||
| 103 | - | Enable daemon pprof explicitly when collecting a daemon profile: | |
| 104 | - | ||
| 105 | - | ```sh | |
| 106 | - | VEV_PPROF_ADDR=127.0.0.1:6060 vev --daemon | |
| 107 | - | ``` | |
| @@ -1,107 +1,70 @@ | |||||
| 1 | - | # Performance methodology (M5) | 1 | + | # Performance measurement |
| 2 | - | 2 | + | ||
| 3 | - | This baseline measures in-process daemon work only. It includes coordinator-enabled | 3 | + | ## S5a transport observability |
| 4 | - | fixture scheduling metrics, but deliberately does **not** measure chunk | 4 | + | |
| 5 | - | encoding, compression, GC tuning, or real network latency. Output metrics count frames and encoded output data produced | 5 | + | S5a is measurement-only. It records process-local runtime marks and harness-owned |
| 6 | - | by the daemon; snapshot writes and serialized bytes are local snapshot-store | 6 | + | end-to-end boundaries without changing rendering, pacing, transport policy, wire |
| 7 | - | proxies. These metrics approximate local rendering, IPC payload, and persistence | 7 | + | bytes, or `ProtocolVersion`. The canonical matrix and workload/topology/ |
| 8 | - | work, not a Unix socket, SSH, or UDP transport. Real transport impairment and | 8 | + | transport coverage are in [`testdata/perf/manifest.json`](../testdata/perf/manifest.json): |
| 9 | - | latency benchmarks are deferred. | 9 | + | `1x4`, `4x1`, `4x4`, and `8x1`, all at `120x40` with 10,000 rows per pane; |
| 10 | - | 10 | + | idle, active/all/inactive output, interactive flood, copy search, resize sweep, | |
| 11 | - | ## Large-history daemon baseline | 11 | + | snapshot/output/resize, and attach/restore/tab switch; local, SSH stdio, UDP |
| 12 | - | 12 | + | baseline, UDP 25/100 ms, and UDP 1% loss. It has 216 scenarios (4×9×6), with | |
| 13 | - | Use a single iteration only as a smoke check; it is not suitable for drawing | 13 | + | inapplicable cases required to be explicit. |
| 14 | - | performance conclusions: | 14 | + | |
| 15 | - | 15 | + | ### Trace schema and clock domains | |
| 16 | - | ```sh | 16 | + | |
| 17 | - | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1x -benchmem | 17 | + | Every process JSONL record has exactly these fields: |
| 18 | - | ``` | 18 | + | `schema` (uint), `process_id` (string), `component` (string), `scenario` |
| 19 | - | 19 | + | (string), `run`, `sequence`, `request_id`, `epoch` (uint), `kind` (string), | |
| 20 | - | For before/after comparison, collect repeated samples for the **full matrix** | 20 | + | `tick` (int64), `bytes`, `fragments`, `retransmits`, `pending` (uint), |
| 21 | - | (all workloads and topologies), pinning CPU/governor and recording the host. | 21 | + | `ack_rtt_nanos` (int64), and `valid` (bool). Schema 1 is represented by: |
| 22 | - | The benchmark harness must be committed and present in **both** measured | 22 | + | |
| 23 | - | revisions: collect the baseline after the harness is introduced but before the | 23 | + | ```json |
| 24 | - | optimization, then collect the optimized revision with that same harness. The | 24 | + | {"schema":1,"process_id":"p","component":"daemon","scenario":"s","run":1,"sequence":1,"request_id":1,"epoch":1,"kind":"diff_start","tick":0,"bytes":0,"fragments":0,"retransmits":0,"pending":0,"ack_rtt_nanos":0,"valid":true} |
| 25 | - | fixed duration and count make the output suitable for `benchstat`: | |||
| 26 | - | ||||
| 27 | - | ```sh | |||
| 28 | - | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1s -count=5 -benchmem > /tmp/vev-history-after.txt | |||
| 29 | - | benchstat /tmp/vev-history-before.txt /tmp/vev-history-after.txt | |||
| 30 | ``` | 25 | ``` | ||
| 31 | 26 | ||||
| 32 | - | The matrix exercises live paint, snapshot capture, copy-mode entry, copy search, | 27 | + | `tick` is stamped only by the concrete JSONL observer using its process-local |
| 33 | - | and resize at `120x40` client geometry. Each pane is populated with 10,000 | 28 | + | injected monotonic clock; producers supply no timestamp. Components in one OS |
| 34 | - | full-width deterministic history rows containing a stable `needle-<tab>-<row>` | 29 | + | process share that observer. The harness clock is a separate monotonic domain |
| 35 | - | prefix and patterned text. The layouts are: | 30 | + | and is the sole clock for input-injection → terminal-bytes-successfully-flushed |
| 31 | + | end-to-end samples. Cross-process ticks are never ordered or subtracted; | |||
| 32 | + | records correlate only by scenario/run/sequence/request/epoch IDs. | |||
| 36 | 33 | ||||
| 37 | - | - 1 tab x 1 pane (control) | 34 | + | The named process-local spans are capture, compose, diff, queue wait, |
| 38 | - | - 1 tab x 4 panes | 35 | + | ACK-blocked interval, emit, adapter send, and adapter receive. Their start/end |
| 39 | - | - 4 tabs x 1 pane | 36 | + | marks are paired within one process and summarized by component/adapter with |
| 40 | - | - 4 tabs x 4 panes | 37 | + | raw records, counts, p50/p95/p99, and max. Diagnostic fields (bytes, fragments, |
| 41 | - | - 8 tabs x 1 pane | 38 | + | retransmits, pending, ACK RTT) remain separate from duration calculations. |
| 42 | 39 | ||||
| 43 | - | The fixture starts no PTY readers or transports. It primes the renderer shadow | 40 | + | ### Baseline procedure and validity |
| 44 | - | before timing; live paint then alternates fixed writes through the coordinator. | |||
| 45 | - | The reported `outputframes/op` value counts output messages. `outputbytes/op` | |||
| 46 | - | counts their encoded terminal data and excludes the 24-byte output header; | |||
| 47 | - | `framepayloadbytes/op` counts the complete output payload including that header. | |||
| 48 | - | `snapshotwrites/op` and `snapshotbytes/op` are local snapshot-store writes and | |||
| 49 | - | serialized bytes. `coordinatorinvalidations/op`, `coordinatorwakes/op`, and | |||
| 50 | - | `coordinatorcoalesced/op` respectively count scheduling requests, delivered | |||
| 51 | - | wakes, and requests coalesced into those wakes; `coordinatorcoalescingratio` | |||
| 52 | - | is the average number of coalesced invalidations delivered per wake. They must | |||
| 53 | - | not be presented as network | |||
| 54 | - | throughput or latency. | |||
| 55 | 41 | ||||
| 56 | - | Capture a pre-change result outside the repository, including Go and kernel | 42 | + | Use the public CLI harness (not internal packages): |
| 57 | - | context: | |||
| 58 | 43 | ||||
| 59 | ```sh | 44 | ```sh | ||
| 60 | - | { | 45 | + | go build -o /tmp/vev-perf ./ |
| 61 | - | date -Is | 46 | + | rm -rf testdata/perf/results |
| 62 | - | go version | 47 | + | go run ./cmd/vev-perf-harness --vev-bin /tmp/vev-perf \ |
| 63 | - | uname -a | 48 | + | --manifest testdata/perf/manifest.json --out testdata/perf/results \ |
| 64 | - | git rev-parse HEAD | 49 | + | --warmup 10s --duration 30s --repetitions 10 |
| 65 | - | # Smoke only; do not use this result for comparisons. | |||
| 66 | - | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1x -benchmem | |||
| 67 | - | # Repeated full matrix for every workload/topology used in conclusions. | |||
| 68 | - | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1s -count=5 -benchmem | |||
| 69 | - | } > /tmp/vev-history-before.txt 2>&1 | |||
| 70 | ``` | 50 | ``` | ||
| 71 | 51 | ||||
| 72 | - | ## Immutable document allocation gate | 52 | + | The minimums are 30 seconds measured and 10 independent repetitions; warmup is |
| 53 | + | excluded. Each run must contain `manifest.json`, `raw-harness.jsonl`, per-process | |||
| 54 | + | JSONL, `runs.json`, and `summary.json`. Reject results with missing input/flush | |||
| 55 | + | boundaries, unmatched IDs, bad correlation, nonmonotonic same-process sequence, | |||
| 56 | + | negative same-domain spans, insufficient measured events, invalid denominators, | |||
| 57 | + | any cross-process tick arithmetic, or unmet duration/repetition minima. Raw | |||
| 58 | + | JSONL and run manifests are the evidence files; no result authorizes a policy | |||
| 59 | + | or optimization change. | |||
| 73 | 60 | ||||
| 74 | - | The copy-mode document retains an immutable view of existing scrollback rows and | 61 | + | ## Existing in-process smoke checks |
| 75 | - | clones only the visible frame. Visual-search models share that document; cloning | |||
| 76 | - | a model copies only its mutable input and match state. Check those two paths at | |||
| 77 | - | 10,000 history rows with: | |||
| 78 | 62 | ||||
| 79 | - | ```sh | 63 | + | These commands measure local daemon work only and are not end-to-end transport |
| 80 | - | go test ./internal/usecase/copy ./internal/usecase/visualsearch -run '^$' -bench '^(BenchmarkNewSnapshot10KRows|BenchmarkVisualSearchClone10KRows)$' -benchtime=1s -count=5 -benchmem | 64 | + | results: |
| 81 | - | ``` | |||
| 82 | - | ||||
| 83 | - | Treat `allocs/op` and `B/op` as allocation gates: compare the complete repeated | |||
| 84 | - | output with the accepted baseline before merging, and investigate any increase. | |||
| 85 | - | The automated scaling gates also compare equal row/match counts at narrow and | |||
| 86 | - | wide row widths; `B/op` may grow by at most 2x, a conservative relative | |||
| 87 | - | allowance that catches full cell-row copies without depending on an absolute | |||
| 88 | - | Go-version-specific budget. This scope does not yet include an | |||
| 89 | - | incremental/full-text search index: queries still scan the immutable document | |||
| 90 | - | and produce a fresh match list. Chunk encoding, compression, GC tuning, and real Unix-socket, SSH, or UDP | |||
| 91 | - | impairment/latency remain out of scope. | |||
| 92 | - | ||||
| 93 | - | ## Lower-level smoke baseline | |||
| 94 | - | ||||
| 95 | - | Use this command to check renderer, VT, IPC, and daemon microbenchmarks without | |||
| 96 | - | claiming an end-to-end transport result: | |||
| 97 | 65 | ||||
| 98 | ```sh | 66 | ```sh | ||
| 67 | + | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkDaemonHistory' -benchtime=1x -benchmem | |||
| 99 | go test ./pkg/renderer ./pkg/vt ./internal/adapters/ipc -run '^$' -bench=. -benchmem | 68 | go test ./pkg/renderer ./pkg/vt ./internal/adapters/ipc -run '^$' -bench=. -benchmem | ||
| 100 | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkComposeFloatingFrameCached$' -benchmem | 69 | go test ./internal/usecase/daemon -run '^$' -bench '^BenchmarkComposeFloatingFrameCached$' -benchmem | ||
| 101 | ``` | 70 | ``` | ||
| 102 | - | ||||
| 103 | - | Enable daemon pprof explicitly when collecting a daemon profile: | |||
| 104 | - | ||||
| 105 | - | ```sh | |||
| 106 | - | VEV_PPROF_ADDR=127.0.0.1:6060 vev --daemon | |||
| 107 | - | ``` | |||
M testdata/perf/README.md +43 -15
| @@ -1,24 +1,52 @@ | |||
| 1 | - | # Public transport-observability harness | |
| 1 | + | # S5a transport-observability evidence | |
| 2 | 2 | ||
| 3 | - | `manifest.json` is the complete S5a measurement matrix: 1x4, 4x1, 4x4, and | |
| 4 | - | 8x1 layouts at 120x40 with 10,000 rows per pane; every canonical workload; and | |
| 5 | - | local, SSH stdio, UDP baseline, 25/100ms RTT, and 1% loss transport fixtures. | |
| 6 | - | Each listed scenario is applicable. A future unavailable combination must remain | |
| 7 | - | in the manifest with `inapplicable_reason` and no launched roles; omission is | |
| 8 | - | invalid. | |
| 3 | + | `manifest.json` is the complete, measurement-only public-CLI matrix: four | |
| 4 | + | 120x40 topologies (`1x4`, `4x1`, `4x4`, `8x1`), 10,000 history rows per pane, | |
| 5 | + | nine workloads, and six transports (local, SSH stdio, UDP baseline, UDP 25 ms, | |
| 6 | + | UDP 100 ms, and UDP with 1% loss). It contains 216 scenario combinations; | |
| 7 | + | there are no omitted combinations. A genuinely unavailable combination must | |
| 8 | + | remain listed with `inapplicable_reason` and no launched roles. | |
| 9 | 9 | ||
| 10 | - | Run only after building the public binary: | |
| 10 | + | Build and run the harness with at least a 10-second warmup, 30-second measured | |
| 11 | + | interval, and 10 independent repetitions: | |
| 11 | 12 | ||
| 12 | 13 | ```sh | |
| 13 | 14 | go build -o /tmp/vev-perf ./ | |
| 15 | + | rm -rf testdata/perf/results | |
| 14 | 16 | go run ./cmd/vev-perf-harness --vev-bin /tmp/vev-perf \ | |
| 15 | - | --manifest testdata/perf/manifest.json --out /tmp/vev-perf-results \ | |
| 17 | + | --manifest testdata/perf/manifest.json --out testdata/perf/results \ | |
| 16 | 18 | --warmup 10s --duration 30s --repetitions 10 | |
| 17 | 19 | ``` | |
| 18 | 20 | ||
| 19 | - | The harness creates one `O_EXCL` JSONL path per launched role before launch and | |
| 20 | - | records the assigned process ID, clock domain, role, and path in that run's | |
| 21 | - | manifest. `raw-harness.jsonl` contains only harness-clock input/terminal-flush | |
| 22 | - | pairs. Process traces are correlated by IDs, never by timestamps across process | |
| 23 | - | clock domains. Results with missing pairs, bad order, insufficient samples, or | |
| 24 | - | cross-process tick arithmetic are invalid. | |
| 21 | + | Each run writes `manifest.json`, `raw-harness.jsonl`, one JSONL trace per | |
| 22 | + | launched process, `runs.json`, and `summary.json`. The process manifest records | |
| 23 | + | process ID, role, scenario/run, clock domain, exact exclusive trace path, and | |
| 24 | + | identity. The harness owns only the end-to-end boundaries: input injection to | |
| 25 | + | terminal bytes successfully flushed. It never subtracts ticks from daemon, | |
| 26 | + | client, or adapter processes. | |
| 27 | + | ||
| 28 | + | A process trace is JSONL with exactly these keys and types: | |
| 29 | + | ||
| 30 | + | ```json | |
| 31 | + | {"schema":1,"process_id":"p","component":"daemon","scenario":"s","run":1,"sequence":1,"request_id":1,"epoch":1,"kind":"diff_start","tick":0,"bytes":0,"fragments":0,"retransmits":0,"pending":0,"ack_rtt_nanos":0,"valid":true} | |
| 32 | + | ``` | |
| 33 | + | ||
| 34 | + | `tick` is owned and stamped by that process's injected monotonic clock. Producers | |
| 35 | + | must leave it zero; one observer/clock is shared by components in a process. | |
| 36 | + | Ticks from different `process_id` or clock domains are never ordered or | |
| 37 | + | subtracted. Correlation uses scenario, run, sequence, request ID, and epoch. | |
| 38 | + | ||
| 39 | + | Named process-local spans are: capture, compose, diff (`diff_start/end`), queue | |
| 40 | + | wait (`queue_enqueued/dequeued`), ACK-blocked interval, emit, adapter send, | |
| 41 | + | and adapter receive. Adapter spans are reported by component/transport; bytes, | |
| 42 | + | fragments, retransmits, pending, and ACK RTT remain diagnostics. Reports retain | |
| 43 | + | raw records and counts plus p50/p95/p99 (and max where applicable). End-to-end | |
| 44 | + | reports are harness-clock distributions only. | |
| 45 | + | ||
| 46 | + | Reject a result for missing or unmatched boundaries, missing flush, invalid | |
| 47 | + | correlation, nonmonotonic same-process sequence, negative same-domain duration, | |
| 48 | + | insufficient measured events, invalid denominators, cross-process tick math, | |
| 49 | + | less than 30 seconds measured, or fewer than 10 repetitions. Warmup is excluded | |
| 50 | + | from measured samples. Raw traces are evidence, not a transport-policy or | |
| 51 | + | optimization gate; wire bytes, protocol version, rendering policy, pacing, | |
| 52 | + | compression, and loss behavior are unchanged. | |
| @@ -1,24 +1,52 @@ | |||||
| 1 | - | # Public transport-observability harness | 1 | + | # S5a transport-observability evidence |
| 2 | 2 | ||||
| 3 | - | `manifest.json` is the complete S5a measurement matrix: 1x4, 4x1, 4x4, and | 3 | + | `manifest.json` is the complete, measurement-only public-CLI matrix: four |
| 4 | - | 8x1 layouts at 120x40 with 10,000 rows per pane; every canonical workload; and | 4 | + | 120x40 topologies (`1x4`, `4x1`, `4x4`, `8x1`), 10,000 history rows per pane, |
| 5 | - | local, SSH stdio, UDP baseline, 25/100ms RTT, and 1% loss transport fixtures. | 5 | + | nine workloads, and six transports (local, SSH stdio, UDP baseline, UDP 25 ms, |
| 6 | - | Each listed scenario is applicable. A future unavailable combination must remain | 6 | + | UDP 100 ms, and UDP with 1% loss). It contains 216 scenario combinations; |
| 7 | - | in the manifest with `inapplicable_reason` and no launched roles; omission is | 7 | + | there are no omitted combinations. A genuinely unavailable combination must |
| 8 | - | invalid. | 8 | + | remain listed with `inapplicable_reason` and no launched roles. |
| 9 | 9 | ||||
| 10 | - | Run only after building the public binary: | 10 | + | Build and run the harness with at least a 10-second warmup, 30-second measured |
| 11 | + | interval, and 10 independent repetitions: | |||
| 11 | 12 | ||||
| 12 | ```sh | 13 | ```sh | ||
| 13 | go build -o /tmp/vev-perf ./ | 14 | go build -o /tmp/vev-perf ./ | ||
| 15 | + | rm -rf testdata/perf/results | |||
| 14 | go run ./cmd/vev-perf-harness --vev-bin /tmp/vev-perf \ | 16 | go run ./cmd/vev-perf-harness --vev-bin /tmp/vev-perf \ | ||
| 15 | - | --manifest testdata/perf/manifest.json --out /tmp/vev-perf-results \ | 17 | + | --manifest testdata/perf/manifest.json --out testdata/perf/results \ |
| 16 | --warmup 10s --duration 30s --repetitions 10 | 18 | --warmup 10s --duration 30s --repetitions 10 | ||
| 17 | ``` | 19 | ``` | ||
| 18 | 20 | ||||
| 19 | - | The harness creates one `O_EXCL` JSONL path per launched role before launch and | 21 | + | Each run writes `manifest.json`, `raw-harness.jsonl`, one JSONL trace per |
| 20 | - | records the assigned process ID, clock domain, role, and path in that run's | 22 | + | launched process, `runs.json`, and `summary.json`. The process manifest records |
| 21 | - | manifest. `raw-harness.jsonl` contains only harness-clock input/terminal-flush | 23 | + | process ID, role, scenario/run, clock domain, exact exclusive trace path, and |
| 22 | - | pairs. Process traces are correlated by IDs, never by timestamps across process | 24 | + | identity. The harness owns only the end-to-end boundaries: input injection to |
| 23 | - | clock domains. Results with missing pairs, bad order, insufficient samples, or | 25 | + | terminal bytes successfully flushed. It never subtracts ticks from daemon, |
| 24 | - | cross-process tick arithmetic are invalid. | 26 | + | client, or adapter processes. |
| 27 | + | ||||
| 28 | + | A process trace is JSONL with exactly these keys and types: | |||
| 29 | + | ||||
| 30 | + | ```json | |||
| 31 | + | {"schema":1,"process_id":"p","component":"daemon","scenario":"s","run":1,"sequence":1,"request_id":1,"epoch":1,"kind":"diff_start","tick":0,"bytes":0,"fragments":0,"retransmits":0,"pending":0,"ack_rtt_nanos":0,"valid":true} | |||
| 32 | + | ``` | |||
| 33 | + | ||||
| 34 | + | `tick` is owned and stamped by that process's injected monotonic clock. Producers | |||
| 35 | + | must leave it zero; one observer/clock is shared by components in a process. | |||
| 36 | + | Ticks from different `process_id` or clock domains are never ordered or | |||
| 37 | + | subtracted. Correlation uses scenario, run, sequence, request ID, and epoch. | |||
| 38 | + | ||||
| 39 | + | Named process-local spans are: capture, compose, diff (`diff_start/end`), queue | |||
| 40 | + | wait (`queue_enqueued/dequeued`), ACK-blocked interval, emit, adapter send, | |||
| 41 | + | and adapter receive. Adapter spans are reported by component/transport; bytes, | |||
| 42 | + | fragments, retransmits, pending, and ACK RTT remain diagnostics. Reports retain | |||
| 43 | + | raw records and counts plus p50/p95/p99 (and max where applicable). End-to-end | |||
| 44 | + | reports are harness-clock distributions only. | |||
| 45 | + | ||||
| 46 | + | Reject a result for missing or unmatched boundaries, missing flush, invalid | |||
| 47 | + | correlation, nonmonotonic same-process sequence, negative same-domain duration, | |||
| 48 | + | insufficient measured events, invalid denominators, cross-process tick math, | |||
| 49 | + | less than 30 seconds measured, or fewer than 10 repetitions. Warmup is excluded | |||
| 50 | + | from measured samples. Raw traces are evidence, not a transport-policy or | |||
| 51 | + | optimization gate; wire bytes, protocol version, rendering policy, pacing, | |||
| 52 | + | compression, and loss behavior are unchanged. | |||