docs(perf): align observability manifest evidence

commit: 7f9117924b82a74ec7417777ab948045042104dc
author: brice · 3mo ago · 2026-07-13 00:22 UTC
parent: 9597b55
signature: GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email

2 files · +37 -11

M docs/performance.md +14 -5


@@ -9,8 +9,10 @@ transport coverage are in [`testdata/perf/manifest.json`](../testdata/perf/manif
9	9		`1x4`, `4x1`, `4x4`, and `8x1`, all at `120x40` with 10,000 rows per pane;
10	10		idle, active/all/inactive output, interactive flood, copy search, resize sweep,
11	11		snapshot/output/resize, and attach/restore/tab switch; local, SSH stdio, UDP
12		-	baseline, UDP 25/100 ms, and UDP 1% loss. It has 216 scenarios (4×9×6), with
13		-	inapplicable cases required to be explicit.
	12	+	baseline, UDP 25/100 ms, and UDP 0%/1% loss. It has 252 scenarios (4×9×7),
	13	+	with inapplicable cases required to be explicit. The seven transport entries are
	14	+	`local`, `ssh_stdio`, `udp_baseline`, `udp_loss_0pct`, `udp_25ms`, `udp_100ms`,
	15	+	and `udp_loss_1pct`.
14	16
15	17		### Trace schema and clock domains
16	18

@@ -25,7 +27,10 @@ Every process JSONL record has exactly these fields:
25	27		```
26	28
27	29		`tick` is stamped only by the concrete JSONL observer using its process-local
28		-	injected monotonic clock; producers supply no timestamp. Components in one OS
	30	+	injected monotonic clock; producers supply no timestamp. The harness sets `VEV_PERF_TRACE`,
	31	+	`VEV_PERF_PROCESS_ID`, `VEV_PERF_SCENARIO`, and `VEV_PERF_RUN` for every
	32	+	launched role, identifying its exclusive trace path and process/scenario/run
	33	+	mapping. Components in one OS
29	34		process share that observer. The harness clock is a separate monotonic domain
30	35		and is the sole clock for input-injection → terminal-bytes-successfully-flushed
31	36		end-to-end samples. Cross-process ticks are never ordered or subtracted;

@@ -50,10 +55,14 @@ go run ./cmd/vev-perf-harness --vev-bin /tmp/vev-perf \
50	55		```
51	56
52	57		The minimums are 30 seconds measured and 10 independent repetitions; warmup is
53		-	excluded. Each run must contain `manifest.json`, `raw-harness.jsonl`, per-process
	58	+	excluded. During the measured interval the harness injects at most one event per
	59	+	second. At least 10 complete event pairs are required: both injection and
	60	+	successful terminal flush must fall inside the measured interval; straddling or
	61	+	failed events are retained only as raw diagnostics and are not samples. Each run must contain `manifest.json`, `raw-harness.jsonl`, per-process
54	62		JSONL, `runs.json`, and `summary.json`. Reject results with missing input/flush
55	63		boundaries, unmatched IDs, bad correlation, nonmonotonic same-process sequence,
56		-	negative same-domain spans, insufficient measured events, invalid denominators,
	64	+	negative same-domain spans, fewer than 10 complete in-interval event pairs,
	65	+	invalid denominators,
57	66		any cross-process tick arithmetic, or unmet duration/repetition minima. Raw
58	67		JSONL and run manifests are the evidence files; no result authorizes a policy
59	68		or optimization change.

 `1x4`, `4x1`, `4x4`, and `8x1`, all at `120x40` with 10,000 rows per pane;
 idle, active/all/inactive output, interactive flood, copy search, resize sweep,
 snapshot/output/resize, and attach/restore/tab switch; local, SSH stdio, UDP
-baseline, UDP 25/100 ms, and UDP 1% loss. It has 216 scenarios (4×9×6), with
+baseline, UDP 25/100 ms, and UDP 0%/1% loss. It has 252 scenarios (4×9×7),
-inapplicable cases required to be explicit.
+with inapplicable cases required to be explicit. The seven transport entries are
+`local`, `ssh_stdio`, `udp_baseline`, `udp_loss_0pct`, `udp_25ms`, `udp_100ms`,
+and `udp_loss_1pct`.
 ### Trace schema and clock domains
 ```
 `tick` is stamped only by the concrete JSONL observer using its process-local
-injected monotonic clock; producers supply no timestamp. Components in one OS
+injected monotonic clock; producers supply no timestamp. The harness sets `VEV_PERF_TRACE`,
+`VEV_PERF_PROCESS_ID`, `VEV_PERF_SCENARIO`, and `VEV_PERF_RUN` for every
+launched role, identifying its exclusive trace path and process/scenario/run
+mapping. Components in one OS
 process share that observer. The harness clock is a separate monotonic domain
 and is the sole clock for input-injection → terminal-bytes-successfully-flushed
 end-to-end samples. Cross-process ticks are never ordered or subtracted;
 ```
 The minimums are 30 seconds measured and 10 independent repetitions; warmup is
-excluded. Each run must contain `manifest.json`, `raw-harness.jsonl`, per-process
+excluded. During the measured interval the harness injects at most one event per
+second. At least 10 complete event pairs are required: both injection and
+successful terminal flush must fall inside the measured interval; straddling or
+failed events are retained only as raw diagnostics and are not samples. Each run must contain `manifest.json`, `raw-harness.jsonl`, per-process
 JSONL, `runs.json`, and `summary.json`. Reject results with missing input/flush
 boundaries, unmatched IDs, bad correlation, nonmonotonic same-process sequence,
-negative same-domain spans, insufficient measured events, invalid denominators,
+negative same-domain spans, fewer than 10 complete in-interval event pairs,
+invalid denominators,
 any cross-process tick arithmetic, or unmet duration/repetition minima. Raw
 JSONL and run manifests are the evidence files; no result authorizes a policy
 or optimization change.

M testdata/perf/README.md +23 -6


@@ -21,7 +21,9 @@ go run ./cmd/vev-perf-harness --vev-bin /tmp/vev-perf \
21	21		Each run writes `manifest.json`, `raw-harness.jsonl`, one JSONL trace per
22	22		launched process, `runs.json`, and `summary.json`. The process manifest records
23	23		process ID, role, scenario/run, clock domain, exact exclusive trace path, and
24		-	identity. The harness owns only the end-to-end boundaries: input injection to
	24	+	identity. Every launched role receives `VEV_PERF_TRACE`, `VEV_PERF_PROCESS_ID`,
	25	+	`VEV_PERF_SCENARIO`, and `VEV_PERF_RUN`; the first is exclusive to that process
	26	+	and the remaining values correlate it to the manifest. The harness owns only the end-to-end boundaries: input injection to
25	27		terminal bytes successfully flushed. It never subtracts ticks from daemon,
26	28		client, or adapter processes.
27	29

@@ -43,10 +45,25 @@ fragments, retransmits, pending, and ACK RTT remain diagnostics. Reports retain
43	45		raw records and counts plus p50/p95/p99 (and max where applicable). End-to-end
44	46		reports are harness-clock distributions only.
45	47
46		-	Reject a result for missing or unmatched boundaries, missing flush, invalid
47		-	correlation, nonmonotonic same-process sequence, negative same-domain duration,
48		-	insufficient measured events, invalid denominators, cross-process tick math,
49		-	less than 30 seconds measured, or fewer than 10 repetitions. Warmup is excluded
50		-	from measured samples. Raw traces are evidence, not a transport-policy or
	48	+	The harness injects at most one measured event per second and requires at least
	49	+	10 complete in-interval event pairs. Both the input-injection and successful
	50	+	terminal-flush boundaries must be within the measured interval; warmup and
	51	+	straddling pairs are excluded from percentile samples (but retained as raw
	52	+	evidence). Reject a result for missing or unmatched boundaries, missing flush,
	53	+	invalid correlation, nonmonotonic same-process sequence, negative same-domain
	54	+	duration, fewer than 10 complete in-interval pairs, invalid denominators,
	55	+	cross-process tick math, less than 30 seconds measured, or fewer than 10
	56	+	repetitions. Warmup is excluded from measured samples. Raw traces are evidence, not a transport-policy or
51	57		optimization gate; wire bytes, protocol version, rendering policy, pacing,
52	58		compression, and loss behavior are unchanged.
	59	+
	60	+	## Bounded manifest and harness checks
	61	+
	62	+	Before a measurement, validate the manifest shape and run the harness tests without launching the full matrix:
	63	+
	64	+	```sh
	65	+	jq -e '(.topologies\|length)==4 and (.workloads\|length)==9 and (.transports\|length)==7 and (.scenarios\|length)==252' testdata/perf/manifest.json
	66	+	go test ./cmd/vev-perf-harness
	67	+	```
	68	+
	69	+	The full command above is the measurement procedure; its 10-second warmup, 30-second measured interval, and 10 repetitions are the minimum accepted settings.


@@ -21,7 +21,9 @@ go run ./cmd/vev-perf-harness --vev-bin /tmp/vev-perf \
21		Each run writes `manifest.json`, `raw-harness.jsonl`, one JSONL trace per	21		Each run writes `manifest.json`, `raw-harness.jsonl`, one JSONL trace per
22		launched process, `runs.json`, and `summary.json`. The process manifest records	22		launched process, `runs.json`, and `summary.json`. The process manifest records
23		process ID, role, scenario/run, clock domain, exact exclusive trace path, and	23		process ID, role, scenario/run, clock domain, exact exclusive trace path, and
24	-	identity. The harness owns only the end-to-end boundaries: input injection to	24	+	identity. Every launched role receives `VEV_PERF_TRACE`, `VEV_PERF_PROCESS_ID`,
			25	+	`VEV_PERF_SCENARIO`, and `VEV_PERF_RUN`; the first is exclusive to that process
			26	+	and the remaining values correlate it to the manifest. The harness owns only the end-to-end boundaries: input injection to
25		terminal bytes successfully flushed. It never subtracts ticks from daemon,	27		terminal bytes successfully flushed. It never subtracts ticks from daemon,
26		client, or adapter processes.	28		client, or adapter processes.
27			29

@@ -43,10 +45,25 @@ fragments, retransmits, pending, and ACK RTT remain diagnostics. Reports retain
43		raw records and counts plus p50/p95/p99 (and max where applicable). End-to-end	45		raw records and counts plus p50/p95/p99 (and max where applicable). End-to-end
44		reports are harness-clock distributions only.	46		reports are harness-clock distributions only.
45			47
46	-	Reject a result for missing or unmatched boundaries, missing flush, invalid	48	+	The harness injects at most one measured event per second and requires at least
47	-	correlation, nonmonotonic same-process sequence, negative same-domain duration,	49	+	10 complete in-interval event pairs. Both the input-injection and successful
48	-	insufficient measured events, invalid denominators, cross-process tick math,	50	+	terminal-flush boundaries must be within the measured interval; warmup and
49	-	less than 30 seconds measured, or fewer than 10 repetitions. Warmup is excluded	51	+	straddling pairs are excluded from percentile samples (but retained as raw
50	-	from measured samples. Raw traces are evidence, not a transport-policy or	52	+	evidence). Reject a result for missing or unmatched boundaries, missing flush,
			53	+	invalid correlation, nonmonotonic same-process sequence, negative same-domain
			54	+	duration, fewer than 10 complete in-interval pairs, invalid denominators,
			55	+	cross-process tick math, less than 30 seconds measured, or fewer than 10
			56	+	repetitions. Warmup is excluded from measured samples. Raw traces are evidence, not a transport-policy or
51		optimization gate; wire bytes, protocol version, rendering policy, pacing,	57		optimization gate; wire bytes, protocol version, rendering policy, pacing,
52		compression, and loss behavior are unchanged.	58		compression, and loss behavior are unchanged.
			59	+
			60	+	## Bounded manifest and harness checks
			61	+
			62	+	Before a measurement, validate the manifest shape and run the harness tests without launching the full matrix:
			63	+
			64	+	```sh
			65	+	jq -e '(.topologies\|length)==4 and (.workloads\|length)==9 and (.transports\|length)==7 and (.scenarios\|length)==252' testdata/perf/manifest.json
			66	+	go test ./cmd/vev-perf-harness
			67	+	```
			68	+
			69	+	The full command above is the measurement procedure; its 10-second warmup, 30-second measured interval, and 10 repetitions are the minimum accepted settings.