detached be629c4

#3 WIP: feat(hint): keyboard hint mode for links and buttons

state
closed
branch
feat/hint-mode → main
opened
bnema ·
updated

Adds a hint action: NeferWL captures the focused output internally (no client, no capture indicator), finds links, URLs and buttons with bnema.dev/hintscan, shows short labels, and clicks the chosen target through its own pointer path.

  • Core: hint state, keyboard grab (letters, BackSpace, Escape), synthetic click guarded against held modifiers, pointer constraints, grabs, held buttons and drags; refused while locked.
  • App: internal capture IDs (bit 63) and a reply demux; hints adapter wired.
  • Vulkan: labels rasterized with Go Mono (internal/adapters/hintlabel), drawn over everything, never in captures; direct scanout off while labels show.
  • Config: hint action (no default bind), hint.targets = clickable|links.

Blocker before merge: go.mod has replace bnema.dev/hintscan => ../../../hintscan/.worktrees/scanner. CI fails until hintscan v0.1.0 is published (hintscan!1) and the replace is removed.

Checks: make check, make race, make mocks-check, staticcheck ./... pass locally. Headless e2e (Chromium, 1280x720): labels page 11 targets (9 links, 2 buttons), capture ~1 ms, scan ~2 ms, no indicator; click page Super+g, f → page logs hint-clicked; Escape cancels without a click.

Known visual follow-ups: labels cover the first glyph of their target; label font rendering to check.

CommitSubjectAuthorDateSignatureStatus
be629c4feat(hints): dump scanned frames when NEFERWL_HINT_DEBUG is setbriceGPG Verified
8ed066edocs(config): document hint modebriceGPG Verified
2a96044feat(app): wire the hints adapterbriceGPG Verified
4b455b2feat(hints): capture and scan the output for hint targetsbriceGPG Verified
df4b2febuild(deps): use local hintscan modulebriceGPG Verified
09a5c8cfix(core): make hint clicks safe against modifiers, constraints and grabsbriceGPG Verified
90d23d4fix(capture): keep hint labels out of captured framesbriceGPG Verified
9ad3054feat(vulkan): draw hint labelsbriceGPG Verified
a329128feat(core): add keyboard hint modebriceGPG Verified
1824dbdfeat(app): route internal capture repliesbriceGPG Verified
7f28d9cfeat(ports): add hint scan messages and internal capture idsbriceGPG Verified

Checks 1

44 files · +2287 -24
0/1
M .hexcheck.yaml +1 -1
@@ -25,7 +25,7 @@ allow:
2525 ruleSettings:
2626 missing-generated-mock-for-port:
2727 # Sealed message unions are channel data, not ports; tested by sending messages (ADR 005).
28- excludePaths: [internal/ports/client_events.go, internal/ports/client_commands.go, internal/ports/input_events.go, internal/ports/output_events.go, internal/ports/lease.go, internal/ports/session_lock_messages.go, internal/ports/screencast_messages.go, internal/ports/session_backend.go]
28+ excludePaths: [internal/ports/client_events.go, internal/ports/client_commands.go, internal/ports/input_events.go, internal/ports/output_events.go, internal/ports/lease.go, internal/ports/session_lock_messages.go, internal/ports/screencast_messages.go, internal/ports/session_backend.go, internal/ports/hint_messages.go]
2929 suspicious-business-logic-in-adapter:
3030 # app is the composition root; starting core and wiring channels is its job.
3131 excludePaths: [internal/app/app.go, internal/app/session.go]
@@ -25,7 +25,7 @@ allow:
25 ruleSettings:25 ruleSettings:
26 missing-generated-mock-for-port:26 missing-generated-mock-for-port:
27 # Sealed message unions are channel data, not ports; tested by sending messages (ADR 005).27 # Sealed message unions are channel data, not ports; tested by sending messages (ADR 005).
28- excludePaths: [internal/ports/client_events.go, internal/ports/client_commands.go, internal/ports/input_events.go, internal/ports/output_events.go, internal/ports/lease.go, internal/ports/session_lock_messages.go, internal/ports/screencast_messages.go, internal/ports/session_backend.go]28+ excludePaths: [internal/ports/client_events.go, internal/ports/client_commands.go, internal/ports/input_events.go, internal/ports/output_events.go, internal/ports/lease.go, internal/ports/session_lock_messages.go, internal/ports/screencast_messages.go, internal/ports/session_backend.go, internal/ports/hint_messages.go]
29 suspicious-business-logic-in-adapter:29 suspicious-business-logic-in-adapter:
30 # app is the composition root; starting core and wiring channels is its job.30 # app is the composition root; starting core and wiring channels is its job.
31 excludePaths: [internal/app/app.go, internal/app/session.go]31 excludePaths: [internal/app/app.go, internal/app/session.go]
M Makefile +1 -1
@@ -61,7 +61,7 @@ arch: adapter-imports
6161 # Adapters stay independent: production code may import only the shared
6262 # adapter libraries below. outputkit holds what both output backends (drm,
6363 # headless) run; the others are leaf helpers.
64-ADAPTER_LIBS := fdwait|busretry|outputkit/capture|outputkit/corewake|outputkit/presented|outputkit/surfaces|captureallow|clock|logging|peerexe|sessionsecurity|syncfile|workspaceid|xkb
64+ADAPTER_LIBS := fdwait|busretry|hintlabel|outputkit/capture|outputkit/corewake|outputkit/presented|outputkit/surfaces|captureallow|clock|logging|peerexe|sessionsecurity|syncfile|workspaceid|xkb
6565 adapter-imports:
6666 @edges=$$(CGO_ENABLED=0 go list -f '{{$$p := .ImportPath}}{{range .Imports}}{{$$p}} {{.}}{{"\n"}}{{end}}' ./internal/adapters/...) || exit 1; \
6767 bad=$$(echo "$$edges" | grep -E ' \S+/internal/adapters/' | grep -vE '/internal/adapters/($(ADAPTER_LIBS))$$' \
@@ -61,7 +61,7 @@ arch: adapter-imports
61 # Adapters stay independent: production code may import only the shared61 # Adapters stay independent: production code may import only the shared
62 # adapter libraries below. outputkit holds what both output backends (drm,62 # adapter libraries below. outputkit holds what both output backends (drm,
63 # headless) run; the others are leaf helpers.63 # headless) run; the others are leaf helpers.
64-ADAPTER_LIBS := fdwait|busretry|outputkit/capture|outputkit/corewake|outputkit/presented|outputkit/surfaces|captureallow|clock|logging|peerexe|sessionsecurity|syncfile|workspaceid|xkb64+ADAPTER_LIBS := fdwait|busretry|hintlabel|outputkit/capture|outputkit/corewake|outputkit/presented|outputkit/surfaces|captureallow|clock|logging|peerexe|sessionsecurity|syncfile|workspaceid|xkb
65 adapter-imports:65 adapter-imports:
66 @edges=$$(CGO_ENABLED=0 go list -f '{{$$p := .ImportPath}}{{range .Imports}}{{$$p}} {{.}}{{"\n"}}{{end}}' ./internal/adapters/...) || exit 1; \66 @edges=$$(CGO_ENABLED=0 go list -f '{{$$p := .ImportPath}}{{range .Imports}}{{$$p}} {{.}}{{"\n"}}{{end}}' ./internal/adapters/...) || exit 1; \
67 bad=$$(echo "$$edges" | grep -E ' \S+/internal/adapters/' | grep -vE '/internal/adapters/($(ADAPTER_LIBS))$$' \67 bad=$$(echo "$$edges" | grep -E ' \S+/internal/adapters/' | grep -vE '/internal/adapters/($(ADAPTER_LIBS))$$' \
M README.md +1 -0
@@ -281,6 +281,7 @@ bind.cmd+shift+f = toggle-fullscreen
281281 bind.cmd+s = toggle-stash-visible
282282 bind.cmd+shift+s = toggle-window-stash
283283 bind.cmd+o = toggle-overview
284+# bind.cmd+g = hint
284285 bind.cmd+tab = switch-column-next
285286 bind.cmd+shift+tab = switch-column-prev
286287
@@ -281,6 +281,7 @@ bind.cmd+shift+f = toggle-fullscreen
281 bind.cmd+s = toggle-stash-visible281 bind.cmd+s = toggle-stash-visible
282 bind.cmd+shift+s = toggle-window-stash282 bind.cmd+shift+s = toggle-window-stash
283 bind.cmd+o = toggle-overview283 bind.cmd+o = toggle-overview
284+# bind.cmd+g = hint
284 bind.cmd+tab = switch-column-next285 bind.cmd+tab = switch-column-next
285 bind.cmd+shift+tab = switch-column-prev286 bind.cmd+shift+tab = switch-column-prev
286 287
M docs/config.md +18 -0
@@ -55,6 +55,7 @@ See [examples/config](../examples/config) for every key with its default.
5555 | `focus.flash-opacity` | `0.6` | Starting opacity of the flash, `0.1` to `1` |
5656 | `focus.flash-duration` | `200ms` | Fade duration, `50ms` to `1s`; not scaled by `animations.speed` |
5757 | `focus.flash-color` | `border.active` | Flash color `#rrggbb`; empty uses `border.active` |
58+| `hint.targets` | `clickable` | What [hint mode](#hint-mode) labels: `clickable` (links, URLs and buttons) or `links` (links and URLs only) |
5859 | `workspace.<name>.monitor` | focused output | Home monitor of a named workspace |
5960 | `workspace.<name>.size` | `inherit` | [Smaller viewport](#workspace-size), e.g. `1920x1080` |
6061 | `workspace.<name>.max-columns`, `.overflow` | output values | Per-workspace layout |
@@ -191,6 +192,7 @@ bind.XF86MonBrightnessDown = spawn brightnessctl set 5%-
191192 | `toggle-window-stash` | Move the focused window to the [stash](#stash), or back to its column |
192193 | `toggle-stash-visible` | Show or hide the stash |
193194 | `toggle-overview` | Open or close the [overview](#overview) |
195+| `hint` | Label the links and buttons on the focused screen, then click the one you type ([hint mode](#hint-mode)) |
194196 | `switch-column-next` / `-prev` | Switch to the previously used column. Hold `cmd` and press again to go further back |
195197 | `maximize-column` | Toggle full width for the focused column |
196198 | `cycle-column-width` | Grow the focused column by one [cell](#column-widths), then back to equal widths |
@@ -295,3 +297,19 @@ A workspace's stash shows as a pile of cards. Selecting the pile unfolds it into
295297 With `layout.overflow = cascade`, workspaces sit side by side, each band of columns is a lane, and the stash sits under the lanes. Down (or fingers down on the touchpad, with natural scroll off) moves to the lane below; past the last lane it unfolds the stash. Left/right walk the cards; where the carousel does not wrap round, they go to the neighbor workspace past either end. Up leaves the stash for the last lane, at the column you came from.
296298
297299 Otherwise, left from the first column unfolds the stash. Up/down walk the cards, the focus-workspace binds change workspace, and right goes back to the columns.
300+
301+### Hint mode
302+
303+`hint` labels the links, URLs and buttons visible on the focused screen, so you can click one from the keyboard. No bind is set by default; for example `bind.cmd+g = hint`.
304+
305+- Each target gets a label of lowercase letters. With up to 26 targets the labels are one letter, with more they are two.
306+- Type a label to click its target: the pointer moves to the center of the target and presses the left button. With two-letter labels, the labels that do not match what you typed disappear and the typed letters are dimmed.
307+- `BackSpace` removes the last typed letter. `Escape` cancels, and so does any other key, which no window receives.
308+- Only the visible area of windows is scanned: a target covered by another window or outside the screen has no label. Targets on other screens are not labeled.
309+- `hint.targets = links` labels links and URLs only, not buttons.
310+- `hint` does nothing while the session is locked, on a rotated or flipped screen, while the overview is open, while a pointer button, drag or pointer constraint is active, or while hint mode is already active. Hint mode ends when the workspace on that screen changes or the overview opens.
311+- No capture indicator appears: the scan happens inside NeferWL and no client receives the pixels. Screen captures do not contain the labels.
312+
313+#### Debugging hint mode
314+
315+With `NEFERWL_HINT_DEBUG=1` set when NeferWL starts (for example `NEFERWL_HINT_DEBUG=1 make tty`, or the variable in the launch environment), every successful scan writes three files to `$XDG_STATE_HOME/neferwl/hints/` (`~/.local/state/neferwl/hints/` when unset): `<time>-<output>.png`, the frame as scanned; `<time>-<output>.annotated.png`, the same with each target outlined (links green, buttons blue, text grey); and `<time>-<output>.json`, the targets and timings. NeferWL keeps the 50 most recent scans and logs a warning at startup. The files contain the screen in clear: delete them after use.
@@ -55,6 +55,7 @@ See [examples/config](../examples/config) for every key with its default.
55 | `focus.flash-opacity` | `0.6` | Starting opacity of the flash, `0.1` to `1` |55 | `focus.flash-opacity` | `0.6` | Starting opacity of the flash, `0.1` to `1` |
56 | `focus.flash-duration` | `200ms` | Fade duration, `50ms` to `1s`; not scaled by `animations.speed` |56 | `focus.flash-duration` | `200ms` | Fade duration, `50ms` to `1s`; not scaled by `animations.speed` |
57 | `focus.flash-color` | `border.active` | Flash color `#rrggbb`; empty uses `border.active` |57 | `focus.flash-color` | `border.active` | Flash color `#rrggbb`; empty uses `border.active` |
58+| `hint.targets` | `clickable` | What [hint mode](#hint-mode) labels: `clickable` (links, URLs and buttons) or `links` (links and URLs only) |
58 | `workspace.<name>.monitor` | focused output | Home monitor of a named workspace |59 | `workspace.<name>.monitor` | focused output | Home monitor of a named workspace |
59 | `workspace.<name>.size` | `inherit` | [Smaller viewport](#workspace-size), e.g. `1920x1080` |60 | `workspace.<name>.size` | `inherit` | [Smaller viewport](#workspace-size), e.g. `1920x1080` |
60 | `workspace.<name>.max-columns`, `.overflow` | output values | Per-workspace layout |61 | `workspace.<name>.max-columns`, `.overflow` | output values | Per-workspace layout |
@@ -191,6 +192,7 @@ bind.XF86MonBrightnessDown = spawn brightnessctl set 5%-
191 | `toggle-window-stash` | Move the focused window to the [stash](#stash), or back to its column |192 | `toggle-window-stash` | Move the focused window to the [stash](#stash), or back to its column |
192 | `toggle-stash-visible` | Show or hide the stash |193 | `toggle-stash-visible` | Show or hide the stash |
193 | `toggle-overview` | Open or close the [overview](#overview) |194 | `toggle-overview` | Open or close the [overview](#overview) |
195+| `hint` | Label the links and buttons on the focused screen, then click the one you type ([hint mode](#hint-mode)) |
194 | `switch-column-next` / `-prev` | Switch to the previously used column. Hold `cmd` and press again to go further back |196 | `switch-column-next` / `-prev` | Switch to the previously used column. Hold `cmd` and press again to go further back |
195 | `maximize-column` | Toggle full width for the focused column |197 | `maximize-column` | Toggle full width for the focused column |
196 | `cycle-column-width` | Grow the focused column by one [cell](#column-widths), then back to equal widths |198 | `cycle-column-width` | Grow the focused column by one [cell](#column-widths), then back to equal widths |
@@ -295,3 +297,19 @@ A workspace's stash shows as a pile of cards. Selecting the pile unfolds it into
295 With `layout.overflow = cascade`, workspaces sit side by side, each band of columns is a lane, and the stash sits under the lanes. Down (or fingers down on the touchpad, with natural scroll off) moves to the lane below; past the last lane it unfolds the stash. Left/right walk the cards; where the carousel does not wrap round, they go to the neighbor workspace past either end. Up leaves the stash for the last lane, at the column you came from.297 With `layout.overflow = cascade`, workspaces sit side by side, each band of columns is a lane, and the stash sits under the lanes. Down (or fingers down on the touchpad, with natural scroll off) moves to the lane below; past the last lane it unfolds the stash. Left/right walk the cards; where the carousel does not wrap round, they go to the neighbor workspace past either end. Up leaves the stash for the last lane, at the column you came from.
296 298
297 Otherwise, left from the first column unfolds the stash. Up/down walk the cards, the focus-workspace binds change workspace, and right goes back to the columns.299 Otherwise, left from the first column unfolds the stash. Up/down walk the cards, the focus-workspace binds change workspace, and right goes back to the columns.
300+
301+### Hint mode
302+
303+`hint` labels the links, URLs and buttons visible on the focused screen, so you can click one from the keyboard. No bind is set by default; for example `bind.cmd+g = hint`.
304+
305+- Each target gets a label of lowercase letters. With up to 26 targets the labels are one letter, with more they are two.
306+- Type a label to click its target: the pointer moves to the center of the target and presses the left button. With two-letter labels, the labels that do not match what you typed disappear and the typed letters are dimmed.
307+- `BackSpace` removes the last typed letter. `Escape` cancels, and so does any other key, which no window receives.
308+- Only the visible area of windows is scanned: a target covered by another window or outside the screen has no label. Targets on other screens are not labeled.
309+- `hint.targets = links` labels links and URLs only, not buttons.
310+- `hint` does nothing while the session is locked, on a rotated or flipped screen, while the overview is open, while a pointer button, drag or pointer constraint is active, or while hint mode is already active. Hint mode ends when the workspace on that screen changes or the overview opens.
311+- No capture indicator appears: the scan happens inside NeferWL and no client receives the pixels. Screen captures do not contain the labels.
312+
313+#### Debugging hint mode
314+
315+With `NEFERWL_HINT_DEBUG=1` set when NeferWL starts (for example `NEFERWL_HINT_DEBUG=1 make tty`, or the variable in the launch environment), every successful scan writes three files to `$XDG_STATE_HOME/neferwl/hints/` (`~/.local/state/neferwl/hints/` when unset): `<time>-<output>.png`, the frame as scanned; `<time>-<output>.annotated.png`, the same with each target outlined (links green, buttons blue, text grey); and `<time>-<output>.json`, the targets and timings. NeferWL keeps the 50 most recent scans and logs a warning at startup. The files contain the screen in clear: delete them after use.
M examples/config +5 -0
@@ -94,6 +94,9 @@ focus.flash-width = 2
9494 focus.flash-opacity = 0.6
9595 focus.flash-duration = 200ms
9696 focus.flash-color =
97+# Hint mode labels the links, URLs and buttons on the focused screen:
98+# clickable (links, URLs and buttons) or links (links and URLs only).
99+# hint.targets = clickable
97100
98101 # Named workspaces are outside the numbered list and require a bind. The bind
99102 # shows one; pressing it again returns to the previous workspace.
@@ -195,6 +198,8 @@ bind.cmd+shift+f = toggle-fullscreen
195198 bind.cmd+s = toggle-stash-visible
196199 bind.cmd+shift+s = toggle-window-stash
197200 bind.cmd+o = toggle-overview
201+# Label the links and buttons on screen and click one by typing its label.
202+# bind.cmd+g = hint
198203 bind.cmd+q = close-window
199204 bind.ctrl+alt+backspace = quit
200205 bind.ctrl+cmd+space = spawn fuzzel
@@ -94,6 +94,9 @@ focus.flash-width = 2
94 focus.flash-opacity = 0.694 focus.flash-opacity = 0.6
95 focus.flash-duration = 200ms95 focus.flash-duration = 200ms
96 focus.flash-color =96 focus.flash-color =
97+# Hint mode labels the links, URLs and buttons on the focused screen:
98+# clickable (links, URLs and buttons) or links (links and URLs only).
99+# hint.targets = clickable
97 100
98 # Named workspaces are outside the numbered list and require a bind. The bind101 # Named workspaces are outside the numbered list and require a bind. The bind
99 # shows one; pressing it again returns to the previous workspace.102 # shows one; pressing it again returns to the previous workspace.
@@ -195,6 +198,8 @@ bind.cmd+shift+f = toggle-fullscreen
195 bind.cmd+s = toggle-stash-visible198 bind.cmd+s = toggle-stash-visible
196 bind.cmd+shift+s = toggle-window-stash199 bind.cmd+shift+s = toggle-window-stash
197 bind.cmd+o = toggle-overview200 bind.cmd+o = toggle-overview
201+# Label the links and buttons on screen and click one by typing its label.
202+# bind.cmd+g = hint
198 bind.cmd+q = close-window203 bind.cmd+q = close-window
199 bind.ctrl+alt+backspace = quit204 bind.ctrl+alt+backspace = quit
200 bind.ctrl+cmd+space = spawn fuzzel205 bind.ctrl+cmd+space = spawn fuzzel
M go.mod +5 -0
@@ -15,14 +15,19 @@ require (
1515 github.com/godbus/dbus/v5 v5.2.2
1616 github.com/rs/zerolog v1.35.1
1717 github.com/stretchr/testify v1.12.1
18+ golang.org/x/image v0.46.0
1819 golang.org/x/sys v0.48.0
1920 golang.org/x/term v0.46.0
2021 )
2122
2223 require (
24+ bnema.dev/hintscan v0.0.0-00010101000000-000000000000
2325 github.com/mattn/go-colorable v0.1.15 // indirect
2426 github.com/mattn/go-isatty v0.0.22 // indirect
2527 github.com/stretchr/objx v0.5.3 // indirect
2628 go.yaml.in/yaml/v3 v3.0.5 // indirect
29+ golang.org/x/text v0.42.0 // indirect
2730 gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
2831 )
32+
33+replace bnema.dev/hintscan => ../../../hintscan/.worktrees/scanner
@@ -15,14 +15,19 @@ require (
15 github.com/godbus/dbus/v5 v5.2.215 github.com/godbus/dbus/v5 v5.2.2
16 github.com/rs/zerolog v1.35.116 github.com/rs/zerolog v1.35.1
17 github.com/stretchr/testify v1.12.117 github.com/stretchr/testify v1.12.1
18+ golang.org/x/image v0.46.0
18 golang.org/x/sys v0.48.019 golang.org/x/sys v0.48.0
19 golang.org/x/term v0.46.020 golang.org/x/term v0.46.0
20 )21 )
21 22
22 require (23 require (
24+ bnema.dev/hintscan v0.0.0-00010101000000-000000000000
23 github.com/mattn/go-colorable v0.1.15 // indirect25 github.com/mattn/go-colorable v0.1.15 // indirect
24 github.com/mattn/go-isatty v0.0.22 // indirect26 github.com/mattn/go-isatty v0.0.22 // indirect
25 github.com/stretchr/objx v0.5.3 // indirect27 github.com/stretchr/objx v0.5.3 // indirect
26 go.yaml.in/yaml/v3 v3.0.5 // indirect28 go.yaml.in/yaml/v3 v3.0.5 // indirect
29+ golang.org/x/text v0.42.0 // indirect
27 gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect30 gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
28 )31 )
32+
33+replace bnema.dev/hintscan => ../../../hintscan/.worktrees/scanner
M go.sum +4 -0 generated file: collapsed
M internal/adapters/config/config.go +10 -0
@@ -114,6 +114,7 @@ var actions = map[string]bool{
114114 "move-workspace-prev": true,
115115 "move-workspace-next": true,
116116 "toggle-floating": true,
117+ "hint": true,
117118 }
118119 var components = map[string]bool{"core": true, "wayland": true, "input": true, "drm": true, "seat": true, "render": true, "sync": true, "config": true, "app": true}
119120 var color = regexp.MustCompile(`^#[0-9a-fA-F]{6}$`)
@@ -847,6 +848,15 @@ func set(c *ports.Config, key, v string) error {
847848 return err
848849 }
849850 c.Focus.Flash = b
851+ case "hint.targets":
852+ switch v {
853+ case "clickable":
854+ c.Hint.LinksOnly = false
855+ case "links":
856+ c.Hint.LinksOnly = true
857+ default:
858+ return fmt.Errorf("must be clickable or links")
859+ }
850860 case "focus.flash-width":
851861 n, err := strconv.Atoi(v)
852862 if err != nil || n < 1 || n > 8 {
@@ -114,6 +114,7 @@ var actions = map[string]bool{
114 "move-workspace-prev": true,114 "move-workspace-prev": true,
115 "move-workspace-next": true,115 "move-workspace-next": true,
116 "toggle-floating": true,116 "toggle-floating": true,
117+ "hint": true,
117 }118 }
118 var components = map[string]bool{"core": true, "wayland": true, "input": true, "drm": true, "seat": true, "render": true, "sync": true, "config": true, "app": true}119 var components = map[string]bool{"core": true, "wayland": true, "input": true, "drm": true, "seat": true, "render": true, "sync": true, "config": true, "app": true}
119 var color = regexp.MustCompile(`^#[0-9a-fA-F]{6}$`)120 var color = regexp.MustCompile(`^#[0-9a-fA-F]{6}$`)
@@ -847,6 +848,15 @@ func set(c *ports.Config, key, v string) error {
847 return err848 return err
848 }849 }
849 c.Focus.Flash = b850 c.Focus.Flash = b
851+ case "hint.targets":
852+ switch v {
853+ case "clickable":
854+ c.Hint.LinksOnly = false
855+ case "links":
856+ c.Hint.LinksOnly = true
857+ default:
858+ return fmt.Errorf("must be clickable or links")
859+ }
850 case "focus.flash-width":860 case "focus.flash-width":
851 n, err := strconv.Atoi(v)861 n, err := strconv.Atoi(v)
852 if err != nil || n < 1 || n > 8 {862 if err != nil || n < 1 || n > 8 {
M internal/adapters/config/config_test.go +23 -0
@@ -953,3 +953,26 @@ func TestLoadRejectsFIFO(t *testing.T) {
953953 t.Fatal("Load blocked on a FIFO")
954954 }
955955 }
956+
957+func TestHintTargets(t *testing.T) {
958+ if Defaults().Hint.LinksOnly {
959+ t.Fatal("hint targets default must be clickable")
960+ }
961+ c, w := parseString(t, "hint.targets = links\n")
962+ if !c.Hint.LinksOnly || len(w) != 0 {
963+ t.Fatal(c.Hint, w)
964+ }
965+ c, w = parseString(t, "hint.targets = clickable\n")
966+ if c.Hint.LinksOnly || len(w) != 0 {
967+ t.Fatal(c.Hint, w)
968+ }
969+ // An invalid value warns and keeps the default.
970+ c, w = parseString(t, "hint.targets = buttons\n")
971+ if c.Hint.LinksOnly || len(w) != 1 {
972+ t.Fatal(c.Hint, w)
973+ }
974+ c, w = parseString(t, "bind.cmd+g = hint\n")
975+ if len(w) != 0 || c.Binds["Cmd+g"] != "hint" {
976+ t.Fatal(c.Binds["Cmd+g"], w)
977+ }
978+}
@@ -953,3 +953,26 @@ func TestLoadRejectsFIFO(t *testing.T) {
953 t.Fatal("Load blocked on a FIFO")953 t.Fatal("Load blocked on a FIFO")
954 }954 }
955 }955 }
956+
957+func TestHintTargets(t *testing.T) {
958+ if Defaults().Hint.LinksOnly {
959+ t.Fatal("hint targets default must be clickable")
960+ }
961+ c, w := parseString(t, "hint.targets = links\n")
962+ if !c.Hint.LinksOnly || len(w) != 0 {
963+ t.Fatal(c.Hint, w)
964+ }
965+ c, w = parseString(t, "hint.targets = clickable\n")
966+ if c.Hint.LinksOnly || len(w) != 0 {
967+ t.Fatal(c.Hint, w)
968+ }
969+ // An invalid value warns and keeps the default.
970+ c, w = parseString(t, "hint.targets = buttons\n")
971+ if c.Hint.LinksOnly || len(w) != 1 {
972+ t.Fatal(c.Hint, w)
973+ }
974+ c, w = parseString(t, "bind.cmd+g = hint\n")
975+ if len(w) != 0 || c.Binds["Cmd+g"] != "hint" {
976+ t.Fatal(c.Binds["Cmd+g"], w)
977+ }
978+}
M internal/adapters/drm/overlay.go +3 -0
@@ -70,6 +70,9 @@ func overlayCandidate(s ports.Scene, surfaces map[ports.WindowID]ports.SurfaceCo
7070 if s.Flash != nil {
7171 return ports.SceneWindow{}, ports.SurfaceContent{}, colorBypass, "focus_flash"
7272 }
73+ if len(s.HintLabels) > 0 {
74+ return ports.SceneWindow{}, ports.SurfaceContent{}, colorBypass, "hint_labels"
75+ }
7376 if s.Transform != 0 {
7477 return ports.SceneWindow{}, ports.SurfaceContent{}, colorBypass, "output_transform"
7578 }
@@ -70,6 +70,9 @@ func overlayCandidate(s ports.Scene, surfaces map[ports.WindowID]ports.SurfaceCo
70 if s.Flash != nil {70 if s.Flash != nil {
71 return ports.SceneWindow{}, ports.SurfaceContent{}, colorBypass, "focus_flash"71 return ports.SceneWindow{}, ports.SurfaceContent{}, colorBypass, "focus_flash"
72 }72 }
73+ if len(s.HintLabels) > 0 {
74+ return ports.SceneWindow{}, ports.SurfaceContent{}, colorBypass, "hint_labels"
75+ }
73 if s.Transform != 0 {76 if s.Transform != 0 {
74 return ports.SceneWindow{}, ports.SurfaceContent{}, colorBypass, "output_transform"77 return ports.SceneWindow{}, ports.SurfaceContent{}, colorBypass, "output_transform"
75 }78 }
M internal/adapters/drm/scanout.go +4 -0
@@ -94,6 +94,10 @@ func scanoutCandidate(s ports.Scene, surfaces map[ports.WindowID]ports.SurfaceCo
9494 if s.Flash != nil {
9595 return ports.SurfaceContent{}, "focus_flash"
9696 }
97+ // Hint labels are composed over the window.
98+ if len(s.HintLabels) > 0 {
99+ return ports.SurfaceContent{}, "hint_labels"
100+ }
97101 if s.Transform != 0 {
98102 // The scanout buffer would have to be pre-rotated: rotated outputs compose.
99103 return ports.SurfaceContent{}, "output_transform"
@@ -94,6 +94,10 @@ func scanoutCandidate(s ports.Scene, surfaces map[ports.WindowID]ports.SurfaceCo
94 if s.Flash != nil {94 if s.Flash != nil {
95 return ports.SurfaceContent{}, "focus_flash"95 return ports.SurfaceContent{}, "focus_flash"
96 }96 }
97+ // Hint labels are composed over the window.
98+ if len(s.HintLabels) > 0 {
99+ return ports.SurfaceContent{}, "hint_labels"
100+ }
97 if s.Transform != 0 {101 if s.Transform != 0 {
98 // The scanout buffer would have to be pre-rotated: rotated outputs compose.102 // The scanout buffer would have to be pre-rotated: rotated outputs compose.
99 return ports.SurfaceContent{}, "output_transform"103 return ports.SurfaceContent{}, "output_transform"
A internal/adapters/hintlabel/hintlabel.go +131 -0
@@ -0,0 +1,131 @@
1+// Package hintlabel rasterizes the keyboard hint labels on the CPU: a small
2+// box with the label text in Go Mono, as premultiplied BGRA pixels for the
3+// renderer to upload and draw.
4+package hintlabel
5+
6+import (
7+ "image"
8+ "image/color"
9+ "image/draw"
10+ "math"
11+ "sync"
12+
13+ "golang.org/x/image/font"
14+ "golang.org/x/image/font/gofont/gomono"
15+ "golang.org/x/image/font/opentype"
16+ "golang.org/x/image/math/fixed"
17+)
18+
19+const (
20+ // fontSize is the text size in logical pixels.
21+ fontSize = 13
22+ // padX and padY are the padding around the text in logical pixels.
23+ padX, padY = 3, 1
24+ // maxEntries bounds the cache: it is cleared when it grows past it.
25+ maxEntries = 1024
26+)
27+
28+var (
29+ background = color.RGBA{0xf9, 0xe2, 0xaf, 0xff}
30+ ink = color.RGBA{0x1e, 0x1e, 0x2e, 0xff}
31+ typedInk = color.RGBA{0x7f, 0x84, 0x9c, 0xff}
32+
33+ parseOnce sync.Once
34+ parsed *opentype.Font
35+ parseErr error
36+)
37+
38+type key struct {
39+ text string
40+ typed int
41+ scale float64
42+}
43+
44+type entry struct {
45+ pix []byte
46+ w, h int
47+}
48+
49+// Raster draws and caches labels. It is not safe for concurrent use: the
50+// renderer owns it.
51+type Raster struct {
52+ faces map[float64]font.Face
53+ cache map[key]entry
54+}
55+
56+// Label returns the label box for text, whose first typed characters are
57+// drawn dimmed, as premultiplied BGRA at scale physical pixels per logical
58+// pixel. The slice is owned by the cache and stays valid until the cache is
59+// cleared by a later call: do not modify it. Empty text, or a font that does
60+// not load, yields a nil slice.
61+func (r *Raster) Label(text string, typed int, scale float64) (pix []byte, w, h int) {
62+ if text == "" {
63+ return nil, 0, 0
64+ }
65+ if scale <= 0 || math.IsNaN(scale) || math.IsInf(scale, 0) {
66+ scale = 1
67+ }
68+ typed = max(0, min(typed, len(text)))
69+ k := key{text, typed, scale}
70+ if e, ok := r.cache[k]; ok {
71+ return e.pix, e.w, e.h
72+ }
73+ face := r.face(scale)
74+ if face == nil {
75+ return nil, 0, 0
76+ }
77+ if r.cache == nil || len(r.cache) >= maxEntries {
78+ r.cache = map[key]entry{}
79+ }
80+ e := render(face, text, typed, scale)
81+ r.cache[k] = e
82+ return e.pix, e.w, e.h
83+}
84+
85+func (r *Raster) face(scale float64) font.Face {
86+ if f, ok := r.faces[scale]; ok {
87+ return f
88+ }
89+ parseOnce.Do(func() { parsed, parseErr = opentype.Parse(gomono.TTF) })
90+ if parseErr != nil {
91+ return nil
92+ }
93+ f, err := opentype.NewFace(parsed, &opentype.FaceOptions{Size: fontSize * scale, DPI: 72, Hinting: font.HintingFull})
94+ if err != nil {
95+ return nil
96+ }
97+ if r.faces == nil || len(r.faces) >= 16 {
98+ for _, old := range r.faces {
99+ _ = old.Close()
100+ }
101+ r.faces = map[float64]font.Face{}
102+ }
103+ r.faces[scale] = f
104+ return f
105+}
106+
107+// render draws the box. Colours are opaque, so the result is premultiplied
108+// by construction.
109+func render(face font.Face, text string, typed int, scale float64) entry {
110+ m := face.Metrics()
111+ px := int(math.Round(padX * scale))
112+ py := int(math.Round(padY * scale))
113+ border := 1
114+ adv := font.MeasureString(face, text).Ceil()
115+ w := adv + 2*px + 2*border
116+ h := m.Height.Ceil() + 2*py + 2*border
117+ img := image.NewRGBA(image.Rect(0, 0, w, h))
118+ draw.Draw(img, img.Bounds(), image.NewUniform(ink), image.Point{}, draw.Src)
119+ draw.Draw(img, img.Bounds().Inset(border), image.NewUniform(background), image.Point{}, draw.Src)
120+ d := font.Drawer{Dst: img, Face: face, Dot: fixed.Point26_6{X: fixed.I(border + px), Y: fixed.I(border+py) + m.Ascent}}
121+ d.Src = image.NewUniform(typedInk)
122+ d.DrawString(text[:typed])
123+ d.Src = image.NewUniform(ink)
124+ d.DrawString(text[typed:])
125+ // RGBA to BGRA in place; alpha is 0xff everywhere.
126+ pix := img.Pix
127+ for i := 0; i+3 < len(pix); i += 4 {
128+ pix[i], pix[i+2] = pix[i+2], pix[i]
129+ }
130+ return entry{pix: pix, w: w, h: h}
131+}
@@ -0,0 +1,131 @@
1+// Package hintlabel rasterizes the keyboard hint labels on the CPU: a small
2+// box with the label text in Go Mono, as premultiplied BGRA pixels for the
3+// renderer to upload and draw.
4+package hintlabel
5+
6+import (
7+ "image"
8+ "image/color"
9+ "image/draw"
10+ "math"
11+ "sync"
12+
13+ "golang.org/x/image/font"
14+ "golang.org/x/image/font/gofont/gomono"
15+ "golang.org/x/image/font/opentype"
16+ "golang.org/x/image/math/fixed"
17+)
18+
19+const (
20+ // fontSize is the text size in logical pixels.
21+ fontSize = 13
22+ // padX and padY are the padding around the text in logical pixels.
23+ padX, padY = 3, 1
24+ // maxEntries bounds the cache: it is cleared when it grows past it.
25+ maxEntries = 1024
26+)
27+
28+var (
29+ background = color.RGBA{0xf9, 0xe2, 0xaf, 0xff}
30+ ink = color.RGBA{0x1e, 0x1e, 0x2e, 0xff}
31+ typedInk = color.RGBA{0x7f, 0x84, 0x9c, 0xff}
32+
33+ parseOnce sync.Once
34+ parsed *opentype.Font
35+ parseErr error
36+)
37+
38+type key struct {
39+ text string
40+ typed int
41+ scale float64
42+}
43+
44+type entry struct {
45+ pix []byte
46+ w, h int
47+}
48+
49+// Raster draws and caches labels. It is not safe for concurrent use: the
50+// renderer owns it.
51+type Raster struct {
52+ faces map[float64]font.Face
53+ cache map[key]entry
54+}
55+
56+// Label returns the label box for text, whose first typed characters are
57+// drawn dimmed, as premultiplied BGRA at scale physical pixels per logical
58+// pixel. The slice is owned by the cache and stays valid until the cache is
59+// cleared by a later call: do not modify it. Empty text, or a font that does
60+// not load, yields a nil slice.
61+func (r *Raster) Label(text string, typed int, scale float64) (pix []byte, w, h int) {
62+ if text == "" {
63+ return nil, 0, 0
64+ }
65+ if scale <= 0 || math.IsNaN(scale) || math.IsInf(scale, 0) {
66+ scale = 1
67+ }
68+ typed = max(0, min(typed, len(text)))
69+ k := key{text, typed, scale}
70+ if e, ok := r.cache[k]; ok {
71+ return e.pix, e.w, e.h
72+ }
73+ face := r.face(scale)
74+ if face == nil {
75+ return nil, 0, 0
76+ }
77+ if r.cache == nil || len(r.cache) >= maxEntries {
78+ r.cache = map[key]entry{}
79+ }
80+ e := render(face, text, typed, scale)
81+ r.cache[k] = e
82+ return e.pix, e.w, e.h
83+}
84+
85+func (r *Raster) face(scale float64) font.Face {
86+ if f, ok := r.faces[scale]; ok {
87+ return f
88+ }
89+ parseOnce.Do(func() { parsed, parseErr = opentype.Parse(gomono.TTF) })
90+ if parseErr != nil {
91+ return nil
92+ }
93+ f, err := opentype.NewFace(parsed, &opentype.FaceOptions{Size: fontSize * scale, DPI: 72, Hinting: font.HintingFull})
94+ if err != nil {
95+ return nil
96+ }
97+ if r.faces == nil || len(r.faces) >= 16 {
98+ for _, old := range r.faces {
99+ _ = old.Close()
100+ }
101+ r.faces = map[float64]font.Face{}
102+ }
103+ r.faces[scale] = f
104+ return f
105+}
106+
107+// render draws the box. Colours are opaque, so the result is premultiplied
108+// by construction.
109+func render(face font.Face, text string, typed int, scale float64) entry {
110+ m := face.Metrics()
111+ px := int(math.Round(padX * scale))
112+ py := int(math.Round(padY * scale))
113+ border := 1
114+ adv := font.MeasureString(face, text).Ceil()
115+ w := adv + 2*px + 2*border
116+ h := m.Height.Ceil() + 2*py + 2*border
117+ img := image.NewRGBA(image.Rect(0, 0, w, h))
118+ draw.Draw(img, img.Bounds(), image.NewUniform(ink), image.Point{}, draw.Src)
119+ draw.Draw(img, img.Bounds().Inset(border), image.NewUniform(background), image.Point{}, draw.Src)
120+ d := font.Drawer{Dst: img, Face: face, Dot: fixed.Point26_6{X: fixed.I(border + px), Y: fixed.I(border+py) + m.Ascent}}
121+ d.Src = image.NewUniform(typedInk)
122+ d.DrawString(text[:typed])
123+ d.Src = image.NewUniform(ink)
124+ d.DrawString(text[typed:])
125+ // RGBA to BGRA in place; alpha is 0xff everywhere.
126+ pix := img.Pix
127+ for i := 0; i+3 < len(pix); i += 4 {
128+ pix[i], pix[i+2] = pix[i+2], pix[i]
129+ }
130+ return entry{pix: pix, w: w, h: h}
131+}
A internal/adapters/hintlabel/hintlabel_test.go +60 -0
@@ -0,0 +1,60 @@
1+package hintlabel
2+
3+import (
4+ "testing"
5+
6+ "github.com/stretchr/testify/require"
7+)
8+
9+func TestHintLabelSize(t *testing.T) {
10+ var r Raster
11+ _, w1, h1 := r.Label("f", 0, 1)
12+ _, w2, h2 := r.Label("fj", 0, 1)
13+ require.Greater(t, w1, 0)
14+ require.Greater(t, w2, w1, "a longer label is wider")
15+ require.Equal(t, h1, h2)
16+ _, w3, h3 := r.Label("fj", 0, 2)
17+ // The 1 px border does not scale: compare the boxes without it.
18+ require.InDelta(t, 2*(w2-2), w3-2, 1)
19+ require.InDelta(t, 2*(h2-2), h3-2, 1)
20+}
21+
22+func TestHintLabelCache(t *testing.T) {
23+ var r Raster
24+ a, w, h := r.Label("fj", 1, 1)
25+ b, _, _ := r.Label("fj", 1, 1)
26+ require.Len(t, a, w*h*4)
27+ require.Same(t, &a[0], &b[0], "same key returns the cached slice")
28+ c, _, _ := r.Label("fj", 0, 1)
29+ require.NotSame(t, &a[0], &c[0], "typed count is part of the key")
30+}
31+
32+func TestHintLabelPremultiplied(t *testing.T) {
33+ var r Raster
34+ pix, w, h := r.Label("ab", 1, 1.5)
35+ require.Len(t, pix, w*h*4)
36+ for i := 0; i < len(pix); i += 4 {
37+ a := pix[i+3]
38+ require.GreaterOrEqual(t, a, pix[i])
39+ require.GreaterOrEqual(t, a, pix[i+1])
40+ require.GreaterOrEqual(t, a, pix[i+2])
41+ }
42+}
43+
44+func TestHintLabelTypedIsDimmed(t *testing.T) {
45+ var r Raster
46+ plain, _, _ := r.Label("fj", 0, 1)
47+ dim, _, _ := r.Label("fj", 1, 1)
48+ require.NotEqual(t, plain, dim)
49+}
50+
51+func TestHintLabelEmptyAndCacheBound(t *testing.T) {
52+ var r Raster
53+ pix, w, h := r.Label("", 0, 1)
54+ require.Nil(t, pix)
55+ require.Zero(t, w+h)
56+ for i := 0; i < maxEntries+10; i++ {
57+ r.Label("a", 0, 1+float64(i)/100000)
58+ }
59+ require.LessOrEqual(t, len(r.cache), maxEntries)
60+}
@@ -0,0 +1,60 @@
1+package hintlabel
2+
3+import (
4+ "testing"
5+
6+ "github.com/stretchr/testify/require"
7+)
8+
9+func TestHintLabelSize(t *testing.T) {
10+ var r Raster
11+ _, w1, h1 := r.Label("f", 0, 1)
12+ _, w2, h2 := r.Label("fj", 0, 1)
13+ require.Greater(t, w1, 0)
14+ require.Greater(t, w2, w1, "a longer label is wider")
15+ require.Equal(t, h1, h2)
16+ _, w3, h3 := r.Label("fj", 0, 2)
17+ // The 1 px border does not scale: compare the boxes without it.
18+ require.InDelta(t, 2*(w2-2), w3-2, 1)
19+ require.InDelta(t, 2*(h2-2), h3-2, 1)
20+}
21+
22+func TestHintLabelCache(t *testing.T) {
23+ var r Raster
24+ a, w, h := r.Label("fj", 1, 1)
25+ b, _, _ := r.Label("fj", 1, 1)
26+ require.Len(t, a, w*h*4)
27+ require.Same(t, &a[0], &b[0], "same key returns the cached slice")
28+ c, _, _ := r.Label("fj", 0, 1)
29+ require.NotSame(t, &a[0], &c[0], "typed count is part of the key")
30+}
31+
32+func TestHintLabelPremultiplied(t *testing.T) {
33+ var r Raster
34+ pix, w, h := r.Label("ab", 1, 1.5)
35+ require.Len(t, pix, w*h*4)
36+ for i := 0; i < len(pix); i += 4 {
37+ a := pix[i+3]
38+ require.GreaterOrEqual(t, a, pix[i])
39+ require.GreaterOrEqual(t, a, pix[i+1])
40+ require.GreaterOrEqual(t, a, pix[i+2])
41+ }
42+}
43+
44+func TestHintLabelTypedIsDimmed(t *testing.T) {
45+ var r Raster
46+ plain, _, _ := r.Label("fj", 0, 1)
47+ dim, _, _ := r.Label("fj", 1, 1)
48+ require.NotEqual(t, plain, dim)
49+}
50+
51+func TestHintLabelEmptyAndCacheBound(t *testing.T) {
52+ var r Raster
53+ pix, w, h := r.Label("", 0, 1)
54+ require.Nil(t, pix)
55+ require.Zero(t, w+h)
56+ for i := 0; i < maxEntries+10; i++ {
57+ r.Label("a", 0, 1+float64(i)/100000)
58+ }
59+ require.LessOrEqual(t, len(r.cache), maxEntries)
60+}
A internal/adapters/hints/dump.go +214 -0
@@ -0,0 +1,214 @@
1+package hints
2+
3+import (
4+ "encoding/json"
5+ "fmt"
6+ "image"
7+ "image/color"
8+ "image/png"
9+ "os"
10+ "path/filepath"
11+ "sort"
12+ "strings"
13+ "time"
14+
15+ "github.com/bnema/neferwl/internal/ports"
16+)
17+
18+const (
19+ // maxDumps is the number of dump groups kept in the directory.
20+ maxDumps = 50
21+ outlineSize = 2
22+)
23+
24+// Options tunes Run.
25+type Options struct {
26+ // DumpDir, when not empty, receives the scanned frames (PNG, annotated
27+ // PNG and JSON). They contain the screen in clear.
28+ DumpDir string
29+}
30+
31+// dumpJob is one scan to write. Pix is the worker buffer, not a copy: it
32+// stays untouched until the worker serves the next request.
33+type dumpJob struct {
34+ when time.Time
35+ req ports.HintScanRequest
36+ pix []byte
37+ stride int
38+ targets []ports.HintTarget
39+ captureD time.Duration
40+ scanD time.Duration
41+}
42+
43+type dumpTarget struct {
44+ Kind string `json:"kind"`
45+ Rect [4]int `json:"rect"`
46+}
47+
48+type dumpMeta struct {
49+ Output string `json:"output"`
50+ Width int `json:"width"`
51+ Height int `json:"height"`
52+ Stride int `json:"stride"`
53+ Scale float64 `json:"scale"`
54+ LinksOnly bool `json:"links_only"`
55+ CaptureMS float64 `json:"capture_ms"`
56+ ScanMS float64 `json:"scan_ms"`
57+ Targets []dumpTarget `json:"targets"`
58+}
59+
60+func kindName(k ports.HintKind) string {
61+ switch k {
62+ case ports.HintLink:
63+ return "link"
64+ case ports.HintButton:
65+ return "button"
66+ default:
67+ return "text"
68+ }
69+}
70+
71+func kindColor(k ports.HintKind) color.RGBA {
72+ switch k {
73+ case ports.HintLink:
74+ return color.RGBA{0x00, 0xc8, 0x53, 0xff}
75+ case ports.HintButton:
76+ return color.RGBA{0x29, 0x62, 0xff, 0xff}
77+ default:
78+ return color.RGBA{0x80, 0x80, 0x80, 0xff}
79+ }
80+}
81+
82+func ms(d time.Duration) float64 { return float64(d.Microseconds()) / 1000 }
83+
84+// writeDump writes the three files of j into dir, then prunes old groups.
85+func writeDump(dir string, j *dumpJob) error {
86+ if err := os.MkdirAll(dir, 0o700); err != nil {
87+ return err
88+ }
89+ prefix := filepath.Join(dir, j.when.Format("20060102-150405.000")+"-"+safeName(j.req.Output))
90+
91+ img := &image.RGBA{Pix: make([]byte, j.req.Height*j.stride), Stride: j.stride, Rect: image.Rect(0, 0, j.req.Width, j.req.Height)}
92+ for y := 0; y < j.req.Height; y++ {
93+ row := j.pix[y*j.stride : y*j.stride+j.req.Width*4]
94+ out := img.Pix[y*j.stride:]
95+ for x := 0; x < len(row); x += 4 {
96+ out[x], out[x+1], out[x+2], out[x+3] = row[x+2], row[x+1], row[x], 0xff
97+ }
98+ }
99+ if err := writePNG(prefix+".png", img); err != nil {
100+ return err
101+ }
102+
103+ scale := j.req.Scale
104+ if scale <= 0 {
105+ scale = 1
106+ }
107+ for _, t := range j.targets {
108+ r := image.Rect(int(float64(t.Rect.X)*scale), int(float64(t.Rect.Y)*scale),
109+ int(float64(t.Rect.X+t.Rect.W)*scale), int(float64(t.Rect.Y+t.Rect.H)*scale))
110+ outline(img, r, kindColor(t.Kind))
111+ }
112+ if err := writePNG(prefix+".annotated.png", img); err != nil {
113+ return err
114+ }
115+
116+ meta := dumpMeta{
117+ Output: j.req.Output, Width: j.req.Width, Height: j.req.Height, Stride: j.stride,
118+ Scale: j.req.Scale, LinksOnly: j.req.LinksOnly,
119+ CaptureMS: ms(j.captureD), ScanMS: ms(j.scanD),
120+ Targets: make([]dumpTarget, 0, len(j.targets)),
121+ }
122+ for _, t := range j.targets {
123+ meta.Targets = append(meta.Targets, dumpTarget{Kind: kindName(t.Kind), Rect: [4]int{t.Rect.X, t.Rect.Y, t.Rect.X + t.Rect.W, t.Rect.Y + t.Rect.H}})
124+ }
125+ data, err := json.Marshal(meta)
126+ if err != nil {
127+ return err
128+ }
129+ if err := writeFile(prefix+".json", func(f *os.File) error { _, err := f.Write(append(data, '\n')); return err }); err != nil {
130+ return err
131+ }
132+ return prune(dir, maxDumps)
133+}
134+
135+func safeName(s string) string {
136+ return strings.Map(func(r rune) rune {
137+ if r == '/' || r == 0 {
138+ return '_'
139+ }
140+ return r
141+ }, s)
142+}
143+
144+func writePNG(path string, img image.Image) error {
145+ enc := png.Encoder{CompressionLevel: png.BestSpeed}
146+ return writeFile(path, func(f *os.File) error { return enc.Encode(f, img) })
147+}
148+
149+func writeFile(path string, fill func(*os.File) error) error {
150+ f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
151+ if err != nil {
152+ return err
153+ }
154+ if err := fill(f); err != nil {
155+ _ = f.Close()
156+ return err
157+ }
158+ return f.Close()
159+}
160+
161+// outline draws a 2 px frame inside r, clipped to img.
162+func outline(img *image.RGBA, r image.Rectangle, c color.RGBA) {
163+ r = r.Intersect(img.Rect)
164+ if r.Empty() {
165+ return
166+ }
167+ for y := r.Min.Y; y < r.Max.Y; y++ {
168+ for x := r.Min.X; x < r.Max.X; x++ {
169+ if x-r.Min.X < outlineSize || r.Max.X-1-x < outlineSize || y-r.Min.Y < outlineSize || r.Max.Y-1-y < outlineSize {
170+ img.SetRGBA(x, y, c)
171+ }
172+ }
173+ }
174+}
175+
176+// groupOf returns the "<timestamp>-<output>" prefix of a dump file name.
177+func groupOf(name string) string {
178+ for _, suf := range []string{".annotated.png", ".png", ".json"} {
179+ if strings.HasSuffix(name, suf) {
180+ return strings.TrimSuffix(name, suf)
181+ }
182+ }
183+ return ""
184+}
185+
186+// prune keeps the keep most recent groups of dir; timestamps sort by name.
187+func prune(dir string, keep int) error {
188+ entries, err := os.ReadDir(dir)
189+ if err != nil {
190+ return err
191+ }
192+ groups := map[string][]string{}
193+ for _, e := range entries {
194+ if g := groupOf(e.Name()); g != "" && !e.IsDir() {
195+ groups[g] = append(groups[g], e.Name())
196+ }
197+ }
198+ if len(groups) <= keep {
199+ return nil
200+ }
201+ names := make([]string, 0, len(groups))
202+ for g := range groups {
203+ names = append(names, g)
204+ }
205+ sort.Strings(names)
206+ for _, g := range names[:len(names)-keep] {
207+ for _, f := range groups[g] {
208+ if err := os.Remove(filepath.Join(dir, f)); err != nil {
209+ return fmt.Errorf("prune: %w", err)
210+ }
211+ }
212+ }
213+ return nil
214+}
@@ -0,0 +1,214 @@
1+package hints
2+
3+import (
4+ "encoding/json"
5+ "fmt"
6+ "image"
7+ "image/color"
8+ "image/png"
9+ "os"
10+ "path/filepath"
11+ "sort"
12+ "strings"
13+ "time"
14+
15+ "github.com/bnema/neferwl/internal/ports"
16+)
17+
18+const (
19+ // maxDumps is the number of dump groups kept in the directory.
20+ maxDumps = 50
21+ outlineSize = 2
22+)
23+
24+// Options tunes Run.
25+type Options struct {
26+ // DumpDir, when not empty, receives the scanned frames (PNG, annotated
27+ // PNG and JSON). They contain the screen in clear.
28+ DumpDir string
29+}
30+
31+// dumpJob is one scan to write. Pix is the worker buffer, not a copy: it
32+// stays untouched until the worker serves the next request.
33+type dumpJob struct {
34+ when time.Time
35+ req ports.HintScanRequest
36+ pix []byte
37+ stride int
38+ targets []ports.HintTarget
39+ captureD time.Duration
40+ scanD time.Duration
41+}
42+
43+type dumpTarget struct {
44+ Kind string `json:"kind"`
45+ Rect [4]int `json:"rect"`
46+}
47+
48+type dumpMeta struct {
49+ Output string `json:"output"`
50+ Width int `json:"width"`
51+ Height int `json:"height"`
52+ Stride int `json:"stride"`
53+ Scale float64 `json:"scale"`
54+ LinksOnly bool `json:"links_only"`
55+ CaptureMS float64 `json:"capture_ms"`
56+ ScanMS float64 `json:"scan_ms"`
57+ Targets []dumpTarget `json:"targets"`
58+}
59+
60+func kindName(k ports.HintKind) string {
61+ switch k {
62+ case ports.HintLink:
63+ return "link"
64+ case ports.HintButton:
65+ return "button"
66+ default:
67+ return "text"
68+ }
69+}
70+
71+func kindColor(k ports.HintKind) color.RGBA {
72+ switch k {
73+ case ports.HintLink:
74+ return color.RGBA{0x00, 0xc8, 0x53, 0xff}
75+ case ports.HintButton:
76+ return color.RGBA{0x29, 0x62, 0xff, 0xff}
77+ default:
78+ return color.RGBA{0x80, 0x80, 0x80, 0xff}
79+ }
80+}
81+
82+func ms(d time.Duration) float64 { return float64(d.Microseconds()) / 1000 }
83+
84+// writeDump writes the three files of j into dir, then prunes old groups.
85+func writeDump(dir string, j *dumpJob) error {
86+ if err := os.MkdirAll(dir, 0o700); err != nil {
87+ return err
88+ }
89+ prefix := filepath.Join(dir, j.when.Format("20060102-150405.000")+"-"+safeName(j.req.Output))
90+
91+ img := &image.RGBA{Pix: make([]byte, j.req.Height*j.stride), Stride: j.stride, Rect: image.Rect(0, 0, j.req.Width, j.req.Height)}
92+ for y := 0; y < j.req.Height; y++ {
93+ row := j.pix[y*j.stride : y*j.stride+j.req.Width*4]
94+ out := img.Pix[y*j.stride:]
95+ for x := 0; x < len(row); x += 4 {
96+ out[x], out[x+1], out[x+2], out[x+3] = row[x+2], row[x+1], row[x], 0xff
97+ }
98+ }
99+ if err := writePNG(prefix+".png", img); err != nil {
100+ return err
101+ }
102+
103+ scale := j.req.Scale
104+ if scale <= 0 {
105+ scale = 1
106+ }
107+ for _, t := range j.targets {
108+ r := image.Rect(int(float64(t.Rect.X)*scale), int(float64(t.Rect.Y)*scale),
109+ int(float64(t.Rect.X+t.Rect.W)*scale), int(float64(t.Rect.Y+t.Rect.H)*scale))
110+ outline(img, r, kindColor(t.Kind))
111+ }
112+ if err := writePNG(prefix+".annotated.png", img); err != nil {
113+ return err
114+ }
115+
116+ meta := dumpMeta{
117+ Output: j.req.Output, Width: j.req.Width, Height: j.req.Height, Stride: j.stride,
118+ Scale: j.req.Scale, LinksOnly: j.req.LinksOnly,
119+ CaptureMS: ms(j.captureD), ScanMS: ms(j.scanD),
120+ Targets: make([]dumpTarget, 0, len(j.targets)),
121+ }
122+ for _, t := range j.targets {
123+ meta.Targets = append(meta.Targets, dumpTarget{Kind: kindName(t.Kind), Rect: [4]int{t.Rect.X, t.Rect.Y, t.Rect.X + t.Rect.W, t.Rect.Y + t.Rect.H}})
124+ }
125+ data, err := json.Marshal(meta)
126+ if err != nil {
127+ return err
128+ }
129+ if err := writeFile(prefix+".json", func(f *os.File) error { _, err := f.Write(append(data, '\n')); return err }); err != nil {
130+ return err
131+ }
132+ return prune(dir, maxDumps)
133+}
134+
135+func safeName(s string) string {
136+ return strings.Map(func(r rune) rune {
137+ if r == '/' || r == 0 {
138+ return '_'
139+ }
140+ return r
141+ }, s)
142+}
143+
144+func writePNG(path string, img image.Image) error {
145+ enc := png.Encoder{CompressionLevel: png.BestSpeed}
146+ return writeFile(path, func(f *os.File) error { return enc.Encode(f, img) })
147+}
148+
149+func writeFile(path string, fill func(*os.File) error) error {
150+ f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
151+ if err != nil {
152+ return err
153+ }
154+ if err := fill(f); err != nil {
155+ _ = f.Close()
156+ return err
157+ }
158+ return f.Close()
159+}
160+
161+// outline draws a 2 px frame inside r, clipped to img.
162+func outline(img *image.RGBA, r image.Rectangle, c color.RGBA) {
163+ r = r.Intersect(img.Rect)
164+ if r.Empty() {
165+ return
166+ }
167+ for y := r.Min.Y; y < r.Max.Y; y++ {
168+ for x := r.Min.X; x < r.Max.X; x++ {
169+ if x-r.Min.X < outlineSize || r.Max.X-1-x < outlineSize || y-r.Min.Y < outlineSize || r.Max.Y-1-y < outlineSize {
170+ img.SetRGBA(x, y, c)
171+ }
172+ }
173+ }
174+}
175+
176+// groupOf returns the "<timestamp>-<output>" prefix of a dump file name.
177+func groupOf(name string) string {
178+ for _, suf := range []string{".annotated.png", ".png", ".json"} {
179+ if strings.HasSuffix(name, suf) {
180+ return strings.TrimSuffix(name, suf)
181+ }
182+ }
183+ return ""
184+}
185+
186+// prune keeps the keep most recent groups of dir; timestamps sort by name.
187+func prune(dir string, keep int) error {
188+ entries, err := os.ReadDir(dir)
189+ if err != nil {
190+ return err
191+ }
192+ groups := map[string][]string{}
193+ for _, e := range entries {
194+ if g := groupOf(e.Name()); g != "" && !e.IsDir() {
195+ groups[g] = append(groups[g], e.Name())
196+ }
197+ }
198+ if len(groups) <= keep {
199+ return nil
200+ }
201+ names := make([]string, 0, len(groups))
202+ for g := range groups {
203+ names = append(names, g)
204+ }
205+ sort.Strings(names)
206+ for _, g := range names[:len(names)-keep] {
207+ for _, f := range groups[g] {
208+ if err := os.Remove(filepath.Join(dir, f)); err != nil {
209+ return fmt.Errorf("prune: %w", err)
210+ }
211+ }
212+ }
213+ return nil
214+}
A internal/adapters/hints/hints.go +200 -0
@@ -0,0 +1,200 @@
1+// Package hints finds clickable targets for hint mode: it captures the last
2+// frame of an output through the capture pipeline and scans it with hintscan.
3+package hints
4+
5+import (
6+ "context"
7+ "errors"
8+ "fmt"
9+ "image"
10+ "time"
11+
12+ "bnema.dev/hintscan"
13+
14+ "github.com/bnema/neferwl/internal/adapters/logging"
15+ "github.com/bnema/neferwl/internal/ports"
16+)
17+
18+// ErrBusy answers a request that arrives while another one is in flight.
19+var ErrBusy = errors.New("hints: scan in progress")
20+
21+// ErrTimeout is the error of a capture that is not answered in time.
22+var ErrTimeout = errors.New("hints: capture timed out")
23+
24+// ErrBadRequest is the error of a request without a usable buffer size.
25+var ErrBadRequest = errors.New("hints: invalid output size")
26+
27+const (
28+ captureTimeout = time.Second
29+ // formatXRGB8888 is the capture format: opaque BGRA in memory.
30+ formatXRGB8888 = 1
31+ maxPixels = 1 << 28
32+)
33+
34+// Run serves requests until ctx is done. It handles one request at a time;
35+// a request that arrives meanwhile is answered with ErrBusy. Every request
36+// gets exactly one result on results.
37+func Run(ctx context.Context, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, opts Options) error {
38+ return run(ctx, requests, captures, captured, results, captureTimeout, opts)
39+}
40+
41+type worker struct {
42+ scanner *hintscan.Scanner
43+ buf []byte
44+ dst []hintscan.Target
45+ n uint64
46+ dumpDir string
47+ pending *dumpJob
48+}
49+
50+func run(ctx context.Context, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, timeout time.Duration, opts Options) error {
51+ w := &worker{scanner: hintscan.NewScanner(), dumpDir: opts.DumpDir}
52+ for {
53+ select {
54+ case <-ctx.Done():
55+ return nil
56+ case req := <-requests:
57+ res := w.serve(ctx, req, requests, captures, captured, results, timeout)
58+ if !send(ctx, results, res) {
59+ return nil
60+ }
61+ w.dump(ctx)
62+ }
63+ }
64+}
65+
66+// dump writes the last successful scan, after its result was sent.
67+func (w *worker) dump(ctx context.Context) {
68+ j := w.pending
69+ w.pending = nil
70+ if j == nil {
71+ return
72+ }
73+ if err := writeDump(w.dumpDir, j); err != nil {
74+ log := logging.For(ctx, "hints")
75+ log.Warn().Err(err).Str("dir", w.dumpDir).Msg("hint debug dump failed")
76+ }
77+}
78+
79+func send(ctx context.Context, results chan<- ports.HintScanResult, res ports.HintScanResult) bool {
80+ select {
81+ case results <- res:
82+ return true
83+ case <-ctx.Done():
84+ return false
85+ }
86+}
87+
88+func (w *worker) serve(ctx context.Context, req ports.HintScanRequest, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, timeout time.Duration) ports.HintScanResult {
89+ log := logging.For(ctx, "hints")
90+ res := ports.HintScanResult{Seq: req.Seq, Output: req.Output}
91+ fail := func(err error) ports.HintScanResult {
92+ log.Info().Err(err).Str("output", req.Output).Msg("hint scan failed")
93+ res.Err = err
94+ return res
95+ }
96+
97+ if req.Width <= 0 || req.Height <= 0 || req.Width*req.Height > maxPixels {
98+ return fail(fmt.Errorf("%w: %dx%d", ErrBadRequest, req.Width, req.Height))
99+ }
100+ size := req.Width * req.Height * 4
101+ if cap(w.buf) < size {
102+ w.buf = make([]byte, size)
103+ }
104+ buf := w.buf[:size]
105+
106+ w.n++
107+ id := ports.CaptureIDInternal | w.n
108+ stride := req.Width * 4
109+ start := time.Now()
110+ creq := ports.CaptureRequest{
111+ ID: id,
112+ Output: req.Output,
113+ Region: image.Rect(0, 0, req.Width, req.Height),
114+ Mapped: buf,
115+ Width: req.Width,
116+ Height: req.Height,
117+ Stride: stride,
118+ Format: formatXRGB8888,
119+ Indicate: false,
120+ }
121+
122+ timer := time.NewTimer(timeout)
123+ defer timer.Stop()
124+
125+ select {
126+ case captures <- creq:
127+ case <-timer.C:
128+ return fail(ErrTimeout)
129+ case <-ctx.Done():
130+ return fail(ctx.Err())
131+ }
132+
133+ for {
134+ select {
135+ case done := <-captured:
136+ if done.ID != id {
137+ continue
138+ }
139+ if done.Err != nil {
140+ return fail(fmt.Errorf("hints: capture: %w", done.Err))
141+ }
142+ return w.scan(ctx, req, buf, stride, time.Since(start), res)
143+ case other := <-requests:
144+ // The buffer is in use until the capture is done: answer now.
145+ busy := ports.HintScanResult{Seq: other.Seq, Output: other.Output, Err: ErrBusy}
146+ if !send(ctx, results, busy) {
147+ return fail(ctx.Err())
148+ }
149+ case <-timer.C:
150+ // The output may still write into buf: do not reuse it.
151+ w.buf = nil
152+ return fail(ErrTimeout)
153+ case <-ctx.Done():
154+ w.buf = nil
155+ return fail(ctx.Err())
156+ }
157+ }
158+}
159+
160+func (w *worker) scan(ctx context.Context, req ports.HintScanRequest, buf []byte, stride int, captureDur time.Duration, res ports.HintScanResult) ports.HintScanResult {
161+ log := logging.For(ctx, "hints")
162+ kinds := hintscan.Clickable
163+ if req.LinksOnly {
164+ kinds = hintscan.KindSet(1 << hintscan.KindLink)
165+ }
166+ frame := hintscan.Frame{Pix: buf, Stride: stride, Width: req.Width, Height: req.Height, Order: hintscan.OrderBGRA, Scale: req.Scale}
167+
168+ t0 := time.Now()
169+ targets, err := w.scanner.Scan(ctx, frame, hintscan.Options{Kinds: kinds}, w.dst)
170+ scanDur := time.Since(t0)
171+ if err != nil {
172+ log.Info().Err(err).Str("output", req.Output).Msg("hint scan failed")
173+ res.Err = err
174+ return res
175+ }
176+ w.dst = targets
177+
178+ out := make([]ports.HintTarget, 0, len(targets))
179+ for _, t := range targets {
180+ kind := ports.HintLink
181+ if t.Kind == hintscan.KindButton {
182+ kind = ports.HintButton
183+ }
184+ out = append(out, ports.HintTarget{
185+ Rect: ports.Rect{X: t.Rect.Min.X, Y: t.Rect.Min.Y, W: t.Rect.Dx(), H: t.Rect.Dy()},
186+ Kind: kind,
187+ })
188+ }
189+ res.Targets = out
190+ if w.dumpDir != "" {
191+ w.pending = &dumpJob{when: time.Now(), req: req, pix: buf, stride: stride, targets: out, captureD: captureDur, scanD: scanDur}
192+ }
193+ log.Info().
194+ Str("output", req.Output).
195+ Int("targets", len(out)).
196+ Dur("scan", scanDur).
197+ Dur("capture", captureDur).
198+ Msg("hint scan")
199+ return res
200+}
@@ -0,0 +1,200 @@
1+// Package hints finds clickable targets for hint mode: it captures the last
2+// frame of an output through the capture pipeline and scans it with hintscan.
3+package hints
4+
5+import (
6+ "context"
7+ "errors"
8+ "fmt"
9+ "image"
10+ "time"
11+
12+ "bnema.dev/hintscan"
13+
14+ "github.com/bnema/neferwl/internal/adapters/logging"
15+ "github.com/bnema/neferwl/internal/ports"
16+)
17+
18+// ErrBusy answers a request that arrives while another one is in flight.
19+var ErrBusy = errors.New("hints: scan in progress")
20+
21+// ErrTimeout is the error of a capture that is not answered in time.
22+var ErrTimeout = errors.New("hints: capture timed out")
23+
24+// ErrBadRequest is the error of a request without a usable buffer size.
25+var ErrBadRequest = errors.New("hints: invalid output size")
26+
27+const (
28+ captureTimeout = time.Second
29+ // formatXRGB8888 is the capture format: opaque BGRA in memory.
30+ formatXRGB8888 = 1
31+ maxPixels = 1 << 28
32+)
33+
34+// Run serves requests until ctx is done. It handles one request at a time;
35+// a request that arrives meanwhile is answered with ErrBusy. Every request
36+// gets exactly one result on results.
37+func Run(ctx context.Context, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, opts Options) error {
38+ return run(ctx, requests, captures, captured, results, captureTimeout, opts)
39+}
40+
41+type worker struct {
42+ scanner *hintscan.Scanner
43+ buf []byte
44+ dst []hintscan.Target
45+ n uint64
46+ dumpDir string
47+ pending *dumpJob
48+}
49+
50+func run(ctx context.Context, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, timeout time.Duration, opts Options) error {
51+ w := &worker{scanner: hintscan.NewScanner(), dumpDir: opts.DumpDir}
52+ for {
53+ select {
54+ case <-ctx.Done():
55+ return nil
56+ case req := <-requests:
57+ res := w.serve(ctx, req, requests, captures, captured, results, timeout)
58+ if !send(ctx, results, res) {
59+ return nil
60+ }
61+ w.dump(ctx)
62+ }
63+ }
64+}
65+
66+// dump writes the last successful scan, after its result was sent.
67+func (w *worker) dump(ctx context.Context) {
68+ j := w.pending
69+ w.pending = nil
70+ if j == nil {
71+ return
72+ }
73+ if err := writeDump(w.dumpDir, j); err != nil {
74+ log := logging.For(ctx, "hints")
75+ log.Warn().Err(err).Str("dir", w.dumpDir).Msg("hint debug dump failed")
76+ }
77+}
78+
79+func send(ctx context.Context, results chan<- ports.HintScanResult, res ports.HintScanResult) bool {
80+ select {
81+ case results <- res:
82+ return true
83+ case <-ctx.Done():
84+ return false
85+ }
86+}
87+
88+func (w *worker) serve(ctx context.Context, req ports.HintScanRequest, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, timeout time.Duration) ports.HintScanResult {
89+ log := logging.For(ctx, "hints")
90+ res := ports.HintScanResult{Seq: req.Seq, Output: req.Output}
91+ fail := func(err error) ports.HintScanResult {
92+ log.Info().Err(err).Str("output", req.Output).Msg("hint scan failed")
93+ res.Err = err
94+ return res
95+ }
96+
97+ if req.Width <= 0 || req.Height <= 0 || req.Width*req.Height > maxPixels {
98+ return fail(fmt.Errorf("%w: %dx%d", ErrBadRequest, req.Width, req.Height))
99+ }
100+ size := req.Width * req.Height * 4
101+ if cap(w.buf) < size {
102+ w.buf = make([]byte, size)
103+ }
104+ buf := w.buf[:size]
105+
106+ w.n++
107+ id := ports.CaptureIDInternal | w.n
108+ stride := req.Width * 4
109+ start := time.Now()
110+ creq := ports.CaptureRequest{
111+ ID: id,
112+ Output: req.Output,
113+ Region: image.Rect(0, 0, req.Width, req.Height),
114+ Mapped: buf,
115+ Width: req.Width,
116+ Height: req.Height,
117+ Stride: stride,
118+ Format: formatXRGB8888,
119+ Indicate: false,
120+ }
121+
122+ timer := time.NewTimer(timeout)
123+ defer timer.Stop()
124+
125+ select {
126+ case captures <- creq:
127+ case <-timer.C:
128+ return fail(ErrTimeout)
129+ case <-ctx.Done():
130+ return fail(ctx.Err())
131+ }
132+
133+ for {
134+ select {
135+ case done := <-captured:
136+ if done.ID != id {
137+ continue
138+ }
139+ if done.Err != nil {
140+ return fail(fmt.Errorf("hints: capture: %w", done.Err))
141+ }
142+ return w.scan(ctx, req, buf, stride, time.Since(start), res)
143+ case other := <-requests:
144+ // The buffer is in use until the capture is done: answer now.
145+ busy := ports.HintScanResult{Seq: other.Seq, Output: other.Output, Err: ErrBusy}
146+ if !send(ctx, results, busy) {
147+ return fail(ctx.Err())
148+ }
149+ case <-timer.C:
150+ // The output may still write into buf: do not reuse it.
151+ w.buf = nil
152+ return fail(ErrTimeout)
153+ case <-ctx.Done():
154+ w.buf = nil
155+ return fail(ctx.Err())
156+ }
157+ }
158+}
159+
160+func (w *worker) scan(ctx context.Context, req ports.HintScanRequest, buf []byte, stride int, captureDur time.Duration, res ports.HintScanResult) ports.HintScanResult {
161+ log := logging.For(ctx, "hints")
162+ kinds := hintscan.Clickable
163+ if req.LinksOnly {
164+ kinds = hintscan.KindSet(1 << hintscan.KindLink)
165+ }
166+ frame := hintscan.Frame{Pix: buf, Stride: stride, Width: req.Width, Height: req.Height, Order: hintscan.OrderBGRA, Scale: req.Scale}
167+
168+ t0 := time.Now()
169+ targets, err := w.scanner.Scan(ctx, frame, hintscan.Options{Kinds: kinds}, w.dst)
170+ scanDur := time.Since(t0)
171+ if err != nil {
172+ log.Info().Err(err).Str("output", req.Output).Msg("hint scan failed")
173+ res.Err = err
174+ return res
175+ }
176+ w.dst = targets
177+
178+ out := make([]ports.HintTarget, 0, len(targets))
179+ for _, t := range targets {
180+ kind := ports.HintLink
181+ if t.Kind == hintscan.KindButton {
182+ kind = ports.HintButton
183+ }
184+ out = append(out, ports.HintTarget{
185+ Rect: ports.Rect{X: t.Rect.Min.X, Y: t.Rect.Min.Y, W: t.Rect.Dx(), H: t.Rect.Dy()},
186+ Kind: kind,
187+ })
188+ }
189+ res.Targets = out
190+ if w.dumpDir != "" {
191+ w.pending = &dumpJob{when: time.Now(), req: req, pix: buf, stride: stride, targets: out, captureD: captureDur, scanD: scanDur}
192+ }
193+ log.Info().
194+ Str("output", req.Output).
195+ Int("targets", len(out)).
196+ Dur("scan", scanDur).
197+ Dur("capture", captureDur).
198+ Msg("hint scan")
199+ return res
200+}
A internal/adapters/hints/hints_test.go +287 -0
@@ -0,0 +1,287 @@
1+package hints
2+
3+import (
4+ "context"
5+ "encoding/json"
6+ "errors"
7+ "fmt"
8+ "image"
9+ "image/draw"
10+ _ "image/png"
11+ "os"
12+ "path/filepath"
13+ "strings"
14+ "testing"
15+ "time"
16+
17+ "github.com/stretchr/testify/require"
18+
19+ "github.com/bnema/neferwl/internal/ports"
20+)
21+
22+type harness struct {
23+ requests chan ports.HintScanRequest
24+ captures chan ports.CaptureRequest
25+ captured chan ports.CaptureDone
26+ results chan ports.HintScanResult
27+}
28+
29+func start(t *testing.T, timeout time.Duration) *harness {
30+ return startWith(t, timeout, Options{})
31+}
32+
33+func startWith(t *testing.T, timeout time.Duration, opts Options) *harness {
34+ t.Helper()
35+ h := &harness{
36+ requests: make(chan ports.HintScanRequest, 1),
37+ captures: make(chan ports.CaptureRequest, 4),
38+ captured: make(chan ports.CaptureDone, 1),
39+ results: make(chan ports.HintScanResult, 4),
40+ }
41+ ctx, cancel := context.WithCancel(context.Background())
42+ exited := make(chan struct{})
43+ go func() {
44+ defer close(exited)
45+ _ = run(ctx, h.requests, h.captures, h.captured, h.results, timeout, opts)
46+ }()
47+ t.Cleanup(func() { cancel(); <-exited })
48+ return h
49+}
50+
51+func loadBGRA(t *testing.T) (pix []byte, w, hgt int) {
52+ t.Helper()
53+ f, err := os.Open("testdata/browser.png")
54+ require.NoError(t, err)
55+ defer f.Close()
56+ img, _, err := image.Decode(f)
57+ require.NoError(t, err)
58+ b := img.Bounds()
59+ rgba := image.NewRGBA(image.Rect(0, 0, b.Dx(), b.Dy()))
60+ draw.Draw(rgba, rgba.Bounds(), img, b.Min, draw.Src)
61+ for i := 0; i < len(rgba.Pix); i += 4 {
62+ rgba.Pix[i], rgba.Pix[i+2] = rgba.Pix[i+2], rgba.Pix[i]
63+ rgba.Pix[i+3] = 0xff
64+ }
65+ return rgba.Pix, b.Dx(), b.Dy()
66+}
67+
68+func TestHintScan(t *testing.T) {
69+ pix, w, hgt := loadBGRA(t)
70+ h := start(t, 5*time.Second)
71+
72+ // Plays the output owner.
73+ go func() {
74+ creq := <-h.captures
75+ copy(creq.Mapped, pix)
76+ h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
77+ }()
78+
79+ h.requests <- ports.HintScanRequest{Seq: 7, Output: "HEADLESS-1", Width: w, Height: hgt, Scale: 1}
80+ res := <-h.results
81+ require.NoError(t, res.Err)
82+ require.Equal(t, uint64(7), res.Seq)
83+ require.Equal(t, "HEADLESS-1", res.Output)
84+ var links int
85+ for _, tg := range res.Targets {
86+ require.Positive(t, tg.Rect.W)
87+ require.Positive(t, tg.Rect.H)
88+ if tg.Kind == ports.HintLink {
89+ links++
90+ }
91+ }
92+ require.GreaterOrEqual(t, links, 1)
93+}
94+
95+func TestHintRequestShape(t *testing.T) {
96+ h := start(t, 5*time.Second)
97+ got := make(chan ports.CaptureRequest, 1)
98+ go func() {
99+ creq := <-h.captures
100+ got <- creq
101+ h.captured <- ports.CaptureDone{ID: creq.ID + 1} // another ID: ignored
102+ h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output} // blank frame
103+ }()
104+ h.requests <- ports.HintScanRequest{Seq: 1, Output: "o", Width: 64, Height: 32, Scale: 1}
105+ res := <-h.results
106+ require.NoError(t, res.Err)
107+ creq := <-got
108+ require.NotZero(t, creq.ID&ports.CaptureIDInternal)
109+ require.False(t, creq.Indicate)
110+ require.Equal(t, image.Rect(0, 0, 64, 32), creq.Region)
111+ require.Len(t, creq.Mapped, 64*32*4)
112+ require.Equal(t, 64*4, creq.Stride)
113+ require.Equal(t, uint32(1), creq.Format)
114+}
115+
116+func TestHintCaptureError(t *testing.T) {
117+ h := start(t, 5*time.Second)
118+ boom := errors.New("boom")
119+ go func() {
120+ creq := <-h.captures
121+ h.captured <- ports.CaptureDone{ID: creq.ID, Err: boom}
122+ }()
123+ h.requests <- ports.HintScanRequest{Seq: 3, Output: "o", Width: 64, Height: 32, Scale: 1}
124+ res := <-h.results
125+ require.ErrorIs(t, res.Err, boom)
126+ require.Equal(t, uint64(3), res.Seq)
127+ require.Empty(t, res.Targets)
128+}
129+
130+func TestHintTimeout(t *testing.T) {
131+ h := start(t, 50*time.Millisecond)
132+ h.requests <- ports.HintScanRequest{Seq: 4, Output: "o", Width: 64, Height: 32, Scale: 1}
133+ res := <-h.results
134+ require.ErrorIs(t, res.Err, ErrTimeout)
135+ require.Equal(t, uint64(4), res.Seq)
136+}
137+
138+func TestHintBusy(t *testing.T) {
139+ h := start(t, 5*time.Second)
140+ h.requests <- ports.HintScanRequest{Seq: 1, Output: "o", Width: 64, Height: 32, Scale: 1}
141+ creq := <-h.captures
142+ h.requests <- ports.HintScanRequest{Seq: 2, Output: "o", Width: 64, Height: 32, Scale: 1}
143+ res := <-h.results
144+ require.ErrorIs(t, res.Err, ErrBusy)
145+ require.Equal(t, uint64(2), res.Seq)
146+ h.captured <- ports.CaptureDone{ID: creq.ID}
147+ res = <-h.results
148+ require.NoError(t, res.Err)
149+ require.Equal(t, uint64(1), res.Seq)
150+}
151+
152+func TestHintBadSize(t *testing.T) {
153+ h := start(t, time.Second)
154+ h.requests <- ports.HintScanRequest{Seq: 5, Output: "o"}
155+ res := <-h.results
156+ require.ErrorIs(t, res.Err, ErrBadRequest)
157+}
158+
159+func scanOnce(t *testing.T, h *harness, pix []byte, w, hgt int, seq uint64) ports.HintScanResult {
160+ t.Helper()
161+ go func() {
162+ creq := <-h.captures
163+ copy(creq.Mapped, pix)
164+ h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
165+ }()
166+ h.requests <- ports.HintScanRequest{Seq: seq, Output: "HEADLESS-1", Width: w, Height: hgt, Scale: 1}
167+ res := <-h.results
168+ require.NoError(t, res.Err)
169+ return res
170+}
171+
172+// waitFiles waits for the dump the worker writes after sending the result.
173+func waitFiles(t *testing.T, dir string, n int) []os.DirEntry {
174+ t.Helper()
175+ var entries []os.DirEntry
176+ require.Eventually(t, func() bool {
177+ entries, _ = os.ReadDir(dir)
178+ return len(entries) >= n
179+ }, 5*time.Second, 10*time.Millisecond)
180+ return entries
181+}
182+
183+func TestHintDump(t *testing.T) {
184+ pix, w, hgt := loadBGRA(t)
185+ dir := filepath.Join(t.TempDir(), "hints")
186+ h := startWith(t, 5*time.Second, Options{DumpDir: dir})
187+ res := scanOnce(t, h, pix, w, hgt, 1)
188+ require.NotEmpty(t, res.Targets)
189+
190+ entries := waitFiles(t, dir, 3)
191+ // A second request is only served once the dump is written.
192+ _ = scanOnceDone(t, h)
193+ var png, ann, js string
194+ for _, e := range entries {
195+ info, err := e.Info()
196+ require.NoError(t, err)
197+ require.Equal(t, os.FileMode(0o600), info.Mode().Perm())
198+ switch {
199+ case strings.HasSuffix(e.Name(), ".annotated.png"):
200+ ann = e.Name()
201+ case strings.HasSuffix(e.Name(), ".png"):
202+ png = e.Name()
203+ case strings.HasSuffix(e.Name(), ".json"):
204+ js = e.Name()
205+ }
206+ }
207+ require.NotEmpty(t, png)
208+ require.NotEmpty(t, ann)
209+ require.Regexp(t, `^\d{8}-\d{6}\.\d{3}-HEADLESS-1\.json$`, js)
210+
211+ st, err := os.Stat(dir)
212+ require.NoError(t, err)
213+ require.Equal(t, os.FileMode(0o700), st.Mode().Perm())
214+
215+ data, err := os.ReadFile(filepath.Join(dir, js))
216+ require.NoError(t, err)
217+ var meta struct {
218+ Output string `json:"output"`
219+ Width int `json:"width"`
220+ Height int `json:"height"`
221+ Stride int `json:"stride"`
222+ Scale float64 `json:"scale"`
223+ LinksOnly bool `json:"links_only"`
224+ Targets []struct {
225+ Kind string `json:"kind"`
226+ Rect [4]int `json:"rect"`
227+ } `json:"targets"`
228+ }
229+ require.NoError(t, json.Unmarshal(data, &meta))
230+ require.Equal(t, "HEADLESS-1", meta.Output)
231+ require.Equal(t, w, meta.Width)
232+ require.Equal(t, hgt, meta.Height)
233+ require.Equal(t, w*4, meta.Stride)
234+ require.Equal(t, 1.0, meta.Scale)
235+ require.Len(t, meta.Targets, len(res.Targets))
236+ for i, tg := range res.Targets {
237+ require.Equal(t, [4]int{tg.Rect.X, tg.Rect.Y, tg.Rect.X + tg.Rect.W, tg.Rect.Y + tg.Rect.H}, meta.Targets[i].Rect)
238+ }
239+
240+ f, err := os.Open(filepath.Join(dir, png))
241+ require.NoError(t, err)
242+ defer f.Close()
243+ img, _, err := image.Decode(f)
244+ require.NoError(t, err)
245+ require.Equal(t, image.Rect(0, 0, w, hgt), img.Bounds())
246+ r, g, b, _ := img.At(0, 0).RGBA()
247+ require.Equal(t, uint32(pix[2])*0x101, r)
248+ require.Equal(t, uint32(pix[1])*0x101, g)
249+ require.Equal(t, uint32(pix[0])*0x101, b)
250+}
251+
252+// scanOnceDone sends a request that fails fast on a blank capture and waits
253+// for its result, which proves the previous dump finished.
254+func scanOnceDone(t *testing.T, h *harness) ports.HintScanResult {
255+ t.Helper()
256+ go func() {
257+ creq := <-h.captures
258+ h.captured <- ports.CaptureDone{ID: creq.ID, Err: errors.New("stop")}
259+ }()
260+ h.requests <- ports.HintScanRequest{Seq: 99, Output: "x", Width: 8, Height: 8, Scale: 1}
261+ return <-h.results
262+}
263+
264+func TestHintDumpDisabled(t *testing.T) {
265+ pix, w, hgt := loadBGRA(t)
266+ dir := filepath.Join(t.TempDir(), "hints")
267+ h := start(t, 5*time.Second)
268+ scanOnce(t, h, pix, w, hgt, 1)
269+ scanOnceDone(t, h)
270+ _, err := os.Stat(dir)
271+ require.ErrorIs(t, err, os.ErrNotExist)
272+}
273+
274+func TestHintDumpRetention(t *testing.T) {
275+ dir := t.TempDir()
276+ for i := 0; i < 60; i++ {
277+ for _, ext := range []string{".png", ".annotated.png", ".json"} {
278+ name := fmt.Sprintf("20260101-000000.%03d-o%s", i, ext)
279+ require.NoError(t, os.WriteFile(filepath.Join(dir, name), nil, 0o600))
280+ }
281+ }
282+ require.NoError(t, prune(dir, maxDumps))
283+ entries, err := os.ReadDir(dir)
284+ require.NoError(t, err)
285+ require.Len(t, entries, maxDumps*3)
286+ require.Equal(t, "20260101-000000.010-o.annotated.png", entries[0].Name())
287+}
@@ -0,0 +1,287 @@
1+package hints
2+
3+import (
4+ "context"
5+ "encoding/json"
6+ "errors"
7+ "fmt"
8+ "image"
9+ "image/draw"
10+ _ "image/png"
11+ "os"
12+ "path/filepath"
13+ "strings"
14+ "testing"
15+ "time"
16+
17+ "github.com/stretchr/testify/require"
18+
19+ "github.com/bnema/neferwl/internal/ports"
20+)
21+
22+type harness struct {
23+ requests chan ports.HintScanRequest
24+ captures chan ports.CaptureRequest
25+ captured chan ports.CaptureDone
26+ results chan ports.HintScanResult
27+}
28+
29+func start(t *testing.T, timeout time.Duration) *harness {
30+ return startWith(t, timeout, Options{})
31+}
32+
33+func startWith(t *testing.T, timeout time.Duration, opts Options) *harness {
34+ t.Helper()
35+ h := &harness{
36+ requests: make(chan ports.HintScanRequest, 1),
37+ captures: make(chan ports.CaptureRequest, 4),
38+ captured: make(chan ports.CaptureDone, 1),
39+ results: make(chan ports.HintScanResult, 4),
40+ }
41+ ctx, cancel := context.WithCancel(context.Background())
42+ exited := make(chan struct{})
43+ go func() {
44+ defer close(exited)
45+ _ = run(ctx, h.requests, h.captures, h.captured, h.results, timeout, opts)
46+ }()
47+ t.Cleanup(func() { cancel(); <-exited })
48+ return h
49+}
50+
51+func loadBGRA(t *testing.T) (pix []byte, w, hgt int) {
52+ t.Helper()
53+ f, err := os.Open("testdata/browser.png")
54+ require.NoError(t, err)
55+ defer f.Close()
56+ img, _, err := image.Decode(f)
57+ require.NoError(t, err)
58+ b := img.Bounds()
59+ rgba := image.NewRGBA(image.Rect(0, 0, b.Dx(), b.Dy()))
60+ draw.Draw(rgba, rgba.Bounds(), img, b.Min, draw.Src)
61+ for i := 0; i < len(rgba.Pix); i += 4 {
62+ rgba.Pix[i], rgba.Pix[i+2] = rgba.Pix[i+2], rgba.Pix[i]
63+ rgba.Pix[i+3] = 0xff
64+ }
65+ return rgba.Pix, b.Dx(), b.Dy()
66+}
67+
68+func TestHintScan(t *testing.T) {
69+ pix, w, hgt := loadBGRA(t)
70+ h := start(t, 5*time.Second)
71+
72+ // Plays the output owner.
73+ go func() {
74+ creq := <-h.captures
75+ copy(creq.Mapped, pix)
76+ h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
77+ }()
78+
79+ h.requests <- ports.HintScanRequest{Seq: 7, Output: "HEADLESS-1", Width: w, Height: hgt, Scale: 1}
80+ res := <-h.results
81+ require.NoError(t, res.Err)
82+ require.Equal(t, uint64(7), res.Seq)
83+ require.Equal(t, "HEADLESS-1", res.Output)
84+ var links int
85+ for _, tg := range res.Targets {
86+ require.Positive(t, tg.Rect.W)
87+ require.Positive(t, tg.Rect.H)
88+ if tg.Kind == ports.HintLink {
89+ links++
90+ }
91+ }
92+ require.GreaterOrEqual(t, links, 1)
93+}
94+
95+func TestHintRequestShape(t *testing.T) {
96+ h := start(t, 5*time.Second)
97+ got := make(chan ports.CaptureRequest, 1)
98+ go func() {
99+ creq := <-h.captures
100+ got <- creq
101+ h.captured <- ports.CaptureDone{ID: creq.ID + 1} // another ID: ignored
102+ h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output} // blank frame
103+ }()
104+ h.requests <- ports.HintScanRequest{Seq: 1, Output: "o", Width: 64, Height: 32, Scale: 1}
105+ res := <-h.results
106+ require.NoError(t, res.Err)
107+ creq := <-got
108+ require.NotZero(t, creq.ID&ports.CaptureIDInternal)
109+ require.False(t, creq.Indicate)
110+ require.Equal(t, image.Rect(0, 0, 64, 32), creq.Region)
111+ require.Len(t, creq.Mapped, 64*32*4)
112+ require.Equal(t, 64*4, creq.Stride)
113+ require.Equal(t, uint32(1), creq.Format)
114+}
115+
116+func TestHintCaptureError(t *testing.T) {
117+ h := start(t, 5*time.Second)
118+ boom := errors.New("boom")
119+ go func() {
120+ creq := <-h.captures
121+ h.captured <- ports.CaptureDone{ID: creq.ID, Err: boom}
122+ }()
123+ h.requests <- ports.HintScanRequest{Seq: 3, Output: "o", Width: 64, Height: 32, Scale: 1}
124+ res := <-h.results
125+ require.ErrorIs(t, res.Err, boom)
126+ require.Equal(t, uint64(3), res.Seq)
127+ require.Empty(t, res.Targets)
128+}
129+
130+func TestHintTimeout(t *testing.T) {
131+ h := start(t, 50*time.Millisecond)
132+ h.requests <- ports.HintScanRequest{Seq: 4, Output: "o", Width: 64, Height: 32, Scale: 1}
133+ res := <-h.results
134+ require.ErrorIs(t, res.Err, ErrTimeout)
135+ require.Equal(t, uint64(4), res.Seq)
136+}
137+
138+func TestHintBusy(t *testing.T) {
139+ h := start(t, 5*time.Second)
140+ h.requests <- ports.HintScanRequest{Seq: 1, Output: "o", Width: 64, Height: 32, Scale: 1}
141+ creq := <-h.captures
142+ h.requests <- ports.HintScanRequest{Seq: 2, Output: "o", Width: 64, Height: 32, Scale: 1}
143+ res := <-h.results
144+ require.ErrorIs(t, res.Err, ErrBusy)
145+ require.Equal(t, uint64(2), res.Seq)
146+ h.captured <- ports.CaptureDone{ID: creq.ID}
147+ res = <-h.results
148+ require.NoError(t, res.Err)
149+ require.Equal(t, uint64(1), res.Seq)
150+}
151+
152+func TestHintBadSize(t *testing.T) {
153+ h := start(t, time.Second)
154+ h.requests <- ports.HintScanRequest{Seq: 5, Output: "o"}
155+ res := <-h.results
156+ require.ErrorIs(t, res.Err, ErrBadRequest)
157+}
158+
159+func scanOnce(t *testing.T, h *harness, pix []byte, w, hgt int, seq uint64) ports.HintScanResult {
160+ t.Helper()
161+ go func() {
162+ creq := <-h.captures
163+ copy(creq.Mapped, pix)
164+ h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
165+ }()
166+ h.requests <- ports.HintScanRequest{Seq: seq, Output: "HEADLESS-1", Width: w, Height: hgt, Scale: 1}
167+ res := <-h.results
168+ require.NoError(t, res.Err)
169+ return res
170+}
171+
172+// waitFiles waits for the dump the worker writes after sending the result.
173+func waitFiles(t *testing.T, dir string, n int) []os.DirEntry {
174+ t.Helper()
175+ var entries []os.DirEntry
176+ require.Eventually(t, func() bool {
177+ entries, _ = os.ReadDir(dir)
178+ return len(entries) >= n
179+ }, 5*time.Second, 10*time.Millisecond)
180+ return entries
181+}
182+
183+func TestHintDump(t *testing.T) {
184+ pix, w, hgt := loadBGRA(t)
185+ dir := filepath.Join(t.TempDir(), "hints")
186+ h := startWith(t, 5*time.Second, Options{DumpDir: dir})
187+ res := scanOnce(t, h, pix, w, hgt, 1)
188+ require.NotEmpty(t, res.Targets)
189+
190+ entries := waitFiles(t, dir, 3)
191+ // A second request is only served once the dump is written.
192+ _ = scanOnceDone(t, h)
193+ var png, ann, js string
194+ for _, e := range entries {
195+ info, err := e.Info()
196+ require.NoError(t, err)
197+ require.Equal(t, os.FileMode(0o600), info.Mode().Perm())
198+ switch {
199+ case strings.HasSuffix(e.Name(), ".annotated.png"):
200+ ann = e.Name()
201+ case strings.HasSuffix(e.Name(), ".png"):
202+ png = e.Name()
203+ case strings.HasSuffix(e.Name(), ".json"):
204+ js = e.Name()
205+ }
206+ }
207+ require.NotEmpty(t, png)
208+ require.NotEmpty(t, ann)
209+ require.Regexp(t, `^\d{8}-\d{6}\.\d{3}-HEADLESS-1\.json$`, js)
210+
211+ st, err := os.Stat(dir)
212+ require.NoError(t, err)
213+ require.Equal(t, os.FileMode(0o700), st.Mode().Perm())
214+
215+ data, err := os.ReadFile(filepath.Join(dir, js))
216+ require.NoError(t, err)
217+ var meta struct {
218+ Output string `json:"output"`
219+ Width int `json:"width"`
220+ Height int `json:"height"`
221+ Stride int `json:"stride"`
222+ Scale float64 `json:"scale"`
223+ LinksOnly bool `json:"links_only"`
224+ Targets []struct {
225+ Kind string `json:"kind"`
226+ Rect [4]int `json:"rect"`
227+ } `json:"targets"`
228+ }
229+ require.NoError(t, json.Unmarshal(data, &meta))
230+ require.Equal(t, "HEADLESS-1", meta.Output)
231+ require.Equal(t, w, meta.Width)
232+ require.Equal(t, hgt, meta.Height)
233+ require.Equal(t, w*4, meta.Stride)
234+ require.Equal(t, 1.0, meta.Scale)
235+ require.Len(t, meta.Targets, len(res.Targets))
236+ for i, tg := range res.Targets {
237+ require.Equal(t, [4]int{tg.Rect.X, tg.Rect.Y, tg.Rect.X + tg.Rect.W, tg.Rect.Y + tg.Rect.H}, meta.Targets[i].Rect)
238+ }
239+
240+ f, err := os.Open(filepath.Join(dir, png))
241+ require.NoError(t, err)
242+ defer f.Close()
243+ img, _, err := image.Decode(f)
244+ require.NoError(t, err)
245+ require.Equal(t, image.Rect(0, 0, w, hgt), img.Bounds())
246+ r, g, b, _ := img.At(0, 0).RGBA()
247+ require.Equal(t, uint32(pix[2])*0x101, r)
248+ require.Equal(t, uint32(pix[1])*0x101, g)
249+ require.Equal(t, uint32(pix[0])*0x101, b)
250+}
251+
252+// scanOnceDone sends a request that fails fast on a blank capture and waits
253+// for its result, which proves the previous dump finished.
254+func scanOnceDone(t *testing.T, h *harness) ports.HintScanResult {
255+ t.Helper()
256+ go func() {
257+ creq := <-h.captures
258+ h.captured <- ports.CaptureDone{ID: creq.ID, Err: errors.New("stop")}
259+ }()
260+ h.requests <- ports.HintScanRequest{Seq: 99, Output: "x", Width: 8, Height: 8, Scale: 1}
261+ return <-h.results
262+}
263+
264+func TestHintDumpDisabled(t *testing.T) {
265+ pix, w, hgt := loadBGRA(t)
266+ dir := filepath.Join(t.TempDir(), "hints")
267+ h := start(t, 5*time.Second)
268+ scanOnce(t, h, pix, w, hgt, 1)
269+ scanOnceDone(t, h)
270+ _, err := os.Stat(dir)
271+ require.ErrorIs(t, err, os.ErrNotExist)
272+}
273+
274+func TestHintDumpRetention(t *testing.T) {
275+ dir := t.TempDir()
276+ for i := 0; i < 60; i++ {
277+ for _, ext := range []string{".png", ".annotated.png", ".json"} {
278+ name := fmt.Sprintf("20260101-000000.%03d-o%s", i, ext)
279+ require.NoError(t, os.WriteFile(filepath.Join(dir, name), nil, 0o600))
280+ }
281+ }
282+ require.NoError(t, prune(dir, maxDumps))
283+ entries, err := os.ReadDir(dir)
284+ require.NoError(t, err)
285+ require.Len(t, entries, maxDumps*3)
286+ require.Equal(t, "20260101-000000.010-o.annotated.png", entries[0].Name())
287+}
A internal/adapters/hints/testdata/browser.png +0 -0 binary file: no text diff

binary file: no text diff

M internal/adapters/outputkit/capture/excluded.go +5 -5
@@ -10,7 +10,7 @@ import (
1010
1111 // A request with Exclude leaves out the exclusion's HUD layers and popups;
1212 // every other capture shows them. No capture shows the capture indicator
13-// (Scene.CaptureIndicators): a scene that carries it is never the source of a
13+// (Scene.CaptureIndicators): a scene that carries it, or hint labels, is never the source of a
1414 // capture image. Requests of the displayed frame are then served from the
1515 // frame rendered without it first (SubmitPlain, ExcludedScene).
1616
@@ -64,10 +64,10 @@ func excludedShown(s ports.Scene) bool {
6464 return false
6565 }
6666
67-// plain returns s without its capture state and indicators, Seq 0 so the
67+// plain returns s without its capture state, indicators and hint labels, Seq 0 so the
6868 // renderer redraws the whole target instead of reusing a displayed frame.
6969 func plain(s ports.Scene) ports.Scene {
70- s.Capture, s.CaptureIndicators, s.Seq = nil, nil, 0
70+ s.Capture, s.CaptureIndicators, s.HintLabels, s.Seq = nil, nil, nil, 0
7171 return s
7272 }
7373
@@ -85,7 +85,7 @@ func plain(s ports.Scene) ports.Scene {
8585 // the plain render reads client buffers even when the requests are rejected,
8686 // so the caller owns it (hold or wait it) whatever happened, error included.
8787 func (p *Pipeline) SubmitPlain(r ports.Renderer, s ports.Scene, surfaces map[ports.WindowID]ports.SurfaceContent, normal []ports.CaptureRequest) (took bool, done *os.File, err error) {
88- if len(normal) == 0 || len(s.CaptureIndicators) == 0 {
88+ if len(normal) == 0 || len(s.CaptureIndicators) == 0 && len(s.HintLabels) == 0 {
8989 return false, nil, nil
9090 }
9191 done, err = r.Render(plain(s), surfaces)
@@ -164,7 +164,7 @@ func (p *Pipeline) Split(s ports.Scene, reqs []ports.CaptureRequest) (normal, ex
164164 }
165165 clear(reqs[n:])
166166 reqs = reqs[:n]
167- excl := excludedShown(s) || len(s.CaptureIndicators) > 0
167+ excl := excludedShown(s) || len(s.CaptureIndicators) > 0 || len(s.HintLabels) > 0
168168 group := func(q ports.CaptureRequest) int {
169169 switch {
170170 case q.OffScreen:
@@ -10,7 +10,7 @@ import (
10 10
11 // A request with Exclude leaves out the exclusion's HUD layers and popups;11 // A request with Exclude leaves out the exclusion's HUD layers and popups;
12 // every other capture shows them. No capture shows the capture indicator12 // every other capture shows them. No capture shows the capture indicator
13-// (Scene.CaptureIndicators): a scene that carries it is never the source of a13+// (Scene.CaptureIndicators): a scene that carries it, or hint labels, is never the source of a
14 // capture image. Requests of the displayed frame are then served from the14 // capture image. Requests of the displayed frame are then served from the
15 // frame rendered without it first (SubmitPlain, ExcludedScene).15 // frame rendered without it first (SubmitPlain, ExcludedScene).
16 16
@@ -64,10 +64,10 @@ func excludedShown(s ports.Scene) bool {
64 return false64 return false
65 }65 }
66 66
67-// plain returns s without its capture state and indicators, Seq 0 so the67+// plain returns s without its capture state, indicators and hint labels, Seq 0 so the
68 // renderer redraws the whole target instead of reusing a displayed frame.68 // renderer redraws the whole target instead of reusing a displayed frame.
69 func plain(s ports.Scene) ports.Scene {69 func plain(s ports.Scene) ports.Scene {
70- s.Capture, s.CaptureIndicators, s.Seq = nil, nil, 070+ s.Capture, s.CaptureIndicators, s.HintLabels, s.Seq = nil, nil, nil, 0
71 return s71 return s
72 }72 }
73 73
@@ -85,7 +85,7 @@ func plain(s ports.Scene) ports.Scene {
85 // the plain render reads client buffers even when the requests are rejected,85 // the plain render reads client buffers even when the requests are rejected,
86 // so the caller owns it (hold or wait it) whatever happened, error included.86 // so the caller owns it (hold or wait it) whatever happened, error included.
87 func (p *Pipeline) SubmitPlain(r ports.Renderer, s ports.Scene, surfaces map[ports.WindowID]ports.SurfaceContent, normal []ports.CaptureRequest) (took bool, done *os.File, err error) {87 func (p *Pipeline) SubmitPlain(r ports.Renderer, s ports.Scene, surfaces map[ports.WindowID]ports.SurfaceContent, normal []ports.CaptureRequest) (took bool, done *os.File, err error) {
88- if len(normal) == 0 || len(s.CaptureIndicators) == 0 {88+ if len(normal) == 0 || len(s.CaptureIndicators) == 0 && len(s.HintLabels) == 0 {
89 return false, nil, nil89 return false, nil, nil
90 }90 }
91 done, err = r.Render(plain(s), surfaces)91 done, err = r.Render(plain(s), surfaces)
@@ -164,7 +164,7 @@ func (p *Pipeline) Split(s ports.Scene, reqs []ports.CaptureRequest) (normal, ex
164 }164 }
165 clear(reqs[n:])165 clear(reqs[n:])
166 reqs = reqs[:n]166 reqs = reqs[:n]
167- excl := excludedShown(s) || len(s.CaptureIndicators) > 0167+ excl := excludedShown(s) || len(s.CaptureIndicators) > 0 || len(s.HintLabels) > 0
168 group := func(q ports.CaptureRequest) int {168 group := func(q ports.CaptureRequest) int {
169 switch {169 switch {
170 case q.OffScreen:170 case q.OffScreen:
M internal/adapters/outputkit/capture/indicator_test.go +40 -0
@@ -112,3 +112,43 @@ func TestSubmitPlainRejectsProtectedEpoch(t *testing.T) {
112112 _, statErr := file.Stat()
113113 require.ErrorIs(t, statErr, os.ErrClosed)
114114 }
115+
116+// Hint labels are never captured either: a scene with only labels is served
117+// from the plain frame, and an Exclude request is ordered to the excluded scene.
118+func TestSubmitPlainServesHintLabelScenesWithoutLabels(t *testing.T) {
119+ replies := make(chan ports.CaptureDone, 1)
120+ p := NewPipeline(context.Background(), replies)
121+ r := portsmocks.NewMockRenderer(t)
122+ frame := portsmocks.NewMockCaptureFrame(t)
123+ frame.EXPECT().Done().Return((*os.File)(nil))
124+ frame.EXPECT().Read(image.Rect(0, 0, 2, 2), mock.Anything, 8).Return(nil).Once()
125+ var rendered []ports.Scene
126+ r.EXPECT().Render(mock.Anything, mock.Anything).RunAndReturn(func(s ports.Scene, _ map[ports.WindowID]ports.SurfaceContent) (*os.File, error) {
127+ rendered = append(rendered, s)
128+ return nil, nil
129+ }).Once()
130+ r.EXPECT().BeginCapture().Return(frame, nil).Once()
131+ r.EXPECT().EndCapture(frame).Return().Once()
132+ s := excludedTestScene()
133+ s.Capture = nil
134+ s.HintLabels = []ports.HintLabel{{Text: "f"}}
135+ reqs := []ports.CaptureRequest{pipelineRequest(t, 1)}
136+ require.False(t, reqs[0].Indicate)
137+ took, done, err := p.SubmitPlain(r, s, nil, reqs)
138+ require.NoError(t, err)
139+ require.Nil(t, done)
140+ require.True(t, took)
141+ require.Len(t, rendered, 1)
142+ require.Nil(t, rendered[0].HintLabels)
143+ require.Zero(t, rendered[0].Seq)
144+ require.Equal(t, uint64(1), awaitCapture(t, replies).ID)
145+ p.Close(r)
146+ frame.Calls = nil
147+
148+ withSession := s
149+ withSession.Capture = &ports.SceneCapture{Session: 7}
150+ normal, excluded, _ := p.Split(withSession, []ports.CaptureRequest{{ID: 2, Exclude: true, Session: 7}})
151+ require.Empty(t, normal)
152+ require.Equal(t, []uint64{2}, ids(excluded), "labels force the Exclude request to the excluded scene")
153+ require.Nil(t, p.ExcludedScene(s).HintLabels)
154+}
@@ -112,3 +112,43 @@ func TestSubmitPlainRejectsProtectedEpoch(t *testing.T) {
112 _, statErr := file.Stat()112 _, statErr := file.Stat()
113 require.ErrorIs(t, statErr, os.ErrClosed)113 require.ErrorIs(t, statErr, os.ErrClosed)
114 }114 }
115+
116+// Hint labels are never captured either: a scene with only labels is served
117+// from the plain frame, and an Exclude request is ordered to the excluded scene.
118+func TestSubmitPlainServesHintLabelScenesWithoutLabels(t *testing.T) {
119+ replies := make(chan ports.CaptureDone, 1)
120+ p := NewPipeline(context.Background(), replies)
121+ r := portsmocks.NewMockRenderer(t)
122+ frame := portsmocks.NewMockCaptureFrame(t)
123+ frame.EXPECT().Done().Return((*os.File)(nil))
124+ frame.EXPECT().Read(image.Rect(0, 0, 2, 2), mock.Anything, 8).Return(nil).Once()
125+ var rendered []ports.Scene
126+ r.EXPECT().Render(mock.Anything, mock.Anything).RunAndReturn(func(s ports.Scene, _ map[ports.WindowID]ports.SurfaceContent) (*os.File, error) {
127+ rendered = append(rendered, s)
128+ return nil, nil
129+ }).Once()
130+ r.EXPECT().BeginCapture().Return(frame, nil).Once()
131+ r.EXPECT().EndCapture(frame).Return().Once()
132+ s := excludedTestScene()
133+ s.Capture = nil
134+ s.HintLabels = []ports.HintLabel{{Text: "f"}}
135+ reqs := []ports.CaptureRequest{pipelineRequest(t, 1)}
136+ require.False(t, reqs[0].Indicate)
137+ took, done, err := p.SubmitPlain(r, s, nil, reqs)
138+ require.NoError(t, err)
139+ require.Nil(t, done)
140+ require.True(t, took)
141+ require.Len(t, rendered, 1)
142+ require.Nil(t, rendered[0].HintLabels)
143+ require.Zero(t, rendered[0].Seq)
144+ require.Equal(t, uint64(1), awaitCapture(t, replies).ID)
145+ p.Close(r)
146+ frame.Calls = nil
147+
148+ withSession := s
149+ withSession.Capture = &ports.SceneCapture{Session: 7}
150+ normal, excluded, _ := p.Split(withSession, []ports.CaptureRequest{{ID: 2, Exclude: true, Session: 7}})
151+ require.Empty(t, normal)
152+ require.Equal(t, []uint64{2}, ids(excluded), "labels force the Exclude request to the excluded scene")
153+ require.Nil(t, p.ExcludedScene(s).HintLabels)
154+}
M internal/adapters/vulkan/damage.go +3 -1
@@ -238,9 +238,11 @@ func (tg *target) hold(s ports.Scene, d *damageRegion) {
238238 tg.heldLayers = append(tg.heldLayers[:0], s.Layers...)
239239 tg.heldHints = append(tg.heldHints[:0], s.DropHints...)
240240 tg.heldIndicators = append(tg.heldIndicators[:0], s.CaptureIndicators...)
241+ tg.heldLabels = append(tg.heldLabels[:0], s.HintLabels...)
241242 tg.valid, tg.sceneSeq, tg.scene, tg.windows = true, s.Seq, s, d.drawn
242243 tg.scene.Windows, tg.scene.Separators, tg.scene.Layers = tg.heldWindows, tg.heldSeparators, tg.heldLayers
243244 tg.scene.DropHints, tg.scene.CaptureIndicators = tg.heldHints, tg.heldIndicators
245+ tg.scene.HintLabels = tg.heldLabels
244246 // Only whether they are set matters (any capture redraws everything):
245247 // sentinels, so no pointer into the caller's capture state is kept.
246248 tg.scene.Capture, tg.scene.CaptureScene = nil, nil
@@ -275,7 +277,7 @@ func sceneDelta(old, cur ports.Scene, phys func(ports.Rect) image.Rectangle, lim
275277 old.WorkspaceClip != cur.WorkspaceClip || old.TileInset != cur.TileInset || old.Dim != cur.Dim || old.DimBehind != cur.DimBehind ||
276278 old.Capture != nil || cur.Capture != nil || old.CaptureScene != nil || cur.CaptureScene != nil ||
277279 !slices.Equal(old.Layers, cur.Layers) || !slices.Equal(old.DropHints, cur.DropHints) ||
278- !slices.Equal(old.CaptureIndicators, cur.CaptureIndicators) ||
280+ !slices.Equal(old.CaptureIndicators, cur.CaptureIndicators) || !slices.Equal(old.HintLabels, cur.HintLabels) ||
279281 len(old.Windows) != len(cur.Windows) {
280282 return image.Rectangle{}, false
281283 }
@@ -238,9 +238,11 @@ func (tg *target) hold(s ports.Scene, d *damageRegion) {
238 tg.heldLayers = append(tg.heldLayers[:0], s.Layers...)238 tg.heldLayers = append(tg.heldLayers[:0], s.Layers...)
239 tg.heldHints = append(tg.heldHints[:0], s.DropHints...)239 tg.heldHints = append(tg.heldHints[:0], s.DropHints...)
240 tg.heldIndicators = append(tg.heldIndicators[:0], s.CaptureIndicators...)240 tg.heldIndicators = append(tg.heldIndicators[:0], s.CaptureIndicators...)
241+ tg.heldLabels = append(tg.heldLabels[:0], s.HintLabels...)
241 tg.valid, tg.sceneSeq, tg.scene, tg.windows = true, s.Seq, s, d.drawn242 tg.valid, tg.sceneSeq, tg.scene, tg.windows = true, s.Seq, s, d.drawn
242 tg.scene.Windows, tg.scene.Separators, tg.scene.Layers = tg.heldWindows, tg.heldSeparators, tg.heldLayers243 tg.scene.Windows, tg.scene.Separators, tg.scene.Layers = tg.heldWindows, tg.heldSeparators, tg.heldLayers
243 tg.scene.DropHints, tg.scene.CaptureIndicators = tg.heldHints, tg.heldIndicators244 tg.scene.DropHints, tg.scene.CaptureIndicators = tg.heldHints, tg.heldIndicators
245+ tg.scene.HintLabels = tg.heldLabels
244 // Only whether they are set matters (any capture redraws everything):246 // Only whether they are set matters (any capture redraws everything):
245 // sentinels, so no pointer into the caller's capture state is kept.247 // sentinels, so no pointer into the caller's capture state is kept.
246 tg.scene.Capture, tg.scene.CaptureScene = nil, nil248 tg.scene.Capture, tg.scene.CaptureScene = nil, nil
@@ -275,7 +277,7 @@ func sceneDelta(old, cur ports.Scene, phys func(ports.Rect) image.Rectangle, lim
275 old.WorkspaceClip != cur.WorkspaceClip || old.TileInset != cur.TileInset || old.Dim != cur.Dim || old.DimBehind != cur.DimBehind ||277 old.WorkspaceClip != cur.WorkspaceClip || old.TileInset != cur.TileInset || old.Dim != cur.Dim || old.DimBehind != cur.DimBehind ||
276 old.Capture != nil || cur.Capture != nil || old.CaptureScene != nil || cur.CaptureScene != nil ||278 old.Capture != nil || cur.Capture != nil || old.CaptureScene != nil || cur.CaptureScene != nil ||
277 !slices.Equal(old.Layers, cur.Layers) || !slices.Equal(old.DropHints, cur.DropHints) ||279 !slices.Equal(old.Layers, cur.Layers) || !slices.Equal(old.DropHints, cur.DropHints) ||
278- !slices.Equal(old.CaptureIndicators, cur.CaptureIndicators) ||280+ !slices.Equal(old.CaptureIndicators, cur.CaptureIndicators) || !slices.Equal(old.HintLabels, cur.HintLabels) ||
279 len(old.Windows) != len(cur.Windows) {281 len(old.Windows) != len(cur.Windows) {
280 return image.Rectangle{}, false282 return image.Rectangle{}, false
281 }283 }
M internal/adapters/vulkan/damage_delta_test.go +2 -2
@@ -17,9 +17,9 @@ import (
1717 // TestSceneFieldCount fails when Scene gains or loses a field: sceneDelta
1818 // lists the fields that force a full redraw.
1919 func TestSceneFieldCount(t *testing.T) {
20- // 23 with Animating, which sceneDelta intentionally ignores: it only
20+ // 24 with Animating, which sceneDelta intentionally ignores: it only
2121 // schedules the next frame, it draws nothing.
22- const fields = 23
22+ const fields = 24
2323 assert.Equal(t, fields, len(reflect.VisibleFields(reflect.TypeOf(ports.Scene{}))),
2424 "Scene fields changed: check sceneDelta (a field it does not compare must force a full redraw or be covered), then this count")
2525 }
@@ -17,9 +17,9 @@ import (
17 // TestSceneFieldCount fails when Scene gains or loses a field: sceneDelta17 // TestSceneFieldCount fails when Scene gains or loses a field: sceneDelta
18 // lists the fields that force a full redraw.18 // lists the fields that force a full redraw.
19 func TestSceneFieldCount(t *testing.T) {19 func TestSceneFieldCount(t *testing.T) {
20- // 23 with Animating, which sceneDelta intentionally ignores: it only20+ // 24 with Animating, which sceneDelta intentionally ignores: it only
21 // schedules the next frame, it draws nothing.21 // schedules the next frame, it draws nothing.
22- const fields = 2322+ const fields = 24
23 assert.Equal(t, fields, len(reflect.VisibleFields(reflect.TypeOf(ports.Scene{}))),23 assert.Equal(t, fields, len(reflect.VisibleFields(reflect.TypeOf(ports.Scene{}))),
24 "Scene fields changed: check sceneDelta (a field it does not compare must force a full redraw or be covered), then this count")24 "Scene fields changed: check sceneDelta (a field it does not compare must force a full redraw or be covered), then this count")
25 }25 }
A internal/adapters/vulkan/hint_labels.go +44 -0
@@ -0,0 +1,44 @@
1+package vulkan
2+
3+import (
4+ "image"
5+)
6+
7+// hintLabels draws the keyboard hint labels (ports.Scene.HintLabels) over
8+// everything, after the capture indicators. Each label is rasterized on the CPU
9+// (hintlabel) at the scene's scale and drawn from the SHM copy path: the GPU
10+// buffer is cached under a key naming the label, so a label that stays the
11+// same uploads once. Captures are rendered from scenes without labels.
12+func (w *sceneWalk) hintLabels() {
13+ if len(w.s.HintLabels) == 0 {
14+ return
15+ }
16+ r := w.r
17+ for _, l := range w.s.HintLabels {
18+ pix, lw, lh := r.labels.Label(l.Text, l.Typed, w.scale)
19+ if lw <= 0 || lh <= 0 {
20+ continue
21+ }
22+ // Inside the output, whatever the target's position.
23+ x := min(max(w.phys(l.Rect.X), w.bounds.Min.X), max(w.bounds.Max.X-lw, w.bounds.Min.X))
24+ y := min(max(w.phys(l.Rect.Y), w.bounds.Min.Y), max(w.bounds.Max.Y-lh, w.bounds.Min.Y))
25+ rect := image.Rect(x, y, x+lw, y+lh)
26+ if rect.Intersect(w.bounds).Empty() {
27+ continue
28+ }
29+ key := shmKey{label: l.Text, typed: l.Typed, scale: w.scale}
30+ st := shmState{w: lw, h: lh, seq: 1}
31+ c := r.shmCached(key, st)
32+ if c == nil {
33+ var err error
34+ if c, err = r.shmCopyFor(key, st, pix, 0, lw*4, nil); err != nil {
35+ continue
36+ }
37+ }
38+ dr := r.contentDraw(rect, rect, lw, lh, [4]float32{}, 0, modeBuffer, true)
39+ dr.pc.color[3] = 1
40+ dr.shm = c
41+ dr.pc.buf = [4]uint32{0, uint32(lw), uint32(lh), 0}
42+ w.draws = append(w.draws, dr)
43+ }
44+}
@@ -0,0 +1,44 @@
1+package vulkan
2+
3+import (
4+ "image"
5+)
6+
7+// hintLabels draws the keyboard hint labels (ports.Scene.HintLabels) over
8+// everything, after the capture indicators. Each label is rasterized on the CPU
9+// (hintlabel) at the scene's scale and drawn from the SHM copy path: the GPU
10+// buffer is cached under a key naming the label, so a label that stays the
11+// same uploads once. Captures are rendered from scenes without labels.
12+func (w *sceneWalk) hintLabels() {
13+ if len(w.s.HintLabels) == 0 {
14+ return
15+ }
16+ r := w.r
17+ for _, l := range w.s.HintLabels {
18+ pix, lw, lh := r.labels.Label(l.Text, l.Typed, w.scale)
19+ if lw <= 0 || lh <= 0 {
20+ continue
21+ }
22+ // Inside the output, whatever the target's position.
23+ x := min(max(w.phys(l.Rect.X), w.bounds.Min.X), max(w.bounds.Max.X-lw, w.bounds.Min.X))
24+ y := min(max(w.phys(l.Rect.Y), w.bounds.Min.Y), max(w.bounds.Max.Y-lh, w.bounds.Min.Y))
25+ rect := image.Rect(x, y, x+lw, y+lh)
26+ if rect.Intersect(w.bounds).Empty() {
27+ continue
28+ }
29+ key := shmKey{label: l.Text, typed: l.Typed, scale: w.scale}
30+ st := shmState{w: lw, h: lh, seq: 1}
31+ c := r.shmCached(key, st)
32+ if c == nil {
33+ var err error
34+ if c, err = r.shmCopyFor(key, st, pix, 0, lw*4, nil); err != nil {
35+ continue
36+ }
37+ }
38+ dr := r.contentDraw(rect, rect, lw, lh, [4]float32{}, 0, modeBuffer, true)
39+ dr.pc.color[3] = 1
40+ dr.shm = c
41+ dr.pc.buf = [4]uint32{0, uint32(lw), uint32(lh), 0}
42+ w.draws = append(w.draws, dr)
43+ }
44+}
A internal/adapters/vulkan/hint_labels_test.go +97 -0
@@ -0,0 +1,97 @@
1+package vulkan
2+
3+import (
4+ "image"
5+ "testing"
6+
7+ "github.com/stretchr/testify/require"
8+
9+ "github.com/bnema/neferwl/internal/ports"
10+)
11+
12+func TestHintLabelsDrawnAfterIndicators(t *testing.T) {
13+ r, err := New(256, 256)
14+ if err != nil {
15+ t.Skipf("Vulkan unavailable: %v", err)
16+ }
17+ defer r.Close()
18+ for _, scale := range []int{1, 2} {
19+ s := ports.Scene{Scale: float64(scale),
20+ CaptureIndicators: []ports.CaptureIndicator{{Rect: ports.Rect{W: 64, H: 64}}},
21+ HintLabels: []ports.HintLabel{
22+ {Rect: ports.Rect{X: 10, Y: 12}, Text: "f"},
23+ {Rect: ports.Rect{X: 30, Y: 40}, Text: "jk", Typed: 1},
24+ }}
25+ bare := s
26+ bare.HintLabels = nil
27+ before := len(r.draws(bare, nil, newDamage(&target{}, bare, image.Rect(0, 0, 256, 256))))
28+ ds := r.draws(s, nil, newDamage(&target{}, s, image.Rect(0, 0, 256, 256)))
29+ require.Len(t, ds, before+2, "one draw per label, after the indicator's")
30+ for i, want := range []image.Point{{10 * scale, 12 * scale}, {30 * scale, 40 * scale}} {
31+ d := ds[before+i]
32+ require.NotNil(t, d.shm)
33+ require.Equal(t, int32(want.X), d.pc.rect[0])
34+ require.Equal(t, int32(want.Y), d.pc.rect[1])
35+ require.Greater(t, d.pc.rect[2], d.pc.rect[0])
36+ require.Greater(t, d.pc.rect[3], d.pc.rect[1])
37+ require.Equal(t, float32(1), d.pc.color[3])
38+ }
39+ }
40+}
41+
42+func TestHintLabelsAbsentDrawNothing(t *testing.T) {
43+ r := &Renderer{width: 64, height: 64}
44+ s := ports.Scene{Scale: 1}
45+ require.Empty(t, r.draws(s, nil, newDamage(&target{}, s, image.Rect(0, 0, 64, 64))))
46+}
47+
48+func TestHintLabelsClampedInsideOutput(t *testing.T) {
49+ r, err := New(128, 128)
50+ if err != nil {
51+ t.Skipf("Vulkan unavailable: %v", err)
52+ }
53+ defer r.Close()
54+ s := ports.Scene{Scale: 1, HintLabels: []ports.HintLabel{{Rect: ports.Rect{X: 127, Y: 127}, Text: "ff"}}}
55+ ds := r.draws(s, nil, newDamage(&target{}, s, image.Rect(0, 0, 128, 128)))
56+ require.Len(t, ds, 1)
57+ require.LessOrEqual(t, ds[0].pc.rect[2], int32(128))
58+ require.LessOrEqual(t, ds[0].pc.rect[3], int32(128))
59+}
60+
61+func TestHintLabelsRenderAndUploadOnce(t *testing.T) {
62+ r, err := New(128, 128)
63+ if err != nil {
64+ t.Skipf("Vulkan unavailable: %v", err)
65+ }
66+ defer r.Close()
67+ s := ports.Scene{Scale: 1, Seq: 1, HintLabels: []ports.HintLabel{{Rect: ports.Rect{X: 4, Y: 4}, Text: "fj"}}}
68+ require.NoError(t, render(r, s, nil))
69+ first := r.copied
70+ require.Positive(t, first)
71+ s.Seq = 2
72+ require.NoError(t, render(r, s, nil))
73+ require.Equal(t, first, r.copied, "an unchanged label is not uploaded again")
74+}
75+
76+func TestHintSceneDeltaOnlyHintLabelsChange(t *testing.T) {
77+ phys := func(r ports.Rect) image.Rectangle { return image.Rect(r.X, r.Y, r.X+r.W, r.Y+r.H) }
78+ base := ports.Scene{Scale: 1, OutputWidth: 100, OutputHeight: 100}
79+ with := base
80+ with.HintLabels = []ports.HintLabel{{Rect: ports.Rect{X: 1, Y: 1}, Text: "f"}}
81+ _, ok := sceneDelta(base, with, phys, 1<<30)
82+ require.False(t, ok, "labels appearing redraw the output")
83+ typed := with
84+ typed.HintLabels = []ports.HintLabel{{Rect: ports.Rect{X: 1, Y: 1}, Text: "f", Typed: 1}}
85+ _, ok = sceneDelta(with, typed, phys, 1<<30)
86+ require.False(t, ok)
87+ _, ok = sceneDelta(with, with, phys, 1<<30)
88+ require.True(t, ok, "same labels, no damage")
89+}
90+
91+func TestHintLabelsHeldByTarget(t *testing.T) {
92+ s := ports.Scene{Seq: 1, HintLabels: []ports.HintLabel{{Text: "f"}}}
93+ tg := &target{}
94+ tg.hold(s, &damageRegion{drawn: map[ports.WindowID]heldWindow{}})
95+ s.HintLabels[0].Text = "j"
96+ require.Equal(t, "f", tg.scene.HintLabels[0].Text, "the held scene owns its labels")
97+}
@@ -0,0 +1,97 @@
1+package vulkan
2+
3+import (
4+ "image"
5+ "testing"
6+
7+ "github.com/stretchr/testify/require"
8+
9+ "github.com/bnema/neferwl/internal/ports"
10+)
11+
12+func TestHintLabelsDrawnAfterIndicators(t *testing.T) {
13+ r, err := New(256, 256)
14+ if err != nil {
15+ t.Skipf("Vulkan unavailable: %v", err)
16+ }
17+ defer r.Close()
18+ for _, scale := range []int{1, 2} {
19+ s := ports.Scene{Scale: float64(scale),
20+ CaptureIndicators: []ports.CaptureIndicator{{Rect: ports.Rect{W: 64, H: 64}}},
21+ HintLabels: []ports.HintLabel{
22+ {Rect: ports.Rect{X: 10, Y: 12}, Text: "f"},
23+ {Rect: ports.Rect{X: 30, Y: 40}, Text: "jk", Typed: 1},
24+ }}
25+ bare := s
26+ bare.HintLabels = nil
27+ before := len(r.draws(bare, nil, newDamage(&target{}, bare, image.Rect(0, 0, 256, 256))))
28+ ds := r.draws(s, nil, newDamage(&target{}, s, image.Rect(0, 0, 256, 256)))
29+ require.Len(t, ds, before+2, "one draw per label, after the indicator's")
30+ for i, want := range []image.Point{{10 * scale, 12 * scale}, {30 * scale, 40 * scale}} {
31+ d := ds[before+i]
32+ require.NotNil(t, d.shm)
33+ require.Equal(t, int32(want.X), d.pc.rect[0])
34+ require.Equal(t, int32(want.Y), d.pc.rect[1])
35+ require.Greater(t, d.pc.rect[2], d.pc.rect[0])
36+ require.Greater(t, d.pc.rect[3], d.pc.rect[1])
37+ require.Equal(t, float32(1), d.pc.color[3])
38+ }
39+ }
40+}
41+
42+func TestHintLabelsAbsentDrawNothing(t *testing.T) {
43+ r := &Renderer{width: 64, height: 64}
44+ s := ports.Scene{Scale: 1}
45+ require.Empty(t, r.draws(s, nil, newDamage(&target{}, s, image.Rect(0, 0, 64, 64))))
46+}
47+
48+func TestHintLabelsClampedInsideOutput(t *testing.T) {
49+ r, err := New(128, 128)
50+ if err != nil {
51+ t.Skipf("Vulkan unavailable: %v", err)
52+ }
53+ defer r.Close()
54+ s := ports.Scene{Scale: 1, HintLabels: []ports.HintLabel{{Rect: ports.Rect{X: 127, Y: 127}, Text: "ff"}}}
55+ ds := r.draws(s, nil, newDamage(&target{}, s, image.Rect(0, 0, 128, 128)))
56+ require.Len(t, ds, 1)
57+ require.LessOrEqual(t, ds[0].pc.rect[2], int32(128))
58+ require.LessOrEqual(t, ds[0].pc.rect[3], int32(128))
59+}
60+
61+func TestHintLabelsRenderAndUploadOnce(t *testing.T) {
62+ r, err := New(128, 128)
63+ if err != nil {
64+ t.Skipf("Vulkan unavailable: %v", err)
65+ }
66+ defer r.Close()
67+ s := ports.Scene{Scale: 1, Seq: 1, HintLabels: []ports.HintLabel{{Rect: ports.Rect{X: 4, Y: 4}, Text: "fj"}}}
68+ require.NoError(t, render(r, s, nil))
69+ first := r.copied
70+ require.Positive(t, first)
71+ s.Seq = 2
72+ require.NoError(t, render(r, s, nil))
73+ require.Equal(t, first, r.copied, "an unchanged label is not uploaded again")
74+}
75+
76+func TestHintSceneDeltaOnlyHintLabelsChange(t *testing.T) {
77+ phys := func(r ports.Rect) image.Rectangle { return image.Rect(r.X, r.Y, r.X+r.W, r.Y+r.H) }
78+ base := ports.Scene{Scale: 1, OutputWidth: 100, OutputHeight: 100}
79+ with := base
80+ with.HintLabels = []ports.HintLabel{{Rect: ports.Rect{X: 1, Y: 1}, Text: "f"}}
81+ _, ok := sceneDelta(base, with, phys, 1<<30)
82+ require.False(t, ok, "labels appearing redraw the output")
83+ typed := with
84+ typed.HintLabels = []ports.HintLabel{{Rect: ports.Rect{X: 1, Y: 1}, Text: "f", Typed: 1}}
85+ _, ok = sceneDelta(with, typed, phys, 1<<30)
86+ require.False(t, ok)
87+ _, ok = sceneDelta(with, with, phys, 1<<30)
88+ require.True(t, ok, "same labels, no damage")
89+}
90+
91+func TestHintLabelsHeldByTarget(t *testing.T) {
92+ s := ports.Scene{Seq: 1, HintLabels: []ports.HintLabel{{Text: "f"}}}
93+ tg := &target{}
94+ tg.hold(s, &damageRegion{drawn: map[ports.WindowID]heldWindow{}})
95+ s.HintLabels[0].Text = "j"
96+ require.Equal(t, "f", tg.scene.HintLabels[0].Text, "the held scene owns its labels")
97+}
M internal/adapters/vulkan/renderer.go +3 -0
@@ -9,6 +9,7 @@ import (
99 "strconv"
1010 "unsafe"
1111
12+ "github.com/bnema/neferwl/internal/adapters/hintlabel"
1213 "github.com/bnema/neferwl/internal/ports"
1314 "github.com/bnema/zerowrap"
1415
@@ -85,6 +86,8 @@ type Renderer struct {
8586 // the bytes copied into them, for tests.
8687 shm map[shmKey]*shmSurface
8788 copied int
89+ // labels rasterizes the hint labels (hint_labels.go); the zero value works.
90+ labels hintlabel.Raster
8891 // shmRects is scratch for shmCopyFor.
8992 shmRects []image.Rectangle
9093 // marks date the frames for Trim's wall-clock eviction (trim.go).
@@ -9,6 +9,7 @@ import (
9 "strconv"9 "strconv"
10 "unsafe"10 "unsafe"
11 11
12+ "github.com/bnema/neferwl/internal/adapters/hintlabel"
12 "github.com/bnema/neferwl/internal/ports"13 "github.com/bnema/neferwl/internal/ports"
13 "github.com/bnema/zerowrap"14 "github.com/bnema/zerowrap"
14 15
@@ -85,6 +86,8 @@ type Renderer struct {
85 // the bytes copied into them, for tests.86 // the bytes copied into them, for tests.
86 shm map[shmKey]*shmSurface87 shm map[shmKey]*shmSurface
87 copied int88 copied int
89+ // labels rasterizes the hint labels (hint_labels.go); the zero value works.
90+ labels hintlabel.Raster
88 // shmRects is scratch for shmCopyFor.91 // shmRects is scratch for shmCopyFor.
89 shmRects []image.Rectangle92 shmRects []image.Rectangle
90 // marks date the frames for Trim's wall-clock eviction (trim.go).93 // marks date the frames for Trim's wall-clock eviction (trim.go).
M internal/adapters/vulkan/scene.go +4 -3
@@ -60,6 +60,7 @@ func (r *Renderer) draws(s ports.Scene, contents map[ports.WindowID]ports.Surfac
6060 w.popups(true)
6161 w.focusFlash()
6262 w.captureIndicators()
63+ w.hintLabels()
6364 r.scratchDraws = w.draws
6465 return w.draws
6566 }
@@ -331,14 +332,14 @@ func (w *sceneWalk) place(id ports.WindowID, content *ports.SurfaceContent, x, y
331332 for i := range content.Children {
332333 if ch := &content.Children[i]; ch.Below {
333334 cx, cy := at(ch)
334- w.surface(&ch.SurfaceContent, cx, cy, clip, shmKey{id, ch.Surface}, ch.Version, covers, false)
335+ w.surface(&ch.SurfaceContent, cx, cy, clip, shmKey{win: id, surface: ch.Surface}, ch.Version, covers, false)
335336 }
336337 }
337- w.surface(content, ox, oy, clip, shmKey{id, content.Surface}, content.Version, covers, true)
338+ w.surface(content, ox, oy, clip, shmKey{win: id, surface: content.Surface}, content.Version, covers, true)
338339 for i := range content.Children {
339340 if ch := &content.Children[i]; !ch.Below {
340341 cx, cy := at(ch)
341- w.surface(&ch.SurfaceContent, cx, cy, clip, shmKey{id, ch.Surface}, ch.Version, nil, false)
342+ w.surface(&ch.SurfaceContent, cx, cy, clip, shmKey{win: id, surface: ch.Surface}, ch.Version, nil, false)
342343 }
343344 }
344345 }
@@ -60,6 +60,7 @@ func (r *Renderer) draws(s ports.Scene, contents map[ports.WindowID]ports.Surfac
60 w.popups(true)60 w.popups(true)
61 w.focusFlash()61 w.focusFlash()
62 w.captureIndicators()62 w.captureIndicators()
63+ w.hintLabels()
63 r.scratchDraws = w.draws64 r.scratchDraws = w.draws
64 return w.draws65 return w.draws
65 }66 }
@@ -331,14 +332,14 @@ func (w *sceneWalk) place(id ports.WindowID, content *ports.SurfaceContent, x, y
331 for i := range content.Children {332 for i := range content.Children {
332 if ch := &content.Children[i]; ch.Below {333 if ch := &content.Children[i]; ch.Below {
333 cx, cy := at(ch)334 cx, cy := at(ch)
334- w.surface(&ch.SurfaceContent, cx, cy, clip, shmKey{id, ch.Surface}, ch.Version, covers, false)335+ w.surface(&ch.SurfaceContent, cx, cy, clip, shmKey{win: id, surface: ch.Surface}, ch.Version, covers, false)
335 }336 }
336 }337 }
337- w.surface(content, ox, oy, clip, shmKey{id, content.Surface}, content.Version, covers, true)338+ w.surface(content, ox, oy, clip, shmKey{win: id, surface: content.Surface}, content.Version, covers, true)
338 for i := range content.Children {339 for i := range content.Children {
339 if ch := &content.Children[i]; !ch.Below {340 if ch := &content.Children[i]; !ch.Below {
340 cx, cy := at(ch)341 cx, cy := at(ch)
341- w.surface(&ch.SurfaceContent, cx, cy, clip, shmKey{id, ch.Surface}, ch.Version, nil, false)342+ w.surface(&ch.SurfaceContent, cx, cy, clip, shmKey{win: id, surface: ch.Surface}, ch.Version, nil, false)
342 }343 }
343 }344 }
344 }345 }
M internal/adapters/vulkan/shm.go +6 -0
M internal/adapters/vulkan/targets.go +1 -0
A internal/app/captured.go +41 -0
A internal/app/captured_test.go +72 -0
A internal/app/hintdebug.go +31 -0
M internal/app/session.go +11 -0
M internal/app/wiring.go +27 -6
M internal/core/action.go +6 -0
M internal/core/core.go +24 -1
M internal/core/drag.go +1 -1
A internal/core/hint.go +250 -0
A internal/core/hint_test.go +577 -0
M internal/core/input.go +4 -0
M internal/core/scene.go +1 -0
M internal/ports/capture_messages.go +6 -0
M internal/ports/config.go +6 -0
A internal/ports/hint_messages.go +49 -0
M internal/ports/scene.go +6 -1
M internal/ports/scene_same_test.go +3 -2

Comments0 open / 0

no comments yet.

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
l
merge request: the list of files (j k move, enter opens, esc back)
M
merge request: the merge menu; there 1–3 pick the mode, D deletes the branch, enter merges
R
merge request: mark the head read
esc
close a dialog, leave a field

Branches 25

Tags 10

↑↓ moveenter openesc close