#31 WIP: feat(hints): detect hint targets with the nefer-hints detector

state
draft
branch
feat/hint-detector → main
opened
bnema ·
updated

Summary

Hint mode (experimental, off by default) finds click targets with the local nefer-hints detector instead of AT-SPI and OCR. NeferWL captures the output, sends the frame to the detector subprocess over its binary protocol, and turns the returned boxes into labelled hints. Screen content stays on the host.

Change

  • Detector subprocess: started on first scan, ready handshake, per-request deadline, process group killed and reaped on timeout or cancellation; a response that races the deadline is rejected and the process replaced.
  • Detector output is untrusted: boxes are validated (finite, scored in [0,1], non-empty), clamped to the frame, capped to a work budget, and duplicate or nested boxes are merged.
  • Wide outputs are tiled into 16:9 regions; boxes cut by a tile edge are dropped in favour of the neighbouring tile.
  • Labels never overlap: higher scores win; reloading hint.size cancels shown hints.
  • Hint clicks are sent without held modifiers.
  • hint.threshold defaults to the model's own threshold (model.json in nefer-hints).
  • Each scan logs regions, raw and prepared box counts, and timings; log.debug accepts hints.
  • AT-SPI and OCR target sources are removed.

Model

Tested with nefer-hints terminal-v2 (experimental). It ignores terminal text except links (98% link recall, under 0.5 false boxes per screen) and finds more web and native GTK targets than the stock OmniParser model, but it fails the nefer-hints release gate on GUI precision (57% vs 64% at matched settings). See the nefer-hints MR for the evaluation.

Checks

make check, make race, make mocks-check, staticcheck ./...; unit tests for the detector lifecycle (including deadline races with synctest), box validation, tiling, merging, label spreading and reload; headless smoke run. Wrap-up review done; its 3 findings are fixed and re-reviewed.

Not covered end to end: changing hint.size while hints are shown (unit-tested only).

CommitSubjectAuthorDateSignatureStatus
c706457docs(hints): clarify size changes on reloadbriceGPG Verified
68ee891fix(hints): log raw and prepared detection counts separatelybriceGPG Verified
4b3bafffix(core): cancel hints when reloading label sizebriceGPG Verified
8f581ccfix(hints): discard detector processes interrupted at response boundariesbriceGPG Verified
1ade7b5fix(hints): bound detector work and account for clamped labelsbriceGPG Verified
6fafd11feat(hints): use the model's threshold when hint.threshold is unsetbriceGPG Verified
0e762e1feat(hints): log regions and raw boxes per scanbriceGPG Verified
400321dfeat(hints): tile wide outputs into 16:9 regionsbriceGPG Verified
ab2b2affix(core): keep hint labels from overlappingbriceGPG Verified
8a43081fix(hints): merge duplicate and nested detector boxesbriceGPG Verified
ea2f35efix(hints): validate and clamp detector boxes to the framebriceGPG Verified
e8e812dfix(hints): kill the detector's process group on timeoutbriceGPG Verified
88edc52fix(core): send hint clicks without held modifiersbriceGPG Verified
492a5a9fix(config): accept hints in log.debugbriceGPG Verified
03202abdocs(config): describe the hint detectorbriceGPG Verified
2b7a35ffeat(hints): detect targets with the nefer-hints subprocessbriceGPG Verified

Checks 1

note: a failing or unfinished check blocks the merge.

50 files · +2126 -4857
1/1
M .mockery.yml +2 -19
@@ -138,28 +138,11 @@ packages:
138 filename: proc_tree_mock_test.go138 filename: proc_tree_mock_test.go
139 github.com/bnema/neferwl/internal/adapters/hints:139 github.com/bnema/neferwl/internal/adapters/hints:
140 interfaces:140 interfaces:
141- languageRunner:141+ detector:
142 config:142 config:
143 dir: internal/adapters/hints143 dir: internal/adapters/hints
144 pkgname: hints144 pkgname: hints
145- filename: language_runner_mock_test.go145+ filename: detector_mock_test.go
146- targetSource:
147- config:
148- dir: internal/adapters/hints
149- pkgname: hints
150- filename: target_source_mock_test.go
151- github.com/bnema/neferwl/internal/adapters/atspi:
152- interfaces:
153- transport:
154- config:
155- dir: internal/adapters/atspi
156- pkgname: atspi
157- filename: transport_mock_test.go
158- connector:
159- config:
160- dir: internal/adapters/atspi
161- pkgname: atspi
162- filename: connector_mock_test.go
163 github.com/bnema/neferwl/internal/adapters/screencast:146 github.com/bnema/neferwl/internal/adapters/screencast:
164 interfaces:147 interfaces:
165 bus:148 bus:
M README.md +1 -1
@@ -21,7 +21,7 @@ https://github.com/user-attachments/assets/1ff42ce5-47b3-4cf8-a3a7-5fc5abde4af1
21 - **Games:** direct scanout, VRR, tearing, explicit sync and [HDR](docs/config.md#hdr) give games the whole GPU instead of spending it on composition; see [Performance](docs/performance.md).21 - **Games:** direct scanout, VRR, tearing, explicit sync and [HDR](docs/config.md#hdr) give games the whole GPU instead of spending it on composition; see [Performance](docs/performance.md).
22 - **Overview:** `cmd+o` or a four-finger swipe up shows live workspace previews. Select a window with the keyboard, touchpad or mouse; see [Overview](docs/config.md#overview).22 - **Overview:** `cmd+o` or a four-finger swipe up shows live workspace previews. Select a window with the keyboard, touchpad or mouse; see [Overview](docs/config.md#overview).
23 - **Stash:** `cmd+s` toggles a floating strip of windows set aside for the current workspace; `cmd+shift+s` moves a window into or out of it. See [Stash](docs/config.md#stash).23 - **Stash:** `cmd+s` toggles a floating strip of windows set aside for the current workspace; `cmd+shift+s` moves a window into or out of it. See [Stash](docs/config.md#stash).
24-- **Hint mode:** click links and buttons by typing their labels, without reaching for the mouse. Enable experimental `hint.enable` and add a bind; see [Hint mode](docs/config.md#hint-mode).24+- **Hint mode:** click what is on screen by typing its label, without reaching for the mouse; targets come from the `nefer-hints` detector. Enable experimental `hint.enable` and add a bind; see [Hint mode](docs/config.md#hint-mode).
25 - **Animations:** short, cheap and off by default. Opt in with experimental `animations = on`; see [Configuration](docs/config.md#keys).25 - **Animations:** short, cheap and off by default. Opt in with experimental `animations = on`; see [Configuration](docs/config.md#keys).
26 26
27 Written in Go, without cgo. Each piece of state has one owner goroutine, connected to the others by channels; hot paths reuse memory and client buffers stay off the Go heap. See [Architecture](docs/dev/architecture.md) for the concurrency model, the GC strategy and the reusable Go libraries.27 Written in Go, without cgo. Each piece of state has one owner goroutine, connected to the others by channels; hot paths reuse memory and client buffers stay off the Go heap. See [Architecture](docs/dev/architecture.md) for the concurrency model, the GC strategy and the reusable Go libraries.
M docs/config.md +27 -29
@@ -60,8 +60,9 @@ See [examples/config](../examples/config) for every key with its default.
60 | `focus.flash-duration` | `200ms` | Fade duration, `50ms` to `1s`; not scaled by `animations.speed` |60 | `focus.flash-duration` | `200ms` | Fade duration, `50ms` to `1s`; not scaled by `animations.speed` |
61 | `focus.flash-color` | `border.active` | Flash color `#rrggbb`; empty uses `border.active` |61 | `focus.flash-color` | `border.active` | Flash color `#rrggbb`; empty uses `border.active` |
62 | `hint.enable` | `off` | Turn on the experimental [hint mode](#hint-mode) (restart required) |62 | `hint.enable` | `off` | Turn on the experimental [hint mode](#hint-mode) (restart required) |
63-| `hint.targets` | `clickable` | What [hint mode](#hint-mode) labels: `clickable` (links, URLs and buttons) or `links` (links and URLs only) |63+| `hint.detector` | `nefer-hints` | The [hint mode](#hint-mode) detector binary: a path, or a name looked up in `PATH` (restart required) |
64-| `hint.text-click` | `ctrl+shift` | Modifiers held for clicks on links found by OCR (terminals open links on a modified click); `none` sends a plain click |64+| `hint.model` | `$XDG_DATA_HOME/nefer-hints/models` | Directory of the detector's model files; `~/.local/share/nefer-hints/models` when `XDG_DATA_HOME` is unset; a leading `~/` is expanded (restart required) |
65+| `hint.threshold` | model default | Optional minimum detection score override, 0.01 to 0.99; unset or empty uses the model’s own threshold (`model.json`), else the runner default. Lower finds more targets and more false ones (restart required) |
65 | `hint.capture` | `off` | Show hint labels in captures requested by Wayland clients (screenshots, screencasts); the compositor's own hint scans never see them; applies on reload |66 | `hint.capture` | `off` | Show hint labels in captures requested by Wayland clients (screenshots, screencasts); the compositor's own hint scans never see them; applies on reload |
66 | `hint.size` | `auto` | Label text size in logical pixels (6 to 72), or `auto` for screen pixel density |67 | `hint.size` | `auto` | Label text size in logical pixels (6 to 72), or `auto` for screen pixel density |
67 | `hint.background` | empty | Label background (`#rrggbb`); empty uses `border.active` |68 | `hint.background` | empty | Label background (`#rrggbb`); empty uses `border.active` |
@@ -239,7 +240,7 @@ bind.XF86MonBrightnessDown = spawn brightnessctl set 5%-
239 | `toggle-window-stash` | Move the focused window to the [stash](#stash), or back to its column |240 | `toggle-window-stash` | Move the focused window to the [stash](#stash), or back to its column |
240 | `toggle-stash-visible` | Show or hide the stash |241 | `toggle-stash-visible` | Show or hide the stash |
241 | `toggle-overview` | Open or close the [overview](#overview) |242 | `toggle-overview` | Open or close the [overview](#overview) |
242-| `hint` | Label the links and buttons on the focused screen, then click the one you type ([hint mode](#hint-mode)) |243+| `hint` | Label the clickable elements on the focused screen, then click the one you type ([hint mode](#hint-mode)) |
243 | `switch-column-next` / `-prev` | Switch to the previously used column. Hold `cmd` and press again to go further back |244 | `switch-column-next` / `-prev` | Switch to the previously used column. Hold `cmd` and press again to go further back |
244 | `maximize-column` | Toggle full width for the focused column |245 | `maximize-column` | Toggle full width for the focused column |
245 | `cycle-column-width` | Grow the focused column by one [cell](#column-widths), then back to equal widths |246 | `cycle-column-width` | Grow the focused column by one [cell](#column-widths), then back to equal widths |
@@ -353,46 +354,43 @@ Hint mode is experimental and off by default: set `hint.enable = on` and restart
353 While it is off, nothing of hint mode runs or uses memory, and the `hint` bind354 While it is off, nothing of hint mode runs or uses memory, and the `hint` bind
354 logs a notice once and does nothing. A reload does not change it.355 logs a notice once and does nothing. A reload does not change it.
355 356
356-`hint` labels the links and buttons on the focused output. Type a label to357+`hint` labels the clickable elements (buttons, links, icons, fields) on the
357-click its target; BackSpace erases a letter, Escape cancels. No bind is set358+focused output. Type a label to click its target; BackSpace erases a letter,
358-by default; for example `bind.cmd+g = hint`.359+Escape cancels. No bind is set by default; for example `bind.cmd+g = hint`.
359 360
360 Labels use the `border.active` colour with a contrasting text colour, and361 Labels use the `border.active` colour with a contrasting text colour, and
361-their size follows the screen's pixel density. Each label sits just outside362+their size follows the screen's pixel density when `hint.size = auto`.
362-the top-left corner of its target. Links found by OCR (terminal text) are363+`hint.size` also accepts a text size from 6 to 72 logical pixels. Changing it
363-clicked with the `hint.text-click` modifiers held, which is how terminals open364+on reload cancels the shown labels; the next `hint` scan uses the new size.
364-a link; accessibility targets get a plain click.365+Each label sits just outside
366+the top-left corner of its target. The click is a plain left click.
365 367
366 Labels are not part of screenshots and screencasts by default. With368 Labels are not part of screenshots and screencasts by default. With
367 `hint.capture = on` they appear in the captures that Wayland clients request369 `hint.capture = on` they appear in the captures that Wayland clients request
368-(including Exclude captures); NeferWL's own text recognition never sees them.370+(including Exclude captures); the detector never sees them.
369 The capture indicator and the position hint stay out of every capture.371 The capture indicator and the position hint stay out of every capture.
370 A reload applies `hint.capture` to the next frame.372 A reload applies `hint.capture` to the next frame.
371 373
372 ### Where targets come from374 ### Where targets come from
373 375
374-| Source | Apps | Accuracy | Speed |376+Targets come from `nefer-hints`, a separate program that looks at the pixels of
375-|---|---|---|---|377+the whole output with a neural network and returns a box for each clickable
376-| Accessibility tree (AT-SPI) | Firefox, Chromium, GTK, Qt, Electron | Exact: the app reports its own links and buttons | A few milliseconds |378+element. It works the same for every app, toolkit and terminal. It sees what is
377-| Text recognition (OCR) | Everything else: terminals, apps without accessibility | Best effort: URLs, domains and underlined text written on screen | 150 ms or more, more on large screens |379+drawn: it does not know what an element does, and it can miss or invent boxes
380+(`hint.threshold` trades one for the other). Boxes under 4×4 logical pixels
381+are dropped.
378 382
379-NeferWL asks the accessibility tree first. Windows it does not cover are read383+NeferWL starts `nefer-hints` at the first hint scan (loading takes about half a
380-with OCR, which finds visible URLs, domains, email addresses and underlined384+second), keeps it running, and stops it on exit. If it dies, fails or takes
381-text. Underlined words get a plain click, and the words of one continuous385+more than 2 seconds on a scan, that scan is cancelled and a new process starts
382-underline form one target; URLs keep the `hint.text-click` modifiers. OCR386+at the next one. Its device (GPU or CPU) is logged at startup, at info level.
383-cannot see a link whose address is hidden behind plain text, and it can misread
384-small or low-contrast text.
385 387
386 ### Requirements388 ### Requirements
387 389
388-- Accessibility: NeferWL enables it lazily on the first hint scan that queries390+- The `nefer-hints` binary, found in `PATH` or named by `hint.detector`.
389- app windows. Apps started before then may need restarting. Some apps need a flag:391+- Its model files, in `hint.model`.
390- - Chromium: `--force-renderer-accessibility`392+- A Vulkan GPU for speed; without one the detector falls back to the CPU, which is slower.
391- - Electron: `ACCESSIBILITY_ENABLED=1`
392- - Qt: `QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1`
393-- OCR: `tesseract` and `tesseract-data-eng`. Without tesseract, OCR is off
394- and hint mode labels only apps that expose the accessibility tree.
395 393
396 ### Debugging hint mode394 ### Debugging hint mode
397 395
398-With `hint.enable = on` and `NEFERWL_HINT_DEBUG=1` set when NeferWL starts (for example `NEFERWL_HINT_DEBUG=1 make tty`, or the variable in the launch environment), every successful scan writes three files to `$XDG_STATE_HOME/neferwl/hints/` (`~/.local/state/neferwl/hints/` when unset): `<time>-<output>.png`, the frame as scanned; `<time>-<output>.annotated.png`, the same with each target outlined (OCR links green, accessibility targets magenta, OCR window crops grey); and `<time>-<output>.json`, the targets (with their `source`, `atspi` or `ocr`) and timings. NeferWL keeps the 50 most recent scans and logs a warning at startup. The files contain the screen in clear: delete them after use.396+With `hint.enable = on` and `NEFERWL_HINT_DEBUG=1` set when NeferWL starts (for example `NEFERWL_HINT_DEBUG=1 make tty`, or the variable in the launch environment), every successful scan writes three files to `$XDG_STATE_HOME/neferwl/hints/` (`~/.local/state/neferwl/hints/` when unset): `<time>-<output>.png`, the frame as scanned; `<time>-<output>.annotated.png`, the same with the detector boxes outlined; and `<time>-<output>.json`, the targets (rectangle and score) and the capture, detection and total timings. NeferWL keeps the 50 most recent scans and logs a warning at startup. The detector's own messages are logged at debug level (`log.debug = hints`). The files contain the screen in clear: delete them after use.
M examples/config +6 -3
@@ -98,9 +98,12 @@ focus.flash-duration = 200ms
98 focus.flash-color =98 focus.flash-color =
99 # Hint mode is experimental and off: turn it on here (restart required).99 # Hint mode is experimental and off: turn it on here (restart required).
100 # hint.enable = off100 # hint.enable = off
101-# Hint mode labels the links, URLs and buttons on the focused screen:101+# Hint mode finds its targets with the nefer-hints detector (restart required).
102-# clickable (links, URLs and buttons) or links (links and URLs only).102+# hint.detector = nefer-hints
103-# hint.targets = clickable103+# hint.model = ~/.local/share/nefer-hints/models
104+# Unset or empty uses the model threshold (model.json), else the runner default.
105+# Set 0.01..0.99 to override it.
106+# hint.threshold =
104 # Show the labels in screenshots and screencasts taken by clients.107 # Show the labels in screenshots and screencasts taken by clients.
105 # hint.capture = off108 # hint.capture = off
106 109
D internal/adapters/atspi/bus_test.go +0 -197
@@ -1,197 +0,0 @@
1-package atspi
2-
3-import (
4- "bufio"
5- "os"
6- "os/exec"
7- "strings"
8- "testing"
9- "time"
10-
11- "github.com/bnema/zerobus"
12- "github.com/stretchr/testify/require"
13-)
14-
15-// startBus starts a private dbus-daemon, so the test never touches the
16-// desktop's buses. It skips when the daemon is not installed.
17-func startBus(t *testing.T) string {
18- t.Helper()
19- bin, err := exec.LookPath("dbus-daemon")
20- if err != nil {
21- if os.Getenv("CI") != "" {
22- t.Fatal("dbus-daemon not installed")
23- }
24- t.Skip("dbus-daemon not installed")
25- }
26- cmd := exec.Command(bin, "--session", "--nofork", "--nopidfile", "--print-address=1")
27- out, err := cmd.StdoutPipe()
28- require.NoError(t, err)
29- require.NoError(t, cmd.Start())
30- t.Cleanup(func() { _ = cmd.Process.Kill(); _ = cmd.Wait() })
31- line, err := bufio.NewReader(out).ReadString('\n')
32- require.NoError(t, err)
33- return strings.TrimSpace(line)
34-}
35-
36-// TestDoDecodesMatchesPipelined runs Do against an app on a real bus: the
37-// GetMatches reply is decoded, and replies to several calls sent before any
38-// is read are matched by serial.
39-func TestDoDecodesMatchesPipelined(t *testing.T) {
40- addr := startBus(t)
41- app, err := zerobus.Dial(addr)
42- require.NoError(t, err)
43- t.Cleanup(func() { _ = app.Close() })
44- _, err = app.RequestName("org.test.App", 0)
45- require.NoError(t, err)
46-
47- done := make(chan struct{})
48- go func() {
49- defer close(done)
50- for {
51- m, err := app.ReadMessage()
52- if err != nil {
53- return
54- }
55- if m.Type != zerobus.TypeMethodCall {
56- continue
57- }
58- req := &zerobus.Message{Sender: strings.Clone(m.Sender), Serial: m.Serial}
59- switch m.Member {
60- case "GetMatches":
61- // Check the rule: struct, then sort, count, traverse.
62- r := m.Body()
63- r.Struct()
64- end := r.Array('i')
65- var states []int32
66- for r.More(end) {
67- states = append(states, r.Int32())
68- }
69- _ = r.Int32()
70- ad := r.Array('{')
71- _ = ad
72- _ = r.Int32()
73- end = r.Array('i')
74- var roles []int32
75- for r.More(end) {
76- roles = append(roles, r.Int32())
77- }
78- ok := r.Err() == nil && len(states) == 2 && states[0] == 1<<25|1<<30 && len(roles) == 5 && uint32(roles[2]) == 1<<24|1<<18|1<<31
79- if !ok {
80- app.NewError(req, "org.test.Error.BadRule", "s").Str("bad rule")
81- _, _ = app.Send()
82- continue
83- }
84- e := app.NewReply(req, "a(so)")
85- a := e.BeginArray('(')
86- for _, p := range []string{"/one", "/two"} {
87- e.Struct()
88- e.Str(":1.99")
89- e.ObjectPath(p)
90- }
91- e.EndArray(a)
92- case "GetExtents":
93- e := app.NewReply(req, "(iiii)")
94- e.Struct()
95- e.Int32(1)
96- e.Int32(2)
97- e.Int32(30)
98- e.Int32(40)
99- case "GetRole":
100- app.NewReply(req, "u").Uint32(roleLink)
101- case "GetState":
102- e := app.NewReply(req, "au")
103- a := e.BeginArray('u')
104- e.Uint32(stShowing)
105- e.Uint32(0)
106- e.EndArray(a)
107- default:
108- app.NewError(req, "org.freedesktop.DBus.Error.UnknownMethod", "s").Str("no")
109- }
110- _, _ = app.Send()
111- }
112- }()
113-
114- cc, err := zerobus.Dial(addr)
115- require.NoError(t, err)
116- b, err := newConn(cc)
117- require.NoError(t, err)
118- t.Cleanup(func() { _ = b.Close() })
119-
120- calls := []call{
121- {op: opMatches, name: "org.test.App", path: "/frame", rule: matchRule{linksOnly: true}},
122- {op: opExtents, name: "org.test.App", path: "/one", arg: coordWindow},
123- {op: opRole, name: "org.test.App", path: "/one"},
124- {op: opState, name: "org.test.App", path: "/one"},
125- {op: opChildren, name: "org.test.App", path: "/one"},
126- }
127- rep := b.Do(time.Now().Add(2*time.Second), time.Second, calls)
128- require.Len(t, rep, 5)
129- require.NoError(t, rep[0].err)
130- require.Equal(t, []ref{{":1.99", "/one"}, {":1.99", "/two"}}, rep[0].refs)
131- require.NoError(t, rep[1].err)
132- require.Equal(t, [4]int32{1, 2, 30, 40}, rep[1].rect)
133- require.Equal(t, uint32(roleLink), rep[2].u)
134- require.Equal(t, uint64(stShowing), rep[3].state)
135- var be *busError
136- require.ErrorAs(t, rep[4].err, &be)
137- require.Equal(t, "org.freedesktop.DBus.Error.UnknownMethod", be.name)
138-
139- // A call to a name nobody owns fails fast, others still succeed.
140- rep = b.Do(time.Now().Add(2*time.Second), time.Second, []call{
141- {op: opRole, name: "org.test.Nobody", path: "/x"},
142- {op: opRole, name: "org.test.App", path: "/x"},
143- })
144- require.Error(t, rep[0].err)
145- require.NoError(t, rep[1].err)
146-}
147-
148-func TestDoTimesOutOnSilentApp(t *testing.T) {
149- addr := startBus(t)
150- app, err := zerobus.Dial(addr)
151- require.NoError(t, err)
152- t.Cleanup(func() { _ = app.Close() })
153- _, err = app.RequestName("org.test.Mute", 0)
154- require.NoError(t, err)
155-
156- cc, err := zerobus.Dial(addr)
157- require.NoError(t, err)
158- b, err := newConn(cc)
159- require.NoError(t, err)
160- t.Cleanup(func() { _ = b.Close() })
161-
162- t0 := time.Now()
163- rep := b.Do(time.Now().Add(time.Second), 50*time.Millisecond, []call{{op: opRole, name: "org.test.Mute", path: "/x"}})
164- require.ErrorIs(t, rep[0].err, ErrTimeout)
165- require.Less(t, time.Since(t0), 500*time.Millisecond)
166-}
167-
168-// A call left unanswered for idle gets ErrTimeout without breaking the
169-// connection; a batch that hits its deadline breaks it, and Do returns near
170-// the deadline even though the idle timeout is longer.
171-func TestDoDeadlineBreaksConnection(t *testing.T) {
172- addr := startBus(t)
173- app, err := zerobus.Dial(addr)
174- require.NoError(t, err)
175- t.Cleanup(func() { _ = app.Close() })
176- _, err = app.RequestName("org.test.Mute", 0)
177- require.NoError(t, err)
178-
179- cc, err := zerobus.Dial(addr)
180- require.NoError(t, err)
181- b, err := newConn(cc)
182- require.NoError(t, err)
183- t.Cleanup(func() { _ = b.Close() })
184-
185- mute := []call{{op: opRole, name: "org.test.Mute", path: "/x"}}
186- rep := b.Do(time.Now().Add(time.Second), 40*time.Millisecond, mute)
187- require.ErrorIs(t, rep[0].err, ErrTimeout)
188- require.NotErrorIs(t, rep[0].err, ErrBroken)
189-
190- t0 := time.Now()
191- rep = b.Do(time.Now().Add(60*time.Millisecond), time.Second, mute)
192- require.ErrorIs(t, rep[0].err, ErrBroken)
193- require.Less(t, time.Since(t0), 300*time.Millisecond)
194- // The socket is closed: the next batch fails at once.
195- rep = b.Do(time.Now().Add(time.Second), time.Second, mute)
196- require.ErrorIs(t, rep[0].err, ErrBroken)
197-}
D internal/adapters/atspi/connector_mock_test.go +0 -109
@@ -1,109 +0,0 @@
1-// Code generated by mockery; DO NOT EDIT.
2-// github.com/vektra/mockery
3-// template: testify
4-
5-package atspi
6-
7-import (
8- "context"
9-
10- mock "github.com/stretchr/testify/mock"
11-)
12-
13-// newMockconnector creates a new instance of mockconnector. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
14-// The first argument is typically a *testing.T value.
15-func newMockconnector(t interface {
16- mock.TestingT
17- Cleanup(func())
18-}) *mockconnector {
19- if helper, ok := t.(interface{ Helper() }); ok {
20- helper.Helper()
21- }
22-
23- mock := &mockconnector{}
24- mock.Mock.Test(t)
25-
26- t.Cleanup(func() {
27- if helper, ok := t.(interface{ Helper() }); ok {
28- helper.Helper()
29- }
30- mock.AssertExpectations(t)
31- })
32-
33- return mock
34-}
35-
36-// mockconnector is an autogenerated mock type for the connector type
37-type mockconnector struct {
38- mock.Mock
39-}
40-
41-type mockconnector_Expecter struct {
42- mock *mock.Mock
43-}
44-
45-func (_m *mockconnector) EXPECT() *mockconnector_Expecter {
46- return &mockconnector_Expecter{mock: &_m.Mock}
47-}
48-
49-// Connect provides a mock function for the type mockconnector
50-func (_mock *mockconnector) Connect(ctx context.Context) (transport, error) {
51- ret := _mock.Called(ctx)
52-
53- if len(ret) == 0 {
54- panic("no return value specified for Connect")
55- }
56-
57- var r0 transport
58- var r1 error
59- if returnFunc, ok := ret.Get(0).(func(context.Context) (transport, error)); ok {
60- return returnFunc(ctx)
61- }
62- if returnFunc, ok := ret.Get(0).(func(context.Context) transport); ok {
63- r0 = returnFunc(ctx)
64- } else {
65- if ret.Get(0) != nil {
66- r0 = ret.Get(0).(transport)
67- }
68- }
69- if returnFunc, ok := ret.Get(1).(func(context.Context) error); ok {
70- r1 = returnFunc(ctx)
71- } else {
72- r1 = ret.Error(1)
73- }
74- return r0, r1
75-}
76-
77-// mockconnector_Connect_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Connect'
78-type mockconnector_Connect_Call struct {
79- *mock.Call
80-}
81-
82-// Connect is a helper method to define mock.On call
83-// - ctx context.Context
84-func (_e *mockconnector_Expecter) Connect(ctx any) *mockconnector_Connect_Call {
85- return &mockconnector_Connect_Call{Call: _e.mock.On("Connect", ctx)}
86-}
87-
88-func (_c *mockconnector_Connect_Call) Run(run func(ctx context.Context)) *mockconnector_Connect_Call {
89- _c.Call.Run(func(args mock.Arguments) {
90- var arg0 context.Context
91- if args[0] != nil {
92- arg0 = args[0].(context.Context)
93- }
94- run(
95- arg0,
96- )
97- })
98- return _c
99-}
100-
101-func (_c *mockconnector_Connect_Call) Return(transportMoqParam transport, err error) *mockconnector_Connect_Call {
102- _c.Call.Return(transportMoqParam, err)
103- return _c
104-}
105-
106-func (_c *mockconnector_Connect_Call) RunAndReturn(run func(ctx context.Context) (transport, error)) *mockconnector_Connect_Call {
107- _c.Call.Return(run)
108- return _c
109-}
D internal/adapters/atspi/doc.go +0 -10
@@ -1,10 +0,0 @@
1-// Package atspi finds hint targets in the accessibility tree of the apps on
2-// screen. It talks to the AT-SPI registry over the accessibility bus with
3-// zerobus, from one connection used by one goroutine at a time, and pipelines
4-// the calls it needs: several are sent before their replies are read, and the
5-// replies are matched by serial.
6-//
7-// The approach (Collection.GetMatches on the window frame, bounded walk when
8-// the toolkit has no Collection, window-relative extents) is adapted from
9-// neru's AT-SPI client, https://github.com/y3owk1n/neru (MIT licence).
10-package atspi
D internal/adapters/atspi/model.go +0 -206
@@ -1,206 +0,0 @@
1-package atspi
2-
3-import "github.com/bnema/neferwl/internal/ports"
4-
5-// AT-SPI constants (atspi-constants.h). The role ids are the AtspiRole enum
6-// values; the state ids are bit indices of the state set.
7-const (
8- roleCheckBox = 7
9- roleComboBox = 11
10- roleDialog = 16
11- roleFrame = 23
12- roleMenuItem = 35
13- rolePageTab = 37
14- rolePushButton = 43
15- roleRadioButton = 44
16- roleToggleButton = 62
17- roleWindow = 69
18- roleLink = 88
19- roleDocumentFrame = 82
20- roleDocumentWeb = 95
21-
22- stateActive = 1
23- stateShowing = 25
24- stateVisible = 30
25-
26- roleWords = 5 // int32 words of a role set in a MatchRule
27- stateWords = 2 // int32 words of a state set in a MatchRule
28-
29- // AtspiCollectionMatchType and AtspiCollectionSortOrder values.
30- matchAll = 1
31- matchAny = 2
32- sortCanonical = 1
33- matchCount = 500
34- maxDepth = 30
35- maxWalkNodes = 3000
36- maxFrames = 32
37- sizeTolerance = 40
38- maxMatchesRead = matchCount
39-)
40-
41-// clickableRoles are the roles that become hint targets.
42-var clickableRoles = []uint32{
43- roleLink, rolePushButton, roleToggleButton, roleCheckBox,
44- roleRadioButton, roleMenuItem, rolePageTab, roleComboBox,
45-}
46-
47-// matchRule is the Collection query: showing and visible objects of the
48-// clickable roles (only links when linksOnly), plus web documents for coverage.
49-type matchRule struct{ linksOnly bool }
50-
51-func (r matchRule) roles() []uint32 {
52- if r.linksOnly {
53- return []uint32{roleLink, roleDocumentFrame, roleDocumentWeb}
54- }
55- return append(append([]uint32(nil), clickableRoles...), roleDocumentFrame, roleDocumentWeb)
56-}
57-
58-// roleBits packs role ids into the bit set of a MatchRule: bit id%32 of word
59-// id/32.
60-func roleBits(ids []uint32) []int32 {
61- words := make([]int32, roleWords)
62- for _, id := range ids {
63- if w := int(id / 32); w < len(words) {
64- words[w] |= int32(uint32(1) << (id % 32))
65- }
66- }
67- return words
68-}
69-
70-// stateBits packs state ids into the bit set of a MatchRule.
71-func stateBits(ids ...uint32) []int32 {
72- words := make([]int32, stateWords)
73- for _, id := range ids {
74- if w := int(id / 32); w < len(words) {
75- words[w] |= int32(uint32(1) << (id % 32))
76- }
77- }
78- return words
79-}
80-
81-func hasState(set uint64, state uint) bool { return set&(1<<state) != 0 }
82-
83-// isClickable reports whether role makes a target.
84-func isClickable(role uint32, linksOnly bool) bool {
85- if linksOnly {
86- return role == roleLink
87- }
88- for _, r := range clickableRoles {
89- if r == role {
90- return true
91- }
92- }
93- return false
94-}
95-
96-// isWindowRole reports whether role is a top-level window role.
97-func isWindowRole(role uint32) bool {
98- return role == roleFrame || role == roleWindow || role == roleDialog
99-}
100-
101-// kindOf maps a role to the hint kind: links are links, the rest buttons.
102-func kindOf(role uint32) ports.HintKind {
103- if role == roleLink {
104- return ports.HintLink
105- }
106- return ports.HintButton
107-}
108-
109-// frameInfo is a showing top-level window of an app.
110-type frameInfo struct {
111- ref ref
112- active bool
113- w, h int // size of the frame, from its window extents
114-}
115-
116-// matchFrames pairs the windows of one app with its frames, one to one. A
117-// window is paired with a frame when the pairing is certain:
118-// - one window and one frame;
119-// - the focused window with the one ACTIVE frame; that frame is then
120-// reserved and no other window can take it;
121-// - otherwise the one unreserved frame whose size is within sizeTolerance
122-// of the window, provided no other unmatched window claims it as well.
123-//
124-// Ambiguous windows are left out. The result maps the index of a window in
125-// wins to the index of its frame in frames.
126-func matchFrames(wins []ports.HintWindow, frames []frameInfo) map[int]int {
127- out := map[int]int{}
128- if len(wins) == 1 && len(frames) == 1 {
129- out[0] = 0
130- return out
131- }
132- // Active matches first: they are reserved against the size matches.
133- active, n := -1, 0
134- for j, f := range frames {
135- if f.active {
136- active, n = j, n+1
137- }
138- }
139- reserved := make([]bool, len(frames))
140- done := make([]bool, len(wins))
141- if n == 1 {
142- focused := -1
143- count := 0
144- for i, w := range wins {
145- if w.Focused {
146- focused, count = i, count+1
147- }
148- }
149- if count == 1 {
150- out[focused] = active
151- reserved[active] = true
152- done[focused] = true
153- }
154- }
155- claims := make([][]int, len(wins)) // size candidates per window
156- owners := make([]int, len(frames)) // windows that size-match a frame
157- for i, w := range wins {
158- if done[i] {
159- continue
160- }
161- for j, f := range frames {
162- if !reserved[j] && absInt(f.w-w.Rect.W) <= sizeTolerance && absInt(f.h-w.Rect.H) <= sizeTolerance {
163- claims[i] = append(claims[i], j)
164- owners[j]++
165- }
166- }
167- }
168- for i := range wins {
169- if !done[i] && len(claims[i]) == 1 && owners[claims[i][0]] == 1 {
170- out[i] = claims[i][0]
171- }
172- }
173- return out
174-}
175-
176-func absInt(v int) int {
177- if v < 0 {
178- return -v
179- }
180- return v
181-}
182-
183-// element is a match with its extents.
184-type element struct {
185- role uint32
186- x, y, w, h int32 // window-relative
187-}
188-
189-// toTarget converts an element of a window into a logical output-local
190-// target: the element extents are relative to the top-level frame, whose own
191-// window-relative origin is (fx, fy); the result is clipped to the window.
192-// It reports false for an empty result.
193-func toTarget(win ports.Rect, fx, fy int32, e element) (ports.HintTarget, bool) {
194- if e.w <= 0 || e.h <= 0 {
195- return ports.HintTarget{}, false
196- }
197- r := ports.Rect{X: win.X + int(e.x-fx), Y: win.Y + int(e.y-fy), W: int(e.w), H: int(e.h)}
198- x0, y0 := max(r.X, win.X), max(r.Y, win.Y)
199- x1, y1 := min(r.X+r.W, win.X+win.W), min(r.Y+r.H, win.Y+win.H)
200- if x1 <= x0 || y1 <= y0 {
201- return ports.HintTarget{}, false
202- }
203- return ports.HintTarget{Rect: ports.Rect{X: x0, Y: y0, W: x1 - x0, H: y1 - y0}, Kind: kindOf(e.role)}, true
204-}
205-
206-func isDocument(role uint32) bool { return role == roleDocumentFrame || role == roleDocumentWeb }
D internal/adapters/atspi/model_test.go +0 -117
@@ -1,117 +0,0 @@
1-package atspi
2-
3-import (
4- "testing"
5-
6- "github.com/stretchr/testify/require"
7-
8- "github.com/bnema/neferwl/internal/ports"
9-)
10-
11-func TestToTargetConvertsWindowCoordinates(t *testing.T) {
12- win := ports.Rect{X: 100, Y: 50, W: 800, H: 600}
13- // The frame origin (fx, fy) is subtracted: toolkit offsets cancel out.
14- got, ok := toTarget(win, 10, 20, element{role: roleLink, x: 110, y: 120, w: 40, h: 16})
15- require.True(t, ok)
16- require.Equal(t, ports.HintTarget{Rect: ports.Rect{X: 200, Y: 150, W: 40, H: 16}, Kind: ports.HintLink}, got)
17-}
18-
19-func TestToTargetClipsToWindow(t *testing.T) {
20- win := ports.Rect{X: 100, Y: 50, W: 200, H: 100}
21- got, ok := toTarget(win, 0, 0, element{role: rolePushButton, x: 180, y: 90, w: 60, h: 30})
22- require.True(t, ok)
23- require.Equal(t, ports.Rect{X: 280, Y: 140, W: 20, H: 10}, got.Rect)
24- require.Equal(t, ports.HintButton, got.Kind)
25-}
26-
27-func TestToTargetDropsEmptyAndOutside(t *testing.T) {
28- win := ports.Rect{X: 0, Y: 0, W: 100, H: 100}
29- for name, e := range map[string]element{
30- "zero width": {role: roleLink, x: 1, y: 1, w: 0, h: 10},
31- "neg height": {role: roleLink, x: 1, y: 1, w: 10, h: -1},
32- "outside": {role: roleLink, x: 500, y: 1, w: 10, h: 10},
33- "above": {role: roleLink, x: 1, y: -50, w: 10, h: 10},
34- "left of win": {role: roleLink, x: -30, y: 1, w: 10, h: 10},
35- } {
36- _, ok := toTarget(win, 0, 0, e)
37- require.False(t, ok, name)
38- }
39-}
40-
41-func TestRoleKindMapping(t *testing.T) {
42- require.Equal(t, ports.HintLink, kindOf(roleLink))
43- for _, r := range []uint32{rolePushButton, roleToggleButton, roleCheckBox, roleRadioButton, roleMenuItem, rolePageTab, roleComboBox} {
44- require.Equal(t, ports.HintButton, kindOf(r), r)
45- require.True(t, isClickable(r, false), r)
46- require.False(t, isClickable(r, true), r)
47- }
48- require.True(t, isClickable(roleLink, true))
49- require.False(t, isClickable(roleFrame, false))
50- require.True(t, isWindowRole(roleFrame) && isWindowRole(roleWindow) && isWindowRole(roleDialog))
51- require.False(t, isWindowRole(roleLink))
52-}
53-
54-func TestRoleAndStateBits(t *testing.T) {
55- require.Equal(t, []int32{0, 0, 0, 0, 0}, roleBits(nil))
56- // link is role 88: word 2, bit 24; push button is 43: word 1, bit 11.
57- require.Equal(t, []int32{0, 1 << 11, 1 << 24, 0, 0}, roleBits([]uint32{roleLink, rolePushButton}))
58- // showing is state 25, visible 30: both in word 0.
59- require.Equal(t, []int32{1<<25 | 1<<30, 0}, stateBits(stateShowing, stateVisible))
60-}
61-
62-func wins(ws ...ports.HintWindow) []ports.HintWindow { return ws }
63-
64-func win(id ports.WindowID, w, h int, focused bool) ports.HintWindow {
65- return ports.HintWindow{ID: id, PID: 1, Rect: ports.Rect{W: w, H: h}, Focused: focused}
66-}
67-
68-func TestMatchFramesSingle(t *testing.T) {
69- got := matchFrames(wins(win(1, 800, 600, false)), []frameInfo{{w: 100, h: 100}})
70- require.Equal(t, map[int]int{0: 0}, got)
71-}
72-
73-func TestMatchFramesActive(t *testing.T) {
74- frames := []frameInfo{{w: 800, h: 600}, {w: 800, h: 600, active: true}}
75- got := matchFrames(wins(win(1, 800, 600, true), win(2, 800, 600, false)), frames)
76- // The focused window takes the ACTIVE frame and reserves it; the other
77- // window then only has the remaining frame to match.
78- require.Equal(t, map[int]int{0: 1, 1: 0}, got)
79-}
80-
81-func TestMatchFramesSize(t *testing.T) {
82- frames := []frameInfo{{w: 1000, h: 700}, {w: 400, h: 300}}
83- got := matchFrames(wins(win(1, 1000, 700, false), win(2, 420, 290, false)), frames)
84- require.Equal(t, map[int]int{0: 0, 1: 1}, got)
85-}
86-
87-func TestMatchFramesSizeTolerance(t *testing.T) {
88- frames := []frameInfo{{w: 840, h: 640}, {w: 100, h: 100}}
89- other := win(2, 90, 90, false)
90- // 40 px off still matches; 41 px does not.
91- require.Equal(t, map[int]int{0: 0, 1: 1}, matchFrames(wins(win(1, 800, 600, false), other), frames))
92- require.Equal(t, map[int]int{1: 1}, matchFrames(wins(win(1, 799, 600, false), other), frames))
93-}
94-
95-func TestMatchFramesAmbiguous(t *testing.T) {
96- // Two windows and two frames of the same size, none active: skipped.
97- frames := []frameInfo{{w: 800, h: 600}, {w: 810, h: 600}}
98- got := matchFrames(wins(win(1, 800, 600, false), win(2, 800, 600, false)), frames)
99- require.Empty(t, got)
100- // One window, two same-size frames, not focused: skipped.
101- require.Empty(t, matchFrames(wins(win(1, 800, 600, false)), frames))
102- // No frame close enough.
103- require.Empty(t, matchFrames(wins(win(1, 200, 200, false), win(2, 300, 300, false)), frames))
104-}
105-
106-// The active frame is reserved by the focused window: an unfocused window of
107-// the same size as that frame must not be paired with it as well.
108-func TestMatchFramesOneToOne(t *testing.T) {
109- frames := []frameInfo{{ref: ref{"a", "/active"}, active: true, w: 400, h: 300}, {ref: ref{"a", "/idle"}, w: 800, h: 600}}
110- got := matchFrames(wins(win(1, 800, 600, true), win(2, 400, 300, false)), frames)
111- seen := map[int]bool{}
112- for _, f := range got {
113- require.False(t, seen[f], "frame %d assigned twice: %v", f, got)
114- seen[f] = true
115- }
116- require.Equal(t, map[int]int{0: 0}, got)
117-}
D internal/adapters/atspi/source.go +0 -481
@@ -1,481 +0,0 @@
1-package atspi
2-
3-import (
4- "context"
5- "errors"
6- "fmt"
7- "strings"
8- "time"
9-
10- "github.com/bnema/zerobus"
11-
12- "github.com/bnema/neferwl/internal/adapters/logging"
13- "github.com/bnema/neferwl/internal/ports"
14-)
15-
16-const (
17- // callTimeout is how long a batch waits for the next reply.
18- callTimeout = 100 * time.Millisecond
19- // budget is the whole time Targets may take.
20- budget = 300 * time.Millisecond
21-)
22-
23-// connectBudget bounds the whole connection setup (session bus,
24-// GetAddress, IsEnabled, dial of the accessibility bus) of one attempt.
25-const connectBudget = 300 * time.Millisecond
26-
27-// retryEvery is the minimum time between two connection attempts.
28-const retryEvery = 5 * time.Second
29-
30-// connector opens a transport on the accessibility bus.
31-type connector interface {
32- Connect(ctx context.Context) (transport, error)
33-}
34-
35-// stallWarnAfter is how long an attempt may be in flight before it is
36-// reported as a stalled accessibility bus.
37-const stallWarnAfter = 5 * time.Second
38-
39-// dialResult is the outcome of a connection attempt.
40-type dialResult struct {
41- tr transport
42- err error
43- at time.Time // when the attempt finished
44-}
45-
46-// Source reads hint targets from the accessibility tree. It owns one
47-// connection to the accessibility bus. It is NOT safe for concurrent use:
48-// it is owned by the hints worker goroutine, which calls Targets and,
49-// when it returns, Close. The connection is made on the first Targets call
50-// that needs it, never at startup, and made again after a failure, at most
51-// once every retryEvery; meanwhile Targets finds nothing and OCR takes over.
52-//
53-// At most one connection attempt (one goroutine and one socket) exists at
54-// any time: an attempt that outlives the wait of ensure stays in flight, is
55-// polled by the later Targets calls and is never doubled.
56-type Source struct {
57- tr transport
58- dial connector
59- // attempt delivers the outcome of the attempt in flight (buffered, so
60- // the attempt goroutine never blocks); nil when none runs.
61- attempt chan dialResult
62- attemptStarted time.Time
63- warnedStall bool
64- retryAt time.Time
65- closed bool
66- // wait is how long ensure waits for a new attempt; connectBudget if 0.
67- wait time.Duration
68-}
69-
70-// New returns a Source. It does not touch the bus: the connection is made by
71-// the first Targets call.
72-func New() *Source { return &Source{dial: busConnector{}} }
73-
74-// busConnector connects to the session bus, enables accessibility and dials
75-// the accessibility bus.
76-type busConnector struct{}
77-
78-func (busConnector) Connect(ctx context.Context) (transport, error) {
79- log := logging.For(ctx, "atspi")
80- // zerobus has no deadline or context API: SessionBus and Dial run the
81- // socket dial, SASL auth and Hello without a bound, and the watchdog
82- // below cannot exist before the connection does. A stall there is
83- // bounded by the single-in-flight rule of Source.ensure, not by a
84- // deadline. Follow-up: a deadline-capable zerobus API.
85- sess, err := zerobus.SessionBus()
86- if err != nil {
87- return nil, fmt.Errorf("atspi: session bus: %w", err)
88- }
89- defer func() { _ = sess.Close() }()
90- // Call has no deadline: closing the connection ends the wait for a
91- // silent service.
92- stop := time.AfterFunc(connectBudget, func() { _ = sess.Close() })
93- defer stop.Stop()
94-
95- sess.NewCall(a11yBusName, a11yBusPath, a11yBusName, "GetAddress", "")
96- m, err := sess.Call()
97- if err != nil {
98- return nil, fmt.Errorf("atspi: org.a11y.Bus GetAddress: %w", err)
99- }
100- r := m.Body()
101- addr := r.Str()
102- if r.Err() != nil || addr == "" {
103- return nil, errors.New("atspi: no accessibility bus address")
104- }
105- addr = strings.Clone(addr)
106-
107- e := sess.NewCall(a11yBusName, a11yBusPath, ifaceProps, "Set", "ssv")
108- e.Str(a11yStatus)
109- e.Str("IsEnabled")
110- e.Variant("b")
111- e.Bool(true)
112- if _, err := sess.Call(); err != nil {
113- // Toolkits that already see the bus keep working.
114- log.Warn().Err(err).Msg("could not enable accessibility")
115- } else {
116- log.Info().Msg("accessibility enabled")
117- }
118-
119- zc, err := zerobus.Dial(addr)
120- if err != nil {
121- return nil, fmt.Errorf("atspi: accessibility bus: %w", err)
122- }
123- tr, err := newConn(zc)
124- if err != nil {
125- return nil, err
126- }
127- log.Info().Msg("accessibility bus connected")
128- return tr, nil
129-}
130-
131-// ensure returns the connection, making one if none exists and the last
132-// attempt is old enough. It waits at most the budget for a new attempt; an
133-// attempt that does not answer in time stays in flight (abandoned, never
134-// doubled) and is polled without blocking by the next calls.
135-func (s *Source) ensure(ctx context.Context) transport {
136- if s.closed {
137- return nil
138- }
139- if s.tr != nil {
140- return s.tr
141- }
142- if s.dial == nil {
143- return nil
144- }
145- now := time.Now()
146- if s.attempt != nil {
147- select {
148- case r := <-s.attempt:
149- s.finish(ctx, r)
150- default:
151- if !s.warnedStall && now.Sub(s.attemptStarted) > stallWarnAfter {
152- s.warnedStall = true
153- log := logging.For(ctx, "atspi")
154- log.Warn().Dur("for", now.Sub(s.attemptStarted)).Msg("accessibility bus not answering")
155- }
156- }
157- return s.tr
158- }
159- if now.Before(s.retryAt) {
160- return nil
161- }
162- ch := make(chan dialResult, 1)
163- s.attempt = ch
164- s.attemptStarted = now
165- s.warnedStall = false
166- dial := s.dial
167- go func() {
168- tr, err := dial.Connect(ctx)
169- ch <- dialResult{tr: tr, err: err, at: time.Now()}
170- }()
171- wait := s.wait
172- if wait <= 0 {
173- wait = connectBudget
174- }
175- timer := time.NewTimer(wait)
176- defer timer.Stop()
177- select {
178- case r := <-ch:
179- s.finish(ctx, r)
180- case <-timer.C:
181- log := logging.For(ctx, "atspi")
182- log.Debug().Dur("after", wait).Msg("accessibility connect attempt abandoned in flight")
183- case <-ctx.Done():
184- }
185- return s.tr
186-}
187-
188-// finish takes the outcome of the attempt in flight: a transport is kept,
189-// an error starts the backoff from the time the attempt finished.
190-func (s *Source) finish(ctx context.Context, r dialResult) {
191- s.attempt = nil
192- if r.err != nil || r.tr == nil {
193- s.retryAt = r.at.Add(retryEvery)
194- err := r.err
195- if err == nil {
196- err = errors.New("atspi: no transport")
197- }
198- log := logging.For(ctx, "atspi")
199- log.Warn().Err(err).Msg("accessibility hint targets unavailable, using OCR only")
200- return
201- }
202- s.tr = r.tr
203-}
204-
205-// drop closes a broken connection; the next attempt reconnects.
206-func (s *Source) drop() {
207- s.retryAt = time.Now().Add(retryEvery)
208- if s.tr != nil {
209- _ = s.tr.Close()
210- s.tr = nil
211- }
212-}
213-
214-// Close closes the accessibility bus connection, and the one an attempt
215-// still in flight may deliver late. After Close, Targets finds nothing and
216-// never connects. Like Targets it is called by the owner goroutine only.
217-func (s *Source) Close() error {
218- if s.closed {
219- return nil
220- }
221- s.closed = true
222- var err error
223- if s.tr != nil {
224- err = s.tr.Close()
225- s.tr = nil
226- }
227- if ch := s.attempt; ch != nil {
228- s.attempt = nil
229- // One reaper at most: Close runs once, and one attempt exists.
230- go func() {
231- if r := <-ch; r.tr != nil {
232- _ = r.tr.Close()
233- }
234- }()
235- }
236- return err
237-}
238-
239-// appInfo is an application of the registry whose process shows a window.
240-type appInfo struct {
241- root ref
242- pid int
243-}
244-
245-// winMatch is a window with the frame it was paired with.
246-type winMatch struct {
247- win ports.HintWindow
248- frame ref
249- fx int32
250- fy int32
251-}
252-
253-// Targets returns the clickable targets of the accessibility tree for
254-// windows (logical output-local, clipped to each window) and the set of
255-// windows those targets cover. A window is covered when its tree answered
256-// with a link target or web document; uncovered windows also get OCR. It
257-// returns what is ready within the budget.
258-func (s *Source) Targets(ctx context.Context, windows []ports.HintWindow, linksOnly bool) ([]ports.HintTarget, map[ports.WindowID]bool, error) {
259- covered := map[ports.WindowID]bool{}
260- deadline := time.Now().Add(budget)
261- if d, ok := ctx.Deadline(); ok && d.Before(deadline) {
262- deadline = d
263- }
264- wanted := map[int]bool{}
265- for _, w := range windows {
266- if w.PID > 0 {
267- wanted[w.PID] = true
268- }
269- }
270- if len(wanted) == 0 {
271- return nil, covered, nil
272- }
273- tr := s.ensure(ctx)
274- if tr == nil {
275- return nil, covered, nil
276- }
277- broken := false
278- do := func(calls []call) []reply {
279- if len(calls) == 0 || ctx.Err() != nil {
280- return make([]reply, len(calls))
281- }
282- if broken {
283- out := make([]reply, len(calls))
284- for i := range out {
285- out[i].err = ErrBroken
286- }
287- return out
288- }
289- out := tr.Do(deadline, callTimeout, calls)
290- for _, r := range out {
291- if errors.Is(r.err, ErrBroken) {
292- broken = true
293- s.drop()
294- break
295- }
296- }
297- return out
298- }
299-
300- // Applications of the registry, kept when their process shows a window.
301- rep := do([]call{{op: opChildren, name: registryName, path: registryRoot}})
302- if len(rep) == 0 || rep[0].err != nil {
303- if len(rep) > 0 {
304- return nil, covered, fmt.Errorf("atspi: registry: %w", rep[0].err)
305- }
306- return nil, covered, nil
307- }
308- apps := rep[0].refs
309- calls := make([]call, len(apps))
310- for i, a := range apps {
311- calls[i] = call{op: opPID, name: a.name}
312- }
313- var kept []appInfo
314- for i, r := range do(calls) {
315- if r.err == nil && wanted[int(r.u)] {
316- kept = append(kept, appInfo{root: apps[i], pid: int(r.u)})
317- }
318- }
319-
320- // The showing windows of each kept app.
321- calls = calls[:0]
322- for _, a := range kept {
323- calls = append(calls, call{op: opChildren, name: a.root.name, path: a.root.path})
324- }
325- var cands []ref
326- var candApp []int
327- for i, r := range do(calls) {
328- if r.err != nil {
329- continue
330- }
331- for _, c := range r.refs[:min(len(r.refs), maxFrames)] {
332- cands = append(cands, c)
333- candApp = append(candApp, i)
334- }
335- }
336- calls = calls[:0]
337- for _, c := range cands {
338- calls = append(calls,
339- call{op: opRole, name: c.name, path: c.path},
340- call{op: opState, name: c.name, path: c.path},
341- call{op: opExtents, name: c.name, path: c.path, arg: coordWindow})
342- }
343- info := do(calls)
344- type frame struct {
345- frameInfo
346- pid int
347- fx, fy int32
348- }
349- var frames []frame
350- for i, c := range cands {
351- role, state, ext := info[3*i], info[3*i+1], info[3*i+2]
352- if role.err != nil || state.err != nil || ext.err != nil || !isWindowRole(role.u) || !hasState(state.state, stateShowing) {
353- continue
354- }
355- frames = append(frames, frame{
356- frameInfo: frameInfo{ref: c, active: hasState(state.state, stateActive), w: int(ext.rect[2]), h: int(ext.rect[3])},
357- pid: kept[candApp[i]].pid, fx: ext.rect[0], fy: ext.rect[1],
358- })
359- }
360-
361- // Pair the windows with the frames, process by process.
362- var matched []winMatch
363- for pid := range wanted {
364- var wins []ports.HintWindow
365- for _, w := range windows {
366- if w.PID == pid {
367- wins = append(wins, w)
368- }
369- }
370- var fs []frameInfo
371- var origin []frame
372- for _, f := range frames {
373- if f.pid == pid {
374- fs = append(fs, f.frameInfo)
375- origin = append(origin, f)
376- }
377- }
378- for wi, fi := range matchFrames(wins, fs) {
379- matched = append(matched, winMatch{win: wins[wi], frame: origin[fi].ref, fx: origin[fi].fx, fy: origin[fi].fy})
380- }
381- }
382-
383- targets := s.elements(do, matched, linksOnly, covered)
384- log := logging.For(ctx, "atspi")
385- log.Debug().Int("apps", len(kept)).Int("frames", len(frames)).Int("windows", len(matched)).Int("targets", len(targets)).Msg("atspi targets")
386- return targets, covered, nil
387-}
388-
389-// elements reads the clickable elements of the matched frames.
390-func (s *Source) elements(do func([]call) []reply, matched []winMatch, linksOnly bool, covered map[ports.WindowID]bool) []ports.HintTarget {
391- calls := make([]call, len(matched))
392- for i, m := range matched {
393- calls[i] = call{op: opMatches, name: m.frame.name, path: m.frame.path, rule: matchRule{linksOnly: linksOnly}}
394- }
395- found := make([][]ref, len(matched))
396- var walk []int // frames without Collection
397- for i, r := range do(calls) {
398- if r.err != nil {
399- var be *busError
400- if errors.As(r.err, &be) {
401- walk = append(walk, i)
402- }
403- continue
404- }
405- found[i] = r.refs
406- }
407- for _, i := range walk {
408- found[i] = s.walk(do, matched[i].frame, linksOnly)
409- }
410-
411- calls = calls[:0]
412- var owner []int
413- for i, refs := range found {
414- for _, r := range refs {
415- calls = append(calls,
416- call{op: opRole, name: r.name, path: r.path},
417- call{op: opExtents, name: r.name, path: r.path, arg: coordWindow})
418- owner = append(owner, i)
419- }
420- }
421- info := do(calls)
422- var out []ports.HintTarget
423- for k, i := range owner {
424- role, ext := info[2*k], info[2*k+1]
425- if role.err == nil && isDocument(role.u) {
426- covered[matched[i].win.ID] = true
427- continue
428- }
429- if role.err != nil || ext.err != nil || !isClickable(role.u, linksOnly) {
430- continue
431- }
432- m := matched[i]
433- t, ok := toTarget(m.win.Rect, m.fx, m.fy, element{role: role.u, x: ext.rect[0], y: ext.rect[1], w: ext.rect[2], h: ext.rect[3]})
434- if !ok {
435- continue
436- }
437- t.Window = m.win.ID
438- out = append(out, t)
439- if t.Kind == ports.HintLink {
440- covered[m.win.ID] = true
441- }
442- }
443- return out
444-}
445-
446-// walk is the fallback for toolkits without Collection: a breadth-first
447-// walk of at most maxDepth levels and maxWalkNodes nodes, returning the
448-// clickable nodes.
449-func (s *Source) walk(do func([]call) []reply, root ref, linksOnly bool) []ref {
450- var out []ref
451- level := []ref{root}
452- seen := 0
453- for depth := 0; depth < maxDepth && len(level) > 0 && seen < maxWalkNodes; depth++ {
454- calls := make([]call, 0, 3*len(level))
455- for _, n := range level {
456- calls = append(calls,
457- call{op: opChildren, name: n.name, path: n.path},
458- call{op: opRole, name: n.name, path: n.path},
459- call{op: opState, name: n.name, path: n.path})
460- }
461- rep := do(calls)
462- room := max(0, maxWalkNodes-seen-len(level))
463- next := make([]ref, 0, room)
464- for i, n := range level {
465- ch, role, state := rep[3*i], rep[3*i+1], rep[3*i+2]
466- if state.err == nil && !hasState(state.state, stateShowing) {
467- continue
468- }
469- if role.err == nil && (isClickable(role.u, linksOnly) || isDocument(role.u)) && state.err == nil && hasState(state.state, stateVisible) {
470- out = append(out, n)
471- }
472- if ch.err == nil {
473- take := min(len(ch.refs), room-len(next))
474- next = append(next, ch.refs[:take]...)
475- }
476- }
477- seen += len(level)
478- level = next
479- }
480- return out
481-}
D internal/adapters/atspi/source_test.go +0 -355
@@ -1,355 +0,0 @@
1-package atspi
2-
3-import (
4- "context"
5- "errors"
6- "fmt"
7- "runtime"
8- "testing"
9- "time"
10-
11- "github.com/stretchr/testify/mock"
12- "github.com/stretchr/testify/require"
13-
14- "github.com/bnema/neferwl/internal/ports"
15-)
16-
17-const (
18- stActive = 1 << stateActive
19- stShowing = 1 << stateShowing
20-)
21-
22-// tree answers calls like an app: registry with one app (:1.5, pid 77) with
23-// one showing frame whose links sit at window-relative extents.
24-func answerTree(frameExt [4]int32, matches []ref, ext map[string][4]int32, roles map[string]uint32, noCollection bool) func(time.Time, time.Duration, []call) []reply {
25- return func(_ time.Time, _ time.Duration, calls []call) []reply {
26- out := make([]reply, len(calls))
27- for i, c := range calls {
28- switch {
29- case c.op == opChildren && c.name == registryName:
30- out[i].refs = []ref{{":1.5", "/root"}, {":1.9", "/root"}}
31- case c.op == opPID:
32- out[i].u = map[string]uint32{":1.5": 77, ":1.9": 99}[c.name]
33- case c.op == opChildren && c.path == "/root":
34- out[i].refs = []ref{{":1.5", "/frame"}}
35- case c.op == opRole && c.path == "/frame":
36- out[i].u = roleFrame
37- case c.op == opState && c.path == "/frame":
38- out[i].state = stShowing | stActive
39- case c.op == opExtents && c.path == "/frame":
40- out[i].rect = frameExt
41- case c.op == opMatches:
42- if noCollection {
43- out[i].err = &busError{name: "org.freedesktop.DBus.Error.UnknownMethod"}
44- } else {
45- out[i].refs = matches
46- }
47- case c.op == opChildren && c.path == "/frame":
48- out[i].refs = matches
49- case c.op == opChildren:
50- case c.op == opRole:
51- out[i].u = roles[c.path]
52- case c.op == opState:
53- out[i].state = stShowing | 1<<stateVisible
54- case c.op == opExtents:
55- out[i].rect = ext[c.path]
56- }
57- }
58- return out
59- }
60-}
61-
62-func TestTargetsCoversWindowAndConverts(t *testing.T) {
63- tr := newMocktransport(t)
64- matches := []ref{{":1.5", "/a"}, {":1.5", "/b"}, {":1.5", "/zero"}}
65- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(answerTree(
66- [4]int32{4, 30, 800, 600}, matches,
67- map[string][4]int32{"/a": {14, 130, 50, 16}, "/b": {104, 230, 20, 20}, "/zero": {5, 5, 0, 0}},
68- map[string]uint32{"/a": roleLink, "/b": rolePushButton, "/zero": roleLink}, false))
69- s := &Source{tr: tr}
70-
71- windows := []ports.HintWindow{
72- {ID: 1, PID: 77, Rect: ports.Rect{X: 100, Y: 200, W: 800, H: 600}, Focused: true},
73- {ID: 2, PID: 5, Rect: ports.Rect{X: 0, Y: 0, W: 50, H: 50}},
74- }
75- targets, covered, err := s.Targets(context.Background(), windows, false)
76- require.NoError(t, err)
77- require.Equal(t, map[ports.WindowID]bool{1: true}, covered)
78- require.Equal(t, []ports.HintTarget{
79- {Rect: ports.Rect{X: 110, Y: 300, W: 50, H: 16}, Kind: ports.HintLink, Window: 1},
80- {Rect: ports.Rect{X: 200, Y: 400, W: 20, H: 20}, Kind: ports.HintButton, Window: 1},
81- }, targets)
82-}
83-
84-func TestTargetsLinksOnlyDropsButtons(t *testing.T) {
85- tr := newMocktransport(t)
86- matches := []ref{{":1.5", "/a"}, {":1.5", "/b"}}
87- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(answerTree(
88- [4]int32{0, 0, 800, 600}, matches,
89- map[string][4]int32{"/a": {14, 130, 50, 16}, "/b": {104, 230, 20, 20}},
90- map[string]uint32{"/a": roleLink, "/b": rolePushButton}, false))
91- s := &Source{tr: tr}
92- targets, _, err := s.Targets(context.Background(), []ports.HintWindow{{ID: 1, PID: 77, Rect: ports.Rect{W: 800, H: 600}}}, true)
93- require.NoError(t, err)
94- require.Len(t, targets, 1)
95- require.Equal(t, ports.HintLink, targets[0].Kind)
96-}
97-
98-func TestTargetsWalkWithoutCollection(t *testing.T) {
99- tr := newMocktransport(t)
100- matches := []ref{{":1.5", "/a"}}
101- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(answerTree(
102- [4]int32{0, 0, 800, 600}, matches,
103- map[string][4]int32{"/a": {14, 130, 50, 16}},
104- map[string]uint32{"/a": roleLink}, true))
105- s := &Source{tr: tr}
106- targets, covered, err := s.Targets(context.Background(), []ports.HintWindow{{ID: 1, PID: 77, Rect: ports.Rect{W: 800, H: 600}}}, false)
107- require.NoError(t, err)
108- require.Len(t, targets, 1)
109- require.True(t, covered[1])
110-}
111-
112-func TestTargetsNoPIDMatchCoversNothing(t *testing.T) {
113- tr := newMocktransport(t)
114- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(answerTree([4]int32{}, nil, nil, nil, false))
115- s := &Source{tr: tr}
116- targets, covered, err := s.Targets(context.Background(), []ports.HintWindow{{ID: 1, PID: 1234, Rect: ports.Rect{W: 8, H: 6}}}, false)
117- require.NoError(t, err)
118- require.Empty(t, targets)
119- require.Empty(t, covered)
120-}
121-
122-func TestTargetsWithoutPIDsAsksNothing(t *testing.T) {
123- s := &Source{tr: newMocktransport(t)}
124- _, covered, err := s.Targets(context.Background(), []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: 8, H: 6}}}, false)
125- require.NoError(t, err)
126- require.Empty(t, covered)
127-}
128-
129-func TestCoverageRequiresLinkOrDocument(t *testing.T) {
130- for _, collection := range []bool{true, false} {
131- for _, role := range []uint32{rolePushButton, roleDocumentFrame, roleDocumentWeb, roleLink} {
132- t.Run(fmt.Sprintf("collection=%t/role=%d", collection, role), func(t *testing.T) {
133- tr := newMocktransport(t)
134- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(answerTree(
135- [4]int32{0, 0, 800, 600}, []ref{{":1.5", "/a"}}, map[string][4]int32{"/a": {10, 20, 30, 40}}, map[string]uint32{"/a": role}, !collection))
136- s := &Source{tr: tr}
137- targets, covered, err := s.Targets(context.Background(), []ports.HintWindow{{ID: 1, PID: 77, Rect: ports.Rect{W: 800, H: 600}}}, false)
138- require.NoError(t, err)
139- require.Equal(t, role != rolePushButton, covered[1])
140- if isDocument(role) {
141- require.Empty(t, targets)
142- } else {
143- require.Len(t, targets, 1)
144- }
145- })
146- }
147- }
148-}
149-
150-var oneWindow = []ports.HintWindow{{ID: 1, PID: 77, Rect: ports.Rect{W: 800, H: 600}}}
151-
152-// New does not touch the bus; the first Targets call connects, once.
153-func TestConnectsLazilyOnFirstTargets(t *testing.T) {
154- tr := newMocktransport(t)
155- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(answerTree(
156- [4]int32{0, 0, 800, 600}, []ref{{":1.5", "/a"}}, map[string][4]int32{"/a": {1, 1, 9, 9}}, map[string]uint32{"/a": roleLink}, false))
157- c := newMockconnector(t)
158- c.EXPECT().Connect(mock.Anything).Return(tr, nil).Once()
159- s := &Source{dial: c}
160- for range 2 {
161- targets, _, err := s.Targets(context.Background(), oneWindow, false)
162- require.NoError(t, err)
163- require.Len(t, targets, 1)
164- }
165-}
166-
167-// Without PIDs nothing is asked and nothing is connected.
168-func TestNoConnectWithoutWindows(t *testing.T) {
169- s := &Source{dial: newMockconnector(t)}
170- _, _, err := s.Targets(context.Background(), []ports.HintWindow{{ID: 1}}, false)
171- require.NoError(t, err)
172-}
173-
174-// A failed connection is retried no sooner than retryEvery; meanwhile no
175-// targets are returned.
176-func TestConnectRetriesWithBackoff(t *testing.T) {
177- c := newMockconnector(t)
178- c.EXPECT().Connect(mock.Anything).Return(nil, errors.New("down")).Once()
179- s := &Source{dial: c}
180- for range 3 {
181- targets, covered, err := s.Targets(context.Background(), oneWindow, false)
182- require.NoError(t, err)
183- require.Empty(t, targets)
184- require.Empty(t, covered)
185- }
186- tr := newMocktransport(t)
187- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(answerTree(
188- [4]int32{0, 0, 800, 600}, []ref{{":1.5", "/a"}}, map[string][4]int32{"/a": {1, 1, 9, 9}}, map[string]uint32{"/a": roleLink}, false))
189- c.EXPECT().Connect(mock.Anything).Return(tr, nil).Once()
190- s.retryAt = time.Now().Add(-time.Second)
191- targets, _, err := s.Targets(context.Background(), oneWindow, false)
192- require.NoError(t, err)
193- require.Len(t, targets, 1)
194-}
195-
196-// A broken connection is closed and dropped; the next attempt reconnects.
197-func TestBrokenConnectionIsDropped(t *testing.T) {
198- tr := newMocktransport(t)
199- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(
200- func(_ time.Time, _ time.Duration, calls []call) []reply {
201- out := make([]reply, len(calls))
202- for i := range out {
203- out[i].err = ErrBroken
204- }
205- return out
206- }).Once()
207- tr.EXPECT().Close().Return(nil).Once()
208- c := newMockconnector(t)
209- c.EXPECT().Connect(mock.Anything).Return(tr, nil).Once()
210- s := &Source{dial: c}
211- _, _, err := s.Targets(context.Background(), oneWindow, false)
212- require.Error(t, err)
213- require.Nil(t, s.tr)
214- targets, _, err := s.Targets(context.Background(), oneWindow, false)
215- require.NoError(t, err)
216- require.Empty(t, targets) // backoff: no new attempt yet
217-}
218-
219-const testWait = 20 * time.Millisecond
220-
221-// blockedConnector returns a connector whose Connect blocks until release is
222-func blockedConnector(t *testing.T, release <-chan struct{}, tr transport, err error, times int) *mockconnector {
223- t.Helper()
224- c := newMockconnector(t)
225- c.EXPECT().Connect(mock.Anything).RunAndReturn(func(context.Context) (transport, error) {
226- <-release
227- return tr, err
228- }).Times(times)
229- return c
230-}
231-
232-// A silent service cannot hold Targets beyond the wait, is not doubled while
233-// in flight, and its late transport is used.
234-func TestConnectIsBoundedAndNotDoubled(t *testing.T) {
235- release := make(chan struct{})
236- tr := newMocktransport(t)
237- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(answerTree(
238- [4]int32{0, 0, 800, 600}, []ref{{":1.5", "/a"}}, map[string][4]int32{"/a": {1, 1, 9, 9}}, map[string]uint32{"/a": roleLink}, false))
239- c := blockedConnector(t, release, tr, nil, 1)
240- s := &Source{dial: c, wait: testWait}
241- t0 := time.Now()
242- targets, _, err := s.Targets(context.Background(), oneWindow, false)
243- require.NoError(t, err)
244- require.Empty(t, targets)
245- require.Less(t, time.Since(t0), testWait+200*time.Millisecond)
246- require.NotNil(t, s.attempt)
247- // Still in flight: no second Connect (Times(1)), and no blocking.
248- targets, _, err = s.Targets(context.Background(), oneWindow, false)
249- require.NoError(t, err)
250- require.Empty(t, targets)
251- // It connects late: the next call uses it.
252- close(release)
253- require.Eventually(t, func() bool { return len(s.attempt) == 1 }, time.Second, time.Millisecond)
254- targets, _, err = s.Targets(context.Background(), oneWindow, false)
255- require.NoError(t, err)
256- require.Len(t, targets, 1)
257- require.Nil(t, s.attempt)
258-}
259-
260-// An abandoned attempt that fails applies the backoff, then one new attempt.
261-func TestAbandonedAttemptErrorBacksOff(t *testing.T) {
262- release := make(chan struct{})
263- tr := newMocktransport(t)
264- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(answerTree(
265- [4]int32{0, 0, 800, 600}, []ref{{":1.5", "/a"}}, map[string][4]int32{"/a": {1, 1, 9, 9}}, map[string]uint32{"/a": roleLink}, false))
266- c := newMockconnector(t)
267- c.EXPECT().Connect(mock.Anything).RunAndReturn(func(context.Context) (transport, error) {
268- <-release
269- return nil, errors.New("down")
270- }).Once()
271- c.EXPECT().Connect(mock.Anything).Return(tr, nil).Once()
272- s := &Source{dial: c, wait: testWait}
273- targets, _, err := s.Targets(context.Background(), oneWindow, false)
274- require.NoError(t, err)
275- require.Empty(t, targets)
276- close(release)
277- require.Eventually(t, func() bool { return len(s.attempt) == 1 }, time.Second, time.Millisecond)
278- // The error is collected; the backoff runs from its finish time.
279- targets, _, err = s.Targets(context.Background(), oneWindow, false)
280- require.NoError(t, err)
281- require.Empty(t, targets)
282- require.Nil(t, s.attempt)
283- require.WithinDuration(t, time.Now().Add(retryEvery), s.retryAt, time.Second)
284- targets, _, err = s.Targets(context.Background(), oneWindow, false) // backoff: no Connect
285- require.NoError(t, err)
286- require.Empty(t, targets)
287- s.retryAt = time.Now().Add(-time.Second)
288- targets, _, err = s.Targets(context.Background(), oneWindow, false)
289- require.NoError(t, err)
290- require.Len(t, targets, 1)
291-}
292-
293-// Close with an attempt in flight closes the late transport; afterwards
294-// Targets finds nothing and never connects.
295-func TestCloseReapsLateTransport(t *testing.T) {
296- release := make(chan struct{})
297- closed := make(chan struct{})
298- tr := newMocktransport(t)
299- tr.EXPECT().Close().RunAndReturn(func() error { close(closed); return nil }).Once()
300- c := blockedConnector(t, release, tr, nil, 1)
301- s := &Source{dial: c, wait: testWait}
302- _, _, err := s.Targets(context.Background(), oneWindow, false)
303- require.NoError(t, err)
304- require.NoError(t, s.Close())
305- close(release)
306- select {
307- case <-closed:
308- case <-time.After(5 * time.Second):
309- t.Fatal("late transport not closed")
310- }
311- targets, covered, err := s.Targets(context.Background(), oneWindow, false)
312- require.NoError(t, err)
313- require.Empty(t, targets)
314- require.Empty(t, covered)
315- require.NoError(t, s.Close()) // idempotent
316-}
317-
318-func TestWalkWideTreeStaysWithinBudget(t *testing.T) {
319- tr := newMocktransport(t)
320- children := make([]ref, 4096)
321- for i := range children {
322- children[i] = ref{":1.5", fmt.Sprintf("/node%d", i)}
323- }
324- visited := 0
325- tr.EXPECT().Do(mock.Anything, mock.Anything, mock.Anything).RunAndReturn(
326- func(_ time.Time, _ time.Duration, calls []call) []reply {
327- visited += len(calls) / 3
328- require.LessOrEqual(t, visited, maxWalkNodes)
329- rep := make([]reply, len(calls))
330- for i, c := range calls {
331- switch c.op {
332- case opChildren:
333- rep[i].refs = children
334- case opRole:
335- rep[i].u = roleLink
336- case opState:
337- rep[i].state = stShowing | 1<<stateVisible
338- }
339- }
340- return rep
341- }).Twice()
342- s := &Source{tr: tr}
343- // A full second level replies with another 4096 children each. There is
344- // no remaining budget, so none of those millions of refs may be copied.
345- var out []ref
346- var before, after runtime.MemStats
347- runtime.ReadMemStats(&before)
348- out = s.walk(func(calls []call) []reply { return tr.Do(time.Time{}, time.Second, calls) }, ref{":1.5", "/frame"}, false)
349- runtime.ReadMemStats(&after)
350- require.Len(t, out, maxWalkNodes)
351- require.Equal(t, maxWalkNodes, visited)
352- // Normal replies and call batches take only a few MB; the old append-
353- // then-truncate path copied hundreds of MB for this tree.
354- require.Less(t, after.TotalAlloc-before.TotalAlloc, uint64(32<<20))
355-}
D internal/adapters/atspi/transport.go +0 -362
@@ -1,362 +0,0 @@
1-package atspi
2-
3-import (
4- "errors"
5- "fmt"
6- "strings"
7- "sync/atomic"
8- "syscall"
9- "time"
10-
11- "github.com/bnema/zerobus"
12- "golang.org/x/sys/unix"
13-)
14-
15-// D-Bus names, paths and interfaces.
16-const (
17- busDaemon = "org.freedesktop.DBus"
18- busDaemonPath = "/org/freedesktop/DBus"
19- ifaceProps = "org.freedesktop.DBus.Properties"
20-
21- a11yBusName = "org.a11y.Bus"
22- a11yBusPath = "/org/a11y/bus"
23- a11yStatus = "org.a11y.Status"
24- registryName = "org.a11y.atspi.Registry"
25- registryRoot = "/org/a11y/atspi/accessible/root"
26-
27- ifaceAccessible = "org.a11y.atspi.Accessible"
28- ifaceComponent = "org.a11y.atspi.Component"
29- ifaceCollection = "org.a11y.atspi.Collection"
30-
31- // coordWindow is ATSPI_COORD_TYPE_WINDOW: extents relative to the
32- // top-level window.
33- coordWindow = 1
34-
35- // maxInflight bounds the calls sent before a reply is read, so the bus
36- // never waits on a full socket while this side waits on it.
37- maxInflight = 64
38- // maxRefs bounds the objects read from one reply.
39- maxRefs = 4096
40-)
41-
42-// op is a call the source makes.
43-type op uint8
44-
45-const (
46- opChildren op = iota // Accessible.GetChildren -> refs
47- opPID // GetConnectionUnixProcessID(name) -> u
48- opRole // Accessible.GetRole -> u
49- opState // Accessible.GetState -> state
50- opExtents // Component.GetExtents(arg) -> rect
51- opMatches // Collection.GetMatches(rule) -> refs
52-)
53-
54-// ref is an accessible object: a bus name and a path.
55-type ref struct{ name, path string }
56-
57-// call is one request of a batch. name and path address the object; for
58-// opPID, name is the bus name asked about.
59-type call struct {
60- op op
61- name, path string
62- arg uint32
63- rule matchRule
64-}
65-
66-// reply answers the call at the same index of the batch.
67-type reply struct {
68- err error
69- refs []ref
70- u uint32
71- state uint64
72- rect [4]int32 // x, y, w, h
73-}
74-
75-// transport sends batches of calls on one bus connection. It is used by one
76-// goroutine at a time.
77-type transport interface {
78- // Do sends calls pipelined and returns one reply per call, in order.
79- // The batch ends at deadline. A call not answered idle after it was
80- // sent gets ErrTimeout. When the connection fails or the deadline
81- // finds calls unanswered, those calls get an error wrapping
82- // ErrBroken: the connection must be dropped.
83- Do(deadline time.Time, idle time.Duration, calls []call) []reply
84- Close() error
85-}
86-
87-// ErrTimeout is the error of a call not answered in time.
88-var ErrTimeout = errors.New("atspi: call timed out")
89-
90-// busError is a D-Bus error reply.
91-type busError struct{ name, msg string }
92-
93-func (e *busError) Error() string {
94- if e.msg == "" {
95- return e.name
96- }
97- return e.name + ": " + e.msg
98-}
99-
100-// conn is a transport on a zerobus connection.
101-type conn struct {
102- c *zerobus.Conn
103- raw syscall.RawConn
104-}
105-
106-func newConn(c *zerobus.Conn) (*conn, error) {
107- raw, err := c.SyscallConn()
108- if err != nil {
109- _ = c.Close()
110- return nil, err
111- }
112- return &conn{c: c, raw: raw}, nil
113-}
114-
115-func (b *conn) Close() error { return b.c.Close() }
116-
117-// readable waits until a message can be read or d passes.
118-func (b *conn) readable(d time.Duration) (bool, error) {
119- if b.c.Buffered() {
120- return true, nil
121- }
122- ms := int((d + time.Millisecond - 1) / time.Millisecond)
123- if ms < 1 {
124- ms = 1
125- }
126- var n int
127- var perr error
128- if err := b.raw.Control(func(fd uintptr) {
129- fds := []unix.PollFd{{Fd: int32(fd), Events: unix.POLLIN}}
130- for {
131- n, perr = unix.Poll(fds, ms)
132- if perr != unix.EINTR {
133- return
134- }
135- }
136- }); err != nil {
137- return false, err
138- }
139- return n > 0, perr
140-}
141-
142-// ErrBroken is the error of calls on a connection that failed or was closed
143-// at the deadline; the owner must drop it and connect again.
144-var ErrBroken = errors.New("atspi: connection broken")
145-
146-// errDeadline is the error of calls still unanswered at the batch deadline:
147-// late replies and half-read messages leave the connection unusable.
148-var errDeadline = fmt.Errorf("%w: %w", ErrBroken, ErrTimeout)
149-
150-// watchdogGrace is how long after the deadline the watchdog closes a socket
151-// that the batch loop could not leave by itself (a stuck read or write).
152-const watchdogGrace = 20 * time.Millisecond
153-
154-// Do sends the calls pipelined. A watchdog closes the socket shortly after
155-// deadline, so neither a blocked write nor a read stuck in the middle of a
156-// message can outlive the batch; the connection is then broken. Each call
157-// has its own idle timeout, counted from its send: traffic for other calls
158-// does not extend it.
159-func (b *conn) Do(deadline time.Time, idle time.Duration, calls []call) []reply {
160- out := make([]reply, len(calls))
161- answered := make([]bool, len(calls))
162- type inflight struct {
163- idx int
164- sent time.Time
165- }
166- pending := make(map[uint32]inflight, min(len(calls), maxInflight))
167- next, done := 0, 0
168- var fired atomic.Bool
169- watchdog := time.AfterFunc(time.Until(deadline)+watchdogGrace, func() {
170- fired.Store(true)
171- _ = b.c.Close()
172- })
173- defer watchdog.Stop()
174- // finish gives err to every call without an answer.
175- finish := func(err error) []reply {
176- if !errors.Is(err, ErrBroken) && (fired.Load() || !errors.Is(err, ErrTimeout)) {
177- err = fmt.Errorf("%w: %w", ErrBroken, err)
178- }
179- if errors.Is(err, ErrBroken) {
180- _ = b.c.Close() // late replies would desynchronize the next batch
181- }
182- for i := range out {
183- if !answered[i] {
184- out[i].err = err
185- }
186- }
187- return out
188- }
189- for done < len(calls) {
190- for next < len(calls) && len(pending) < maxInflight {
191- if !time.Now().Before(deadline) {
192- return finish(errDeadline)
193- }
194- serial, err := b.send(calls[next])
195- if err != nil {
196- return finish(err)
197- }
198- pending[serial] = inflight{idx: next, sent: time.Now()}
199- next++
200- }
201- // Calls that waited longer than idle get no reply.
202- now := time.Now()
203- var oldest time.Time
204- for serial, p := range pending {
205- if now.Sub(p.sent) >= idle {
206- out[p.idx].err = ErrTimeout
207- answered[p.idx] = true
208- done++
209- delete(pending, serial)
210- } else if oldest.IsZero() || p.sent.Before(oldest) {
211- oldest = p.sent
212- }
213- }
214- if len(pending) == 0 {
215- continue
216- }
217- wait := min(time.Until(oldest.Add(idle)), time.Until(deadline))
218- if wait <= 0 {
219- if !time.Now().Before(deadline) {
220- break
221- }
222- continue
223- }
224- ok, err := b.readable(wait)
225- if err != nil {
226- return finish(err)
227- }
228- if !ok {
229- continue
230- }
231- m, err := b.c.ReadMessage()
232- if err != nil {
233- return finish(err)
234- }
235- p, found := pending[m.ReplySerial]
236- if !found || (m.Type != zerobus.TypeMethodReturn && m.Type != zerobus.TypeError) {
237- // A signal, or the late reply of an earlier batch.
238- continue
239- }
240- delete(pending, m.ReplySerial)
241- decode(m, calls[p.idx], &out[p.idx])
242- answered[p.idx] = true
243- done++
244- }
245- if done == len(calls) {
246- return out
247- }
248- return finish(errDeadline)
249-}
250-
251-// send builds and sends one call and returns its serial.
252-func (b *conn) send(c call) (uint32, error) {
253- switch c.op {
254- case opChildren:
255- b.c.NewCall(c.name, c.path, ifaceAccessible, "GetChildren", "")
256- case opPID:
257- b.c.NewCall(busDaemon, busDaemonPath, busDaemon, "GetConnectionUnixProcessID", "s").Str(c.name)
258- case opRole:
259- b.c.NewCall(c.name, c.path, ifaceAccessible, "GetRole", "")
260- case opState:
261- b.c.NewCall(c.name, c.path, ifaceAccessible, "GetState", "")
262- case opExtents:
263- b.c.NewCall(c.name, c.path, ifaceComponent, "GetExtents", "u").Uint32(c.arg)
264- case opMatches:
265- e := b.c.NewCall(c.name, c.path, ifaceCollection, "GetMatches", "(aiia{ss}iaiiasib)uib")
266- encodeRule(e, c.rule)
267- e.Uint32(sortCanonical)
268- e.Int32(matchCount)
269- e.Bool(true)
270- }
271- return b.c.Send()
272-}
273-
274-// encodeRule writes the MatchRule struct (aiia{ss}iaiiasib): states, state
275-// match, attributes, attribute match, roles, role match, interfaces,
276-// interface match, invert.
277-func encodeRule(e *zerobus.Encoder, r matchRule) {
278- e.Struct()
279- a := e.BeginArray('i')
280- for _, w := range stateBits(stateShowing, stateVisible) {
281- e.Int32(w)
282- }
283- e.EndArray(a)
284- e.Int32(matchAll)
285- m := e.BeginArray('{')
286- e.EndArray(m)
287- e.Int32(matchAll)
288- a = e.BeginArray('i')
289- for _, w := range roleBits(r.roles()) {
290- e.Int32(w)
291- }
292- e.EndArray(a)
293- e.Int32(matchAny)
294- s := e.BeginArray('s')
295- e.EndArray(s)
296- e.Int32(matchAll)
297- e.Bool(false)
298-}
299-
300-// decode reads the reply m of c into out.
301-func decode(m *zerobus.Message, c call, out *reply) {
302- if m.Type == zerobus.TypeError {
303- be := &busError{name: strings.Clone(m.ErrorName)}
304- if strings.HasPrefix(m.Signature, "s") {
305- be.msg = strings.Clone(m.Body().Str())
306- }
307- out.err = be
308- return
309- }
310- r := m.Body()
311- want := func(sig string) bool {
312- if m.Signature != sig {
313- out.err = fmt.Errorf("atspi: reply signature %q, want %q", m.Signature, sig)
314- return false
315- }
316- return true
317- }
318- switch c.op {
319- case opChildren, opMatches:
320- if !want("a(so)") {
321- return
322- }
323- end := r.Array('(')
324- for r.More(end) {
325- if len(out.refs) >= maxRefs {
326- break
327- }
328- r.Struct()
329- name := r.Str()
330- path := r.ObjectPath()
331- if r.Err() != nil {
332- break
333- }
334- out.refs = append(out.refs, ref{strings.Clone(name), strings.Clone(path)})
335- }
336- case opPID, opRole:
337- if want("u") {
338- out.u = r.Uint32()
339- }
340- case opState:
341- if !want("au") {
342- return
343- }
344- end := r.Array('u')
345- var words [2]uint32
346- for i := 0; r.More(end); i++ {
347- w := r.Uint32()
348- if i < len(words) {
349- words[i] = w
350- }
351- }
352- out.state = uint64(words[0]) | uint64(words[1])<<32
353- case opExtents:
354- if want("(iiii)") {
355- r.Struct()
356- out.rect = [4]int32{r.Int32(), r.Int32(), r.Int32(), r.Int32()}
357- }
358- }
359- if out.err == nil && r.Err() != nil {
360- out.err = r.Err()
361- }
362-}
D internal/adapters/atspi/transport_mock_test.go +0 -156
@@ -1,156 +0,0 @@
1-// Code generated by mockery; DO NOT EDIT.
2-// github.com/vektra/mockery
3-// template: testify
4-
5-package atspi
6-
7-import (
8- "time"
9-
10- mock "github.com/stretchr/testify/mock"
11-)
12-
13-// newMocktransport creates a new instance of mocktransport. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
14-// The first argument is typically a *testing.T value.
15-func newMocktransport(t interface {
16- mock.TestingT
17- Cleanup(func())
18-}) *mocktransport {
19- if helper, ok := t.(interface{ Helper() }); ok {
20- helper.Helper()
21- }
22-
23- mock := &mocktransport{}
24- mock.Mock.Test(t)
25-
26- t.Cleanup(func() {
27- if helper, ok := t.(interface{ Helper() }); ok {
28- helper.Helper()
29- }
30- mock.AssertExpectations(t)
31- })
32-
33- return mock
34-}
35-
36-// mocktransport is an autogenerated mock type for the transport type
37-type mocktransport struct {
38- mock.Mock
39-}
40-
41-type mocktransport_Expecter struct {
42- mock *mock.Mock
43-}
44-
45-func (_m *mocktransport) EXPECT() *mocktransport_Expecter {
46- return &mocktransport_Expecter{mock: &_m.Mock}
47-}
48-
49-// Close provides a mock function for the type mocktransport
50-func (_mock *mocktransport) Close() error {
51- ret := _mock.Called()
52-
53- if len(ret) == 0 {
54- panic("no return value specified for Close")
55- }
56-
57- var r0 error
58- if returnFunc, ok := ret.Get(0).(func() error); ok {
59- r0 = returnFunc()
60- } else {
61- r0 = ret.Error(0)
62- }
63- return r0
64-}
65-
66-// mocktransport_Close_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Close'
67-type mocktransport_Close_Call struct {
68- *mock.Call
69-}
70-
71-// Close is a helper method to define mock.On call
72-func (_e *mocktransport_Expecter) Close() *mocktransport_Close_Call {
73- return &mocktransport_Close_Call{Call: _e.mock.On("Close")}
74-}
75-
76-func (_c *mocktransport_Close_Call) Run(run func()) *mocktransport_Close_Call {
77- _c.Call.Run(func(args mock.Arguments) {
78- run()
79- })
80- return _c
81-}
82-
83-func (_c *mocktransport_Close_Call) Return(err error) *mocktransport_Close_Call {
84- _c.Call.Return(err)
85- return _c
86-}
87-
88-func (_c *mocktransport_Close_Call) RunAndReturn(run func() error) *mocktransport_Close_Call {
89- _c.Call.Return(run)
90- return _c
91-}
92-
93-// Do provides a mock function for the type mocktransport
94-func (_mock *mocktransport) Do(deadline time.Time, idle time.Duration, calls []call) []reply {
95- ret := _mock.Called(deadline, idle, calls)
96-
97- if len(ret) == 0 {
98- panic("no return value specified for Do")
99- }
100-
101- var r0 []reply
102- if returnFunc, ok := ret.Get(0).(func(time.Time, time.Duration, []call) []reply); ok {
103- r0 = returnFunc(deadline, idle, calls)
104- } else {
105- if ret.Get(0) != nil {
106- r0 = ret.Get(0).([]reply)
107- }
108- }
109- return r0
110-}
111-
112-// mocktransport_Do_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Do'
113-type mocktransport_Do_Call struct {
114- *mock.Call
115-}
116-
117-// Do is a helper method to define mock.On call
118-// - deadline time.Time
119-// - idle time.Duration
120-// - calls []call
121-func (_e *mocktransport_Expecter) Do(deadline any, idle any, calls any) *mocktransport_Do_Call {
122- return &mocktransport_Do_Call{Call: _e.mock.On("Do", deadline, idle, calls)}
123-}
124-
125-func (_c *mocktransport_Do_Call) Run(run func(deadline time.Time, idle time.Duration, calls []call)) *mocktransport_Do_Call {
126- _c.Call.Run(func(args mock.Arguments) {
127- var arg0 time.Time
128- if args[0] != nil {
129- arg0 = args[0].(time.Time)
130- }
131- var arg1 time.Duration
132- if args[1] != nil {
133- arg1 = args[1].(time.Duration)
134- }
135- var arg2 []call
136- if args[2] != nil {
137- arg2 = args[2].([]call)
138- }
139- run(
140- arg0,
141- arg1,
142- arg2,
143- )
144- })
145- return _c
146-}
147-
148-func (_c *mocktransport_Do_Call) Return(replyMoqParams []reply) *mocktransport_Do_Call {
149- _c.Call.Return(replyMoqParams)
150- return _c
151-}
152-
153-func (_c *mocktransport_Do_Call) RunAndReturn(run func(deadline time.Time, idle time.Duration, calls []call) []reply) *mocktransport_Do_Call {
154- _c.Call.Return(run)
155- return _c
156-}
M internal/adapters/config/config.go +43 -40
@@ -116,7 +116,7 @@ var actions = map[string]bool{
116 "toggle-floating": true,116 "toggle-floating": true,
117 "hint": true,117 "hint": true,
118 }118 }
119-var components = map[string]bool{"core": true, "wayland": true, "input": true, "drm": true, "seat": true, "render": true, "sync": true, "config": true, "app": true}119+var components = map[string]bool{"core": true, "wayland": true, "input": true, "drm": true, "seat": true, "render": true, "sync": true, "config": true, "app": true, "hints": true}
120 var color = regexp.MustCompile(`^#[0-9a-fA-F]{6}$`)120 var color = regexp.MustCompile(`^#[0-9a-fA-F]{6}$`)
121 121
122 // columnStep is the +N or -N step of set-column-width, in cells, N from 1122 // columnStep is the +N or -N step of set-column-width, in cells, N from 1
@@ -166,7 +166,8 @@ func Defaults() ports.Config {
166 c.Terminal.AutoOpen = "first"166 c.Terminal.AutoOpen = "first"
167 // Hint mode is experimental: off unless the config turns it on.167 // Hint mode is experimental: off unless the config turns it on.
168 c.Hint.Enable = false168 c.Hint.Enable = false
169- c.Hint.TextClick = ports.ModCtrl | ports.ModShift169+ c.Hint.Detector = defaultHintDetector
170+ c.Hint.Model = defaultHintModel()
170 c.Xwayland = "xwayland-satellite"171 c.Xwayland = "xwayland-satellite"
171 c.Background.Color = "#111111"172 c.Background.Color = "#111111"
172 c.Floating.Dim = 0.3173 c.Floating.Dim = 0.3
@@ -895,21 +896,27 @@ func set(c *ports.Config, key, v string) error {
895 } else {896 } else {
896 c.Hint.Foreground = v897 c.Hint.Foreground = v
897 }898 }
898- case "hint.targets":899+ case "hint.detector":
899- switch v {900+ if v == "" {
900- case "clickable":901+ v = defaultHintDetector
901- c.Hint.LinksOnly = false
902- case "links":
903- c.Hint.LinksOnly = true
904- default:
905- return fmt.Errorf("must be clickable or links")
906 }902 }
907- case "hint.text-click":903+ c.Hint.Detector = expandHome(v)
908- m, err := parseTextClick(v)904+ case "hint.model":
909- if err != nil {905+ if v == "" {
910- return err906+ c.Hint.Model = defaultHintModel()
907+ break
911 }908 }
912- c.Hint.TextClick = m909+ c.Hint.Model = expandHome(v)
910+ case "hint.threshold":
911+ if v == "" {
912+ c.Hint.Threshold = 0
913+ break
914+ }
915+ n, err := strconv.ParseFloat(v, 64)
916+ if err != nil || math.IsNaN(n) || n < 0.01 || n > 0.99 {
917+ return fmt.Errorf("must be a number from 0.01 to 0.99")
918+ }
919+ c.Hint.Threshold = n
913 case "focus.flash-width":920 case "focus.flash-width":
914 n, err := strconv.Atoi(v)921 n, err := strconv.Atoi(v)
915 if err != nil || n < 1 || n > 8 {922 if err != nil || n < 1 || n > 8 {
@@ -1333,34 +1340,30 @@ func checkAction(v string) error {
1333 return nil1340 return nil
1334 }1341 }
1335 1342
1336-// parseTextClick parses hint.text-click: `+`-separated modifier names as in1343+const defaultHintDetector = "nefer-hints"
1337-// binds, or none. cmd is an alias of super here (it is not resolved through1344+
1338-// keyboard.cmd, so the result does not depend on key order).1345+// defaultHintModel is $XDG_DATA_HOME/nefer-hints/models, or
1339-func parseTextClick(v string) (ports.Mods, error) {1346+// ~/.local/share/nefer-hints/models when XDG_DATA_HOME is unset.
1340- if strings.ToLower(v) == "none" {1347+func defaultHintModel() string {
1341- return 0, nil1348+ base := os.Getenv("XDG_DATA_HOME")
1342- }1349+ if base == "" {
1343- var mods ports.Mods1350+ home, err := os.UserHomeDir()
1344- for _, p := range strings.Split(v, "+") {1351+ if err != nil {
1345- var m ports.Mods1352+ return ""
1346- switch strings.ToLower(strings.TrimSpace(p)) {
1347- case "shift":
1348- m = ports.ModShift
1349- case "ctrl":
1350- m = ports.ModCtrl
1351- case "alt":
1352- m = ports.ModAlt
1353- case "super", "cmd":
1354- m = ports.ModSuper
1355- default:
1356- return 0, fmt.Errorf("unknown modifier %q (use shift, ctrl, alt, super or cmd joined by +, or none)", p)
1357 }1353 }
1358- if mods&m != 0 {1354+ base = filepath.Join(home, ".local", "share")
1359- return 0, fmt.Errorf("duplicate modifier %q", p)1355+ }
1356+ return filepath.Join(base, "nefer-hints", "models")
1357+}
1358+
1359+// expandHome replaces a leading ~/ with the home directory.
1360+func expandHome(p string) string {
1361+ if rest, ok := strings.CutPrefix(p, "~/"); ok {
1362+ if home, err := os.UserHomeDir(); err == nil {
1363+ return filepath.Join(home, rest)
1360 }1364 }
1361- mods |= m
1362 }1365 }
1363- return mods, nil1366+ return p
1364 }1367 }
1365 1368
1366 // parseCombo turns `ctrl+cmd+é` into the canonical `Cmd+Ctrl+eacute`:1369 // parseCombo turns `ctrl+cmd+é` into the canonical `Cmd+Ctrl+eacute`:
M internal/adapters/config/config_test.go +51 -41
@@ -954,26 +954,61 @@ func TestLoadRejectsFIFO(t *testing.T) {
954 }954 }
955 }955 }
956 956
957-func TestHintTargets(t *testing.T) {957+func TestHintBind(t *testing.T) {
958- if Defaults().Hint.LinksOnly {958+ c, w := parseString(t, "bind.cmd+g = hint\n")
959- t.Fatal("hint targets default must be clickable")959+ if len(w) != 0 || c.Binds["Cmd+g"] != "hint" {
960+ t.Fatal(c.Binds["Cmd+g"], w)
960 }961 }
961- c, w := parseString(t, "hint.targets = links\n")962+}
962- if !c.Hint.LinksOnly || len(w) != 0 {963+
963- t.Fatal(c.Hint, w)964+func TestHintRemovedKeysAreUnknown(t *testing.T) {
965+ for _, line := range []string{"hint.targets = links", "hint.text-click = none"} {
966+ _, w := parseString(t, line+"\n")
967+ if len(w) != 1 {
968+ t.Errorf("%q: %v", line, w)
969+ }
964 }970 }
965- c, w = parseString(t, "hint.targets = clickable\n")971+}
966- if c.Hint.LinksOnly || len(w) != 0 {972+
967- t.Fatal(c.Hint, w)973+func TestHintDetectorConfig(t *testing.T) {
974+ t.Setenv("HOME", "/home/u")
975+ t.Setenv("XDG_DATA_HOME", "")
976+ d := Defaults().Hint
977+ if d.Detector != "nefer-hints" || d.Model != "/home/u/.local/share/nefer-hints/models" || d.Threshold != 0 {
978+ t.Fatalf("defaults: %+v", d)
968 }979 }
969- // An invalid value warns and keeps the default.980+ t.Setenv("XDG_DATA_HOME", "/data")
970- c, w = parseString(t, "hint.targets = buttons\n")981+ if m := Defaults().Hint.Model; m != "/data/nefer-hints/models" {
971- if c.Hint.LinksOnly || len(w) != 1 {982+ t.Fatal(m)
972- t.Fatal(c.Hint, w)
973 }983 }
974- c, w = parseString(t, "bind.cmd+g = hint\n")984+ c, w := parseString(t, "hint.detector = ~/bin/nh\nhint.model = ~/m\nhint.threshold = 0.3\n")
975- if len(w) != 0 || c.Binds["Cmd+g"] != "hint" {985+ if len(w) != 0 || c.Hint.Detector != "/home/u/bin/nh" || c.Hint.Model != "/home/u/m" || c.Hint.Threshold != 0.3 {
976- t.Fatal(c.Binds["Cmd+g"], w)986+ t.Fatalf("%+v %v", c.Hint, w)
987+ }
988+ c, w = parseString(t, "hint.detector = /opt/nh\nhint.model = /opt/models\n")
989+ if len(w) != 0 || c.Hint.Detector != "/opt/nh" || c.Hint.Model != "/opt/models" {
990+ t.Fatalf("%+v %v", c.Hint, w)
991+ }
992+ for _, input := range []string{"", "hint.threshold =\n"} {
993+ c, w := parseString(t, input)
994+ if len(w) != 0 || c.Hint.Threshold != 0 {
995+ t.Errorf("unset threshold %q: %v %v", input, c.Hint.Threshold, w)
996+ }
997+ }
998+ c, w = parseString(t, "hint.threshold = 0.3\nhint.threshold =\n")
999+ if len(w) != 1 || c.Hint.Threshold != 0 { // duplicate-key warning only
1000+ t.Fatalf("cleared threshold: %v %v", c.Hint.Threshold, w)
1001+ }
1002+ for _, v := range []string{"0", "0.009", "1", "0.995", "NaN", "Inf", "-Inf", "high", "-1"} {
1003+ c, w := parseString(t, "hint.threshold = "+v+"\n")
1004+ if len(w) != 1 || c.Hint.Threshold != 0 {
1005+ t.Errorf("hint.threshold = %s: %v %v", v, c.Hint.Threshold, w)
1006+ }
1007+ }
1008+ for _, v := range []string{"0.01", "0.99"} {
1009+ if c, w := parseString(t, "hint.threshold = "+v+"\n"); len(w) != 0 || c.Hint.Threshold == 0 {
1010+ t.Errorf("hint.threshold = %s: %v", v, w)
1011+ }
977 }1012 }
978 }1013 }
979 1014
@@ -1021,31 +1056,6 @@ func TestHintSizeBounds(t *testing.T) {
1021 }1056 }
1022 }1057 }
1023 1058
1024-func TestHintTextClick(t *testing.T) {
1025- if got := Defaults().Hint.TextClick; got != ports.ModCtrl|ports.ModShift {
1026- t.Fatalf("default %d", got)
1027- }
1028- for in, want := range map[string]ports.Mods{
1029- "ctrl+shift": ports.ModCtrl | ports.ModShift,
1030- "Alt": ports.ModAlt,
1031- "super": ports.ModSuper,
1032- "cmd+shift": ports.ModSuper | ports.ModShift,
1033- "none": 0,
1034- "shift+ctrl+alt": ports.ModShift | ports.ModCtrl | ports.ModAlt,
1035- } {
1036- c, w := parseString(t, "hint.text-click = "+in+"\n")
1037- if c.Hint.TextClick != want || len(w) != 0 {
1038- t.Errorf("%q: %d %v", in, c.Hint.TextClick, w)
1039- }
1040- }
1041- for _, bad := range []string{"hyper", "ctrl+", "ctrl+ctrl", "ctrl+none", ""} {
1042- c, w := parseString(t, "hint.text-click = "+bad+"\n")
1043- if len(w) != 1 || c.Hint.TextClick != ports.ModCtrl|ports.ModShift {
1044- t.Errorf("%q: %d %v", bad, c.Hint.TextClick, w)
1045- }
1046- }
1047-}
1048-
1049 func TestHintEnable(t *testing.T) {1059 func TestHintEnable(t *testing.T) {
1050 if Defaults().Hint.Enable {1060 if Defaults().Hint.Enable {
1051 t.Fatal("hint mode must be off by default")1061 t.Fatal("hint mode must be off by default")
D internal/adapters/hints/benchmark_test.go +0 -81
@@ -1,81 +0,0 @@
1-package hints
2-
3-import (
4- "context"
5- "image"
6- "image/draw"
7- "image/png"
8- "os"
9- "os/exec"
10- "path/filepath"
11- "strconv"
12- "testing"
13- "time"
14-)
15-
16-// TestFrameBenchmark logs counts and timings, never filenames or OCR text.
17-func TestFrameBenchmark(t *testing.T) {
18- dir := os.Getenv("NEFERWL_HINT_BENCH_DIR")
19- if dir == "" {
20- t.Skip("set NEFERWL_HINT_BENCH_DIR to a local PNG directory")
21- }
22- if _, err := exec.LookPath("tesseract"); err != nil {
23- t.Skip("tesseract not on PATH")
24- }
25- files, err := filepath.Glob(filepath.Join(dir, "*.png"))
26- if err != nil || len(files) == 0 {
27- t.Fatal("no PNG frames")
28- }
29- // Settings under test: NEFERWL_HINT_BENCH_WORKERS=<n>, NEFERWL_HINT_BENCH_PNG=speed|none.
30- if v, _ := strconv.Atoi(os.Getenv("NEFERWL_HINT_BENCH_WORKERS")); v > 0 {
31- ocrWorkers = v
32- }
33- if os.Getenv("NEFERWL_HINT_BENCH_PGM") == "0" {
34- rawPGM = false
35- }
36- if os.Getenv("NEFERWL_HINT_BENCH_PNG") == "none" {
37- pngLevel = png.NoCompression
38- }
39- for i, file := range files {
40- f, err := os.Open(file)
41- if err != nil {
42- t.Fatal("open frame failed")
43- }
44- src, err := png.Decode(f)
45- _ = f.Close()
46- if err != nil {
47- t.Fatal("decode frame failed")
48- }
49- img := image.NewRGBA(image.Rect(0, 0, src.Bounds().Dx(), src.Bounds().Dy()))
50- draw.Draw(img, img.Bounds(), src, src.Bounds().Min, draw.Src)
51- start := time.Now()
52- words, bands, err := readWords(context.Background(), toGray(img))
53- if err != nil {
54- t.Fatal("recognition failed")
55- }
56- t.Logf("frame=%d px=%d ocr_ms=%.1f words=%d links=%d bands=%d", i, img.Bounds().Dx()*img.Bounds().Dy(), ms(time.Since(start)), len(words), len(textLinks(words, 1)), bands)
57- if os.Getenv("NEFERWL_HINT_BENCH_STRIPS") == "" {
58- continue
59- }
60- // Experimental strips use known OCR word regions, not pixel targets.
61- // This is an oracle comparison, not a production optimization.
62- lines := make([]image.Rectangle, 0, len(words))
63- for _, word := range words {
64- lines = append(lines, word.Rect)
65- }
66- for _, normalized := range []bool{false, true} {
67- begin := time.Now()
68- compact, _, err := readStrips(context.Background(), img, lines, 8, normalized)
69- if err != nil {
70- t.Fatal("strip recognition failed")
71- }
72- t.Logf("frame=%d strips normalized=%t ocr_ms=%.1f words=%d links=%d", i, normalized, ms(time.Since(begin)), len(compact), len(textLinks(compact, 1)))
73- }
74- }
75-}
76-
77-func toGray(img *image.RGBA) *image.Gray {
78- g := image.NewGray(img.Bounds())
79- draw.Draw(g, g.Bounds(), img, img.Bounds().Min, draw.Src)
80- return g
81-}
A internal/adapters/hints/detector.go +448 -0
@@ -0,0 +1,448 @@
1+package hints
2+
3+import (
4+ "bufio"
5+ "bytes"
6+ "context"
7+ "encoding/binary"
8+ "errors"
9+ "fmt"
10+ "io"
11+ "math"
12+ "os"
13+ "os/exec"
14+ "strconv"
15+ "sync"
16+ "syscall"
17+ "time"
18+
19+ "github.com/bnema/neferwl/internal/adapters/logging"
20+)
21+
22+// detFrame is a captured output as the detector receives it: BGRA pixels in
23+// m, valid only during the Detect call. It holds the mapping, not the bytes,
24+// so that recording a call (mocks print their arguments) never reads memory
25+// that was unmapped since.
26+type detFrame struct {
27+ width, height, stride int
28+ m *mapping
29+ // regions are the crops detected independently; none means the whole
30+ // frame.
31+ regions []region
32+}
33+
34+// detection is one box found by the detector, in frame (physical) pixels.
35+type detection struct {
36+ x1, y1, x2, y2, score float32
37+ // region is the index of the request region the box comes from.
38+ region int
39+}
40+
41+// detector finds clickable elements in a frame. The worker goroutine is its
42+// only caller.
43+type detector interface {
44+ // Detect returns the boxes found in f. An error is for this call only:
45+ // the detector recovers by itself on the next one.
46+ Detect(ctx context.Context, f detFrame) ([]detection, error)
47+ // Close releases the detector. It is called once, by the worker
48+ // goroutine, when the worker returns.
49+ Close() error
50+}
51+
52+const (
53+ protocolVersion = 1
54+ helloTimeout = 10 * time.Second
55+ requestTimeout = 2 * time.Second
56+ imgsz = 1280
57+ // maxString and maxBoxes bound what a misbehaving child can make us
58+ // allocate.
59+ maxString = 1 << 16
60+ maxBoxes = 64 * 1000 // runner default max-det per region, at most 64 regions
61+)
62+
63+var (
64+ magicHello = [4]byte{'N', 'H', 'N', 'T'}
65+ magicRequest = [4]byte{'N', 'H', 'R', 'Q'}
66+ magicResponse = [4]byte{'N', 'H', 'R', 'S'}
67+
68+ // errProtocol is a framing error: the stream cannot be trusted any more.
69+ errProtocol = errors.New("hints: detector protocol error")
70+ // errDetector is an error the detector reported itself; it keeps serving.
71+ errDetector = errors.New("hints: detector error")
72+)
73+
74+var le = binary.LittleEndian
75+
76+// writeRequest writes one request (the regions of f, default confidence)
77+// followed by the pixels.
78+func writeRequest(w io.Writer, seq uint64, f detFrame) error {
79+ var h [4 + 4 + 8 + 5*4]byte
80+ copy(h[0:], magicRequest[:])
81+ le.PutUint32(h[4:], protocolVersion)
82+ le.PutUint64(h[8:], seq)
83+ le.PutUint32(h[16:], uint32(f.width))
84+ le.PutUint32(h[20:], uint32(f.height))
85+ le.PutUint32(h[24:], uint32(f.stride))
86+ le.PutUint32(h[28:], formatXRGB8888)
87+ le.PutUint32(h[32:], uint32(len(f.regions)))
88+ if _, err := w.Write(h[:]); err != nil {
89+ return err
90+ }
91+ var rec [16]byte
92+ for _, r := range f.regions {
93+ le.PutUint32(rec[0:], uint32(int32(r.x)))
94+ le.PutUint32(rec[4:], uint32(int32(r.y)))
95+ le.PutUint32(rec[8:], uint32(r.w))
96+ le.PutUint32(rec[12:], uint32(r.h))
97+ if _, err := w.Write(rec[:]); err != nil {
98+ return err
99+ }
100+ }
101+ var conf [4]byte // confidence: startup default
102+ le.PutUint32(conf[:], math.Float32bits(0))
103+ if _, err := w.Write(conf[:]); err != nil {
104+ return err
105+ }
106+ _, err := w.Write(f.m.b[:f.height*f.stride])
107+ return err
108+}
109+
110+func readMagic(r io.Reader, want [4]byte) error {
111+ var m [4]byte
112+ if _, err := io.ReadFull(r, m[:]); err != nil {
113+ return err
114+ }
115+ if m != want {
116+ return fmt.Errorf("%w: bad magic %q", errProtocol, m[:])
117+ }
118+ return nil
119+}
120+
121+func readU32(r io.Reader) (uint32, error) {
122+ var b [4]byte
123+ _, err := io.ReadFull(r, b[:])
124+ return le.Uint32(b[:]), err
125+}
126+
127+func readString(r io.Reader) (string, error) {
128+ n, err := readU32(r)
129+ if err != nil {
130+ return "", err
131+ }
132+ if n > maxString {
133+ return "", fmt.Errorf("%w: string of %d bytes", errProtocol, n)
134+ }
135+ b := make([]byte, n)
136+ _, err = io.ReadFull(r, b)
137+ return string(b), err
138+}
139+
140+// readHello reads the hello: the device name, or the error of a failed start.
141+func readHello(r io.Reader) (device string, err error) {
142+ if err := readMagic(r, magicHello); err != nil {
143+ return "", err
144+ }
145+ version, err := readU32(r)
146+ if err != nil {
147+ return "", err
148+ }
149+ if version != protocolVersion {
150+ return "", fmt.Errorf("%w: version %d", errProtocol, version)
151+ }
152+ status, err := readU32(r)
153+ if err != nil {
154+ return "", err
155+ }
156+ msg, err := readString(r)
157+ if err != nil {
158+ return "", err
159+ }
160+ if status != 0 {
161+ return "", fmt.Errorf("%w: start failed: %s", errDetector, msg)
162+ }
163+ if _, err := readU32(r); err != nil { // imgsz
164+ return "", err
165+ }
166+ return msg, nil
167+}
168+
169+// readResponse reads the response to request seq.
170+func readResponse(r io.Reader, seq uint64) ([]detection, error) {
171+ if err := readMagic(r, magicResponse); err != nil {
172+ return nil, err
173+ }
174+ var h [16]byte
175+ if _, err := io.ReadFull(r, h[:]); err != nil {
176+ return nil, err
177+ }
178+ if v := le.Uint32(h[0:]); v != protocolVersion {
179+ return nil, fmt.Errorf("%w: version %d", errProtocol, v)
180+ }
181+ if got := le.Uint64(h[4:]); got != seq {
182+ return nil, fmt.Errorf("%w: response %d to request %d", errProtocol, got, seq)
183+ }
184+ if status := le.Uint32(h[12:]); status != 0 {
185+ msg, err := readString(r)
186+ if err != nil {
187+ return nil, err
188+ }
189+ return nil, fmt.Errorf("%w: status %d: %s", errDetector, status, msg)
190+ }
191+ var t [8]byte // inference ms, nboxes
192+ if _, err := io.ReadFull(r, t[:]); err != nil {
193+ return nil, err
194+ }
195+ n := le.Uint32(t[4:])
196+ if n > maxBoxes {
197+ return nil, fmt.Errorf("%w: %d boxes", errProtocol, n)
198+ }
199+ out := make([]detection, 0, min(n, 4096))
200+ var rec [24]byte
201+ for range n {
202+ if _, err := io.ReadFull(r, rec[:]); err != nil {
203+ return nil, err
204+ }
205+ f := func(i int) float32 { return math.Float32frombits(le.Uint32(rec[i*4:])) }
206+ out = append(out, detection{x1: f(0), y1: f(1), x2: f(2), y2: f(3), score: f(4), region: int(le.Uint32(rec[20:]))})
207+ }
208+ return out, nil
209+}
210+
211+// procDetector runs the nefer-hints binary as a persistent child. It starts
212+// on the first Detect, and again after the child died or misbehaved.
213+type procDetector struct {
214+ command, model string
215+ conf float64
216+ helloTimeout time.Duration
217+ reqTimeout time.Duration
218+
219+ proc *process
220+ seq uint64
221+}
222+
223+func newProcDetector(command, model string, threshold float64) *procDetector {
224+ return &procDetector{command: command, model: model, conf: threshold, helloTimeout: helloTimeout, reqTimeout: requestTimeout}
225+}
226+
227+// waitDelay bounds cmd.Wait once the child is dead: a descendant that kept
228+// stderr open must not block the reaping.
229+const (
230+ waitDelay = time.Second
231+ closeGrace = time.Second
232+)
233+
234+// process is one running child, alone in its process group. It keeps both
235+// pipe ends so that closing them unblocks a pending read or write.
236+type process struct {
237+ cmd *exec.Cmd
238+ in *os.File // write end of the child's stdin
239+ out *os.File // read end of the child's stdout
240+ rd *bufio.Reader
241+
242+ mu sync.Mutex
243+ reaped bool
244+ once sync.Once
245+}
246+
247+// interrupt kills the whole process group and closes our pipe ends, without
248+// waiting: it unblocks a pending read or write from a timer or a context
249+// callback. kill reaps the child afterwards.
250+func (p *process) interrupt() {
251+ p.mu.Lock()
252+ if !p.reaped {
253+ _ = syscall.Kill(-p.cmd.Process.Pid, syscall.SIGKILL)
254+ }
255+ p.mu.Unlock()
256+ _ = p.out.Close()
257+ _ = p.in.Close()
258+}
259+
260+// kill stops the child and its descendants and reaps it; safe to call more
261+// than once. Wait is bounded by cmd.WaitDelay.
262+func (p *process) kill() {
263+ p.interrupt()
264+ p.wait()
265+}
266+
267+// wait reaps the child (once); bounded by cmd.WaitDelay after its death.
268+func (p *process) wait() {
269+ p.once.Do(func() {
270+ _ = p.cmd.Wait()
271+ p.mu.Lock()
272+ p.reaped = true
273+ p.mu.Unlock()
274+ })
275+}
276+
277+// cmd leaves confidence selection to the runner unless explicitly configured.
278+func (d *procDetector) cmd() *exec.Cmd {
279+ args := []string{"--model", d.model, "--imgsz", strconv.Itoa(imgsz), "--gpu", "auto"}
280+ if d.conf != 0 {
281+ args = append(args, "--conf", strconv.FormatFloat(d.conf, 'f', -1, 64))
282+ }
283+ return exec.Command(d.command, args...)
284+}
285+
286+// interruptOnDeadline binds both callbacks to this process only. Stopping joins
287+// any callback that already started, so its result can never be accepted.
288+func interruptOnDeadline(ctx context.Context, timeout time.Duration, p *process) func() (timedOut, canceled bool) {
289+ timerDone, contextDone := make(chan struct{}), make(chan struct{})
290+ timer := time.AfterFunc(timeout, func() { defer close(timerDone); p.interrupt() })
291+ stop := context.AfterFunc(ctx, func() { defer close(contextDone); p.interrupt() })
292+ return func() (bool, bool) {
293+ timedOut, canceled := !timer.Stop(), !stop()
294+ if timedOut {
295+ <-timerDone
296+ }
297+ if canceled {
298+ <-contextDone
299+ }
300+ return timedOut, canceled
301+ }
302+}
303+
304+func (d *procDetector) start(ctx context.Context) error {
305+ log := logging.For(ctx, "hints")
306+ cmd := d.cmd()
307+ cmd.Stderr = &lineLog{log: func(line string) { log.Debug().Str("detector", d.command).Msg(line) }}
308+ cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
309+ cmd.WaitDelay = waitDelay
310+ inR, inW, err := os.Pipe()
311+ if err != nil {
312+ return err
313+ }
314+ outR, outW, err := os.Pipe()
315+ if err != nil {
316+ _ = inR.Close()
317+ _ = inW.Close()
318+ return err
319+ }
320+ cmd.Stdin, cmd.Stdout = inR, outW
321+ err = cmd.Start()
322+ // The child has its ends; ours are not needed any more.
323+ _ = inR.Close()
324+ _ = outW.Close()
325+ if err != nil {
326+ _ = inW.Close()
327+ _ = outR.Close()
328+ return fmt.Errorf("hints: start detector: %w", err)
329+ }
330+ p := &process{cmd: cmd, in: inW, out: outR, rd: bufio.NewReaderSize(outR, 1<<16)}
331+ t0 := time.Now()
332+ stop := interruptOnDeadline(ctx, d.helloTimeout, p)
333+ device, err := readHello(p.rd)
334+ timedOut, canceled := stop()
335+ if err != nil || timedOut || canceled {
336+ p.kill()
337+ if ctx.Err() != nil {
338+ err = ctx.Err()
339+ } else if timedOut {
340+ err = errors.New("hello timed out")
341+ }
342+ return fmt.Errorf("hints: detector hello: %w", err)
343+ }
344+ d.proc = p
345+ log.Info().Str("device", device).Float64("start_ms", ms(time.Since(t0))).Msg("hint detector ready")
346+ return nil
347+}
348+
349+func (d *procDetector) Detect(ctx context.Context, f detFrame) ([]detection, error) {
350+ if d.proc == nil {
351+ if err := d.start(ctx); err != nil {
352+ return nil, err
353+ }
354+ }
355+ p := d.proc
356+ d.seq++
357+ stop := interruptOnDeadline(ctx, d.reqTimeout, p)
358+ dets, err := d.roundTrip(p, f)
359+ timedOut, canceled := stop()
360+ if timedOut || canceled {
361+ d.drop()
362+ if ctx.Err() != nil {
363+ return nil, ctx.Err()
364+ }
365+ return nil, fmt.Errorf("hints: detector timed out after %s", d.reqTimeout)
366+ }
367+ if err == nil {
368+ return dets, nil
369+ }
370+ if errors.Is(err, errDetector) {
371+ return nil, err // the child answered and keeps serving
372+ }
373+ d.drop()
374+ switch {
375+ case timedOut:
376+ return nil, fmt.Errorf("hints: detector timed out after %s", d.reqTimeout)
377+ case ctx.Err() != nil:
378+ return nil, ctx.Err()
379+ }
380+ return nil, fmt.Errorf("hints: detector failed: %w", err)
381+}
382+
383+func (d *procDetector) roundTrip(p *process, f detFrame) ([]detection, error) {
384+ bw := bufio.NewWriterSize(p.in, 1<<16)
385+ if err := writeRequest(bw, d.seq, f); err != nil {
386+ return nil, err
387+ }
388+ if err := bw.Flush(); err != nil {
389+ return nil, err
390+ }
391+ return readResponse(p.rd, d.seq)
392+}
393+
394+// drop kills the current child: the next Detect starts a new one.
395+func (d *procDetector) drop() {
396+ if d.proc != nil {
397+ d.proc.kill()
398+ d.proc = nil
399+ }
400+}
401+
402+// Close asks the child to stop by closing its stdin (it exits 0 on EOF), then
403+// kills it if it does not.
404+func (d *procDetector) Close() error {
405+ p := d.proc
406+ d.proc = nil
407+ if p == nil {
408+ return nil
409+ }
410+ defer func() { _ = p.out.Close() }()
411+ _ = p.in.Close()
412+ done := make(chan struct{})
413+ go func() { p.wait(); close(done) }()
414+ select {
415+ case <-done:
416+ case <-time.After(closeGrace):
417+ p.interrupt()
418+ select {
419+ case <-done:
420+ case <-time.After(2 * waitDelay):
421+ }
422+ }
423+ return nil
424+}
425+
426+// lineLog is an io.Writer that hands each complete line to log.
427+type lineLog struct {
428+ log func(string)
429+ buf []byte
430+}
431+
432+func (l *lineLog) Write(b []byte) (int, error) {
433+ l.buf = append(l.buf, b...)
434+ for {
435+ i := bytes.IndexByte(l.buf, '\n')
436+ if i < 0 {
437+ break
438+ }
439+ if line := string(l.buf[:i]); line != "" {
440+ l.log(line)
441+ }
442+ l.buf = l.buf[i+1:]
443+ }
444+ if len(l.buf) > maxString {
445+ l.buf = l.buf[:0]
446+ }
447+ return len(b), nil
448+}
A internal/adapters/hints/detector_mock_test.go +159 -0
@@ -0,0 +1,159 @@
1+// Code generated by mockery; DO NOT EDIT.
2+// github.com/vektra/mockery
3+// template: testify
4+
5+package hints
6+
7+import (
8+ "context"
9+
10+ mock "github.com/stretchr/testify/mock"
11+)
12+
13+// newMockdetector creates a new instance of mockdetector. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
14+// The first argument is typically a *testing.T value.
15+func newMockdetector(t interface {
16+ mock.TestingT
17+ Cleanup(func())
18+}) *mockdetector {
19+ if helper, ok := t.(interface{ Helper() }); ok {
20+ helper.Helper()
21+ }
22+
23+ mock := &mockdetector{}
24+ mock.Mock.Test(t)
25+
26+ t.Cleanup(func() {
27+ if helper, ok := t.(interface{ Helper() }); ok {
28+ helper.Helper()
29+ }
30+ mock.AssertExpectations(t)
31+ })
32+
33+ return mock
34+}
35+
36+// mockdetector is an autogenerated mock type for the detector type
37+type mockdetector struct {
38+ mock.Mock
39+}
40+
41+type mockdetector_Expecter struct {
42+ mock *mock.Mock
43+}
44+
45+func (_m *mockdetector) EXPECT() *mockdetector_Expecter {
46+ return &mockdetector_Expecter{mock: &_m.Mock}
47+}
48+
49+// Close provides a mock function for the type mockdetector
50+func (_mock *mockdetector) Close() error {
51+ ret := _mock.Called()
52+
53+ if len(ret) == 0 {
54+ panic("no return value specified for Close")
55+ }
56+
57+ var r0 error
58+ if returnFunc, ok := ret.Get(0).(func() error); ok {
59+ r0 = returnFunc()
60+ } else {
61+ r0 = ret.Error(0)
62+ }
63+ return r0
64+}
65+
66+// mockdetector_Close_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Close'
67+type mockdetector_Close_Call struct {
68+ *mock.Call
69+}
70+
71+// Close is a helper method to define mock.On call
72+func (_e *mockdetector_Expecter) Close() *mockdetector_Close_Call {
73+ return &mockdetector_Close_Call{Call: _e.mock.On("Close")}
74+}
75+
76+func (_c *mockdetector_Close_Call) Run(run func()) *mockdetector_Close_Call {
77+ _c.Call.Run(func(args mock.Arguments) {
78+ run()
79+ })
80+ return _c
81+}
82+
83+func (_c *mockdetector_Close_Call) Return(err error) *mockdetector_Close_Call {
84+ _c.Call.Return(err)
85+ return _c
86+}
87+
88+func (_c *mockdetector_Close_Call) RunAndReturn(run func() error) *mockdetector_Close_Call {
89+ _c.Call.Return(run)
90+ return _c
91+}
92+
93+// Detect provides a mock function for the type mockdetector
94+func (_mock *mockdetector) Detect(ctx context.Context, f detFrame) ([]detection, error) {
95+ ret := _mock.Called(ctx, f)
96+
97+ if len(ret) == 0 {
98+ panic("no return value specified for Detect")
99+ }
100+
101+ var r0 []detection
102+ var r1 error
103+ if returnFunc, ok := ret.Get(0).(func(context.Context, detFrame) ([]detection, error)); ok {
104+ return returnFunc(ctx, f)
105+ }
106+ if returnFunc, ok := ret.Get(0).(func(context.Context, detFrame) []detection); ok {
107+ r0 = returnFunc(ctx, f)
108+ } else {
109+ if ret.Get(0) != nil {
110+ r0 = ret.Get(0).([]detection)
111+ }
112+ }
113+ if returnFunc, ok := ret.Get(1).(func(context.Context, detFrame) error); ok {
114+ r1 = returnFunc(ctx, f)
115+ } else {
116+ r1 = ret.Error(1)
117+ }
118+ return r0, r1
119+}
120+
121+// mockdetector_Detect_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Detect'
122+type mockdetector_Detect_Call struct {
123+ *mock.Call
124+}
125+
126+// Detect is a helper method to define mock.On call
127+// - ctx context.Context
128+// - f detFrame
129+func (_e *mockdetector_Expecter) Detect(ctx any, f any) *mockdetector_Detect_Call {
130+ return &mockdetector_Detect_Call{Call: _e.mock.On("Detect", ctx, f)}
131+}
132+
133+func (_c *mockdetector_Detect_Call) Run(run func(ctx context.Context, f detFrame)) *mockdetector_Detect_Call {
134+ _c.Call.Run(func(args mock.Arguments) {
135+ var arg0 context.Context
136+ if args[0] != nil {
137+ arg0 = args[0].(context.Context)
138+ }
139+ var arg1 detFrame
140+ if args[1] != nil {
141+ arg1 = args[1].(detFrame)
142+ }
143+ run(
144+ arg0,
145+ arg1,
146+ )
147+ })
148+ return _c
149+}
150+
151+func (_c *mockdetector_Detect_Call) Return(detectionMoqParams []detection, err error) *mockdetector_Detect_Call {
152+ _c.Call.Return(detectionMoqParams, err)
153+ return _c
154+}
155+
156+func (_c *mockdetector_Detect_Call) RunAndReturn(run func(ctx context.Context, f detFrame) ([]detection, error)) *mockdetector_Detect_Call {
157+ _c.Call.Return(run)
158+ return _c
159+}
A internal/adapters/hints/detector_proc_test.go +172 -0
@@ -0,0 +1,172 @@
1+package hints
2+
3+import (
4+ "context"
5+ "io"
6+ "os"
7+ "os/exec"
8+ "strconv"
9+ "strings"
10+ "testing"
11+ "time"
12+
13+ "github.com/stretchr/testify/require"
14+)
15+
16+// The detector child of these tests is the test binary itself, re-executed:
17+// a real process, started with the detector's own arguments. TestMain turns
18+// it into helperChild when helperEnv is set.
19+const (
20+ helperEnv = "NEFERWL_HINTS_HELPER"
21+ helperPIDFile = "NEFERWL_HINTS_HELPER_PIDFILE"
22+ helperChildPID = "NEFERWL_HINTS_HELPER_CHILDPID"
23+)
24+
25+func TestMain(m *testing.M) {
26+ if mode := os.Getenv(helperEnv); mode != "" {
27+ helperChild(mode)
28+ os.Exit(0)
29+ }
30+ os.Exit(m.Run())
31+}
32+
33+func helperHello() []byte {
34+ return cat(magicHello[:], u32(protocolVersion), u32(0), u32(4), []byte("test"), u32(imgsz))
35+}
36+
37+// helperChild plays the detector for mode:
38+//
39+// nohello never says hello
40+// silent says hello, then never answers
41+// grandchild like nohello, after spawning a grandchild that shares the
42+// child's stdout, stderr and process group
43+// ok says hello and answers every request with no box
44+func helperChild(mode string) {
45+ if f := os.Getenv(helperChildPID); f != "" {
46+ _ = os.WriteFile(f, []byte(strconv.Itoa(os.Getpid())), 0o600)
47+ }
48+ if mode == "grandchild" {
49+ gc := exec.Command("sleep", "60")
50+ gc.Stdout, gc.Stderr = os.Stdout, os.Stderr
51+ if err := gc.Start(); err != nil {
52+ os.Exit(2)
53+ }
54+ _ = os.WriteFile(os.Getenv(helperPIDFile), []byte(strconv.Itoa(gc.Process.Pid)), 0o600)
55+ mode = "nohello"
56+ }
57+ if mode != "nohello" {
58+ _, _ = os.Stdout.Write(helperHello())
59+ }
60+ if mode != "ok" {
61+ time.Sleep(time.Minute)
62+ return
63+ }
64+ for {
65+ var h [40]byte
66+ if _, err := io.ReadFull(os.Stdin, h[:]); err != nil {
67+ return
68+ }
69+ _, err := io.CopyN(io.Discard, os.Stdin, int64(le.Uint32(h[20:])*le.Uint32(h[24:])))
70+ if err != nil {
71+ return
72+ }
73+ _, _ = os.Stdout.Write(cat(magicResponse[:], u32(protocolVersion), h[8:16], u32(0), u32(0), u32(0)))
74+ }
75+}
76+
77+func helperDetector(t *testing.T, mode string) *procDetector {
78+ t.Helper()
79+ t.Setenv(helperEnv, mode)
80+ d := newProcDetector(os.Args[0], t.TempDir(), 0.05)
81+ d.helloTimeout, d.reqTimeout = 300*time.Millisecond, 300*time.Millisecond
82+ t.Cleanup(func() { _ = d.Close() })
83+ return d
84+}
85+
86+func tinyFrame() detFrame {
87+ return detFrame{width: 1, height: 1, stride: 4, m: &mapping{b: make([]byte, 4), live: new(int)}}
88+}
89+
90+// processGone reports whether pid is dead (or a zombie waiting for its
91+// reaper).
92+func processGone(pid int) bool {
93+ b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
94+ if err != nil {
95+ return true
96+ }
97+ s := string(b)
98+ return strings.HasPrefix(s[strings.LastIndexByte(s, ')')+2:], "Z")
99+}
100+
101+func readPID(t *testing.T, file string) int {
102+ t.Helper()
103+ var pid int
104+ require.Eventually(t, func() bool {
105+ b, err := os.ReadFile(file)
106+ if err != nil {
107+ return false
108+ }
109+ pid, err = strconv.Atoi(string(b))
110+ return err == nil
111+ }, time.Second, 10*time.Millisecond)
112+ return pid
113+}
114+
115+func TestProcDetectorHungChildIsKilledAndRestarted(t *testing.T) {
116+ for _, mode := range []string{"nohello", "silent"} {
117+ t.Run(mode, func(t *testing.T) {
118+ pidFile := t.TempDir() + "/child"
119+ t.Setenv(helperChildPID, pidFile)
120+ d := helperDetector(t, mode)
121+ ctx := context.Background()
122+ start := time.Now()
123+ _, err := d.Detect(ctx, tinyFrame())
124+ require.ErrorContains(t, err, "timed out")
125+ if mode == "nohello" {
126+ require.ErrorContains(t, err, "hello")
127+ }
128+ require.Less(t, time.Since(start), 2*time.Second)
129+ require.Nil(t, d.proc)
130+ pid := readPID(t, pidFile)
131+ require.True(t, processGone(pid), "child %d survives", pid)
132+
133+ // The next Detect starts a new, healthy child.
134+ t.Setenv(helperEnv, "ok")
135+ t.Setenv(helperChildPID, "")
136+ _, err = d.Detect(ctx, tinyFrame())
137+ require.NoError(t, err)
138+ require.NotNil(t, d.proc)
139+ start = time.Now()
140+ require.NoError(t, d.Close())
141+ require.Less(t, time.Since(start), 3*time.Second)
142+ })
143+ }
144+}
145+
146+// A grandchild holding the child's stdout and stderr open must neither
147+// survive the kill nor keep Wait blocked.
148+func TestProcDetectorKillsGrandchild(t *testing.T) {
149+ pidFile := t.TempDir() + "/pid"
150+ t.Setenv(helperPIDFile, pidFile)
151+ d := helperDetector(t, "grandchild")
152+ start := time.Now()
153+ _, err := d.Detect(context.Background(), tinyFrame())
154+ require.ErrorContains(t, err, "hello timed out")
155+ require.Less(t, time.Since(start), 2500*time.Millisecond)
156+ pid := readPID(t, pidFile)
157+ require.Eventually(t, func() bool { return processGone(pid) }, 2*time.Second, 20*time.Millisecond, "grandchild %d survives", pid)
158+ require.Nil(t, d.proc)
159+}
160+
161+// Cancelling the context interrupts a pending read.
162+func TestProcDetectorContextCancel(t *testing.T) {
163+ d := helperDetector(t, "silent")
164+ d.reqTimeout = time.Minute
165+ ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
166+ defer cancel()
167+ start := time.Now()
168+ _, err := d.Detect(ctx, tinyFrame())
169+ require.ErrorIs(t, err, context.DeadlineExceeded)
170+ require.Less(t, time.Since(start), 2*time.Second)
171+ require.Nil(t, d.proc)
172+}
A internal/adapters/hints/detector_test.go +335 -0
@@ -0,0 +1,335 @@
1+package hints
2+
3+import (
4+ "bytes"
5+ "context"
6+ "encoding/binary"
7+ "fmt"
8+ "io"
9+ "math"
10+ "net"
11+ "os"
12+ "path/filepath"
13+ "testing"
14+ "testing/synctest"
15+ "time"
16+
17+ "github.com/stretchr/testify/require"
18+)
19+
20+// u32, u64 and f32 build the byte fixtures of the protocol.
21+func u32(v uint32) []byte { return binary.LittleEndian.AppendUint32(nil, v) }
22+func u64(v uint64) []byte { return binary.LittleEndian.AppendUint64(nil, v) }
23+func f32(v float32) []byte { return u32(math.Float32bits(v)) }
24+func cat(parts ...[]byte) []byte {
25+ return bytes.Join(parts, nil)
26+}
27+
28+func TestWriteRequest(t *testing.T) {
29+ var buf bytes.Buffer
30+ pix := []byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
31+ require.NoError(t, writeRequest(&buf, 0x0102030405, detFrame{width: 2, height: 1, stride: 12, m: &mapping{b: append(pix, 99)}}))
32+ want := cat([]byte("NHRQ"), u32(1), u64(0x0102030405), u32(2), u32(1), u32(12), u32(1), u32(0), f32(0), pix)
33+ require.Equal(t, want, buf.Bytes())
34+}
35+
36+func TestWriteRequestRegions(t *testing.T) {
37+ var buf bytes.Buffer
38+ pix := []byte{1, 2, 3, 4}
39+ f := detFrame{width: 1, height: 1, stride: 4, m: &mapping{b: pix}, regions: []region{{0, 1, 2, 3}, {4, 5, 6, 7}}}
40+ require.NoError(t, writeRequest(&buf, 1, f))
41+ want := cat([]byte("NHRQ"), u32(1), u64(1), u32(1), u32(1), u32(4), u32(1), u32(2),
42+ u32(0), u32(1), u32(2), u32(3), u32(4), u32(5), u32(6), u32(7), f32(0), pix)
43+ require.Equal(t, want, buf.Bytes())
44+}
45+
46+func TestReadHello(t *testing.T) {
47+ dev, err := readHello(bytes.NewReader(cat([]byte("NHNT"), u32(1), u32(0), u32(6), []byte("Vulkan"), u32(1280))))
48+ require.NoError(t, err)
49+ require.Equal(t, "Vulkan", dev)
50+
51+ _, err = readHello(bytes.NewReader(cat([]byte("NHNT"), u32(1), u32(1), u32(4), []byte("boom"))))
52+ require.ErrorIs(t, err, errDetector)
53+ require.ErrorContains(t, err, "boom")
54+
55+ _, err = readHello(bytes.NewReader(cat([]byte("XXXX"), u32(1))))
56+ require.ErrorIs(t, err, errProtocol)
57+ _, err = readHello(bytes.NewReader(cat([]byte("NHNT"), u32(2), u32(0))))
58+ require.ErrorIs(t, err, errProtocol)
59+ _, err = readHello(bytes.NewReader(cat([]byte("NHNT"), u32(1), u32(0), u32(1<<20))))
60+ require.ErrorIs(t, err, errProtocol)
61+ _, err = readHello(bytes.NewReader([]byte("NHNT")))
62+ require.Error(t, err)
63+}
64+
65+func TestReadResponse(t *testing.T) {
66+ head := func(seq uint64, status uint32) []byte { return cat([]byte("NHRS"), u32(1), u64(seq), u32(status)) }
67+
68+ ok := cat(head(5, 0), f32(12.5), u32(2),
69+ f32(1), f32(2), f32(30), f32(40), f32(0.9), u32(0),
70+ f32(5), f32(6), f32(7), f32(8), f32(0.25), u32(3))
71+ got, err := readResponse(bytes.NewReader(ok), 5)
72+ require.NoError(t, err)
73+ require.Equal(t, []detection{{x1: 1, y1: 2, x2: 30, y2: 40, score: 0.9}, {x1: 5, y1: 6, x2: 7, y2: 8, score: 0.25, region: 3}}, got)
74+
75+ got, err = readResponse(bytes.NewReader(cat(head(5, 0), f32(1), u32(0))), 5)
76+ require.NoError(t, err)
77+ require.Empty(t, got)
78+
79+ _, err = readResponse(bytes.NewReader(cat(head(5, 1), u32(3), []byte("bad"))), 5)
80+ require.ErrorIs(t, err, errDetector)
81+ require.ErrorContains(t, err, "bad")
82+
83+ _, err = readResponse(bytes.NewReader(ok), 6) // wrong sequence
84+ require.ErrorIs(t, err, errProtocol)
85+ _, err = readResponse(bytes.NewReader(cat([]byte("NHRX"), u32(1))), 5)
86+ require.ErrorIs(t, err, errProtocol)
87+ _, err = readResponse(bytes.NewReader(cat([]byte("NHRS"), u32(9), u64(5), u32(0))), 5)
88+ require.ErrorIs(t, err, errProtocol)
89+ _, err = readResponse(bytes.NewReader(cat(head(5, 0), f32(1), u32(1<<30))), 5)
90+ require.ErrorIs(t, err, errProtocol)
91+ _, err = readResponse(bytes.NewReader(ok[:len(ok)-3]), 5) // truncated
92+ require.Error(t, err)
93+}
94+
95+func TestLineLog(t *testing.T) {
96+ var lines []string
97+ l := &lineLog{log: func(s string) { lines = append(lines, s) }}
98+ _, _ = l.Write([]byte("one\ntw"))
99+ _, _ = l.Write([]byte("o\n\nthree"))
100+ require.Equal(t, []string{"one", "two"}, lines)
101+}
102+
103+// The real detector, when the environment names it:
104+// NEFERWL_TEST_DETECTOR (binary) and NEFERWL_TEST_DETECTOR_MODEL (models dir).
105+func TestProcDetectorIntegration(t *testing.T) {
106+ bin, model := os.Getenv("NEFERWL_TEST_DETECTOR"), os.Getenv("NEFERWL_TEST_DETECTOR_MODEL")
107+ if bin == "" || model == "" {
108+ t.Skip("NEFERWL_TEST_DETECTOR and NEFERWL_TEST_DETECTOR_MODEL not set")
109+ }
110+ d := newProcDetector(bin, model, 0.05)
111+ t.Cleanup(func() { _ = d.Close() })
112+ m := &mapping{b: make([]byte, 1280*720*4), live: new(int)}
113+ for i := range m.b {
114+ m.b[i] = 0xf0 // a blank light frame
115+ }
116+ f := detFrame{width: 1280, height: 720, stride: 1280 * 4, m: m}
117+ ctx := context.Background()
118+
119+ dets, err := d.Detect(ctx, f)
120+ require.NoError(t, err)
121+ require.Empty(t, dets) // nothing clickable on a blank frame
122+ first := d.proc
123+ require.NotNil(t, first)
124+ _, err = d.Detect(ctx, f)
125+ require.NoError(t, err)
126+ require.Same(t, first, d.proc, "the child is kept alive between scans")
127+
128+ // A dead child fails one scan, and the next one starts a new child.
129+ require.NoError(t, first.cmd.Process.Kill())
130+ time.Sleep(50 * time.Millisecond)
131+ _, err = d.Detect(ctx, f)
132+ require.Error(t, err)
133+ require.Nil(t, d.proc)
134+ _, err = d.Detect(ctx, f)
135+ require.NoError(t, err)
136+ require.NotSame(t, first, d.proc)
137+
138+ // A wrong geometry is the detector's own error: the child stays.
139+ proc := d.proc
140+ _, err = d.Detect(ctx, detFrame{width: 1280, height: 720, stride: 4, m: m})
141+ require.ErrorIs(t, err, errDetector)
142+ require.Same(t, proc, d.proc)
143+ require.NoError(t, d.Close())
144+}
145+
146+func TestProcDetectorMissingBinary(t *testing.T) {
147+ d := newProcDetector("/nonexistent/nefer-hints", t.TempDir(), 0.05)
148+ _, err := d.Detect(context.Background(), detFrame{width: 1, height: 1, stride: 4, m: &mapping{b: make([]byte, 4)}})
149+ require.Error(t, err)
150+ require.Nil(t, d.proc)
151+ require.NoError(t, d.Close())
152+}
153+
154+func TestProcDetectorCommand(t *testing.T) {
155+ for _, threshold := range []float64{0, 0.3} {
156+ d := newProcDetector("nefer-hints", "/models", threshold)
157+ want := []string{"nefer-hints", "--model", "/models", "--imgsz", "1280", "--gpu", "auto"}
158+ if threshold != 0 {
159+ want = append(want, "--conf", "0.3")
160+ }
161+ require.Equal(t, want, d.cmd().Args)
162+ }
163+}
164+
165+func TestResponseBoxLimit(t *testing.T) {
166+ _, err := readResponse(bytes.NewReader(cat([]byte("NHRS"), u32(1), u64(1), u32(0), f32(0), u32(64001))), 1)
167+ require.ErrorIs(t, err, errProtocol)
168+}
169+
170+func TestPrepareDetections(t *testing.T) {
171+ regions := []region{{0, 0, 100, 100}, {50, 0, 100, 100}}
172+ valid := detection{x1: 90, y1: 10, x2: 100, y2: 30, score: .5}
173+ invalid := detection{x1: 80, y1: 0, x2: 120, y2: 40, score: float32(math.NaN()), region: 1}
174+ badRegion := valid
175+ badRegion.region = 2
176+ got := prepareDetections([]detection{valid, invalid, badRegion}, regions, 150, 100)
177+ require.Equal(t, []detection{valid}, dropCutBoxes(got, regions))
178+ many := make([]detection, maxWorkBoxes+5)
179+ for i := range many {
180+ many[i] = valid
181+ many[i].score = float32(i) / float32(len(many))
182+ }
183+ got = prepareDetections(many, nil, 150, 100)
184+ require.Len(t, got, maxWorkBoxes)
185+ require.Equal(t, many[len(many)-1].score, got[0].score)
186+ require.Equal(t, many[5].score, got[len(got)-1].score)
187+}
188+
189+// TestDetectorChild is a real subprocess speaking just enough protocol to
190+// synchronize cancellation without timers or project-interface doubles.
191+func TestDetectorChild(t *testing.T) {
192+ mode := os.Getenv("HINT_CHILD_MODE")
193+ if mode == "" {
194+ return
195+ }
196+ conn, err := net.Dial("unix", os.Getenv("HINT_CHILD_SOCKET"))
197+ if err != nil {
198+ os.Exit(2)
199+ }
200+ defer conn.Close()
201+ if mode != "hello" {
202+ _, _ = os.Stdout.Write(cat([]byte("NHNT"), u32(1), u32(0), u32(3), []byte("cpu"), u32(1280)))
203+ }
204+ if mode == "request" {
205+ b := make([]byte, 44) // request header, confidence and one BGRA pixel
206+ if _, err := io.ReadFull(os.Stdin, b); err != nil {
207+ os.Exit(3)
208+ }
209+ }
210+ _, _ = conn.Write([]byte{1})
211+ _, _ = io.Copy(io.Discard, os.Stdin)
212+ os.Exit(0)
213+}
214+
215+func childDetector(t *testing.T, mode string) (*procDetector, net.Listener) {
216+ t.Helper()
217+ dir := t.TempDir()
218+ socket := filepath.Join(dir, "ready")
219+ listener, err := net.Listen("unix", socket)
220+ require.NoError(t, err)
221+ t.Cleanup(func() { _ = listener.Close() })
222+ t.Setenv("HINT_CHILD_MODE", mode)
223+ t.Setenv("HINT_CHILD_SOCKET", socket)
224+ exe, err := os.Executable()
225+ require.NoError(t, err)
226+ script := filepath.Join(dir, "detector")
227+ require.NoError(t, os.WriteFile(script, fmt.Appendf(nil, "#!/bin/sh\nexec %q -test.run=^TestDetectorChild$\n", exe), 0700))
228+ d := newProcDetector(script, dir, 0)
229+ t.Cleanup(func() { _ = d.Close() })
230+ return d, listener
231+}
232+
233+func TestDetectorCancellation(t *testing.T) {
234+ for _, mode := range []string{"hello", "request"} {
235+ t.Run(mode, func(t *testing.T) {
236+ d, l := childDetector(t, mode)
237+ ctx, cancel := context.WithCancel(context.Background())
238+ defer cancel()
239+ done := make(chan error, 1)
240+ go func() {
241+ _, err := d.Detect(ctx, detFrame{width: 1, height: 1, stride: 4, m: &mapping{b: make([]byte, 4)}})
242+ done <- err
243+ }()
244+ require.NoError(t, l.(*net.UnixListener).SetDeadline(time.Now().Add(5*time.Second)))
245+ conn, err := l.Accept()
246+ require.NoError(t, err)
247+ require.NoError(t, conn.SetReadDeadline(time.Now().Add(5*time.Second)))
248+ var ready [1]byte
249+ _, err = io.ReadFull(conn, ready[:])
250+ _ = conn.Close()
251+ require.NoError(t, err)
252+ cancel()
253+ select {
254+ case err := <-done:
255+ require.ErrorIs(t, err, context.Canceled)
256+ case <-time.After(5 * time.Second):
257+ t.Fatal("cancellation did not finish")
258+ }
259+ })
260+ }
261+}
262+
263+func TestDetectorCloseClosesOutput(t *testing.T) {
264+ d, l := childDetector(t, "close")
265+ require.NoError(t, d.start(context.Background()))
266+ conn, err := l.Accept()
267+ require.NoError(t, err)
268+ _ = conn.Close()
269+ p := d.proc
270+ require.NoError(t, d.Close())
271+ _, err = p.out.Read(make([]byte, 1))
272+ require.ErrorIs(t, err, os.ErrClosed)
273+ require.NoError(t, d.Close())
274+}
275+
276+// A complete buffered response must not override an already launched context
277+// callback, including a detector-reported error that normally keeps the child.
278+func TestDetectorCanceledBufferedResponse(t *testing.T) {
279+ for _, status := range []uint32{0, 1} {
280+ t.Run(fmt.Sprint(status), func(t *testing.T) {
281+ d, l := childDetector(t, "close")
282+ require.NoError(t, d.start(context.Background()))
283+ conn, err := l.Accept()
284+ require.NoError(t, err)
285+ defer conn.Close()
286+ p := d.proc
287+ response := cat([]byte("NHRS"), u32(1), u64(1), u32(status))
288+ if status == 0 {
289+ response = cat(response, f32(0), u32(0))
290+ } else {
291+ response = cat(response, u32(3), []byte("bad"))
292+ }
293+ p.rd.Reset(bytes.NewReader(response))
294+ ctx, cancel := context.WithCancel(context.Background())
295+ cancel()
296+ _, err = d.Detect(ctx, detFrame{width: 1, height: 1, stride: 4, m: &mapping{b: make([]byte, 4)}})
297+ require.ErrorIs(t, err, context.Canceled)
298+ require.Nil(t, d.proc)
299+ require.True(t, p.reaped)
300+ })
301+ }
302+}
303+
304+func TestInterruptDeadlineBoundary(t *testing.T) {
305+ for _, cancelContext := range []bool{false, true} {
306+ t.Run(fmt.Sprint(cancelContext), func(t *testing.T) {
307+ synctest.Test(t, func(t *testing.T) {
308+ inR, inW, err := os.Pipe()
309+ require.NoError(t, err)
310+ defer inR.Close()
311+ outR, outW, err := os.Pipe()
312+ require.NoError(t, err)
313+ defer outW.Close()
314+ // Already reaped: only the real pipe endpoints remain to interrupt.
315+ p := &process{in: inW, out: outR, reaped: true}
316+ ctx, cancel := context.WithCancel(context.Background())
317+ defer cancel()
318+ stop := interruptOnDeadline(ctx, time.Second, p)
319+ if cancelContext {
320+ cancel()
321+ } else {
322+ time.Sleep(time.Second)
323+ }
324+ synctest.Wait()
325+ timedOut, canceled := stop()
326+ require.Equal(t, !cancelContext, timedOut)
327+ require.Equal(t, cancelContext, canceled)
328+ _, err = inW.Write([]byte{1})
329+ require.ErrorIs(t, err, os.ErrClosed)
330+ _, err = outR.Read(make([]byte, 1))
331+ require.ErrorIs(t, err, os.ErrClosed)
332+ })
333+ })
334+ }
335+}
M internal/adapters/hints/dump.go +38 -79
@@ -7,6 +7,7 @@ import (
7 "image"7 "image"
8 "image/color"8 "image/color"
9 "image/png"9 "image/png"
10+ "math"
10 "os"11 "os"
11 "path/filepath"12 "path/filepath"
12 "sort"13 "sort"
@@ -25,78 +26,54 @@ const (
25 26
26 // Options tunes Run.27 // Options tunes Run.
27 type Options struct {28 type Options struct {
29+ // Command is the nefer-hints binary: a path, or a name looked up in PATH.
30+ Command string
31+ // ModelDir is the directory of the detector's model files.
32+ ModelDir string
33+ // Threshold overrides the minimum detection score, 0.01 to 0.99.
34+ // Zero uses model.json or, if absent, the runner default.
35+ Threshold float64
28 // DumpDir, when not empty, receives the scanned frames (PNG, annotated36 // DumpDir, when not empty, receives the scanned frames (PNG, annotated
29 // PNG and JSON). They contain the screen in clear.37 // PNG and JSON). They contain the screen in clear.
30 DumpDir string38 DumpDir string
31- // Source, when not nil, finds targets in the accessibility tree of the39+
32- // windows on screen; OCR reads the rest.40+ // detector replaces the subprocess in tests.
33- Source targetSource41+ detector detector
34 }42 }
35 43
36 // dumpJob is one scan to write. pix is the capture buffer of the scan, owned44 // dumpJob is one scan to write. pix is the capture buffer of the scan, owned
37 // by the job: writeDump converts it in place.45 // by the job: writeDump converts it in place.
38 type dumpJob struct {46 type dumpJob struct {
39- when time.Time47+ when time.Time
40- req ports.HintScanRequest48+ req ports.HintScanRequest
41- pix []byte49+ pix []byte
42- m *mapping // owns pix; released after the dump50+ m *mapping // owns pix; released after the dump
43- stride int51+ stride int
44- targets []ports.HintTarget52+ targets []ports.HintTarget
45- crops []image.Rectangle53+ dets []detection // detector boxes, physical pixels
46- words []OCRWord
47- // atspi is the number of leading targets that come from the
48- // accessibility tree; the others come from OCR.
49- atspi int
50 captureD time.Duration54 captureD time.Duration
51- scanD time.Duration55+ detectD time.Duration
52- atspiD time.Duration
53 }56 }
54 57
55 type dumpTarget struct {58 type dumpTarget struct {
56- Kind string `json:"kind"`59+ Rect [4]int `json:"rect"` // x1, y1, x2, y2, logical output-local
57- Source string `json:"source"`60+ Score float32 `json:"score"`
58- Rect [4]int `json:"rect"`
59 }61 }
60 62
61 type dumpMeta struct {63 type dumpMeta struct {
62- Output string `json:"output"`64+ Output string `json:"output"`
63- Width int `json:"width"`65+ Width int `json:"width"`
64- Height int `json:"height"`66+ Height int `json:"height"`
65- Stride int `json:"stride"`67+ Stride int `json:"stride"`
66- Scale float64 `json:"scale"`68+ Scale float64 `json:"scale"`
67- LinksOnly bool `json:"links_only"`69+ CaptureMS float64 `json:"capture_ms"`
68- CaptureMS float64 `json:"capture_ms"`70+ DetectMS float64 `json:"detect_ms"`
69- ScanMS float64 `json:"scan_ms"`71+ TotalMS float64 `json:"total_ms"`
70- AtspiMS float64 `json:"atspi_ms"`72+ Targets []dumpTarget `json:"targets"`
71- Crops []image.Rectangle `json:"ocr_crops"`
72- Words []OCRWord `json:"words"`
73- Targets []dumpTarget `json:"targets"`
74-}
75-
76-func kindName(k ports.HintKind) string {
77- switch k {
78- case ports.HintLink:
79- return "link"
80- case ports.HintButton:
81- return "button"
82- default:
83- return "text"
84- }
85 }73 }
86 74
87-func kindColor(k ports.HintKind) color.RGBA {75+// boxColor is the outline of the detector boxes.
88- switch k {76+var boxColor = color.RGBA{0x00, 0xc8, 0x53, 0xff}
89- case ports.HintLink:
90- return color.RGBA{0x00, 0xc8, 0x53, 0xff}
91- case ports.HintButton:
92- return color.RGBA{0x29, 0x62, 0xff, 0xff}
93- default:
94- return color.RGBA{0x80, 0x80, 0x80, 0xff}
95- }
96-}
97-
98-// atspiColor is the outline of targets found in the accessibility tree.
99-var atspiColor = color.RGBA{0xff, 0x00, 0xff, 0xff}
100 77
101 func ms(d time.Duration) float64 { return float64(d.Microseconds()) / 1000 }78 func ms(d time.Duration) float64 { return float64(d.Microseconds()) / 1000 }
102 79
@@ -121,21 +98,8 @@ func writeDump(dir string, j *dumpJob) error {
121 return err98 return err
122 }99 }
123 100
124- scale := j.req.Scale101+ for _, d := range j.dets {
125- if scale <= 0 {102+ outline(img, image.Rect(int(d.x1), int(d.y1), int(math.Ceil(float64(d.x2))), int(math.Ceil(float64(d.y2)))), boxColor)
126- scale = 1
127- }
128- for _, r := range j.crops {
129- outline(img, r, color.RGBA{0x80, 0x80, 0x80, 0xff})
130- }
131- for i, t := range j.targets {
132- r := image.Rect(int(float64(t.Rect.X)*scale), int(float64(t.Rect.Y)*scale),
133- int(float64(t.Rect.X+t.Rect.W)*scale), int(float64(t.Rect.Y+t.Rect.H)*scale))
134- c := kindColor(t.Kind)
135- if i < j.atspi {
136- c = atspiColor
137- }
138- outline(img, r, c)
139 }103 }
140 if err := writePNG(prefix+".annotated.png", img); err != nil {104 if err := writePNG(prefix+".annotated.png", img); err != nil {
141 return err105 return err
@@ -143,17 +107,12 @@ func writeDump(dir string, j *dumpJob) error {
143 107
144 meta := dumpMeta{108 meta := dumpMeta{
145 Output: j.req.Output, Width: j.req.Width, Height: j.req.Height, Stride: j.stride,109 Output: j.req.Output, Width: j.req.Width, Height: j.req.Height, Stride: j.stride,
146- Scale: j.req.Scale, LinksOnly: j.req.LinksOnly,110+ Scale: j.req.Scale,
147- CaptureMS: ms(j.captureD), ScanMS: ms(j.scanD), AtspiMS: ms(j.atspiD),111+ CaptureMS: ms(j.captureD), DetectMS: ms(j.detectD), TotalMS: ms(j.captureD + j.detectD),
148- Crops: j.crops, Words: j.words,
149 Targets: make([]dumpTarget, 0, len(j.targets)),112 Targets: make([]dumpTarget, 0, len(j.targets)),
150 }113 }
151- for i, t := range j.targets {114+ for _, t := range j.targets {
152- src := "ocr"115+ meta.Targets = append(meta.Targets, dumpTarget{Rect: [4]int{t.Rect.X, t.Rect.Y, t.Rect.X + t.Rect.W, t.Rect.Y + t.Rect.H}, Score: t.Score})
153- if i < j.atspi {
154- src = "atspi"
155- }
156- meta.Targets = append(meta.Targets, dumpTarget{Kind: kindName(t.Kind), Source: src, Rect: [4]int{t.Rect.X, t.Rect.Y, t.Rect.X + t.Rect.W, t.Rect.Y + t.Rect.H}})
157 }116 }
158 data, err := json.Marshal(meta)117 data, err := json.Marshal(meta)
159 if err != nil {118 if err != nil {
M internal/adapters/hints/hints.go +99 -292
@@ -1,15 +1,16 @@
1 // Package hints finds clickable targets for hint mode: it captures the last1 // Package hints finds clickable targets for hint mode: it captures the last
2-// frame of an output through the capture pipeline and reads uncovered windows with OCR.2+// frame of an output through the capture pipeline and sends it to the
3+// nefer-hints detector, a persistent subprocess.
3 package hints4 package hints
4 5
5 import (6 import (
7+ "cmp"
6 "context"8 "context"
7 "errors"9 "errors"
8 "fmt"10 "fmt"
9- "hash/maphash"
10 "image"11 "image"
11 "math"12 "math"
12- "strings"13+ "slices"
13 "time"14 "time"
14 15
15 "github.com/bnema/neferwl/internal/adapters/logging"16 "github.com/bnema/neferwl/internal/adapters/logging"
@@ -39,39 +40,18 @@ func Run(ctx context.Context, requests <-chan ports.HintScanRequest, captures ch
39 return run(ctx, requests, captures, captured, results, captureTimeout, opts)40 return run(ctx, requests, captures, captured, results, captureTimeout, opts)
40 }41 }
41 42
42-// targetSource finds hint targets in the accessibility tree of windows. It
43-// returns the targets (logical output-local) and the windows they cover.
44-type targetSource interface {
45- Targets(ctx context.Context, windows []ports.HintWindow, linksOnly bool) ([]ports.HintTarget, map[ports.WindowID]bool, error)
46- // Close releases the source. It is called once, by the worker goroutine,
47- // when the worker returns; the source is not safe for concurrent use.
48- Close() error
49-}
50-
51-// atspiResult is what the source found for one request.
52-type atspiResult struct {
53- targets []ports.HintTarget
54- covered map[ports.WindowID]bool
55- dur time.Duration
56- err error
57-}
58-
59 type worker struct {43 type worker struct {
60- source targetSource44+ det detector
61- cache map[bandKey]bandCache45+ n uint64
62- cacheSeed maphash.Seed46+ dumpDir string
63- ocr ocrState47+ pending *dumpJob
64- warnedSource bool
65- n uint64
66- dumpDir string
67- pending *dumpJob
68 // orphans are capture buffers the output may still write: they are48 // orphans are capture buffers the output may still write: they are
69 // released when its CaptureDone finally arrives.49 // released when its CaptureDone finally arrives.
70 orphans map[uint64]*mapping50 orphans map[uint64]*mapping
71 live int // mappings not released yet51 live int // mappings not released yet
72 }52 }
73 53
74-// frame is the captured output: BGRA pixels, nil when no capture was needed.54+// frame is the captured output: BGRA pixels.
75 type frame struct {55 type frame struct {
76 pix []byte56 pix []byte
77 stride int57 stride int
@@ -79,11 +59,12 @@ type frame struct {
79 }59 }
80 60
81 func run(ctx context.Context, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, timeout time.Duration, opts Options) error {61 func run(ctx context.Context, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, timeout time.Duration, opts Options) error {
82- w := &worker{dumpDir: opts.DumpDir, source: opts.Source}62+ w := &worker{dumpDir: opts.DumpDir, det: opts.detector}
83- if opts.Source != nil {63+ if w.det == nil {
84- // The worker goroutine is the only one to touch the source.64+ w.det = newProcDetector(opts.Command, opts.ModelDir, opts.Threshold)
85- defer func() { _ = opts.Source.Close() }()
86 }65 }
66+ // The worker goroutine is the only one to touch the detector.
67+ defer func() { _ = w.det.Close() }()
87 for {68 for {
88 select {69 select {
89 case <-ctx.Done():70 case <-ctx.Done():
@@ -130,25 +111,6 @@ func send(ctx context.Context, results chan<- ports.HintScanResult, res ports.Hi
130 }111 }
131 }112 }
132 113
133-// needFrame reports whether the scan must capture the output: OCR has a
134-// window to read, or the debug dump wants the frame. With OCR unavailable
135-// (tesseract missing) and no dump, the accessibility targets are the result.
136-func (w *worker) needFrame(ctx context.Context, req ports.HintScanRequest, ar atspiResult) bool {
137- ocr := w.ocrReady(ctx)
138- if w.dumpDir != "" {
139- return true
140- }
141- if !ocr {
142- return false
143- }
144- for _, win := range req.Windows {
145- if !ar.covered[win.ID] && !windowCrop(win.Rect, req.Scale, req.Width, req.Height).Empty() {
146- return true
147- }
148- }
149- return false
150-}
151-
152 func (w *worker) serve(ctx context.Context, req ports.HintScanRequest, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, timeout time.Duration) ports.HintScanResult {114 func (w *worker) serve(ctx context.Context, req ports.HintScanRequest, requests <-chan ports.HintScanRequest, captures chan<- ports.CaptureRequest, captured <-chan ports.CaptureDone, results chan<- ports.HintScanResult, timeout time.Duration) ports.HintScanResult {
153 log := logging.For(ctx, "hints")115 log := logging.For(ctx, "hints")
154 res := ports.HintScanResult{Seq: req.Seq, Output: req.Output}116 res := ports.HintScanResult{Seq: req.Seq, Output: req.Output}
@@ -161,16 +123,8 @@ func (w *worker) serve(ctx context.Context, req ports.HintScanRequest, requests
161 if req.Width <= 0 || req.Height <= 0 || req.Width*req.Height > maxPixels {123 if req.Width <= 0 || req.Height <= 0 || req.Width*req.Height > maxPixels {
162 return fail(fmt.Errorf("%w: %dx%d", ErrBadRequest, req.Width, req.Height))124 return fail(fmt.Errorf("%w: %dx%d", ErrBadRequest, req.Width, req.Height))
163 }125 }
164- // The accessibility query is always waited for: the source is not shared
165- // between requests. The capture only starts once it is known whether any
166- // window needs OCR.
167- atspi := w.startSource(ctx, req)
168- defer func() { <-atspi.done }()
169- if !w.needFrame(ctx, req, atspi.wait()) {
170- return w.scan(ctx, req, frame{}, 0, res, atspi)
171- }
172 // The capture buffer is anonymous mapped memory, outside the Go heap. It126 // The capture buffer is anonymous mapped memory, outside the Go heap. It
173- // is released once the crops are extracted, or by the dump job.127+ // is released once the detector answered, or by the dump job.
174 m := w.alloc(req.Width * req.Height * 4)128 m := w.alloc(req.Width * req.Height * 4)
175 keep := false // the mapping outlives this call: dump job or pending capture129 keep := false // the mapping outlives this call: dump job or pending capture
176 defer func() {130 defer func() {
@@ -225,7 +179,7 @@ func (w *worker) serve(ctx context.Context, req ports.HintScanRequest, requests
225 if done.Err != nil {179 if done.Err != nil {
226 return fail(fmt.Errorf("hints: capture: %w", done.Err))180 return fail(fmt.Errorf("hints: capture: %w", done.Err))
227 }181 }
228- res = w.scan(ctx, req, frame{pix: buf, stride: stride, m: m}, time.Since(start), res, atspi)182+ res = w.scan(ctx, req, frame{pix: buf, stride: stride, m: m}, time.Since(start), res)
229 keep = w.pending != nil && w.pending.m == m183 keep = w.pending != nil && w.pending.m == m
230 return res184 return res
231 case other := <-requests:185 case other := <-requests:
@@ -255,253 +209,106 @@ func (w *worker) reap(id uint64) {
255 }209 }
256 }210 }
257 211
258-// bandJob is one OCR band of one window crop.212+// minTarget is the smallest target side kept, in logical pixels.
259-type bandJob struct {213+const minTarget = 4
260- win int // index into the scan's windows
261- key bandKey //
262- ext image.Rectangle // crop-local rows read, with context
263- own [2]int // crop-local rows owned: words centred there are kept
264- hash uint64
265- hit bool
266- gray *image.Gray
267- words []OCRWord
268- err error
269-}
270 214
271-func (w *worker) scan(ctx context.Context, req ports.HintScanRequest, fr frame, captureDur time.Duration, res ports.HintScanResult, atspi *pendingSource) ports.HintScanResult {215+// maxWorkBoxes bounds the quadratic cut-box and merge passes.
216+const maxWorkBoxes = 2000
217+
218+// scan sends the frame to the detector and turns the boxes into targets. It
219+// releases the frame, or hands it to the dump job.
220+func (w *worker) scan(ctx context.Context, req ports.HintScanRequest, fr frame, captureDur time.Duration, res ports.HintScanResult) ports.HintScanResult {
272 log := logging.For(ctx, "hints")221 log := logging.For(ctx, "hints")
273- ar := atspi.wait()
274- if ar.err != nil && !w.warnedSource {
275- log.Warn().Err(ar.err).Msg("accessibility targets unavailable; using OCR only")
276- w.warnedSource = true
277- }
278- out := make([]ports.HintTarget, 0, len(ar.targets))
279- for _, t := range ar.targets {
280- if !req.LinksOnly || t.Kind == ports.HintLink {
281- out = append(out, t)
282- }
283- }
284- nAtspi := len(out)
285 t0 := time.Now()222 t0 := time.Now()
286- if w.cacheSeed == (maphash.Seed{}) {223+ regions := tileRegions(req.Width, req.Height)
287- w.cacheSeed = maphash.MakeSeed()224+ dets, err := w.det.Detect(ctx, detFrame{width: req.Width, height: req.Height, stride: fr.stride, m: fr.m, regions: regions})
288- }225+ rawN := len(dets)
289- nextCache := make(map[bandKey]bandCache)226+ detectDur := time.Since(t0)
290- var crops []image.Rectangle227+ if err != nil {
291- var allWords []OCRWord228+ fr.m.unmap()
292- pixels, links := 0, 0229+ log.Info().Err(err).Str("output", req.Output).Float64("detect_ms", ms(detectDur)).Msg("hint scan failed")
293- ocrOn := fr.pix != nil && w.ocrReady(ctx)230+ res.Err = err
294- // Phase 1 reads the frame: the bands that miss the cache are copied, as231+ return res
295- // grayscale, into one arena. Phase 2 (OCR) does not need the frame.
296- var jobs []*bandJob
297- var rects []image.Rectangle // crop of each window that has bands
298- var arena *mapping
299- if ocrOn {
300- type pending struct {
301- r image.Rectangle
302- jobs []*bandJob
303- }
304- var wins []pending
305- need := 0
306- for _, win := range req.Windows {
307- if ar.covered[win.ID] {
308- continue
309- }
310- r := windowCrop(win.Rect, req.Scale, req.Width, req.Height)
311- if r.Empty() {
312- continue
313- }
314- crops = append(crops, r)
315- pixels += r.Dx() * r.Dy()
316- p := pending{r: r}
317- for _, b := range bandsOf(r.Dy()) {
318- ext := image.Rect(0, b.top, r.Dx(), b.bottom)
319- j := &bandJob{win: len(wins), ext: ext, own: [2]int{b.ownTop, b.ownBottom},
320- key: bandKey{output: req.Output, window: win.ID, band: b.idx, bands: b.n}}
321- j.hash = w.cropHash(fr.pix, fr.stride, ext.Add(r.Min))
322- entry, ok := w.cache[j.key]
323- if ok && entry.hash == j.hash && entry.ext == ext {
324- j.hit, j.words = true, entry.words
325- } else {
326- need += ext.Dx() * ext.Dy()
327- }
328- p.jobs = append(p.jobs, j)
329- }
330- wins = append(wins, p)
331- }
332- if need > 0 {
333- arena = w.alloc(need)
334- }
335- off := 0
336- for _, p := range wins {
337- rects = append(rects, p.r)
338- for _, j := range p.jobs {
339- if !j.hit {
340- n := j.ext.Dx() * j.ext.Dy()
341- j.gray = cropGray(fr.pix, fr.stride, j.ext.Add(p.r.Min), arena.b[off:off+n])
342- off += n
343- }
344- jobs = append(jobs, j)
345- }
346- }
347- } else if w.dumpDir != "" {
348- for _, win := range req.Windows {
349- if !ar.covered[win.ID] {
350- if r := windowCrop(win.Rect, req.Scale, req.Width, req.Height); !r.Empty() {
351- crops = append(crops, r)
352- }
353- }
354- }
355 }232 }
356- if w.dumpDir != "" && fr.pix != nil {233+ // Boxes cut by a tile's inner edge are dropped when a neighbour tile
234+ // saw the element whole; then, on the union of all tiles and after
235+ // rounding to logical pixels, duplicates and nested boxes are merged.
236+ dets = prepareDetections(dets, regions, req.Width, req.Height)
237+ res.Targets = mergeTargets(targetsOf(dropCutBoxes(dets, regions), req.Scale, req.Width, req.Height))
238+ if w.dumpDir != "" {
357 // The dump reads the frame after the result is sent and releases it.239 // The dump reads the frame after the result is sent and releases it.
358- w.pending = &dumpJob{when: time.Now(), req: req, pix: fr.pix, m: fr.m, stride: fr.stride, crops: crops, captureD: captureDur, atspiD: ar.dur}240+ w.pending = &dumpJob{when: time.Now(), req: req, pix: fr.pix, m: fr.m, stride: fr.stride, targets: res.Targets, dets: dets, captureD: captureDur, detectD: detectDur}
359 } else {241 } else {
360- fr.m.unmap() // the crops are extracted: give the frame back242+ fr.m.unmap()
361 }243 }
362- fr.pix = nil244+ log.Info().Str("output", req.Output).Float64("capture_ms", ms(captureDur)).Float64("detect_ms", ms(detectDur)).
363-245+ Int("regions", len(regions)).Int("raw_boxes", rawN).Int("prepared_boxes", len(dets)).Int("targets", len(res.Targets)).Msg("hint scan")
364- hits := 0
365- if len(jobs) > 0 {
366- w.recognize(ctx, jobs)
367- for _, j := range jobs {
368- if j.hit {
369- hits++
370- }
371- }
372- }
373- arena.unmap()
374-
375- // Merge the bands of each window, in order.
376- k := 0
377- for wi, r := range rects {
378- var words []OCRWord
379- for ; k < len(jobs) && jobs[k].win == wi; k++ {
380- j := jobs[k]
381- if j.err != nil {
382- failureWarning.Do(func() { log.Warn().Err(j.err).Msg("OCR unavailable; install tesseract and tesseract-data-eng") })
383- continue
384- }
385- nextCache[j.key] = bandCache{hash: j.hash, ext: j.ext, words: j.words}
386- for _, word := range j.words {
387- word.Rect = word.Rect.Add(r.Min)
388- if word.Underlined {
389- word.UnderL, word.UnderR = word.UnderL+r.Min.X, word.UnderR+r.Min.X
390- }
391- if !duplicateWord(words, word) {
392- words = append(words, word)
393- }
394- }
395- }
396- allWords = append(allWords, words...)
397- targets := textLinks(words, req.Scale)
398- links += len(targets)
399- out = append(out, targets...)
400- }
401- w.cache = nextCache
402- ocrDur := time.Since(t0)
403- res.Targets = out
404- if p := w.pending; p != nil {
405- p.targets, p.atspi, p.words, p.scanD = out, nAtspi, allWords, ocrDur
406- }
407- log.Info().Str("output", req.Output).
408- Float64("atspi_ms", ms(ar.dur)).Int("atspi_windows", len(ar.covered)).Int("atspi_targets", nAtspi).
409- Int("ocr_windows", len(rects)).Float64("ocr_ms", ms(ocrDur)).Int("ocr_px", pixels).
410- Int("ocr_bands", len(jobs)).Int("band_hits", hits).
411- Int("words", len(allWords)).Int("ocr_links", links).Bool("cache_hit", len(jobs) > 0 && hits == len(jobs)).
412- Int("targets", len(out)).Msg("hint scan")
413 return res246 return res
414 }247 }
415 248
416-// duplicateWord reports whether words already holds word (the same text at249+// targetsOf converts boxes in physical pixels to logical output-local
417-// the same place), as happens where two bands overlap.250+// targets. A box with a non-finite coordinate or score, a score outside
418-func duplicateWord(words []OCRWord, word OCRWord) bool {251+// [0,1] or no area is dropped; the others are clamped to the width x height
419- for _, old := range words {252+// frame (physical), rounded outwards to logical pixels, and dropped when
420- if strings.EqualFold(word.Text, old.Text) && overlap(word.Rect, old.Rect) > .5 {253+// under minTarget on a side.
421- return true254+func targetsOf(dets []detection, scale float64, width, height int) []ports.HintTarget {
422- }
423- }
424- return false
425-}
426-
427-// bandKey identifies one OCR band of a window on an output.
428-type bandKey struct {
429- output string
430- window ports.WindowID
431- band int
432- bands int
433-}
434-
435-// bandCache is the OCR result of a band whose pixels hash to hash. words are
436-// crop-local and only the ones the band owns.
437-type bandCache struct {
438- hash uint64
439- ext image.Rectangle
440- words []OCRWord
441-}
442-
443-// windowCrop rounds outwards in physical pixels, then intersects the output.
444-func windowCrop(r ports.Rect, scale float64, width, height int) image.Rectangle {
445 if scale <= 0 {255 if scale <= 0 {
446 scale = 1256 scale = 1
447 }257 }
448- if r.W <= 0 || r.H <= 0 {258+ fw, fh := float64(width), float64(height)
449- return image.Rectangle{}259+ out := make([]ports.HintTarget, 0, len(dets))
450- }260+ for _, d := range dets {
451- return image.Rect(int(math.Floor(float64(r.X)*scale)), int(math.Floor(float64(r.Y)*scale)),261+ v := [4]float64{float64(d.x1), float64(d.y1), float64(d.x2), float64(d.y2)}
452- int(math.Ceil(float64(r.X+r.W)*scale)), int(math.Ceil(float64(r.Y+r.H)*scale))).Intersect(image.Rect(0, 0, width, height))262+ ok := d.score >= 0 && d.score <= 1 // false for NaN too
453-}263+ for _, f := range v {
454-264+ ok = ok && !math.IsNaN(f) && !math.IsInf(f, 0)
455-// cropHash hashes the raw pixels of r without copying them.265+ }
456-func (w *worker) cropHash(buf []byte, stride int, r image.Rectangle) uint64 {266+ if !ok || v[2] <= v[0] || v[3] <= v[1] {
457- var h maphash.Hash267+ continue
458- h.SetSeed(w.cacheSeed)268+ }
459- for y := r.Min.Y; y < r.Max.Y; y++ {269+ // Clamp before any int conversion: huge finite values stay safe.
460- _, _ = h.Write(buf[y*stride+r.Min.X*4 : y*stride+r.Max.X*4])270+ x1, y1 := math.Max(0, math.Min(v[0], fw)), math.Max(0, math.Min(v[1], fh))
271+ x2, y2 := math.Max(0, math.Min(v[2], fw)), math.Max(0, math.Min(v[3], fh))
272+ lx1, ly1 := int(math.Floor(x1/scale)), int(math.Floor(y1/scale))
273+ lx2, ly2 := int(math.Ceil(x2/scale)), int(math.Ceil(y2/scale))
274+ if lx2-lx1 < minTarget || ly2-ly1 < minTarget {
275+ continue
276+ }
277+ out = append(out, ports.HintTarget{Rect: ports.Rect{X: lx1, Y: ly1, W: lx2 - lx1, H: ly2 - ly1}, Score: d.score})
461 }278 }
462- return h.Sum64()279+ return out
463 }280 }
464 281
465-// cropGray converts the BGRA pixels of r to 8-bit luma into dst, which must282+// prepareDetections validates and clamps boxes before any pairwise work,
466-// hold r.Dx()*r.Dy() bytes; only this band is converted, not the output.283+// preserving their region metadata for cut-edge suppression.
467-func cropGray(buf []byte, stride int, r image.Rectangle, dst []byte) *image.Gray {284+func prepareDetections(in []detection, regions []region, width, height int) []detection {
468- img := &image.Gray{Pix: dst[:r.Dx()*r.Dy()], Stride: r.Dx(), Rect: image.Rect(0, 0, r.Dx(), r.Dy())}285+ out := make([]detection, 0, len(in))
469- for y := 0; y < r.Dy(); y++ {286+ for _, d := range in {
470- src := buf[(r.Min.Y+y)*stride+r.Min.X*4:][:r.Dx()*4]287+ if d.region < 0 || d.region >= max(1, len(regions)) || !(d.score >= 0 && d.score <= 1) {
471- row := img.Pix[y*img.Stride:][:r.Dx()]288+ continue
472- for x := range row {289+ }
473- p := src[x*4 : x*4+4 : x*4+4]290+ valid := true
474- row[x] = uint8((uint32(p[2])*299 + uint32(p[1])*587 + uint32(p[0])*114 + 500) / 1000)291+ for _, v := range []float32{d.x1, d.y1, d.x2, d.y2} {
292+ if math.IsNaN(float64(v)) || math.IsInf(float64(v), 0) {
293+ valid = false
294+ break
295+ }
475 }296 }
297+ if !valid {
298+ continue
299+ }
300+ d.x1 = max(0, min(d.x1, float32(width)))
301+ d.x2 = max(0, min(d.x2, float32(width)))
302+ d.y1 = max(0, min(d.y1, float32(height)))
303+ d.y2 = max(0, min(d.y2, float32(height)))
304+ if d.x2 <= d.x1 || d.y2 <= d.y1 {
305+ continue
306+ }
307+ out = append(out, d)
476 }308 }
477- return img309+ slices.SortStableFunc(out, func(a, b detection) int { return cmp.Compare(b.score, a.score) })
478-}310+ if len(out) > maxWorkBoxes {
479-311+ out = out[:maxWorkBoxes]
480-// pendingSource is an accessibility query in flight.
481-type pendingSource struct {
482- done chan struct{}
483- res atspiResult
484-}
485-
486-// startSource starts the accessibility query of req, if there is a source and
487-// a window to ask about. done is closed when res is set.
488-func (w *worker) startSource(ctx context.Context, req ports.HintScanRequest) *pendingSource {
489- p := &pendingSource{done: make(chan struct{})}
490- if w.source == nil || len(req.Windows) == 0 {
491- close(p.done)
492- return p
493 }312 }
494- go func() {313+ return out
495- defer close(p.done)
496- t0 := time.Now()
497- targets, covered, err := w.source.Targets(ctx, req.Windows, req.LinksOnly)
498- p.res = atspiResult{targets: targets, covered: covered, dur: time.Since(t0), err: err}
499- }()
500- return p
501-}
502-
503-// wait returns the result of the query.
504-func (p *pendingSource) wait() atspiResult {
505- <-p.done
506- return p.res
507 }314 }
M internal/adapters/hints/hints_test.go +160 -178
@@ -1,15 +1,16 @@
1 package hints1 package hints
2 2
3 import (3 import (
4+ "bytes"
4 "context"5 "context"
5 "encoding/json"6 "encoding/json"
6 "errors"7 "errors"
7 "fmt"8 "fmt"
9+ "github.com/bnema/zerowrap"
8 "image"10 "image"
9- "image/draw"
10 _ "image/png"11 _ "image/png"
12+ "math"
11 "os"13 "os"
12- "os/exec"
13 "path/filepath"14 "path/filepath"
14 "strings"15 "strings"
15 "testing"16 "testing"
@@ -39,6 +40,13 @@ func startCapturing(t *testing.T, timeout time.Duration) *harness {
39 40
40 func startWith(t *testing.T, timeout time.Duration, opts Options) *harness {41 func startWith(t *testing.T, timeout time.Duration, opts Options) *harness {
41 t.Helper()42 t.Helper()
43+ if opts.detector == nil {
44+ // Tests of the capture flow do not care what the detector finds.
45+ det := newMockdetector(t)
46+ det.EXPECT().Detect(mock.Anything, mock.Anything).Return(nil, nil).Maybe()
47+ det.EXPECT().Close().Return(nil).Maybe()
48+ opts.detector = det
49+ }
42 h := &harness{50 h := &harness{
43 requests: make(chan ports.HintScanRequest, 1),51 requests: make(chan ports.HintScanRequest, 1),
44 captures: make(chan ports.CaptureRequest, 4),52 captures: make(chan ports.CaptureRequest, 4),
@@ -55,51 +63,99 @@ func startWith(t *testing.T, timeout time.Duration, opts Options) *harness {
55 return h63 return h
56 }64 }
57 65
58-func loadBGRA(t *testing.T) (pix []byte, w, hgt int) {
59- t.Helper()
60- f, err := os.Open("testdata/browser.png")
61- require.NoError(t, err)
62- defer f.Close()
63- img, _, err := image.Decode(f)
64- require.NoError(t, err)
65- b := img.Bounds()
66- rgba := image.NewRGBA(image.Rect(0, 0, b.Dx(), b.Dy()))
67- draw.Draw(rgba, rgba.Bounds(), img, b.Min, draw.Src)
68- for i := 0; i < len(rgba.Pix); i += 4 {
69- rgba.Pix[i], rgba.Pix[i+2] = rgba.Pix[i+2], rgba.Pix[i]
70- rgba.Pix[i+3] = 0xff
71- }
72- return rgba.Pix, b.Dx(), b.Dy()
73-}
74-
75 func TestHintScan(t *testing.T) {66 func TestHintScan(t *testing.T) {
76- if _, err := exec.LookPath("tesseract"); err != nil {67+ det := newMockdetector(t)
77- t.Skip("tesseract not on PATH")68+ det.EXPECT().Close().Return(nil).Once()
78- }69+ var got detFrame
79- pix, w, hgt := loadBGRA(t)70+ det.EXPECT().Detect(mock.Anything, mock.Anything).RunAndReturn(func(_ context.Context, f detFrame) ([]detection, error) {
80- h := start(t, 5*time.Second)71+ got = f
81-72+ return []detection{{x1: 10, y1: 20, x2: 50, y2: 30, score: 0.8}}, nil
82- // Plays the output owner.73+ }).Once()
74+ h := startWith(t, 5*time.Second, Options{detector: det})
83 go func() {75 go func() {
84 creq := <-h.captures76 creq := <-h.captures
85- copy(creq.Mapped, pix)
86 h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}77 h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
87 }()78 }()
88-79+ h.requests <- ports.HintScanRequest{Seq: 7, Output: "HEADLESS-1", Width: 64, Height: 32, Scale: 1}
89- h.requests <- ports.HintScanRequest{Seq: 7, Output: "HEADLESS-1", Width: w, Height: hgt, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: w, H: hgt}}}}
90 res := <-h.results80 res := <-h.results
91 require.NoError(t, res.Err)81 require.NoError(t, res.Err)
92 require.Equal(t, uint64(7), res.Seq)82 require.Equal(t, uint64(7), res.Seq)
93 require.Equal(t, "HEADLESS-1", res.Output)83 require.Equal(t, "HEADLESS-1", res.Output)
94- var links int84+ require.Equal(t, []ports.HintTarget{{Rect: ports.Rect{X: 10, Y: 20, W: 40, H: 10}, Score: 0.8}}, res.Targets)
95- for _, tg := range res.Targets {85+ require.Equal(t, 64, got.width)
96- require.Positive(t, tg.Rect.W)86+ require.Equal(t, 32, got.height)
97- require.Positive(t, tg.Rect.H)87+ require.Equal(t, 64*4, got.stride)
98- if tg.Kind == ports.HintLink {88+}
99- links++89+
90+func TestTargetsOf(t *testing.T) {
91+ nan := float32(math.NaN())
92+ inf := float32(math.Inf(1))
93+ r := func(x, y, w, h int, score float32) []ports.HintTarget {
94+ return []ports.HintTarget{{Rect: ports.Rect{X: x, Y: y, W: w, H: h}, Score: score}}
95+ }
96+ for _, tc := range []struct {
97+ name string
98+ det detection
99+ scale float64
100+ want []ports.HintTarget
101+ }{
102+ {"inside", detection{10, 20, 50, 40, 0.5, 0}, 1, r(10, 20, 40, 20, 0.5)},
103+ {"scale 2 rounds outwards, clamped at 100", detection{11, 21, 100.2, 61, 0.5, 0}, 2, r(5, 10, 45, 21, 0.5)},
104+ {"zero scale reads as 1", detection{0, 0, 10, 10, 0.5, 0}, 0, r(0, 0, 10, 10, 0.5)},
105+ {"exactly 4x4 kept", detection{0, 0, 8, 8, 0.4, 0}, 2, r(0, 0, 4, 4, 0.4)},
106+ {"3 logical wide dropped", detection{0, 0, 5, 40, 0.9, 0}, 2, nil},
107+ {"3 logical high dropped", detection{0, 0, 40, 5, 0.9, 0}, 2, nil},
108+ {"fully outside", detection{200, 10, 300, 50, 0.5, 0}, 1, nil},
109+ {"fully before origin", detection{-50, -50, -10, -10, 0.5, 0}, 1, nil},
110+ {"partly outside is clamped", detection{-20, 70, 150, 120, 0.5, 0}, 1, r(0, 70, 100, 30, 0.5)},
111+ {"partly outside, scale 2", detection{-20, 70, 150, 120, 0.5, 0}, 2, r(0, 35, 50, 15, 0.5)},
112+ {"inverted x", detection{50, 10, 20, 40, 0.5, 0}, 1, nil},
113+ {"inverted y", detection{10, 40, 50, 10, 0.5, 0}, 1, nil},
114+ {"empty x", detection{10, 10, 10, 40, 0.5, 0}, 1, nil},
115+ {"empty y", detection{10, 10, 40, 10, 0.5, 0}, 1, nil},
116+ {"NaN coordinate", detection{nan, 0, 50, 50, 0.5, 0}, 1, nil},
117+ {"Inf coordinate", detection{0, 0, inf, 50, 0.5, 0}, 1, nil},
118+ {"NaN score", detection{0, 0, 50, 50, nan, 0}, 1, nil},
119+ {"Inf score", detection{0, 0, 50, 50, inf, 0}, 1, nil},
120+ {"score above 1", detection{0, 0, 50, 50, 1.5, 0}, 1, nil},
121+ {"negative score", detection{0, 0, 50, 50, -0.1, 0}, 1, nil},
122+ {"score 0 and 1 kept", detection{0, 0, 50, 50, 1, 0}, 1, r(0, 0, 50, 50, 1)},
123+ {"huge coordinates", detection{-1e30, -1e30, 1e30, 1e30, 0.5, 0}, 1, r(0, 0, 100, 100, 0.5)},
124+ {"huge positive only", detection{10, 10, 1e30, 1e30, 0.5, 0}, 2, r(5, 5, 45, 45, 0.5)},
125+ {"huge outside", detection{1e30, 1e30, 3e30, 3e30, 0.5, 0}, 1, nil},
126+ } {
127+ t.Run(tc.name, func(t *testing.T) {
128+ got := targetsOf([]detection{tc.det}, tc.scale, 100, 100)
129+ if tc.want == nil {
130+ require.Empty(t, got)
131+ return
132+ }
133+ require.Equal(t, tc.want, got)
134+ })
135+ }
136+}
137+
138+// A detector error fails the scan; the worker goes on and the buffer is released.
139+func TestHintDetectorErrorFailsScan(t *testing.T) {
140+ boom := errors.New("detector gone")
141+ det := newMockdetector(t)
142+ det.EXPECT().Close().Return(nil).Once()
143+ det.EXPECT().Detect(mock.Anything, mock.Anything).Return(nil, boom).Once()
144+ det.EXPECT().Detect(mock.Anything, mock.Anything).Return(nil, nil).Once()
145+ h := startWith(t, 5*time.Second, Options{detector: det})
146+ for i, want := range []error{boom, nil} {
147+ go func() {
148+ creq := <-h.captures
149+ h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
150+ }()
151+ h.requests <- ports.HintScanRequest{Seq: uint64(i + 1), Output: "o", Width: 16, Height: 16, Scale: 1}
152+ res := <-h.results
153+ if want != nil {
154+ require.ErrorIs(t, res.Err, want)
155+ } else {
156+ require.NoError(t, res.Err)
100 }157 }
101 }158 }
102- require.GreaterOrEqual(t, links, 1)
103 }159 }
104 160
105 func TestHintRequestShape(t *testing.T) {161 func TestHintRequestShape(t *testing.T) {
@@ -166,19 +222,6 @@ func TestHintBadSize(t *testing.T) {
166 require.ErrorIs(t, res.Err, ErrBadRequest)222 require.ErrorIs(t, res.Err, ErrBadRequest)
167 }223 }
168 224
169-func scanOnce(t *testing.T, h *harness, pix []byte, w, hgt int, seq uint64) ports.HintScanResult {
170- t.Helper()
171- go func() {
172- creq := <-h.captures
173- copy(creq.Mapped, pix)
174- h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
175- }()
176- h.requests <- ports.HintScanRequest{Seq: seq, Output: "HEADLESS-1", Width: w, Height: hgt, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: w, H: hgt}}}}
177- res := <-h.results
178- require.NoError(t, res.Err)
179- return res
180-}
181-
182 // waitFiles waits for the dump the worker writes after sending the result.225 // waitFiles waits for the dump the worker writes after sending the result.
183 func waitFiles(t *testing.T, dir string, n int) []os.DirEntry {226 func waitFiles(t *testing.T, dir string, n int) []os.DirEntry {
184 t.Helper()227 t.Helper()
@@ -190,19 +233,32 @@ func waitFiles(t *testing.T, dir string, n int) []os.DirEntry {
190 return entries233 return entries
191 }234 }
192 235
236+// scanOnce runs one scan on a 64x32 frame and waits for its result.
237+func scanOnce(t *testing.T, h *harness, seq uint64) ports.HintScanResult {
238+ t.Helper()
239+ go func() {
240+ creq := <-h.captures
241+ for i := 0; i < len(creq.Mapped); i += 4 {
242+ creq.Mapped[i], creq.Mapped[i+1], creq.Mapped[i+2] = 10, 20, 30
243+ }
244+ h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
245+ }()
246+ h.requests <- ports.HintScanRequest{Seq: seq, Output: "HEADLESS-1", Width: 64, Height: 32, Scale: 1}
247+ res := <-h.results
248+ require.NoError(t, res.Err)
249+ return res
250+}
251+
193 func TestHintDump(t *testing.T) {252 func TestHintDump(t *testing.T) {
194- if _, err := exec.LookPath("tesseract"); err != nil {253+ det := newMockdetector(t)
195- t.Skip("tesseract not on PATH")254+ det.EXPECT().Close().Return(nil).Once()
196- }255+ det.EXPECT().Detect(mock.Anything, mock.Anything).Return([]detection{{x1: 8, y1: 4, x2: 40, y2: 20, score: 0.75}}, nil).Once()
197- pix, w, hgt := loadBGRA(t)
198 dir := filepath.Join(t.TempDir(), "hints")256 dir := filepath.Join(t.TempDir(), "hints")
199- h := startWith(t, 5*time.Second, Options{DumpDir: dir})257+ h := startWith(t, 5*time.Second, Options{DumpDir: dir, detector: det})
200- res := scanOnce(t, h, pix, w, hgt, 1)258+ res := scanOnce(t, h, 1)
201- require.NotEmpty(t, res.Targets)259+ require.Len(t, res.Targets, 1)
202 260
203 entries := waitFiles(t, dir, 3)261 entries := waitFiles(t, dir, 3)
204- // A second request is only served once the dump is written.
205- _ = scanOnceDone(t, h)
206 var png, ann, js string262 var png, ann, js string
207 for _, e := range entries {263 for _, e := range entries {
208 info, err := e.Info()264 info, err := e.Info()
@@ -228,61 +284,45 @@ func TestHintDump(t *testing.T) {
228 data, err := os.ReadFile(filepath.Join(dir, js))284 data, err := os.ReadFile(filepath.Join(dir, js))
229 require.NoError(t, err)285 require.NoError(t, err)
230 var meta struct {286 var meta struct {
231- Output string `json:"output"`287+ Output string `json:"output"`
232- Width int `json:"width"`288+ Width int `json:"width"`
233- Height int `json:"height"`289+ Height int `json:"height"`
234- Stride int `json:"stride"`290+ Stride int `json:"stride"`
235- Scale float64 `json:"scale"`291+ Scale float64 `json:"scale"`
236- LinksOnly bool `json:"links_only"`292+ DetectMS float64 `json:"detect_ms"`
237- Targets []struct {293+ TotalMS float64 `json:"total_ms"`
238- Kind string `json:"kind"`294+ Targets []struct {
239- Rect [4]int `json:"rect"`295+ Rect [4]int `json:"rect"`
296+ Score float32 `json:"score"`
240 } `json:"targets"`297 } `json:"targets"`
241 }298 }
242 require.NoError(t, json.Unmarshal(data, &meta))299 require.NoError(t, json.Unmarshal(data, &meta))
243 require.Equal(t, "HEADLESS-1", meta.Output)300 require.Equal(t, "HEADLESS-1", meta.Output)
244- require.Equal(t, w, meta.Width)301+ require.Equal(t, 64, meta.Width)
245- require.Equal(t, hgt, meta.Height)302+ require.Equal(t, 32, meta.Height)
246- require.Equal(t, w*4, meta.Stride)303+ require.Equal(t, 64*4, meta.Stride)
247 require.Equal(t, 1.0, meta.Scale)304 require.Equal(t, 1.0, meta.Scale)
248- require.Len(t, meta.Targets, len(res.Targets))305+ require.GreaterOrEqual(t, meta.TotalMS, meta.DetectMS)
249- for i, tg := range res.Targets {306+ require.Len(t, meta.Targets, 1)
250- require.Equal(t, [4]int{tg.Rect.X, tg.Rect.Y, tg.Rect.X + tg.Rect.W, tg.Rect.Y + tg.Rect.H}, meta.Targets[i].Rect)307+ require.Equal(t, [4]int{8, 4, 40, 20}, meta.Targets[0].Rect)
251- }308+ require.Equal(t, float32(0.75), meta.Targets[0].Score)
252 309
253 f, err := os.Open(filepath.Join(dir, png))310 f, err := os.Open(filepath.Join(dir, png))
254 require.NoError(t, err)311 require.NoError(t, err)
255 defer f.Close()312 defer f.Close()
256 img, _, err := image.Decode(f)313 img, _, err := image.Decode(f)
257 require.NoError(t, err)314 require.NoError(t, err)
258- require.Equal(t, image.Rect(0, 0, w, hgt), img.Bounds())315+ require.Equal(t, image.Rect(0, 0, 64, 32), img.Bounds())
259 r, g, b, _ := img.At(0, 0).RGBA()316 r, g, b, _ := img.At(0, 0).RGBA()
260- require.Equal(t, uint32(pix[2])*0x101, r)317+ require.Equal(t, uint32(30)*0x101, r)
261- require.Equal(t, uint32(pix[1])*0x101, g)318+ require.Equal(t, uint32(20)*0x101, g)
262- require.Equal(t, uint32(pix[0])*0x101, b)319+ require.Equal(t, uint32(10)*0x101, b)
263-}
264-
265-// scanOnceDone sends a request that fails fast on a blank capture and waits
266-// for its result, which proves the previous dump finished.
267-func scanOnceDone(t *testing.T, h *harness) ports.HintScanResult {
268- t.Helper()
269- go func() {
270- creq := <-h.captures
271- h.captured <- ports.CaptureDone{ID: creq.ID, Err: errors.New("stop")}
272- }()
273- h.requests <- ports.HintScanRequest{Seq: 99, Output: "x", Width: 8, Height: 8, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: 8, H: 8}}}}
274- return <-h.results
275 }320 }
276 321
277 func TestHintDumpDisabled(t *testing.T) {322 func TestHintDumpDisabled(t *testing.T) {
278- if _, err := exec.LookPath("tesseract"); err != nil {
279- t.Skip("tesseract not on PATH")
280- }
281- pix, w, hgt := loadBGRA(t)
282 dir := filepath.Join(t.TempDir(), "hints")323 dir := filepath.Join(t.TempDir(), "hints")
283 h := start(t, 5*time.Second)324 h := start(t, 5*time.Second)
284- scanOnce(t, h, pix, w, hgt, 1)325+ scanOnce(t, h, 1)
285- scanOnceDone(t, h)
286 _, err := os.Stat(dir)326 _, err := os.Stat(dir)
287 require.ErrorIs(t, err, os.ErrNotExist)327 require.ErrorIs(t, err, os.ErrNotExist)
288 }328 }
@@ -302,67 +342,25 @@ func TestHintDumpRetention(t *testing.T) {
302 require.Equal(t, "20260101-000000.010-o.annotated.png", entries[0].Name())342 require.Equal(t, "20260101-000000.010-o.annotated.png", entries[0].Name())
303 }343 }
304 344
305-func TestScanMergesSource(t *testing.T) {
306- pix, w, hgt := loadBGRA(t)
307- src := newMocktargetSource(t)
308- src.EXPECT().Close().Return(nil).Once() // the worker closes its source on return
309- wins := []ports.HintWindow{{ID: 4, PID: 9, Rect: ports.Rect{X: 0, Y: 0, W: w, H: hgt}}}
310- at := []ports.HintTarget{{Rect: ports.Rect{X: 5, Y: 5, W: 30, H: 10}, Kind: ports.HintLink}}
311- src.EXPECT().Targets(mock.Anything, wins, false).Return(at, map[ports.WindowID]bool{4: true}, nil).Once()
312- h := startWith(t, 5*time.Second, Options{Source: src})
313- go func() {
314- creq := <-h.captures
315- copy(creq.Mapped, pix)
316- h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
317- }()
318- h.requests <- ports.HintScanRequest{Seq: 1, Output: "HEADLESS-1", Width: w, Height: hgt, Scale: 1, Windows: wins}
319- res := <-h.results
320- require.NoError(t, res.Err)
321- // The whole frame is covered: only the accessibility target remains.
322- require.Equal(t, at, res.Targets)
323-}
324-
325-func TestScanSourceErrorUsesOCR(t *testing.T) {
326- if _, err := exec.LookPath("tesseract"); err != nil {
327- t.Skip("tesseract not on PATH")
328- }
329- pix, w, hgt := loadBGRA(t)
330- src := newMocktargetSource(t)
331- src.EXPECT().Close().Return(nil).Once() // the worker closes its source on return
332- wins := []ports.HintWindow{{ID: 4, PID: 9, Rect: ports.Rect{W: w, H: hgt}}}
333- src.EXPECT().Targets(mock.Anything, wins, false).Return(nil, nil, errors.New("bus gone")).Once()
334- h := startWith(t, 5*time.Second, Options{Source: src})
335- go func() {
336- creq := <-h.captures
337- copy(creq.Mapped, pix)
338- h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
339- }()
340- h.requests <- ports.HintScanRequest{Seq: 1, Output: "HEADLESS-1", Width: w, Height: hgt, Scale: 1, Windows: wins}
341- res := <-h.results
342- require.NoError(t, res.Err)
343- require.NotEmpty(t, res.Targets)
344-}
345-
346 // The worker keeps no frame: the mapped buffer belongs to the scan, and to the345 // The worker keeps no frame: the mapped buffer belongs to the scan, and to the
347 // dump job until it is written; every mapping is released afterwards.346 // dump job until it is written; every mapping is released afterwards.
348 func TestWorkerHoldsNoFrameAfterScan(t *testing.T) {347 func TestWorkerHoldsNoFrameAfterScan(t *testing.T) {
349 for _, dumpDir := range []string{"", filepath.Join(t.TempDir(), "hints")} {348 for _, dumpDir := range []string{"", filepath.Join(t.TempDir(), "hints")} {
350- // OCR is "installed" but its binary cannot run: only the memory matters.349+ det := newMockdetector(t)
351- w := &worker{dumpDir: dumpDir, ocr: ocrState{done: true, binary: "/nonexistent/tesseract", lang: "eng"}}350+ det.EXPECT().Detect(mock.Anything, mock.Anything).Return(nil, nil).Once()
352- requests := make(chan ports.HintScanRequest)351+ w := &worker{dumpDir: dumpDir, det: det}
353 captures := make(chan ports.CaptureRequest, 1)352 captures := make(chan ports.CaptureRequest, 1)
354 captured := make(chan ports.CaptureDone, 1)353 captured := make(chan ports.CaptureDone, 1)
355- results := make(chan ports.HintScanResult, 1)
356 go func() {354 go func() {
357 creq := <-captures355 creq := <-captures
358 captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}356 captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
359 }()357 }()
360- req := ports.HintScanRequest{Output: "o", Width: 64, Height: 32, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: 64, H: 32}}}}358+ req := ports.HintScanRequest{Output: "o", Width: 64, Height: 32, Scale: 1}
361- res := w.serve(context.Background(), req, requests, captures, captured, results, 5*time.Second)359+ res := w.serve(context.Background(), req, make(chan ports.HintScanRequest), captures, captured, make(chan ports.HintScanResult, 1), 5*time.Second)
362 require.NoError(t, res.Err)360 require.NoError(t, res.Err)
363 if dumpDir == "" {361 if dumpDir == "" {
364 require.Nil(t, w.pending)362 require.Nil(t, w.pending)
365- require.Zero(t, w.live, "frame and crops are released before the result")363+ require.Zero(t, w.live, "the frame is released before the result")
366 } else {364 } else {
367 require.NotNil(t, w.pending)365 require.NotNil(t, w.pending)
368 require.Equal(t, 1, w.live, "only the frame, kept for the dump")366 require.Equal(t, 1, w.live, "only the frame, kept for the dump")
@@ -370,7 +368,6 @@ func TestWorkerHoldsNoFrameAfterScan(t *testing.T) {
370 w.dump(context.Background())368 w.dump(context.Background())
371 require.Nil(t, w.pending)369 require.Nil(t, w.pending)
372 require.Zero(t, w.live)370 require.Zero(t, w.live)
373- require.Empty(t, w.cache) // OCR failed: nothing cached
374 }371 }
375 }372 }
376 373
@@ -389,39 +386,6 @@ func TestTimedOutCaptureIsReleasedWhenDone(t *testing.T) {
389 require.Zero(t, w.live)386 require.Zero(t, w.live)
390 }387 }
391 388
392-// Without tesseract OCR is off for good: no capture, only accessibility targets.
393-func TestNoTesseractSkipsCapture(t *testing.T) {
394- t.Setenv("PATH", t.TempDir())
395- button := ports.HintTarget{Kind: ports.HintButton, Rect: ports.Rect{W: 5, H: 5}}
396- covered := ports.HintTarget{Kind: ports.HintLink, Rect: ports.Rect{X: 30, W: 5, H: 5}}
397- wins := []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: 20, H: 20}}, {ID: 2, Rect: ports.Rect{X: 20, W: 20, H: 20}}}
398- src := newMocktargetSource(t)
399- src.EXPECT().Close().Return(nil).Once() // the worker closes its source on return
400- src.EXPECT().Targets(mock.Anything, wins, false).Return([]ports.HintTarget{button, covered}, map[ports.WindowID]bool{2: true}, nil).Times(2)
401- h := startWith(t, time.Second, Options{Source: src})
402- for seq := uint64(1); seq <= 2; seq++ {
403- h.requests <- ports.HintScanRequest{Seq: seq, Output: "o", Width: 40, Height: 20, Scale: 1, Windows: wins}
404- res := <-h.results
405- require.NoError(t, res.Err)
406- require.Equal(t, []ports.HintTarget{button, covered}, res.Targets)
407- }
408- require.Empty(t, h.captures)
409-}
410-
411-// With debug dumps on, the frame is still captured (for the dump only).
412-func TestNoTesseractDumpStillCaptures(t *testing.T) {
413- t.Setenv("PATH", t.TempDir())
414- dir := filepath.Join(t.TempDir(), "hints")
415- h := startWith(t, 5*time.Second, Options{DumpDir: dir})
416- go func() {
417- creq := <-h.captures
418- h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
419- }()
420- h.requests <- ports.HintScanRequest{Seq: 1, Output: "o", Width: 40, Height: 20, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: 40, H: 20}}}}
421- require.NoError(t, (<-h.results).Err)
422- waitFiles(t, dir, 3)
423-}
424-
425 // A capture that completes after its scan timed out is released while the389 // A capture that completes after its scan timed out is released while the
426 // worker is idle, without another scan, and its reply does not linger.390 // worker is idle, without another scan, and its reply does not linger.
427 func TestHintLateCaptureReaped(t *testing.T) {391 func TestHintLateCaptureReaped(t *testing.T) {
@@ -434,3 +398,21 @@ func TestHintLateCaptureReaped(t *testing.T) {
434 h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}398 h.captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
435 require.Eventually(t, func() bool { return len(h.captured) == 0 }, time.Second, time.Millisecond)399 require.Eventually(t, func() bool { return len(h.captured) == 0 }, time.Second, time.Millisecond)
436 }400 }
401+
402+func TestScanLogsRawAndPreparedCounts(t *testing.T) {
403+ dets := make([]detection, maxWorkBoxes+5)
404+ for i := range dets {
405+ dets[i] = detection{x1: 10, y1: 10, x2: 30, y2: 30, score: .9}
406+ }
407+ det := newMockdetector(t)
408+ det.EXPECT().Detect(mock.Anything, mock.Anything).Return(dets, nil).Once()
409+ var out bytes.Buffer // scan logs synchronously on this goroutine
410+ ctx := zerowrap.WithCtx(context.Background(), zerowrap.New(zerowrap.Config{Level: "info", Format: "json", Output: &out}))
411+ w := &worker{det: det, dumpDir: "enabled"}
412+ w.scan(ctx, ports.HintScanRequest{Width: 100, Height: 100, Scale: 1}, frame{m: &mapping{}}, 0, ports.HintScanResult{})
413+ var fields map[string]any
414+ require.NoError(t, json.Unmarshal(out.Bytes(), &fields))
415+ require.Equal(t, float64(maxWorkBoxes+5), fields["raw_boxes"])
416+ require.Equal(t, float64(maxWorkBoxes), fields["prepared_boxes"])
417+ require.Equal(t, "hints", fields["component"])
418+}
D internal/adapters/hints/hybrid_test.go +0 -177
@@ -1,177 +0,0 @@
1-package hints
2-
3-import (
4- "context"
5- "hash/maphash"
6- "image"
7- "image/draw"
8- "image/png"
9- "os"
10- "os/exec"
11- "runtime"
12- "testing"
13-
14- "github.com/bnema/neferwl/internal/ports"
15- "github.com/stretchr/testify/require"
16-)
17-
18-func TestWindowCrop(t *testing.T) {
19- for _, tc := range []struct {
20- name string
21- r ports.Rect
22- scale float64
23- want image.Rectangle
24- }{
25- {"unit", ports.Rect{X: 10, Y: 20, W: 100, H: 50}, 1, image.Rect(10, 20, 110, 70)},
26- {"fractional", ports.Rect{X: 11, Y: 21, W: 100, H: 50}, 1.2, image.Rect(13, 25, 134, 86)},
27- {"off output", ports.Rect{X: -10, Y: 20, W: 100, H: 100}, 1.2, image.Rect(0, 24, 108, 100)},
28- {"outside", ports.Rect{X: 1000, Y: 20, W: 100, H: 100}, 1, image.Rectangle{}},
29- } {
30- t.Run(tc.name, func(t *testing.T) { require.Equal(t, tc.want, windowCrop(tc.r, tc.scale, 200, 100)) })
31- }
32-}
33-
34-func sourceResult(ar atspiResult) *pendingSource {
35- p := &pendingSource{done: make(chan struct{}), res: ar}
36- close(p.done)
37- return p
38-}
39-
40-func TestHybridCacheAndMerge(t *testing.T) {
41- t.Setenv("PATH", t.TempDir()) // Cached OCR must not need tesseract.
42- buf := make([]byte, 200*100*4)
43- wins := []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: 100, H: 100}}, {ID: 2, Rect: ports.Rect{X: 100, W: 100, H: 100}}}
44- req := ports.HintScanRequest{Output: "o", Width: 200, Height: 100, Scale: 1, Windows: wins}
45- seed := maphash.MakeSeed()
46- r := windowCrop(wins[1].Rect, 1, 200, 100)
47- oldWorkers := ocrWorkers
48- ocrWorkers = 1 // one band covers the crop
49- t.Cleanup(func() { ocrWorkers = oldWorkers })
50- ext := image.Rect(0, 0, 100, 100)
51- entry := bandCache{hash: (&worker{cacheSeed: seed}).cropHash(buf, 800, r), ext: ext, words: []OCRWord{{Text: "example.org", Rect: image.Rect(10, 20, 90, 40)}}}
52- button := ports.HintTarget{Kind: ports.HintButton, Rect: ports.Rect{X: 1, Y: 1, W: 10, H: 10}}
53- link := ports.HintTarget{Kind: ports.HintLink, Text: true, Rect: ports.Rect{X: 110, Y: 20, W: 80, H: 20}}
54- uncoveredButton := ports.HintTarget{Kind: ports.HintButton, Rect: ports.Rect{X: 101, Y: 1, W: 10, H: 10}}
55- ar := atspiResult{targets: []ports.HintTarget{button, uncoveredButton}, covered: map[ports.WindowID]bool{1: true}}
56- w := worker{ocr: ocrState{done: true, binary: "tesseract", lang: "eng"}, cacheSeed: seed, cache: map[bandKey]bandCache{{output: "o", window: 2, band: 0, bands: 1}: entry}}
57- got := w.scan(context.Background(), req, frame{pix: buf, stride: 800}, 0, ports.HintScanResult{}, sourceResult(ar))
58- require.Equal(t, []ports.HintTarget{button, uncoveredButton, link}, got.Targets)
59- req.LinksOnly = true
60- got = w.scan(context.Background(), req, frame{pix: buf, stride: 800}, 0, ports.HintScanResult{}, sourceResult(ar))
61- require.Equal(t, []ports.HintTarget{link}, got.Targets)
62- // Changing one pixel invalidates only the crop, never leaving stale links.
63- buf[100*4]++
64- got = w.scan(context.Background(), req, frame{pix: buf, stride: 800}, 0, ports.HintScanResult{}, sourceResult(ar))
65- require.Empty(t, got.Targets)
66-}
67-
68-func TestMissingTesseractKeepsAccessibilityButtons(t *testing.T) {
69- t.Setenv("PATH", t.TempDir())
70- w := worker{}
71- button := ports.HintTarget{Kind: ports.HintButton, Rect: ports.Rect{W: 5, H: 5}}
72- req := ports.HintScanRequest{Width: 20, Height: 20, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: 20, H: 20}}}}
73- got := w.scan(context.Background(), req, frame{pix: make([]byte, 1600), stride: 80}, 0, ports.HintScanResult{}, sourceResult(atspiResult{targets: []ports.HintTarget{button}}))
74- require.NoError(t, got.Err)
75- require.Equal(t, []ports.HintTarget{button}, got.Targets)
76-}
77-
78-func TestNoWindowsProducesNoPixelTargets(t *testing.T) {
79- pix, width, height := loadBGRA(t)
80- w := worker{}
81- got := w.scan(context.Background(), ports.HintScanRequest{Width: width, Height: height}, frame{pix: pix, stride: width * 4}, 0, ports.HintScanResult{}, sourceResult(atspiResult{}))
82- require.Empty(t, got.Targets)
83-}
84-
85-func TestStripMappingAndPolarity(t *testing.T) {
86- img := image.NewRGBA(image.Rect(0, 0, 200, 200))
87- strips := packLines(img, []image.Rectangle{image.Rect(20, 20, 160, 40), image.Rect(20, 80, 160, 112)}, 2, true)
88- require.Len(t, strips, 2)
89- for _, s := range strips {
90- for _, off := range s.offsets {
91- require.Equal(t, 1, off.factor)
92- require.Equal(t, uint8(255), s.img.GrayAt(off.packed.Min.X, off.packed.Min.Y).Y)
93- }
94- }
95- small := packLines(img, []image.Rectangle{image.Rect(20, 20, 100, 30)}, 1, true)
96- require.Equal(t, 2, small[0].offsets[0].factor)
97-}
98-
99-func TestCropGrayAndHash(t *testing.T) {
100- buf := make([]byte, 4*4*4)
101- for y := 0; y < 4; y++ {
102- for x := 0; x < 4; x++ {
103- copy(buf[y*16+x*4:], []byte{0, 0, 255, 255}) // BGRA red
104- }
105- }
106- g := cropGray(buf, 16, image.Rect(1, 1, 3, 3), make([]byte, 4))
107- require.Equal(t, image.Rect(0, 0, 2, 2), g.Bounds())
108- require.Equal(t, uint8(76), g.GrayAt(1, 1).Y)
109- w := worker{cacheSeed: maphash.MakeSeed()}
110- a := w.cropHash(buf, 16, image.Rect(1, 1, 3, 3))
111- buf[0]++ // outside the crop
112- require.Equal(t, a, w.cropHash(buf, 16, image.Rect(1, 1, 3, 3)))
113- buf[16+4]++
114- require.NotEqual(t, a, w.cropHash(buf, 16, image.Rect(1, 1, 3, 3)))
115-}
116-
117-func TestUpscaleBandGray(t *testing.T) {
118- g := image.NewGray(image.Rect(0, 0, 3, 2))
119- for i := range g.Pix {
120- g.Pix[i] = uint8(i + 1)
121- }
122- sub := g.SubImage(image.Rect(1, 1, 3, 2)).(*image.Gray)
123- up := upscaleBand(sub)
124- require.Equal(t, image.Rect(0, 0, 4, 2), up.Bounds())
125- require.Equal(t, []uint8{5, 5, 6, 6, 5, 5, 6, 6}, up.Pix)
126-}
127-
128-// A blinking cursor changes one band: the other bands keep their cached
129-// words, and only the changed one is recognized again.
130-func TestBandCacheSurvivesCursorBlink(t *testing.T) {
131- binary, err := exec.LookPath("tesseract")
132- if err != nil {
133- t.Skip("tesseract not on PATH")
134- }
135- old := ocrWorkers
136- ocrWorkers = 4
137- t.Cleanup(func() { ocrWorkers = old })
138- f, err := os.Open("testdata/terminal.png")
139- require.NoError(t, err)
140- defer f.Close()
141- src, err := png.Decode(f)
142- require.NoError(t, err)
143- w, h := src.Bounds().Dx(), src.Bounds().Dy()
144- rgba := image.NewRGBA(image.Rect(0, 0, w, h))
145- draw.Draw(rgba, rgba.Bounds(), src, src.Bounds().Min, draw.Src)
146- buf := make([]byte, w*h*4)
147- for i := 0; i < len(buf); i += 4 {
148- buf[i], buf[i+1], buf[i+2], buf[i+3] = rgba.Pix[i+2], rgba.Pix[i+1], rgba.Pix[i], 0xff
149- }
150- _ = binary
151- wk := &worker{}
152- req := ports.HintScanRequest{Output: "o", Width: w, Height: h, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: w, H: h}}}}
153- first := wk.scan(context.Background(), req, frame{pix: buf, stride: w * 4}, 0, ports.HintScanResult{}, sourceResult(atspiResult{}))
154- require.NotEmpty(t, first.Targets)
155- require.Len(t, wk.cache, min(runtime.GOMAXPROCS(0), 4))
156-
157- // The same frame: every band hits.
158- again := wk.scan(context.Background(), req, frame{pix: buf, stride: w * 4}, 0, ports.HintScanResult{}, sourceResult(atspiResult{}))
159- require.Equal(t, first.Targets, again.Targets)
160-
161- // One pixel at the bottom blinks: only the last band (and the context of
162- // none other) is invalidated.
163- before := map[bandKey]uint64{}
164- for k, v := range wk.cache {
165- before[k] = v.hash
166- }
167- buf[((h-1)*w+3)*4] ^= 0xff
168- blink := wk.scan(context.Background(), req, frame{pix: buf, stride: w * 4}, 0, ports.HintScanResult{}, sourceResult(atspiResult{}))
169- require.Equal(t, first.Targets, blink.Targets)
170- changed := 0
171- for k, v := range wk.cache {
172- if before[k] != v.hash {
173- changed++
174- }
175- }
176- require.Equal(t, 1, changed)
177-}
D internal/adapters/hints/language_runner_mock_test.go +0 -130
@@ -1,130 +0,0 @@
1-// Code generated by mockery; DO NOT EDIT.
2-// github.com/vektra/mockery
3-// template: testify
4-
5-package hints
6-
7-import (
8- "context"
9-
10- mock "github.com/stretchr/testify/mock"
11-)
12-
13-// newMocklanguageRunner creates a new instance of mocklanguageRunner. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
14-// The first argument is typically a *testing.T value.
15-func newMocklanguageRunner(t interface {
16- mock.TestingT
17- Cleanup(func())
18-}) *mocklanguageRunner {
19- if helper, ok := t.(interface{ Helper() }); ok {
20- helper.Helper()
21- }
22-
23- mock := &mocklanguageRunner{}
24- mock.Mock.Test(t)
25-
26- t.Cleanup(func() {
27- if helper, ok := t.(interface{ Helper() }); ok {
28- helper.Helper()
29- }
30- mock.AssertExpectations(t)
31- })
32-
33- return mock
34-}
35-
36-// mocklanguageRunner is an autogenerated mock type for the languageRunner type
37-type mocklanguageRunner struct {
38- mock.Mock
39-}
40-
41-type mocklanguageRunner_Expecter struct {
42- mock *mock.Mock
43-}
44-
45-func (_m *mocklanguageRunner) EXPECT() *mocklanguageRunner_Expecter {
46- return &mocklanguageRunner_Expecter{mock: &_m.Mock}
47-}
48-
49-// Output provides a mock function for the type mocklanguageRunner
50-func (_mock *mocklanguageRunner) Output(context1 context.Context, s string, strings ...string) ([]byte, error) {
51- var tmpRet mock.Arguments
52- if len(strings) > 0 {
53- tmpRet = _mock.Called(context1, s, strings)
54- } else {
55- tmpRet = _mock.Called(context1, s)
56- }
57- ret := tmpRet
58-
59- if len(ret) == 0 {
60- panic("no return value specified for Output")
61- }
62-
63- var r0 []byte
64- var r1 error
65- if returnFunc, ok := ret.Get(0).(func(context.Context, string, ...string) ([]byte, error)); ok {
66- return returnFunc(context1, s, strings...)
67- }
68- if returnFunc, ok := ret.Get(0).(func(context.Context, string, ...string) []byte); ok {
69- r0 = returnFunc(context1, s, strings...)
70- } else {
71- if ret.Get(0) != nil {
72- r0 = ret.Get(0).([]byte)
73- }
74- }
75- if returnFunc, ok := ret.Get(1).(func(context.Context, string, ...string) error); ok {
76- r1 = returnFunc(context1, s, strings...)
77- } else {
78- r1 = ret.Error(1)
79- }
80- return r0, r1
81-}
82-
83-// mocklanguageRunner_Output_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Output'
84-type mocklanguageRunner_Output_Call struct {
85- *mock.Call
86-}
87-
88-// Output is a helper method to define mock.On call
89-// - context1 context.Context
90-// - s string
91-// - strings ...string
92-func (_e *mocklanguageRunner_Expecter) Output(context1 any, s any, strings ...any) *mocklanguageRunner_Output_Call {
93- return &mocklanguageRunner_Output_Call{Call: _e.mock.On("Output",
94- append([]any{context1, s}, strings...)...)}
95-}
96-
97-func (_c *mocklanguageRunner_Output_Call) Run(run func(context1 context.Context, s string, strings ...string)) *mocklanguageRunner_Output_Call {
98- _c.Call.Run(func(args mock.Arguments) {
99- var arg0 context.Context
100- if args[0] != nil {
101- arg0 = args[0].(context.Context)
102- }
103- var arg1 string
104- if args[1] != nil {
105- arg1 = args[1].(string)
106- }
107- var arg2 []string
108- var variadicArgs []string
109- if len(args) > 2 {
110- variadicArgs = args[2].([]string)
111- }
112- arg2 = variadicArgs
113- run(
114- arg0,
115- arg1,
116- arg2...,
117- )
118- })
119- return _c
120-}
121-
122-func (_c *mocklanguageRunner_Output_Call) Return(bytes []byte, err error) *mocklanguageRunner_Output_Call {
123- _c.Call.Return(bytes, err)
124- return _c
125-}
126-
127-func (_c *mocklanguageRunner_Output_Call) RunAndReturn(run func(context1 context.Context, s string, strings ...string) ([]byte, error)) *mocklanguageRunner_Output_Call {
128- _c.Call.Return(run)
129- return _c
130-}
A internal/adapters/hints/merge.go +82 -0
A internal/adapters/hints/merge_test.go +38 -0
D internal/adapters/hints/ocr.go +0 -460
D internal/adapters/hints/ocr_test.go +0 -272
D internal/adapters/hints/scan_bench_test.go +0 -49
D internal/adapters/hints/strips_test.go +0 -124
D internal/adapters/hints/target_source_mock_test.go +0 -174
D internal/adapters/hints/testdata/browser.png +0 -0 binary file: no text diff
D internal/adapters/hints/testdata/terminal.png +0 -0 binary file: no text diff
D internal/adapters/hints/testdata/underline.png +0 -0 binary file: no text diff
A internal/adapters/hints/tile.go +122 -0
A internal/adapters/hints/tile_test.go +75 -0
D internal/adapters/hints/underline.go +0 -246
D internal/adapters/hints/underline_test.go +0 -252
M internal/adapters/logging/logging.go +1 -1
M internal/app/app.go +6 -6
R internal/app/hintdebug.go → internal/app/hint.go +7 -10 renamed, 66% similar
M internal/app/hintoptin_test.go +18 -2
M internal/app/session.go +2 -2
M internal/core/core.go +3 -0
M internal/core/hint.go +84 -35
A internal/core/hint_internal_test.go +45 -0
M internal/core/hint_test.go +92 -124
M internal/ports/config.go +8 -6
M internal/ports/hint_messages.go +2 -31

Comments0 open / 0

no comments yet.

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
l
merge request: the list of files (j k move, enter opens, esc back)
M
merge request: the merge menu; there 1–3 pick the mode, D deletes the branch, enter merges
R
merge request: mark the head read
esc
close a dialog, leave a field

Branches 23

Tags 10

↑↓ moveenter openesc close