feat/hint-detector c706457

#31 WIP: feat(hints): detect hint targets with the nefer-hints detector

state
draft
branch
feat/hint-detector → main
opened
bnema ·
updated

Summary

Hint mode (experimental, off by default) finds click targets with the local nefer-hints detector instead of AT-SPI and OCR. NeferWL captures the output, sends the frame to the detector subprocess over its binary protocol, and turns the returned boxes into labelled hints. Screen content stays on the host.

Change

  • Detector subprocess: started on first scan, ready handshake, per-request deadline, process group killed and reaped on timeout or cancellation; a response that races the deadline is rejected and the process replaced.
  • Detector output is untrusted: boxes are validated (finite, scored in [0,1], non-empty), clamped to the frame, capped to a work budget, and duplicate or nested boxes are merged.
  • Wide outputs are tiled into 16:9 regions; boxes cut by a tile edge are dropped in favour of the neighbouring tile.
  • Labels never overlap: higher scores win; reloading hint.size cancels shown hints.
  • Hint clicks are sent without held modifiers.
  • hint.threshold defaults to the model's own threshold (model.json in nefer-hints).
  • Each scan logs regions, raw and prepared box counts, and timings; log.debug accepts hints.
  • AT-SPI and OCR target sources are removed.

Model

Tested with nefer-hints terminal-v2 (experimental). It ignores terminal text except links (98% link recall, under 0.5 false boxes per screen) and finds more web and native GTK targets than the stock OmniParser model, but it fails the nefer-hints release gate on GUI precision (57% vs 64% at matched settings). See the nefer-hints MR for the evaluation.

Checks

make check, make race, make mocks-check, staticcheck ./...; unit tests for the detector lifecycle (including deadline races with synctest), box validation, tiling, merging, label spreading and reload; headless smoke run. Wrap-up review done; its 3 findings are fixed and re-reviewed.

Not covered end to end: changing hint.size while hints are shown (unit-tested only).

CommitSubjectAuthorDateSignatureStatus
c706457docs(hints): clarify size changes on reloadbriceGPG Verified
68ee891fix(hints): log raw and prepared detection counts separatelybriceGPG Verified
4b3bafffix(core): cancel hints when reloading label sizebriceGPG Verified
8f581ccfix(hints): discard detector processes interrupted at response boundariesbriceGPG Verified
1ade7b5fix(hints): bound detector work and account for clamped labelsbriceGPG Verified
6fafd11feat(hints): use the model's threshold when hint.threshold is unsetbriceGPG Verified
0e762e1feat(hints): log regions and raw boxes per scanbriceGPG Verified
400321dfeat(hints): tile wide outputs into 16:9 regionsbriceGPG Verified
ab2b2affix(core): keep hint labels from overlappingbriceGPG Verified
8a43081fix(hints): merge duplicate and nested detector boxesbriceGPG Verified
ea2f35efix(hints): validate and clamp detector boxes to the framebriceGPG Verified
e8e812dfix(hints): kill the detector's process group on timeoutbriceGPG Verified
88edc52fix(core): send hint clicks without held modifiersbriceGPG Verified
492a5a9fix(config): accept hints in log.debugbriceGPG Verified
03202abdocs(config): describe the hint detectorbriceGPG Verified
2b7a35ffeat(hints): detect targets with the nefer-hints subprocessbriceGPG Verified

Checks 1

note: a failing or unfinished check blocks the merge.

50 files · +2126 -4857
Unified
UnifiedSplit
1/1
⋯

Merge settings

modes
fast-forward, squash, merge commit
checks
failing or unfinished checks block the merge
M .mockery.yml +2 -19
show this file
M README.md +1 -1
show this file
M docs/config.md +27 -29
show this file
M examples/config +6 -3
show this file
D internal/adapters/atspi/bus_test.go +0 -197
show this file
D internal/adapters/atspi/connector_mock_test.go +0 -109
show this file
D internal/adapters/atspi/doc.go +0 -10
show this file
D internal/adapters/atspi/model.go +0 -206
show this file
D internal/adapters/atspi/model_test.go +0 -117
show this file
D internal/adapters/atspi/source.go +0 -481
show this file
D internal/adapters/atspi/source_test.go +0 -355
show this file
D internal/adapters/atspi/transport.go +0 -362
show this file
D internal/adapters/atspi/transport_mock_test.go +0 -156
show this file
M internal/adapters/config/config.go +43 -40
show this file
M internal/adapters/config/config_test.go +51 -41
show this file
D internal/adapters/hints/benchmark_test.go +0 -81
show this file
A internal/adapters/hints/detector.go +448 -0
show this file
A internal/adapters/hints/detector_mock_test.go +159 -0
show this file
A internal/adapters/hints/detector_proc_test.go +172 -0
show this file
A internal/adapters/hints/detector_test.go +335 -0
show this file
M internal/adapters/hints/dump.go +38 -79
show this file
M internal/adapters/hints/hints.go +99 -292
show this file
M internal/adapters/hints/hints_test.go +160 -178
show this file
D internal/adapters/hints/hybrid_test.go +0 -177
show this file
D internal/adapters/hints/language_runner_mock_test.go +0 -130
show this file
A internal/adapters/hints/merge.go +82 -0
@@ -0,0 +1,82 @@
1+package hints
2+
3+import (
4+ "cmp"
5+ "slices"
6+
7+ "github.com/bnema/neferwl/internal/ports"
8+)
9+
10+const (
11+ // dupIoU is the IoU above which two boxes are the same element: the
12+ // higher score is kept.
13+ dupIoU = 0.5
14+ // nestRatio is the share of the smaller box that must lie inside the
15+ // other for the two to be nested.
16+ nestRatio = 0.85
17+)
18+
19+func area(r ports.Rect) int { return r.W * r.H }
20+
21+// interArea is the area of the intersection of a and b.
22+func interArea(a, b ports.Rect) int {
23+ w := min(a.X+a.W, b.X+b.W) - max(a.X, b.X)
24+ h := min(a.Y+a.H, b.Y+b.H) - max(a.Y, b.Y)
25+ if w <= 0 || h <= 0 {
26+ return 0
27+ }
28+ return w * h
29+}
30+
31+// nested reports whether the smaller of a and b lies for at least nestRatio
32+// inside the other.
33+func nested(a, b ports.Rect) bool {
34+ return float64(interArea(a, b)) >= nestRatio*float64(min(area(a), area(b)))
35+}
36+
37+// mergeTargets removes the duplicate and nested boxes the detector returns
38+// for one element. It works on logical targets (after rounding), in two
39+// passes so that the result does not depend on the processing order:
40+//
41+// 1. duplicates: by descending score, a box whose IoU with an already kept
42+// box is over dupIoU is dropped;
43+// 2. nesting: a box that contains another surviving box (that box lies
44+// nestRatio inside it) is dropped, whatever the scores: the smallest of
45+// a nested chain is kept (a row loses against its icon and its text).
46+//
47+// The result is ordered by descending score, then position.
48+func mergeTargets(in []ports.HintTarget) []ports.HintTarget {
49+ s := slices.Clone(in)
50+ slices.SortStableFunc(s, func(a, b ports.HintTarget) int {
51+ return cmp.Or(cmp.Compare(b.Score, a.Score), cmp.Compare(a.Rect.Y, b.Rect.Y), cmp.Compare(a.Rect.X, b.Rect.X),
52+ cmp.Compare(area(a.Rect), area(b.Rect)))
53+ })
54+ kept := s[:0]
55+ for _, t := range s {
56+ dup := false
57+ for _, k := range kept {
58+ inter := interArea(t.Rect, k.Rect)
59+ if inter > 0 && float64(inter) > dupIoU*float64(area(t.Rect)+area(k.Rect)-inter) {
60+ dup = true
61+ break
62+ }
63+ }
64+ if !dup {
65+ kept = append(kept, t)
66+ }
67+ }
68+ out := make([]ports.HintTarget, 0, len(kept))
69+ for i, t := range kept {
70+ contains := false
71+ for j, k := range kept {
72+ if i != j && area(k.Rect) < area(t.Rect) && nested(t.Rect, k.Rect) {
73+ contains = true
74+ break
75+ }
76+ }
77+ if !contains {
78+ out = append(out, t)
79+ }
80+ }
81+ return out
82+}
@@ -0,0 +1,82 @@
1+package hints
2+
3+import (
4+ "cmp"
5+ "slices"
6+
7+ "github.com/bnema/neferwl/internal/ports"
8+)
9+
10+const (
11+ // dupIoU is the IoU above which two boxes are the same element: the
12+ // higher score is kept.
13+ dupIoU = 0.5
14+ // nestRatio is the share of the smaller box that must lie inside the
15+ // other for the two to be nested.
16+ nestRatio = 0.85
17+)
18+
19+func area(r ports.Rect) int { return r.W * r.H }
20+
21+// interArea is the area of the intersection of a and b.
22+func interArea(a, b ports.Rect) int {
23+ w := min(a.X+a.W, b.X+b.W) - max(a.X, b.X)
24+ h := min(a.Y+a.H, b.Y+b.H) - max(a.Y, b.Y)
25+ if w <= 0 || h <= 0 {
26+ return 0
27+ }
28+ return w * h
29+}
30+
31+// nested reports whether the smaller of a and b lies for at least nestRatio
32+// inside the other.
33+func nested(a, b ports.Rect) bool {
34+ return float64(interArea(a, b)) >= nestRatio*float64(min(area(a), area(b)))
35+}
36+
37+// mergeTargets removes the duplicate and nested boxes the detector returns
38+// for one element. It works on logical targets (after rounding), in two
39+// passes so that the result does not depend on the processing order:
40+//
41+// 1. duplicates: by descending score, a box whose IoU with an already kept
42+// box is over dupIoU is dropped;
43+// 2. nesting: a box that contains another surviving box (that box lies
44+// nestRatio inside it) is dropped, whatever the scores: the smallest of
45+// a nested chain is kept (a row loses against its icon and its text).
46+//
47+// The result is ordered by descending score, then position.
48+func mergeTargets(in []ports.HintTarget) []ports.HintTarget {
49+ s := slices.Clone(in)
50+ slices.SortStableFunc(s, func(a, b ports.HintTarget) int {
51+ return cmp.Or(cmp.Compare(b.Score, a.Score), cmp.Compare(a.Rect.Y, b.Rect.Y), cmp.Compare(a.Rect.X, b.Rect.X),
52+ cmp.Compare(area(a.Rect), area(b.Rect)))
53+ })
54+ kept := s[:0]
55+ for _, t := range s {
56+ dup := false
57+ for _, k := range kept {
58+ inter := interArea(t.Rect, k.Rect)
59+ if inter > 0 && float64(inter) > dupIoU*float64(area(t.Rect)+area(k.Rect)-inter) {
60+ dup = true
61+ break
62+ }
63+ }
64+ if !dup {
65+ kept = append(kept, t)
66+ }
67+ }
68+ out := make([]ports.HintTarget, 0, len(kept))
69+ for i, t := range kept {
70+ contains := false
71+ for j, k := range kept {
72+ if i != j && area(k.Rect) < area(t.Rect) && nested(t.Rect, k.Rect) {
73+ contains = true
74+ break
75+ }
76+ }
77+ if !contains {
78+ out = append(out, t)
79+ }
80+ }
81+ return out
82+}
A internal/adapters/hints/merge_test.go +38 -0
@@ -0,0 +1,38 @@
1+package hints
2+
3+import (
4+ "testing"
5+
6+ "github.com/stretchr/testify/require"
7+
8+ "github.com/bnema/neferwl/internal/ports"
9+)
10+
11+func tg(x, y, w, h int, score float32) ports.HintTarget {
12+ return ports.HintTarget{Rect: ports.Rect{X: x, Y: y, W: w, H: h}, Score: score}
13+}
14+
15+func TestMergeTargets(t *testing.T) {
16+ tests := []struct {
17+ name string
18+ in []ports.HintTarget
19+ want []ports.HintTarget
20+ }{
21+ {"empty", nil, []ports.HintTarget{}},
22+ {"duplicate keeps the best score", []ports.HintTarget{tg(10, 10, 100, 40, 0.3), tg(12, 11, 100, 40, 0.9)}, []ports.HintTarget{tg(12, 11, 100, 40, 0.9)}},
23+ {"duplicate, input order irrelevant", []ports.HintTarget{tg(12, 11, 100, 40, 0.9), tg(10, 10, 100, 40, 0.3)}, []ports.HintTarget{tg(12, 11, 100, 40, 0.9)}},
24+ {"row with icon and text keeps both small ones", []ports.HintTarget{
25+ tg(0, 0, 400, 30, 0.95), tg(4, 4, 22, 22, 0.5), tg(40, 6, 200, 18, 0.4),
26+ }, []ports.HintTarget{tg(4, 4, 22, 22, 0.5), tg(40, 6, 200, 18, 0.4)}},
27+ {"nested keeps the smaller even with a lower score", []ports.HintTarget{tg(0, 0, 200, 100, 0.9), tg(20, 20, 50, 30, 0.1)}, []ports.HintTarget{tg(20, 20, 50, 30, 0.1)}},
28+ {"partly overlapping distinct buttons are kept", []ports.HintTarget{tg(0, 0, 100, 30, 0.8), tg(80, 0, 100, 30, 0.7)}, []ports.HintTarget{tg(0, 0, 100, 30, 0.8), tg(80, 0, 100, 30, 0.7)}},
29+ {"mostly (not 85%) inside is kept", []ports.HintTarget{tg(0, 0, 100, 100, 0.8), tg(60, 0, 100, 20, 0.7)}, []ports.HintTarget{tg(0, 0, 100, 100, 0.8), tg(60, 0, 100, 20, 0.7)}},
30+ {"chain of three nested keeps the innermost", []ports.HintTarget{tg(0, 0, 300, 200, 0.99), tg(10, 10, 200, 100, 0.5), tg(20, 20, 50, 30, 0.2)}, []ports.HintTarget{tg(20, 20, 50, 30, 0.2)}},
31+ {"disjoint", []ports.HintTarget{tg(0, 0, 10, 10, 0.2), tg(50, 50, 10, 10, 0.6)}, []ports.HintTarget{tg(50, 50, 10, 10, 0.6), tg(0, 0, 10, 10, 0.2)}},
32+ }
33+ for _, tc := range tests {
34+ t.Run(tc.name, func(t *testing.T) {
35+ require.Equal(t, tc.want, mergeTargets(tc.in))
36+ })
37+ }
38+}
@@ -0,0 +1,38 @@
1+package hints
2+
3+import (
4+ "testing"
5+
6+ "github.com/stretchr/testify/require"
7+
8+ "github.com/bnema/neferwl/internal/ports"
9+)
10+
11+func tg(x, y, w, h int, score float32) ports.HintTarget {
12+ return ports.HintTarget{Rect: ports.Rect{X: x, Y: y, W: w, H: h}, Score: score}
13+}
14+
15+func TestMergeTargets(t *testing.T) {
16+ tests := []struct {
17+ name string
18+ in []ports.HintTarget
19+ want []ports.HintTarget
20+ }{
21+ {"empty", nil, []ports.HintTarget{}},
22+ {"duplicate keeps the best score", []ports.HintTarget{tg(10, 10, 100, 40, 0.3), tg(12, 11, 100, 40, 0.9)}, []ports.HintTarget{tg(12, 11, 100, 40, 0.9)}},
23+ {"duplicate, input order irrelevant", []ports.HintTarget{tg(12, 11, 100, 40, 0.9), tg(10, 10, 100, 40, 0.3)}, []ports.HintTarget{tg(12, 11, 100, 40, 0.9)}},
24+ {"row with icon and text keeps both small ones", []ports.HintTarget{
25+ tg(0, 0, 400, 30, 0.95), tg(4, 4, 22, 22, 0.5), tg(40, 6, 200, 18, 0.4),
26+ }, []ports.HintTarget{tg(4, 4, 22, 22, 0.5), tg(40, 6, 200, 18, 0.4)}},
27+ {"nested keeps the smaller even with a lower score", []ports.HintTarget{tg(0, 0, 200, 100, 0.9), tg(20, 20, 50, 30, 0.1)}, []ports.HintTarget{tg(20, 20, 50, 30, 0.1)}},
28+ {"partly overlapping distinct buttons are kept", []ports.HintTarget{tg(0, 0, 100, 30, 0.8), tg(80, 0, 100, 30, 0.7)}, []ports.HintTarget{tg(0, 0, 100, 30, 0.8), tg(80, 0, 100, 30, 0.7)}},
29+ {"mostly (not 85%) inside is kept", []ports.HintTarget{tg(0, 0, 100, 100, 0.8), tg(60, 0, 100, 20, 0.7)}, []ports.HintTarget{tg(0, 0, 100, 100, 0.8), tg(60, 0, 100, 20, 0.7)}},
30+ {"chain of three nested keeps the innermost", []ports.HintTarget{tg(0, 0, 300, 200, 0.99), tg(10, 10, 200, 100, 0.5), tg(20, 20, 50, 30, 0.2)}, []ports.HintTarget{tg(20, 20, 50, 30, 0.2)}},
31+ {"disjoint", []ports.HintTarget{tg(0, 0, 10, 10, 0.2), tg(50, 50, 10, 10, 0.6)}, []ports.HintTarget{tg(50, 50, 10, 10, 0.6), tg(0, 0, 10, 10, 0.2)}},
32+ }
33+ for _, tc := range tests {
34+ t.Run(tc.name, func(t *testing.T) {
35+ require.Equal(t, tc.want, mergeTargets(tc.in))
36+ })
37+ }
38+}
D internal/adapters/hints/ocr.go +0 -460
@@ -1,460 +0,0 @@
1-package hints
2-
3-import (
4- "bufio"
5- "bytes"
6- "context"
7- "fmt"
8- "image"
9- "image/png"
10- "io"
11- "math"
12- "net"
13- "os"
14- "os/exec"
15- "path/filepath"
16- "regexp"
17- "runtime"
18- "sort"
19- "strconv"
20- "strings"
21- "sync"
22- "time"
23-
24- "github.com/bnema/neferwl/internal/adapters/logging"
25- "github.com/bnema/neferwl/internal/ports"
26-)
27-
28-// OCRWord is a recognized word in physical output coordinates.
29-type OCRWord struct {
30- Text string `json:"text"`
31- Rect image.Rectangle `json:"rect"`
32- Confidence float64 `json:"confidence"`
33- // Underlined is set when a thin line runs under the word. UnderL and
34- // UnderR are the x extent of that line (crop-local), which can reach the
35- // neighbouring words of one underlined link.
36- Underlined bool `json:"underlined,omitempty"`
37- UnderL int `json:"under_l,omitempty"`
38- UnderR int `json:"under_r,omitempty"`
39-}
40-
41-var languageWarning sync.Once
42-var failureWarning sync.Once
43-
44-// languageRunner bounds the external language-discovery process independently
45-// of recognition, which has not started yet when discovery runs.
46-type languageRunner interface {
47- Output(context.Context, string, ...string) ([]byte, error)
48-}
49-
50-type execLanguageRunner struct{}
51-
52-func (execLanguageRunner) Output(ctx context.Context, binary string, args ...string) ([]byte, error) {
53- return exec.CommandContext(ctx, binary, args...).Output()
54-}
55-
56-const languageTimeout = 5 * time.Second
57-
58-func ocrLanguage(ctx context.Context, binary string) (string, error) {
59- return discoverLanguage(ctx, binary, execLanguageRunner{}, languageTimeout)
60-}
61-
62-func discoverLanguage(ctx context.Context, binary string, runner languageRunner, timeout time.Duration) (string, error) {
63- dir := os.Getenv("TESSDATA_PREFIX")
64- if dir == "" {
65- dir = "/usr/share/tessdata"
66- }
67- if _, err := os.Stat(filepath.Join(dir, "eng.traineddata")); err == nil {
68- return "eng", nil
69- }
70- ctx, cancel := context.WithTimeout(ctx, timeout)
71- defer cancel()
72- out, err := runner.Output(ctx, binary, "--list-langs")
73- if err != nil {
74- return "", err
75- }
76- for _, line := range strings.Split(string(out), "\n")[1:] {
77- lang := strings.TrimSpace(line)
78- if lang != "" && lang != "osd" {
79- languageWarning.Do(func() {
80- log := logging.For(ctx, "hints")
81- log.Warn().Str("language", lang).Msg("English OCR data missing; install tesseract-data-eng")
82- })
83- return lang, nil
84- }
85- }
86- return "", fmt.Errorf("no OCR language available")
87-}
88-
89-func parseTSV(data []byte, offset image.Point) ([]OCRWord, error) {
90- var words []OCRWord
91- scanner := bufio.NewScanner(bytes.NewReader(data))
92- scanner.Buffer(make([]byte, 4096), 1024*1024)
93- for scanner.Scan() {
94- fields := strings.SplitN(scanner.Text(), "\t", 12)
95- if len(fields) != 12 || fields[0] != "5" {
96- continue
97- }
98- var values [4]int
99- for i := range values {
100- v, err := strconv.Atoi(fields[6+i])
101- if err != nil {
102- return nil, fmt.Errorf("OCR TSV coordinate: %w", err)
103- }
104- values[i] = v
105- }
106- conf, err := strconv.ParseFloat(fields[10], 64)
107- if err != nil {
108- return nil, fmt.Errorf("OCR TSV confidence: %w", err)
109- }
110- if values[2] <= 0 || values[3] <= 0 || strings.TrimSpace(fields[11]) == "" {
111- continue
112- }
113- r := image.Rect(values[0], values[1], values[0]+values[2], values[1]+values[3]).Add(offset)
114- words = append(words, OCRWord{Text: fields[11], Rect: r, Confidence: conf})
115- }
116- return words, scanner.Err()
117-}
118-
119-// ocrWorkers is the maximum number of bands recognized in parallel. 4 keeps
120-// the tesseract processes at about 60% of the memory of 8 for the same time.
121-var ocrWorkers = 4
122-
123-// pngLevel is the compression of non-grayscale images piped to tesseract.
124-var pngLevel = png.BestSpeed
125-
126-// rawPGM feeds grayscale images to tesseract as binary PGM instead of PNG.
127-var rawPGM = true
128-
129-// ocrState is the tesseract installation, resolved once.
130-type ocrState struct {
131- done bool
132- binary string
133- lang string
134- err error
135-}
136-
137-func (s *ocrState) resolve(ctx context.Context) error {
138- if !s.done {
139- s.done = true
140- if s.binary, s.err = exec.LookPath("tesseract"); s.err == nil {
141- s.lang, s.err = ocrLanguage(ctx, s.binary)
142- }
143- }
144- return s.err
145-}
146-
147-// ocrReady resolves tesseract on the first call. When it is missing, OCR is
148-// disabled for the life of the process and the warning is logged once.
149-func (w *worker) ocrReady(ctx context.Context) bool {
150- if err := w.ocr.resolve(ctx); err != nil {
151- failureWarning.Do(func() {
152- log := logging.For(ctx, "hints")
153- log.Warn().Err(err).Msg("OCR unavailable; install tesseract and tesseract-data-eng")
154- })
155- return false
156- }
157- return true
158-}
159-
160-// band is a horizontal slice of a window crop recognized by one tesseract
161-// process. It reads rows [top,bottom) and owns the words centred in
162-// [ownTop,ownBottom); the margin only supplies context.
163-type band struct {
164- idx, n int
165- top, bottom int
166- ownTop, ownBottom int
167-}
168-
169-func bandsOf(height int) []band {
170- n := min(runtime.GOMAXPROCS(0), ocrWorkers, height)
171- bands := make([]band, 0, max(n, 0))
172- for i := range n {
173- ownTop, ownBottom := i*height/n, (i+1)*height/n
174- bands = append(bands, band{idx: i, n: n, top: max(0, ownTop-40), bottom: min(height, ownBottom+40), ownTop: ownTop, ownBottom: ownBottom})
175- }
176- return bands
177-}
178-
179-// recognize reads the bands of jobs that missed the cache, in parallel.
180-func (w *worker) recognize(ctx context.Context, jobs []*bandJob) {
181- ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
182- defer cancel()
183- // One scan-wide pool: at most ocrWorkers tesseract processes run at once,
184- // whatever the number of windows.
185- sem := make(chan struct{}, max(ocrWorkers, 1))
186- var wg sync.WaitGroup
187- for _, j := range jobs {
188- if j.hit {
189- continue
190- }
191- select {
192- case sem <- struct{}{}:
193- case <-ctx.Done():
194- j.err = ctx.Err()
195- continue
196- }
197- wg.Go(func() {
198- defer func() { <-sem }()
199- j.words, j.err = readBand(ctx, &w.ocr, j)
200- })
201- }
202- wg.Wait()
203-}
204-
205-// readBand recognizes one band; the words are crop-local and owned by it.
206-func readBand(ctx context.Context, st *ocrState, j *bandJob) ([]OCRWord, error) {
207- words, err := recognizeBand(ctx, st.binary, st.lang, j.gray)
208- if err != nil {
209- return nil, err
210- }
211- heights := make([]int, 0, len(words))
212- for _, word := range words {
213- if len(word.Text) >= 3 {
214- heights = append(heights, word.Rect.Dy())
215- }
216- }
217- sort.Ints(heights)
218- if len(heights) > 0 && heights[len(heights)/2] < 12 {
219- words, err = recognizeBand(ctx, st.binary, st.lang, upscaleBand(j.gray))
220- if err != nil {
221- return nil, err
222- }
223- for i := range words {
224- r := words[i].Rect
225- words[i].Rect = image.Rect(r.Min.X/2, r.Min.Y/2, (r.Max.X+1)/2, (r.Max.Y+1)/2)
226- }
227- }
228- markUnderlined(j.gray, words)
229- owned := words[:0]
230- for _, word := range words {
231- // Overlap supplies context, but each word belongs to exactly one band.
232- word.Rect = word.Rect.Add(image.Pt(0, j.ext.Min.Y))
233- if center := (word.Rect.Min.Y + word.Rect.Max.Y) / 2; center >= j.own[0] && center < j.own[1] {
234- owned = append(owned, word)
235- }
236- }
237- return owned, nil
238-}
239-
240-// readWords recognizes the words of img, a window crop, in parallel bands.
241-func readWords(ctx context.Context, img *image.Gray) ([]OCRWord, int, error) {
242- w := &worker{}
243- if err := w.ocr.resolve(ctx); err != nil {
244- return nil, 0, err
245- }
246- var jobs []*bandJob
247- for _, b := range bandsOf(img.Bounds().Dy()) {
248- ext := image.Rect(0, b.top, img.Bounds().Dx(), b.bottom)
249- jobs = append(jobs, &bandJob{ext: ext, own: [2]int{b.ownTop, b.ownBottom}, gray: img.SubImage(ext).(*image.Gray)})
250- }
251- w.recognize(ctx, jobs)
252- var words []OCRWord
253- for _, j := range jobs {
254- if j.err != nil {
255- return nil, len(jobs), j.err
256- }
257- for _, word := range j.words {
258- if !duplicateWord(words, word) {
259- words = append(words, word)
260- }
261- }
262- }
263- return words, len(jobs), nil
264-}
265-
266-// The link patterns compile on the first OCR link check, not at process
267-// start: with hint mode off nothing here costs memory or CPU.
268-var domainPattern = sync.OnceValue(func() *regexp.Regexp {
269- return regexp.MustCompile(`(?i)^(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+([a-z]{2,24})(?::[0-9]+)?(?:[/?#]\S*)?$`)
270-})
271-var schemePattern = sync.OnceValue(func() *regexp.Regexp { return regexp.MustCompile(`(?i)^(?:https?|htps|file|ftp):/{1,2}\S+$`) })
272-var emailPattern = sync.OnceValue(func() *regexp.Regexp {
273- return regexp.MustCompile(`(?i)^[a-z0-9.!#$%&'*+/=?^_` + "`" + `{|}~-]+@(?:[a-z0-9-]+\.)+[a-z]{2,24}$`)
274-})
275-var fileExtensions = sync.OnceValue(func() []string {
276- return strings.Fields("go md txt toml json yaml yml rs py js ts sh c h png jpg log conf lock mod sum html css")
277-})
278-
279-func isTextLink(text string) bool {
280- text = strings.Trim(text, "()[]<>.,;:'\"")
281- if schemePattern().MatchString(text) || (strings.HasPrefix(strings.ToLower(text), "www.") && len(text) > 4) || emailPattern().MatchString(text) {
282- return true
283- }
284- host := text
285- if h, _, err := net.SplitHostPort(text); err == nil {
286- host = h
287- }
288- if ip := net.ParseIP(host); ip != nil && ip.To4() != nil {
289- return true
290- }
291- m := domainPattern().FindStringSubmatch(text)
292- if m == nil {
293- return false
294- }
295- for _, ext := range fileExtensions() {
296- if strings.EqualFold(m[1], ext) {
297- return false
298- }
299- }
300- return true
301-}
302-
303-func textLinks(words []OCRWord, scale float64) []ports.HintTarget {
304- if scale <= 0 {
305- scale = 1
306- }
307- words = append([]OCRWord(nil), words...)
308- sort.Slice(words, func(i, j int) bool {
309- if words[i].Rect.Min.Y != words[j].Rect.Min.Y {
310- return words[i].Rect.Min.Y < words[j].Rect.Min.Y
311- }
312- return words[i].Rect.Min.X < words[j].Rect.Min.X
313- })
314- var out []ports.HintTarget
315- var urls []image.Rectangle // boxes of the URL targets, in pixels
316- used := make([]bool, len(words))
317- for i, word := range words {
318- if used[i] {
319- continue
320- }
321- text, rect := word.Text, word.Rect
322- // Grow only touching words on the same baseline; ordinary prose is not joined.
323- for {
324- next := -1
325- for j, candidate := range words {
326- if j == i || used[j] || candidate.Rect.Min.X < rect.Max.X-2 {
327- continue
328- }
329- gap := candidate.Rect.Min.X - rect.Max.X
330- if gap > max(3, min(rect.Dy(), candidate.Rect.Dy())/5) {
331- continue
332- }
333- if abs(candidate.Rect.Max.Y-rect.Max.Y) > max(3, rect.Dy()/3) {
334- continue
335- }
336- if next < 0 || candidate.Rect.Min.X < words[next].Rect.Min.X {
337- next = j
338- }
339- }
340- if next < 0 {
341- break
342- }
343- text += words[next].Text
344- rect = rect.Union(words[next].Rect)
345- used[next] = true
346- }
347- if !isTextLink(text) {
348- continue
349- }
350- x, y := int(math.Floor(float64(rect.Min.X)/scale)), int(math.Floor(float64(rect.Min.Y)/scale))
351- right, bottom := int(math.Ceil(float64(rect.Max.X)/scale)), int(math.Ceil(float64(rect.Max.Y)/scale))
352- out = append(out, ports.HintTarget{Kind: ports.HintLink, Text: true, Rect: ports.Rect{X: x, Y: y, W: right - x, H: bottom - y}})
353- urls = append(urls, rect)
354- }
355- return append(out, underlinedLinks(words, urls, scale)...)
356-}
357-
358-func abs(x int) int {
359- if x < 0 {
360- return -x
361- }
362- return x
363-}
364-func overlap(a, b image.Rectangle) float64 {
365- intersection := a.Intersect(b)
366- area := min(a.Dx()*a.Dy(), b.Dx()*b.Dy())
367- if intersection.Empty() || area <= 0 {
368- return 0
369- }
370- return float64(intersection.Dx()*intersection.Dy()) / float64(area)
371-}
372-func targetRect(t ports.HintTarget) image.Rectangle {
373- return image.Rect(t.Rect.X, t.Rect.Y, t.Rect.X+t.Rect.W, t.Rect.Y+t.Rect.H)
374-}
375-func recognizeBand(ctx context.Context, binary, lang string, img image.Image) ([]OCRWord, error) {
376- return recognizeImage(ctx, binary, lang, img, "11", false)
377-}
378-
379-// pgmReader streams a grayscale image as binary PGM (P5), row by row.
380-type pgmReader struct {
381- head []byte
382- img *image.Gray
383- y int
384- row []byte
385-}
386-
387-func newPGMReader(img *image.Gray) *pgmReader {
388- b := img.Bounds()
389- return &pgmReader{head: fmt.Appendf(nil, "P5\n%d %d\n255\n", b.Dx(), b.Dy()), img: img}
390-}
391-
392-func (r *pgmReader) Read(p []byte) (int, error) {
393- n := 0
394- for n < len(p) {
395- if len(r.head) > 0 {
396- c := copy(p[n:], r.head)
397- r.head, n = r.head[c:], n+c
398- continue
399- }
400- if len(r.row) == 0 {
401- b := r.img.Bounds()
402- if r.y >= b.Dy() {
403- break
404- }
405- r.row = r.img.Pix[(r.y+b.Min.Y-r.img.Rect.Min.Y)*r.img.Stride+(b.Min.X-r.img.Rect.Min.X):][:b.Dx()]
406- r.y++
407- }
408- c := copy(p[n:], r.row)
409- r.row, n = r.row[c:], n+c
410- }
411- if n == 0 {
412- return 0, io.EOF
413- }
414- return n, nil
415-}
416-
417-func recognizeImage(ctx context.Context, binary, lang string, img image.Image, psm string, normalized bool) ([]OCRWord, error) {
418- args := []string{"stdin", "stdout", "--psm", psm, "-l", lang}
419- if normalized {
420- args = append(args, "-c", "tessedit_do_invert=0")
421- }
422- args = append(args, "tsv")
423- cmd := exec.CommandContext(ctx, binary, args...)
424- cmd.Env = append(os.Environ(), "OMP_THREAD_LIMIT=1")
425- if g, ok := img.(*image.Gray); ok && rawPGM {
426- // Raw grayscale: no deflate, no encoder buffers.
427- cmd.Stdin = newPGMReader(g)
428- } else {
429- // The PNG streams into tesseract; no encoded copy is held.
430- pr, pw := io.Pipe()
431- defer func() { _ = pr.Close() }()
432- go func() {
433- enc := png.Encoder{CompressionLevel: pngLevel}
434- _ = pw.CloseWithError(enc.Encode(pw, img))
435- }()
436- cmd.Stdin = pr
437- }
438- var stderr bytes.Buffer
439- cmd.Stderr = &stderr
440- output, err := cmd.Output()
441- if err != nil {
442- return nil, fmt.Errorf("tesseract: %w: %s", err, stderr.String())
443- }
444- return parseTSV(output, image.Point{})
445-}
446-
447-// upscaleBand doubles a grayscale band with nearest-neighbour sampling.
448-func upscaleBand(src *image.Gray) *image.Gray {
449- b := src.Bounds()
450- dst := image.NewGray(image.Rect(0, 0, b.Dx()*2, b.Dy()*2))
451- for y := 0; y < b.Dy(); y++ {
452- row := src.Pix[(y+b.Min.Y-src.Rect.Min.Y)*src.Stride+(b.Min.X-src.Rect.Min.X):][:b.Dx()]
453- top := dst.Pix[2*y*dst.Stride:][:dst.Stride]
454- for x, v := range row {
455- top[2*x], top[2*x+1] = v, v
456- }
457- copy(dst.Pix[(2*y+1)*dst.Stride:], top)
458- }
459- return dst
460-}
@@ -1,460 +0,0 @@
1-package hints
2-
3-import (
4- "bufio"
5- "bytes"
6- "context"
7- "fmt"
8- "image"
9- "image/png"
10- "io"
11- "math"
12- "net"
13- "os"
14- "os/exec"
15- "path/filepath"
16- "regexp"
17- "runtime"
18- "sort"
19- "strconv"
20- "strings"
21- "sync"
22- "time"
23-
24- "github.com/bnema/neferwl/internal/adapters/logging"
25- "github.com/bnema/neferwl/internal/ports"
26-)
27-
28-// OCRWord is a recognized word in physical output coordinates.
29-type OCRWord struct {
30- Text string `json:"text"`
31- Rect image.Rectangle `json:"rect"`
32- Confidence float64 `json:"confidence"`
33- // Underlined is set when a thin line runs under the word. UnderL and
34- // UnderR are the x extent of that line (crop-local), which can reach the
35- // neighbouring words of one underlined link.
36- Underlined bool `json:"underlined,omitempty"`
37- UnderL int `json:"under_l,omitempty"`
38- UnderR int `json:"under_r,omitempty"`
39-}
40-
41-var languageWarning sync.Once
42-var failureWarning sync.Once
43-
44-// languageRunner bounds the external language-discovery process independently
45-// of recognition, which has not started yet when discovery runs.
46-type languageRunner interface {
47- Output(context.Context, string, ...string) ([]byte, error)
48-}
49-
50-type execLanguageRunner struct{}
51-
52-func (execLanguageRunner) Output(ctx context.Context, binary string, args ...string) ([]byte, error) {
53- return exec.CommandContext(ctx, binary, args...).Output()
54-}
55-
56-const languageTimeout = 5 * time.Second
57-
58-func ocrLanguage(ctx context.Context, binary string) (string, error) {
59- return discoverLanguage(ctx, binary, execLanguageRunner{}, languageTimeout)
60-}
61-
62-func discoverLanguage(ctx context.Context, binary string, runner languageRunner, timeout time.Duration) (string, error) {
63- dir := os.Getenv("TESSDATA_PREFIX")
64- if dir == "" {
65- dir = "/usr/share/tessdata"
66- }
67- if _, err := os.Stat(filepath.Join(dir, "eng.traineddata")); err == nil {
68- return "eng", nil
69- }
70- ctx, cancel := context.WithTimeout(ctx, timeout)
71- defer cancel()
72- out, err := runner.Output(ctx, binary, "--list-langs")
73- if err != nil {
74- return "", err
75- }
76- for _, line := range strings.Split(string(out), "\n")[1:] {
77- lang := strings.TrimSpace(line)
78- if lang != "" && lang != "osd" {
79- languageWarning.Do(func() {
80- log := logging.For(ctx, "hints")
81- log.Warn().Str("language", lang).Msg("English OCR data missing; install tesseract-data-eng")
82- })
83- return lang, nil
84- }
85- }
86- return "", fmt.Errorf("no OCR language available")
87-}
88-
89-func parseTSV(data []byte, offset image.Point) ([]OCRWord, error) {
90- var words []OCRWord
91- scanner := bufio.NewScanner(bytes.NewReader(data))
92- scanner.Buffer(make([]byte, 4096), 1024*1024)
93- for scanner.Scan() {
94- fields := strings.SplitN(scanner.Text(), "\t", 12)
95- if len(fields) != 12 || fields[0] != "5" {
96- continue
97- }
98- var values [4]int
99- for i := range values {
100- v, err := strconv.Atoi(fields[6+i])
101- if err != nil {
102- return nil, fmt.Errorf("OCR TSV coordinate: %w", err)
103- }
104- values[i] = v
105- }
106- conf, err := strconv.ParseFloat(fields[10], 64)
107- if err != nil {
108- return nil, fmt.Errorf("OCR TSV confidence: %w", err)
109- }
110- if values[2] <= 0 || values[3] <= 0 || strings.TrimSpace(fields[11]) == "" {
111- continue
112- }
113- r := image.Rect(values[0], values[1], values[0]+values[2], values[1]+values[3]).Add(offset)
114- words = append(words, OCRWord{Text: fields[11], Rect: r, Confidence: conf})
115- }
116- return words, scanner.Err()
117-}
118-
119-// ocrWorkers is the maximum number of bands recognized in parallel. 4 keeps
120-// the tesseract processes at about 60% of the memory of 8 for the same time.
121-var ocrWorkers = 4
122-
123-// pngLevel is the compression of non-grayscale images piped to tesseract.
124-var pngLevel = png.BestSpeed
125-
126-// rawPGM feeds grayscale images to tesseract as binary PGM instead of PNG.
127-var rawPGM = true
128-
129-// ocrState is the tesseract installation, resolved once.
130-type ocrState struct {
131- done bool
132- binary string
133- lang string
134- err error
135-}
136-
137-func (s *ocrState) resolve(ctx context.Context) error {
138- if !s.done {
139- s.done = true
140- if s.binary, s.err = exec.LookPath("tesseract"); s.err == nil {
141- s.lang, s.err = ocrLanguage(ctx, s.binary)
142- }
143- }
144- return s.err
145-}
146-
147-// ocrReady resolves tesseract on the first call. When it is missing, OCR is
148-// disabled for the life of the process and the warning is logged once.
149-func (w *worker) ocrReady(ctx context.Context) bool {
150- if err := w.ocr.resolve(ctx); err != nil {
151- failureWarning.Do(func() {
152- log := logging.For(ctx, "hints")
153- log.Warn().Err(err).Msg("OCR unavailable; install tesseract and tesseract-data-eng")
154- })
155- return false
156- }
157- return true
158-}
159-
160-// band is a horizontal slice of a window crop recognized by one tesseract
161-// process. It reads rows [top,bottom) and owns the words centred in
162-// [ownTop,ownBottom); the margin only supplies context.
163-type band struct {
164- idx, n int
165- top, bottom int
166- ownTop, ownBottom int
167-}
168-
169-func bandsOf(height int) []band {
170- n := min(runtime.GOMAXPROCS(0), ocrWorkers, height)
171- bands := make([]band, 0, max(n, 0))
172- for i := range n {
173- ownTop, ownBottom := i*height/n, (i+1)*height/n
174- bands = append(bands, band{idx: i, n: n, top: max(0, ownTop-40), bottom: min(height, ownBottom+40), ownTop: ownTop, ownBottom: ownBottom})
175- }
176- return bands
177-}
178-
179-// recognize reads the bands of jobs that missed the cache, in parallel.
180-func (w *worker) recognize(ctx context.Context, jobs []*bandJob) {
181- ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
182- defer cancel()
183- // One scan-wide pool: at most ocrWorkers tesseract processes run at once,
184- // whatever the number of windows.
185- sem := make(chan struct{}, max(ocrWorkers, 1))
186- var wg sync.WaitGroup
187- for _, j := range jobs {
188- if j.hit {
189- continue
190- }
191- select {
192- case sem <- struct{}{}:
193- case <-ctx.Done():
194- j.err = ctx.Err()
195- continue
196- }
197- wg.Go(func() {
198- defer func() { <-sem }()
199- j.words, j.err = readBand(ctx, &w.ocr, j)
200- })
201- }
202- wg.Wait()
203-}
204-
205-// readBand recognizes one band; the words are crop-local and owned by it.
206-func readBand(ctx context.Context, st *ocrState, j *bandJob) ([]OCRWord, error) {
207- words, err := recognizeBand(ctx, st.binary, st.lang, j.gray)
208- if err != nil {
209- return nil, err
210- }
211- heights := make([]int, 0, len(words))
212- for _, word := range words {
213- if len(word.Text) >= 3 {
214- heights = append(heights, word.Rect.Dy())
215- }
216- }
217- sort.Ints(heights)
218- if len(heights) > 0 && heights[len(heights)/2] < 12 {
219- words, err = recognizeBand(ctx, st.binary, st.lang, upscaleBand(j.gray))
220- if err != nil {
221- return nil, err
222- }
223- for i := range words {
224- r := words[i].Rect
225- words[i].Rect = image.Rect(r.Min.X/2, r.Min.Y/2, (r.Max.X+1)/2, (r.Max.Y+1)/2)
226- }
227- }
228- markUnderlined(j.gray, words)
229- owned := words[:0]
230- for _, word := range words {
231- // Overlap supplies context, but each word belongs to exactly one band.
232- word.Rect = word.Rect.Add(image.Pt(0, j.ext.Min.Y))
233- if center := (word.Rect.Min.Y + word.Rect.Max.Y) / 2; center >= j.own[0] && center < j.own[1] {
234- owned = append(owned, word)
235- }
236- }
237- return owned, nil
238-}
239-
240-// readWords recognizes the words of img, a window crop, in parallel bands.
241-func readWords(ctx context.Context, img *image.Gray) ([]OCRWord, int, error) {
242- w := &worker{}
243- if err := w.ocr.resolve(ctx); err != nil {
244- return nil, 0, err
245- }
246- var jobs []*bandJob
247- for _, b := range bandsOf(img.Bounds().Dy()) {
248- ext := image.Rect(0, b.top, img.Bounds().Dx(), b.bottom)
249- jobs = append(jobs, &bandJob{ext: ext, own: [2]int{b.ownTop, b.ownBottom}, gray: img.SubImage(ext).(*image.Gray)})
250- }
251- w.recognize(ctx, jobs)
252- var words []OCRWord
253- for _, j := range jobs {
254- if j.err != nil {
255- return nil, len(jobs), j.err
256- }
257- for _, word := range j.words {
258- if !duplicateWord(words, word) {
259- words = append(words, word)
260- }
261- }
262- }
263- return words, len(jobs), nil
264-}
265-
266-// The link patterns compile on the first OCR link check, not at process
267-// start: with hint mode off nothing here costs memory or CPU.
268-var domainPattern = sync.OnceValue(func() *regexp.Regexp {
269- return regexp.MustCompile(`(?i)^(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+([a-z]{2,24})(?::[0-9]+)?(?:[/?#]\S*)?$`)
270-})
271-var schemePattern = sync.OnceValue(func() *regexp.Regexp { return regexp.MustCompile(`(?i)^(?:https?|htps|file|ftp):/{1,2}\S+$`) })
272-var emailPattern = sync.OnceValue(func() *regexp.Regexp {
273- return regexp.MustCompile(`(?i)^[a-z0-9.!#$%&'*+/=?^_` + "`" + `{|}~-]+@(?:[a-z0-9-]+\.)+[a-z]{2,24}$`)
274-})
275-var fileExtensions = sync.OnceValue(func() []string {
276- return strings.Fields("go md txt toml json yaml yml rs py js ts sh c h png jpg log conf lock mod sum html css")
277-})
278-
279-func isTextLink(text string) bool {
280- text = strings.Trim(text, "()[]<>.,;:'\"")
281- if schemePattern().MatchString(text) || (strings.HasPrefix(strings.ToLower(text), "www.") && len(text) > 4) || emailPattern().MatchString(text) {
282- return true
283- }
284- host := text
285- if h, _, err := net.SplitHostPort(text); err == nil {
286- host = h
287- }
288- if ip := net.ParseIP(host); ip != nil && ip.To4() != nil {
289- return true
290- }
291- m := domainPattern().FindStringSubmatch(text)
292- if m == nil {
293- return false
294- }
295- for _, ext := range fileExtensions() {
296- if strings.EqualFold(m[1], ext) {
297- return false
298- }
299- }
300- return true
301-}
302-
303-func textLinks(words []OCRWord, scale float64) []ports.HintTarget {
304- if scale <= 0 {
305- scale = 1
306- }
307- words = append([]OCRWord(nil), words...)
308- sort.Slice(words, func(i, j int) bool {
309- if words[i].Rect.Min.Y != words[j].Rect.Min.Y {
310- return words[i].Rect.Min.Y < words[j].Rect.Min.Y
311- }
312- return words[i].Rect.Min.X < words[j].Rect.Min.X
313- })
314- var out []ports.HintTarget
315- var urls []image.Rectangle // boxes of the URL targets, in pixels
316- used := make([]bool, len(words))
317- for i, word := range words {
318- if used[i] {
319- continue
320- }
321- text, rect := word.Text, word.Rect
322- // Grow only touching words on the same baseline; ordinary prose is not joined.
323- for {
324- next := -1
325- for j, candidate := range words {
326- if j == i || used[j] || candidate.Rect.Min.X < rect.Max.X-2 {
327- continue
328- }
329- gap := candidate.Rect.Min.X - rect.Max.X
330- if gap > max(3, min(rect.Dy(), candidate.Rect.Dy())/5) {
331- continue
332- }
333- if abs(candidate.Rect.Max.Y-rect.Max.Y) > max(3, rect.Dy()/3) {
334- continue
335- }
336- if next < 0 || candidate.Rect.Min.X < words[next].Rect.Min.X {
337- next = j
338- }
339- }
340- if next < 0 {
341- break
342- }
343- text += words[next].Text
344- rect = rect.Union(words[next].Rect)
345- used[next] = true
346- }
347- if !isTextLink(text) {
348- continue
349- }
350- x, y := int(math.Floor(float64(rect.Min.X)/scale)), int(math.Floor(float64(rect.Min.Y)/scale))
351- right, bottom := int(math.Ceil(float64(rect.Max.X)/scale)), int(math.Ceil(float64(rect.Max.Y)/scale))
352- out = append(out, ports.HintTarget{Kind: ports.HintLink, Text: true, Rect: ports.Rect{X: x, Y: y, W: right - x, H: bottom - y}})
353- urls = append(urls, rect)
354- }
355- return append(out, underlinedLinks(words, urls, scale)...)
356-}
357-
358-func abs(x int) int {
359- if x < 0 {
360- return -x
361- }
362- return x
363-}
364-func overlap(a, b image.Rectangle) float64 {
365- intersection := a.Intersect(b)
366- area := min(a.Dx()*a.Dy(), b.Dx()*b.Dy())
367- if intersection.Empty() || area <= 0 {
368- return 0
369- }
370- return float64(intersection.Dx()*intersection.Dy()) / float64(area)
371-}
372-func targetRect(t ports.HintTarget) image.Rectangle {
373- return image.Rect(t.Rect.X, t.Rect.Y, t.Rect.X+t.Rect.W, t.Rect.Y+t.Rect.H)
374-}
375-func recognizeBand(ctx context.Context, binary, lang string, img image.Image) ([]OCRWord, error) {
376- return recognizeImage(ctx, binary, lang, img, "11", false)
377-}
378-
379-// pgmReader streams a grayscale image as binary PGM (P5), row by row.
380-type pgmReader struct {
381- head []byte
382- img *image.Gray
383- y int
384- row []byte
385-}
386-
387-func newPGMReader(img *image.Gray) *pgmReader {
388- b := img.Bounds()
389- return &pgmReader{head: fmt.Appendf(nil, "P5\n%d %d\n255\n", b.Dx(), b.Dy()), img: img}
390-}
391-
392-func (r *pgmReader) Read(p []byte) (int, error) {
393- n := 0
394- for n < len(p) {
395- if len(r.head) > 0 {
396- c := copy(p[n:], r.head)
397- r.head, n = r.head[c:], n+c
398- continue
399- }
400- if len(r.row) == 0 {
401- b := r.img.Bounds()
402- if r.y >= b.Dy() {
403- break
404- }
405- r.row = r.img.Pix[(r.y+b.Min.Y-r.img.Rect.Min.Y)*r.img.Stride+(b.Min.X-r.img.Rect.Min.X):][:b.Dx()]
406- r.y++
407- }
408- c := copy(p[n:], r.row)
409- r.row, n = r.row[c:], n+c
410- }
411- if n == 0 {
412- return 0, io.EOF
413- }
414- return n, nil
415-}
416-
417-func recognizeImage(ctx context.Context, binary, lang string, img image.Image, psm string, normalized bool) ([]OCRWord, error) {
418- args := []string{"stdin", "stdout", "--psm", psm, "-l", lang}
419- if normalized {
420- args = append(args, "-c", "tessedit_do_invert=0")
421- }
422- args = append(args, "tsv")
423- cmd := exec.CommandContext(ctx, binary, args...)
424- cmd.Env = append(os.Environ(), "OMP_THREAD_LIMIT=1")
425- if g, ok := img.(*image.Gray); ok && rawPGM {
426- // Raw grayscale: no deflate, no encoder buffers.
427- cmd.Stdin = newPGMReader(g)
428- } else {
429- // The PNG streams into tesseract; no encoded copy is held.
430- pr, pw := io.Pipe()
431- defer func() { _ = pr.Close() }()
432- go func() {
433- enc := png.Encoder{CompressionLevel: pngLevel}
434- _ = pw.CloseWithError(enc.Encode(pw, img))
435- }()
436- cmd.Stdin = pr
437- }
438- var stderr bytes.Buffer
439- cmd.Stderr = &stderr
440- output, err := cmd.Output()
441- if err != nil {
442- return nil, fmt.Errorf("tesseract: %w: %s", err, stderr.String())
443- }
444- return parseTSV(output, image.Point{})
445-}
446-
447-// upscaleBand doubles a grayscale band with nearest-neighbour sampling.
448-func upscaleBand(src *image.Gray) *image.Gray {
449- b := src.Bounds()
450- dst := image.NewGray(image.Rect(0, 0, b.Dx()*2, b.Dy()*2))
451- for y := 0; y < b.Dy(); y++ {
452- row := src.Pix[(y+b.Min.Y-src.Rect.Min.Y)*src.Stride+(b.Min.X-src.Rect.Min.X):][:b.Dx()]
453- top := dst.Pix[2*y*dst.Stride:][:dst.Stride]
454- for x, v := range row {
455- top[2*x], top[2*x+1] = v, v
456- }
457- copy(dst.Pix[(2*y+1)*dst.Stride:], top)
458- }
459- return dst
460-}
D internal/adapters/hints/ocr_test.go +0 -272
@@ -1,272 +0,0 @@
1-package hints
2-
3-import (
4- "context"
5- "errors"
6- "image"
7- "image/draw"
8- "image/png"
9- "io"
10- "os"
11- "os/exec"
12- "path/filepath"
13- "reflect"
14- "strings"
15- "testing"
16- "time"
17-
18- "github.com/stretchr/testify/mock"
19-
20- "github.com/bnema/neferwl/internal/ports"
21-)
22-
23-func TestTextLink(t *testing.T) {
24- yes := []string{"ermite.bnema.dev", "google.fr", "example.org/path?q=1", "https.google.fr", "https://example.org", "file:///tmp/main.go", "ftp://host/file", "www.main.go", "user@example.org", "127.0.0.1", "192.168.1.1:8080", "(https://example.org),", "htps://google.fr", "https:/example.org"}
25- no := []string{"ermite-runner", "main.go", "README.md", "v1.2.3", "1.5", "e.g.", "report.txt", "source.rs", "alpha", "999.999.999.999"}
26- for _, s := range yes {
27- t.Run(s, func(t *testing.T) {
28- if !isTextLink(s) {
29- t.Errorf("not accepted: %s", s)
30- }
31- })
32- }
33- for _, s := range no {
34- t.Run(s, func(t *testing.T) {
35- if isTextLink(s) {
36- t.Errorf("incorrectly accepted: %s", s)
37- }
38- })
39- }
40-}
41-func TestParseTSV(t *testing.T) {
42- data := "level\tpage_num\tblock_num\tpar_num\tline_num\tword_num\tleft\ttop\twidth\theight\tconf\ttext\n5\t1\t1\t1\t1\t1\t10\t20\t30\t12\t91.5\texample.org\n"
43- words, err := parseTSV([]byte(data), image.Pt(0, 40))
44- if err != nil || len(words) != 1 {
45- t.Fatalf("%v %v", words, err)
46- }
47- if words[0].Rect != image.Rect(10, 60, 40, 72) || words[0].Confidence != 91.5 {
48- t.Fatal(words)
49- }
50- _, err = parseTSV([]byte(strings.ReplaceAll(data, "\t10\t", "\tbad\t")), image.Point{})
51- if err == nil {
52- t.Fatal("expected malformed coordinate error")
53- }
54-}
55-func TestJoinedOCRLink(t *testing.T) {
56- words := []OCRWord{{Text: "https:/", Rect: image.Rect(20, 20, 90, 40)}, {Text: "/example.org", Rect: image.Rect(91, 20, 200, 40)}}
57- got := textLinks(words, 2)
58- if len(got) != 1 || got[0].Rect != (ports.Rect{X: 10, Y: 10, W: 90, H: 10}) {
59- t.Fatal(got)
60- }
61-}
62-func TestRealOCR(t *testing.T) {
63- if _, err := exec.LookPath("tesseract"); err != nil {
64- t.Skip("tesseract not on PATH")
65- }
66- for _, name := range []string{"browser", "terminal"} {
67- t.Run(name, func(t *testing.T) {
68- f, err := os.Open("testdata/" + name + ".png")
69- if err != nil {
70- t.Fatal(err)
71- }
72- defer f.Close()
73- src, err := png.Decode(f)
74- if err != nil {
75- t.Fatal(err)
76- }
77- img := image.NewRGBA(src.Bounds())
78- draw.Draw(img, img.Bounds(), src, src.Bounds().Min, draw.Src)
79- start := time.Now()
80- words, bands, err := readWords(context.Background(), toGray(img))
81- if err != nil {
82- t.Fatal(err)
83- }
84- links := textLinks(words, 1)
85- t.Logf("%s %dx%d ocr_ms=%.1f words=%d links=%d bands=%d", name, img.Bounds().Dx(), img.Bounds().Dy(), ms(time.Since(start)), len(words), len(links), bands)
86- for _, w := range words {
87- t.Logf("%s %v", w.Text, w.Rect)
88- }
89- if name == "terminal" {
90- for _, r := range []image.Rectangle{image.Rect(160, 65, 380, 85), image.Rect(140, 90, 490, 110), image.Rect(95, 115, 630, 138), image.Rect(120, 140, 330, 163), image.Rect(385, 140, 740, 163)} {
91- found := false
92- for _, l := range links {
93- if overlap(targetRect(l), r) > .5 {
94- found = true
95- }
96- }
97- if !found {
98- t.Errorf("missing terminal URL near %v", r)
99- }
100- }
101- if len(links) != 5 {
102- t.Errorf("want 5 terminal links, got %d: %v", len(links), links)
103- }
104- } else if len(links) == 0 {
105- t.Error("no browser links")
106- }
107- })
108- }
109-}
110-
111-func TestSmallGlyphOCR(t *testing.T) {
112- binary, err := exec.LookPath("tesseract")
113- if err != nil {
114- t.Skip("tesseract not on PATH")
115- }
116- f, err := os.Open("testdata/terminal.png")
117- if err != nil {
118- t.Fatal(err)
119- }
120- defer f.Close()
121- src, err := png.Decode(f)
122- if err != nil {
123- t.Fatal(err)
124- }
125- small := image.NewGray(image.Rect(0, 0, 640, 200))
126- for y := 0; y < 200; y++ {
127- for x := 0; x < 640; x++ {
128- small.Set(x, y, src.At(x*2, y*2))
129- }
130- }
131- lang, err := ocrLanguage(context.Background(), binary)
132- if err != nil {
133- t.Fatal(err)
134- }
135- for _, factor := range []int{1, 2} {
136- var img image.Image = small
137- if factor == 2 {
138- img = upscaleBand(small)
139- }
140- start := time.Now()
141- words, err := recognizeBand(context.Background(), binary, lang, img)
142- if err != nil {
143- t.Fatal(err)
144- }
145- t.Logf("glyph experiment %dx: %.1f ms, words=%d links=%d", factor, ms(time.Since(start)), len(words), len(textLinks(words, 1)))
146- }
147-}
148-
149-func TestPGMReader(t *testing.T) {
150- g := image.NewGray(image.Rect(0, 0, 3, 3))
151- for i := range g.Pix {
152- g.Pix[i] = uint8(i + 1)
153- }
154- sub := g.SubImage(image.Rect(1, 1, 3, 3)).(*image.Gray)
155- got, err := io.ReadAll(newPGMReader(sub))
156- if err != nil {
157- t.Fatal(err)
158- }
159- if want := "P5\n2 2\n255\n" + string([]byte{5, 6, 8, 9}); string(got) != want {
160- t.Fatalf("%q", got)
161- }
162-}
163-
164-// Tesseract must read the raw PGM exactly as it reads the PNG.
165-func TestPGMMatchesPNG(t *testing.T) {
166- binary, err := exec.LookPath("tesseract")
167- if err != nil {
168- t.Skip("tesseract not on PATH")
169- }
170- f, err := os.Open("testdata/terminal.png")
171- if err != nil {
172- t.Fatal(err)
173- }
174- defer f.Close()
175- src, err := png.Decode(f)
176- if err != nil {
177- t.Fatal(err)
178- }
179- rgba := image.NewRGBA(src.Bounds())
180- draw.Draw(rgba, rgba.Bounds(), src, src.Bounds().Min, draw.Src)
181- gray := toGray(rgba)
182- lang, err := ocrLanguage(context.Background(), binary)
183- if err != nil {
184- t.Fatal(err)
185- }
186- defer func(v bool) { rawPGM = v }(rawPGM)
187- var res [2][]OCRWord
188- for i, raw := range []bool{false, true} {
189- rawPGM = raw
190- if res[i], err = recognizeBand(context.Background(), binary, lang, gray); err != nil {
191- t.Fatal(err)
192- }
193- }
194- if len(res[0]) == 0 || !reflect.DeepEqual(res[0], res[1]) {
195- t.Fatalf("PNG %d words, PGM %d words", len(res[0]), len(res[1]))
196- }
197-}
198-
199-// TestRecognizeCapsProcesses runs the bands of several windows through a
200-// stand-in tesseract that logs when each process starts and ends.
201-func TestRecognizeCapsProcesses(t *testing.T) {
202- dir := t.TempDir()
203- logPath := filepath.Join(dir, "log")
204- script := filepath.Join(dir, "tesseract")
205- body := "#!/bin/sh\ncat >/dev/null\necho s >> " + logPath + "\nsleep 0.1\necho e >> " + logPath + "\nprintf 'level\\tpage_num\\tblock_num\\tpar_num\\tline_num\\tword_num\\tleft\\ttop\\twidth\\theight\\tconf\\ttext\\n'\n"
206- if err := os.WriteFile(script, []byte(body), 0o700); err != nil {
207- t.Fatal(err)
208- }
209- old := ocrWorkers
210- ocrWorkers = 2
211- t.Cleanup(func() { ocrWorkers = old })
212- w := &worker{ocr: ocrState{done: true, binary: script, lang: "eng"}}
213- var jobs []*bandJob
214- for range 8 { // several windows' bands
215- g := image.NewGray(image.Rect(0, 0, 8, 8))
216- jobs = append(jobs, &bandJob{ext: g.Rect, own: [2]int{0, 8}, gray: g})
217- }
218- w.recognize(context.Background(), jobs)
219- data, err := os.ReadFile(logPath)
220- if err != nil {
221- t.Fatal(err)
222- }
223- cur, peak, starts := 0, 0, 0
224- for _, l := range strings.Fields(string(data)) {
225- if l == "s" {
226- cur++
227- starts++
228- peak = max(peak, cur)
229- } else {
230- cur--
231- }
232- }
233- if starts != 8 || peak > 2 {
234- t.Fatalf("starts=%d peak=%d", starts, peak)
235- }
236- for _, j := range jobs {
237- if j.err != nil {
238- t.Fatal(j.err)
239- }
240- }
241-}
242-
243-func TestLanguageDiscoveryTimeout(t *testing.T) {
244- t.Setenv("TESSDATA_PREFIX", t.TempDir())
245- runner := newMocklanguageRunner(t)
246- parent := context.Background()
247- runner.EXPECT().Output(mock.Anything, "tesseract", []string{"--list-langs"}).RunAndReturn(
248- func(ctx context.Context, _ string, _ ...string) ([]byte, error) {
249- deadline, ok := ctx.Deadline()
250- if !ok || time.Until(deadline) > languageTimeout {
251- t.Fatal("language discovery has no bounded deadline")
252- }
253- <-ctx.Done()
254- return nil, ctx.Err()
255- }).Once()
256- lang, err := discoverLanguage(parent, "tesseract", runner, 0)
257- if lang != "" || !errors.Is(err, context.DeadlineExceeded) || parent.Err() != nil {
258- t.Fatalf("language=%q error=%v parent=%v", lang, err, parent.Err())
259- }
260-}
261-
262-func TestLanguageDiscoveryFailureKeepsAccessibilityOnly(t *testing.T) {
263- w := &worker{ocr: ocrState{done: true, err: context.DeadlineExceeded}}
264- ar := atspiResult{targets: []ports.HintTarget{{Window: 1, Rect: ports.Rect{W: 20, H: 10}}}}
265- req := ports.HintScanRequest{Width: 100, Height: 100, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: 100, H: 100}}}}
266- if w.needFrame(context.Background(), req, ar) {
267- t.Fatal("timed out discovery must not require an OCR capture")
268- }
269- if err := w.ocr.resolve(context.Background()); !errors.Is(err, context.DeadlineExceeded) {
270- t.Fatalf("discovery failure was not retained: %v", err)
271- }
272-}
@@ -1,272 +0,0 @@
1-package hints
2-
3-import (
4- "context"
5- "errors"
6- "image"
7- "image/draw"
8- "image/png"
9- "io"
10- "os"
11- "os/exec"
12- "path/filepath"
13- "reflect"
14- "strings"
15- "testing"
16- "time"
17-
18- "github.com/stretchr/testify/mock"
19-
20- "github.com/bnema/neferwl/internal/ports"
21-)
22-
23-func TestTextLink(t *testing.T) {
24- yes := []string{"ermite.bnema.dev", "google.fr", "example.org/path?q=1", "https.google.fr", "https://example.org", "file:///tmp/main.go", "ftp://host/file", "www.main.go", "user@example.org", "127.0.0.1", "192.168.1.1:8080", "(https://example.org),", "htps://google.fr", "https:/example.org"}
25- no := []string{"ermite-runner", "main.go", "README.md", "v1.2.3", "1.5", "e.g.", "report.txt", "source.rs", "alpha", "999.999.999.999"}
26- for _, s := range yes {
27- t.Run(s, func(t *testing.T) {
28- if !isTextLink(s) {
29- t.Errorf("not accepted: %s", s)
30- }
31- })
32- }
33- for _, s := range no {
34- t.Run(s, func(t *testing.T) {
35- if isTextLink(s) {
36- t.Errorf("incorrectly accepted: %s", s)
37- }
38- })
39- }
40-}
41-func TestParseTSV(t *testing.T) {
42- data := "level\tpage_num\tblock_num\tpar_num\tline_num\tword_num\tleft\ttop\twidth\theight\tconf\ttext\n5\t1\t1\t1\t1\t1\t10\t20\t30\t12\t91.5\texample.org\n"
43- words, err := parseTSV([]byte(data), image.Pt(0, 40))
44- if err != nil || len(words) != 1 {
45- t.Fatalf("%v %v", words, err)
46- }
47- if words[0].Rect != image.Rect(10, 60, 40, 72) || words[0].Confidence != 91.5 {
48- t.Fatal(words)
49- }
50- _, err = parseTSV([]byte(strings.ReplaceAll(data, "\t10\t", "\tbad\t")), image.Point{})
51- if err == nil {
52- t.Fatal("expected malformed coordinate error")
53- }
54-}
55-func TestJoinedOCRLink(t *testing.T) {
56- words := []OCRWord{{Text: "https:/", Rect: image.Rect(20, 20, 90, 40)}, {Text: "/example.org", Rect: image.Rect(91, 20, 200, 40)}}
57- got := textLinks(words, 2)
58- if len(got) != 1 || got[0].Rect != (ports.Rect{X: 10, Y: 10, W: 90, H: 10}) {
59- t.Fatal(got)
60- }
61-}
62-func TestRealOCR(t *testing.T) {
63- if _, err := exec.LookPath("tesseract"); err != nil {
64- t.Skip("tesseract not on PATH")
65- }
66- for _, name := range []string{"browser", "terminal"} {
67- t.Run(name, func(t *testing.T) {
68- f, err := os.Open("testdata/" + name + ".png")
69- if err != nil {
70- t.Fatal(err)
71- }
72- defer f.Close()
73- src, err := png.Decode(f)
74- if err != nil {
75- t.Fatal(err)
76- }
77- img := image.NewRGBA(src.Bounds())
78- draw.Draw(img, img.Bounds(), src, src.Bounds().Min, draw.Src)
79- start := time.Now()
80- words, bands, err := readWords(context.Background(), toGray(img))
81- if err != nil {
82- t.Fatal(err)
83- }
84- links := textLinks(words, 1)
85- t.Logf("%s %dx%d ocr_ms=%.1f words=%d links=%d bands=%d", name, img.Bounds().Dx(), img.Bounds().Dy(), ms(time.Since(start)), len(words), len(links), bands)
86- for _, w := range words {
87- t.Logf("%s %v", w.Text, w.Rect)
88- }
89- if name == "terminal" {
90- for _, r := range []image.Rectangle{image.Rect(160, 65, 380, 85), image.Rect(140, 90, 490, 110), image.Rect(95, 115, 630, 138), image.Rect(120, 140, 330, 163), image.Rect(385, 140, 740, 163)} {
91- found := false
92- for _, l := range links {
93- if overlap(targetRect(l), r) > .5 {
94- found = true
95- }
96- }
97- if !found {
98- t.Errorf("missing terminal URL near %v", r)
99- }
100- }
101- if len(links) != 5 {
102- t.Errorf("want 5 terminal links, got %d: %v", len(links), links)
103- }
104- } else if len(links) == 0 {
105- t.Error("no browser links")
106- }
107- })
108- }
109-}
110-
111-func TestSmallGlyphOCR(t *testing.T) {
112- binary, err := exec.LookPath("tesseract")
113- if err != nil {
114- t.Skip("tesseract not on PATH")
115- }
116- f, err := os.Open("testdata/terminal.png")
117- if err != nil {
118- t.Fatal(err)
119- }
120- defer f.Close()
121- src, err := png.Decode(f)
122- if err != nil {
123- t.Fatal(err)
124- }
125- small := image.NewGray(image.Rect(0, 0, 640, 200))
126- for y := 0; y < 200; y++ {
127- for x := 0; x < 640; x++ {
128- small.Set(x, y, src.At(x*2, y*2))
129- }
130- }
131- lang, err := ocrLanguage(context.Background(), binary)
132- if err != nil {
133- t.Fatal(err)
134- }
135- for _, factor := range []int{1, 2} {
136- var img image.Image = small
137- if factor == 2 {
138- img = upscaleBand(small)
139- }
140- start := time.Now()
141- words, err := recognizeBand(context.Background(), binary, lang, img)
142- if err != nil {
143- t.Fatal(err)
144- }
145- t.Logf("glyph experiment %dx: %.1f ms, words=%d links=%d", factor, ms(time.Since(start)), len(words), len(textLinks(words, 1)))
146- }
147-}
148-
149-func TestPGMReader(t *testing.T) {
150- g := image.NewGray(image.Rect(0, 0, 3, 3))
151- for i := range g.Pix {
152- g.Pix[i] = uint8(i + 1)
153- }
154- sub := g.SubImage(image.Rect(1, 1, 3, 3)).(*image.Gray)
155- got, err := io.ReadAll(newPGMReader(sub))
156- if err != nil {
157- t.Fatal(err)
158- }
159- if want := "P5\n2 2\n255\n" + string([]byte{5, 6, 8, 9}); string(got) != want {
160- t.Fatalf("%q", got)
161- }
162-}
163-
164-// Tesseract must read the raw PGM exactly as it reads the PNG.
165-func TestPGMMatchesPNG(t *testing.T) {
166- binary, err := exec.LookPath("tesseract")
167- if err != nil {
168- t.Skip("tesseract not on PATH")
169- }
170- f, err := os.Open("testdata/terminal.png")
171- if err != nil {
172- t.Fatal(err)
173- }
174- defer f.Close()
175- src, err := png.Decode(f)
176- if err != nil {
177- t.Fatal(err)
178- }
179- rgba := image.NewRGBA(src.Bounds())
180- draw.Draw(rgba, rgba.Bounds(), src, src.Bounds().Min, draw.Src)
181- gray := toGray(rgba)
182- lang, err := ocrLanguage(context.Background(), binary)
183- if err != nil {
184- t.Fatal(err)
185- }
186- defer func(v bool) { rawPGM = v }(rawPGM)
187- var res [2][]OCRWord
188- for i, raw := range []bool{false, true} {
189- rawPGM = raw
190- if res[i], err = recognizeBand(context.Background(), binary, lang, gray); err != nil {
191- t.Fatal(err)
192- }
193- }
194- if len(res[0]) == 0 || !reflect.DeepEqual(res[0], res[1]) {
195- t.Fatalf("PNG %d words, PGM %d words", len(res[0]), len(res[1]))
196- }
197-}
198-
199-// TestRecognizeCapsProcesses runs the bands of several windows through a
200-// stand-in tesseract that logs when each process starts and ends.
201-func TestRecognizeCapsProcesses(t *testing.T) {
202- dir := t.TempDir()
203- logPath := filepath.Join(dir, "log")
204- script := filepath.Join(dir, "tesseract")
205- body := "#!/bin/sh\ncat >/dev/null\necho s >> " + logPath + "\nsleep 0.1\necho e >> " + logPath + "\nprintf 'level\\tpage_num\\tblock_num\\tpar_num\\tline_num\\tword_num\\tleft\\ttop\\twidth\\theight\\tconf\\ttext\\n'\n"
206- if err := os.WriteFile(script, []byte(body), 0o700); err != nil {
207- t.Fatal(err)
208- }
209- old := ocrWorkers
210- ocrWorkers = 2
211- t.Cleanup(func() { ocrWorkers = old })
212- w := &worker{ocr: ocrState{done: true, binary: script, lang: "eng"}}
213- var jobs []*bandJob
214- for range 8 { // several windows' bands
215- g := image.NewGray(image.Rect(0, 0, 8, 8))
216- jobs = append(jobs, &bandJob{ext: g.Rect, own: [2]int{0, 8}, gray: g})
217- }
218- w.recognize(context.Background(), jobs)
219- data, err := os.ReadFile(logPath)
220- if err != nil {
221- t.Fatal(err)
222- }
223- cur, peak, starts := 0, 0, 0
224- for _, l := range strings.Fields(string(data)) {
225- if l == "s" {
226- cur++
227- starts++
228- peak = max(peak, cur)
229- } else {
230- cur--
231- }
232- }
233- if starts != 8 || peak > 2 {
234- t.Fatalf("starts=%d peak=%d", starts, peak)
235- }
236- for _, j := range jobs {
237- if j.err != nil {
238- t.Fatal(j.err)
239- }
240- }
241-}
242-
243-func TestLanguageDiscoveryTimeout(t *testing.T) {
244- t.Setenv("TESSDATA_PREFIX", t.TempDir())
245- runner := newMocklanguageRunner(t)
246- parent := context.Background()
247- runner.EXPECT().Output(mock.Anything, "tesseract", []string{"--list-langs"}).RunAndReturn(
248- func(ctx context.Context, _ string, _ ...string) ([]byte, error) {
249- deadline, ok := ctx.Deadline()
250- if !ok || time.Until(deadline) > languageTimeout {
251- t.Fatal("language discovery has no bounded deadline")
252- }
253- <-ctx.Done()
254- return nil, ctx.Err()
255- }).Once()
256- lang, err := discoverLanguage(parent, "tesseract", runner, 0)
257- if lang != "" || !errors.Is(err, context.DeadlineExceeded) || parent.Err() != nil {
258- t.Fatalf("language=%q error=%v parent=%v", lang, err, parent.Err())
259- }
260-}
261-
262-func TestLanguageDiscoveryFailureKeepsAccessibilityOnly(t *testing.T) {
263- w := &worker{ocr: ocrState{done: true, err: context.DeadlineExceeded}}
264- ar := atspiResult{targets: []ports.HintTarget{{Window: 1, Rect: ports.Rect{W: 20, H: 10}}}}
265- req := ports.HintScanRequest{Width: 100, Height: 100, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: 100, H: 100}}}}
266- if w.needFrame(context.Background(), req, ar) {
267- t.Fatal("timed out discovery must not require an OCR capture")
268- }
269- if err := w.ocr.resolve(context.Background()); !errors.Is(err, context.DeadlineExceeded) {
270- t.Fatalf("discovery failure was not retained: %v", err)
271- }
272-}
D internal/adapters/hints/scan_bench_test.go +0 -49
@@ -1,49 +0,0 @@
1-package hints
2-
3-import (
4- "context"
5- "os"
6- "os/exec"
7- "runtime"
8- "testing"
9- "time"
10-
11- "github.com/bnema/neferwl/internal/ports"
12-)
13-
14-// BenchmarkScan runs whole scans of a 5120x2160 frame with a noisy window
15-// (the cache never hits) and reports the Go heap bytes allocated per scan.
16-func BenchmarkScan(b *testing.B) {
17- if _, err := exec.LookPath("tesseract"); err != nil {
18- b.Skip("tesseract not on PATH")
19- }
20- const w, h = 5120, 2160
21- ctx := context.Background()
22- w0 := &worker{}
23- requests := make(chan ports.HintScanRequest)
24- captures := make(chan ports.CaptureRequest, 1)
25- captured := make(chan ports.CaptureDone, 1)
26- results := make(chan ports.HintScanResult, 1)
27- req := ports.HintScanRequest{Output: "o", Width: w, Height: h, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: w, H: h}}}}
28- if os.Getenv("NEFERWL_HINT_BENCH_PGM") == "0" {
29- rawPGM = false
30- }
31- b.ReportAllocs()
32- var peak uint64
33- for i := range b.N {
34- go func() {
35- creq := <-captures
36- creq.Mapped[0] = byte(i) // each scan differs: no cache hit
37- creq.Mapped[1] = byte(i >> 8)
38- captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
39- }()
40- res := w0.serve(ctx, req, requests, captures, captured, results, 5*time.Second)
41- if res.Err != nil {
42- b.Fatal(res.Err)
43- }
44- var m runtime.MemStats
45- runtime.ReadMemStats(&m)
46- peak = max(peak, m.HeapSys-m.HeapReleased)
47- }
48- b.ReportMetric(float64(peak)/1e6, "heapMB")
49-}
@@ -1,49 +0,0 @@
1-package hints
2-
3-import (
4- "context"
5- "os"
6- "os/exec"
7- "runtime"
8- "testing"
9- "time"
10-
11- "github.com/bnema/neferwl/internal/ports"
12-)
13-
14-// BenchmarkScan runs whole scans of a 5120x2160 frame with a noisy window
15-// (the cache never hits) and reports the Go heap bytes allocated per scan.
16-func BenchmarkScan(b *testing.B) {
17- if _, err := exec.LookPath("tesseract"); err != nil {
18- b.Skip("tesseract not on PATH")
19- }
20- const w, h = 5120, 2160
21- ctx := context.Background()
22- w0 := &worker{}
23- requests := make(chan ports.HintScanRequest)
24- captures := make(chan ports.CaptureRequest, 1)
25- captured := make(chan ports.CaptureDone, 1)
26- results := make(chan ports.HintScanResult, 1)
27- req := ports.HintScanRequest{Output: "o", Width: w, Height: h, Scale: 1, Windows: []ports.HintWindow{{ID: 1, Rect: ports.Rect{W: w, H: h}}}}
28- if os.Getenv("NEFERWL_HINT_BENCH_PGM") == "0" {
29- rawPGM = false
30- }
31- b.ReportAllocs()
32- var peak uint64
33- for i := range b.N {
34- go func() {
35- creq := <-captures
36- creq.Mapped[0] = byte(i) // each scan differs: no cache hit
37- creq.Mapped[1] = byte(i >> 8)
38- captured <- ports.CaptureDone{ID: creq.ID, Output: creq.Output}
39- }()
40- res := w0.serve(ctx, req, requests, captures, captured, results, 5*time.Second)
41- if res.Err != nil {
42- b.Fatal(res.Err)
43- }
44- var m runtime.MemStats
45- runtime.ReadMemStats(&m)
46- peak = max(peak, m.HeapSys-m.HeapReleased)
47- }
48- b.ReportMetric(float64(peak)/1e6, "heapMB")
49-}
D internal/adapters/hints/strips_test.go +0 -124
@@ -1,124 +0,0 @@
1-package hints
2-
3-import (
4- "context"
5- "image"
6- "image/color"
7- "image/draw"
8- "os/exec"
9- "runtime"
10- "sort"
11- "sync"
12- "time"
13-)
14-
15-type stripOffset struct {
16- source, packed image.Rectangle
17- factor int
18-}
19-type ocrStrip struct {
20- img *image.Gray
21- offsets []stripOffset
22-}
23-
24-func packLines(src *image.RGBA, lines []image.Rectangle, workers int, normalize bool) []ocrStrip {
25- n := min(workers, len(lines))
26- if n == 0 {
27- return nil
28- }
29- groups := make([][]image.Rectangle, n)
30- areas := make([]int, n)
31- lines = append([]image.Rectangle(nil), lines...)
32- sort.Slice(lines, func(i, j int) bool { return lines[i].Dx()*lines[i].Dy() > lines[j].Dx()*lines[j].Dy() })
33- for _, r := range lines {
34- best := 0
35- for i := range n {
36- if areas[i] < areas[best] {
37- best = i
38- }
39- }
40- groups[best] = append(groups[best], r)
41- areas[best] += r.Dx() * r.Dy()
42- }
43- strips := make([]ocrStrip, n)
44- for i, group := range groups {
45- width, height := 0, 8
46- for _, r := range group {
47- factor := 1
48- if r.Dy() < 12 {
49- factor = 2
50- }
51- crop := r.Inset(-4).Intersect(src.Bounds())
52- packed := image.Rect(4, height, 4+crop.Dx()*factor, height+crop.Dy()*factor)
53- strips[i].offsets = append(strips[i].offsets, stripOffset{crop, packed, factor})
54- width = max(width, packed.Max.X+4)
55- height = packed.Max.Y + 8
56- }
57- dst := image.NewGray(image.Rect(0, 0, width, height))
58- draw.Draw(dst, dst.Bounds(), image.NewUniform(color.White), image.Point{}, draw.Src)
59- for _, off := range strips[i].offsets {
60- // The padded perimeter estimates the background without depending on text colour.
61- var sum, count int
62- r := off.source
63- for x := r.Min.X; x < r.Max.X; x++ {
64- sum += int(color.GrayModel.Convert(src.At(x, r.Min.Y)).(color.Gray).Y)
65- count++
66- }
67- invert := normalize && count > 0 && sum/count < 128
68- for y := off.packed.Min.Y; y < off.packed.Max.Y; y++ {
69- for x := off.packed.Min.X; x < off.packed.Max.X; x++ {
70- v := color.GrayModel.Convert(src.At(r.Min.X+(x-off.packed.Min.X)/off.factor, r.Min.Y+(y-off.packed.Min.Y)/off.factor)).(color.Gray).Y
71- if invert {
72- v = 255 - v
73- }
74- dst.SetGray(x, y, color.Gray{Y: v})
75- }
76- }
77- }
78- strips[i].img = dst
79- }
80- return strips
81-}
82-
83-func readStrips(ctx context.Context, img *image.RGBA, lines []image.Rectangle, workers int, normalize bool) ([]OCRWord, int, error) {
84- binary, err := exec.LookPath("tesseract")
85- if err != nil {
86- return nil, 0, err
87- }
88- ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
89- defer cancel()
90- lang, err := ocrLanguage(ctx, binary)
91- if err != nil {
92- return nil, 0, err
93- }
94- strips := packLines(img, lines, min(workers, runtime.GOMAXPROCS(0), 16), normalize)
95- results := make([][]OCRWord, len(strips))
96- errs := make([]error, len(strips))
97- var wg sync.WaitGroup
98- for i, s := range strips {
99- wg.Go(func() {
100- words, e := recognizeImage(ctx, binary, lang, s.img, "6", normalize)
101- errs[i] = e
102- for _, w := range words {
103- center := image.Pt((w.Rect.Min.X+w.Rect.Max.X)/2, (w.Rect.Min.Y+w.Rect.Max.Y)/2)
104- for _, off := range s.offsets {
105- if center.In(off.packed) {
106- r := w.Rect.Sub(off.packed.Min)
107- w.Rect = image.Rect(r.Min.X/off.factor, r.Min.Y/off.factor, (r.Max.X+off.factor-1)/off.factor, (r.Max.Y+off.factor-1)/off.factor).Add(off.source.Min)
108- results[i] = append(results[i], w)
109- break
110- }
111- }
112- }
113- })
114- }
115- wg.Wait()
116- var words []OCRWord
117- for i := range strips {
118- if errs[i] != nil {
119- return nil, len(strips), errs[i]
120- }
121- words = append(words, results[i]...)
122- }
123- return words, len(strips), nil
124-}
@@ -1,124 +0,0 @@
1-package hints
2-
3-import (
4- "context"
5- "image"
6- "image/color"
7- "image/draw"
8- "os/exec"
9- "runtime"
10- "sort"
11- "sync"
12- "time"
13-)
14-
15-type stripOffset struct {
16- source, packed image.Rectangle
17- factor int
18-}
19-type ocrStrip struct {
20- img *image.Gray
21- offsets []stripOffset
22-}
23-
24-func packLines(src *image.RGBA, lines []image.Rectangle, workers int, normalize bool) []ocrStrip {
25- n := min(workers, len(lines))
26- if n == 0 {
27- return nil
28- }
29- groups := make([][]image.Rectangle, n)
30- areas := make([]int, n)
31- lines = append([]image.Rectangle(nil), lines...)
32- sort.Slice(lines, func(i, j int) bool { return lines[i].Dx()*lines[i].Dy() > lines[j].Dx()*lines[j].Dy() })
33- for _, r := range lines {
34- best := 0
35- for i := range n {
36- if areas[i] < areas[best] {
37- best = i
38- }
39- }
40- groups[best] = append(groups[best], r)
41- areas[best] += r.Dx() * r.Dy()
42- }
43- strips := make([]ocrStrip, n)
44- for i, group := range groups {
45- width, height := 0, 8
46- for _, r := range group {
47- factor := 1
48- if r.Dy() < 12 {
49- factor = 2
50- }
51- crop := r.Inset(-4).Intersect(src.Bounds())
52- packed := image.Rect(4, height, 4+crop.Dx()*factor, height+crop.Dy()*factor)
53- strips[i].offsets = append(strips[i].offsets, stripOffset{crop, packed, factor})
54- width = max(width, packed.Max.X+4)
55- height = packed.Max.Y + 8
56- }
57- dst := image.NewGray(image.Rect(0, 0, width, height))
58- draw.Draw(dst, dst.Bounds(), image.NewUniform(color.White), image.Point{}, draw.Src)
59- for _, off := range strips[i].offsets {
60- // The padded perimeter estimates the background without depending on text colour.
61- var sum, count int
62- r := off.source
63- for x := r.Min.X; x < r.Max.X; x++ {
64- sum += int(color.GrayModel.Convert(src.At(x, r.Min.Y)).(color.Gray).Y)
65- count++
66- }
67- invert := normalize && count > 0 && sum/count < 128
68- for y := off.packed.Min.Y; y < off.packed.Max.Y; y++ {
69- for x := off.packed.Min.X; x < off.packed.Max.X; x++ {
70- v := color.GrayModel.Convert(src.At(r.Min.X+(x-off.packed.Min.X)/off.factor, r.Min.Y+(y-off.packed.Min.Y)/off.factor)).(color.Gray).Y
71- if invert {
72- v = 255 - v
73- }
74- dst.SetGray(x, y, color.Gray{Y: v})
75- }
76- }
77- }
78- strips[i].img = dst
79- }
80- return strips
81-}
82-
83-func readStrips(ctx context.Context, img *image.RGBA, lines []image.Rectangle, workers int, normalize bool) ([]OCRWord, int, error) {
84- binary, err := exec.LookPath("tesseract")
85- if err != nil {
86- return nil, 0, err
87- }
88- ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
89- defer cancel()
90- lang, err := ocrLanguage(ctx, binary)
91- if err != nil {
92- return nil, 0, err
93- }
94- strips := packLines(img, lines, min(workers, runtime.GOMAXPROCS(0), 16), normalize)
95- results := make([][]OCRWord, len(strips))
96- errs := make([]error, len(strips))
97- var wg sync.WaitGroup
98- for i, s := range strips {
99- wg.Go(func() {
100- words, e := recognizeImage(ctx, binary, lang, s.img, "6", normalize)
101- errs[i] = e
102- for _, w := range words {
103- center := image.Pt((w.Rect.Min.X+w.Rect.Max.X)/2, (w.Rect.Min.Y+w.Rect.Max.Y)/2)
104- for _, off := range s.offsets {
105- if center.In(off.packed) {
106- r := w.Rect.Sub(off.packed.Min)
107- w.Rect = image.Rect(r.Min.X/off.factor, r.Min.Y/off.factor, (r.Max.X+off.factor-1)/off.factor, (r.Max.Y+off.factor-1)/off.factor).Add(off.source.Min)
108- results[i] = append(results[i], w)
109- break
110- }
111- }
112- }
113- })
114- }
115- wg.Wait()
116- var words []OCRWord
117- for i := range strips {
118- if errs[i] != nil {
119- return nil, len(strips), errs[i]
120- }
121- words = append(words, results[i]...)
122- }
123- return words, len(strips), nil
124-}
D internal/adapters/hints/target_source_mock_test.go +0 -174
@@ -1,174 +0,0 @@
1-// Code generated by mockery; DO NOT EDIT.
2-// github.com/vektra/mockery
3-// template: testify
4-
5-package hints
6-
7-import (
8- "context"
9-
10- "github.com/bnema/neferwl/internal/ports"
11- mock "github.com/stretchr/testify/mock"
12-)
13-
14-// newMocktargetSource creates a new instance of mocktargetSource. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
15-// The first argument is typically a *testing.T value.
16-func newMocktargetSource(t interface {
17- mock.TestingT
18- Cleanup(func())
19-}) *mocktargetSource {
20- if helper, ok := t.(interface{ Helper() }); ok {
21- helper.Helper()
22- }
23-
24- mock := &mocktargetSource{}
25- mock.Mock.Test(t)
26-
27- t.Cleanup(func() {
28- if helper, ok := t.(interface{ Helper() }); ok {
29- helper.Helper()
30- }
31- mock.AssertExpectations(t)
32- })
33-
34- return mock
35-}
36-
37-// mocktargetSource is an autogenerated mock type for the targetSource type
38-type mocktargetSource struct {
39- mock.Mock
40-}
41-
42-type mocktargetSource_Expecter struct {
43- mock *mock.Mock
44-}
45-
46-func (_m *mocktargetSource) EXPECT() *mocktargetSource_Expecter {
47- return &mocktargetSource_Expecter{mock: &_m.Mock}
48-}
49-
50-// Close provides a mock function for the type mocktargetSource
51-func (_mock *mocktargetSource) Close() error {
52- ret := _mock.Called()
53-
54- if len(ret) == 0 {
55- panic("no return value specified for Close")
56- }
57-
58- var r0 error
59- if returnFunc, ok := ret.Get(0).(func() error); ok {
60- r0 = returnFunc()
61- } else {
62- r0 = ret.Error(0)
63- }
64- return r0
65-}
66-
67-// mocktargetSource_Close_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Close'
68-type mocktargetSource_Close_Call struct {
69- *mock.Call
70-}
71-
72-// Close is a helper method to define mock.On call
73-func (_e *mocktargetSource_Expecter) Close() *mocktargetSource_Close_Call {
74- return &mocktargetSource_Close_Call{Call: _e.mock.On("Close")}
75-}
76-
77-func (_c *mocktargetSource_Close_Call) Run(run func()) *mocktargetSource_Close_Call {
78- _c.Call.Run(func(args mock.Arguments) {
79- run()
80- })
81- return _c
82-}
83-
84-func (_c *mocktargetSource_Close_Call) Return(err error) *mocktargetSource_Close_Call {
85- _c.Call.Return(err)
86- return _c
87-}
88-
89-func (_c *mocktargetSource_Close_Call) RunAndReturn(run func() error) *mocktargetSource_Close_Call {
90- _c.Call.Return(run)
91- return _c
92-}
93-
94-// Targets provides a mock function for the type mocktargetSource
95-func (_mock *mocktargetSource) Targets(ctx context.Context, windows []ports.HintWindow, linksOnly bool) ([]ports.HintTarget, map[ports.WindowID]bool, error) {
96- ret := _mock.Called(ctx, windows, linksOnly)
97-
98- if len(ret) == 0 {
99- panic("no return value specified for Targets")
100- }
101-
102- var r0 []ports.HintTarget
103- var r1 map[ports.WindowID]bool
104- var r2 error
105- if returnFunc, ok := ret.Get(0).(func(context.Context, []ports.HintWindow, bool) ([]ports.HintTarget, map[ports.WindowID]bool, error)); ok {
106- return returnFunc(ctx, windows, linksOnly)
107- }
108- if returnFunc, ok := ret.Get(0).(func(context.Context, []ports.HintWindow, bool) []ports.HintTarget); ok {
109- r0 = returnFunc(ctx, windows, linksOnly)
110- } else {
111- if ret.Get(0) != nil {
112- r0 = ret.Get(0).([]ports.HintTarget)
113- }
114- }
115- if returnFunc, ok := ret.Get(1).(func(context.Context, []ports.HintWindow, bool) map[ports.WindowID]bool); ok {
116- r1 = returnFunc(ctx, windows, linksOnly)
117- } else {
118- if ret.Get(1) != nil {
119- r1 = ret.Get(1).(map[ports.WindowID]bool)
120- }
121- }
122- if returnFunc, ok := ret.Get(2).(func(context.Context, []ports.HintWindow, bool) error); ok {
123- r2 = returnFunc(ctx, windows, linksOnly)
124- } else {
125- r2 = ret.Error(2)
126- }
127- return r0, r1, r2
128-}
129-
130-// mocktargetSource_Targets_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Targets'
131-type mocktargetSource_Targets_Call struct {
132- *mock.Call
133-}
134-
135-// Targets is a helper method to define mock.On call
136-// - ctx context.Context
137-// - windows []ports.HintWindow
138-// - linksOnly bool
139-func (_e *mocktargetSource_Expecter) Targets(ctx any, windows any, linksOnly any) *mocktargetSource_Targets_Call {
140- return &mocktargetSource_Targets_Call{Call: _e.mock.On("Targets", ctx, windows, linksOnly)}
141-}
142-
143-func (_c *mocktargetSource_Targets_Call) Run(run func(ctx context.Context, windows []ports.HintWindow, linksOnly bool)) *mocktargetSource_Targets_Call {
144- _c.Call.Run(func(args mock.Arguments) {
145- var arg0 context.Context
146- if args[0] != nil {
147- arg0 = args[0].(context.Context)
148- }
149- var arg1 []ports.HintWindow
150- if args[1] != nil {
151- arg1 = args[1].([]ports.HintWindow)
152- }
153- var arg2 bool
154- if args[2] != nil {
155- arg2 = args[2].(bool)
156- }
157- run(
158- arg0,
159- arg1,
160- arg2,
161- )
162- })
163- return _c
164-}
165-
166-func (_c *mocktargetSource_Targets_Call) Return(hintTargets []ports.HintTarget, windowIDToBool map[ports.WindowID]bool, err error) *mocktargetSource_Targets_Call {
167- _c.Call.Return(hintTargets, windowIDToBool, err)
168- return _c
169-}
170-
171-func (_c *mocktargetSource_Targets_Call) RunAndReturn(run func(ctx context.Context, windows []ports.HintWindow, linksOnly bool) ([]ports.HintTarget, map[ports.WindowID]bool, error)) *mocktargetSource_Targets_Call {
172- _c.Call.Return(run)
173- return _c
174-}
@@ -1,174 +0,0 @@
1-// Code generated by mockery; DO NOT EDIT.
2-// github.com/vektra/mockery
3-// template: testify
4-
5-package hints
6-
7-import (
8- "context"
9-
10- "github.com/bnema/neferwl/internal/ports"
11- mock "github.com/stretchr/testify/mock"
12-)
13-
14-// newMocktargetSource creates a new instance of mocktargetSource. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
15-// The first argument is typically a *testing.T value.
16-func newMocktargetSource(t interface {
17- mock.TestingT
18- Cleanup(func())
19-}) *mocktargetSource {
20- if helper, ok := t.(interface{ Helper() }); ok {
21- helper.Helper()
22- }
23-
24- mock := &mocktargetSource{}
25- mock.Mock.Test(t)
26-
27- t.Cleanup(func() {
28- if helper, ok := t.(interface{ Helper() }); ok {
29- helper.Helper()
30- }
31- mock.AssertExpectations(t)
32- })
33-
34- return mock
35-}
36-
37-// mocktargetSource is an autogenerated mock type for the targetSource type
38-type mocktargetSource struct {
39- mock.Mock
40-}
41-
42-type mocktargetSource_Expecter struct {
43- mock *mock.Mock
44-}
45-
46-func (_m *mocktargetSource) EXPECT() *mocktargetSource_Expecter {
47- return &mocktargetSource_Expecter{mock: &_m.Mock}
48-}
49-
50-// Close provides a mock function for the type mocktargetSource
51-func (_mock *mocktargetSource) Close() error {
52- ret := _mock.Called()
53-
54- if len(ret) == 0 {
55- panic("no return value specified for Close")
56- }
57-
58- var r0 error
59- if returnFunc, ok := ret.Get(0).(func() error); ok {
60- r0 = returnFunc()
61- } else {
62- r0 = ret.Error(0)
63- }
64- return r0
65-}
66-
67-// mocktargetSource_Close_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Close'
68-type mocktargetSource_Close_Call struct {
69- *mock.Call
70-}
71-
72-// Close is a helper method to define mock.On call
73-func (_e *mocktargetSource_Expecter) Close() *mocktargetSource_Close_Call {
74- return &mocktargetSource_Close_Call{Call: _e.mock.On("Close")}
75-}
76-
77-func (_c *mocktargetSource_Close_Call) Run(run func()) *mocktargetSource_Close_Call {
78- _c.Call.Run(func(args mock.Arguments) {
79- run()
80- })
81- return _c
82-}
83-
84-func (_c *mocktargetSource_Close_Call) Return(err error) *mocktargetSource_Close_Call {
85- _c.Call.Return(err)
86- return _c
87-}
88-
89-func (_c *mocktargetSource_Close_Call) RunAndReturn(run func() error) *mocktargetSource_Close_Call {
90- _c.Call.Return(run)
91- return _c
92-}
93-
94-// Targets provides a mock function for the type mocktargetSource
95-func (_mock *mocktargetSource) Targets(ctx context.Context, windows []ports.HintWindow, linksOnly bool) ([]ports.HintTarget, map[ports.WindowID]bool, error) {
96- ret := _mock.Called(ctx, windows, linksOnly)
97-
98- if len(ret) == 0 {
99- panic("no return value specified for Targets")
100- }
101-
102- var r0 []ports.HintTarget
103- var r1 map[ports.WindowID]bool
104- var r2 error
105- if returnFunc, ok := ret.Get(0).(func(context.Context, []ports.HintWindow, bool) ([]ports.HintTarget, map[ports.WindowID]bool, error)); ok {
106- return returnFunc(ctx, windows, linksOnly)
107- }
108- if returnFunc, ok := ret.Get(0).(func(context.Context, []ports.HintWindow, bool) []ports.HintTarget); ok {
109- r0 = returnFunc(ctx, windows, linksOnly)
110- } else {
111- if ret.Get(0) != nil {
112- r0 = ret.Get(0).([]ports.HintTarget)
113- }
114- }
115- if returnFunc, ok := ret.Get(1).(func(context.Context, []ports.HintWindow, bool) map[ports.WindowID]bool); ok {
116- r1 = returnFunc(ctx, windows, linksOnly)
117- } else {
118- if ret.Get(1) != nil {
119- r1 = ret.Get(1).(map[ports.WindowID]bool)
120- }
121- }
122- if returnFunc, ok := ret.Get(2).(func(context.Context, []ports.HintWindow, bool) error); ok {
123- r2 = returnFunc(ctx, windows, linksOnly)
124- } else {
125- r2 = ret.Error(2)
126- }
127- return r0, r1, r2
128-}
129-
130-// mocktargetSource_Targets_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Targets'
131-type mocktargetSource_Targets_Call struct {
132- *mock.Call
133-}
134-
135-// Targets is a helper method to define mock.On call
136-// - ctx context.Context
137-// - windows []ports.HintWindow
138-// - linksOnly bool
139-func (_e *mocktargetSource_Expecter) Targets(ctx any, windows any, linksOnly any) *mocktargetSource_Targets_Call {
140- return &mocktargetSource_Targets_Call{Call: _e.mock.On("Targets", ctx, windows, linksOnly)}
141-}
142-
143-func (_c *mocktargetSource_Targets_Call) Run(run func(ctx context.Context, windows []ports.HintWindow, linksOnly bool)) *mocktargetSource_Targets_Call {
144- _c.Call.Run(func(args mock.Arguments) {
145- var arg0 context.Context
146- if args[0] != nil {
147- arg0 = args[0].(context.Context)
148- }
149- var arg1 []ports.HintWindow
150- if args[1] != nil {
151- arg1 = args[1].([]ports.HintWindow)
152- }
153- var arg2 bool
154- if args[2] != nil {
155- arg2 = args[2].(bool)
156- }
157- run(
158- arg0,
159- arg1,
160- arg2,
161- )
162- })
163- return _c
164-}
165-
166-func (_c *mocktargetSource_Targets_Call) Return(hintTargets []ports.HintTarget, windowIDToBool map[ports.WindowID]bool, err error) *mocktargetSource_Targets_Call {
167- _c.Call.Return(hintTargets, windowIDToBool, err)
168- return _c
169-}
170-
171-func (_c *mocktargetSource_Targets_Call) RunAndReturn(run func(ctx context.Context, windows []ports.HintWindow, linksOnly bool) ([]ports.HintTarget, map[ports.WindowID]bool, error)) *mocktargetSource_Targets_Call {
172- _c.Call.Return(run)
173- return _c
174-}
D internal/adapters/hints/testdata/browser.png +0 -0 binary file: no text diff

binary file: no text diff

D internal/adapters/hints/testdata/terminal.png +0 -0 binary file: no text diff

binary file: no text diff

D internal/adapters/hints/testdata/underline.png +0 -0 binary file: no text diff

binary file: no text diff

A internal/adapters/hints/tile.go +122 -0
@@ -0,0 +1,122 @@
1+package hints
2+
3+const (
4+ // maxRegions is the protocol limit of regions per request.
5+ maxRegions = 64
6+ // wholeMaxSide is the longest frame side sent whole when the frame is not
7+ // 16:9: up to there the square letterbox of the detector loses little.
8+ wholeMaxSide = 2560
9+ // tileMaxH is the tallest tile; its width is 16/9 of it.
10+ tileMaxH = 1440
11+ // tileCutMargin is the distance, in physical pixels, to a tile's inner
12+ // edge under which a box counts as cut by it.
13+ tileCutMargin = 2
14+)
15+
16+// region is a crop of the frame, in physical pixels.
17+type region struct{ x, y, w, h int }
18+
19+// tileRegions returns the crops to detect separately on a width x height
20+// frame, or nil for one whole-frame detection (a 16:9 frame, or one whose
21+// long side is at most wholeMaxSide). Otherwise the frame is covered by exact
22+// 16:9 tiles (w*9 == h*16; explicit crops always use square letterboxing) of
23+// height min(height, 1440) rounded down to a multiple of 9, overlapping by at
24+// least 10 % of the tile side on each axis; the last tile of a row or column
25+// is aligned to the frame edge. Tiles are listed row by row. A frame that
26+// would need more than maxRegions tiles is detected whole.
27+func tileRegions(width, height int) []region {
28+ if width <= 0 || height <= 0 || width*9 == height*16 || max(width, height) <= wholeMaxSide {
29+ return nil
30+ }
31+ th := min(height, tileMaxH) / 9 * 9
32+ tw := th / 9 * 16
33+ if tw > width { // a tall frame: the width bounds the tile
34+ tw = width / 16 * 16
35+ th = tw / 16 * 9
36+ }
37+ if tw == 0 || th == 0 {
38+ return nil
39+ }
40+ xs, ys := tileStarts(width, tw), tileStarts(height, th)
41+ if len(xs)*len(ys) > maxRegions {
42+ return nil
43+ }
44+ out := make([]region, 0, len(xs)*len(ys))
45+ for _, y := range ys {
46+ for _, x := range xs {
47+ out = append(out, region{x, y, tw, th})
48+ }
49+ }
50+ return out
51+}
52+
53+// tileStarts returns the start offsets of tiles of size t covering length l,
54+// evenly spread, the last one flush with the end, with consecutive tiles
55+// overlapping by at least a tenth of t.
56+func tileStarts(l, t int) []int {
57+ if l <= t {
58+ return []int{0}
59+ }
60+ d := l - t
61+ maxStep := t * 9 / 10
62+ n := (d+maxStep-1)/maxStep + 1
63+ out := make([]int, n)
64+ for i := range out {
65+ out[i] = (d*i + n - 2) / (n - 1) // ceil(d*i/(n-1)): steps never exceed maxStep
66+ }
67+ return out
68+}
69+
70+// dropCutBoxes drops the boxes that a tile's inner edge cut (an edge of the
71+// tile that is not an edge of the frame) when another tile saw the same
72+// element uncut: the cut box would otherwise survive as a smaller box nested
73+// in the whole one. A box is "the same element" when the other box covers
74+// nestRatio of it. Without regions the boxes are returned as they are.
75+func dropCutBoxes(dets []detection, regions []region) []detection {
76+ if len(regions) == 0 {
77+ return dets
78+ }
79+ fw, fh := 0, 0
80+ for _, r := range regions {
81+ fw, fh = max(fw, r.x+r.w), max(fh, r.y+r.h)
82+ }
83+ cut := func(d detection) bool {
84+ if d.region < 0 || d.region >= len(regions) {
85+ return false
86+ }
87+ r := regions[d.region]
88+ m := float32(tileCutMargin)
89+ return (r.x > 0 && d.x1 < float32(r.x)+m) || (r.x+r.w < fw && d.x2 > float32(r.x+r.w)-m) ||
90+ (r.y > 0 && d.y1 < float32(r.y)+m) || (r.y+r.h < fh && d.y2 > float32(r.y+r.h)-m)
91+ }
92+ isCut := make([]bool, len(dets))
93+ for i, d := range dets {
94+ isCut[i] = cut(d)
95+ }
96+ out := make([]detection, 0, len(dets))
97+ for i, d := range dets {
98+ drop := false
99+ if isCut[i] {
100+ for j, o := range dets {
101+ if j != i && !isCut[j] && o.region != d.region && covered(d, o) {
102+ drop = true
103+ break
104+ }
105+ }
106+ }
107+ if !drop {
108+ out = append(out, d)
109+ }
110+ }
111+ return out
112+}
113+
114+// covered reports whether o covers at least nestRatio of d's area.
115+func covered(d, o detection) bool {
116+ w := min(d.x2, o.x2) - max(d.x1, o.x1)
117+ h := min(d.y2, o.y2) - max(d.y1, o.y1)
118+ if w <= 0 || h <= 0 {
119+ return false
120+ }
121+ return float64(w*h) >= nestRatio*float64((d.x2-d.x1)*(d.y2-d.y1))
122+}
@@ -0,0 +1,122 @@
1+package hints
2+
3+const (
4+ // maxRegions is the protocol limit of regions per request.
5+ maxRegions = 64
6+ // wholeMaxSide is the longest frame side sent whole when the frame is not
7+ // 16:9: up to there the square letterbox of the detector loses little.
8+ wholeMaxSide = 2560
9+ // tileMaxH is the tallest tile; its width is 16/9 of it.
10+ tileMaxH = 1440
11+ // tileCutMargin is the distance, in physical pixels, to a tile's inner
12+ // edge under which a box counts as cut by it.
13+ tileCutMargin = 2
14+)
15+
16+// region is a crop of the frame, in physical pixels.
17+type region struct{ x, y, w, h int }
18+
19+// tileRegions returns the crops to detect separately on a width x height
20+// frame, or nil for one whole-frame detection (a 16:9 frame, or one whose
21+// long side is at most wholeMaxSide). Otherwise the frame is covered by exact
22+// 16:9 tiles (w*9 == h*16; explicit crops always use square letterboxing) of
23+// height min(height, 1440) rounded down to a multiple of 9, overlapping by at
24+// least 10 % of the tile side on each axis; the last tile of a row or column
25+// is aligned to the frame edge. Tiles are listed row by row. A frame that
26+// would need more than maxRegions tiles is detected whole.
27+func tileRegions(width, height int) []region {
28+ if width <= 0 || height <= 0 || width*9 == height*16 || max(width, height) <= wholeMaxSide {
29+ return nil
30+ }
31+ th := min(height, tileMaxH) / 9 * 9
32+ tw := th / 9 * 16
33+ if tw > width { // a tall frame: the width bounds the tile
34+ tw = width / 16 * 16
35+ th = tw / 16 * 9
36+ }
37+ if tw == 0 || th == 0 {
38+ return nil
39+ }
40+ xs, ys := tileStarts(width, tw), tileStarts(height, th)
41+ if len(xs)*len(ys) > maxRegions {
42+ return nil
43+ }
44+ out := make([]region, 0, len(xs)*len(ys))
45+ for _, y := range ys {
46+ for _, x := range xs {
47+ out = append(out, region{x, y, tw, th})
48+ }
49+ }
50+ return out
51+}
52+
53+// tileStarts returns the start offsets of tiles of size t covering length l,
54+// evenly spread, the last one flush with the end, with consecutive tiles
55+// overlapping by at least a tenth of t.
56+func tileStarts(l, t int) []int {
57+ if l <= t {
58+ return []int{0}
59+ }
60+ d := l - t
61+ maxStep := t * 9 / 10
62+ n := (d+maxStep-1)/maxStep + 1
63+ out := make([]int, n)
64+ for i := range out {
65+ out[i] = (d*i + n - 2) / (n - 1) // ceil(d*i/(n-1)): steps never exceed maxStep
66+ }
67+ return out
68+}
69+
70+// dropCutBoxes drops the boxes that a tile's inner edge cut (an edge of the
71+// tile that is not an edge of the frame) when another tile saw the same
72+// element uncut: the cut box would otherwise survive as a smaller box nested
73+// in the whole one. A box is "the same element" when the other box covers
74+// nestRatio of it. Without regions the boxes are returned as they are.
75+func dropCutBoxes(dets []detection, regions []region) []detection {
76+ if len(regions) == 0 {
77+ return dets
78+ }
79+ fw, fh := 0, 0
80+ for _, r := range regions {
81+ fw, fh = max(fw, r.x+r.w), max(fh, r.y+r.h)
82+ }
83+ cut := func(d detection) bool {
84+ if d.region < 0 || d.region >= len(regions) {
85+ return false
86+ }
87+ r := regions[d.region]
88+ m := float32(tileCutMargin)
89+ return (r.x > 0 && d.x1 < float32(r.x)+m) || (r.x+r.w < fw && d.x2 > float32(r.x+r.w)-m) ||
90+ (r.y > 0 && d.y1 < float32(r.y)+m) || (r.y+r.h < fh && d.y2 > float32(r.y+r.h)-m)
91+ }
92+ isCut := make([]bool, len(dets))
93+ for i, d := range dets {
94+ isCut[i] = cut(d)
95+ }
96+ out := make([]detection, 0, len(dets))
97+ for i, d := range dets {
98+ drop := false
99+ if isCut[i] {
100+ for j, o := range dets {
101+ if j != i && !isCut[j] && o.region != d.region && covered(d, o) {
102+ drop = true
103+ break
104+ }
105+ }
106+ }
107+ if !drop {
108+ out = append(out, d)
109+ }
110+ }
111+ return out
112+}
113+
114+// covered reports whether o covers at least nestRatio of d's area.
115+func covered(d, o detection) bool {
116+ w := min(d.x2, o.x2) - max(d.x1, o.x1)
117+ h := min(d.y2, o.y2) - max(d.y1, o.y1)
118+ if w <= 0 || h <= 0 {
119+ return false
120+ }
121+ return float64(w*h) >= nestRatio*float64((d.x2-d.x1)*(d.y2-d.y1))
122+}
A internal/adapters/hints/tile_test.go +75 -0
@@ -0,0 +1,75 @@
1+package hints
2+
3+import (
4+ "testing"
5+
6+ "github.com/stretchr/testify/require"
7+)
8+
9+func TestTileRegions(t *testing.T) {
10+ whole := [][2]int{{3840, 2160}, {1920, 1080}, {2560, 1080}, {2560, 1440}, {1280, 720}}
11+ for _, s := range whole {
12+ require.Nil(t, tileRegions(s[0], s[1]), "%dx%d", s[0], s[1])
13+ }
14+ require.Equal(t, []region{{0, 0, 2560, 1440}, {1280, 0, 2560, 1440}, {2560, 0, 2560, 1440},
15+ {0, 720, 2560, 1440}, {1280, 720, 2560, 1440}, {2560, 720, 2560, 1440}}, tileRegions(5120, 2160))
16+ require.Equal(t, []region{{0, 0, 2560, 1440}, {880, 0, 2560, 1440}}, tileRegions(3440, 1440))
17+
18+ for _, s := range [][2]int{{5120, 2160}, {3440, 1440}, {5120, 1440}, {7680, 2160}, {3000, 2999}, {1000, 3000}, {5121, 2161}, {6000, 1001}} {
19+ w, h := s[0], s[1]
20+ rs := tileRegions(w, h)
21+ require.NotEmpty(t, rs, "%dx%d", w, h)
22+ require.LessOrEqual(t, len(rs), maxRegions)
23+ covered := make([]bool, w*h)
24+ for _, r := range rs {
25+ require.Equal(t, r.h*16, r.w*9, "%dx%d tile %+v is not 16:9", w, h, r)
26+ require.GreaterOrEqual(t, r.x, 0)
27+ require.GreaterOrEqual(t, r.y, 0)
28+ require.LessOrEqual(t, r.x+r.w, w)
29+ require.LessOrEqual(t, r.y+r.h, h)
30+ for y := r.y; y < r.y+r.h; y++ {
31+ for x := r.x; x < r.x+r.w; x++ {
32+ covered[y*w+x] = true
33+ }
34+ }
35+ }
36+ for i, c := range covered {
37+ if !c {
38+ t.Fatalf("%dx%d: pixel %d,%d not covered", w, h, i%w, i/w)
39+ }
40+ }
41+ }
42+}
43+
44+func TestTileStartsOverlap(t *testing.T) {
45+ for _, l := range []int{2561, 3440, 5120, 7680, 10000} {
46+ xs := tileStarts(l, 2560)
47+ require.Equal(t, 0, xs[0])
48+ require.Equal(t, l-2560, xs[len(xs)-1])
49+ for i := 1; i < len(xs); i++ {
50+ require.GreaterOrEqual(t, xs[i-1]+2560-xs[i], 256, "length %d", l) // >= 10 %
51+ }
52+ }
53+}
54+
55+func TestTileRegionsTooMany(t *testing.T) {
56+ require.Nil(t, tileRegions(60000, 20000))
57+}
58+
59+func TestDropCutBoxes(t *testing.T) {
60+ rs := tileRegions(5120, 2160) // tile 0: x 0..2560, tile 1: x 1280..3840
61+ // A button at x 2500..2700 is cut at 2560 by tile 0 and whole in tile 1.
62+ cutBox := detection{x1: 2500, y1: 100, x2: 2560, y2: 140, score: 0.6, region: 0}
63+ whole := detection{x1: 2500, y1: 100, x2: 2700, y2: 140, score: 0.8, region: 1}
64+ require.Equal(t, []detection{whole}, dropCutBoxes([]detection{cutBox, whole}, rs))
65+ // Nobody saw it whole: the cut box stays.
66+ require.Equal(t, []detection{cutBox}, dropCutBoxes([]detection{cutBox}, rs))
67+ // An edge of the frame is not an inner edge.
68+ edge := detection{x1: 0, y1: 0, x2: 60, y2: 30, score: 0.6, region: 0}
69+ // Region 3 starts at y 720: its box at the frame's left edge is not cut
70+ // either, so both stay though they sit in different regions.
71+ other := detection{x1: 0, y1: 800, x2: 60, y2: 830, score: 0.5, region: 3}
72+ require.Equal(t, []detection{edge, other}, dropCutBoxes([]detection{edge, other}, rs))
73+ // No regions: unchanged.
74+ require.Equal(t, []detection{cutBox}, dropCutBoxes([]detection{cutBox}, nil))
75+}
@@ -0,0 +1,75 @@
1+package hints
2+
3+import (
4+ "testing"
5+
6+ "github.com/stretchr/testify/require"
7+)
8+
9+func TestTileRegions(t *testing.T) {
10+ whole := [][2]int{{3840, 2160}, {1920, 1080}, {2560, 1080}, {2560, 1440}, {1280, 720}}
11+ for _, s := range whole {
12+ require.Nil(t, tileRegions(s[0], s[1]), "%dx%d", s[0], s[1])
13+ }
14+ require.Equal(t, []region{{0, 0, 2560, 1440}, {1280, 0, 2560, 1440}, {2560, 0, 2560, 1440},
15+ {0, 720, 2560, 1440}, {1280, 720, 2560, 1440}, {2560, 720, 2560, 1440}}, tileRegions(5120, 2160))
16+ require.Equal(t, []region{{0, 0, 2560, 1440}, {880, 0, 2560, 1440}}, tileRegions(3440, 1440))
17+
18+ for _, s := range [][2]int{{5120, 2160}, {3440, 1440}, {5120, 1440}, {7680, 2160}, {3000, 2999}, {1000, 3000}, {5121, 2161}, {6000, 1001}} {
19+ w, h := s[0], s[1]
20+ rs := tileRegions(w, h)
21+ require.NotEmpty(t, rs, "%dx%d", w, h)
22+ require.LessOrEqual(t, len(rs), maxRegions)
23+ covered := make([]bool, w*h)
24+ for _, r := range rs {
25+ require.Equal(t, r.h*16, r.w*9, "%dx%d tile %+v is not 16:9", w, h, r)
26+ require.GreaterOrEqual(t, r.x, 0)
27+ require.GreaterOrEqual(t, r.y, 0)
28+ require.LessOrEqual(t, r.x+r.w, w)
29+ require.LessOrEqual(t, r.y+r.h, h)
30+ for y := r.y; y < r.y+r.h; y++ {
31+ for x := r.x; x < r.x+r.w; x++ {
32+ covered[y*w+x] = true
33+ }
34+ }
35+ }
36+ for i, c := range covered {
37+ if !c {
38+ t.Fatalf("%dx%d: pixel %d,%d not covered", w, h, i%w, i/w)
39+ }
40+ }
41+ }
42+}
43+
44+func TestTileStartsOverlap(t *testing.T) {
45+ for _, l := range []int{2561, 3440, 5120, 7680, 10000} {
46+ xs := tileStarts(l, 2560)
47+ require.Equal(t, 0, xs[0])
48+ require.Equal(t, l-2560, xs[len(xs)-1])
49+ for i := 1; i < len(xs); i++ {
50+ require.GreaterOrEqual(t, xs[i-1]+2560-xs[i], 256, "length %d", l) // >= 10 %
51+ }
52+ }
53+}
54+
55+func TestTileRegionsTooMany(t *testing.T) {
56+ require.Nil(t, tileRegions(60000, 20000))
57+}
58+
59+func TestDropCutBoxes(t *testing.T) {
60+ rs := tileRegions(5120, 2160) // tile 0: x 0..2560, tile 1: x 1280..3840
61+ // A button at x 2500..2700 is cut at 2560 by tile 0 and whole in tile 1.
62+ cutBox := detection{x1: 2500, y1: 100, x2: 2560, y2: 140, score: 0.6, region: 0}
63+ whole := detection{x1: 2500, y1: 100, x2: 2700, y2: 140, score: 0.8, region: 1}
64+ require.Equal(t, []detection{whole}, dropCutBoxes([]detection{cutBox, whole}, rs))
65+ // Nobody saw it whole: the cut box stays.
66+ require.Equal(t, []detection{cutBox}, dropCutBoxes([]detection{cutBox}, rs))
67+ // An edge of the frame is not an inner edge.
68+ edge := detection{x1: 0, y1: 0, x2: 60, y2: 30, score: 0.6, region: 0}
69+ // Region 3 starts at y 720: its box at the frame's left edge is not cut
70+ // either, so both stay though they sit in different regions.
71+ other := detection{x1: 0, y1: 800, x2: 60, y2: 830, score: 0.5, region: 3}
72+ require.Equal(t, []detection{edge, other}, dropCutBoxes([]detection{edge, other}, rs))
73+ // No regions: unchanged.
74+ require.Equal(t, []detection{cutBox}, dropCutBoxes([]detection{cutBox}, nil))
75+}
D internal/adapters/hints/underline.go +0 -246
@@ -1,246 +0,0 @@
1-package hints
2-
3-import (
4- "image"
5- "math"
6- "sort"
7-
8- "github.com/bnema/neferwl/internal/ports"
9-)
10-
11-// underlineCandidate is a word with a thin line under it.
12-type underlineCandidate struct {
13- word int
14- row int // the line's row
15- from, to int // x extent of the continuous line through the word
16-}
17-
18-// markUnderlined sets Underlined, UnderL and UnderR on the words of a band
19-// that have a thin line under them. words are in the coordinates of g.
20-// Lines that run on far beyond the word (rules, table borders, input boxes)
21-// do not count, unless the extension is made of underlined words too.
22-func markUnderlined(src *image.Gray, words []OCRWord) {
23- // Words are relative to the band's own origin, whatever the image is a
24- // view of.
25- b := src.Bounds()
26- g := &image.Gray{Pix: src.Pix[src.PixOffset(b.Min.X, b.Min.Y):], Stride: src.Stride, Rect: image.Rect(0, 0, b.Dx(), b.Dy())}
27- var cands []underlineCandidate
28- for i, w := range words {
29- // A single glyph, or a box narrower than tall, has a flat bottom
30- // stroke (8, L, 2) that is not an underline.
31- if len([]rune(w.Text)) < 2 || w.Rect.Dx() < w.Rect.Dy() {
32- continue
33- }
34- if c, ok := underlineOf(g, w.Rect); ok {
35- c.word = i
36- cands = append(cands, c)
37- }
38- }
39- for _, c := range cands {
40- w := words[c.word]
41- h := w.Rect.Dy()
42- if (w.Rect.Min.X-c.from > h && !covered(words, cands, c, -1)) ||
43- (c.to-w.Rect.Max.X > h && !covered(words, cands, c, 1)) {
44- continue
45- }
46- words[c.word].Underlined, words[c.word].UnderL, words[c.word].UnderR = true, c.from, c.to
47- }
48-}
49-
50-// covered reports whether the line of c, on the given side of its word, is
51-// explained by underlined words on the same line up to within a glyph of its
52-// end.
53-func covered(words []OCRWord, cands []underlineCandidate, c underlineCandidate, dir int) bool {
54- w := words[c.word].Rect
55- h := w.Dy()
56- cur := w.Min.X
57- if dir > 0 {
58- cur = w.Max.X
59- }
60- for {
61- if (dir < 0 && cur-c.from <= h) || (dir > 0 && c.to-cur <= h) {
62- return true
63- }
64- next := -1
65- for _, o := range cands {
66- r := words[o.word].Rect
67- if abs(r.Max.Y-w.Max.Y) > max(3, h/3) {
68- continue
69- }
70- var gap, reach int
71- if dir < 0 {
72- gap, reach = cur-r.Max.X, r.Min.X
73- } else {
74- gap, reach = r.Min.X-cur, r.Max.X
75- }
76- if gap < -2 || gap > h {
77- continue
78- }
79- if next < 0 || (dir < 0 && reach < next) || (dir > 0 && reach > next) {
80- next = reach
81- }
82- }
83- if next < 0 || (dir < 0 && next >= cur) || (dir > 0 && next <= cur) {
84- return false
85- }
86- cur = next
87- }
88-}
89-
90-// underlineOf looks for an underline under the word box r in g.
91-func underlineOf(g *image.Gray, r image.Rectangle) (underlineCandidate, bool) {
92- b := g.Bounds()
93- box := r.Intersect(b)
94- h := r.Dy()
95- if box.Dx() < 4 || h < 4 {
96- return underlineCandidate{}, false
97- }
98- bg, ink := contrastOf(g, box)
99- thr := min(40, max(20, ink/2))
100- if ink < thr+10 {
101- return underlineCandidate{}, false
102- }
103- top := max(b.Min.Y, r.Max.Y-(h*15+99)/100)
104- bottom := min(b.Max.Y, r.Max.Y+(h*35+99)/100+1)
105- maxThick := max(2, h/6)
106- rowOK := func(y int) bool {
107- row := g.Pix[(y-g.Rect.Min.Y)*g.Stride:]
108- n := 0
109- for x := box.Min.X; x < box.Max.X; x++ {
110- if abs(int(row[x-g.Rect.Min.X])-bg) >= thr {
111- n++
112- }
113- }
114- return n*100 >= box.Dx()*85
115- }
116- thick, first := 0, 0
117- for y := top; y <= bottom; y++ {
118- if y < bottom && rowOK(y) {
119- if thick == 0 {
120- first = y
121- }
122- thick++
123- continue
124- }
125- if thick > 0 {
126- if thick > maxThick {
127- return underlineCandidate{}, false // a block, not a line
128- }
129- row := first + thick/2
130- from, to := box.Min.X, box.Max.X
131- line := g.Pix[(row-g.Rect.Min.Y)*g.Stride:]
132- on := func(x int) bool { return abs(int(line[x-g.Rect.Min.X])-bg) >= thr }
133- for from > b.Min.X && on(from-1) {
134- from--
135- }
136- for to < b.Max.X && on(to) {
137- to++
138- }
139- return underlineCandidate{row: row, from: from, to: to}, true
140- }
141- }
142- return underlineCandidate{}, false
143-}
144-
145-// contrastOf returns the background level of box (the most common level,
146-// found in bins of 16 so antialiasing does not split it, then refined to the
147-// mean of that bin) and the contrast of its ink (the deviation reached by
148-// the strongest 3% of pixels).
149-func contrastOf(g *image.Gray, box image.Rectangle) (bg, ink int) {
150- var hist [256]int
151- for y := box.Min.Y; y < box.Max.Y; y++ {
152- for _, v := range g.Pix[(y-g.Rect.Min.Y)*g.Stride+box.Min.X-g.Rect.Min.X:][:box.Dx()] {
153- hist[v]++
154- }
155- }
156- total := box.Dx() * box.Dy()
157- var bins [16]int
158- for v, c := range hist {
159- bins[v>>4] += c
160- }
161- best := 0
162- for i, c := range bins {
163- if c > bins[best] {
164- best = i
165- }
166- }
167- sum, n := 0, 0
168- for v := best << 4; v < (best+1)<<4; v++ {
169- sum += v * hist[v]
170- n += hist[v]
171- }
172- bg = sum / max(n, 1)
173- var dev [256]int
174- for v, c := range hist {
175- dev[abs(v-bg)] += c
176- }
177- n = 0
178- for d := 255; d >= 0; d-- {
179- n += dev[d]
180- if n*100 >= total*3 {
181- return bg, d
182- }
183- }
184- return bg, 0
185-}
186-
187-// underlinedLinks returns the targets for underlined words, joining the
188-// words of one continuous underline. A group that overlaps a URL target (in
189-// pixels) is left to it. Coordinates are scaled to logical.
190-func underlinedLinks(words []OCRWord, urls []image.Rectangle, scale float64) []ports.HintTarget {
191- var us []OCRWord
192- for _, w := range words {
193- if w.Underlined {
194- us = append(us, w)
195- }
196- }
197- sort.Slice(us, func(i, j int) bool { return us[i].Rect.Min.X < us[j].Rect.Min.X })
198- used := make([]bool, len(us))
199- var out []ports.HintTarget
200- for i := range us {
201- if used[i] {
202- continue
203- }
204- used[i] = true
205- rect, last := us[i].Rect, us[i]
206- chars, width := len(last.Text), last.Rect.Dx()
207- for {
208- next := -1
209- for j := range us {
210- c := us[j]
211- if used[j] || c.Rect.Min.X < rect.Max.X-2 || abs(c.Rect.Max.Y-last.Rect.Max.Y) > max(3, rect.Dy()/3) {
212- continue
213- }
214- avg := float64(width+c.Rect.Dx()) / float64(max(1, chars+len(c.Text)))
215- gap := c.Rect.Min.X - rect.Max.X
216- if float64(gap) > 1.2*avg || c.Rect.Min.X > last.UnderR+2 {
217- continue
218- }
219- if next < 0 || c.Rect.Min.X < us[next].Rect.Min.X {
220- next = j
221- }
222- }
223- if next < 0 {
224- break
225- }
226- used[next] = true
227- last = us[next]
228- rect = rect.Union(last.Rect)
229- chars += len(last.Text)
230- width += last.Rect.Dx()
231- }
232- dup := false
233- for _, u := range urls {
234- if overlap(rect, u) > .5 {
235- dup = true
236- }
237- }
238- if dup {
239- continue
240- }
241- x, y := int(math.Floor(float64(rect.Min.X)/scale)), int(math.Floor(float64(rect.Min.Y)/scale))
242- right, bottom := int(math.Ceil(float64(rect.Max.X)/scale)), int(math.Ceil(float64(rect.Max.Y)/scale))
243- out = append(out, ports.HintTarget{Kind: ports.HintLink, Rect: ports.Rect{X: x, Y: y, W: right - x, H: bottom - y}})
244- }
245- return out
246-}
@@ -1,246 +0,0 @@
1-package hints
2-
3-import (
4- "image"
5- "math"
6- "sort"
7-
8- "github.com/bnema/neferwl/internal/ports"
9-)
10-
11-// underlineCandidate is a word with a thin line under it.
12-type underlineCandidate struct {
13- word int
14- row int // the line's row
15- from, to int // x extent of the continuous line through the word
16-}
17-
18-// markUnderlined sets Underlined, UnderL and UnderR on the words of a band
19-// that have a thin line under them. words are in the coordinates of g.
20-// Lines that run on far beyond the word (rules, table borders, input boxes)
21-// do not count, unless the extension is made of underlined words too.
22-func markUnderlined(src *image.Gray, words []OCRWord) {
23- // Words are relative to the band's own origin, whatever the image is a
24- // view of.
25- b := src.Bounds()
26- g := &image.Gray{Pix: src.Pix[src.PixOffset(b.Min.X, b.Min.Y):], Stride: src.Stride, Rect: image.Rect(0, 0, b.Dx(), b.Dy())}
27- var cands []underlineCandidate
28- for i, w := range words {
29- // A single glyph, or a box narrower than tall, has a flat bottom
30- // stroke (8, L, 2) that is not an underline.
31- if len([]rune(w.Text)) < 2 || w.Rect.Dx() < w.Rect.Dy() {
32- continue
33- }
34- if c, ok := underlineOf(g, w.Rect); ok {
35- c.word = i
36- cands = append(cands, c)
37- }
38- }
39- for _, c := range cands {
40- w := words[c.word]
41- h := w.Rect.Dy()
42- if (w.Rect.Min.X-c.from > h && !covered(words, cands, c, -1)) ||
43- (c.to-w.Rect.Max.X > h && !covered(words, cands, c, 1)) {
44- continue
45- }
46- words[c.word].Underlined, words[c.word].UnderL, words[c.word].UnderR = true, c.from, c.to
47- }
48-}
49-
50-// covered reports whether the line of c, on the given side of its word, is
51-// explained by underlined words on the same line up to within a glyph of its
52-// end.
53-func covered(words []OCRWord, cands []underlineCandidate, c underlineCandidate, dir int) bool {
54- w := words[c.word].Rect
55- h := w.Dy()
56- cur := w.Min.X
57- if dir > 0 {
58- cur = w.Max.X
59- }
60- for {
61- if (dir < 0 && cur-c.from <= h) || (dir > 0 && c.to-cur <= h) {
62- return true
63- }
64- next := -1
65- for _, o := range cands {
66- r := words[o.word].Rect
67- if abs(r.Max.Y-w.Max.Y) > max(3, h/3) {
68- continue
69- }
70- var gap, reach int
71- if dir < 0 {
72- gap, reach = cur-r.Max.X, r.Min.X
73- } else {
74- gap, reach = r.Min.X-cur, r.Max.X
75- }
76- if gap < -2 || gap > h {
77- continue
78- }
79- if next < 0 || (dir < 0 && reach < next) || (dir > 0 && reach > next) {
80- next = reach
81- }
82- }
83- if next < 0 || (dir < 0 && next >= cur) || (dir > 0 && next <= cur) {
84- return false
85- }
86- cur = next
87- }
88-}
89-
90-// underlineOf looks for an underline under the word box r in g.
91-func underlineOf(g *image.Gray, r image.Rectangle) (underlineCandidate, bool) {
92- b := g.Bounds()
93- box := r.Intersect(b)
94- h := r.Dy()
95- if box.Dx() < 4 || h < 4 {
96- return underlineCandidate{}, false
97- }
98- bg, ink := contrastOf(g, box)
99- thr := min(40, max(20, ink/2))
100- if ink < thr+10 {
101- return underlineCandidate{}, false
102- }
103- top := max(b.Min.Y, r.Max.Y-(h*15+99)/100)
104- bottom := min(b.Max.Y, r.Max.Y+(h*35+99)/100+1)
105- maxThick := max(2, h/6)
106- rowOK := func(y int) bool {
107- row := g.Pix[(y-g.Rect.Min.Y)*g.Stride:]
108- n := 0
109- for x := box.Min.X; x < box.Max.X; x++ {
110- if abs(int(row[x-g.Rect.Min.X])-bg) >= thr {
111- n++
112- }
113- }
114- return n*100 >= box.Dx()*85
115- }
116- thick, first := 0, 0
117- for y := top; y <= bottom; y++ {
118- if y < bottom && rowOK(y) {
119- if thick == 0 {
120- first = y
121- }
122- thick++
123- continue
124- }
125- if thick > 0 {
126- if thick > maxThick {
127- return underlineCandidate{}, false // a block, not a line
128- }
129- row := first + thick/2
130- from, to := box.Min.X, box.Max.X
131- line := g.Pix[(row-g.Rect.Min.Y)*g.Stride:]
132- on := func(x int) bool { return abs(int(line[x-g.Rect.Min.X])-bg) >= thr }
133- for from > b.Min.X && on(from-1) {
134- from--
135- }
136- for to < b.Max.X && on(to) {
137- to++
138- }
139- return underlineCandidate{row: row, from: from, to: to}, true
140- }
141- }
142- return underlineCandidate{}, false
143-}
144-
145-// contrastOf returns the background level of box (the most common level,
146-// found in bins of 16 so antialiasing does not split it, then refined to the
147-// mean of that bin) and the contrast of its ink (the deviation reached by
148-// the strongest 3% of pixels).
149-func contrastOf(g *image.Gray, box image.Rectangle) (bg, ink int) {
150- var hist [256]int
151- for y := box.Min.Y; y < box.Max.Y; y++ {
152- for _, v := range g.Pix[(y-g.Rect.Min.Y)*g.Stride+box.Min.X-g.Rect.Min.X:][:box.Dx()] {
153- hist[v]++
154- }
155- }
156- total := box.Dx() * box.Dy()
157- var bins [16]int
158- for v, c := range hist {
159- bins[v>>4] += c
160- }
161- best := 0
162- for i, c := range bins {
163- if c > bins[best] {
164- best = i
165- }
166- }
167- sum, n := 0, 0
168- for v := best << 4; v < (best+1)<<4; v++ {
169- sum += v * hist[v]
170- n += hist[v]
171- }
172- bg = sum / max(n, 1)
173- var dev [256]int
174- for v, c := range hist {
175- dev[abs(v-bg)] += c
176- }
177- n = 0
178- for d := 255; d >= 0; d-- {
179- n += dev[d]
180- if n*100 >= total*3 {
181- return bg, d
182- }
183- }
184- return bg, 0
185-}
186-
187-// underlinedLinks returns the targets for underlined words, joining the
188-// words of one continuous underline. A group that overlaps a URL target (in
189-// pixels) is left to it. Coordinates are scaled to logical.
190-func underlinedLinks(words []OCRWord, urls []image.Rectangle, scale float64) []ports.HintTarget {
191- var us []OCRWord
192- for _, w := range words {
193- if w.Underlined {
194- us = append(us, w)
195- }
196- }
197- sort.Slice(us, func(i, j int) bool { return us[i].Rect.Min.X < us[j].Rect.Min.X })
198- used := make([]bool, len(us))
199- var out []ports.HintTarget
200- for i := range us {
201- if used[i] {
202- continue
203- }
204- used[i] = true
205- rect, last := us[i].Rect, us[i]
206- chars, width := len(last.Text), last.Rect.Dx()
207- for {
208- next := -1
209- for j := range us {
210- c := us[j]
211- if used[j] || c.Rect.Min.X < rect.Max.X-2 || abs(c.Rect.Max.Y-last.Rect.Max.Y) > max(3, rect.Dy()/3) {
212- continue
213- }
214- avg := float64(width+c.Rect.Dx()) / float64(max(1, chars+len(c.Text)))
215- gap := c.Rect.Min.X - rect.Max.X
216- if float64(gap) > 1.2*avg || c.Rect.Min.X > last.UnderR+2 {
217- continue
218- }
219- if next < 0 || c.Rect.Min.X < us[next].Rect.Min.X {
220- next = j
221- }
222- }
223- if next < 0 {
224- break
225- }
226- used[next] = true
227- last = us[next]
228- rect = rect.Union(last.Rect)
229- chars += len(last.Text)
230- width += last.Rect.Dx()
231- }
232- dup := false
233- for _, u := range urls {
234- if overlap(rect, u) > .5 {
235- dup = true
236- }
237- }
238- if dup {
239- continue
240- }
241- x, y := int(math.Floor(float64(rect.Min.X)/scale)), int(math.Floor(float64(rect.Min.Y)/scale))
242- right, bottom := int(math.Ceil(float64(rect.Max.X)/scale)), int(math.Ceil(float64(rect.Max.Y)/scale))
243- out = append(out, ports.HintTarget{Kind: ports.HintLink, Rect: ports.Rect{X: x, Y: y, W: right - x, H: bottom - y}})
244- }
245- return out
246-}
D internal/adapters/hints/underline_test.go +0 -252
@@ -1,252 +0,0 @@
1-package hints
2-
3-import (
4- "image"
5- "image/color"
6- "os"
7- "os/exec"
8- "testing"
9-
10- "github.com/stretchr/testify/require"
11-
12- "github.com/bnema/neferwl/internal/ports"
13-)
14-
15-// canvas is a synthetic grayscale band: bg everywhere, ink for text.
16-type canvas struct {
17- g *image.Gray
18- bg, ink uint8
19- s int // scale
20-}
21-
22-func newCanvas(w, h int, bg, ink uint8, s int) *canvas {
23- g := image.NewGray(image.Rect(0, 0, w*s, h*s))
24- for i := range g.Pix {
25- g.Pix[i] = bg
26- }
27- return &canvas{g, bg, ink, s}
28-}
29-
30-func (c *canvas) fill(x0, y0, x1, y1 int, v uint8) {
31- for y := y0 * c.s; y < y1*c.s; y++ {
32- for x := x0 * c.s; x < x1*c.s; x++ {
33- c.g.SetGray(x, y, color.Gray{v})
34- }
35- }
36-}
37-
38-// word draws a 10 px high word of x-height body at (x,y) with the given
39-// width: text-like noisy ink (a stem every 3 px) and optional descenders.
40-// It returns the OCR box (descenders excluded, as a box around the body).
41-func (c *canvas) word(x, y, w int, descender bool) OCRWord {
42- for i := 0; i < w; i += 3 {
43- c.fill(x+i, y, x+i+1, y+10, c.ink)
44- }
45- box := image.Rect(x*c.s, y*c.s, (x+w)*c.s, (y+10)*c.s)
46- if descender {
47- for i := 0; i < w; i += 6 {
48- c.fill(x+i, y+10, x+i+1, y+13, c.ink)
49- }
50- box.Max.Y = (y + 13) * c.s
51- }
52- return OCRWord{Text: "word", Rect: box}
53-}
54-
55-func (c *canvas) underline(x0, x1, y int) { c.fill(x0, y, x1, y+1, c.ink) }
56-
57-func scanUnder(c *canvas, words ...OCRWord) []ports.HintTarget {
58- markUnderlined(c.g, words)
59- return textLinks(words, float64(c.s))
60-}
61-
62-func TestUnderline(t *testing.T) {
63- for _, tc := range []struct {
64- name string
65- bg, ink uint8
66- }{{"dark-bg", 10, 204}, {"light-bg", 245, 30}} {
67- for _, s := range []int{1, 2} {
68- t.Run(tc.name, func(t *testing.T) {
69- mk := func() *canvas { return newCanvas(200, 60, tc.bg, tc.ink, s) }
70-
71- c := mk()
72- w := c.word(10, 20, 40, false)
73- c.underline(10, 50, 32)
74- got := scanUnder(c, w)
75- require.Len(t, got, 1)
76- require.Equal(t, ports.HintLink, got[0].Kind)
77- require.False(t, got[0].Text)
78- require.Equal(t, ports.WindowID(0), got[0].Window)
79- require.Equal(t, 10, got[0].Rect.X)
80-
81- c = mk()
82- require.Empty(t, scanUnder(c, c.word(10, 20, 40, false)), "plain word")
83-
84- c = mk()
85- require.Empty(t, scanUnder(c, c.word(10, 20, 40, true)), "descenders")
86-
87- c = mk()
88- w = c.word(10, 20, 40, false)
89- c.underline(0, 200, 32)
90- require.Empty(t, scanUnder(c, w), "rule across")
91-
92- c = mk()
93- w = c.word(10, 20, 40, false)
94- c.fill(10, 24, 50, 25, c.ink)
95- require.Empty(t, scanUnder(c, w), "strikethrough")
96-
97- // Two words with a continuous underline: one target.
98- c = mk()
99- a, b := c.word(10, 20, 40, false), c.word(56, 20, 40, false)
100- c.underline(10, 96, 32)
101- got = scanUnder(c, a, b)
102- require.Len(t, got, 1)
103- require.Equal(t, 10, got[0].Rect.X)
104- require.Equal(t, 86, got[0].Rect.W)
105-
106- // A gap in the underline: two targets.
107- c = mk()
108- a, b = c.word(10, 20, 40, false), c.word(56, 20, 40, false)
109- c.underline(10, 50, 32)
110- c.underline(56, 96, 32)
111- require.Len(t, scanUnder(c, a, b), 2)
112- })
113- }
114- }
115-}
116-
117-// A box border under a word, with the word a part of a table row, is not a
118-// link; neither is an underline that continues beyond into unrelated text.
119-func TestUnderlineRejectsBox(t *testing.T) {
120- c := newCanvas(300, 60, 10, 204, 1)
121- w := c.word(100, 20, 40, false)
122- c.underline(20, 280, 32)
123- c.fill(20, 10, 21, 33, 204)
124- c.fill(279, 10, 280, 33, 204)
125- require.Empty(t, scanUnder(c, w))
126-}
127-
128-// The line must be thin.
129-func TestUnderlineRejectsBlock(t *testing.T) {
130- c := newCanvas(100, 60, 10, 204, 1)
131- w := c.word(10, 20, 40, false)
132- c.fill(10, 31, 50, 36, 204)
133- require.Empty(t, scanUnder(c, w))
134-}
135-
136-func TestUnderlinedURLIsOneTextTarget(t *testing.T) {
137- c := newCanvas(300, 60, 10, 204, 1)
138- w := c.word(10, 20, 100, false)
139- w.Text = "example.org"
140- c.underline(10, 110, 32)
141- got := scanUnder(c, w)
142- require.Len(t, got, 1)
143- require.True(t, got[0].Text)
144-}
145-
146-// ocrTestdata reads the words of a testdata image with the real tesseract.
147-func ocrTestdata(t *testing.T, name string) (*image.Gray, []OCRWord) {
148- t.Helper()
149- if _, err := exec.LookPath("tesseract"); err != nil {
150- t.Skip("tesseract not on PATH")
151- }
152- f, err := os.Open("testdata/" + name)
153- require.NoError(t, err)
154- defer f.Close()
155- img, _, err := image.Decode(f)
156- require.NoError(t, err)
157- g := image.NewGray(img.Bounds())
158- for y := 0; y < g.Rect.Dy(); y++ {
159- for x := 0; x < g.Rect.Dx(); x++ {
160- g.Set(x, y, img.At(x, y))
161- }
162- }
163- words, _, err := readWords(t.Context(), g)
164- require.NoError(t, err)
165- return g, words
166-}
167-
168-// headingRule finds the thin horizontal rule under the heading of
169-// browser.png: the first row, from y=150, with a 900 px run of non-white
170-// pixels. It returns the rule as a rect.
171-func headingRule(t *testing.T, g *image.Gray) image.Rectangle {
172- t.Helper()
173- for y := 150; y < 220; y++ {
174- run, start := 0, 0
175- for x := 0; x < g.Rect.Dx(); x++ {
176- if g.GrayAt(x, y).Y < 240 {
177- if run == 0 {
178- start = x
179- }
180- run++
181- if run >= 900 {
182- return image.Rect(start, y, start+run, y+1)
183- }
184- } else {
185- run = 0
186- }
187- }
188- }
189- t.Fatal("heading rule not found")
190- return image.Rectangle{}
191-}
192-
193-// browser.png has blue links and no underlined link: the only underline-only
194-// target is a known false positive, and the heading rule is not taken for an
195-// underline.
196-func TestUnderlineOnBrowserTestdata(t *testing.T) {
197- g, words := ocrTestdata(t, "browser.png")
198- rule := headingRule(t, g)
199- // Known false positive: the word "as" (flat glyph bottoms, ~(878,353)) is
200- // flagged underlined by the detector. Anything else is a regression.
201- var under []ports.HintTarget
202- for _, tg := range textLinks(words, 1) {
203- if !tg.Text {
204- under = append(under, tg)
205- }
206- }
207- require.Len(t, under, 1, "underline-only targets: %v", under)
208- require.True(t, under[0].Rect.X > 860 && under[0].Rect.X < 900 && under[0].Rect.Y > 340 && under[0].Rect.Y < 370, "unexpected target %v", under[0].Rect)
209- for _, tg := range textLinks(words, 1) {
210- r := image.Rect(tg.Rect.X, tg.Rect.Y, tg.Rect.X+tg.Rect.W, tg.Rect.Y+tg.Rect.H)
211- require.False(t, r.Overlaps(rule), "target %v overlaps the heading rule %v", tg.Rect, rule)
212- }
213-}
214-
215-// underline.png is a real render (Chromium) of a generic page: underlined
216-// links `demo`, `source` and `ermite-runner hosts`, a table with row rules
217-// and an input box, which must yield nothing.
218-func TestUnderlineOnRealRender(t *testing.T) {
219- _, words := ocrTestdata(t, "underline.png")
220- near := func(r ports.Rect, x0, y0, x1, y1 int) bool {
221- const tol = 8
222- abs := func(v int) int { return max(v, -v) }
223- return abs(r.X-x0) <= tol && abs(r.Y-y0) <= tol && abs(r.X+r.W-x1) <= tol && abs(r.Y+r.H-y1) <= tol
224- }
225- var got []ports.HintTarget
226- for _, tg := range textLinks(words, 1) {
227- if !tg.Text {
228- got = append(got, tg)
229- }
230- }
231- require.Len(t, got, 3, "targets: %v", got)
232- for _, want := range [][4]int{{148, 122, 196, 137}, {306, 126, 364, 137}, {122, 165, 294, 180}} {
233- found := false
234- for _, tg := range got {
235- found = found || near(tg.Rect, want[0], want[1], want[2], want[3])
236- }
237- require.True(t, found, "no target near %v in %v", want, got)
238- }
239- // The table (y 215..335) and the input box (y 360..410) give nothing.
240- for _, tg := range textLinks(words, 1) {
241- require.False(t, tg.Rect.Y > 200 && tg.Rect.Y < 420, "unexpected target %v", tg.Rect)
242- }
243-}
244-
245-// A single glyph with a flat bottom stroke (8, L) is not underlined text.
246-func TestUnderlineIgnoresSingleGlyph(t *testing.T) {
247- c := newCanvas(100, 60, 10, 204, 1)
248- w := c.word(10, 20, 10, false)
249- w.Text = "8"
250- c.underline(10, 20, 32)
251- require.Empty(t, scanUnder(c, w))
252-}
@@ -1,252 +0,0 @@
1-package hints
2-
3-import (
4- "image"
5- "image/color"
6- "os"
7- "os/exec"
8- "testing"
9-
10- "github.com/stretchr/testify/require"
11-
12- "github.com/bnema/neferwl/internal/ports"
13-)
14-
15-// canvas is a synthetic grayscale band: bg everywhere, ink for text.
16-type canvas struct {
17- g *image.Gray
18- bg, ink uint8
19- s int // scale
20-}
21-
22-func newCanvas(w, h int, bg, ink uint8, s int) *canvas {
23- g := image.NewGray(image.Rect(0, 0, w*s, h*s))
24- for i := range g.Pix {
25- g.Pix[i] = bg
26- }
27- return &canvas{g, bg, ink, s}
28-}
29-
30-func (c *canvas) fill(x0, y0, x1, y1 int, v uint8) {
31- for y := y0 * c.s; y < y1*c.s; y++ {
32- for x := x0 * c.s; x < x1*c.s; x++ {
33- c.g.SetGray(x, y, color.Gray{v})
34- }
35- }
36-}
37-
38-// word draws a 10 px high word of x-height body at (x,y) with the given
39-// width: text-like noisy ink (a stem every 3 px) and optional descenders.
40-// It returns the OCR box (descenders excluded, as a box around the body).
41-func (c *canvas) word(x, y, w int, descender bool) OCRWord {
42- for i := 0; i < w; i += 3 {
43- c.fill(x+i, y, x+i+1, y+10, c.ink)
44- }
45- box := image.Rect(x*c.s, y*c.s, (x+w)*c.s, (y+10)*c.s)
46- if descender {
47- for i := 0; i < w; i += 6 {
48- c.fill(x+i, y+10, x+i+1, y+13, c.ink)
49- }
50- box.Max.Y = (y + 13) * c.s
51- }
52- return OCRWord{Text: "word", Rect: box}
53-}
54-
55-func (c *canvas) underline(x0, x1, y int) { c.fill(x0, y, x1, y+1, c.ink) }
56-
57-func scanUnder(c *canvas, words ...OCRWord) []ports.HintTarget {
58- markUnderlined(c.g, words)
59- return textLinks(words, float64(c.s))
60-}
61-
62-func TestUnderline(t *testing.T) {
63- for _, tc := range []struct {
64- name string
65- bg, ink uint8
66- }{{"dark-bg", 10, 204}, {"light-bg", 245, 30}} {
67- for _, s := range []int{1, 2} {
68- t.Run(tc.name, func(t *testing.T) {
69- mk := func() *canvas { return newCanvas(200, 60, tc.bg, tc.ink, s) }
70-
71- c := mk()
72- w := c.word(10, 20, 40, false)
73- c.underline(10, 50, 32)
74- got := scanUnder(c, w)
75- require.Len(t, got, 1)
76- require.Equal(t, ports.HintLink, got[0].Kind)
77- require.False(t, got[0].Text)
78- require.Equal(t, ports.WindowID(0), got[0].Window)
79- require.Equal(t, 10, got[0].Rect.X)
80-
81- c = mk()
82- require.Empty(t, scanUnder(c, c.word(10, 20, 40, false)), "plain word")
83-
84- c = mk()
85- require.Empty(t, scanUnder(c, c.word(10, 20, 40, true)), "descenders")
86-
87- c = mk()
88- w = c.word(10, 20, 40, false)
89- c.underline(0, 200, 32)
90- require.Empty(t, scanUnder(c, w), "rule across")
91-
92- c = mk()
93- w = c.word(10, 20, 40, false)
94- c.fill(10, 24, 50, 25, c.ink)
95- require.Empty(t, scanUnder(c, w), "strikethrough")
96-
97- // Two words with a continuous underline: one target.
98- c = mk()
99- a, b := c.word(10, 20, 40, false), c.word(56, 20, 40, false)
100- c.underline(10, 96, 32)
101- got = scanUnder(c, a, b)
102- require.Len(t, got, 1)
103- require.Equal(t, 10, got[0].Rect.X)
104- require.Equal(t, 86, got[0].Rect.W)
105-
106- // A gap in the underline: two targets.
107- c = mk()
108- a, b = c.word(10, 20, 40, false), c.word(56, 20, 40, false)
109- c.underline(10, 50, 32)
110- c.underline(56, 96, 32)
111- require.Len(t, scanUnder(c, a, b), 2)
112- })
113- }
114- }
115-}
116-
117-// A box border under a word, with the word a part of a table row, is not a
118-// link; neither is an underline that continues beyond into unrelated text.
119-func TestUnderlineRejectsBox(t *testing.T) {
120- c := newCanvas(300, 60, 10, 204, 1)
121- w := c.word(100, 20, 40, false)
122- c.underline(20, 280, 32)
123- c.fill(20, 10, 21, 33, 204)
124- c.fill(279, 10, 280, 33, 204)
125- require.Empty(t, scanUnder(c, w))
126-}
127-
128-// The line must be thin.
129-func TestUnderlineRejectsBlock(t *testing.T) {
130- c := newCanvas(100, 60, 10, 204, 1)
131- w := c.word(10, 20, 40, false)
132- c.fill(10, 31, 50, 36, 204)
133- require.Empty(t, scanUnder(c, w))
134-}
135-
136-func TestUnderlinedURLIsOneTextTarget(t *testing.T) {
137- c := newCanvas(300, 60, 10, 204, 1)
138- w := c.word(10, 20, 100, false)
139- w.Text = "example.org"
140- c.underline(10, 110, 32)
141- got := scanUnder(c, w)
142- require.Len(t, got, 1)
143- require.True(t, got[0].Text)
144-}
145-
146-// ocrTestdata reads the words of a testdata image with the real tesseract.
147-func ocrTestdata(t *testing.T, name string) (*image.Gray, []OCRWord) {
148- t.Helper()
149- if _, err := exec.LookPath("tesseract"); err != nil {
150- t.Skip("tesseract not on PATH")
151- }
152- f, err := os.Open("testdata/" + name)
153- require.NoError(t, err)
154- defer f.Close()
155- img, _, err := image.Decode(f)
156- require.NoError(t, err)
157- g := image.NewGray(img.Bounds())
158- for y := 0; y < g.Rect.Dy(); y++ {
159- for x := 0; x < g.Rect.Dx(); x++ {
160- g.Set(x, y, img.At(x, y))
161- }
162- }
163- words, _, err := readWords(t.Context(), g)
164- require.NoError(t, err)
165- return g, words
166-}
167-
168-// headingRule finds the thin horizontal rule under the heading of
169-// browser.png: the first row, from y=150, with a 900 px run of non-white
170-// pixels. It returns the rule as a rect.
171-func headingRule(t *testing.T, g *image.Gray) image.Rectangle {
172- t.Helper()
173- for y := 150; y < 220; y++ {
174- run, start := 0, 0
175- for x := 0; x < g.Rect.Dx(); x++ {
176- if g.GrayAt(x, y).Y < 240 {
177- if run == 0 {
178- start = x
179- }
180- run++
181- if run >= 900 {
182- return image.Rect(start, y, start+run, y+1)
183- }
184- } else {
185- run = 0
186- }
187- }
188- }
189- t.Fatal("heading rule not found")
190- return image.Rectangle{}
191-}
192-
193-// browser.png has blue links and no underlined link: the only underline-only
194-// target is a known false positive, and the heading rule is not taken for an
195-// underline.
196-func TestUnderlineOnBrowserTestdata(t *testing.T) {
197- g, words := ocrTestdata(t, "browser.png")
198- rule := headingRule(t, g)
199- // Known false positive: the word "as" (flat glyph bottoms, ~(878,353)) is
200- // flagged underlined by the detector. Anything else is a regression.
201- var under []ports.HintTarget
202- for _, tg := range textLinks(words, 1) {
203- if !tg.Text {
204- under = append(under, tg)
205- }
206- }
207- require.Len(t, under, 1, "underline-only targets: %v", under)
208- require.True(t, under[0].Rect.X > 860 && under[0].Rect.X < 900 && under[0].Rect.Y > 340 && under[0].Rect.Y < 370, "unexpected target %v", under[0].Rect)
209- for _, tg := range textLinks(words, 1) {
210- r := image.Rect(tg.Rect.X, tg.Rect.Y, tg.Rect.X+tg.Rect.W, tg.Rect.Y+tg.Rect.H)
211- require.False(t, r.Overlaps(rule), "target %v overlaps the heading rule %v", tg.Rect, rule)
212- }
213-}
214-
215-// underline.png is a real render (Chromium) of a generic page: underlined
216-// links `demo`, `source` and `ermite-runner hosts`, a table with row rules
217-// and an input box, which must yield nothing.
218-func TestUnderlineOnRealRender(t *testing.T) {
219- _, words := ocrTestdata(t, "underline.png")
220- near := func(r ports.Rect, x0, y0, x1, y1 int) bool {
221- const tol = 8
222- abs := func(v int) int { return max(v, -v) }
223- return abs(r.X-x0) <= tol && abs(r.Y-y0) <= tol && abs(r.X+r.W-x1) <= tol && abs(r.Y+r.H-y1) <= tol
224- }
225- var got []ports.HintTarget
226- for _, tg := range textLinks(words, 1) {
227- if !tg.Text {
228- got = append(got, tg)
229- }
230- }
231- require.Len(t, got, 3, "targets: %v", got)
232- for _, want := range [][4]int{{148, 122, 196, 137}, {306, 126, 364, 137}, {122, 165, 294, 180}} {
233- found := false
234- for _, tg := range got {
235- found = found || near(tg.Rect, want[0], want[1], want[2], want[3])
236- }
237- require.True(t, found, "no target near %v in %v", want, got)
238- }
239- // The table (y 215..335) and the input box (y 360..410) give nothing.
240- for _, tg := range textLinks(words, 1) {
241- require.False(t, tg.Rect.Y > 200 && tg.Rect.Y < 420, "unexpected target %v", tg.Rect)
242- }
243-}
244-
245-// A single glyph with a flat bottom stroke (8, L) is not underlined text.
246-func TestUnderlineIgnoresSingleGlyph(t *testing.T) {
247- c := newCanvas(100, 60, 10, 204, 1)
248- w := c.word(10, 20, 10, false)
249- w.Text = "8"
250- c.underline(10, 20, 32)
251- require.Empty(t, scanUnder(c, w))
252-}
M internal/adapters/logging/logging.go +1 -1
@@ -22,7 +22,7 @@ import (
2222 var components = map[string]bool{
2323 "core": true, "wayland": true, "input": true, "drm": true, "seat": true,
2424 "render": true, "sync": true, "config": true, "app": true,
25- "screensaver": true, "powerkey": true, "sched": true, "screencast": true,
25+ "screensaver": true, "powerkey": true, "sched": true, "screencast": true, "hints": true,
2626 }
2727
2828 // categories are debug switches narrower than a component, only on when
@@ -22,7 +22,7 @@ import (
22 var components = map[string]bool{22 var components = map[string]bool{
23 "core": true, "wayland": true, "input": true, "drm": true, "seat": true,23 "core": true, "wayland": true, "input": true, "drm": true, "seat": true,
24 "render": true, "sync": true, "config": true, "app": true,24 "render": true, "sync": true, "config": true, "app": true,
25- "screensaver": true, "powerkey": true, "sched": true, "screencast": true,25+ "screensaver": true, "powerkey": true, "sched": true, "screencast": true, "hints": true,
26 }26 }
27 27
28 // categories are debug switches narrower than a component, only on when28 // categories are debug switches narrower than a component, only on when
M internal/app/app.go +6 -6
@@ -84,7 +84,7 @@ const busRetry = 2 * time.Second
8484 func relayConfig(ctx context.Context, cur ports.Config, in <-chan ports.ConfigChanged, out chan<- ports.ConfigChanged, keymaps chan *xkb.Keymap, deviceConfigs chan ports.InputDevicesConfig, curs *cursors, commands chan<- ports.ClientCommand, log zerowrap.Logger) {
8585 kb := cur.Keyboard
8686 devCfg := cur.InputDevicesConfig
87- hintOn := cur.Hint.Enable
87+ hint := cur.Hint
8888 for {
8989 var ev ports.ConfigChanged
9090 select {
@@ -92,11 +92,11 @@ func relayConfig(ctx context.Context, cur ports.Config, in <-chan ports.ConfigCh
9292 return
9393 case ev = <-in:
9494 }
95- // hint.enable is read at startup: nothing of hint mode starts or
96- // stops on a reload.
97- if ev.Config.Hint.Enable != hintOn {
98- log.Info().Bool("running", hintOn).Msg("hint.enable changes apply after restart")
99- ev.Config.Hint.Enable = hintOn
95+ // hint.enable, hint.detector, hint.model and hint.threshold are read
96+ // at startup: nothing of hint mode starts or stops on a reload.
97+ if h := ev.Config.Hint; h.Enable != hint.Enable || h.Detector != hint.Detector || h.Model != hint.Model || h.Threshold != hint.Threshold {
98+ log.Info().Bool("running", hint.Enable).Msg("hint.enable, hint.detector, hint.model and hint.threshold apply after restart")
99+ ev.Config.Hint.Enable, ev.Config.Hint.Detector, ev.Config.Hint.Model, ev.Config.Hint.Threshold = hint.Enable, hint.Detector, hint.Model, hint.Threshold
100100 }
101101 next := ev.Config.Keyboard
102102 layoutChanged := next.Layout != kb.Layout || next.Variant != kb.Variant || next.Options != kb.Options
@@ -84,7 +84,7 @@ const busRetry = 2 * time.Second
84 func relayConfig(ctx context.Context, cur ports.Config, in <-chan ports.ConfigChanged, out chan<- ports.ConfigChanged, keymaps chan *xkb.Keymap, deviceConfigs chan ports.InputDevicesConfig, curs *cursors, commands chan<- ports.ClientCommand, log zerowrap.Logger) {84 func relayConfig(ctx context.Context, cur ports.Config, in <-chan ports.ConfigChanged, out chan<- ports.ConfigChanged, keymaps chan *xkb.Keymap, deviceConfigs chan ports.InputDevicesConfig, curs *cursors, commands chan<- ports.ClientCommand, log zerowrap.Logger) {
85 kb := cur.Keyboard85 kb := cur.Keyboard
86 devCfg := cur.InputDevicesConfig86 devCfg := cur.InputDevicesConfig
87- hintOn := cur.Hint.Enable87+ hint := cur.Hint
88 for {88 for {
89 var ev ports.ConfigChanged89 var ev ports.ConfigChanged
90 select {90 select {
@@ -92,11 +92,11 @@ func relayConfig(ctx context.Context, cur ports.Config, in <-chan ports.ConfigCh
92 return92 return
93 case ev = <-in:93 case ev = <-in:
94 }94 }
95- // hint.enable is read at startup: nothing of hint mode starts or95+ // hint.enable, hint.detector, hint.model and hint.threshold are read
96- // stops on a reload.96+ // at startup: nothing of hint mode starts or stops on a reload.
97- if ev.Config.Hint.Enable != hintOn {97+ if h := ev.Config.Hint; h.Enable != hint.Enable || h.Detector != hint.Detector || h.Model != hint.Model || h.Threshold != hint.Threshold {
98- log.Info().Bool("running", hintOn).Msg("hint.enable changes apply after restart")98+ log.Info().Bool("running", hint.Enable).Msg("hint.enable, hint.detector, hint.model and hint.threshold apply after restart")
99- ev.Config.Hint.Enable = hintOn99+ ev.Config.Hint.Enable, ev.Config.Hint.Detector, ev.Config.Hint.Model, ev.Config.Hint.Threshold = hint.Enable, hint.Detector, hint.Model, hint.Threshold
100 }100 }
101 next := ev.Config.Keyboard101 next := ev.Config.Keyboard
102 layoutChanged := next.Layout != kb.Layout || next.Variant != kb.Variant || next.Options != kb.Options102 layoutChanged := next.Layout != kb.Layout || next.Variant != kb.Variant || next.Options != kb.Options
R internal/app/hintdebug.go → internal/app/hint.go +7 -10 renamed, 66% similar
@@ -5,21 +5,18 @@ import (
55 "os"
66 "path/filepath"
77
8- "github.com/bnema/neferwl/internal/adapters/atspi"
98 "github.com/bnema/neferwl/internal/adapters/hints"
109 "github.com/bnema/neferwl/internal/adapters/logging"
10+ "github.com/bnema/neferwl/internal/ports"
1111 )
1212
13-// hintOptions builds the hint worker options: the accessibility source, and,
14-// when NEFERWL_HINT_DEBUG is "1", the dump directory of scanned frames,
15-// $XDG_STATE_HOME/neferwl/hints. The accessibility bus is only contacted
16-// by the first scan; without it the worker reads windows with OCR only.
17-func hintOptions(ctx context.Context) hints.Options {
13+// hintOptions builds the hint worker options from the configuration (read at
14+// startup) and, when NEFERWL_HINT_DEBUG is "1", the dump directory of scanned
15+// frames, $XDG_STATE_HOME/neferwl/hints. The detector process only starts at
16+// the first scan.
17+func hintOptions(ctx context.Context, cfg ports.Config) hints.Options {
1818 opts := hintDump(ctx)
19- // The source connects on its first scan, and again after a failure. The
20- // hints worker owns it and closes it when it returns.
21- src := atspi.New()
22- opts.Source = src
19+ opts.Command, opts.ModelDir, opts.Threshold = cfg.Hint.Detector, cfg.Hint.Model, cfg.Hint.Threshold
2320 return opts
2421 }
2522
@@ -5,21 +5,18 @@ import (
5 "os"5 "os"
6 "path/filepath"6 "path/filepath"
7 7
8- "github.com/bnema/neferwl/internal/adapters/atspi"
9 "github.com/bnema/neferwl/internal/adapters/hints"8 "github.com/bnema/neferwl/internal/adapters/hints"
10 "github.com/bnema/neferwl/internal/adapters/logging"9 "github.com/bnema/neferwl/internal/adapters/logging"
10+ "github.com/bnema/neferwl/internal/ports"
11 )11 )
12 12
13-// hintOptions builds the hint worker options: the accessibility source, and,13+// hintOptions builds the hint worker options from the configuration (read at
14-// when NEFERWL_HINT_DEBUG is "1", the dump directory of scanned frames,14+// startup) and, when NEFERWL_HINT_DEBUG is "1", the dump directory of scanned
15-// $XDG_STATE_HOME/neferwl/hints. The accessibility bus is only contacted15+// frames, $XDG_STATE_HOME/neferwl/hints. The detector process only starts at
16-// by the first scan; without it the worker reads windows with OCR only.16+// the first scan.
17-func hintOptions(ctx context.Context) hints.Options {17+func hintOptions(ctx context.Context, cfg ports.Config) hints.Options {
18 opts := hintDump(ctx)18 opts := hintDump(ctx)
19- // The source connects on its first scan, and again after a failure. The19+ opts.Command, opts.ModelDir, opts.Threshold = cfg.Hint.Detector, cfg.Hint.Model, cfg.Hint.Threshold
20- // hints worker owns it and closes it when it returns.
21- src := atspi.New()
22- opts.Source = src
23 return opts20 return opts
24 }21 }
25 22
M internal/app/hintoptin_test.go +18 -2
@@ -45,7 +45,6 @@ func TestHintOffStartsNoHintGoroutine(t *testing.T) {
4545 buf := make([]byte, 1<<20)
4646 dump := string(buf[:runtime.Stack(buf, true)])
4747 require.NotContains(t, dump, "adapters/hints.")
48- require.NotContains(t, dump, "adapters/atspi.")
4948 require.NotContains(t, dump, "routeCaptured")
5049 select {
5150 case <-done:
@@ -64,5 +63,22 @@ func TestRelayConfigKeepsStartupHintEnable(t *testing.T) {
6463 next := config.Defaults()
6564 next.Hint.Enable = true
6665 in <- ports.ConfigChanged{Config: next}
67- require.False(t, (<-out).Config.Hint.Enable)
66+ got := (<-out).Config.Hint
67+ require.False(t, got.Enable)
68+}
69+
70+func TestRelayConfigKeepsStartupHintDetector(t *testing.T) {
71+ ctx, cancel := context.WithCancel(context.Background())
72+ defer cancel()
73+ cur := config.Defaults()
74+ in := make(chan ports.ConfigChanged, 1)
75+ out := make(chan ports.ConfigChanged, 1)
76+ go relayConfig(ctx, cur, in, out, make(chan *xkb.Keymap, 1), make(chan ports.InputDevicesConfig, 1), newCursors(clock.System{}, 0), make(chan ports.ClientCommand, 1), logging.For(ctx, "config"))
77+ next := config.Defaults()
78+ next.Hint.Detector, next.Hint.Model, next.Hint.Threshold = "/other/nh", "/other/models", 0.5
79+ in <- ports.ConfigChanged{Config: next}
80+ got := (<-out).Config.Hint
81+ require.Equal(t, cur.Hint.Detector, got.Detector)
82+ require.Equal(t, cur.Hint.Model, got.Model)
83+ require.Equal(t, cur.Hint.Threshold, got.Threshold)
6884 }
@@ -45,7 +45,6 @@ func TestHintOffStartsNoHintGoroutine(t *testing.T) {
45 buf := make([]byte, 1<<20)45 buf := make([]byte, 1<<20)
46 dump := string(buf[:runtime.Stack(buf, true)])46 dump := string(buf[:runtime.Stack(buf, true)])
47 require.NotContains(t, dump, "adapters/hints.")47 require.NotContains(t, dump, "adapters/hints.")
48- require.NotContains(t, dump, "adapters/atspi.")
49 require.NotContains(t, dump, "routeCaptured")48 require.NotContains(t, dump, "routeCaptured")
50 select {49 select {
51 case <-done:50 case <-done:
@@ -64,5 +63,22 @@ func TestRelayConfigKeepsStartupHintEnable(t *testing.T) {
64 next := config.Defaults()63 next := config.Defaults()
65 next.Hint.Enable = true64 next.Hint.Enable = true
66 in <- ports.ConfigChanged{Config: next}65 in <- ports.ConfigChanged{Config: next}
67- require.False(t, (<-out).Config.Hint.Enable)66+ got := (<-out).Config.Hint
67+ require.False(t, got.Enable)
68+}
69+
70+func TestRelayConfigKeepsStartupHintDetector(t *testing.T) {
71+ ctx, cancel := context.WithCancel(context.Background())
72+ defer cancel()
73+ cur := config.Defaults()
74+ in := make(chan ports.ConfigChanged, 1)
75+ out := make(chan ports.ConfigChanged, 1)
76+ go relayConfig(ctx, cur, in, out, make(chan *xkb.Keymap, 1), make(chan ports.InputDevicesConfig, 1), newCursors(clock.System{}, 0), make(chan ports.ClientCommand, 1), logging.For(ctx, "config"))
77+ next := config.Defaults()
78+ next.Hint.Detector, next.Hint.Model, next.Hint.Threshold = "/other/nh", "/other/models", 0.5
79+ in <- ports.ConfigChanged{Config: next}
80+ got := (<-out).Config.Hint
81+ require.Equal(t, cur.Hint.Detector, got.Detector)
82+ require.Equal(t, cur.Hint.Model, got.Model)
83+ require.Equal(t, cur.Hint.Threshold, got.Threshold)
68 }84 }
M internal/app/session.go +2 -2
@@ -102,9 +102,9 @@ func assembleSession(ctx context.Context, opts Options) (s *session, err error)
102102 return s, err
103103 }
104104 runtimeDir := os.Getenv("XDG_RUNTIME_DIR")
105- // Hint mode off: no accessibility connection, no debug env, nothing.
105+ // Hint mode off: no detector, no debug env, nothing.
106106 if opts.Config.Hint.Enable {
107- s.hintOpts = hintOptions(ctx)
107+ s.hintOpts = hintOptions(ctx, opts.Config)
108108 }
109109 // Every output renders on the same GPU: its formats are the clients'.
110110 s.dmabuf = vulkan.Probe()
@@ -102,9 +102,9 @@ func assembleSession(ctx context.Context, opts Options) (s *session, err error)
102 return s, err102 return s, err
103 }103 }
104 runtimeDir := os.Getenv("XDG_RUNTIME_DIR")104 runtimeDir := os.Getenv("XDG_RUNTIME_DIR")
105- // Hint mode off: no accessibility connection, no debug env, nothing.105+ // Hint mode off: no detector, no debug env, nothing.
106 if opts.Config.Hint.Enable {106 if opts.Config.Hint.Enable {
107- s.hintOpts = hintOptions(ctx)107+ s.hintOpts = hintOptions(ctx, opts.Config)
108 }108 }
109 // Every output renders on the same GPU: its formats are the clients'.109 // Every output renders on the same GPU: its formats are the clients'.
110 s.dmabuf = vulkan.Probe()110 s.dmabuf = vulkan.Probe()
M internal/core/core.go +3 -0
@@ -353,6 +353,9 @@ func (c *Core) apply(cfg ports.Config) error {
353353 for _, s := range c.screens {
354354 axes[s.mon] = s.mon.Current().policy()
355355 }
356+ if c.cfg.Hint.Size != cfg.Hint.Size {
357+ c.hintCancel()
358+ }
356359 c.cfg = cfg
357360 c.binds = binds
358361 c.cmdMod = map[string]ports.Mods{"super": ports.ModSuper, "alt": ports.ModAlt, "ctrl": ports.ModCtrl}[cfg.Keyboard.CmdKey]
@@ -353,6 +353,9 @@ func (c *Core) apply(cfg ports.Config) error {
353 for _, s := range c.screens {353 for _, s := range c.screens {
354 axes[s.mon] = s.mon.Current().policy()354 axes[s.mon] = s.mon.Current().policy()
355 }355 }
356+ if c.cfg.Hint.Size != cfg.Hint.Size {
357+ c.hintCancel()
358+ }
356 c.cfg = cfg359 c.cfg = cfg
357 c.binds = binds360 c.binds = binds
358 c.cmdMod = map[string]ports.Mods{"super": ports.ModSuper, "alt": ports.ModAlt, "ctrl": ports.ModCtrl}[cfg.Keyboard.CmdKey]361 c.cmdMod = map[string]ports.Mods{"super": ports.ModSuper, "alt": ports.ModAlt, "ctrl": ports.ModCtrl}[cfg.Keyboard.CmdKey]
M internal/core/hint.go +84 -35
@@ -3,6 +3,7 @@ package core
33 import (
44 "cmp"
55 "context"
6+ "math"
67 "slices"
78
89 "github.com/bnema/neferwl/internal/ports"
@@ -32,11 +33,11 @@ const (
3233 // coordinates of its centre.
3334 type hintTarget struct {
3435 x, y float64
36+ score float32
3537 window WindowID
3638 source hintSourceGeom // source window and its request-time client geometry
3739 label string
3840 rect ports.Rect
39- text bool // found by OCR
4041 }
4142
4243 type hintSourceGeom struct {
@@ -52,7 +53,7 @@ type hintState struct {
5253 // one on screen ends it.
5354 ws *Workspace
5455 geom hintGeom
55- windows []ports.HintWindow // visible windows at request time
56+ windows []hintSourceGeom // visible windows and their client geometry at request time
5657 targets []hintTarget
5758 typed string
5859 }
@@ -91,7 +92,7 @@ func (c *Core) hintRequest() {
9192 return
9293 }
9394 wins := c.hintWindows(sc)
94- req := ports.HintScanRequest{Seq: c.hint.seq + 1, Output: sc.name(), Width: sc.info.Width, Height: sc.info.Height, Scale: sc.scale, LinksOnly: c.cfg.Hint.LinksOnly, Windows: wins}
95+ req := ports.HintScanRequest{Seq: c.hint.seq + 1, Output: sc.name(), Width: sc.info.Width, Height: sc.info.Height, Scale: sc.scale}
9596 select {
9697 case c.ch.HintScans <- req:
9798 default:
@@ -101,10 +102,11 @@ func (c *Core) hintRequest() {
101102 }
102103
103104 // hintWindows lists the windows drawn on sc, bottom to top, with their client
104-// area (placements are output-local).
105-func (c *Core) hintWindows(sc *screen) []ports.HintWindow {
105+// area (placements are output-local). A target is only clicked while the
106+// geometry of its window is the one seen here.
107+func (c *Core) hintWindows(sc *screen) []hintSourceGeom {
106108 view := sc.mon.viewport()
107- var out []ports.HintWindow
109+ var out []hintSourceGeom
108110 for _, p := range sc.shownLayout() {
109111 if p.Hidden || p.Preview > 0 || !view.shows(p) {
110112 continue
@@ -113,10 +115,7 @@ func (c *Core) hintWindows(sc *screen) []ports.HintWindow {
113115 if r.W <= 0 || r.H <= 0 {
114116 continue
115117 }
116- w := ports.HintWindow{ID: p.ID, Rect: r, Focused: p.Focused}
117- rec := c.windows.lookup(p.ID)
118- w.PID, w.AppID = rec.client.PID, rec.client.AppID
119- out = append(out, w)
118+ out = append(out, hintSourceGeom{id: p.ID, rect: r})
120119 }
121120 return out
122121 }
@@ -167,16 +166,17 @@ func (c *Core) hintResult(res ports.HintScanResult) bool {
167166 // A layer surface, or a window of another output.
168167 continue
169168 }
170- source, ok := c.hintSource(sc, t, root)
169+ source, ok := c.hintSource(sc, root)
171170 if !ok {
172171 continue
173172 }
174- kept = append(kept, hintTarget{x: x, y: y, window: id, source: source, rect: t.Rect, text: t.Text})
173+ kept = append(kept, hintTarget{x: x, y: y, window: id, source: source, rect: t.Rect, score: t.Score})
175174 }
176175 if len(kept) == 0 {
177176 c.hintCancel()
178177 return true
179178 }
179+ kept = c.hintSpread(sc, kept)
180180 slices.SortStableFunc(kept, func(a, b hintTarget) int {
181181 return cmp.Or(cmp.Compare(a.rect.Y, b.rect.Y), cmp.Compare(a.rect.X, b.rect.X))
182182 })
@@ -195,15 +195,76 @@ func (c *Core) hintResult(res ports.HintScanResult) bool {
195195 return true
196196 }
197197
198-// hintSource resolves the request-time source window only while its client
199-// geometry is unchanged. This applies equally to accessibility and OCR targets.
200-func (c *Core) hintSource(sc *screen, t ports.HintTarget, root WindowID) (hintSourceGeom, bool) {
201- if t.Window != 0 && root != t.Window {
202- return hintSourceGeom{}, false
198+// hintLabelSize estimates, in logical pixels, the box of a two-letter label on
199+// sc: a monospace face (advance 0.6 em) plus padding and outline, rounded up.
200+// The font size follows the renderer: hint.size, or the output's density when
201+// it is automatic (same rules as hintlabel.FontPixels, which core cannot
202+// import).
203+func (c *Core) hintLabelSize(sc *screen) (w, h int) {
204+ scale := sc.scale
205+ if scale <= 0 {
206+ scale = 1
207+ }
208+ font := c.cfg.Hint.Size // logical
209+ if font <= 0 {
210+ px := 13 * scale * math.Sqrt(math.Max(1, float64(sc.info.Height)/(1080*scale)))
211+ if sc.info.Width > 0 && sc.info.Height > 0 && sc.info.PhysicalW > 0 && sc.info.PhysicalH > 0 {
212+ ratio := (float64(sc.info.Width) / float64(sc.info.Height)) / (float64(sc.info.PhysicalW) / float64(sc.info.PhysicalH))
213+ if math.Abs(ratio-1) <= .2 {
214+ px = math.Min(math.Round(40*scale), math.Max(math.Round(13*scale), math.Round(3.2*float64(sc.info.Width)/float64(sc.info.PhysicalW))))
215+ }
216+ }
217+ font = px / scale
218+ }
219+ font = math.Min(font, 256/scale)
220+ s := font / 13
221+ return int(math.Ceil(2*0.6*font + 6*s + 2/scale + 1)), int(math.Ceil(1.25*font + 2*s + 2/scale + 1))
222+}
223+
224+// hintLabelRect mirrors the renderer's outside top-left anchor and output
225+// clamping, in output-local logical coordinates.
226+func hintLabelRect(sc *screen, target ports.Rect, lw, lh int) ports.Rect {
227+ bounds := sc.rect()
228+ inset := int(math.Round(0.25 * float64(lh)))
229+ return ports.Rect{X: min(max(target.X-lw+inset, 0), bounds.W-lw), Y: min(max(target.Y-lh+inset, 0), bounds.H-lh), W: lw, H: lh}
230+}
231+
232+// hintSpread drops targets whose output-clamped label rectangles intersect.
233+// Highest scores win; four times the label budget gives spreading headroom
234+// while bounding the quadratic pass. hintlabel.FontPixels is the source of
235+// truth for the font rule mirrored above.
236+func (c *Core) hintSpread(sc *screen, ts []hintTarget) []hintTarget {
237+ lw, lh := c.hintLabelSize(sc)
238+ order := slices.Clone(ts)
239+ slices.SortStableFunc(order, func(a, b hintTarget) int {
240+ return cmp.Or(cmp.Compare(b.score, a.score), cmp.Compare(a.rect.Y, b.rect.Y), cmp.Compare(a.rect.X, b.rect.X))
241+ })
242+ if len(order) > 4*hintMaxTargets {
243+ order = order[:4*hintMaxTargets]
244+ }
245+ kept := order[:0]
246+ for _, t := range order {
247+ clash := false
248+ for _, k := range kept {
249+ a, b := hintLabelRect(sc, t.rect, lw, lh), hintLabelRect(sc, k.rect, lw, lh)
250+ if a.X < b.X+b.W && b.X < a.X+a.W && a.Y < b.Y+b.H && b.Y < a.Y+a.H {
251+ clash = true
252+ break
253+ }
254+ }
255+ if !clash {
256+ kept = append(kept, t)
257+ }
203258 }
259+ return kept
260+}
261+
262+// hintSource resolves the request-time source window only while its client
263+// geometry is unchanged.
264+func (c *Core) hintSource(sc *screen, root WindowID) (hintSourceGeom, bool) {
204265 for _, w := range c.hint.windows {
205- if w.ID == root && c.hintSourceUnchanged(sc, hintSourceGeom{id: w.ID, rect: w.Rect}) {
206- return hintSourceGeom{id: w.ID, rect: w.Rect}, true
266+ if w.id == root && c.hintSourceUnchanged(sc, w) {
267+ return w, true
207268 }
208269 }
209270 return hintSourceGeom{}, false
@@ -292,10 +353,7 @@ func (c *Core) hintKey(ctx context.Context, key ports.KeyEvent) (bool, error) {
292353 // synthetic: c.synthetic keeps the physical modifiers (the bind's Cmd may
293354 // still be down) from turning it into a window drag, without touching c.mods.
294355 //
295-// A target found by OCR is clicked with hint.text-click held (terminals open
296-// a link on a modified click). Every click gives the window the keyboard
297-// first, then sends exactly the modifiers to hold (none for other targets),
298-// and the real modifier state is always sent back afterwards.
356+// The window gets the keyboard first.
299357 func (c *Core) hintClick(ctx context.Context, t hintTarget) (err error) {
300358 c.hintCancel()
301359 if err := c.publish(ctx); err != nil {
@@ -304,12 +362,12 @@ func (c *Core) hintClick(ctx context.Context, t hintTarget) (err error) {
304362 if !c.hintClickable(t) {
305363 return nil
306364 }
307- // Every click states its modifiers (possibly none): the bind's Cmd may
308- // still be down and the client must not see it on a plain click.
309365 if err := c.clickFocus(ctx, t.window); err != nil {
310366 return err
311367 }
312- if err := c.command(ctx, ports.SetModifiers{ID: t.window, Mods: c.hintMods(t)}); err != nil {
368+ // The physical modifiers (bind's Super, Ctrl, Shift) must not reach the
369+ // client: state none, and always send the real state back afterwards.
370+ if err := c.command(ctx, ports.SetModifiers{ID: t.window, Mods: 0}); err != nil {
313371 return err
314372 }
315373 defer func() {
@@ -333,15 +391,6 @@ func (c *Core) hintClick(ctx context.Context, t hintTarget) (err error) {
333391 return c.pointerButton(ctx, ports.PointerButton{Button: btnLeft, Time: c.motionTime})
334392 }
335393
336-// hintMods are the modifiers to hold for a click on t, read at click time so
337-// a reloaded configuration applies.
338-func (c *Core) hintMods(t hintTarget) ports.Mods {
339- if !t.text {
340- return 0
341- }
342- return c.cfg.Hint.TextClick
343-}
344-
345394 // hintClickable reports whether a click on t may go out now: unprotected,
346395 // pointer free, and the selected window is still under the target's centre.
347396 func (c *Core) hintClickable(t hintTarget) bool {
@@ -3,6 +3,7 @@ package core
3 import (3 import (
4 "cmp"4 "cmp"
5 "context"5 "context"
6+ "math"
6 "slices"7 "slices"
7 8
8 "github.com/bnema/neferwl/internal/ports"9 "github.com/bnema/neferwl/internal/ports"
@@ -32,11 +33,11 @@ const (
32 // coordinates of its centre.33 // coordinates of its centre.
33 type hintTarget struct {34 type hintTarget struct {
34 x, y float6435 x, y float64
36+ score float32
35 window WindowID37 window WindowID
36 source hintSourceGeom // source window and its request-time client geometry38 source hintSourceGeom // source window and its request-time client geometry
37 label string39 label string
38 rect ports.Rect40 rect ports.Rect
39- text bool // found by OCR
40 }41 }
41 42
42 type hintSourceGeom struct {43 type hintSourceGeom struct {
@@ -52,7 +53,7 @@ type hintState struct {
52 // one on screen ends it.53 // one on screen ends it.
53 ws *Workspace54 ws *Workspace
54 geom hintGeom55 geom hintGeom
55- windows []ports.HintWindow // visible windows at request time56+ windows []hintSourceGeom // visible windows and their client geometry at request time
56 targets []hintTarget57 targets []hintTarget
57 typed string58 typed string
58 }59 }
@@ -91,7 +92,7 @@ func (c *Core) hintRequest() {
91 return92 return
92 }93 }
93 wins := c.hintWindows(sc)94 wins := c.hintWindows(sc)
94- req := ports.HintScanRequest{Seq: c.hint.seq + 1, Output: sc.name(), Width: sc.info.Width, Height: sc.info.Height, Scale: sc.scale, LinksOnly: c.cfg.Hint.LinksOnly, Windows: wins}95+ req := ports.HintScanRequest{Seq: c.hint.seq + 1, Output: sc.name(), Width: sc.info.Width, Height: sc.info.Height, Scale: sc.scale}
95 select {96 select {
96 case c.ch.HintScans <- req:97 case c.ch.HintScans <- req:
97 default:98 default:
@@ -101,10 +102,11 @@ func (c *Core) hintRequest() {
101 }102 }
102 103
103 // hintWindows lists the windows drawn on sc, bottom to top, with their client104 // hintWindows lists the windows drawn on sc, bottom to top, with their client
104-// area (placements are output-local).105+// area (placements are output-local). A target is only clicked while the
105-func (c *Core) hintWindows(sc *screen) []ports.HintWindow {106+// geometry of its window is the one seen here.
107+func (c *Core) hintWindows(sc *screen) []hintSourceGeom {
106 view := sc.mon.viewport()108 view := sc.mon.viewport()
107- var out []ports.HintWindow109+ var out []hintSourceGeom
108 for _, p := range sc.shownLayout() {110 for _, p := range sc.shownLayout() {
109 if p.Hidden || p.Preview > 0 || !view.shows(p) {111 if p.Hidden || p.Preview > 0 || !view.shows(p) {
110 continue112 continue
@@ -113,10 +115,7 @@ func (c *Core) hintWindows(sc *screen) []ports.HintWindow {
113 if r.W <= 0 || r.H <= 0 {115 if r.W <= 0 || r.H <= 0 {
114 continue116 continue
115 }117 }
116- w := ports.HintWindow{ID: p.ID, Rect: r, Focused: p.Focused}118+ out = append(out, hintSourceGeom{id: p.ID, rect: r})
117- rec := c.windows.lookup(p.ID)
118- w.PID, w.AppID = rec.client.PID, rec.client.AppID
119- out = append(out, w)
120 }119 }
121 return out120 return out
122 }121 }
@@ -167,16 +166,17 @@ func (c *Core) hintResult(res ports.HintScanResult) bool {
167 // A layer surface, or a window of another output.166 // A layer surface, or a window of another output.
168 continue167 continue
169 }168 }
170- source, ok := c.hintSource(sc, t, root)169+ source, ok := c.hintSource(sc, root)
171 if !ok {170 if !ok {
172 continue171 continue
173 }172 }
174- kept = append(kept, hintTarget{x: x, y: y, window: id, source: source, rect: t.Rect, text: t.Text})173+ kept = append(kept, hintTarget{x: x, y: y, window: id, source: source, rect: t.Rect, score: t.Score})
175 }174 }
176 if len(kept) == 0 {175 if len(kept) == 0 {
177 c.hintCancel()176 c.hintCancel()
178 return true177 return true
179 }178 }
179+ kept = c.hintSpread(sc, kept)
180 slices.SortStableFunc(kept, func(a, b hintTarget) int {180 slices.SortStableFunc(kept, func(a, b hintTarget) int {
181 return cmp.Or(cmp.Compare(a.rect.Y, b.rect.Y), cmp.Compare(a.rect.X, b.rect.X))181 return cmp.Or(cmp.Compare(a.rect.Y, b.rect.Y), cmp.Compare(a.rect.X, b.rect.X))
182 })182 })
@@ -195,15 +195,76 @@ func (c *Core) hintResult(res ports.HintScanResult) bool {
195 return true195 return true
196 }196 }
197 197
198-// hintSource resolves the request-time source window only while its client198+// hintLabelSize estimates, in logical pixels, the box of a two-letter label on
199-// geometry is unchanged. This applies equally to accessibility and OCR targets.199+// sc: a monospace face (advance 0.6 em) plus padding and outline, rounded up.
200-func (c *Core) hintSource(sc *screen, t ports.HintTarget, root WindowID) (hintSourceGeom, bool) {200+// The font size follows the renderer: hint.size, or the output's density when
201- if t.Window != 0 && root != t.Window {201+// it is automatic (same rules as hintlabel.FontPixels, which core cannot
202- return hintSourceGeom{}, false202+// import).
203+func (c *Core) hintLabelSize(sc *screen) (w, h int) {
204+ scale := sc.scale
205+ if scale <= 0 {
206+ scale = 1
207+ }
208+ font := c.cfg.Hint.Size // logical
209+ if font <= 0 {
210+ px := 13 * scale * math.Sqrt(math.Max(1, float64(sc.info.Height)/(1080*scale)))
211+ if sc.info.Width > 0 && sc.info.Height > 0 && sc.info.PhysicalW > 0 && sc.info.PhysicalH > 0 {
212+ ratio := (float64(sc.info.Width) / float64(sc.info.Height)) / (float64(sc.info.PhysicalW) / float64(sc.info.PhysicalH))
213+ if math.Abs(ratio-1) <= .2 {
214+ px = math.Min(math.Round(40*scale), math.Max(math.Round(13*scale), math.Round(3.2*float64(sc.info.Width)/float64(sc.info.PhysicalW))))
215+ }
216+ }
217+ font = px / scale
218+ }
219+ font = math.Min(font, 256/scale)
220+ s := font / 13
221+ return int(math.Ceil(2*0.6*font + 6*s + 2/scale + 1)), int(math.Ceil(1.25*font + 2*s + 2/scale + 1))
222+}
223+
224+// hintLabelRect mirrors the renderer's outside top-left anchor and output
225+// clamping, in output-local logical coordinates.
226+func hintLabelRect(sc *screen, target ports.Rect, lw, lh int) ports.Rect {
227+ bounds := sc.rect()
228+ inset := int(math.Round(0.25 * float64(lh)))
229+ return ports.Rect{X: min(max(target.X-lw+inset, 0), bounds.W-lw), Y: min(max(target.Y-lh+inset, 0), bounds.H-lh), W: lw, H: lh}
230+}
231+
232+// hintSpread drops targets whose output-clamped label rectangles intersect.
233+// Highest scores win; four times the label budget gives spreading headroom
234+// while bounding the quadratic pass. hintlabel.FontPixels is the source of
235+// truth for the font rule mirrored above.
236+func (c *Core) hintSpread(sc *screen, ts []hintTarget) []hintTarget {
237+ lw, lh := c.hintLabelSize(sc)
238+ order := slices.Clone(ts)
239+ slices.SortStableFunc(order, func(a, b hintTarget) int {
240+ return cmp.Or(cmp.Compare(b.score, a.score), cmp.Compare(a.rect.Y, b.rect.Y), cmp.Compare(a.rect.X, b.rect.X))
241+ })
242+ if len(order) > 4*hintMaxTargets {
243+ order = order[:4*hintMaxTargets]
244+ }
245+ kept := order[:0]
246+ for _, t := range order {
247+ clash := false
248+ for _, k := range kept {
249+ a, b := hintLabelRect(sc, t.rect, lw, lh), hintLabelRect(sc, k.rect, lw, lh)
250+ if a.X < b.X+b.W && b.X < a.X+a.W && a.Y < b.Y+b.H && b.Y < a.Y+a.H {
251+ clash = true
252+ break
253+ }
254+ }
255+ if !clash {
256+ kept = append(kept, t)
257+ }
203 }258 }
259+ return kept
260+}
261+
262+// hintSource resolves the request-time source window only while its client
263+// geometry is unchanged.
264+func (c *Core) hintSource(sc *screen, root WindowID) (hintSourceGeom, bool) {
204 for _, w := range c.hint.windows {265 for _, w := range c.hint.windows {
205- if w.ID == root && c.hintSourceUnchanged(sc, hintSourceGeom{id: w.ID, rect: w.Rect}) {266+ if w.id == root && c.hintSourceUnchanged(sc, w) {
206- return hintSourceGeom{id: w.ID, rect: w.Rect}, true267+ return w, true
207 }268 }
208 }269 }
209 return hintSourceGeom{}, false270 return hintSourceGeom{}, false
@@ -292,10 +353,7 @@ func (c *Core) hintKey(ctx context.Context, key ports.KeyEvent) (bool, error) {
292 // synthetic: c.synthetic keeps the physical modifiers (the bind's Cmd may353 // synthetic: c.synthetic keeps the physical modifiers (the bind's Cmd may
293 // still be down) from turning it into a window drag, without touching c.mods.354 // still be down) from turning it into a window drag, without touching c.mods.
294 //355 //
295-// A target found by OCR is clicked with hint.text-click held (terminals open356+// The window gets the keyboard first.
296-// a link on a modified click). Every click gives the window the keyboard
297-// first, then sends exactly the modifiers to hold (none for other targets),
298-// and the real modifier state is always sent back afterwards.
299 func (c *Core) hintClick(ctx context.Context, t hintTarget) (err error) {357 func (c *Core) hintClick(ctx context.Context, t hintTarget) (err error) {
300 c.hintCancel()358 c.hintCancel()
301 if err := c.publish(ctx); err != nil {359 if err := c.publish(ctx); err != nil {
@@ -304,12 +362,12 @@ func (c *Core) hintClick(ctx context.Context, t hintTarget) (err error) {
304 if !c.hintClickable(t) {362 if !c.hintClickable(t) {
305 return nil363 return nil
306 }364 }
307- // Every click states its modifiers (possibly none): the bind's Cmd may
308- // still be down and the client must not see it on a plain click.
309 if err := c.clickFocus(ctx, t.window); err != nil {365 if err := c.clickFocus(ctx, t.window); err != nil {
310 return err366 return err
311 }367 }
312- if err := c.command(ctx, ports.SetModifiers{ID: t.window, Mods: c.hintMods(t)}); err != nil {368+ // The physical modifiers (bind's Super, Ctrl, Shift) must not reach the
369+ // client: state none, and always send the real state back afterwards.
370+ if err := c.command(ctx, ports.SetModifiers{ID: t.window, Mods: 0}); err != nil {
313 return err371 return err
314 }372 }
315 defer func() {373 defer func() {
@@ -333,15 +391,6 @@ func (c *Core) hintClick(ctx context.Context, t hintTarget) (err error) {
333 return c.pointerButton(ctx, ports.PointerButton{Button: btnLeft, Time: c.motionTime})391 return c.pointerButton(ctx, ports.PointerButton{Button: btnLeft, Time: c.motionTime})
334 }392 }
335 393
336-// hintMods are the modifiers to hold for a click on t, read at click time so
337-// a reloaded configuration applies.
338-func (c *Core) hintMods(t hintTarget) ports.Mods {
339- if !t.text {
340- return 0
341- }
342- return c.cfg.Hint.TextClick
343-}
344-
345 // hintClickable reports whether a click on t may go out now: unprotected,394 // hintClickable reports whether a click on t may go out now: unprotected,
346 // pointer free, and the selected window is still under the target's centre.395 // pointer free, and the selected window is still under the target's centre.
347 func (c *Core) hintClickable(t hintTarget) bool {396 func (c *Core) hintClickable(t hintTarget) bool {
A internal/core/hint_internal_test.go +45 -0
@@ -0,0 +1,45 @@
1+package core
2+
3+import (
4+ "github.com/bnema/neferwl/internal/ports"
5+ "github.com/stretchr/testify/require"
6+ "math"
7+ "testing"
8+)
9+
10+func TestHintLabelGeometry(t *testing.T) {
11+ for _, scale := range []float64{1, 1.5} {
12+ sc := &screen{scale: scale, info: ports.OutputInfo{Width: 1500, Height: 1200}}
13+ sc.mon = newMonitor("", "")
14+ sc.mon.SetOutput(int(float64(sc.info.Width)/sc.scale), int(float64(sc.info.Height)/sc.scale))
15+ c := &Core{}
16+ c.cfg.Hint.Size = 200
17+ w, h := c.hintLabelSize(sc)
18+ font := math.Min(200, 256/scale)
19+ require.Equal(t, int(math.Ceil(1.2*font+6*font/13+2/scale+1)), w)
20+ require.Equal(t, int(math.Ceil(1.25*font+2*font/13+2/scale+1)), h)
21+ bounds := sc.rect()
22+ // Corner separation exceeds label width, but right-edge clamping folds
23+ // the second label back over the first.
24+ a := hintTarget{rect: ports.Rect{X: bounds.W - w - 1, Y: bounds.H - h - 1}, score: 1}
25+ b := hintTarget{rect: ports.Rect{X: bounds.W - 1, Y: bounds.H - 1}, score: .5}
26+ require.Len(t, c.hintSpread(sc, []hintTarget{a, b}), 1)
27+ b.rect = ports.Rect{X: 0, Y: 0}
28+ require.Len(t, c.hintSpread(sc, []hintTarget{a, b}), 2)
29+ }
30+}
31+
32+func TestHintSpreadBudget(t *testing.T) {
33+ sc := &screen{scale: 1, info: ports.OutputInfo{Width: 1000000, Height: 1000}}
34+ sc.mon = newMonitor("", "")
35+ sc.mon.SetOutput(int(float64(sc.info.Width)/sc.scale), int(float64(sc.info.Height)/sc.scale))
36+ c := &Core{}
37+ ts := make([]hintTarget, 4*hintMaxTargets+10)
38+ for i := range ts {
39+ ts[i] = hintTarget{rect: ports.Rect{X: 100 + i*100, Y: 100}, score: float32(i)}
40+ }
41+ got := c.hintSpread(sc, ts)
42+ require.Len(t, got, 4*hintMaxTargets)
43+ require.Equal(t, float32(len(ts)-1), got[0].score)
44+ require.Equal(t, float32(10), got[len(got)-1].score)
45+}
@@ -0,0 +1,45 @@
1+package core
2+
3+import (
4+ "github.com/bnema/neferwl/internal/ports"
5+ "github.com/stretchr/testify/require"
6+ "math"
7+ "testing"
8+)
9+
10+func TestHintLabelGeometry(t *testing.T) {
11+ for _, scale := range []float64{1, 1.5} {
12+ sc := &screen{scale: scale, info: ports.OutputInfo{Width: 1500, Height: 1200}}
13+ sc.mon = newMonitor("", "")
14+ sc.mon.SetOutput(int(float64(sc.info.Width)/sc.scale), int(float64(sc.info.Height)/sc.scale))
15+ c := &Core{}
16+ c.cfg.Hint.Size = 200
17+ w, h := c.hintLabelSize(sc)
18+ font := math.Min(200, 256/scale)
19+ require.Equal(t, int(math.Ceil(1.2*font+6*font/13+2/scale+1)), w)
20+ require.Equal(t, int(math.Ceil(1.25*font+2*font/13+2/scale+1)), h)
21+ bounds := sc.rect()
22+ // Corner separation exceeds label width, but right-edge clamping folds
23+ // the second label back over the first.
24+ a := hintTarget{rect: ports.Rect{X: bounds.W - w - 1, Y: bounds.H - h - 1}, score: 1}
25+ b := hintTarget{rect: ports.Rect{X: bounds.W - 1, Y: bounds.H - 1}, score: .5}
26+ require.Len(t, c.hintSpread(sc, []hintTarget{a, b}), 1)
27+ b.rect = ports.Rect{X: 0, Y: 0}
28+ require.Len(t, c.hintSpread(sc, []hintTarget{a, b}), 2)
29+ }
30+}
31+
32+func TestHintSpreadBudget(t *testing.T) {
33+ sc := &screen{scale: 1, info: ports.OutputInfo{Width: 1000000, Height: 1000}}
34+ sc.mon = newMonitor("", "")
35+ sc.mon.SetOutput(int(float64(sc.info.Width)/sc.scale), int(float64(sc.info.Height)/sc.scale))
36+ c := &Core{}
37+ ts := make([]hintTarget, 4*hintMaxTargets+10)
38+ for i := range ts {
39+ ts[i] = hintTarget{rect: ports.Rect{X: 100 + i*100, Y: 100}, score: float32(i)}
40+ }
41+ got := c.hintSpread(sc, ts)
42+ require.Len(t, got, 4*hintMaxTargets)
43+ require.Equal(t, float32(len(ts)-1), got[0].score)
44+ require.Equal(t, float32(10), got[len(got)-1].score)
45+}
M internal/core/hint_test.go +92 -124
@@ -24,19 +24,15 @@ type hintHarness struct {
2424 hints chan ports.HintScanResult
2525 win ports.Rect // window 1, output-local
2626 // state is the security state input is stamped with.
27- state ports.SecurityState
27+ state ports.SecurityState
28+ config chan ports.ConfigChanged
29+ cfg ports.Config
2830 }
2931
3032 // newHint starts a core with one 1000x800 output "OUT-1" and window 1 mapped.
3133 func newHint(t *testing.T, opts core.Options) *hintHarness {
3234 t.Helper()
33- return newHintMods(t, opts, 0)
34-}
35-
36-// newHintMods is newHint with hint.text-click set to textClick.
37-func newHintMods(t *testing.T, opts core.Options, textClick ports.Mods) *hintHarness {
38- t.Helper()
39- return newHintConfig(t, opts, func(cfg *ports.Config) { cfg.Hint.TextClick = textClick })
35+ return newHintConfig(t, opts, func(*ports.Config) {})
4036 }
4137
4238 // newHintConfig is newHint with the config edited by edit.
@@ -48,6 +44,8 @@ func newHintConfig(t *testing.T, opts core.Options, edit func(*ports.Config)) *h
4844 cfg.Layout.Gaps = 20
4945 cfg.Binds["Cmd+g"] = "hint"
5046 h := &hintHarness{
47+ config: make(chan ports.ConfigChanged),
48+ cfg: cfg,
5149 t: t,
5250 input: make(chan ports.InputEvent, 32),
5351 client: make(chan ports.ClientEvent, 32),
@@ -58,7 +56,7 @@ func newHintConfig(t *testing.T, opts core.Options, edit func(*ports.Config)) *h
5856 // Unbuffered: a send returns once core took the result.
5957 hints: make(chan ports.HintScanResult),
6058 }
61- ch := core.Channels{Client: h.client, Input: h.input, Output: h.output, Commands: h.commands, Scenes: h.scenes, HintScans: h.scans, Hints: h.hints}
59+ ch := core.Channels{Config: h.config, Client: h.client, Input: h.input, Output: h.output, Commands: h.commands, Scenes: h.scenes, HintScans: h.scans, Hints: h.hints}
6260 c, err := core.New(cfg, ch, opts)
6361 if err != nil {
6462 t.Fatal(err)
@@ -138,19 +136,15 @@ func (h *hintHarness) labelled(n, typed int) ports.Scene {
138136
139137 // inside is a target of the given size inside window 1.
140138 func (h *hintHarness) inside(i int) ports.HintTarget {
141- return ports.HintTarget{Rect: ports.Rect{X: h.win.X + 10 + i, Y: h.win.Y + 10 + 12*i, W: 40, H: 10}}
139+ return ports.HintTarget{Rect: ports.Rect{X: h.win.X + 10 + 100*(i%4), Y: h.win.Y + 10 + 40*(i/4), W: 40, H: 10}}
142140 }
143141
144142 func TestHintRequestOnBind(t *testing.T) {
145143 h := newHint(t, core.Options{})
146144 req := h.start()
147- if req.Output != "OUT-1" || req.Width != 1000 || req.Height != 800 || req.Scale != 1 || req.LinksOnly || req.Seq == 0 {
145+ if req.Output != "OUT-1" || req.Width != 1000 || req.Height != 800 || req.Scale != 1 || req.Seq == 0 {
148146 t.Fatalf("request %+v", req)
149147 }
150- want := []ports.HintWindow{{ID: 1, PID: 4242, AppID: "org.test.app", Rect: h.win, Focused: true}}
151- if !slices.Equal(req.Windows, want) {
152- t.Fatalf("windows %+v, want %+v", req.Windows, want)
153- }
154148 // A second bind while pending does not ask again; the pending phase
155149 // cancels on any key.
156150 h.press("g", ports.ModSuper)
@@ -236,7 +230,7 @@ func TestHintTwoLetterLabels(t *testing.T) {
236230 h := newHint(t, core.Options{})
237231 var targets []ports.HintTarget
238232 for i := 0; i < 27; i++ {
239- targets = append(targets, ports.HintTarget{Rect: ports.Rect{X: h.win.X + 5, Y: h.win.Y + 5 + 14*i, W: 30, H: 10}})
233+ targets = append(targets, ports.HintTarget{Rect: ports.Rect{X: h.win.X + 5 + 100*(i%4), Y: h.win.Y + 5 + 40*(i/4), W: 30, H: 10}})
240234 }
241235 s := h.show(targets...)
242236 if len(s.HintLabels) != 27 {
@@ -274,7 +268,7 @@ func TestHintTypingFilters(t *testing.T) {
274268 h := newHint(t, core.Options{})
275269 var targets []ports.HintTarget
276270 for i := 0; i < 27; i++ {
277- targets = append(targets, ports.HintTarget{Rect: ports.Rect{X: h.win.X + 5, Y: h.win.Y + 5 + 14*i, W: 30, H: 10}})
271+ targets = append(targets, ports.HintTarget{Rect: ports.Rect{X: h.win.X + 5 + 100*(i%4), Y: h.win.Y + 5 + 40*(i/4), W: 30, H: 10}})
278272 }
279273 h.show(targets...)
280274 h.press("f", 0)
@@ -472,7 +466,20 @@ func TestHintClickWithBindModifierHeld(t *testing.T) {
472466 h.drain()
473467 h.press("f", ports.ModSuper) // Super is still down
474468 h.labelled(0, 0)
475- presses, releases := h.buttonsSent()
469+ cmds := h.barrier()
470+ if got, want := commandKinds(cmds), []string{"mods:0", "motion", "press", "release", "mods:restore"}; !slices.Equal(got, want) {
471+ t.Fatalf("commands %v, want %v", got, want)
472+ }
473+ presses, releases := 0, 0
474+ for _, c := range cmds {
475+ if b, ok := c.(ports.PointerButtonTo); ok {
476+ if b.Pressed {
477+ presses++
478+ } else {
479+ releases++
480+ }
481+ }
482+ }
476483 if presses != 1 || releases != 1 {
477484 t.Fatalf("client got %d presses, %d releases", presses, releases)
478485 }
@@ -593,12 +600,12 @@ func TestHintMappedWindowMovedAborts(t *testing.T) {
593600 }
594601
595602 // commandsSent drains the commands up to a barrier, keeping the ones that
596-// matter for a click: modifiers, focus, motion and buttons.
603+// matter for a click: focus, motion and buttons.
597604 func (h *hintHarness) commandsSent() []ports.ClientCommand {
598605 var out []ports.ClientCommand
599606 for _, cmd := range h.barrier() {
600607 switch cmd.(type) {
601- case ports.SetModifiers, ports.FocusWindow, ports.PointerMotionTo, ports.PointerButtonTo:
608+ case ports.FocusWindow, ports.SetModifiers, ports.PointerMotionTo, ports.PointerButtonTo:
602609 out = append(out, cmd)
603610 }
604611 }
@@ -609,14 +616,14 @@ func commandKinds(cmds []ports.ClientCommand) []string {
609616 var out []string
610617 for _, c := range cmds {
611618 switch v := c.(type) {
619+ case ports.FocusWindow:
620+ out = append(out, fmt.Sprintf("focus:%d", v.ID))
612621 case ports.SetModifiers:
613622 if v.Restore {
614- out = append(out, "restore")
623+ out = append(out, "mods:restore")
615624 } else {
616625 out = append(out, fmt.Sprintf("mods:%d", v.Mods))
617626 }
618- case ports.FocusWindow:
619- out = append(out, fmt.Sprintf("focus:%d", v.ID))
620627 case ports.PointerMotionTo:
621628 out = append(out, "motion")
622629 case ports.PointerButtonTo:
@@ -630,97 +637,25 @@ func commandKinds(cmds []ports.ClientCommand) []string {
630637 return out
631638 }
632639
633-// An OCR target is clicked with the configured modifiers held: they go out
634-// before the motion and are restored after the release.
635-func TestHintTextClickHoldsModifiers(t *testing.T) {
636- mods := ports.ModCtrl | ports.ModShift
637- h := newHintMods(t, core.Options{}, mods)
638- tg := h.inside(0)
639- tg.Text = true
640- h.show(tg)
640+// A click on the focused window moves the pointer to the target and presses
641+// and releases the left button; it states no modifiers.
642+func TestHintClickSequence(t *testing.T) {
643+ h := newHint(t, core.Options{})
644+ h.show(h.inside(0))
641645 h.drain()
642646 h.press("f", 0)
643647 got := commandKinds(h.commandsSent())
644- want := []string{fmt.Sprintf("mods:%d", mods), "motion", "press", "release", "restore"}
648+ want := []string{"mods:0", "motion", "press", "release", "mods:restore"}
645649 if !slices.Equal(got, want) {
646650 t.Fatalf("commands %v, want %v", got, want)
647651 }
648652 }
649653
650-// Without text targets or without modifiers the click states no modifiers
651-// (depressed 0) rather than leave the held bind modifier to the client.
652-func TestHintClickWithoutModifiers(t *testing.T) {
653- for name, c := range map[string]struct {
654- mods ports.Mods
655- text bool
656- }{"accessibility target": {ports.ModCtrl, false}, "none configured": {0, true}} {
657- t.Run(name, func(t *testing.T) {
658- h := newHintMods(t, core.Options{}, c.mods)
659- tg := h.inside(0)
660- tg.Text = c.text
661- h.show(tg)
662- h.drain()
663- h.press("f", 0)
664- got := commandKinds(h.commandsSent())
665- if want := []string{"mods:0", "motion", "press", "release", "restore"}; !slices.Equal(got, want) {
666- t.Fatalf("commands %v, want %v", got, want)
667- }
668- })
669- }
670-}
671-
672-// The bind's Super is still physically held: an accessibility click states
673-// depressed 0, and the restore carries the real state of the latest key,
674-// swallowed ones included (a latch a label key consumed).
675-func TestHintClickHeldBindModifierAndLatchRestore(t *testing.T) {
676- h := newHint(t, core.Options{})
677- tg := h.inside(0)
678- h.press("g", ports.ModSuper)
679- req := receive(t, h.scans)
680- h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{tg, h.inside(1)}}
681- h.labelled(2, 0)
682- h.drain()
683- real := ports.ModState{Depressed: 64, Latched: 0, Locked: 2}
684- h.send(ports.KeyEvent{Keysym: "f", Base: "f", Mods: ports.ModSuper, Pressed: true, State: real})
685- cmds := h.commandsSent()
686- var restore *ports.SetModifiers
687- for i, c := range cmds {
688- if v, ok := c.(ports.SetModifiers); ok {
689- if i == 0 && (v.Restore || v.Mods != 0) {
690- t.Fatalf("first modifiers %+v, want depressed 0", v)
691- }
692- if v.Restore {
693- restore = &v
694- }
695- }
696- }
697- if restore == nil || restore.State != real {
698- t.Fatalf("restore %+v, want state %+v (commands %v)", restore, real, commandKinds(cmds))
699- }
700-}
701-
702-// A result is kept only for the window it was scanned on.
703-func TestHintResultWindowChecks(t *testing.T) {
704- h := newHint(t, core.Options{})
705- req := h.start()
706- // Accessibility target naming another window: dropped; naming window 1: kept.
707- ok := h.inside(1)
708- ok.Window = 1
709- bad := h.inside(0)
710- bad.Window = 7
711- h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{bad, ok}}
712- s := h.labelled(1, 0)
713- if len(s.HintLabels) != 1 || s.HintLabels[0].Rect.Y != ok.Rect.Y {
714- t.Fatalf("labels %+v", s.HintLabels)
715- }
716-}
717-
718-// A window that moved while the scan ran: an OCR target over it is dropped.
719-func TestHintOCRResultDroppedWhenWindowMoved(t *testing.T) {
654+// A window that moved while the scan ran: a target over it is dropped.
655+func TestHintResultDroppedWhenWindowMoved(t *testing.T) {
720656 h := newHint(t, core.Options{})
721657 req := h.start()
722658 tg := h.inside(0)
723- tg.Text = true
724659 h.client <- ports.WindowMapped{ID: 2}
725660 h.until(func(s ports.Scene) bool { return len(s.Windows) == 2 })
726661 h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{tg}}
@@ -731,12 +666,12 @@ func TestHintOCRResultDroppedWhenWindowMoved(t *testing.T) {
731666 }
732667
733668 // A new window covering the target (not in the snapshot) drops it as well.
734-func TestHintOCRResultDroppedForUnseenWindow(t *testing.T) {
669+func TestHintResultDroppedForUnseenWindow(t *testing.T) {
735670 h := newHint(t, core.Options{})
736671 req := h.start()
737672 h.client <- ports.WindowMapped{ID: 2}
738673 s := h.until(func(s ports.Scene) bool { return len(s.Windows) == 2 })
739- tg := ports.HintTarget{Text: true}
674+ var tg ports.HintTarget
740675 for _, w := range s.Windows {
741676 if w.ID == 2 {
742677 tg.Rect = ports.Rect{X: w.Rect.X + 5, Y: w.Rect.Y + 5, W: 20, H: 10}
@@ -769,9 +704,9 @@ func TestHintCancelsWhenOutputChanges(t *testing.T) {
769704 }
770705 }
771706
772-// A window that does not have the keyboard gets it before the modifiers.
773-func TestHintTextClickFocusesWindowFirst(t *testing.T) {
774- h := newHintMods(t, core.Options{}, ports.ModCtrl)
707+// A window that does not have the keyboard gets it before the click.
708+func TestHintClickFocusesWindowFirst(t *testing.T) {
709+ h := newHint(t, core.Options{})
775710 h.client <- ports.WindowMapped{ID: 2}
776711 s := h.until(func(s ports.Scene) bool { return len(s.Windows) == 2 })
777712 var r ports.Rect
@@ -781,24 +716,20 @@ func TestHintTextClickFocusesWindowFirst(t *testing.T) {
781716 }
782717 }
783718 h.win = r
784- tg := h.inside(0)
785- tg.Text = true
786- h.show(tg)
719+ h.show(h.inside(0))
787720 h.drain()
788721 h.press("f", 0)
789722 got := commandKinds(h.commandsSent())
790- want := []string{"focus:1", fmt.Sprintf("mods:%d", ports.ModCtrl), "motion", "press", "release", "restore"}
723+ want := []string{"focus:1", "mods:0", "motion", "press", "release", "mods:restore"}
791724 if !slices.Equal(got, want) {
792725 t.Fatalf("commands %v, want %v", got, want)
793726 }
794727 }
795728
796-// A click that is no longer allowed sends no modifiers at all.
797-func TestHintTextClickNoModifiersWhenNotClickable(t *testing.T) {
798- h := newHintMods(t, core.Options{}, ports.ModCtrl)
799- tg := h.inside(0)
800- tg.Text = true
801- h.show(tg)
729+// A click that is no longer allowed sends no command at all.
730+func TestHintClickNothingWhenNotClickable(t *testing.T) {
731+ h := newHint(t, core.Options{})
732+ h.show(h.inside(0))
802733 h.constrain(ports.ConstraintLock, ports.Rect{})
803734 h.drain()
804735 h.press("f", 0)
@@ -842,15 +773,11 @@ func TestHintDisabledNotesOnce(t *testing.T) {
842773 // Resizing window 1 leaves the old target centre inside it. Identity alone
843774 // therefore cannot protect either admission or the eventual synthetic click.
844775 func TestHintRejectsResizedSource(t *testing.T) {
845- for _, accessibility := range []bool{false, true} {
776+ {
846777 for _, beforeResult := range []bool{false, true} {
847- t.Run(fmt.Sprintf("accessibility=%v/beforeResult=%v", accessibility, beforeResult), func(t *testing.T) {
778+ t.Run(fmt.Sprintf("beforeResult=%v", beforeResult), func(t *testing.T) {
848779 h := newHint(t, core.Options{})
849780 tg := h.inside(0)
850- tg.Text = !accessibility
851- if accessibility {
852- tg.Window = 1
853- }
854781 req := h.start()
855782 if !beforeResult {
856783 h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{tg}}
@@ -895,3 +822,44 @@ func TestHintCaptureReachesScene(t *testing.T) {
895822 }
896823 }
897824 }
825+
826+// Labels hang off the targets' top-left corners: targets whose corners are
827+// closer than a label drop, the lowest score first, whatever the input order.
828+func TestHintOverlappingLabelsDropLowerScore(t *testing.T) {
829+ at := func(h *hintHarness, dx, dy int, score float32) ports.HintTarget {
830+ return ports.HintTarget{Rect: ports.Rect{X: h.win.X + 20 + dx, Y: h.win.Y + 20 + dy, W: 80, H: 30}, Score: score}
831+ }
832+ for _, rev := range []bool{false, true} {
833+ h := newHint(t, core.Options{})
834+ ts := []ports.HintTarget{
835+ at(h, 0, 0, 0.4), // clashes with the next one, loses
836+ at(h, 3, 2, 0.9), // wins
837+ at(h, 300, 0, 0.1), // far away: kept, whatever its score
838+ at(h, 0, 200, 0.5), // far below
839+ at(h, 4, 200, 0.45), // clashes with the previous one
840+ }
841+ if rev {
842+ slices.Reverse(ts)
843+ }
844+ req := h.start()
845+ h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: ts}
846+ s := h.labelled(3, 0)
847+ var got [][2]int
848+ for _, l := range s.HintLabels {
849+ got = append(got, [2]int{l.Rect.X - h.win.X - 20, l.Rect.Y - h.win.Y - 20})
850+ }
851+ want := [][2]int{{300, 0}, {3, 2}, {0, 200}} // sorted by Y, then X
852+ if !slices.Equal(got, want) {
853+ t.Fatalf("rev=%v: label corners %v, want %v", rev, got, want)
854+ }
855+ }
856+}
857+
858+func TestHintSizeReloadCancelsShownHints(t *testing.T) {
859+ h := newHint(t, core.Options{})
860+ h.show(h.inside(0))
861+ cfg := h.cfg
862+ cfg.Hint.Size = 40
863+ h.config <- ports.ConfigChanged{Config: cfg}
864+ h.until(func(s ports.Scene) bool { return s.HintStyle.Size == 40 && len(s.HintLabels) == 0 })
865+}
@@ -24,19 +24,15 @@ type hintHarness struct {
24 hints chan ports.HintScanResult24 hints chan ports.HintScanResult
25 win ports.Rect // window 1, output-local25 win ports.Rect // window 1, output-local
26 // state is the security state input is stamped with.26 // state is the security state input is stamped with.
27- state ports.SecurityState27+ state ports.SecurityState
28+ config chan ports.ConfigChanged
29+ cfg ports.Config
28 }30 }
29 31
30 // newHint starts a core with one 1000x800 output "OUT-1" and window 1 mapped.32 // newHint starts a core with one 1000x800 output "OUT-1" and window 1 mapped.
31 func newHint(t *testing.T, opts core.Options) *hintHarness {33 func newHint(t *testing.T, opts core.Options) *hintHarness {
32 t.Helper()34 t.Helper()
33- return newHintMods(t, opts, 0)35+ return newHintConfig(t, opts, func(*ports.Config) {})
34-}
35-
36-// newHintMods is newHint with hint.text-click set to textClick.
37-func newHintMods(t *testing.T, opts core.Options, textClick ports.Mods) *hintHarness {
38- t.Helper()
39- return newHintConfig(t, opts, func(cfg *ports.Config) { cfg.Hint.TextClick = textClick })
40 }36 }
41 37
42 // newHintConfig is newHint with the config edited by edit.38 // newHintConfig is newHint with the config edited by edit.
@@ -48,6 +44,8 @@ func newHintConfig(t *testing.T, opts core.Options, edit func(*ports.Config)) *h
48 cfg.Layout.Gaps = 2044 cfg.Layout.Gaps = 20
49 cfg.Binds["Cmd+g"] = "hint"45 cfg.Binds["Cmd+g"] = "hint"
50 h := &hintHarness{46 h := &hintHarness{
47+ config: make(chan ports.ConfigChanged),
48+ cfg: cfg,
51 t: t,49 t: t,
52 input: make(chan ports.InputEvent, 32),50 input: make(chan ports.InputEvent, 32),
53 client: make(chan ports.ClientEvent, 32),51 client: make(chan ports.ClientEvent, 32),
@@ -58,7 +56,7 @@ func newHintConfig(t *testing.T, opts core.Options, edit func(*ports.Config)) *h
58 // Unbuffered: a send returns once core took the result.56 // Unbuffered: a send returns once core took the result.
59 hints: make(chan ports.HintScanResult),57 hints: make(chan ports.HintScanResult),
60 }58 }
61- ch := core.Channels{Client: h.client, Input: h.input, Output: h.output, Commands: h.commands, Scenes: h.scenes, HintScans: h.scans, Hints: h.hints}59+ ch := core.Channels{Config: h.config, Client: h.client, Input: h.input, Output: h.output, Commands: h.commands, Scenes: h.scenes, HintScans: h.scans, Hints: h.hints}
62 c, err := core.New(cfg, ch, opts)60 c, err := core.New(cfg, ch, opts)
63 if err != nil {61 if err != nil {
64 t.Fatal(err)62 t.Fatal(err)
@@ -138,19 +136,15 @@ func (h *hintHarness) labelled(n, typed int) ports.Scene {
138 136
139 // inside is a target of the given size inside window 1.137 // inside is a target of the given size inside window 1.
140 func (h *hintHarness) inside(i int) ports.HintTarget {138 func (h *hintHarness) inside(i int) ports.HintTarget {
141- return ports.HintTarget{Rect: ports.Rect{X: h.win.X + 10 + i, Y: h.win.Y + 10 + 12*i, W: 40, H: 10}}139+ return ports.HintTarget{Rect: ports.Rect{X: h.win.X + 10 + 100*(i%4), Y: h.win.Y + 10 + 40*(i/4), W: 40, H: 10}}
142 }140 }
143 141
144 func TestHintRequestOnBind(t *testing.T) {142 func TestHintRequestOnBind(t *testing.T) {
145 h := newHint(t, core.Options{})143 h := newHint(t, core.Options{})
146 req := h.start()144 req := h.start()
147- if req.Output != "OUT-1" || req.Width != 1000 || req.Height != 800 || req.Scale != 1 || req.LinksOnly || req.Seq == 0 {145+ if req.Output != "OUT-1" || req.Width != 1000 || req.Height != 800 || req.Scale != 1 || req.Seq == 0 {
148 t.Fatalf("request %+v", req)146 t.Fatalf("request %+v", req)
149 }147 }
150- want := []ports.HintWindow{{ID: 1, PID: 4242, AppID: "org.test.app", Rect: h.win, Focused: true}}
151- if !slices.Equal(req.Windows, want) {
152- t.Fatalf("windows %+v, want %+v", req.Windows, want)
153- }
154 // A second bind while pending does not ask again; the pending phase148 // A second bind while pending does not ask again; the pending phase
155 // cancels on any key.149 // cancels on any key.
156 h.press("g", ports.ModSuper)150 h.press("g", ports.ModSuper)
@@ -236,7 +230,7 @@ func TestHintTwoLetterLabels(t *testing.T) {
236 h := newHint(t, core.Options{})230 h := newHint(t, core.Options{})
237 var targets []ports.HintTarget231 var targets []ports.HintTarget
238 for i := 0; i < 27; i++ {232 for i := 0; i < 27; i++ {
239- targets = append(targets, ports.HintTarget{Rect: ports.Rect{X: h.win.X + 5, Y: h.win.Y + 5 + 14*i, W: 30, H: 10}})233+ targets = append(targets, ports.HintTarget{Rect: ports.Rect{X: h.win.X + 5 + 100*(i%4), Y: h.win.Y + 5 + 40*(i/4), W: 30, H: 10}})
240 }234 }
241 s := h.show(targets...)235 s := h.show(targets...)
242 if len(s.HintLabels) != 27 {236 if len(s.HintLabels) != 27 {
@@ -274,7 +268,7 @@ func TestHintTypingFilters(t *testing.T) {
274 h := newHint(t, core.Options{})268 h := newHint(t, core.Options{})
275 var targets []ports.HintTarget269 var targets []ports.HintTarget
276 for i := 0; i < 27; i++ {270 for i := 0; i < 27; i++ {
277- targets = append(targets, ports.HintTarget{Rect: ports.Rect{X: h.win.X + 5, Y: h.win.Y + 5 + 14*i, W: 30, H: 10}})271+ targets = append(targets, ports.HintTarget{Rect: ports.Rect{X: h.win.X + 5 + 100*(i%4), Y: h.win.Y + 5 + 40*(i/4), W: 30, H: 10}})
278 }272 }
279 h.show(targets...)273 h.show(targets...)
280 h.press("f", 0)274 h.press("f", 0)
@@ -472,7 +466,20 @@ func TestHintClickWithBindModifierHeld(t *testing.T) {
472 h.drain()466 h.drain()
473 h.press("f", ports.ModSuper) // Super is still down467 h.press("f", ports.ModSuper) // Super is still down
474 h.labelled(0, 0)468 h.labelled(0, 0)
475- presses, releases := h.buttonsSent()469+ cmds := h.barrier()
470+ if got, want := commandKinds(cmds), []string{"mods:0", "motion", "press", "release", "mods:restore"}; !slices.Equal(got, want) {
471+ t.Fatalf("commands %v, want %v", got, want)
472+ }
473+ presses, releases := 0, 0
474+ for _, c := range cmds {
475+ if b, ok := c.(ports.PointerButtonTo); ok {
476+ if b.Pressed {
477+ presses++
478+ } else {
479+ releases++
480+ }
481+ }
482+ }
476 if presses != 1 || releases != 1 {483 if presses != 1 || releases != 1 {
477 t.Fatalf("client got %d presses, %d releases", presses, releases)484 t.Fatalf("client got %d presses, %d releases", presses, releases)
478 }485 }
@@ -593,12 +600,12 @@ func TestHintMappedWindowMovedAborts(t *testing.T) {
593 }600 }
594 601
595 // commandsSent drains the commands up to a barrier, keeping the ones that602 // commandsSent drains the commands up to a barrier, keeping the ones that
596-// matter for a click: modifiers, focus, motion and buttons.603+// matter for a click: focus, motion and buttons.
597 func (h *hintHarness) commandsSent() []ports.ClientCommand {604 func (h *hintHarness) commandsSent() []ports.ClientCommand {
598 var out []ports.ClientCommand605 var out []ports.ClientCommand
599 for _, cmd := range h.barrier() {606 for _, cmd := range h.barrier() {
600 switch cmd.(type) {607 switch cmd.(type) {
601- case ports.SetModifiers, ports.FocusWindow, ports.PointerMotionTo, ports.PointerButtonTo:608+ case ports.FocusWindow, ports.SetModifiers, ports.PointerMotionTo, ports.PointerButtonTo:
602 out = append(out, cmd)609 out = append(out, cmd)
603 }610 }
604 }611 }
@@ -609,14 +616,14 @@ func commandKinds(cmds []ports.ClientCommand) []string {
609 var out []string616 var out []string
610 for _, c := range cmds {617 for _, c := range cmds {
611 switch v := c.(type) {618 switch v := c.(type) {
619+ case ports.FocusWindow:
620+ out = append(out, fmt.Sprintf("focus:%d", v.ID))
612 case ports.SetModifiers:621 case ports.SetModifiers:
613 if v.Restore {622 if v.Restore {
614- out = append(out, "restore")623+ out = append(out, "mods:restore")
615 } else {624 } else {
616 out = append(out, fmt.Sprintf("mods:%d", v.Mods))625 out = append(out, fmt.Sprintf("mods:%d", v.Mods))
617 }626 }
618- case ports.FocusWindow:
619- out = append(out, fmt.Sprintf("focus:%d", v.ID))
620 case ports.PointerMotionTo:627 case ports.PointerMotionTo:
621 out = append(out, "motion")628 out = append(out, "motion")
622 case ports.PointerButtonTo:629 case ports.PointerButtonTo:
@@ -630,97 +637,25 @@ func commandKinds(cmds []ports.ClientCommand) []string {
630 return out637 return out
631 }638 }
632 639
633-// An OCR target is clicked with the configured modifiers held: they go out640+// A click on the focused window moves the pointer to the target and presses
634-// before the motion and are restored after the release.641+// and releases the left button; it states no modifiers.
635-func TestHintTextClickHoldsModifiers(t *testing.T) {642+func TestHintClickSequence(t *testing.T) {
636- mods := ports.ModCtrl | ports.ModShift643+ h := newHint(t, core.Options{})
637- h := newHintMods(t, core.Options{}, mods)644+ h.show(h.inside(0))
638- tg := h.inside(0)
639- tg.Text = true
640- h.show(tg)
641 h.drain()645 h.drain()
642 h.press("f", 0)646 h.press("f", 0)
643 got := commandKinds(h.commandsSent())647 got := commandKinds(h.commandsSent())
644- want := []string{fmt.Sprintf("mods:%d", mods), "motion", "press", "release", "restore"}648+ want := []string{"mods:0", "motion", "press", "release", "mods:restore"}
645 if !slices.Equal(got, want) {649 if !slices.Equal(got, want) {
646 t.Fatalf("commands %v, want %v", got, want)650 t.Fatalf("commands %v, want %v", got, want)
647 }651 }
648 }652 }
649 653
650-// Without text targets or without modifiers the click states no modifiers654+// A window that moved while the scan ran: a target over it is dropped.
651-// (depressed 0) rather than leave the held bind modifier to the client.655+func TestHintResultDroppedWhenWindowMoved(t *testing.T) {
652-func TestHintClickWithoutModifiers(t *testing.T) {
653- for name, c := range map[string]struct {
654- mods ports.Mods
655- text bool
656- }{"accessibility target": {ports.ModCtrl, false}, "none configured": {0, true}} {
657- t.Run(name, func(t *testing.T) {
658- h := newHintMods(t, core.Options{}, c.mods)
659- tg := h.inside(0)
660- tg.Text = c.text
661- h.show(tg)
662- h.drain()
663- h.press("f", 0)
664- got := commandKinds(h.commandsSent())
665- if want := []string{"mods:0", "motion", "press", "release", "restore"}; !slices.Equal(got, want) {
666- t.Fatalf("commands %v, want %v", got, want)
667- }
668- })
669- }
670-}
671-
672-// The bind's Super is still physically held: an accessibility click states
673-// depressed 0, and the restore carries the real state of the latest key,
674-// swallowed ones included (a latch a label key consumed).
675-func TestHintClickHeldBindModifierAndLatchRestore(t *testing.T) {
676- h := newHint(t, core.Options{})
677- tg := h.inside(0)
678- h.press("g", ports.ModSuper)
679- req := receive(t, h.scans)
680- h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{tg, h.inside(1)}}
681- h.labelled(2, 0)
682- h.drain()
683- real := ports.ModState{Depressed: 64, Latched: 0, Locked: 2}
684- h.send(ports.KeyEvent{Keysym: "f", Base: "f", Mods: ports.ModSuper, Pressed: true, State: real})
685- cmds := h.commandsSent()
686- var restore *ports.SetModifiers
687- for i, c := range cmds {
688- if v, ok := c.(ports.SetModifiers); ok {
689- if i == 0 && (v.Restore || v.Mods != 0) {
690- t.Fatalf("first modifiers %+v, want depressed 0", v)
691- }
692- if v.Restore {
693- restore = &v
694- }
695- }
696- }
697- if restore == nil || restore.State != real {
698- t.Fatalf("restore %+v, want state %+v (commands %v)", restore, real, commandKinds(cmds))
699- }
700-}
701-
702-// A result is kept only for the window it was scanned on.
703-func TestHintResultWindowChecks(t *testing.T) {
704- h := newHint(t, core.Options{})
705- req := h.start()
706- // Accessibility target naming another window: dropped; naming window 1: kept.
707- ok := h.inside(1)
708- ok.Window = 1
709- bad := h.inside(0)
710- bad.Window = 7
711- h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{bad, ok}}
712- s := h.labelled(1, 0)
713- if len(s.HintLabels) != 1 || s.HintLabels[0].Rect.Y != ok.Rect.Y {
714- t.Fatalf("labels %+v", s.HintLabels)
715- }
716-}
717-
718-// A window that moved while the scan ran: an OCR target over it is dropped.
719-func TestHintOCRResultDroppedWhenWindowMoved(t *testing.T) {
720 h := newHint(t, core.Options{})656 h := newHint(t, core.Options{})
721 req := h.start()657 req := h.start()
722 tg := h.inside(0)658 tg := h.inside(0)
723- tg.Text = true
724 h.client <- ports.WindowMapped{ID: 2}659 h.client <- ports.WindowMapped{ID: 2}
725 h.until(func(s ports.Scene) bool { return len(s.Windows) == 2 })660 h.until(func(s ports.Scene) bool { return len(s.Windows) == 2 })
726 h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{tg}}661 h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{tg}}
@@ -731,12 +666,12 @@ func TestHintOCRResultDroppedWhenWindowMoved(t *testing.T) {
731 }666 }
732 667
733 // A new window covering the target (not in the snapshot) drops it as well.668 // A new window covering the target (not in the snapshot) drops it as well.
734-func TestHintOCRResultDroppedForUnseenWindow(t *testing.T) {669+func TestHintResultDroppedForUnseenWindow(t *testing.T) {
735 h := newHint(t, core.Options{})670 h := newHint(t, core.Options{})
736 req := h.start()671 req := h.start()
737 h.client <- ports.WindowMapped{ID: 2}672 h.client <- ports.WindowMapped{ID: 2}
738 s := h.until(func(s ports.Scene) bool { return len(s.Windows) == 2 })673 s := h.until(func(s ports.Scene) bool { return len(s.Windows) == 2 })
739- tg := ports.HintTarget{Text: true}674+ var tg ports.HintTarget
740 for _, w := range s.Windows {675 for _, w := range s.Windows {
741 if w.ID == 2 {676 if w.ID == 2 {
742 tg.Rect = ports.Rect{X: w.Rect.X + 5, Y: w.Rect.Y + 5, W: 20, H: 10}677 tg.Rect = ports.Rect{X: w.Rect.X + 5, Y: w.Rect.Y + 5, W: 20, H: 10}
@@ -769,9 +704,9 @@ func TestHintCancelsWhenOutputChanges(t *testing.T) {
769 }704 }
770 }705 }
771 706
772-// A window that does not have the keyboard gets it before the modifiers.707+// A window that does not have the keyboard gets it before the click.
773-func TestHintTextClickFocusesWindowFirst(t *testing.T) {708+func TestHintClickFocusesWindowFirst(t *testing.T) {
774- h := newHintMods(t, core.Options{}, ports.ModCtrl)709+ h := newHint(t, core.Options{})
775 h.client <- ports.WindowMapped{ID: 2}710 h.client <- ports.WindowMapped{ID: 2}
776 s := h.until(func(s ports.Scene) bool { return len(s.Windows) == 2 })711 s := h.until(func(s ports.Scene) bool { return len(s.Windows) == 2 })
777 var r ports.Rect712 var r ports.Rect
@@ -781,24 +716,20 @@ func TestHintTextClickFocusesWindowFirst(t *testing.T) {
781 }716 }
782 }717 }
783 h.win = r718 h.win = r
784- tg := h.inside(0)719+ h.show(h.inside(0))
785- tg.Text = true
786- h.show(tg)
787 h.drain()720 h.drain()
788 h.press("f", 0)721 h.press("f", 0)
789 got := commandKinds(h.commandsSent())722 got := commandKinds(h.commandsSent())
790- want := []string{"focus:1", fmt.Sprintf("mods:%d", ports.ModCtrl), "motion", "press", "release", "restore"}723+ want := []string{"focus:1", "mods:0", "motion", "press", "release", "mods:restore"}
791 if !slices.Equal(got, want) {724 if !slices.Equal(got, want) {
792 t.Fatalf("commands %v, want %v", got, want)725 t.Fatalf("commands %v, want %v", got, want)
793 }726 }
794 }727 }
795 728
796-// A click that is no longer allowed sends no modifiers at all.729+// A click that is no longer allowed sends no command at all.
797-func TestHintTextClickNoModifiersWhenNotClickable(t *testing.T) {730+func TestHintClickNothingWhenNotClickable(t *testing.T) {
798- h := newHintMods(t, core.Options{}, ports.ModCtrl)731+ h := newHint(t, core.Options{})
799- tg := h.inside(0)732+ h.show(h.inside(0))
800- tg.Text = true
801- h.show(tg)
802 h.constrain(ports.ConstraintLock, ports.Rect{})733 h.constrain(ports.ConstraintLock, ports.Rect{})
803 h.drain()734 h.drain()
804 h.press("f", 0)735 h.press("f", 0)
@@ -842,15 +773,11 @@ func TestHintDisabledNotesOnce(t *testing.T) {
842 // Resizing window 1 leaves the old target centre inside it. Identity alone773 // Resizing window 1 leaves the old target centre inside it. Identity alone
843 // therefore cannot protect either admission or the eventual synthetic click.774 // therefore cannot protect either admission or the eventual synthetic click.
844 func TestHintRejectsResizedSource(t *testing.T) {775 func TestHintRejectsResizedSource(t *testing.T) {
845- for _, accessibility := range []bool{false, true} {776+ {
846 for _, beforeResult := range []bool{false, true} {777 for _, beforeResult := range []bool{false, true} {
847- t.Run(fmt.Sprintf("accessibility=%v/beforeResult=%v", accessibility, beforeResult), func(t *testing.T) {778+ t.Run(fmt.Sprintf("beforeResult=%v", beforeResult), func(t *testing.T) {
848 h := newHint(t, core.Options{})779 h := newHint(t, core.Options{})
849 tg := h.inside(0)780 tg := h.inside(0)
850- tg.Text = !accessibility
851- if accessibility {
852- tg.Window = 1
853- }
854 req := h.start()781 req := h.start()
855 if !beforeResult {782 if !beforeResult {
856 h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{tg}}783 h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: []ports.HintTarget{tg}}
@@ -895,3 +822,44 @@ func TestHintCaptureReachesScene(t *testing.T) {
895 }822 }
896 }823 }
897 }824 }
825+
826+// Labels hang off the targets' top-left corners: targets whose corners are
827+// closer than a label drop, the lowest score first, whatever the input order.
828+func TestHintOverlappingLabelsDropLowerScore(t *testing.T) {
829+ at := func(h *hintHarness, dx, dy int, score float32) ports.HintTarget {
830+ return ports.HintTarget{Rect: ports.Rect{X: h.win.X + 20 + dx, Y: h.win.Y + 20 + dy, W: 80, H: 30}, Score: score}
831+ }
832+ for _, rev := range []bool{false, true} {
833+ h := newHint(t, core.Options{})
834+ ts := []ports.HintTarget{
835+ at(h, 0, 0, 0.4), // clashes with the next one, loses
836+ at(h, 3, 2, 0.9), // wins
837+ at(h, 300, 0, 0.1), // far away: kept, whatever its score
838+ at(h, 0, 200, 0.5), // far below
839+ at(h, 4, 200, 0.45), // clashes with the previous one
840+ }
841+ if rev {
842+ slices.Reverse(ts)
843+ }
844+ req := h.start()
845+ h.hints <- ports.HintScanResult{Seq: req.Seq, Output: req.Output, Targets: ts}
846+ s := h.labelled(3, 0)
847+ var got [][2]int
848+ for _, l := range s.HintLabels {
849+ got = append(got, [2]int{l.Rect.X - h.win.X - 20, l.Rect.Y - h.win.Y - 20})
850+ }
851+ want := [][2]int{{300, 0}, {3, 2}, {0, 200}} // sorted by Y, then X
852+ if !slices.Equal(got, want) {
853+ t.Fatalf("rev=%v: label corners %v, want %v", rev, got, want)
854+ }
855+ }
856+}
857+
858+func TestHintSizeReloadCancelsShownHints(t *testing.T) {
859+ h := newHint(t, core.Options{})
860+ h.show(h.inside(0))
861+ cfg := h.cfg
862+ cfg.Hint.Size = 40
863+ h.config <- ports.ConfigChanged{Config: cfg}
864+ h.until(func(s ports.Scene) bool { return s.HintStyle.Size == 40 && len(s.HintLabels) == 0 })
865+}
M internal/ports/config.go +8 -6
@@ -170,17 +170,19 @@ type Config struct {
170170 // Enable turns the experimental hint mode on (hint.enable). It is
171171 // read at startup: a reload keeps the startup value.
172172 Enable bool
173- // LinksOnly limits the targets to links and URLs; false also
174- // includes buttons (hint.targets = clickable).
175- LinksOnly bool
176- // TextClick are the modifiers held for a click on a link found by
177- // OCR (hint.text-click); zero sends a plain click.
178- TextClick Mods
179173 // Capture shows the labels in the captures requested by Wayland
180174 // clients (hint.capture); the compositor's own scans never see them.
181175 Capture bool
182176 Size float64 // logical pixels; zero selects automatic density sizing
183177 Background, Foreground string // empty derives from Border.Active
178+ // Detector is the nefer-hints binary (hint.detector): a path, or a
179+ // name looked up in PATH. Model is the directory of its model files
180+ // (hint.model), with a leading ~/ already expanded. Threshold is the
181+ // minimum detection score (hint.threshold, 0.01 to 0.99); zero leaves
182+ // it unset, using model.json or the runner default. All three
183+ // are read at startup.
184+ Detector, Model string
185+ Threshold float64
184186 }
185187 // Workspaces are declared with workspace.<name>.* keys, in first-seen order.
186188 Workspaces []WorkspaceConfig
@@ -170,17 +170,19 @@ type Config struct {
170 // Enable turns the experimental hint mode on (hint.enable). It is170 // Enable turns the experimental hint mode on (hint.enable). It is
171 // read at startup: a reload keeps the startup value.171 // read at startup: a reload keeps the startup value.
172 Enable bool172 Enable bool
173- // LinksOnly limits the targets to links and URLs; false also
174- // includes buttons (hint.targets = clickable).
175- LinksOnly bool
176- // TextClick are the modifiers held for a click on a link found by
177- // OCR (hint.text-click); zero sends a plain click.
178- TextClick Mods
179 // Capture shows the labels in the captures requested by Wayland173 // Capture shows the labels in the captures requested by Wayland
180 // clients (hint.capture); the compositor's own scans never see them.174 // clients (hint.capture); the compositor's own scans never see them.
181 Capture bool175 Capture bool
182 Size float64 // logical pixels; zero selects automatic density sizing176 Size float64 // logical pixels; zero selects automatic density sizing
183 Background, Foreground string // empty derives from Border.Active177 Background, Foreground string // empty derives from Border.Active
178+ // Detector is the nefer-hints binary (hint.detector): a path, or a
179+ // name looked up in PATH. Model is the directory of its model files
180+ // (hint.model), with a leading ~/ already expanded. Threshold is the
181+ // minimum detection score (hint.threshold, 0.01 to 0.99); zero leaves
182+ // it unset, using model.json or the runner default. All three
183+ // are read at startup.
184+ Detector, Model string
185+ Threshold float64
184 }186 }
185 // Workspaces are declared with workspace.<name>.* keys, in first-seen order.187 // Workspaces are declared with workspace.<name>.* keys, in first-seen order.
186 Workspaces []WorkspaceConfig188 Workspaces []WorkspaceConfig
M internal/ports/hint_messages.go +2 -31
@@ -1,15 +1,5 @@
11 package ports
22
3-// HintKind is the kind of a clickable hint target.
4-type HintKind uint8
5-
6-const (
7- // HintLink is a link or URL.
8- HintLink HintKind = iota
9- // HintButton is a button or control.
10- HintButton
11-)
12-
133 // HintScanRequest asks the hint scanner to find clickable targets in the last
144 // frame of Output. Core sends it and expects one HintScanResult echoing Seq.
155 type HintScanRequest struct {
@@ -19,31 +9,12 @@ type HintScanRequest struct {
199 // (OutputInfo.Width/Height).
2010 Width, Height int
2111 Scale float64 // screen scale
22- LinksOnly bool
23- // Windows are the windows visible on Output, for sources that read a
24- // toolkit's accessibility tree. Core owns the slice per request.
25- Windows []HintWindow
26-}
27-
28-// HintWindow is a window visible on the hint output.
29-type HintWindow struct {
30- ID WindowID
31- PID int
32- AppID string
33- // Rect is the client area, logical output-local.
34- Rect Rect
35- Focused bool
3612 }
3713
3814 // HintTarget is a clickable region found in a frame.
3915 type HintTarget struct {
40- Rect Rect // logical, output-local
41- Kind HintKind
42- // Text is set for targets found by OCR (read from pixels, not from an
43- // accessibility tree).
44- Text bool
45- // Window is the source window of an accessibility target, 0 for OCR.
46- Window WindowID
16+ Rect Rect // logical, output-local
17+ Score float32 // detector confidence, 0 to 1
4718 }
4819
4920 // HintScanResult answers a HintScanRequest. Core owns Targets.
@@ -1,15 +1,5 @@
1 package ports1 package ports
2 2
3-// HintKind is the kind of a clickable hint target.
4-type HintKind uint8
5-
6-const (
7- // HintLink is a link or URL.
8- HintLink HintKind = iota
9- // HintButton is a button or control.
10- HintButton
11-)
12-
13 // HintScanRequest asks the hint scanner to find clickable targets in the last3 // HintScanRequest asks the hint scanner to find clickable targets in the last
14 // frame of Output. Core sends it and expects one HintScanResult echoing Seq.4 // frame of Output. Core sends it and expects one HintScanResult echoing Seq.
15 type HintScanRequest struct {5 type HintScanRequest struct {
@@ -19,31 +9,12 @@ type HintScanRequest struct {
19 // (OutputInfo.Width/Height).9 // (OutputInfo.Width/Height).
20 Width, Height int10 Width, Height int
21 Scale float64 // screen scale11 Scale float64 // screen scale
22- LinksOnly bool
23- // Windows are the windows visible on Output, for sources that read a
24- // toolkit's accessibility tree. Core owns the slice per request.
25- Windows []HintWindow
26-}
27-
28-// HintWindow is a window visible on the hint output.
29-type HintWindow struct {
30- ID WindowID
31- PID int
32- AppID string
33- // Rect is the client area, logical output-local.
34- Rect Rect
35- Focused bool
36 }12 }
37 13
38 // HintTarget is a clickable region found in a frame.14 // HintTarget is a clickable region found in a frame.
39 type HintTarget struct {15 type HintTarget struct {
40- Rect Rect // logical, output-local16+ Rect Rect // logical, output-local
41- Kind HintKind17+ Score float32 // detector confidence, 0 to 1
42- // Text is set for targets found by OCR (read from pixels, not from an
43- // accessibility tree).
44- Text bool
45- // Window is the source window of an accessibility target, 0 for OCR.
46- Window WindowID
47 }18 }
48 19
49 // HintScanResult answers a HintScanRequest. Core owns Targets.20 // HintScanResult answers a HintScanRequest. Core owns Targets.

Comments0 open / 0

no comments yet.

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
l
merge request: the list of files (j k move, enter opens, esc back)
M
merge request: the merge menu; there 1–3 pick the mode, D deletes the branch, enter merges
R
merge request: mark the head read
esc
close a dialog, leave a field

Branches 23

Tags 10

↑↓ moveenter openesc close