docs(config): describe the hint detector

commit: 8f6a5574d18c6bb1ff58ccdfb0184914660d3ac1
author: brice · 30h ago · 2026-10-21 13:28 UTC
parent: 596b553
signature: GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email

3 files · +28 -32

M README.md +1 -1


@@ -22,7 +22,7 @@ https://github.com/user-attachments/assets/54f25914-8335-4e66-b458-dabc5493d355
22	22		- Games: direct scanout, VRR, tearing, explicit sync and [HDR](docs/config.md#hdr) give games the whole GPU instead of spending it on composition; see [Performance](docs/performance.md).
23	23		- Overview: `cmd+o` or a four-finger swipe up shows live workspace previews. Select a window with the keyboard, touchpad or mouse; see [Overview](docs/config.md#overview).
24	24		- Stash: `cmd+s` toggles a floating strip of windows set aside for the current workspace; `cmd+shift+s` moves a window into or out of it. See [Stash](docs/config.md#stash).
25		-	- Hint mode: click links and buttons by typing their labels, without reaching for the mouse. Enable experimental `hint.enable` and add a bind; see [Hint mode](docs/config.md#hint-mode).
	25	+	- Hint mode: click what is on screen by typing its label, without reaching for the mouse; targets come from the `nefer-hints` detector. Enable experimental `hint.enable` and add a bind; see [Hint mode](docs/config.md#hint-mode).
26	26		- Animations: short, cheap and off by default. Opt in with experimental `animations = on`; see [Configuration](docs/config.md#keys).
27	27
28	28		Written in Go, without cgo. Each piece of state has one owner goroutine, connected to the others by channels; hot paths reuse memory and client buffers stay off the Go heap. See [Architecture](docs/dev/architecture.md) for the concurrency model, the GC strategy and the reusable Go libraries.


@@ -22,7 +22,7 @@ https://github.com/user-attachments/assets/54f25914-8335-4e66-b458-dabc5493d355
22		- Games: direct scanout, VRR, tearing, explicit sync and [HDR](docs/config.md#hdr) give games the whole GPU instead of spending it on composition; see [Performance](docs/performance.md).	22		- Games: direct scanout, VRR, tearing, explicit sync and [HDR](docs/config.md#hdr) give games the whole GPU instead of spending it on composition; see [Performance](docs/performance.md).
23		- Overview: `cmd+o` or a four-finger swipe up shows live workspace previews. Select a window with the keyboard, touchpad or mouse; see [Overview](docs/config.md#overview).	23		- Overview: `cmd+o` or a four-finger swipe up shows live workspace previews. Select a window with the keyboard, touchpad or mouse; see [Overview](docs/config.md#overview).
24		- Stash: `cmd+s` toggles a floating strip of windows set aside for the current workspace; `cmd+shift+s` moves a window into or out of it. See [Stash](docs/config.md#stash).	24		- Stash: `cmd+s` toggles a floating strip of windows set aside for the current workspace; `cmd+shift+s` moves a window into or out of it. See [Stash](docs/config.md#stash).
25	-	- Hint mode: click links and buttons by typing their labels, without reaching for the mouse. Enable experimental `hint.enable` and add a bind; see [Hint mode](docs/config.md#hint-mode).	25	+	- Hint mode: click what is on screen by typing its label, without reaching for the mouse; targets come from the `nefer-hints` detector. Enable experimental `hint.enable` and add a bind; see [Hint mode](docs/config.md#hint-mode).
26		- Animations: short, cheap and off by default. Opt in with experimental `animations = on`; see [Configuration](docs/config.md#keys).	26		- Animations: short, cheap and off by default. Opt in with experimental `animations = on`; see [Configuration](docs/config.md#keys).
27			27
28		Written in Go, without cgo. Each piece of state has one owner goroutine, connected to the others by channels; hot paths reuse memory and client buffers stay off the Go heap. See [Architecture](docs/dev/architecture.md) for the concurrency model, the GC strategy and the reusable Go libraries.	28		Written in Go, without cgo. Each piece of state has one owner goroutine, connected to the others by channels; hot paths reuse memory and client buffers stay off the Go heap. See [Architecture](docs/dev/architecture.md) for the concurrency model, the GC strategy and the reusable Go libraries.

M docs/config.md +23 -28


@@ -60,8 +60,9 @@ See [examples/config](../examples/config) for every key with its default.
60	60		\| `focus.flash-duration` \| `200ms` \| Fade duration, `50ms` to `1s`; not scaled by `animations.speed` \|
61	61		\| `focus.flash-color` \| `border.active` \| Flash color `#rrggbb`; empty uses `border.active` \|
62	62		\| `hint.enable` \| `off` \| Turn on the experimental [hint mode](#hint-mode) (restart required) \|
63		-	\| `hint.targets` \| `clickable` \| What [hint mode](#hint-mode) labels: `clickable` (links, URLs and buttons) or `links` (links and URLs only) \|
64		-	\| `hint.text-click` \| `ctrl+shift` \| Modifiers held for clicks on links found by OCR (terminals open links on a modified click); `none` sends a plain click \|
	63	+	\| `hint.detector` \| `nefer-hints` \| The [hint mode](#hint-mode) detector binary: a path, or a name looked up in `PATH` (restart required) \|
	64	+	\| `hint.model` \| `$XDG_DATA_HOME/nefer-hints/models` \| Directory of the detector's model files; `~/.local/share/nefer-hints/models` when `XDG_DATA_HOME` is unset; a leading `~/` is expanded (restart required) \|
	65	+	\| `hint.threshold` \| `0.05` \| Minimum detection score, 0.01 to 0.99; lower finds more targets and more false ones (restart required) \|
65	66		\| `hint.capture` \| `off` \| Show hint labels in captures requested by Wayland clients (screenshots, screencasts); the compositor's own hint scans never see them; applies on reload \|
66	67		\| `hint.size` \| `auto` \| Label text size in logical pixels (6 to 72), or `auto` for screen pixel density \|
67	68		\| `hint.background` \| empty \| Label background (`#rrggbb`); empty uses `border.active` \|

@@ -239,7 +240,7 @@ bind.XF86MonBrightnessDown = spawn brightnessctl set 5%-
239	240		\| `toggle-window-stash` \| Move the focused window to the [stash](#stash), or back to its column \|
240	241		\| `toggle-stash-visible` \| Show or hide the stash \|
241	242		\| `toggle-overview` \| Open or close the [overview](#overview) \|
242		-	\| `hint` \| Label the links and buttons on the focused screen, then click the one you type ([hint mode](#hint-mode)) \|
	243	+	\| `hint` \| Label the clickable elements on the focused screen, then click the one you type ([hint mode](#hint-mode)) \|
243	244		\| `switch-column-next` / `-prev` \| Switch to the previously used column. Hold `cmd` and press again to go further back \|
244	245		\| `maximize-column` \| Toggle full width for the focused column \|
245	246		\| `cycle-column-width` \| Grow the focused column by one [cell](#column-widths), then back to equal widths \|

@@ -353,46 +354,40 @@ Hint mode is experimental and off by default: set `hint.enable = on` and restart
353	354		While it is off, nothing of hint mode runs or uses memory, and the `hint` bind
354	355		logs a notice once and does nothing. A reload does not change it.
355	356
356		-	`hint` labels the links and buttons on the focused output. Type a label to
357		-	click its target; BackSpace erases a letter, Escape cancels. No bind is set
358		-	by default; for example `bind.cmd+g = hint`.
	357	+	`hint` labels the clickable elements (buttons, links, icons, fields) on the
	358	+	focused output. Type a label to click its target; BackSpace erases a letter,
	359	+	Escape cancels. No bind is set by default; for example `bind.cmd+g = hint`.
359	360
360	361		Labels use the `border.active` colour with a contrasting text colour, and
361	362		their size follows the screen's pixel density. Each label sits just outside
362		-	the top-left corner of its target. Links found by OCR (terminal text) are
363		-	clicked with the `hint.text-click` modifiers held, which is how terminals open
364		-	a link; accessibility targets get a plain click.
	363	+	the top-left corner of its target. The click is a plain left click.
365	364
366	365		Labels are not part of screenshots and screencasts by default. With
367	366		`hint.capture = on` they appear in the captures that Wayland clients request
368		-	(including Exclude captures); NeferWL's own text recognition never sees them.
	367	+	(including Exclude captures); the detector never sees them.
369	368		The capture indicator and the position hint stay out of every capture.
370	369		A reload applies `hint.capture` to the next frame.
371	370
372	371		### Where targets come from
373	372
374		-	\| Source \| Apps \| Accuracy \| Speed \|
375		-	\|---\|---\|---\|---\|
376		-	\| Accessibility tree (AT-SPI) \| Firefox, Chromium, GTK, Qt, Electron \| Exact: the app reports its own links and buttons \| A few milliseconds \|
377		-	\| Text recognition (OCR) \| Everything else: terminals, apps without accessibility \| Best effort: URLs, domains and underlined text written on screen \| 150 ms or more, more on large screens \|
	373	+	Targets come from `nefer-hints`, a separate program that looks at the pixels of
	374	+	the whole output with a neural network and returns a box for each clickable
	375	+	element. It works the same for every app, toolkit and terminal. It sees what is
	376	+	drawn: it does not know what an element does, and it can miss or invent boxes
	377	+	(`hint.threshold` trades one for the other). Boxes under 4×4 logical pixels
	378	+	are dropped.
378	379
379		-	NeferWL asks the accessibility tree first. Windows it does not cover are read
380		-	with OCR, which finds visible URLs, domains, email addresses and underlined
381		-	text. Underlined words get a plain click, and the words of one continuous
382		-	underline form one target; URLs keep the `hint.text-click` modifiers. OCR
383		-	cannot see a link whose address is hidden behind plain text, and it can misread
384		-	small or low-contrast text.
	380	+	NeferWL starts `nefer-hints` at the first hint scan (loading takes about half a
	381	+	second), keeps it running, and stops it on exit. If it dies, fails or takes
	382	+	more than 2 seconds on a scan, that scan is cancelled and a new process starts
	383	+	at the next one. Its device (GPU or CPU) is logged at startup, at info level.
385	384
386	385		### Requirements
387	386
388		-	- Accessibility: NeferWL enables it lazily on the first hint scan that queries
389		-	app windows. Apps started before then may need restarting. Some apps need a flag:
390		-	- Chromium: `--force-renderer-accessibility`
391		-	- Electron: `ACCESSIBILITY_ENABLED=1`
392		-	- Qt: `QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1`
393		-	- OCR: `tesseract` and `tesseract-data-eng`. Without tesseract, OCR is off
394		-	and hint mode labels only apps that expose the accessibility tree.
	387	+	- The `nefer-hints` binary, found in `PATH` or named by `hint.detector`.
	388	+	- Its model files, in `hint.model`.
	389	+	- A Vulkan GPU for speed; without one the detector falls back to the CPU, which is slower.
395	390
396	391		### Debugging hint mode
397	392
398		-	With `hint.enable = on` and `NEFERWL_HINT_DEBUG=1` set when NeferWL starts (for example `NEFERWL_HINT_DEBUG=1 make tty`, or the variable in the launch environment), every successful scan writes three files to `$XDG_STATE_HOME/neferwl/hints/` (`~/.local/state/neferwl/hints/` when unset): `<time>-<output>.png`, the frame as scanned; `<time>-<output>.annotated.png`, the same with each target outlined (OCR links green, accessibility targets magenta, OCR window crops grey); and `<time>-<output>.json`, the targets (with their `source`, `atspi` or `ocr`) and timings. NeferWL keeps the 50 most recent scans and logs a warning at startup. The files contain the screen in clear: delete them after use.
	393	+	With `hint.enable = on` and `NEFERWL_HINT_DEBUG=1` set when NeferWL starts (for example `NEFERWL_HINT_DEBUG=1 make tty`, or the variable in the launch environment), every successful scan writes three files to `$XDG_STATE_HOME/neferwl/hints/` (`~/.local/state/neferwl/hints/` when unset): `<time>-<output>.png`, the frame as scanned; `<time>-<output>.annotated.png`, the same with the detector boxes outlined; and `<time>-<output>.json`, the targets (rectangle and score) and the capture, detection and total timings. NeferWL keeps the 50 most recent scans and logs a warning at startup. The detector's own messages are logged at debug level (`log.debug = hints`). The files contain the screen in clear: delete them after use.

 | `focus.flash-duration` | `200ms` | Fade duration, `50ms` to `1s`; not scaled by `animations.speed` |
 | `focus.flash-color` | `border.active` | Flash color `#rrggbb`; empty uses `border.active` |
 | `hint.enable` | `off` | Turn on the experimental [hint mode](#hint-mode) (restart required) |
-| `hint.targets` | `clickable` | What [hint mode](#hint-mode) labels: `clickable` (links, URLs and buttons) or `links` (links and URLs only) |
+| `hint.detector` | `nefer-hints` | The [hint mode](#hint-mode) detector binary: a path, or a name looked up in `PATH` (restart required) |
-| `hint.text-click` | `ctrl+shift` | Modifiers held for clicks on links found by OCR (terminals open links on a modified click); `none` sends a plain click |
+| `hint.model` | `$XDG_DATA_HOME/nefer-hints/models` | Directory of the detector's model files; `~/.local/share/nefer-hints/models` when `XDG_DATA_HOME` is unset; a leading `~/` is expanded (restart required) |
+| `hint.threshold` | `0.05` | Minimum detection score, 0.01 to 0.99; lower finds more targets and more false ones (restart required) |
 | `hint.capture` | `off` | Show hint labels in captures requested by Wayland clients (screenshots, screencasts); the compositor's own hint scans never see them; applies on reload |
 | `hint.size` | `auto` | Label text size in logical pixels (6 to 72), or `auto` for screen pixel density |
 | `hint.background` | empty | Label background (`#rrggbb`); empty uses `border.active` |
 | `toggle-window-stash` | Move the focused window to the [stash](#stash), or back to its column |
 | `toggle-stash-visible` | Show or hide the stash |
 | `toggle-overview` | Open or close the [overview](#overview) |
-| `hint` | Label the links and buttons on the focused screen, then click the one you type ([hint mode](#hint-mode)) |
+| `hint` | Label the clickable elements on the focused screen, then click the one you type ([hint mode](#hint-mode)) |
 | `switch-column-next` / `-prev` | Switch to the previously used column. Hold `cmd` and press again to go further back |
 | `maximize-column` | Toggle full width for the focused column |
 | `cycle-column-width` | Grow the focused column by one [cell](#column-widths), then back to equal widths |
 While it is off, nothing of hint mode runs or uses memory, and the `hint` bind
 logs a notice once and does nothing. A reload does not change it.
-`hint` labels the links and buttons on the focused output. Type a label to
+`hint` labels the clickable elements (buttons, links, icons, fields) on the
-click its target; BackSpace erases a letter, Escape cancels. No bind is set
+focused output. Type a label to click its target; BackSpace erases a letter,
-by default; for example `bind.cmd+g = hint`.
+Escape cancels. No bind is set by default; for example `bind.cmd+g = hint`.
 Labels use the `border.active` colour with a contrasting text colour, and
 their size follows the screen's pixel density. Each label sits just outside
-the top-left corner of its target. Links found by OCR (terminal text) are
+the top-left corner of its target. The click is a plain left click.
-clicked with the `hint.text-click` modifiers held, which is how terminals open
-a link; accessibility targets get a plain click.
 Labels are not part of screenshots and screencasts by default. With
 `hint.capture = on` they appear in the captures that Wayland clients request
-(including Exclude captures); NeferWL's own text recognition never sees them.
+(including Exclude captures); the detector never sees them.
 The capture indicator and the position hint stay out of every capture.
 A reload applies `hint.capture` to the next frame.
 ### Where targets come from
-| Source | Apps | Accuracy | Speed |
+Targets come from `nefer-hints`, a separate program that looks at the pixels of
-|---|---|---|---|
+the whole output with a neural network and returns a box for each clickable
-| Accessibility tree (AT-SPI) | Firefox, Chromium, GTK, Qt, Electron | Exact: the app reports its own links and buttons | A few milliseconds |
+element. It works the same for every app, toolkit and terminal. It sees what is
-| Text recognition (OCR) | Everything else: terminals, apps without accessibility | Best effort: URLs, domains and underlined text written on screen | 150 ms or more, more on large screens |
+drawn: it does not know what an element does, and it can miss or invent boxes
+(`hint.threshold` trades one for the other). Boxes under 4×4 logical pixels
+are dropped.
-NeferWL asks the accessibility tree first. Windows it does not cover are read
+NeferWL starts `nefer-hints` at the first hint scan (loading takes about half a
-with OCR, which finds visible URLs, domains, email addresses and underlined
+second), keeps it running, and stops it on exit. If it dies, fails or takes
-text. Underlined words get a plain click, and the words of one continuous
+more than 2 seconds on a scan, that scan is cancelled and a new process starts
-underline form one target; URLs keep the `hint.text-click` modifiers. OCR
+at the next one. Its device (GPU or CPU) is logged at startup, at info level.
-cannot see a link whose address is hidden behind plain text, and it can misread
-small or low-contrast text.
 ### Requirements
-- Accessibility: NeferWL enables it lazily on the first hint scan that queries
+- The `nefer-hints` binary, found in `PATH` or named by `hint.detector`.
-  app windows. Apps started before then may need restarting. Some apps need a flag:
+- Its model files, in `hint.model`.
-  - Chromium: `--force-renderer-accessibility`
+- A Vulkan GPU for speed; without one the detector falls back to the CPU, which is slower.
-  - Electron: `ACCESSIBILITY_ENABLED=1`
-  - Qt: `QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1`
-- OCR: `tesseract` and `tesseract-data-eng`. Without tesseract, OCR is off
-  and hint mode labels only apps that expose the accessibility tree.
 ### Debugging hint mode
-With `hint.enable = on` and `NEFERWL_HINT_DEBUG=1` set when NeferWL starts (for example `NEFERWL_HINT_DEBUG=1 make tty`, or the variable in the launch environment), every successful scan writes three files to `$XDG_STATE_HOME/neferwl/hints/` (`~/.local/state/neferwl/hints/` when unset): `<time>-<output>.png`, the frame as scanned; `<time>-<output>.annotated.png`, the same with each target outlined (OCR links green, accessibility targets magenta, OCR window crops grey); and `<time>-<output>.json`, the targets (with their `source`, `atspi` or `ocr`) and timings. NeferWL keeps the 50 most recent scans and logs a warning at startup. The files contain the screen in clear: delete them after use.
+With `hint.enable = on` and `NEFERWL_HINT_DEBUG=1` set when NeferWL starts (for example `NEFERWL_HINT_DEBUG=1 make tty`, or the variable in the launch environment), every successful scan writes three files to `$XDG_STATE_HOME/neferwl/hints/` (`~/.local/state/neferwl/hints/` when unset): `<time>-<output>.png`, the frame as scanned; `<time>-<output>.annotated.png`, the same with the detector boxes outlined; and `<time>-<output>.json`, the targets (rectangle and score) and the capture, detection and total timings. NeferWL keeps the 50 most recent scans and logs a warning at startup. The detector's own messages are logged at debug level (`log.debug = hints`). The files contain the screen in clear: delete them after use.

M examples/config +4 -3


@@ -98,9 +98,10 @@ focus.flash-duration = 200ms
98	98		focus.flash-color =
99	99		# Hint mode is experimental and off: turn it on here (restart required).
100	100		# hint.enable = off
101		-	# Hint mode labels the links, URLs and buttons on the focused screen:
102		-	# clickable (links, URLs and buttons) or links (links and URLs only).
103		-	# hint.targets = clickable
	101	+	# Hint mode finds its targets with the nefer-hints detector (restart required).
	102	+	# hint.detector = nefer-hints
	103	+	# hint.model = ~/.local/share/nefer-hints/models
	104	+	# hint.threshold = 0.05
104	105		# Show the labels in screenshots and screencasts taken by clients.
105	106		# hint.capture = off
106	107

 focus.flash-color =
 # Hint mode is experimental and off: turn it on here (restart required).
 # hint.enable = off
-# Hint mode labels the links, URLs and buttons on the focused screen:
+# Hint mode finds its targets with the nefer-hints detector (restart required).
-# clickable (links, URLs and buttons) or links (links and URLs only).
+# hint.detector = nefer-hints
-# hint.targets = clickable
+# hint.model = ~/.local/share/nefer-hints/models
+# hint.threshold = 0.05
 # Show the labels in screenshots and screencasts taken by clients.
 # hint.capture = off