detached 3a0a985

feat(demo): open the hint scene on fullscreen Firefox

commit
3a0a985de87a8a181a72d84e0797f921c65f252d
author
brice · · 2026-10-21 15:42 UTC
committer
brice · · 2026-10-22 04:47 UTC
parent
d5c3b34
signature
GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email
7 files · +105 -86
Split
UnifiedSplit
M docs/dev/headless.md +1 -1
@@ -30,7 +30,7 @@ It builds the Arch image (apps, tesseract, AT-SPI) and the compositor binary, th
3030 The compositor runs headless inside a disposable container, with the binary and a work directory bind-mounted and `/dev/dri` passed for the GPU.
3131 Firefox opens a live DuckDuckGo page, so the recording needs the network and its content varies.
3232 A FIFO feeds `input.txt` while the script launches the apps on the same clock.
33-The hint step labels the page with `hint`, then `hint-click <word>` reads the scanned frame and types the label of the link showing that word.
33+The hint step makes Firefox fullscreen and labels the page with `hint`; `hint-click <words>` then types the label of the link that shows those words, found from the scan's dump and an OCR read of the still page (`hint-read`).
3434 Temporary captures are removed by default; set `KEEP_CAPTURE=1` to keep successful runs.
3535 The recording and poster replace `docs/media/demo.mp4` and `docs/media/demo.png`.
3636
@@ -30,7 +30,7 @@ It builds the Arch image (apps, tesseract, AT-SPI) and the compositor binary, th
30 The compositor runs headless inside a disposable container, with the binary and a work directory bind-mounted and `/dev/dri` passed for the GPU.30 The compositor runs headless inside a disposable container, with the binary and a work directory bind-mounted and `/dev/dri` passed for the GPU.
31 Firefox opens a live DuckDuckGo page, so the recording needs the network and its content varies.31 Firefox opens a live DuckDuckGo page, so the recording needs the network and its content varies.
32 A FIFO feeds `input.txt` while the script launches the apps on the same clock.32 A FIFO feeds `input.txt` while the script launches the apps on the same clock.
33-The hint step labels the page with `hint`, then `hint-click <word>` reads the scanned frame and types the label of the link showing that word.33+The hint step makes Firefox fullscreen and labels the page with `hint`; `hint-click <words>` then types the label of the link that shows those words, found from the scan's dump and an OCR read of the still page (`hint-read`).
34 Temporary captures are removed by default; set `KEEP_CAPTURE=1` to keep successful runs.34 Temporary captures are removed by default; set `KEEP_CAPTURE=1` to keep successful runs.
35 The recording and poster replace `docs/media/demo.mp4` and `docs/media/demo.png`.35 The recording and poster replace `docs/media/demo.mp4` and `docs/media/demo.png`.
36 36
M docs/media/demo.mp4 +0 -0 binary file: no text diff

binary file: no text diff

M docs/media/demo.png +0 -0 binary file: no text diff

binary file: no text diff

M tools/demo/config +1 -0
@@ -19,6 +19,7 @@ hint.targets = links
1919 # Super+Shift+s sets a window aside.
2020 # Super+s brings your stash into view.
2121 # Super+o opens the overview.
22+# Super+Shift+f makes a window fullscreen.
2223 # Super+g labels the links; type one to click it.
2324 bind.cmd+g = hint
2425 bind.cmd+code:3 = focus-workspace 2
@@ -19,6 +19,7 @@ hint.targets = links
19 # Super+Shift+s sets a window aside.19 # Super+Shift+s sets a window aside.
20 # Super+s brings your stash into view.20 # Super+s brings your stash into view.
21 # Super+o opens the overview.21 # Super+o opens the overview.
22+# Super+Shift+f makes a window fullscreen.
22 # Super+g labels the links; type one to click it.23 # Super+g labels the links; type one to click it.
23 bind.cmd+g = hint24 bind.cmd+g = hint
24 bind.cmd+code:3 = focus-workspace 225 bind.cmd+code:3 = focus-workspace 2
M tools/demo/demo-hint-label +53 -41
@@ -1,61 +1,73 @@
11 #!/usr/bin/env python3
2-"""Print the hint label of the link under a word of the last hint scan.
2+"""Work out the hint label of a link, to type it the way a user reads it.
33
4-usage: demo-hint-label <since-epoch> <word>...
4+ demo-hint-label ocr
5+ Reads the frame on screen now (the compositor's latest screenshot) with
6+ tesseract and keeps the words and their boxes in /tmp/hint-words.json.
7+ Run it while the page is still, before the hint bind, so that the
8+ lookup after the bind needs no OCR.
59
6-Waits for a hint dump (NEFERWL_HINT_DEBUG=1) written after <since-epoch>, reads
7-the scanned frame with tesseract, finds the largest occurrence of the first of
8-the words that is on screen and returns the label of the link target under it.
9-Labels follow internal/core/hint.go: targets sorted by top then left edge, one
10-letter each from the home-row alphabet, two letters past 26 targets.
10+ demo-hint-label label <since-epoch> <word>...
11+ Waits for the hint dump (NEFERWL_HINT_DEBUG=1) of the scan started after
12+ <since-epoch>, finds the link target that shows all the words and prints
13+ its label. Labels follow internal/core/hint.go: the targets whose centre
14+ is on the output, sorted by top then left edge, one letter each from the
15+ home-row alphabet, two letters past 26 targets.
1116 """
12-import csv, glob, io, json, os, re, subprocess, sys, time
17+import csv, glob, io, json, os, re, shutil, subprocess, sys, time
1318
1419 ALPHABET = "fjdkslaghrueiwoqptyvbcnxmz"
15-DUMPS = os.path.join(os.environ.get("XDG_STATE_HOME", "/work/state"), "neferwl", "hints")
20+STATE = os.environ.get("XDG_STATE_HOME", "/work/state")
21+DUMPS = os.path.join(STATE, "neferwl", "hints")
22+SHOT = "/work/shots/latest.png"
23+WORDS = "/tmp/hint-words.json"
1624
1725
18-def newest_dump(since):
19- deadline = time.time() + 15
20- while time.time() < deadline:
21- for path in sorted(glob.glob(os.path.join(DUMPS, "*.json")), reverse=True):
22- png = path[:-5] + ".png"
23- if os.path.getmtime(path) >= since and os.path.exists(png):
24- return path, png
25- time.sleep(0.2)
26- sys.exit("no hint scan dump after the bind")
27-
28-
29-def words(png):
30- out = subprocess.run(["tesseract", png, "stdout", "--psm", "11", "tsv"],
26+def ocr():
27+ shutil.copyfile(SHOT, "/tmp/hint-ocr.png")
28+ out = subprocess.run(["tesseract", "/tmp/hint-ocr.png", "stdout", "--psm", "11", "tsv"],
3129 capture_output=True, text=True, check=True).stdout
30+ words = []
3231 for row in csv.DictReader(io.StringIO(out), delimiter="\t", quoting=csv.QUOTE_NONE):
3332 text = re.sub(r"\W", "", row["text"] or "").lower()
3433 if text and float(row["conf"]) > 30:
3534 x, y, w, h = (int(row[k]) for k in ("left", "top", "width", "height"))
36- yield text, x + w / 2, y + h / 2, h
35+ words.append([text, x + w / 2, y + h / 2])
36+ json.dump(words, open(WORDS, "w"))
37+
38+
39+def newest_dump(since):
40+ deadline = time.time() + 15
41+ while time.time() < deadline:
42+ for path in sorted(glob.glob(os.path.join(DUMPS, "*.json")), reverse=True):
43+ if os.path.getmtime(path) >= since:
44+ return path
45+ time.sleep(0.05)
46+ sys.exit("no hint scan dump after the bind")
3747
3848
39-def main():
40- since, wanted = float(sys.argv[1]), [w.lower() for w in sys.argv[2:]]
41- path, png = newest_dump(since)
42- meta = json.load(open(path))
49+def label(since, wanted):
50+ meta = json.load(open(newest_dump(since)))
51+ words = json.load(open(WORDS))
4352 # Core keeps the targets whose centre is on the output, then sorts them.
4453 width, height = meta["width"], meta["height"]
4554 targets = [t for t in meta["targets"]
4655 if 0 <= (t["rect"][0] + t["rect"][2]) / 2 < width and 0 <= (t["rect"][1] + t["rect"][3]) / 2 < height]
4756 targets.sort(key=lambda t: (t["rect"][1], t["rect"][0]))
48- found = list(words(png))
49- for word in wanted:
50- hits = [(h, cx, cy) for t, cx, cy, h in found if t == word]
51- for _, cx, cy in sorted(hits, reverse=True):
52- for i, t in enumerate(targets):
53- x0, y0, x1, y1 = t["rect"]
54- if t["kind"] == "link" and x0 <= cx < x1 and y0 <= cy < y1:
55- n = len(ALPHABET)
56- print(ALPHABET[i] if len(targets) <= n else ALPHABET[i // n] + ALPHABET[i % n])
57- return
58- sys.exit("none of %s is under a link target (%d targets)" % (wanted, len(targets)))
59-
60-
61-main()
57+ for i, t in enumerate(targets):
58+ x0, y0, x1, y1 = t["rect"]
59+ inside = {w for w, cx, cy in words if x0 <= cx < x1 and y0 <= cy < y1}
60+ if t["kind"] == "link" and set(wanted) <= inside:
61+ n = len(ALPHABET)
62+ print("link %d of %d shows %s" % (i + 1, len(targets), " ".join(wanted)), file=sys.stderr)
63+ print(ALPHABET[i] if len(targets) <= n else ALPHABET[i // n] + ALPHABET[i % n])
64+ return
65+ sys.exit("no link shows %s (%d targets, %d words read)" % (" ".join(wanted), len(targets), len(words)))
66+
67+
68+if sys.argv[1:2] == ["ocr"]:
69+ ocr()
70+elif sys.argv[1:2] == ["label"] and len(sys.argv) > 3:
71+ label(float(sys.argv[2]), [w.lower() for w in sys.argv[3:]])
72+else:
73+ sys.exit(__doc__)
@@ -1,61 +1,73 @@
1 #!/usr/bin/env python31 #!/usr/bin/env python3
2-"""Print the hint label of the link under a word of the last hint scan.2+"""Work out the hint label of a link, to type it the way a user reads it.
3 3
4-usage: demo-hint-label <since-epoch> <word>...4+ demo-hint-label ocr
5+ Reads the frame on screen now (the compositor's latest screenshot) with
6+ tesseract and keeps the words and their boxes in /tmp/hint-words.json.
7+ Run it while the page is still, before the hint bind, so that the
8+ lookup after the bind needs no OCR.
5 9
6-Waits for a hint dump (NEFERWL_HINT_DEBUG=1) written after <since-epoch>, reads10+ demo-hint-label label <since-epoch> <word>...
7-the scanned frame with tesseract, finds the largest occurrence of the first of11+ Waits for the hint dump (NEFERWL_HINT_DEBUG=1) of the scan started after
8-the words that is on screen and returns the label of the link target under it.12+ <since-epoch>, finds the link target that shows all the words and prints
9-Labels follow internal/core/hint.go: targets sorted by top then left edge, one13+ its label. Labels follow internal/core/hint.go: the targets whose centre
10-letter each from the home-row alphabet, two letters past 26 targets.14+ is on the output, sorted by top then left edge, one letter each from the
15+ home-row alphabet, two letters past 26 targets.
11 """16 """
12-import csv, glob, io, json, os, re, subprocess, sys, time17+import csv, glob, io, json, os, re, shutil, subprocess, sys, time
13 18
14 ALPHABET = "fjdkslaghrueiwoqptyvbcnxmz"19 ALPHABET = "fjdkslaghrueiwoqptyvbcnxmz"
15-DUMPS = os.path.join(os.environ.get("XDG_STATE_HOME", "/work/state"), "neferwl", "hints")20+STATE = os.environ.get("XDG_STATE_HOME", "/work/state")
21+DUMPS = os.path.join(STATE, "neferwl", "hints")
22+SHOT = "/work/shots/latest.png"
23+WORDS = "/tmp/hint-words.json"
16 24
17 25
18-def newest_dump(since):26+def ocr():
19- deadline = time.time() + 1527+ shutil.copyfile(SHOT, "/tmp/hint-ocr.png")
20- while time.time() < deadline:28+ out = subprocess.run(["tesseract", "/tmp/hint-ocr.png", "stdout", "--psm", "11", "tsv"],
21- for path in sorted(glob.glob(os.path.join(DUMPS, "*.json")), reverse=True):
22- png = path[:-5] + ".png"
23- if os.path.getmtime(path) >= since and os.path.exists(png):
24- return path, png
25- time.sleep(0.2)
26- sys.exit("no hint scan dump after the bind")
27-
28-
29-def words(png):
30- out = subprocess.run(["tesseract", png, "stdout", "--psm", "11", "tsv"],
31 capture_output=True, text=True, check=True).stdout29 capture_output=True, text=True, check=True).stdout
30+ words = []
32 for row in csv.DictReader(io.StringIO(out), delimiter="\t", quoting=csv.QUOTE_NONE):31 for row in csv.DictReader(io.StringIO(out), delimiter="\t", quoting=csv.QUOTE_NONE):
33 text = re.sub(r"\W", "", row["text"] or "").lower()32 text = re.sub(r"\W", "", row["text"] or "").lower()
34 if text and float(row["conf"]) > 30:33 if text and float(row["conf"]) > 30:
35 x, y, w, h = (int(row[k]) for k in ("left", "top", "width", "height"))34 x, y, w, h = (int(row[k]) for k in ("left", "top", "width", "height"))
36- yield text, x + w / 2, y + h / 2, h35+ words.append([text, x + w / 2, y + h / 2])
36+ json.dump(words, open(WORDS, "w"))
37+
38+
39+def newest_dump(since):
40+ deadline = time.time() + 15
41+ while time.time() < deadline:
42+ for path in sorted(glob.glob(os.path.join(DUMPS, "*.json")), reverse=True):
43+ if os.path.getmtime(path) >= since:
44+ return path
45+ time.sleep(0.05)
46+ sys.exit("no hint scan dump after the bind")
37 47
38 48
39-def main():49+def label(since, wanted):
40- since, wanted = float(sys.argv[1]), [w.lower() for w in sys.argv[2:]]50+ meta = json.load(open(newest_dump(since)))
41- path, png = newest_dump(since)51+ words = json.load(open(WORDS))
42- meta = json.load(open(path))
43 # Core keeps the targets whose centre is on the output, then sorts them.52 # Core keeps the targets whose centre is on the output, then sorts them.
44 width, height = meta["width"], meta["height"]53 width, height = meta["width"], meta["height"]
45 targets = [t for t in meta["targets"]54 targets = [t for t in meta["targets"]
46 if 0 <= (t["rect"][0] + t["rect"][2]) / 2 < width and 0 <= (t["rect"][1] + t["rect"][3]) / 2 < height]55 if 0 <= (t["rect"][0] + t["rect"][2]) / 2 < width and 0 <= (t["rect"][1] + t["rect"][3]) / 2 < height]
47 targets.sort(key=lambda t: (t["rect"][1], t["rect"][0]))56 targets.sort(key=lambda t: (t["rect"][1], t["rect"][0]))
48- found = list(words(png))57+ for i, t in enumerate(targets):
49- for word in wanted:58+ x0, y0, x1, y1 = t["rect"]
50- hits = [(h, cx, cy) for t, cx, cy, h in found if t == word]59+ inside = {w for w, cx, cy in words if x0 <= cx < x1 and y0 <= cy < y1}
51- for _, cx, cy in sorted(hits, reverse=True):60+ if t["kind"] == "link" and set(wanted) <= inside:
52- for i, t in enumerate(targets):61+ n = len(ALPHABET)
53- x0, y0, x1, y1 = t["rect"]62+ print("link %d of %d shows %s" % (i + 1, len(targets), " ".join(wanted)), file=sys.stderr)
54- if t["kind"] == "link" and x0 <= cx < x1 and y0 <= cy < y1:63+ print(ALPHABET[i] if len(targets) <= n else ALPHABET[i // n] + ALPHABET[i % n])
55- n = len(ALPHABET)64+ return
56- print(ALPHABET[i] if len(targets) <= n else ALPHABET[i // n] + ALPHABET[i % n])65+ sys.exit("no link shows %s (%d targets, %d words read)" % (" ".join(wanted), len(targets), len(words)))
57- return66+
58- sys.exit("none of %s is under a link target (%d targets)" % (wanted, len(targets)))67+
59-68+if sys.argv[1:2] == ["ocr"]:
60-69+ ocr()
61-main()70+elif sys.argv[1:2] == ["label"] and len(sys.argv) > 3:
71+ label(float(sys.argv[2]), [w.lower() for w in sys.argv[3:]])
72+else:
73+ sys.exit(__doc__)
M tools/demo/input.txt +32 -31
@@ -1,8 +1,17 @@
11 move 1918 1078
2-sleep 12s
2+sleep 5s
3+# Unseen warm-up while Firefox sits beside the terminal: a hint bind cancelled
4+# at once still runs its scan, without drawing labels. It connects to the
5+# accessibility bus early; more of them below (each right after a key press, in
6+# an existing pause) let Firefox expose its page content by the end.
7+key Super+g
8+key Escape
9+sleep 7s
310 key Super+Up
411 sleep 1500ms
512 key Super+Down
13+key Super+g
14+key Escape
615 sleep 1500ms
716 key Super+Shift+s
817 sleep 500ms
@@ -11,6 +20,8 @@ sleep 1500ms
1120 key Super+2
1221 key Super+Shift+1
1322 key Super+1
23+key Super+g
24+key Escape
1425 sleep 3s
1526 key Super+Shift+s
1627 sleep 500ms
@@ -18,8 +29,12 @@ key Super+s
1829 sleep 1500ms
1930 key Super+s
2031 key Super+Left
32+key Super+g
33+key Escape
2134 sleep 2s
2235 key Super+Right
36+key Super+g
37+key Escape
2338 sleep 2s
2439 key Super+Left
2540 sleep 1s
@@ -40,34 +55,20 @@ sleep 1500ms
4055 key Up
4156 sleep 1s
4257 key Return
43-sleep 2s
44-# Back on workspace 1. Firefox builds its accessibility tree for page content
45-# only after a few scans; a hint bind cancelled at once still runs its scan
46-# without drawing labels, so these warm it up unseen.
47-key Super+1
48-sleep 2s
49-key Super+g
50-key Escape
51-sleep 2s
52-key Super+g
53-key Escape
54-sleep 2s
55-key Super+g
56-key Escape
57-sleep 2s
58-# Clear the terminal's listing (OCR would take file names for links) and print
59-# a URL, which hint mode labels like Firefox's links. foot opens a URL only from
60-# its own keyboard mode, so the terminal's label is shown, not clicked.
61-move 400 600
62-click
63-type clear; echo https://github.com/bnema/neferwl
64-key Return
65-sleep 500ms
66-move 1918 1078
67-sleep 2s
68-# Hint mode: Firefox's links come from the accessibility tree, the URL from OCR.
58+sleep 700ms
59+# Back on workspace 1: focus Firefox and fullscreen it. Once the page is still,
60+# read it, then hint mode starts and the label of the Wikipedia result
61+# ("Wayland (protocol) - Wikipedia") is typed.
62+key Super+Right
63+sleep 600ms
64+key Super+Shift+f
65+sleep 900ms
66+hint-read
67+sleep 200ms
6968 key Super+g
70-sleep 6s
71-# Types the label of the Wikipedia result: Firefox opens it.
72-hint-click Wikipedia
73-sleep 26s
69+sleep 1500ms
70+hint-click wikipedia protocol
71+# Firefox loads the page; the compositor writes no frame while it is still, so
72+# a pointer move marks the end of the hold.
73+sleep 4s
74+move 1500 900
@@ -1,8 +1,17 @@
1 move 1918 10781 move 1918 1078
2-sleep 12s2+sleep 5s
3+# Unseen warm-up while Firefox sits beside the terminal: a hint bind cancelled
4+# at once still runs its scan, without drawing labels. It connects to the
5+# accessibility bus early; more of them below (each right after a key press, in
6+# an existing pause) let Firefox expose its page content by the end.
7+key Super+g
8+key Escape
9+sleep 7s
3 key Super+Up10 key Super+Up
4 sleep 1500ms11 sleep 1500ms
5 key Super+Down12 key Super+Down
13+key Super+g
14+key Escape
6 sleep 1500ms15 sleep 1500ms
7 key Super+Shift+s16 key Super+Shift+s
8 sleep 500ms17 sleep 500ms
@@ -11,6 +20,8 @@ sleep 1500ms
11 key Super+220 key Super+2
12 key Super+Shift+121 key Super+Shift+1
13 key Super+122 key Super+1
23+key Super+g
24+key Escape
14 sleep 3s25 sleep 3s
15 key Super+Shift+s26 key Super+Shift+s
16 sleep 500ms27 sleep 500ms
@@ -18,8 +29,12 @@ key Super+s
18 sleep 1500ms29 sleep 1500ms
19 key Super+s30 key Super+s
20 key Super+Left31 key Super+Left
32+key Super+g
33+key Escape
21 sleep 2s34 sleep 2s
22 key Super+Right35 key Super+Right
36+key Super+g
37+key Escape
23 sleep 2s38 sleep 2s
24 key Super+Left39 key Super+Left
25 sleep 1s40 sleep 1s
@@ -40,34 +55,20 @@ sleep 1500ms
40 key Up55 key Up
41 sleep 1s56 sleep 1s
42 key Return57 key Return
43-sleep 2s58+sleep 700ms
44-# Back on workspace 1. Firefox builds its accessibility tree for page content59+# Back on workspace 1: focus Firefox and fullscreen it. Once the page is still,
45-# only after a few scans; a hint bind cancelled at once still runs its scan60+# read it, then hint mode starts and the label of the Wikipedia result
46-# without drawing labels, so these warm it up unseen.61+# ("Wayland (protocol) - Wikipedia") is typed.
47-key Super+162+key Super+Right
48-sleep 2s63+sleep 600ms
49-key Super+g64+key Super+Shift+f
50-key Escape65+sleep 900ms
51-sleep 2s66+hint-read
52-key Super+g67+sleep 200ms
53-key Escape
54-sleep 2s
55-key Super+g
56-key Escape
57-sleep 2s
58-# Clear the terminal's listing (OCR would take file names for links) and print
59-# a URL, which hint mode labels like Firefox's links. foot opens a URL only from
60-# its own keyboard mode, so the terminal's label is shown, not clicked.
61-move 400 600
62-click
63-type clear; echo https://github.com/bnema/neferwl
64-key Return
65-sleep 500ms
66-move 1918 1078
67-sleep 2s
68-# Hint mode: Firefox's links come from the accessibility tree, the URL from OCR.
69 key Super+g68 key Super+g
70-sleep 6s69+sleep 1500ms
71-# Types the label of the Wikipedia result: Firefox opens it.70+hint-click wikipedia protocol
72-hint-click Wikipedia71+# Firefox loads the page; the compositor writes no frame while it is still, so
73-sleep 26s72+# a pointer move marks the end of the hold.
73+sleep 4s
74+move 1500 900
M tools/demo/record.sh +18 -13
@@ -53,7 +53,7 @@ docker run -d --name "$container" --hostname neferwl-demo "${gpu[@]}" \
5353 -e XDG_STATE_HOME=/work/state -e NEFERWL_HINT_DEBUG=1 \
5454 -v "$work:/work" -v "$work/neferwl:/usr/local/bin/neferwl:ro" neferwl-demo \
5555 --backend=headless --size 1920x1080 --config /root/config --no-xwayland \
56- --screenshot /work/shots --input /work/input --timeout 120s >/dev/null
56+ --screenshot /work/shots --input /work/input --timeout 75s >/dev/null
5757 display=''
5858 for ((attempt = 0; attempt < 300; attempt++)); do
5959 [[ $(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null) == true ]] ||
@@ -90,11 +90,14 @@ launch_apps() {
9090 }
9191 feed() {
9292 # Forwards input.txt to the compositor in real time. `sleep` lines are
93- # served here (not forwarded), so a host-side step can sit between two compositor steps:
94- # `hint-click <word>...` types the label that hint mode gave the link
95- # showing one of the words, from the dump of the scan the last `hint` bind
96- # started (demo-hint-label reproduces the labelling of internal/core/hint.go).
97- local line cmd d mark label
93+ # served here (not forwarded), so host-side steps can sit between two
94+ # compositor steps:
95+ # hint-read reads the still page with OCR, in the background
96+ # hint-click <word>... types the label of the link that shows all the
97+ # words, from the dump of the scan the last
98+ # `hint` bind started (demo-hint-label reproduces
99+ # the labelling of internal/core/hint.go)
100+ local line cmd d mark label reader=''
98101 while IFS= read -r line || [[ -n $line ]]; do
99102 cmd=${line%% *}
100103 case $cmd in
@@ -103,9 +106,13 @@ feed() {
103106 d=${line#sleep }
104107 if [[ $d == *ms ]]; then d=$(printf '%d.%03d' $((${d%ms} / 1000)) $((${d%ms} % 1000))); fi
105108 sleep "$d" ;;
109+ hint-read)
110+ docker exec "$container" demo-hint-label ocr &
111+ reader=$! ;;
106112 hint-click)
113+ [[ -n $reader ]] && wait "$reader"
107114 # shellcheck disable=SC2086
108- label=$(docker exec "$container" demo-hint-label "$mark" ${line#hint-click }) ||
115+ label=$(docker exec "$container" demo-hint-label label "$mark" ${line#hint-click }) ||
109116 { echo "No hint label for: ${line#hint-click }" >&2; docker kill "$container" >/dev/null; return 1; }
110117 echo "Hint label for ${line#hint-click }: $label" >&2
111118 printf 'type %s\n' "$label" ;;
@@ -117,9 +124,7 @@ feed() {
117124 printf 'key Ctrl+Alt+BackSpace\n'
118125 }
119126 app foot --title 'System / NeferWL' bash -c \
120- 'fastfetch --logo small --structure OS:Kernel:WM:Terminal:Memory:Colors; eza -la --no-user --icons --color=always /etc | head -24
121- # A bare shell for the end of input.txt: no prompt, no echo, no line editing.
122- stty -echo; exec env PS1= bash --norc --noediting'
127+ 'fastfetch --logo small --structure OS:Kernel:WM:Terminal:Memory:Colors; eza -la --no-user --icons --color=always /etc | head -24; exec sleep infinity'
123128 # The first app is launched before t=0. FIFO input and shell launches now
124129 # share this epoch, independent of compositor/image/container startup time.
125130 SECONDS=0
@@ -138,10 +143,10 @@ wait "$feeder" || { echo 'Scripted input failed.' >&2; exit 1; }
138143 feeder=
139144 wait "$launcher"
140145 launcher=
141-# Hint mode needs both sources: the accessibility tree (Firefox) and OCR (terminal).
146+# Firefox's links come from the accessibility tree; OCR (tesseract) must still load.
142147 if grep -qE 'OCR unavailable|English OCR data missing' "$log"; then echo 'tesseract is missing in the image.' >&2; exit 1; fi
143-grep -qE '"targets":[1-9][0-9]*,.*"message":"hint scan"' "$log" ||
144- { echo 'No hint scan found targets:' >&2; grep -E '"component":"(hints|atspi)"' "$log" | cut -c1-300 >&2; exit 1; }
148+grep -qE '"atspi_targets":[1-9][0-9]*,.*"message":"hint scan"' "$log" ||
149+ { echo 'No hint scan found accessibility targets:' >&2; grep -E '"component":"(hints|atspi)"' "$log" | cut -c1-300 >&2; exit 1; }
145150 grep '"message":"hint scan"' "$log" | sed -E 's/.*"atspi_targets":([0-9]+).*"ocr_links":([0-9]+).*"targets":([0-9]+).*/hint scan: \1 accessibility + \2 OCR links = \3 targets/' | tail -3
146151 ffmpeg -hide_banner -loglevel warning -f concat -safe 0 \
147152 -i "$work/shots/frames.ffconcat" -vf 'fps=30,format=yuv420p' \
@@ -53,7 +53,7 @@ docker run -d --name "$container" --hostname neferwl-demo "${gpu[@]}" \
53 -e XDG_STATE_HOME=/work/state -e NEFERWL_HINT_DEBUG=1 \53 -e XDG_STATE_HOME=/work/state -e NEFERWL_HINT_DEBUG=1 \
54 -v "$work:/work" -v "$work/neferwl:/usr/local/bin/neferwl:ro" neferwl-demo \54 -v "$work:/work" -v "$work/neferwl:/usr/local/bin/neferwl:ro" neferwl-demo \
55 --backend=headless --size 1920x1080 --config /root/config --no-xwayland \55 --backend=headless --size 1920x1080 --config /root/config --no-xwayland \
56- --screenshot /work/shots --input /work/input --timeout 120s >/dev/null56+ --screenshot /work/shots --input /work/input --timeout 75s >/dev/null
57 display=''57 display=''
58 for ((attempt = 0; attempt < 300; attempt++)); do58 for ((attempt = 0; attempt < 300; attempt++)); do
59 [[ $(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null) == true ]] ||59 [[ $(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null) == true ]] ||
@@ -90,11 +90,14 @@ launch_apps() {
90 }90 }
91 feed() {91 feed() {
92 # Forwards input.txt to the compositor in real time. `sleep` lines are92 # Forwards input.txt to the compositor in real time. `sleep` lines are
93- # served here (not forwarded), so a host-side step can sit between two compositor steps:93+ # served here (not forwarded), so host-side steps can sit between two
94- # `hint-click <word>...` types the label that hint mode gave the link94+ # compositor steps:
95- # showing one of the words, from the dump of the scan the last `hint` bind95+ # hint-read reads the still page with OCR, in the background
96- # started (demo-hint-label reproduces the labelling of internal/core/hint.go).96+ # hint-click <word>... types the label of the link that shows all the
97- local line cmd d mark label97+ # words, from the dump of the scan the last
98+ # `hint` bind started (demo-hint-label reproduces
99+ # the labelling of internal/core/hint.go)
100+ local line cmd d mark label reader=''
98 while IFS= read -r line || [[ -n $line ]]; do101 while IFS= read -r line || [[ -n $line ]]; do
99 cmd=${line%% *}102 cmd=${line%% *}
100 case $cmd in103 case $cmd in
@@ -103,9 +106,13 @@ feed() {
103 d=${line#sleep }106 d=${line#sleep }
104 if [[ $d == *ms ]]; then d=$(printf '%d.%03d' $((${d%ms} / 1000)) $((${d%ms} % 1000))); fi107 if [[ $d == *ms ]]; then d=$(printf '%d.%03d' $((${d%ms} / 1000)) $((${d%ms} % 1000))); fi
105 sleep "$d" ;;108 sleep "$d" ;;
109+ hint-read)
110+ docker exec "$container" demo-hint-label ocr &
111+ reader=$! ;;
106 hint-click)112 hint-click)
113+ [[ -n $reader ]] && wait "$reader"
107 # shellcheck disable=SC2086114 # shellcheck disable=SC2086
108- label=$(docker exec "$container" demo-hint-label "$mark" ${line#hint-click }) ||115+ label=$(docker exec "$container" demo-hint-label label "$mark" ${line#hint-click }) ||
109 { echo "No hint label for: ${line#hint-click }" >&2; docker kill "$container" >/dev/null; return 1; }116 { echo "No hint label for: ${line#hint-click }" >&2; docker kill "$container" >/dev/null; return 1; }
110 echo "Hint label for ${line#hint-click }: $label" >&2117 echo "Hint label for ${line#hint-click }: $label" >&2
111 printf 'type %s\n' "$label" ;;118 printf 'type %s\n' "$label" ;;
@@ -117,9 +124,7 @@ feed() {
117 printf 'key Ctrl+Alt+BackSpace\n'124 printf 'key Ctrl+Alt+BackSpace\n'
118 }125 }
119 app foot --title 'System / NeferWL' bash -c \126 app foot --title 'System / NeferWL' bash -c \
120- 'fastfetch --logo small --structure OS:Kernel:WM:Terminal:Memory:Colors; eza -la --no-user --icons --color=always /etc | head -24127+ 'fastfetch --logo small --structure OS:Kernel:WM:Terminal:Memory:Colors; eza -la --no-user --icons --color=always /etc | head -24; exec sleep infinity'
121- # A bare shell for the end of input.txt: no prompt, no echo, no line editing.
122- stty -echo; exec env PS1= bash --norc --noediting'
123 # The first app is launched before t=0. FIFO input and shell launches now128 # The first app is launched before t=0. FIFO input and shell launches now
124 # share this epoch, independent of compositor/image/container startup time.129 # share this epoch, independent of compositor/image/container startup time.
125 SECONDS=0130 SECONDS=0
@@ -138,10 +143,10 @@ wait "$feeder" || { echo 'Scripted input failed.' >&2; exit 1; }
138 feeder=143 feeder=
139 wait "$launcher"144 wait "$launcher"
140 launcher=145 launcher=
141-# Hint mode needs both sources: the accessibility tree (Firefox) and OCR (terminal).146+# Firefox's links come from the accessibility tree; OCR (tesseract) must still load.
142 if grep -qE 'OCR unavailable|English OCR data missing' "$log"; then echo 'tesseract is missing in the image.' >&2; exit 1; fi147 if grep -qE 'OCR unavailable|English OCR data missing' "$log"; then echo 'tesseract is missing in the image.' >&2; exit 1; fi
143-grep -qE '"targets":[1-9][0-9]*,.*"message":"hint scan"' "$log" ||148+grep -qE '"atspi_targets":[1-9][0-9]*,.*"message":"hint scan"' "$log" ||
144- { echo 'No hint scan found targets:' >&2; grep -E '"component":"(hints|atspi)"' "$log" | cut -c1-300 >&2; exit 1; }149+ { echo 'No hint scan found accessibility targets:' >&2; grep -E '"component":"(hints|atspi)"' "$log" | cut -c1-300 >&2; exit 1; }
145 grep '"message":"hint scan"' "$log" | sed -E 's/.*"atspi_targets":([0-9]+).*"ocr_links":([0-9]+).*"targets":([0-9]+).*/hint scan: \1 accessibility + \2 OCR links = \3 targets/' | tail -3150 grep '"message":"hint scan"' "$log" | sed -E 's/.*"atspi_targets":([0-9]+).*"ocr_links":([0-9]+).*"targets":([0-9]+).*/hint scan: \1 accessibility + \2 OCR links = \3 targets/' | tail -3
146 ffmpeg -hide_banner -loglevel warning -f concat -safe 0 \151 ffmpeg -hide_banner -loglevel warning -f concat -safe 0 \
147 -i "$work/shots/frames.ffconcat" -vf 'fps=30,format=yuv420p' \152 -i "$work/shots/frames.ffconcat" -vf 'fps=30,format=yuv420p' \

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 25

Tags 10

↑↓ moveenter openesc close