main 191acbd

nefer-hints

Local clickable-UI candidate detector for NeferWL hint mode, using OmniParser v2 icon_detect (YOLO). A persistent C++17 subprocess runs ncnn on Vulkan, with CPU fallback; NeferWL stays Go with CGO_ENABLED=0. No screen content leaves the host. This runner proposes boxes, not calibrated clickability: classifier filtering, AT-SPI merging and compositor integration are not implemented here.

Build

Requires a C++17 compiler, CMake and system ncnn built with Vulkan (Arch: ncnn). No Python or torch dependency at runtime; dynamic linking to system ncnn.

cmake -S runner -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j8

Export (offline weights)

Python is only needed for export and validation. Reuse system torch; never replace it with pip. The pinned local additions are from the GPU experiment.

python -m venv --system-site-packages venv
venv/bin/pip install --no-deps -r poc/2026-10-21-gpu-latency/requirements-local.txt
mkdir -p weights
cp /tmp/omniparser-eval/weights/icon_detect/model.pt weights/model.pt
YOLO_OFFLINE=true HF_HUB_OFFLINE=1 OMP_NUM_THREADS=8 venv/bin/python export/export_ncnn.py

Produces ignored models/icon_detect-{640,1280}[-rect].ncnn.{param,bin} via Ultralytics TorchScript and the pnnx CLI, fp16=0. Dynamic pnnx export left an unsupported expression layer; the reproducible fixed-shape fallback uses square and 16:9 rectangular graphs. Explicit crops and non-16:9 frames use square letterboxing; 16:9 whole frames match Ultralytics minimal rectangular padding.

Run and protocol

build/nefer-hints --model models --imgsz 1280 --conf 0.05 --iou 0.7 --gpu auto

An optional model.json beside the model supplies defaults, for example {"name":"terminal-v1","conf":0.075,"iou":0.4}. Precedence is built-in conf 0.05 / IoU 0.7 < model metadata < explicit --conf / --iou flags, regardless of argument order. Effective settings and the optional name appear on stderr. Directory models use DIR/model.json; prefixes and .ncnn.param paths try <prefix>.json first, then model.json in that directory.

This is not a PNG command-line reader: stdin/stdout carry the binary protocol. After load and warmup, NHNT announces readiness and the device. Each NHRQ contains a sequence number, BGRA frame and optional independent crop regions; NHRS returns scored boxes in frame coordinates or an error. All fields are little-endian; stderr is logs only. EOF between requests exits cleanly. See the complete protocol and Python client.

Verify

venv/bin/python -m unittest discover -s tests -v
venv/bin/python tests/validate.py

Seven verified scenes, CPU torch comparison, protocol latency and memory results: poc/2026-10-21-ncnn-runner. Earlier experiments, frames and annotations live under poc/.

License

AGPL-3.0: detector weights and Ultralytics are AGPL. NeferWL (GPLv3) can call it as a subprocess.

Models and data

Base weights are microsoft/OmniParser-v2.0 icon_detect (AGPL-3.0). terminal-v1 and terminal-v2 are fine-tuned derivatives under the same licence. eval/frames contains screenshots of third-party public sites for evaluation, including Wikipedia and MDN content under CC BY-SA. Attribution belongs to their respective owners; scene URLs are recorded in eval/scenes.json.

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. On pages with a side panel: into the panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit; in a file, click a line number to select it, shift-click for a range
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 3

Tags 0

    ↑↓ moveenter openesc close