dandri/meshcore-analyzer
1
0 Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-11-11 12:37:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,132 Commits 85 Branches 42 Tags
d083f9ade91dbcbc7aa577bbf7b2248ce21ebf09
Commit Graph
297 Commits
Author SHA1 Message Date
TeTeHackoandClaude Opus 5.5 417c8dd00a feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) (#2300) 
Closes #2270, closes #2271.

The ingestor took only the confirmed region scopes out of an observer's
`/neighbors` report (#1865) and dropped the rest. Nothing could
therefore say when an observer last reported, or whom it heard.

## Ingestor

`observer_neighbor_reports` (created in `dbschema.Apply`) keeps the
latest report of each observer, one row per observer, replaced by every
live report.

The report is unauthenticated, so `neighbors[]` is stored sanitised:

- only 64-hex keys
- `snr` and `heard_secs_ago` finite and in range
- `status` limited to `[a-z_]`
- `scopes` clamped
- at most 500 rows

`total_neighbors` is believed only when it is not below the number of
rows that arrived.

A **retained replay** only fills a missing row, and it is dated by the
report's own `timestamp`. After an ingestor restart, a long-dead
observer therefore does not look as if it just reported.

## API

The table is optional for the server (no `AssertReady` change); without
it the fields are simply absent.

- `GET /api/observers`: `neighbors_report_at` and `neighbors_count`,
from one batch query in the existing cached builder.
- `GET /api/observers/{id}`: also `neighbor_report`, which is
`{received_at, reported_at, neighbor_count, queried_neighbors,
self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr,
heard_secs_ago, status, scopes}]}`. Names and roles come from the node
table in one query.

## Frontend

`public/observer-neighbors.js` is shared by both pages and escapes
everything.

- **#2270, observers list:** a sortable **Neighbors** column, e.g. `3h
ago · 12`, or `—` when the observer never reported. The same value
appears in the slide-over.
- **#2270, observer page:** **Neighbors Report** and **Neighbors** stat
cards.
- **#2271, observer page:** a **Neighbors** table listing each neighbor
by name (a link to the node) or key prefix, with SNR, heard, scope query
status and scopes. The heading says when the list is shorter than the
count, e.g. `(2 of 3 listed)`.

## Tests

- **ingestor:** sanitising, the cap, live replace, the retained rules
(fill only, dated by the report, undated ignored) and the MQTT topic end
to end. The existing #1865 scope tests are unchanged and still pass.
- **server:** the list summary, the detail with names, and both
endpoints without the table.
- **frontend:**
  - unit tests of the helpers
- an XSS case for every rendered field; I checked that it fails when
escaping is removed
- `test-observers-headings` now expects the new column after `Last
Packet`
- **E2E:** `test-observer-neighbors-2270-e2e.js` intercepts the observer
endpoints, since the fixture has no reports. It is listed in
`scripts/non-unit-tests.json` and on shard 2.

Gates that pass:

- `go test -race` in `cmd/server`, `cmd/ingestor` and
`internal/dbschema`
- `sh test-all.sh`
- `eslint@8`, with 0 errors
- the XSS diff gate
- these E2E suites: the new one,
`test-issue-1639-observers-sort-e2e.js`,
`test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js`

Performance: the list adds one primary-key-sized query per cache fill,
and the detail adds two small queries per request.

Also fixes #2330 (folded in from #2337, second commit).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-29 22:36:53 +01:00
efitenandClaude Opus 5.5 ddce76a685 ci(e2e): run the E2E shards on ubuntu-24.04 and fail on a broken browser install (#2350) 
`ubuntu-latest` now hands out `ubuntu-26.04` runners. Every E2E shard
that lands on one fails before a single test runs:

```
browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium_headless_shell-1208/...
```

The step "Install Playwright browser" runs `npx playwright install
chromium 2>/dev/null || true`, so its own failure is invisible and the
job only breaks at the first browser launch.

## Evidence

- Run 40779894898 (PR #2349, a mail-text change): three attempts, each
failing on `Executable doesn't exist`. The job log shows `Image:
ubuntu-26.04`, OS `26.04.1 LTS`.
- The master run at 2026-10-28 21:31Z passed. Its shard log shows
`Image: ubuntu-24.04`.
- `playwright` in package-lock.json is 1.58.2.

## Change

- `e2e-shard` gets `runs-on: ubuntu-24.04`, with a comment saying when
to unpin. The other jobs do not install Playwright and stay on
`ubuntu-latest`.
- "Install Playwright browser" no longer has `2>/dev/null || true`, so a
failed install fails at that step.

## Not done

- I did not find out why the install fails on 26.04: its stderr was
discarded. Once this is merged, a run with the pin removed would show
it.
- No Playwright upgrade. That is a separate change, and it may make the
pin unnecessary.
- Before this change, `install-deps` errors were also swallowed. If it
fails on 24.04, this PR's own CI will show it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-29 00:07:30 +01:00
TeTeHackoandClaude Opus 5.5 660cbbf5ff deps: golang.org/x/net v0.61.0, x/crypto v0.58.0, paho.mqtt.golang v1.5.1; runtime alpine:3.24 (#2297) 
A Trivy scan of the published image found **19 HIGH and 20 MEDIUM** in
Go dependencies, plus a runtime base past its support window. There is
no Dependabot here, so this collects the bumps in one PR.

| dependency | from | to | why |
|---|---|---|---|
| `golang.org/x/net` (ingestor) | v0.27.0 | v0.61.0 | CVE-2024-45338 and
later advisories, fixed up to v0.60.0 |
| `golang.org/x/crypto` (server, `internal/users`) | v0.31.0 | v0.58.0 |
CVE-2025-22869, CVE-2025-47913 and later; `x/sys` follows to v0.49.0 |
| `github.com/eclipse/paho.mqtt.golang` | v1.5.0 | v1.5.1 |
CVE-2025-10543, an integer overflow in the UTF-8 string encoding |
| runtime image | `alpine:3.20` | `alpine:3.24` | 3.20 is out of
support; its musl 1.2.5-r3 carries CVE-2026-107891 |

The Alpine change covers `Dockerfile`, `Dockerfile.go`, the arm64 smoke
step in `deploy.yml` and the comment in
`docker/test-select-service-user.sh`.

**What changes besides the versions:**

- The new `x/` modules declare `go 1.26.0`, so the `go` directive of
`cmd/server`, `cmd/ingestor` and `internal/users` moves up from 1.22.
The builder image and CI already run 1.27.
- `x/sync` follows paho to v0.17.0.

**What is left:** only `GO-2026-5932`. It flags `x/crypto/openpgp` as
unmaintained; nothing here imports that package and no fix exists.

## Checks

- Every module passes `go mod tidy`, `go build` and `go vet`.
- `go test -race` passes in `internal/users`, `cmd/ingestor` and
`cmd/server`.
- A local image build on alpine 3.24.3:
  - ships mosquitto 2.1.2, caddy v2.11.4 and supervisor 4.3.0
- keeps the caddy and mosquitto accounts at the same uid:gid (100:101,
101:102), so `test-select-service-user.sh` still describes the image
(116 assertions pass)
- comes up healthy, ingests an MQTT status through mosquitto and serves
the frontend

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 21:09:01 +01:00
TeTeHackoandClaude Opus 5.5 77a5f14925 feat(observers): radio setting per observation from the observer's radio history (#1869) (#2292) 
Part of #1869.

## Problem

`observers.radio` keeps only the **last** `MHz,kHz,SF,CR` an observer
reported. Once an observer is retuned (say from SF7 to the EU narrow SF8
setting to listen to a neighbouring mesh), every packet it ever heard
reads as heard on the new setting, and there is no way to tell an SF7
packet from an SF8 one.

## What this does

As #1869 proposes, the packet data model is untouched: the setting of an
observation is resolved from its observer and time.

**Ingestor** (all schema in `dbschema.Apply`)

- `observer_radios`: one row per stretch of time an observer, and each
radio of a multi-radio node, stayed on one setting. Every live status
extends the current row and a change starts a new one; retained replays
never extend a row. On the first run it is seeded with every observer's
current radio, which is what the UI already showed for the past.
- Settings are stored canonical through `dbschema.CanonicalRadio`, so
`869.4320068,62.5,7,5` and `869.432,62.50,7,5` are one setting.
- `observation_radios` (`observation_id → rx_radio_id`): multi-radio
openHop nodes tag each packet with `rx_radio_id` and list `radios: [{id,
radio}]` in their status, so their observations resolve per radio.
- The table is sparse: single-radio nodes never send the field, so
`observations` and every query that scans it stay untouched.
  - The row id comes from the observation upsert via `RETURNING id`.
- Rows are pruned with their transmissions, and a purged observer takes
its history along.

**Server** (read-only)

- No `AssertReady` change: an absent table reads as empty and is probed
again later, the same rule as `advert_route_evidence`.
- The history is held in memory. The live poller checks it on every tick
(two primary-key lookups), so a retune or a new `rx_radio_id` is known
before the observation is broadcast. A full reload runs every 10 min.
- `radio` is added to observations, packets, grouped rows (taken from
the header observation) and the live feed.
- `GET /api/packets?radio=MHz,kHz,SF,CR` filters by setting.
- Combined with `observer=`, it means *that* observer heard it on that
setting.
- A radio query is served from memory instead of falling back to SQL,
which would return the window unfiltered.
- `GET /api/radios` (named struct) and `radio_history` on `GET
/api/observers/{id}`.

**Frontend**

- Packet filter fields `radio`, `freq`, `bw`, `sf`, `cr`, e.g. `sf == 8`
or `freq > 869.5 && bw == 62.5`.
- A Radio row in the packet detail and a Radio column in its
observations table (only when the server reports one).
- A radio history table on the observer page once there is more than one
stretch.
- `RfParams.radioText` formats the setting. It escapes all five
characters and returns non-numeric input as it came, escaped.

## Performance

`txMatchesRadio` runs once per observation inside `filterPackets`.

- Each observation costs one map lookup by observer id.
- Observers with a single stretch (nearly all) answer from that lookup
directly.
- Retuned observers add a binary search over their few stretches.
- Multi-radio observers add one lookup by observation id.
- The snapshot is loaded once per filter call.

`BenchmarkTxMatchesRadio`, 30K packets / 150K observations (20
observers, 2 of them retuned):

```
BenchmarkTxMatchesRadio-24    602    2056924 ns/op    0 B/op    0 allocs/op
```

Memory is one entry per stretch plus one per multi-radio observation;
nothing is added per observation otherwise.

## Tests

- **dbschema:** seed runs once, canonical spellings, the tables exist
without `observers.radio`.
- **ingestor:**
  - stretches extend and split; canonical spellings do not split
- two radios of one node hearing one transmission over two paths each
keep their own radio
  - a retained status does not write history
  - prune and purge
  - `rx_radio_id` reaches `PacketData`
- **server:**
- resolution before, between and after stretches, and with an unknown
radio id
  - `observer` + `radio`, and grouped rows carry `radio`
  - incremental refresh, `/api/radios`, `radio_history`, missing tables
  - the benchmark above
- **frontend:** the new filter fields in
`tests/unit/test-packet-filter.js`, and XSS cases through the real
`RfParams.radioText` and the history renderer.
- **E2E:** `tests/e2e/test-radio-setting-1869-e2e.js` checks the grouped
rows' radio, `sf ==` on the packets page, the detail Radio row and the
observer history. It is added to `scripts/non-unit-tests.json` and to
shard 2.

`go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema`
passes, along with `sh test-all.sh`, `eslint@8` (0 errors) and the XSS
diff gate. The new E2E test and `test-e2e-playwright.js` pass against
the fixture.

## Browser validation

This runs on a live instance with six observers, one of them retuned to
869.618/62.5/SF8/CR8:

- The packet detail shows `Radio: 869.432 MHz · SF7 · BW 62.5 kHz · CR
4/5`.
- `sf == 7` shows 106 of 106 grouped rows and `sf == 8` shows 0 of 107;
the SF8 observer has not heard anything yet.
- `/api/radios` lists both settings.
- No page errors.

## Notes

- **The customizer (rule 8)** gets nothing: the only constants are
server-side refresh intervals.
- **Attribution timing:** packets heard between a retune and the
observer's next status message (minutes) are attributed to the previous
setting, because a stretch starts when the status is handled. This is
documented at `RecordObserverRadios`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 21:08:40 +01:00
n30nex e61ff6e473 fix(observers): keep detail responses tied to the current view (#2323) 
Fixes #2316 and #2320. Related to #1778; progressive loading remains
separate.

Delayed observer responses could replace a newer observer or period,
overwrite its error/status/calibration, and leave detached charts alive.
Each load now captures its observer and days and checks ownership before
success or failure can update the view. Leaving the page invalidates
pending loads, including same-observer revisits.

New visits also reset the request window to the selector's existing
seven-day default. Previously, a new page could display seven days while
requesting the preceding visit's 30-day window.

Production change: 12 added lines and three replacements in one frontend
file. No new requests, caches, settings or dependencies; the ownership
check is O(1). Existing request deduplication, comparison controls,
chart cleanup and current-request errors are preserved.

Validation: 11 unit checks, all 219 standalone suites, 32 controlled
browser cases across desktop/mobile and both themes, comparison 9/9, and
observer-IATA 5/5. Core browser run exited successfully with 149 passes
and three declared fixture skips; existing conditional subchecks remain.
The unchanged backend tree passed both full backend suites. The parent
independently reran the new unit/browser tests.

Test-only RED `a73c227f` produced [nine unit and 28 browser assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40239824064).
Three independent reviews passed. [Final-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40240673369)
passes on `ba52d873` integrated with upstream `11ce1235`, including all
browser shards and architecture builds.
 2026-10-26 15:42:56 +01:00
n30nex 410db29fe6 chore: remove unused legacy Dockerfiles (#2314) 
Fixes #2310. Partial follow-up to #1385 (PR-C).

Remove unused `Dockerfile.go` and `Dockerfile.node`, which describe
retired alternate builds. Current workflows, Makefile, Compose variants
and operator scripts already select the canonical `Dockerfile`.

Remove the obsolete formatter exclusions and label the old Node-to-Go
migration guide historical, linking current deployment instructions.
Historical release prose remains untouched. The canonical image, runtime
files, dependencies and Compose behavior are unchanged.

Validation: audited all 1,501 tracked files; old and new formatter
inputs are exactly the same 633 real Go files. Both formatter commands,
YAML/shell/workflow-reference checks, existing workflow/routing tests,
COPY gate and test inventory pass. All 218 standalone frontend suites
pass. The unchanged backend tree passed full server/ingestor suites, and
the unchanged app passed 12 browser smoke cases.

TDD exemption: behavior-preserving cleanup/refactor plus documentation;
no test files modified. `make` and `actionlint` were unavailable
locally; equivalent formatter and focused workflow checks ran directly.

[CI passed on this
head](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40230911566),
including canonical container builds and all browser shards. Three
independent reviews passed. The published historical guide and its
current-deployment links render correctly in Chromium. No
settings/customizer changes.
 2026-10-26 15:41:34 +01:00
n30nex 4ea80da234 ci: run JavaScript unit tests before backend validation (#2322) 
Fixes #2317. Partial follow-up to #1995.

JavaScript assertion failures currently reach their unit gate only after
the Go validation work. Move the existing `Run JS unit tests` step
directly after Go setup, before gofmt/vet and backend builds/tests.
Failed unit checks then stop later work under the existing default
failure behavior.

This is an exact three-line block relocation within the same job. Its
command remains `sh test-all.sh`. All ten jobs, twenty step
bodies/options/conditions, other step ordering, dependencies, security
checks, coverage outputs and publication gates are unchanged. No new
job, package, timeout, cache or application setting.

Validation: dependency audit found no need for prior Go outputs. All 218
standalone suites passed with no checkout build artifacts or external
Node packages, after earlier shared Go tests had finished and cleaned
up. The parent reran the suite, parsed-workflow equivalence checks and
browser smoke. The unchanged backend tree passed both full backend
suites.

TDD exemption: configuration-only; no test files modified, and CI is
green without test edits. Three independent reviews passed. [Final-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40238885639)
confirms the unit step precedes Go validation; backend checks, all
browser shards and architecture builds passed. No numerical speedup or
successful-pipeline duration claim is made.
 2026-10-26 15:40:11 +01:00
efitenandClaude Opus 5.5 e149568c6c feat(observer-logins): personal MQTT logins for observers (#2306) 
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.

Closes #2319.

The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.

## Changes in this PR

**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.

**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.

**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.

Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.

**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.

**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.

**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.

**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.

## Enabling it on an instance

Full steps in `docs/observer-mqtt-logins.md`. In short:

1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.

## Who can claim an observer

Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.

What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.

## Performance

`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.

## Verification

- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.

- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.

## Known behaviour

EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.

## Not in this PR

- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.

Note on cb9f4ff3: its message says the scheme problem was reported by an
operator. That report turned out to come from before the broker used the
hook, so it does not show this cause; the change rests on the firmware
source only.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 09:50:20 +01:00
efitenandClaude Opus 5.5 dc98886d92 feat(nodes): name and show repeaters known only from the name they reported (#2238) (#2239) 
Part of #2238. Path hops through such a repeater (`/api/nodes`,
`/api/resolve-hops`, the frontend hop resolver) still show a bare
prefix; that part of the issue stays open.

## Summary

A repeater that never sends an advert can still tell a nearby companion
its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and
publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`.
This PR stores those answers, resolves names with them, and shows these
repeaters on the map, in the nodes list and in search, marked as
self-reported.

### Ingestor: store the names
- `367ecde1` adds a new `names` subtopic.
  - It is gated on `clientNodeNames.enabled`, default off.
- Answers go into a new table, `client_node_names`: one row per
observation, `UNIQUE(target, rx_pubkey, observed_at)`.
- **Validation.**
  - `target` must be 64 hex.
- The name is trimmed and rejected when it is empty, over 31 runes
(firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8,
or contains a control character.
- **Retention.** `retention.clientNodeNamesDays` (0 = keep).
`config.example.json` sets 30, like the other client tables: a companion
only asks a repeater for its name when CoreScope has none (`nameMissing`
in CoreDrive's `src/names.js`), so pruning is also how a renamed
repeater gets its new name.
- **Compatibility.** An ingestor without this change drops `/names`
silently, so CoreDrive v1.20.0 works against either version.

### Server: resolve by the reported name
- `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and
`client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys`
(the coverage tooltips) both use it.
- **Which name wins.** The advert name, otherwise the newest reported
name by `observed_at`. A prefix that matches different pubkeys across
the two tables stays ambiguous.
- **Only repeaters heard directly.** A reported name counts only once
its target has at least one `discover` or `advert` reception in
`client_receptions`, keyed by the full pubkey, or by its 8-byte prefix
when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names`
starts with that prefix (`clientNameHeard`). A name for a pubkey nobody
heard adds no candidate, so it cannot make a real node's prefix
ambiguous. Without the uniqueness check, a made-up pubkey that shares a
heard repeater's first 8 bytes borrowed that repeater's receptions and
passed (fixed in `4c569f91`).
- **Filters.** The blacklist and the hidden name prefixes apply to both
sources.
- **Missing table.** A database without the table resolves from `nodes`
alone, and the server checks for the table again until the ingestor has
created it.
- **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?,
name?, ambiguous}`.

### Server: estimated position
- `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`.
It lists every repeater in `client_node_names` that never sent an advert
(no row in `nodes` or `inactive_nodes`) and was heard directly at least
once (see above). A node with an advert but no position stays out: it is
already in `/api/nodes` under its advert name.
- **Gated** like the other endpoints on `client_receptions`: 404 when
`clientRxCoverage.enabled` is off, because the counts and the estimate
come from companion GPS fixes.
- **Which receptions count.** Only direct ones:
- `discover` and `advert` rows, keyed by the full pubkey, or by its
8-byte prefix when no other known pubkey shares it;
  - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone.

  `geo` rows are left out.
- **How the estimate is made.**
- Receptions within 100 m form one spot, with the best SNR of that spot.
  - An estimate needs at least 3 spots, at least 200 m apart.
- The position is the SNR-weighted centroid (`10^(snr/20)`). The radius
is the distance to the farthest spot, at least 500 m.
- The response is cached for 5 minutes, and the compute time is logged.

### Map
- `5fcde554` adds `public/map-estimated.js`, which draws each estimate
on its own layer:
  - a hollow, dashed "≈" marker and a dashed uncertainty circle;
- a popup with an "Estimated position" badge, the method, the caveat
that one-sided driving skews the estimate, and a link to the coverage
map;
- a Display toggle, "Estimated positions": on by default, kept in
localStorage, and also hidden by the repeater role filter and in route
view.

### Nodes page and search
- `12e6b345` adds a "Self-reported repeaters" section under the nodes
table, on the All and Repeaters tabs, filtered by the search box.
- **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a
self-reported detail view replaces "Node not found":
  - who reported the name;
  - the receptions;
  - the estimate, or why there is none;
  - links to the map and the coverage map.

  An unknown key still shows "Node not found".
- **Header search** adds matching self-reported repeaters, tagged
"self-reported".
- **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as
they are. These repeaters have no packets, so the packet filters, the
live view and the channel sender lookup must not match them.

### Docs
- `03580437` corrects the `/regions` section of
`docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the
switch, but the code uses `clientRegions.enabled`.
- The new subtopic, endpoint and config keys are documented in the same
file.

## Trust

Any companion the broker lets publish can report a name for any
repeater, and the newest report wins. This is the same model as
`/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to
linked companions.

A report for a repeater nobody heard directly is stored but not used: it
is not listed and adds no prefix candidate. A companion can still report
a wrong name for a repeater that was heard. The same companion can also
publish a fake `discover` reception for a made-up pubkey, so the "heard
directly" check filters junk, not a malicious companion;
`requireLinkedCompanion` is the control for that (its gate runs before
the `/names` handler too).

## Performance

- **`lookupPrefixes`** adds one term per key on `client_node_names`.
- It matches the prefix as a range, `target >= ? AND target < ?` (upper
bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive
column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX
idx_cnn_target (target>? AND target<?)`, and the reception check is
`SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`.
- It is still one query per chunk of 200 keys: 400 compound terms, under
SQLite's default limit of 500.
  - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks.
- **`/api/nodes/reported`** makes three kinds of query:
  - one grouped query on `client_node_names`;
  - one `lookupPrefixes` call for the short hashes;
- one `IN` query per 400 keys on `client_receptions`, served by
`idx_client_recept_heard_geo`.

The spread check is O(spots²) per repeater, with at most one spot per
100 m visited. The result is cached for 5 minutes; concurrent misses
wait for one computation, and the response is written after the lock is
released.
- **The map does not wait for it.** It draws its markers first and adds
the estimates when the list arrives. The map and the nodes page share
one cached fetch (`ReportedNodes.load()`).
- **Not benchmarked.** On the instance where this runs,
`client_node_names` holds one row today. I expect at most tens of such
repeaters per instance, but have not measured that.

## Notes

- **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2
occurrences: the handler's `msg` parameter and its `gps` field. That is
the decoded MQTT message, passed in the same shape as
`handleClientPacket` and `handleClientRegions` already take it. No API
response or internal structure uses it.
- **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min
TTL. They should move to the customizer later (rule 8).
- **Map filters.** The estimates follow the "Estimated positions" toggle
and the repeater role filter, not the area, region or last-heard
filters. The toggle's tooltip says so.
- **Name characters.** Names are rejected for control characters but not
for Unicode format characters (bidi overrides, zero-width joiners),
because emoji sequences in node names use the zero-width joiner. All
output goes through `escapeHtml`.
- **Not done:** the home-page search box (`home.js`, `/api/nodes/search`
with a Claim button) does not list these repeaters. Claiming a node
without adverts would have nothing to show.

## Tests

On this branch (based on `550829af`):
- `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and
`cmd/ingestor`.
- **New Go tests:**
  - handler and validation cases;
  - idempotent redelivery;
  - newest `observed_at` wins;
  - prune;
  - the gate on and off, without falling through to the observer path;
- client-only node, nameless `nodes` row, advert name wins, ambiguity
across the two sources;
  - hidden and blacklisted nodes;
  - absent-then-created table;
  - spot merging, threshold, weighted centroid and radius floor;
- `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an
advert position and a missing table;
- an advert without a position, a never-heard name, a name heard only in
`rxlog` paths, and the `clientRxCoverage` gate (404). These fail on
`12e6b345` and pass at head.
- a made-up pubkey sharing a heard repeater's 8-byte prefix, and two
reported targets sharing one (`4c569f91`). Both fail without that
commit.
- **New frontend unit suites:** `tests/unit/test-map-estimated.js` and
`tests/unit/test-reported-nodes.js` pass. Both load the real
`escapeHtml` from `app.js`.
- **`test-all.sh`** passes, all suites (with Python available for the
XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the
four changed frontend files.
- **E2E**, against a static server with a mocked API:
- `test-map-estimated-e2e.js`: 6/6, including the repeater role filter
and route view hiding the estimates. Mutating either hide makes its step
fail.
  - `test-reported-nodes-e2e.js`: 6/6.
  - `test-issue-1150-404-state-e2e.js`: 5/5.
- **Real data, API only.** On the fork's staging, `/api/nodes/reported`
lists one repeater: 5 spots, 17 receptions, an estimate with a radius of
2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays
listed under the stricter rules. Not re-checked after `4c569f91`.
- **Not done:** a check of that repeater in a browser.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 15:17:58 +02:00
efitenandClaude Opus 5.5 d43f3c3777 feat(theme): ?theme=dark|light URL parameter for one page load (#2236) (#2251) 
Fixes #2236.

A display that can only be given a URL (digital signage, an iframe
without cookies or login) had no way to choose dark or light. This adds
`theme=dark` or `theme=light` to the hash query, next to `embed=1`:

```
https://<instance>/#/map?embed=1&theme=dark
```

## What changes

**`public/index.html`, the pre-paint script**
- Reads `theme` from the hash query with `URLSearchParams`, as
`shouldEmbedRoute` does for `embed`. Only `dark` and `light` count;
anything else is ignored.
- A valid value wins over the saved choice and the OS preference, and is
set before first paint, so there is no light flash.
- The value is handed to `app.js` as `window.__themeOverride`.
- When `localStorage` throws (a sandboxed iframe), the URL value is
still applied.

**`public/app.js`, the dark-mode setup**
- On load, a URL value is applied through `applyTheme(theme, false)`,
which does not write `meshcore-theme`. The saved choice, and the account
settings that `settings-sync.js` syncs from that key, stay as they were.
- Theme changes from other tabs or from settings sync (the `storage`
listener) are ignored while the URL value holds. Otherwise a logged-in
display would flip within a minute.
- A click on the dark-mode toggle ends the URL value and is saved as
before.

It works on every page, not only on the two `embed=1` pages. The value
is read once per page load, so later hash changes (navigation, map
viewport) keep it.

## Tests

| File | Assertions | Covers |
|---|---|---|
| `tests/unit/test-theme-url-param-2236.js` | 8 | Runs the real
pre-paint script from `index.html`: `dark` and `light` win over a saved
choice and the OS, nothing is written to `localStorage`, any page and
position in the query, an unknown value, no param, `localStorage`
throwing |
| `tests/e2e/test-theme-url-param-2236-e2e.js` | 6 steps |
`#/map?embed=1&theme=dark` over a saved "light" (dark, embed, saved
still "light"), a reload without the param, `theme=light` over a saved
"dark", a storage event while the override holds, the toggle ending it,
`theme=blue` |

How I checked them:
- With `index.html` and `app.js` from master, 6 of the 8 unit assertions
fail, and the E2E fails at step 1 ("expected dark + embed + saved
light").
- With only the `app.js` change reverted, the E2E fails at step 1. With
only the `if (themeOverride) return;` line in the `storage` listener
removed, it fails at step 4.
- `sh test-all.sh` passes, except `test-issue-1956-release-routing.js`,
which I skipped: it needs the workflow's `GITHUB_REF_NAME` and does not
run in local git-bash. `npx eslint public/app.js` reports 0 errors (18
warnings, the same count as master).
- E2E on a fixture seeded as in CI:

  | Suite | Result |
  |---|---|
  | `test-e2e-playwright.js` | 148 passed, 3 skipped |
  | `test-customize-theme-e2e.js` | 8/8 |
  | the new #2236 suite | 6/6 |

- `test-logo-theme-e2e.js` fails locally with "wordmark `<text>`
overflows SVG viewBox", with the same numbers on master. It measures
font metrics, which differ on this Windows machine.

The new E2E is registered in `scripts/non-unit-tests.json` and
`deploy.yml` (shard 2, next to the logo theme suites); the unit test is
in `test-all.sh`.

## Not done

- No screenshot: the E2E checks `data-theme` in headless Chromium, not
how the page looks.
- `live.js` Matrix mode forces dark and, when switched off, saves the
theme it restores. With a URL value active, turning Matrix off saves
that value as the stored choice. I left this as is, because a display
that only shows a URL does not toggle Matrix mode.
- Only `theme=`; the `?darkMode=1` form from the issue text is not
supported, so there is one form to document.
- I placed the E2E in shard 2 without measuring suite times.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 11:40:25 +02:00
efitenandClaude Opus 5.5 3960fadd96 fix(pages): drop late responses that belong to an earlier page visit (#2218, #2230, #2200) (#2248) 
Fixes #2218, fixes #2230, fixes #2200.

Node analytics, the account page and analytics await their data, then
write into the shared `#app` container. A response that lands after the
user has navigated away replaces the new page. In #2218's reproduction,
the Nodes list, with 200 rows, became an Analytics heading.

## The shared part

`public/app.js` keeps one view generation, `viewGeneration()`:
- `navigate()` bumps it before the old page's `destroy()`, so every page
init counts as a new visit, including a switch to another node of the
same page type.
- A page captures the value at init, and after each `await` drops the
result when the value has moved on.

`.eslintrc.json` declares it as a global.

## Per page

**#2218, `node-analytics.js`**
- `loadAnalytics` and `loadBatteryChart` check the generation on both
success and error. A page-local `loadSeq` makes a time-range click a new
load on the same visit, so an older range can't land over a newer one.
- `NodeHopAnalytics.load` already checked `container.isConnected`, so it
is unchanged.

**#2230, `account.js`**
- The render after `CSAuth.ready()` runs only while the generation is
unchanged.
- `destroy()` was empty. It now clears `shown.app` and `shown.userId`.

**#2200, `analytics.js`**
- A page-local `_loadSeq` sits next to the generation. Results and
errors of an older window/region selection or an earlier visit are
dropped, so a late failure of an old load no longer shows an error over
newer results.
- The bulk load renders only the six tabs drawn from its data. A tab
that loads its own data (Prefix Tool, Nodes, Roles, ...) is no longer
rebuilt or replaced by the bulk result: a deep-linked one renders at
init, and a window, region or area change re-renders it directly.
- `renderTab` guards the six tabs drawn from the bulk data (overview,
rf, topology, channels, hashsizes, collisions). Until the data arrives
they show "Loading analytics…", or the current load's error. Overview no
longer dereferences an undefined `rf`.
- Three related fixes in the same lines:
  - The error message is now escaped with `esc()`; it was inserted raw.
- `destroy()` removes the RegionFilter and AreaFilter listeners. They
piled up on every visit and called `loadAnalytics` from other pages.
  - `destroy()` resets the stored error.

## Tests

| File | Assertions | Covers |
|---|---|---|
| `tests/unit/test-issue-2218-node-analytics-stale-render.js` | 12 |
Loads the real router: the generation rises on every init, including
another node. Late success and error after leaving, a stale visit of the
same node, an older range landing last, late battery success and error.
The current response, battery and error still render |
| `tests/unit/test-issue-2230-account-stale-render.js` | 6 | Leaving
before ready (login view and "not enabled"), re-entering before ready,
the current render, an auth change after `destroy()` |
| `tests/unit/test-issue-2200-analytics-stale-load.js` | 19 | An older
selection landing last, a late rejection of an older selection, leave
and re-enter (success and error), the current error rendered escaped and
kept on a tab click, Overview and RF clicked before the data |
| `tests/e2e/test-issue-2218-node-analytics-stale-e2e.js` | 4 steps |
Holds `/api/nodes/*/analytics` with `page.route`, returns to `#/nodes`,
then releases the response as a 200 and as a 500. The 200 rows stay and
no Analytics heading appears; a normal visit still renders |

How I checked them:
- With the production files reverted and the tests kept, 7 of 12, 4 of 6
and 7 of 12 unit assertions fail. With the fix, all pass.
- The E2E fails on the unfixed frontend with "Nodes rows 200 before
release, 0 after".
- Single mutations, each reverted:
- removing the success check in `loadAnalytics` (analytics) fails 2
assertions;
  - removing the check after `CSAuth.ready()` fails 3.

Other test changes:
- `test-analytics-theme-refresh-before-data.js` and
`test-user-management-ui.js` get a `viewGeneration: () => 0` stub.
- The new E2E is registered in `scripts/non-unit-tests.json` and
`deploy.yml` (shard 2, next to the #748 suite). The unit tests are in
`test-all.sh`.
- The broad suite's "Node side panel Details link navigates" test
selected an anchor that now opens Analytics, as #2218 notes. It now
clicks `#nodesRight .node-detail-btn`.

Results:
- `sh test-all.sh` passes, and `npx eslint public/*.js` reports 0
errors.
- E2E on a fixture seeded as in CI:

  | Suite | Result |
  |---|---|
  | `test-e2e-playwright.js` | 148 passed, 3 skipped |
  | `test-user-management-e2e.js` | 28/28 |
  | the new #2218 suite | 4/4 |
  | `test-issue-748-node-analytics-coverage-e2e.js` | 2/2 |
  | `test-analytics-fluid-charts.js` | 8/8 |
  | `test-issue-1657-analytics-channels-group-sprites-e2e.js` | 5/5 |

## Not done

- Two tabs that each fetch their own data can still overwrite each other
after a fast tab switch between them. #2200 does not ask for that.
- `nodes.js` `navigateToNode` calls its own `init` without the router,
so it does not bump the generation. None of the three pages fixed here
depends on that.
- I placed the E2E in shard 2 without measuring suite times, which the
`deploy.yml` comment asks for.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 10:39:15 +02:00
CoderNemesisandClaude Opus 5.5 e55c95db24 1-Byte Hop Path Inference for Packet and Map Analysis (#2245) 
## Summary

On meshes with 1-byte path hashes, most hop prefixes are shared by
several repeaters. The packet detail pane currently resolves each hop on
its own and stores the result by prefix. Two things go wrong as a
result:

- A prefix that appears twice in one path resolves to the same node both
times.
- Nothing checks that the chosen hops form a path a packet could
actually take.

This PR adds `HopResolver.inferPath()`, which infers the most likely
node behind every hop of **one observation's path**, scoring the path as
a whole and anchoring it on the observer that heard it. The packet
detail pane and the map route view now use it, so both show the same
nodes for the selected observation (`?obs=`).

## The problem, on a real packet

Packet `57da37d1636f895a`, observation `1388126`: a 46-hop flood with
1-byte hashes. The server's `resolved_path` resolved 1 hop of 46, so the
client resolver did the rest. Eight prefixes appear twice in the path.

| | Current resolver | `inferPath()` |
|---|---|---|
| Hops longer than 120 km | 13 (21 longer than 60 km, two over 950 km) |
0 |
| Distance from the last hop to the observer | 59 km | 0.1 km |
| Hops reported as unlisted (no listed node fits) | 0, every hop guessed
| 13 |
| Agreement on the shared path prefix across the packet's 23
observations | — | 97.8% |

The current resolver picks badly for three reasons:

1. **Results are keyed by prefix.** Hop 0 and hop 45 share a prefix, so
both show the node near the start of the path, 59 km from the observer.
2. **Picks are greedy.** Each hop looks only at its nearest resolved
neighbours, so nothing penalises a 1,100 km jump.
3. **The detail pane read the wrong `resolved_path`.** It used the
packet row's, not the selected observation's, and when the lengths
matched it never fell back for the hops that were `null`.

## Firmware basis

The model relies on two facts from `firmware/src/Mesh.cpp`:

- **The last hop is the one the observer heard.**
`Mesh::routeRecvPacket` appends the relaying node's own hash to the end
of a flood path.
- **A node appears at most once in a flood path.** A node forwards a
given packet once: `_tables->wasSeen()` / `markSeen()`.

## How the model works

Each hop is either one of the **located** repeaters or room servers
sharing its prefix, or **unlisted**: a node missing from the node list,
or one without GPS. With 1-byte prefixes, unlisted is common, and naming
it beats drawing a 300 km hop.

| Term | Cost |
|---|---|
| A hop between two located nodes | distance / 50 km; past 100 km per
hop, the cost rises 4× faster |
| A hop with a neighbour-graph link | the distance cost × 0.5 |
| An unlisted hop | 1.6, flat (about one 80 km hop) |
| The last hop to the observer | the same distance cost; falls back to
the observer's IATA region centre (with 50 km slack) when the observer
reports no GPS or 0,0 |
| The first hop to the sender | the same distance cost, when the
sender's position is known (e.g. an ADVERT) |

- **Search:** dynamic programming over (hop, candidate) finds the
cheapest sequence. When the best path uses a node twice, the node is
banned from whichever position costs less to give up.
- **Confidence:** each hop's confidence comes from the min-marginal
costs of its alternatives, put through a softmax. A node the path
already uses at another hop is not counted as an alternative.
- **Server hint:** the observation's own `resolved_path` restricts a hop
to the named node, but that hop can still be reported unlisted if the
node doesn't fit.

The tuning values live in `PATH_MODEL` in `hop-resolver.js`, with a TODO
to expose them in the customizer (AGENTS rule 8).

## What changes in the UI

**Packet detail pane** (`packets.js`):

- The Path row, the cross-observer aggregate row and the byte breakdown
render hops **by position**, from the inferred entries.
- The detail pane no longer writes into the list's shared per-prefix
cache (`hopNameCache`), so it can't change what the packets list shows.

**Hop display** (`hop-display.js`):

| Hop state | Display |
|---|---|
| Confident (≥ 70%) | The node's name; the tooltip says `NN% likely` and
how many nodes share the prefix |
| A guess (< 70%) | Dashed underline, plus a candidate badge when there
are alternatives |
| Unlisted | The hex prefix in muted text; the badge lists the rejected
candidates |
| The only known node, without GPS | The name, with `(no GPS)` |

The candidate popover lists each candidate with its likelihood, or `hop
N` when the path uses that node elsewhere. The "Path symbols" legend
gains an entry for unlisted hops.

**Map route view** (`map.js`, `route-view.js`):

- The `#/map?packet=…&obs=…` deep link infers the chosen observation's
hops with the same shared helper, so the map and the detail pane agree.
- Unlisted hops render as `unlisted` placeholders rather than as the
nearest node that happens to share the prefix.
- When inferred hops are available, the `/api/resolve-hops` request is
skipped.

**Shared helpers** (`packet-helpers.js`):

- `inferObservationPath(obs, observerPos, originPos)`
- `ensureHopAffinity()`: loads the neighbour graph once per page, cached
by `api()` for 5 minutes.
- `routeHopFromInferred(hop, entry, role)`

**Cleanup:** removed dead hop resolution from the "View route on map"
button handler. It computed hops, sender and recipient, then navigated
by URL and discarded them.

**Unchanged:** the packets list, live view, node pages and
`HopResolver.resolve()`.

## Performance

AGENTS rule 0 asks for proof of performance claims; the figures follow.

- **Complexity:** O(hops × 8 × candidates²). Hops more than 8 positions
apart aren't compared directly; a longer run of unlisted hops costs a
flat amount, tracked with a running minimum. Hop-to-hop costs are
computed once and reused by every pass.
- **Where it runs:** once per detail render (plus once for the aggregate
row) and once per map deep link. Nothing runs in the list, live feed or
WebSocket paths.
- **Measured** (Node 22, warm JIT):

| Case | Time |
|---|---|
| 64 hops × 16 candidates, synthetic worst case | 9–14 ms median |
| Real 39–48 hop paths, 1,153 nodes (23 observations) | 1–12 ms each |

- **Enforced:** `test-hop-path-inference.js` asserts a median under 40
ms (target 20 ms) for the 64 × 16 case. It times the module outside
Node's `vm` sandbox, because the sandbox makes global lookups about 10×
slower than a browser.
- **Network:** one extra bulk request per page life
(`/api/analytics/neighbor-graph`, cached server-side, TTL 5 min). No
per-hop or per-packet requests.

## Tests

**New:**

- `tests/unit/test-hop-path-inference.js` (37 assertions). The model
itself:
  - a repeated prefix resolves per position, and no node appears twice;
- the observer anchors the last hop, with the IATA fallback when the
observer has no position or reports 0,0;
  - a far-away lone candidate is reported unlisted;
  - a neighbour-graph link breaks a tie;
  - the server hint and the origin anchor are applied;
  - a node without GPS is handled;
  - output shape, and the performance gate.
- `tests/unit/test-hop-path-detail.js` (32 assertions). Rendering and
wiring:
  - inferred-hop rendering and the popover;
  - positional `renderPath`, including with the 1-byte hop filter on;
- `renderDetail` infers the shown observation from its own
`resolved_path` and leaves the list cache alone;
  - the neighbour graph is fetched once;
  - the shared map helpers, and source checks on the map wiring.
- `tests/e2e/test-hop-path-inference-e2e.js` (6 steps, shard 1). Runs on
fixture packet `57224a4a01c9c682`, observation 147, a 22-hop path that
repeats one prefix three times:
  - every hop is inferred;
  - the repeats resolve per position;
  - the popover shows likelihoods;
  - the map's route list matches the detail pane hop for hop.

**Updated, because they pinned the old behaviour:**

- `test-issue-1418-deeplink-hops-channels.js`: it tested a copy of the
old map priority logic. It now tests the real `inferObservationPath()`,
including `resolved_path` as a hint.
- `test-issue-2098-last-hop-region.js`: the detail pane no longer writes
the prefix cache. The test now reads the region-anchored last hop from
the rendered Path row.
- `test-issue-2097-hop-ambiguity-badge.js`: the source check allows the
new `entries` option while still asserting that the detail pane never
uses the summarised form.
- `.eslintrc.json`: declares the three new shared helpers as globals.

**Results on this branch:**

- `sh test-all.sh`: all standalone frontend suites pass.
- `npx eslint public/*.js`: 0 errors. All 85 warnings are pre-existing;
none come from code this PR adds.
- The "Run Playwright E2E tests (fail-fast)" step was run locally with
its script taken verbatim from `deploy.yml` (`bash -eo pipefail`,
fixture prepared as in CI):
  - shard 1: 27 of 27 passed;
  - shard 2: 47 of 47 passed;
  - shard 3: 50 of 50 passed.

Note for reviewers: `test-issue-1692-packets-init-parallel-e2e.js` only
passes while the freshened fixture's newest packets are less than 15
minutes old. The test sets the time window in `localStorage` after the
page has already read it. CI meets that condition because shards start
right after freshening. Unmodified `master` fails the same way when run
late.

**Browser validation:** the local server ran on the fixture DB. Checked
in the browser:

- the Path row, the candidate popover and the byte breakdown;
- the map route view, which matched the detail pane hop for hop with
unlisted placeholders;
- the console showed no errors.

## Known limitations and follow-ups

- **TRACE packets are not modelled yet.** A TRACE's path is its
*planned* route. The header path byte counts the hops actually forwarded
(repeaters append SNR, not their hash, `Mesh.cpp`), and a trace may
revisit a node on its way back (see the comment in
`Packet::calculatePacketHash`). A follow-up will add a `heardAfter` /
`allowRepeats` option, so a trace is anchored after the hops it actually
completed and may repeat nodes.
- **Customizer (AGENTS rule 8):** the `PATH_MODEL` tuning values are
hardcoded for now. The unlisted cost, typical and maximum hop length,
and the confidence threshold should become customizer settings.
- **Server-side `resolved_path`:** the ingestor writes a hop as resolved
whenever its prefix is unique in the node table, even when that node is
hundreds of km from the neighbouring hops. `inferPath()` treats these as
hints and can overrule them, but the stored data is still misleading.
- **The first hop without a sender position is often a close call.** On
encrypted channel messages, hop 0 is constrained only by what follows
it, so 55/45-style splits between a long chain of known nodes and
"unlisted" are expected. The badge and the percentage show this rather
than hide it.

## Screenshots

This is what a normal corescope instance would show for a 1 byte path.
<img width="318" height="338" alt="image"
src="https://github.com/user-attachments/assets/4056bc27-0b37-4107-9b25-31ac0f7864ec"
/>

<img width="430" height="601" alt="image"
src="https://github.com/user-attachments/assets/01da5ee8-babf-4e41-a9ab-836b5d640549"
/>

Now the same packet but with the logic to infer each hope along the path
different nodes are chosen as they're ranked and selected based on
proximity to the next hop,

<img width="436" height="646" alt="image"
src="https://github.com/user-attachments/assets/bcca73b8-daf0-4031-8ea1-8c96d716c8b3"
/>

<img width="436" height="646" alt="image"
src="https://github.com/user-attachments/assets/fc926d9e-158c-48ce-bee6-93086afb0775"
/>

The higher the confidence level then there's no need to show the
exclamation mark to show that multiple nodes share the same prefix but
it does continue to share that

Examples of it currently running can be found at,
https://corescope.baymc.org - modified instance

https://analyzer.comchan.net - unmodified instance
https://analyzer-stage.comchan.net - modified instance

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-19 20:00:07 +02:00
efitenandClaude Opus 5.5 2d09335cfb fix(packets): keep the node filter pick on a slow click or late search (#2243) (#2244) 
Fixes #2243.

## Problem

The packets node filter selected a suggestion on `click`, while the
input's `blur` handler hides the dropdown 200 ms after focus leaves. A
press held longer than that, or a search response that rebuilt the list
between mousedown and mouseup, lost the pick: the typed text stayed and
no `node=` filter was applied.

## Change

- `public/packets.js`: suggestions select on `mousedown` with
`preventDefault`, so the input keeps focus and the pick happens before
any rebuild. `live.js` already does this for its node filter.
- `public/packets.js`: a sequence counter drops search responses for an
older query, bumped on every search and on a pick, so a late response
can't replace or reopen the list.
- `tests/e2e/test-packets-node-filter-e2e.js` (new), registered in
`scripts/non-unit-tests.json` and on shard 1 in `deploy.yml`.

## Tests

The new E2E has three cases: a normal click, a 400 ms press, and a
search response released mid-click.

| Code | Normal click | 400 ms press | Response mid-click |
|---|---|---|---|
| master | pass | fail | fail |
| this PR without the sequence guard | pass | pass | fail (list reopens)
|
| this PR | pass | pass | pass |

Run locally against the freshened e2e fixture. `sh test-all.sh` passes
(locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and
`PYTHONUTF8=1`, unrelated to this change).

## Not done

- Full `test-e2e-playwright.js` not run locally.
- Could not reproduce on the reporter's instance with the browser tool,
whose clicks are shorter than 200 ms. The 400 ms Playwright press
reproduces it on the fixture.
- `live.js` and `packets.js` still carry two separate node-autocomplete
implementations. Merging them is out of scope here.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 21:30:57 +02:00
nullrouten0nullroutenClaude Sonnet 5.5
 		127dd511a7 fix(health): judge favorites and home status cards by the node's role (#2215) 
## The problem

A starred repeater can show two different health statuses. The My
Repeaters tab (#1761) and `getNodeStatus` judge it by the infra
thresholds (degraded after 24 h, silent after 72 h). The favorites
dropdown and the home page cards judge every node by the node thresholds
(degraded after 1 h, silent after 24 h), because they never look at the
node's role.

| repeater last heard | favorites / home cards (before) | My Repeaters
tab | after this PR |
|---|---|---|---|
| 5 h ago | degraded | healthy | healthy |
| 30 h ago | silent | degraded | degraded |
| 80 h ago | silent | silent | silent |

If you set `healthThresholds` in `config.json`, every view follows your
values.

## The change

Three call sites now use `getHealthThresholds(role)`, the helper the My
Repeaters tab already uses:

- `renderFavDropdown` in `app.js`
- the "My Nodes" grid card in `home.js`
- the hero search-result card in `home.js`

The search card's fallback text, "Not heard in 24+ hours", was only true
for non-infra nodes, so it now reads "Not heard recently".

## Scope

- I kept the existing three-state healthy / degraded / silent logic and
changed only which thresholds apply.
- Not included: counting a repeater that is relaying traffic as alive
(#1598, which `getNodeStatus` does). That is a bigger behavior change
and would be its own PR.

## Risk

Low. It is 9 lines in two files, and the helper and thresholds already
exist and are used elsewhere.

## Tests

New `tests/unit/test-health-card-role-thresholds.js`, added to
`test-all.sh`. It checks that infra roles get looser thresholds than
nodes, and that each of the three call sites goes through
`getHealthThresholds(role)` with no node-only comparison left. On
current `master` 5 of its 7 assertions fail; with this change all pass.

`sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning from an
a11y self-test, which it also does on `master`.

## Update after review (commit `0f2376df`)

The text above is the description efiten reviewed on 2026-10-15. This
section describes the code as it is now.

### The change

- New `getHealthStatus(role, heard, now)` in `roles.js`. It returns
`healthy`, `degraded` or `silent` by the role's thresholds, or `null`
when there is no usable timestamp. It only applies the cut-offs; each
caller picks the timestamp.
- All six places that compared an age against `degradedMs` / `silentMs`
now call it:
  - the favorites dropdown (`app.js`);
  - the "My Nodes" card and the home health card (`home.js`);
  - the analytics network status and My Repeaters (`analytics.js`);
  - the live node panel (`live.js`).

Each keeps its own labels and timestamp, so the last three show the same
as before.

`analytics.js` and `live.js` no longer guard with `typeof
getHealthThresholds`. `roles.js` loads before both, and both already
used it without a guard elsewhere: the old network status called
`getHealthThresholds` directly, and `live.js` reads `ROLE_COLORS`.
- The favorites dropdown and both home cards now fall back to the node's
`last_seen` when the packet store has no packet for it. My Repeaters
(`last_heard || last_seen`) and the node page (`nodes.js:729`) already
do this. Without it, with a packet store window under 72 h, a repeater
could be silent on the cards and degraded on My Repeaters. The "ago"
text uses the same timestamp.

With the default thresholds, the fallback changes a status only when the
packet store holds less than 72 h of packets. `retentionHours` unset
(unlimited) or at the example's 168 h holds more than that. Otherwise a
node with no packet in the window is silent either way, and the views
now show how long ago it was last seen instead of "never" or nothing.
- With no timestamp at all, the home health card said "Not heard in 24+
hours". It now says "No packets in the current window".

### Why a new helper and not `getNodeStatus`

`getNodeStatus` has two states (active / stale) and counts a repeater
that relays traffic as alive (#1598). Using it here would change more
than the thresholds. `getHealthStatus` keeps these views' three states.

### Scope

- Not included: counting a repeater that relays traffic as alive (#1598)
in these views. That is a bigger behavior change and would be its own
PR.
- The live node panel still judges by `last_seen` alone, and the node
page stays relay-aware. Both now use the shared thresholds; aligning
their timestamps would also be its own change.

### Tests

- `tests/unit/test-health-status.js` runs `roles.js` in `vm`. It covers:
- repeater, room, companion, sensor, an unknown role and a missing role,
at ages either side of 1 h, 24 h and 72 h;
  - ISO and epoch input;
  - null, empty and invalid timestamps;
  - a `healthThresholds` override through `/api/config/client`.

One source guard remains: no age-vs-threshold comparison outside
`roles.js`.
- `tests/unit/test-my-repeaters-dashboard.js` now uses the real helper
instead of a stub, which gave repeaters the node thresholds. It also
checks that a repeater heard 30 h ago is Degraded.
- New `tests/e2e/test-health-status-e2e.js` mocks
`/api/nodes/<pk>/health` for six nodes. It checks five views: the
favorites dropdown, the "My Nodes" card, the home health card, the
analytics network status and the live node panel. It fails on `master`
and on this PR's first commit. I put it on shard 2, which was the
shortest shard in the last CI run.
- Mutation check: I made 33 deliberate breaks. In the helper they cover
each comparison and boundary. At each of the six call sites they cover
the timestamp field, the role argument and at least one label mapping.
Each one makes the suite fail.
- I checked a `config.json` with `infraDegradedHours: 2` and
`infraSilentHours: 6` in Chromium. Repeaters heard 3 h and 7 h ago show
Degraded and Silent.
- `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning, which
comes from a deliberate fixture in the a11y self-test.

### Cost

The analytics network status loops over every node. In headless
Chromium, 30,000 synthetic nodes take 9.0 to 9.2 ms before this change
and 9.1 ms after (median of 21 runs, on one machine).

### Risk

Low. The change is frontend only, with no API, config or data change.
Rolling back means reverting the commits.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-10-17 09:38:11 +02:00
efitenandClaude Opus 5.5 927a8eec8a fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211) 
Refs #748 (M3a: correct numbers within the in-memory window; the SQL
history in M3b is a separate plan, see the end).

## Problem

The node analytics range picker offers 24h, 7d, 30d and All (365d), but
the store only holds its in-memory window (`retentionHours`, then
eviction). `GetNodeAnalytics` still divided availability and packets/day
by the requested range. Measured on our live instance (14-day window),
repeater BE-BRE-ON8AR:

| Range | Availability | Packets/day |
|---|---|---|
| 7d | 88.7% | 91 |
| 30d | 42.6% | 265 |
| All | 3.5% | 22 |

Nothing on the page said that the data stopped at the edge of the
window.

## What changes

- `coverageStart` returns the edge of the in-memory window: the latest
of `oldestLoaded` (background-load progress), `now - retentionHours`,
and a new `evictedThrough` that eviction sets to the newest `FirstSeen`
it removed, so memory-pressure eviction inside the retention window
moves the edge too.
- `s.packets[0]` is not that edge, and the first version of this PR used
it. Chunks are selected by `last_seen`, so a transmission first heard
weeks ago and heard again recently sits at the head with its old
`FirstSeen`. On staging (7-day window) it returned 21/07.
- When the edge lies more than a minute after the requested start, node
analytics counts only transmissions from the edge, computes availability
and packets/day over that span, and returns `timeRange.dataFrom` and
`timeRange.partial`.
- The minute of slack keeps a range equal to the retention window from
reading as partial. Without it every 7d request on staging (168h) was
partial, because the two edges came from separate clock reads.
- The analytics page shows "Data from 2026-10-07: the server keeps about
7 days of packets in memory. Availability and packets/day cover that
period, not the full 365 days." when partial.
- **UTC fix.** The window bounds of node analytics and node hop
analytics are formatted in UTC. They are compared as strings with UTC
`FirstSeen` values, so on a non-UTC host the 7d window lost its oldest
hours (2 at +02:00). Docker images run in UTC, which is why this did not
show in production.
- `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the
edge is kept.

## Tests

- Go, `node_analytics_coverage_test.go`:
  - full coverage within memory;
  - 30d over 3 days in memory;
  - empty store;
- a re-heard 85-day-old transmission at the head of `s.packets` (edge
stays at retention, packet not counted, no bucket before `dataFrom`);
  - memory-eviction edge;
  - eviction records `evictedThrough`;
  - retention 30 s shorter than the range is not partial, while 30d is;
  - window bounds under a +02:00 `time.Local`.
- Two tests were checked to fail on the old code: the local-zone test
(166 of 168 packets, a `+02:00` bound) and the retention test without
the slack (`dataFrom 09:57:08` vs `from 09:56:38`).
- Full `cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master). One run of five reported `FAIL` and four reruns were green; I
did not capture which test failed.
- Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js`
(5 assertions on the real `node-analytics.js` in `vm`), in
`test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js`
(registered in `deploy.yml` and `scripts/non-unit-tests.json`), against
the real fixture response with no interception: 7d is not partial and
shows no notice; All is partial and the notice names the real start
date. 2/2 locally.

## Validation

Staging (retentionHours 168), same repeater, after the background load
completed:

| Range | Before | After |
|---|---|---|
| 7d | 82.7% | 83.3%, not partial |
| 30d | 19.3% | 83.3%, partial from 2026-10-07 |
| All | 1.6% | 83.3%, partial from 2026-10-07 |

In the browser the notice appears under All and not under 7d.

## Not done

- M3b, older history from SQLite. In the database only the sender is
indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay
hops only in `observations.resolved_path` (filled for about 27% of
observations on our live instance, not indexed). A fallback today would
show only a node's own packets for older days, a different meaning than
the in-memory part. Doing it properly needs a per-node index written by
the ingestor (pubkey, transmission, role); that is a separate plan.
- The other analytics endpoints with a range (observer analytics, RF)
were not checked for the same dilution.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 09:27:21 +02:00
efitenandClaude Opus 5.5 cab4f71c64 feat(nodes): LoRa radio settings per node from the MeshCore map (#1268) (#2209) 
Fixes #1268.

Frequency, spreading factor, bandwidth and coding rate are per-device
settings that never travel in a packet. The public MeshCore map is the
only source. This PR adds them to the node detail views, opt-in, and
answers the open policy questions from the triage comments.

## Policy decisions

| Question (triage) | Answer in this PR |
|---|---|
| Opt-out for self-hosted instances | Opt-**in**: `rfParams.nodesUrl`
empty (default) = nothing fetched, `/api/rf-params` returns `enabled:
false`. Same model as `knownChannelsUrl`. |
| 33 MB every 6 h, thundering herd | Random 0-60 s delay before the
first fetch, then every `refreshHours` (default 6). The map sends gzip
(17 MB on the wire). It sends no ETag or Last-Modified, so conditional
requests are not possible. |
| Size cap | 128 MB on the decoded body (it is 51.5 MB today). Exceeding
it fails the fetch instead of parsing a truncated body. |
| Field shapes at the parse boundary | 64-hex key; 100-1100 MHz; SF
5-12; bandwidth from the LoRa set (7.8 to 500 kHz); CR 5-8. Invalid rows
are dropped. A body that is not a well-formed array fails the fetch. |
| Keep stale cache on anomaly | Any failure keeps the previous snapshot;
a failed presets fetch keeps the previous presets. `fetched_at` is
returned so an outage is visible rather than hidden. |
| Freshness | Only adverts within `maxAgeDays` (default 7) are kept at
fetch time and re-checked at serve time. |
| TLS pinning | Not done: Go's certificate validation plus the field
validation above. |

## What changes

- `cmd/server/rf_params_cache.go`: stream decoder, validation, cap
reader, preset parser (presets with identical values are merged, titles
joined), background refresh. In memory only, no DB writes, so the
read-only server invariant holds.
- `GET /api/rf-params`: `{enabled, fetched_at, source, presets, nodes}`;
`nodes` holds only nodes this instance knows, keyed by lower-case
pubkey. `/api/nodes` is unchanged, so its payload does not grow for
every consumer and the 500-row paging is unaffected.
- Node detail (side panel and full page): `LoRa` row with the four
values, the preset name when all four match one, and the age of the map
advert. Shown only when the feature is on and the server has fetched the
map at least once.
- No column in the Nodes table, on purpose: on our network such a column
showed one preset title in 814 of 1,366 rows and nothing in 443. Showing
only the nodes that differ from their region is the follow-up PR.
- `config.example.json` documents the block with the two URLs to use.

## Cost

- Measured on the real map payload (51.5 MB, 66,336 nodes): parse 0.52
s, 15 MB allocated in total, 26,815 nodes kept (advert within 7 days).
Runs in a background goroutine every 6 h.
- The endpoint walks the cached node list once with map lookups:
O(nodes).
- Frontend: one bulk fetch of `/api/rf-params` when a node detail opens
(client TTL 5 min), not one per node.

## Tests

- Go (`rf_params_cache_test.go`): parse with every rejection reason,
malformed bodies (not an array, truncated, wrong type, empty), preset
merge and invalid presets, cap reader (one byte over refused, exactly at
the cap accepted), fail-soft against an `httptest` upstream (503, HTML
body, presets-only failure), config defaults, route (disabled shape,
only known nodes, entry aged past max age dropped at serve time). Full
`cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master).
- Frontend unit: `tests/unit/test-issue-1268-rf-params.js` (7 assertions
on the real file in `vm`), in `test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-1268-rf-params-e2e.js` (registered in
`deploy.yml` and `scripts/non-unit-tests.json`). It checks the real
disabled state on the fixture server (no detail row), then serves
`/api/rf-params` through `page.route`: no table column, full detail page
with preset, side panel with raw values, and the hint for a node that is
not on the map. 4/4 locally, plus the existing node table and node
detail E2E files.

## Validation

On a staging instance with `rfParams` set to the map and preset URLs:

- Log: `[rf-params] 26821 nodes within max age, 20 presets`, about 6 s
after start.
- `/api/rf-params` returned 1,115 of our nodes: 988 on
869.618/SF8/BW62.5/CR8, 111 on the same with CR5, 15 on 433.65 MHz, 1 on
SF7.
- In the browser the detail row shows in the side panel and on the full
page, and the Nodes table keeps its original columns.

## Not done

- Highlighting nodes whose settings differ from their region's majority
(a badge in the Nodes list plus a filter) is the follow-up (separate
PR). In that follow-up a CR-only difference stays informational until it
is confirmed whether MeshCore radios use the explicit LoRa header (then
CR travels in the header and a mismatch still decodes).
- `maxAgeDays` is config-only; the customizer is a later step.
- No data is written to the DB, so the RF parameters are gone after a
restart until the next fetch (at most 60 s plus the fetch time).

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 09:26:45 +02:00
nullrouten0Claude Mythos 5.1nullrouten
 		a164a44e7f fix(docker): run the Go services as an unprivileged user (#2125) 
Runs `corescope-server` and `corescope-ingestor` unprivileged, opt-in
with `RUN_AS_UID`. A code-execution bug in either one then no longer
lands as root next to the mounted data and `config.json`. `supervisord`
stays root, for mosquitto and caddy.

| `RUN_AS_UID` | Go services run as |
|---|---|
| unset (default) | root, as before. Upgrades change nothing |
| set | `RUN_AS_UID` / `RUN_AS_GID` |

With `RUN_AS_UID` set:
- Each start hands the data directory to that uid (`chown -h`), except
`.env` and anything mounted into it on its own. Failures are logged.
- `.env` is read only if it is root-owned and only root can write it.
`RUN_AS_UID` and `RUN_AS_GID` are never taken from it.
- `/app` stays root's. **Depends on #2137.**
- A uid with no account in the image gets one at start; `caddy` and
`mosquitto` are reused. A uid that can't be provided fails startup
instead of falling back to root.

The logic is in `docker/select-service-user.sh`, tested by
`docker/test-select-service-user.sh`, which needs no Docker or root and
runs in CI. See also "Running the Go services unprivileged" in
`docs/deployment.md`.

Verified in the built image (master + this + #2137) across 14 cases.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Mythos 5.1 <noreply@anthropic.com>
Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
 2026-10-15 09:49:29 +02:00
efitenandClaude Opus 5.5 1ae4c74913 feat(observers): rolling 1 h TX airtime and a duty-cycle warning (#661) (#2208) 
Fixes #661 (the narrowed scope from the 2026-10-14 comment).

## What changes

- `GET /api/observers/metrics/summary` adds `tx_airtime_pct_1h` and
`rx_airtime_pct_1h` per observer: the airtime share over the hour before
now, whatever `window` is, from the cumulative `tx_air_secs` /
`rx_air_secs` counters in `observer_metrics`.
- A counter that goes down marks a reboot; only the stretch after the
last reset counts.
- Under 15 minutes of samples (one sample, or a reboot late in the hour)
returns `null`.
  - Clamped to 0..100, rounded to 2 decimals.
- The response also carries `duty_cycle_warn_pct`, from the new config
key `dutyCycleWarnPct` (default 10, the EU 868 MHz g3 sub-band limit).
Added to `config.example.json`.
- RF Health tiles show `TX 1h: x%`; the observer list gets a sortable
numeric `TX 1h` column. Both use the shared `public/observer-airtime.js`
helper and flag values at or over the threshold with `.airtime-over`
(`var(--status-red-text)`).
- The handler now returns a named `MetricsSummaryResponse` struct
instead of `map[string]interface{}` (one occurrence fewer).

## Why the warning matters

Repeater and room-server firmware default to `airtime_factor = 1.0`
(`examples/simple_repeater/MyMesh.cpp:890`), which `Dispatcher.cpp:26`
turns into a 50% TX budget per 1 h window. The device therefore does not
keep itself under the EU 10% limit; this view is where an operator can
see it.

## Cost

- Server: one extra query per summary request, `SELECT observer_id,
timestamp, tx_air_secs, rx_air_secs FROM observer_metrics WHERE
timestamp >= now-1h`, on the existing `idx_observer_metrics_timestamp`,
then one O(rows) pass. On our instance that is 35 observers x about 12
samples (5 min interval).
- Observer list: one extra bulk call `summary?window=1h` per load
(client TTL 30 s, failure ignored), not a call per observer.

## Tests

- Go: `TestRollingAirtimePct` (steady, reboot mid-hour, single sample,
span under 15 min, reboot leaving under 15 min, nil counters, clamp,
rounding), `TestGetMetricsSummaryAirtime1h` (samples older than 1 h
ignored inside a 24 h window, single-sample observer gets null),
`TestMetricsSummaryDutyCycleWarnPct` (default 10, config override,
`observers` is `[]` not `null`). Full `cmd/server` suite passes locally
(skipping `TestSaveGeoFilterPreservesFileMode`, which fails on Windows
on clean master too).
- Frontend unit: `tests/unit/test-issue-661-observer-airtime.js` (8
assertions, real file in `vm`), added to `test-all.sh`.
`test-observers-headings.js` updated for the new column. Full
`test-all.sh` passes locally.
- E2E: `tests/e2e/test-issue-661-airtime-e2e.js`, registered in
`deploy.yml` and `scripts/non-unit-tests.json`. The fixture DB has no
`observer_metrics` table, so the summary is served through `page.route`
with known values for the fixture's own observer ids. It checks the
column header, over/under/unknown cells, sort order and the RF Health
tile. 4/4 locally. It caught a real bug during development:
`loadRFHealthData` declares a local `const window`, so the tile calls
the helper as the global `ObserverAirtime`.

## Validation

Deployed to a staging instance with live MQTT data (35 observers with
metrics) and checked in the browser:

- The observer list shows the column for 42 rows (33 with a value, 9
without enough samples). Sorting puts the highest share (5.3%) first.
- RF Health shows `TX 1h` on all 35 tiles. With a fourth stat the line
wrapped inside items on narrow tiles; the stats row now wraps between
items only.
- No observer was over 10% at the time, so the red state was checked by
rendering one cell client-side at 12.5% (computed colour `rgb(185, 28,
28)` in light mode), and by the E2E test.

## Not done

- The threshold is config-only; exposing it in the customizer is a
follow-up.
- No airtime comparison across observers (that is #671) and no
per-packet airtime (Relay Airtime Share covers it).
- Dark mode was not checked in the browser; `--status-red-text` is
defined for both themes.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-14 21:07:20 +02:00
efitenandClaude Opus 5.5 fbeb9e4d71 ci: frontend coverage from every E2E suite, merged across the shards (#2204) 
Fixes #2203.

## Problem

The frontend coverage badge (35.16% on run 38413268001) came from two
sources on shard 2: one dump part-way through `test-e2e-playwright.js`
and the `collect-frontend-coverage.js` tour. The other E2E suites run
against the instrumented build but never save `window.__coverage__`,
including the suites written for single pages (`path-inspector.js`
4.72%, `touch-gestures.js` 14% on that run).

## Change

1. `9d3bfad5` `test(e2e): red:` `tests/unit/test-e2e-coverage.js`, with
fake pages, contexts and browsers (no Chromium needed). On the
placeholder module the first assertion fails.
2. `c00bbb12` `feat(e2e):` the collector.
- `scripts/e2e-coverage-hook.js` is a preload (`node --require`) that
wraps `chromium.launch` from `playwright`. `playwright` re-exports
`playwright-core`, so a suite's own `const { chromium } =
require('playwright')` gets the same, wrapped object.
- `scripts/e2e-coverage.js` reads `window.__coverage__` before every
call that replaces or ends a page's document: `goto`, `reload`,
`setContent`, `goBack`, `goForward`, `close`, and context and browser
`close`. Pages from `newPage`, `newContext` and the context `page` event
are all tracked.
- Each read zeroes the page's counters, so a document read twice is not
counted twice. Maps travel once per file; later reads carry counts only.
- The page returns a JSON string. Returning the object cost about 10 s
per suite, because Playwright serialises it node by node.
- A read is given up after 5 s, so a busy page cannot hold up a suite's
cleanup.
- On exit the hook writes one file per suite to `.nyc_output/`, or
nothing without coverage. A failed write is logged and never changes the
exit code.
3. `64487c4c` `ci:` the workflow.
- On master pushes the E2E run step sets `NODE_OPTIONS=--require
./scripts/e2e-coverage-hook.js`. On PRs it is empty, so PR runs are
unchanged.
- Each shard runs `nyc merge` (one file per suite is about 6 MB, mostly
the same maps) and uploads `coverage-shard.json` with its results
instead of `.badges/`.
- The `e2e-test` job merges the three shards and makes the frontend
badge (it used to come from shard 2 alone).
- The merge, setup-node and badge steps are `continue-on-error`. When
`nyc report` gives no statement percentage the step warns and writes no
badge, so a registry hiccup cannot publish 0%.
- `test-e2e-playwright.js` no longer writes its own `e2e-coverage.json`:
the hook covers that suite completely, including the tests after the old
dump. `collect-frontend-coverage.js` stays as an extra source.

## Performance

Measured locally against an instrumented server (Windows, Chrome):

| Suite | Without hook | With hook |
|---|---|---|
| `test-path-inspector-coverage-e2e.js` | 1.9 s | 2.2 s |
| `test-touch-gestures-coverage-e2e.js` | 17 s | 16 s |
| `test-e2e-playwright.js` (stops at the same local-fixture failure in
both runs) | 52 s | 52 s |
| `browser.close()` on a page in a 30 s busy loop | 153 ms | 5.2 s |

The hook only loads on master pushes.

## Testing

- `tests/unit/test-e2e-coverage.js` covers:
  - reads before each document-replacing call and before close;
  - pages opened by the page itself;
  - context and browser close;
  - summing without double counts;
  - a failed read, and a read that never answers;
  - the file writer, including a failing write.
- Nine deliberate breaks of the library each turn it red:
  - dropping the counter reset;
- dropping the `page` event, the `goto` read, the `goForward` read, the
`close` read or the browser `close` read;
  - removing the timeout or the write guard, or writing an empty store.
- The test also fails if it stops before its end. Without that check, a
read that never returns would let node exit 0 half-way.
- Real suites with the hook, measured on their own (the badge run had
all suites):

| File | Badge run 38413268001 | Own suite with the hook |
|---|---|---|
| `path-inspector.js` | 4.72% | 52.75% |
| `touch-gestures.js` | 14% | 72.96% |

- Exit codes and pass counts were unchanged with the hook for these
suites (the first three were run before the 5 s read timeout was added):
  - `test-slideover-1056-e2e.js` (6 contexts);
  - `test-a11y-axe-1668.js` (`@axe-core/playwright`);
  - `test-issue-1150-404-state-e2e.js`;
  - the three above.
- The badge step, run locally against two merged shard files:
  - It gives the same per-file numbers as `nyc report` on the inputs.
- With no shard coverage, or with `nyc report` failing, it warns and
writes no badge.
- `sh test-all.sh` passes locally, apart from
`test-issue-1956-release-routing.js`, which needs the workflow's
`GITHUB_REF_NAME` and was skipped. The test inventory classifies 321
suites. `actionlint` 1.7.7 reports only the self-hosted runner label, as
on master.

**Not tested:** the shard and badge steps in CI. They only run on a
master push, so the first master run after merge is the real test. I do
not know what the merged number will be. I will compare the badge with
the merged report in that run's `e2e-test` log.

**Not counted:** a full page load the suite does not start through one
of the wrapped calls, such as a link that leaves the SPA or
`location.reload()` from page code.

## Not in this PR

- The `tests/unit/` suites, which load frontend files through
`vm.createContext` without instrumentation (noted in #2203).
- Coverage on PR runs. The overhead above is small, so this could follow
if wanted.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-13 22:04:03 +02:00
efitenandClaude Opus 5.5 59708ee72e ci: publish the README badges to a badges branch with GITHUB_TOKEN (#2202) 
Fixes #2136.

## Problem

The README badges are stale: `e2e-tests` shows 821 passed (1161 when
#2136 was filed), both Go coverage badges have not changed since
2026-03-30 and frontend coverage since June. There are two causes.

1. **The PAT.** The `publish` job wrote the badge JSON to `master`
through the Contents API with `BADGE_PUSH_TOKEN`, a PAT that could push
past the branch protection on `master`. That token returns 401: run
38360160932 on 2026-10-13 logs `gh: Bad credentials (HTTP 401)` for
every badge. The step is `continue-on-error` and only echoed `Failed to
update`, so every run stayed green.
2. **Old badges inside `e2e-badges`.** `.badges/` is tracked on
`master`, so every e2e shard checkout and the aggregate checkout already
hold the old files. The shard upload and the aggregate `cp` shipped them
in `e2e-badges`, and `publish` downloaded `e2e-badges` after
`go-badges`, so the old values overwrote the fresh ones. In run
38407143126, `go-badges` holds 85.8% and 77.1%, while `e2e-badges` holds
85.4%, 70.2% and frontend 36.64%, the same as `master`'s `.badges/`.
Fixing only the token would have left three of the four README badges
frozen.

## Change

1. `5f430966` `ci:` the `publish` job pushes the badges to an
unprotected `badges` branch with its own `GITHUB_TOKEN`.
- Job-level `permissions: contents: write`. No other job gains a
permission. The checkout uses `persist-credentials: false`, since the
push passes its own token.
- The branch is a single orphan commit, force-pushed, so it does not
grow. Master push runs are already serialised by the workflow
`concurrency` group.
- Each run starts from the branch as it is and overwrites only the
badges it downloaded, so a missing artifact keeps those badges at their
previous value. The first time, when `git ls-remote --exit-code` reports
the branch absent (exit 2), it starts from `master`'s `.badges/`.
- No downloaded artifacts, or any other `ls-remote`, fetch, extract,
commit or push failure, ends in a `::warning::` in the run summary and
leaves the branch alone. The step stays `continue-on-error`, so a badge
problem never turns the pipeline red.
   - `go-badges` is downloaded after `e2e-badges`, so its files win.
2. `2134c1e5` `docs:` the four README badge URLs read
`.../CoreScope/badges/<name>.json` instead of
`.../master/.badges/<name>.json`.
3. `47e57482` `ci:` the frontend-coverage step in the e2e shards and the
aggregate badge step start from an empty `.badges/`, so `e2e-badges`
carries only what the run produced. This also stops shard 3's old
`frontend-coverage.json` from replacing the one shard 2 generates.

No PAT is involved any more, so nothing expires.

## Testing

- The publish step script, taken from the YAML, run against a local bare
repository standing in for `origin`:

| Run | Downloaded | Result on `badges` |
|---|---|---|
| 1, branch absent | a, b | a, b new; c from `.badges/`; one commit |
| 2 | c only | a, b kept from run 1; c new; still one commit |
| 3 | none | `::warning::`, branch unchanged, step exit 0 |
| 4, `origin` unreachable for the read | a | `::warning::`, branch
unchanged, step exit 0 |
| 5, push to a missing remote | a | `::warning::`, branch unchanged,
step exit 0 |

- The artifacts of run 38407143126 land flat in the download path, so
`badges-new/*.json` matches them.
- `shellcheck` at style level on the step script: no findings.
- `actionlint` 1.7.7 (shellcheck off): one finding, the self-hosted
runner label `meshcore-runner-2`, which `master` reports as well.

**Not tested:** the e2e change and the push with a real `GITHUB_TOKEN`.
Both only run on a push to `master`, so the first master run after merge
is the real test. The `badges` branch already exists (`80fda6c0`, a copy
of `master`'s `.badges/`), so the README badges keep working from the
merge on, with the old values until that run. After it, I will check the
`badges` branch contents against that run's go and frontend numbers, the
step log and the rendered README.

## Not in this PR

- Removing the now unused `.badges/` from `master`: the first run seeds
the branch from it, so that goes in a follow-up once the branch exists.
- Deleting the `BADGE_PUSH_TOKEN` secret: that needs a repository admin.
- `frontend-tests.json`: no CI step produces it and no README badge uses
it. It is carried over to the branch unchanged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-13 16:25:42 +02:00
Alex BandClaude Opus 5.5 352a0ccd19 feat(region-filter): configurable region quick picks (#2170) 
## Summary

**Independent of every other open PR.** Choosing a pick saves the
selection, redraws the control and notifies the page's listeners itself,
exactly as a click in the control does, so it re-queries straight away
whether or not #2168 (`setSelected` notifies) is merged; with both in,
nothing fires twice.

Region quick picks are named groups of region (IATA) codes, offered as
one-tap choices inside the region filter, e.g. "Northern California" for
a dozen airport codes instead of ticking each box. One deployment has
been injecting these into the page through an nginx `sub_filter` and a
script that wraps `window.RegionFilter`; this makes them a configured
feature.

```jsonc
"regionQuickPicks": [
  { "name": "Northern California", "description": "Bay Area and Sacramento observers.", "regions": ["SJC", "SFO", "OAK", "SMF"] }
]
```

- **Server:** `GET /api/config/region-quick-picks` returns the groups
cleaned up: names and codes trimmed, codes upper-cased and de-duplicated
in their configured order, groups without a name or codes dropped, at
most 20 groups of 200 codes, always a list. `/api/config/regions` is
unchanged.
- **Region filter:** loads the groups with the regions and offers every
group with at least one observer today, inside the region control:
- dropdown layout: rows of the same kind as **All** (checkbox and bold
name), directly under All with a divider before the single regions; the
row is ticked while that pick is the selection, ticking it selects the
pick and closes the menu, unticking goes back to all;
- pill layout: the picks lead the same bar, set off from the single
regions by a thin divider.
- A tap selects the group's codes that have an observer and re-queries
straight away.
- The selector then reads the group's name ("South Bay ▾") instead of "2
Regions", and the button shows as pressed. Selecting the same codes by
hand does the same.
  - Tapping the active group again goes back to all regions.
- A failed fetch, or no configured groups, leaves the filter exactly as
before.
- **Mobile:** the rows take the menu's 44 px row height under `(pointer:
coarse)`; the pill-bar picks reuse `.region-pill` and its touch target.
Inside `.filter-bar` (Packets) the existing touch rule for inputs made
every region checkbox a 44 px square; the checkbox is now 20 px and the
44 px row is the tap target.
- **Semantics:** like every region filter, a group means "heard by
observers in these regions", not "located there"; `config.example.json`
says so.

No existing issue covers this.

## Screenshot

Nodes page with two picks configured (example config below), after
tapping **South Bay**: the pick and its codes (SJC, MRY) are selected,
and the list re-queried for them. Taken against
`test-fixtures/e2e-fixture.db`.

![Region quick picks on the Nodes
page](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/region-quick-picks.png)

## Config example

```json
"regionQuickPicks": [
  { "name": "South Bay", "description": "San Jose and Monterey observers", "regions": ["SJC", "MRY"] },
  { "name": "East and North Bay", "description": "Oakland and San Francisco observers", "regions": ["OAK", "SFO"] }
]
```

Each pick needs a `name` and at least one code; `description` is
optional (shown as the tooltip). Codes are trimmed, upper-cased and
de-duplicated; a pick with no code that has an observer today is not
shown. At most 20 picks of 200 codes.

## Browser validation

The branch's server was run locally against a migrated copy of
`test-fixtures/e2e-fixture.db` (observers in MRY, OAK, SFO, SJC) with
three groups configured: "South Bay" (SJC, MRY, SCZ), "East and North
Bay" (OAK, SFO, sts) and "Nowhere yet" (XXX). Checked in headless
Chromium:

| check | desktop 1280 px | phone 390 px (touch) |
|---|---|---|
| groups shown | South Bay, East and North Bay ("Nowhere yet" hidden) |
same |
| tap South Bay selects | SJC, MRY (SCZ has no observer) | same |
| next `/api/packets` request | `region=SJC,MRY` | `region=SJC,MRY` |
| selector label | "South Bay ▾" | "South Bay ▾" |
| second tap | back to all regions | same |
| button height | 24 px | 44 px |
| page wider than viewport | no | no |
| page errors | none | none |

Checked on Packets (dropdown layout) and Nodes (pill layout). Three
layout problems found this way on the phone were fixed before opening: a
sideways-scrolling row clipped at the screen edge, the selector squeezed
into a narrow column (the filter group is a non-wrapping flex row), and
the two wrapped rows spread apart on Nodes (`align-content`).

**Revision 1:** the first version put the picks in a separate row beside
the selector; they moved inside the dropdown. Re-checked with a
14-region list and three picks on Packets: desktop 1280 px dark and
phone 390 px light both keep the menu on screen (menu 340 px wide, page
not wider than the viewport), the pick closes the menu and the trigger
names the pick.

**Revision 2:** picks are checkbox rows like **All** instead of chips.
Re-checked on Packets at 1280 px dark and 390 px light: rows render bold
under All, the divider separates the single regions, ticking a pick
ticks its codes and names the trigger, no sideways scroll.

Unrelated and not changed: on Nodes at 390 px the region row (existing
pills included) starts flush with the screen edge.

## Tests

- Red commit `81ea67d6`: the config type, a stub normaliser and endpoint
returning nothing, and tests.
- Go: cleaning (trim, upper-case, de-duplicate, drop incomplete groups),
caps, nil config, and the endpoint's JSON.
- Frontend, `tests/e2e/test-region-quick-picks.js` (moved from
`tests/unit/` in `fced69a7`; it needs jsdom, so it runs in the E2E job
like `test-table-sort.js`): the real `region-filter.js` in **jsdom**
against faked config endpoints. Nine cases: rendering and hiding groups
with no observer, selecting and re-querying, the selector label,
toggling back, a hand-made selection marked active, the pill layout, no
groups, a failed fetch, HTML escaping. Seven fail on the stub, all on
assertions.
- Green commit `0e33210e`: the implementation, styles,
`config.example.json` (`regionQuickPicks` +
`_comment_regionQuickPicks`), `docs/api-spec.md`, and the OpenAPI route
list (`TestOpenAPICompleteness` requires it). The jsdom test also had a
cross-realm comparison bug fixed in this commit (`Array.from` on arrays
coming out of the jsdom window).
- `make test`: 16 of 16 modules pass. Frontend suites: 195 of 196; the
one failure, `test-issue-1956-release-routing.js`, needs CI's
`GITHUB_REF_NAME` and fails the same way on `master` locally.

- Red commit `f991060d` (revision): picks live inside the dropdown menu
before the checkboxes, choosing one closes the menu, and in the pill
layout they share the region bar; two fail on assertions against the old
layout. Green commit `1f720e35`. Frontend 195 of 196 (same
environment-only failure).

- Red `134f5256` / green `da0adb50` (second revision): picks are
checkbox rows between All and the single regions with a divider, no
chips in the menu; ticking and unticking a pick row. The green commit
also repairs `style.css`: `1f720e35` had prepended its block to the top
of the file (its edit matched an earlier `.region-pill {`) and left the
previous rules in place. The file now differs from `master` by 13 lines.

## Review notes

- Groups are offered only when at least one of their codes appears in
`/api/config/regions`, and only those codes are selected. The stored
selection is pruned to known codes on load anyway, so selecting absent
codes would not survive a reload.
- Customizer (AGENTS.md rule 8): groups are operator configuration; a
customizer editor could come later.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `b84ace4e`:
- `normalizeQuickPickCodes` split out of the config normaliser (S3776,
17 > 15);
  - a comment on why a failed quick-pick fetch is ignored (S2486);
  - `for…of` in `activePick` (S4138);
- in the tests: `node:` specifiers, `includes()`, and an explicit string
comparator for sorted selections (S7772, S7765, S2871).
- **Kept on purpose:**
- S2486 is still reported on that `catch`. Swallowing the error is the
intended behaviour: the filter works as before, just without picks.
  - S1523 on the `vm` test loader, as above.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 21:23:05 +02:00
efitenandClaude Opus 5.5 de364a8bb6 feat: mail notifications for watched nodes (user management part E) (#2140) 
Part E of #2128: a logged-in user watches nodes and gets one mail when a
watched node goes offline, comes back, or reports a low battery. Admins
can add instance events: a new foreign node, and an observer going
offline or back. This covers the per-user mail part of #775.

Stacked on #2139 (part D, which builds on #2138). Review the commits
after that branch.

## The situation

A sysop learns that a repeater went silent, or that its battery is
running down, only by opening CoreScope. #775 asks for notifications.
Accounts (part A) now give a verified address to mail and a mailer with
delivery status.

## What this PR adds

**Events.** They use the instance thresholds, so a mail does not
disagree with the node page:
- `node.offline`: silent for the role's `healthThresholds` window. Last
heard comes from the packet store, and repeaters and rooms count relayed
traffic (#1598).
- `node.battery`: advert telemetry below `batteryThresholds.lowMv`,
recovered at `lowMv + 100`.
- Admins only: `foreign.new` (once per node) and `observer.offline`.

**Store.** Schema v5 with `notification_prefs`, `notification_watches`
and `notification_state`.

**Server** (only with `userManagement.notifications.enabled`)
- A notifier next to the janitor, every `intervalMinutes` (5). The
evaluator is a pure function of watches, prefs, stored states and node
snapshots.
- No burst after a restart or a new watch: a subject's first evaluation
stores its state without mailing. The loop waits for the startup load.
While ingest is stale (newest packet older than 30 minutes) the offline
checks pause, and after recovery they wait one silent window.
- One mail per user per check. Limits: 20 per user and 100 per instance
per rolling 24 hours, 50 watches per user. A change over a limit is
recorded and never mailed later.
- Every mail has a one-click unsubscribe (`List-Unsubscribe` and
`List-Unsubscribe-Post`). The GET only redirects to a confirm page, so
link scanners cannot unsubscribe anyone. Node names are cleaned of
control and bidi characters before they go into a mail.
- Routes: `GET`/`PUT /api/account/notifications`, `PUT`/`DELETE
/api/account/notifications/watches/{pubkey}`, `POST
/api/account/notifications/watch-my-nodes` (copies the synced "my nodes"
list), `GET`/`POST /api/notifications/unsubscribe`. `mailer.Message`
gains `Headers`.

**Frontend.** A "Notify me" toggle on the node pages, a Notifications
section on the account page (`#/account?section=notifications`), an
unsubscribe page, and the notification mail count on the admin overview.

Spec:
[`docs/specs/2026-10-07-node-notifications-design.md`](https://github.com/efiten/CoreScope/blob/feat/notifications/docs/specs/2026-10-07-node-notifications-design.md).

## Performance

Each check reads watches, prefs and states from `users.db` (one query
each), the watched nodes from the analyzer DB in chunks of 500, and
last-heard times from the packet store.

| Measurement | Result |
|---|---|
| One check on our staging instance, 130,000 packets in memory, 1
watcher | 6.9 ms, of which 0.93 ms holding the store read lock |
| Evaluator benchmark, 100 users with 50 watches each, 2,000 nodes |
1.75 ms |

No work is added to ingest, broadcast or any request path.

## Verification

- `internal/users` and `internal/mailer` with `-race`; the `cmd/server`
suite plus `-race` on the notifier tests; 74 new Go tests. The same
Windows-only failure as noted in #2138 applies.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E: 26 of 26 steps locally. With notifications off
the new steps fail, so they do test the flag.
- On our staging and production instance since 7 October 2026.

## Not in this PR

- Other channels (Discord, Telegram, webhooks). Detection is separate
from delivery, so one can be added.
- The topology, RF and anomaly alerts of #775.
- If the whole server starts after a feed outage that already ended, the
grace window is unknown and a watcher can get one wrong offline mail.
The user guide says so.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:02:39 +02:00
efitenandClaude Opus 5.5 abbe6846ff feat: propose and approve hashtag channels (user management part D) (#2139) 
Part D of #2128: logged-in users propose a hashtag channel, an admin
approves, rejects or later revokes it, and the ingestor decrypts
approved channels without a restart. It is the first consumer of a
generic proposal table, and answers the request in #2092.

Stacked on #2138 (part C). Review the commits after that branch; GitHub
shows C's commits here until it is merged.

## The situation

- An instance decrypts only the hashtag channels listed in
`hashChannels`. Users who want a city or club channel shown have to ask
the operator, who edits `config.json` and restarts the ingestor.
- #2092 asked for a suggest-and-approve flow; a downstream fork
(dborup/CoreScope#99) runs one with anonymous suggestions and a file
queue.

## What this PR adds

**Name rules (`internal/channel`).** `ValidateHashtagName`, mirrored in
the browser:
- at most 31 UTF-8 bytes including `#` (firmware
`ChannelDetails.name[32]`), case preserved, `Public` refused in any
case;
- control, bidi, separator and format characters refused except ZWJ,
plus invisible fillers (`Other_Default_Ignorable_Code_Point`, U+2800,
non-ASCII spaces).

Go and JS give the same verdict for every code point.

**Store.** Schema v4, one generic `proposals` table (kind, subject,
status pending/approved/rejected/revoked, proposer, reviewer, note). One
row per (kind, subject); limits are checked in the same transaction as
the change.

**Server** (only with `userManagement.channelProposals.enabled`)
- `POST /api/proposals`, `GET /api/account/proposals`, `GET
/api/admin/proposals`, `POST
/api/admin/proposals/{id}/approve|reject|revoke`.
- 400 for an invalid name. 409 for a duplicate, a rejected name, or a
name already in `hashChannels`/`channelKeys` ("this channel is already
decrypted on this instance"). 429 over a limit (`maxPending` 100,
`maxApproved` 128, `perUserPerDay` 5).
- `GET /api/channels` gains `approvedChannels`, so approved channels are
listed before they have traffic. With the feature off the response is
byte-identical.

**Ingestor.** Opens `users.db` with `mode=ro` every 60 seconds,
re-validates the names, and swaps its key map through an
`atomic.Pointer`. Configured channels always win, and a failed read
keeps the last good set. The ingestor never writes `users.db`; AGENTS.md
now says so.

**Frontend.** "Propose for everyone" in the add-channel dialog, an
"approved" marker in the channel list, "My proposals" on the account
page, and an admin "Proposals" tab with a warning on approve. The packet
detail pane now escapes the channel name: until now only the operator
chose those names.

Spec:
[`docs/specs/2026-10-07-channel-proposals-design.md`](https://github.com/efiten/CoreScope/blob/feat/channel-proposals/docs/specs/2026-10-07-channel-proposals-design.md).

## Performance

Every key is tried on each GRP_TXT that no key opens (`decodeGrpTxt`).
Benchmark of that worst case:

| Keys | Time per packet |
|---|---|
| 320 configured | 204 to 216 µs |
| 320 configured + 128 approved | 272 to 320 µs |

That is about +33%, linear in the key count and capped by `maxApproved`.
With the feature off the decoder gets the identical map and no ticker
runs.

## Verification

- `internal/channel`, `internal/users` (with `-race`), `cmd/server` and
`cmd/ingestor` pass locally, 53 new Go tests. The ingestor's
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced` needs symlink rights on
Windows and fails on `master` too.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E: 21 of 21 steps locally (propose, approve, the
channel appears, revoke removes it).
- On our staging and production instance since 7 October 2026.

## Not in this PR

- Private (PSK) channels: their keys stay in the browser (#725).
- Anonymous proposals.
- Names that need ZWNJ (some Persian and Urdu spellings) and subdivision
flags are refused, because ZWNJ and tag characters are format
characters. Allowing them later only widens the rule, so no approved
name gets stranded.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:51:42 +02:00
efitenandClaude Opus 5.5 a4445b4985 ci: run E2E, Go tests and the image build side by side (#2135) 
Closes #2134.

## What changes

The test jobs and the image build now run side by side. Only the GHCR
push waits for all of them.

```
changes ─┬─ go-test ─────────────────────────┐
         ├─ race-test (ingestor changes only)  │
         ├─ e2e-shard ×3 ── e2e-test (gate) ───┼─ build-and-publish → deploy, publish
         └─ image-check (two-arch build) ──────┘
```

- **`e2e-shard` (new, matrix of 3).** It needs only `changes`. The suite
list stays in the workflow: each line now reads `suite <shard> <file>
[VAR=value ...]`, with the same environment variables as before.
- Shards are balanced on the measured suite times, about 6 minutes each.
- A shard number outside 1..3 fails the step, so a typo cannot drop a
suite silently.
- `test-issue-1648-m4-icons-e2e.js` stays in the same shard as, and
after, the a11y suite. Its distance-tab check only counts once the lazy
distance index (#1011) has been built, and the a11y suite is what first
requests it.
- **`e2e-test`** is now a gate under the old check name. It runs only if
all three shards passed, checks that three results arrived, and builds
the same `e2e-badges` artifact as before.
- **`image-check` (new).** It runs the two-arch build (no push) and the
arm64 QEMU smoke on every event, beside the tests. It also computes the
build metadata once, which the push reuses.
- **`build-and-publish`** now needs `go-test`, `e2e-test` and
`image-check`. On a PR every step is skipped. On push and tag refs it
only pushes, with the same build args as `image-check`, so every layer
should be a cache hit.
- **The unused `docker compose build` of the staging image is gone.**
`docker compose config --quiet` still validates the file.
- `AXE_SCREENSHOT_DIR` now actually reaches the a11y suite.
- `tests/unit/test-issue-1956-release-routing.js` follows the new jobs,
and checks more than before:
- a failure in `go-test`, `e2e-shard`, `e2e-test` or `image-check` skips
`build-and-publish`;
- the only non-publishing build lives in `image-check` and runs on every
event;
  - the push takes its build args from `image-check`.

## Measurements

The old numbers are from upstream; the new ones are from PR runs on my
fork (a branch on top of this commit, plus one temporary commit that
also triggers the workflow for PRs into a test base branch).

| | Before (PR run 37766553643) | After (fork runs 37812660655 /
37814554422) |
|---|---|---|
| Go Build & Test | 4.9 min | 7.2 / 7.5 min, in parallel |
| E2E | 18.4 min | 3 shards of 7.1–9.0 min, in parallel |
| Image build | 6.9 min, after E2E | 5.9 / 5.4 min, in parallel |
| **Total** | **30.5 min** | **9.6 / 10.6 min** |

**Same tests:** I compared the second fork run with the E2E log of the
upstream run, suite by suite. All 118 suites ran, and each one passed
the same number of checks (1161 in total).

**Cost:** more runner minutes, because each shard repeats about 2.5
minutes of setup. Standard runners are free for a public repository.

## Not verified here

- **A master push.** On a fork the GHCR push cannot run. The expectation
is that `Build and push to GHCR` gets cache hits from `image-check` and
finishes in a minute or two instead of 4–4.5. The first master run after
merge will show whether it does.
- **Branch protection.** If it requires other job names than `🎭
Playwright E2E Tests` and `🏗️ Build & Publish Docker Image`, those may
need adding. I cannot read the protection settings, so I could not
check.

## Tests run locally

- `sh test-all.sh` passes (on Windows with `PYTHONUTF8=1`).
- `test-issue-1956-release-routing.js` passes, and fails as expected
when `go-test` is removed from the `needs` of `build-and-publish`.
- actionlint 1.7.7 reports only the existing self-hosted runner label.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-08 20:13:09 +02:00
efitenandClaude Opus 5.5 d232072f85 feat: optional user accounts (part A: foundation) (#2129) 
Part A of #2128: optional, off-by-default user accounts. With the
feature off nothing changes; with it on, visitors can register and log
in, and admins manage users and use the operator actions without the API
key.

PR #2130 (settings sync) builds on this one. The two are meant to be
merged together.

## The situation

- Operator actions (geofilter save and prune, backup, perf reset) need
the shared `apiKey`. There is no per-person right.
- Nothing in CoreScope knows who a visitor is, so the requests in #2128
that need that (#1835, #2092, #1508, #730) have nothing to build on.

## What this PR adds

**Two new Go modules**
- `internal/users`: a separate `users.db` (SQLite through
`modernc.org/sqlite`) with users, sessions, single-use tokens, an audit
log and a mail log. Passwords use argon2id.
- `internal/mailer`: a `Mailer` interface with a Brevo client (send,
delivery events, webhook parsing) and an in-memory fake for tests.

**Server (`cmd/server`)**, active only with `userManagement.enabled`
- 24 routes, all documented in OpenAPI under the `users` tag
([`auth_routes.go`](https://github.com/efiten/CoreScope/blob/feat/user-management/cmd/server/auth_routes.go)):
  - auth: register, activate, login, logout, me, forgot, reset;
- account: profile, password, email change with confirmation, sessions,
self-delete;
- admin: list, detail, disable, enable, delete, role, resend activation,
manual activation, mail status refresh;
  - a Brevo webhook, registered only when `mail.webhookSecret` is set.
- `requireAdmin` replaces `requireAPIKey` at the 7 operator call sites:
the API key **or** an admin session. With the feature off it is the old
API-key gate (`TestRequireAdminWithoutUserManagementIsAPIKeyGate`).
- `/api/config/client` gets `userManagement: {enabled: true}` only when
the service started; with the feature off the response is
byte-identical.

**Frontend**
- `auth.js` (header account control, request helper that adds the CSRF
header), `account.js` (login, register, activate, forgot, reset, confirm
email, my account), `admin-users.js` (`#/admin/users`, deep-linked
filters), `account.css` (theme tokens only).
- On phones the top-bar control is hidden, so a conditional entry goes
into the bottom-nav "More" sheet and the nav drawer.
- The customizer geofilter tab and the Perf "Reset stats" button use the
admin session when there is one.

**Config.** A `userManagement` block (`config.example.json`,
[`docs/user-guide/accounts.md`](https://github.com/efiten/CoreScope/blob/feat/user-management/docs/user-guide/accounts.md)).
The Brevo key can come from `CORESCOPE_BREVO_API_KEY`. The server
refuses to start when the block is enabled but incomplete.

## Security choices

- Session cookie `cs_session`: HttpOnly, SameSite=Lax, Secure when
`publicBaseUrl` is https. Every cookie-authenticated state change needs
the `X-CS-CSRF` header and a matching Origin.
- Activation needs the token **and** the account password. Without the
password, an attacker who keeps re-registering a known address could get
the owner to activate an account that carries the attacker's password.
- Register, forgot and email change answer identically for known and
unknown addresses. A password reset ends all sessions, a password change
ends all other sessions, and both end outstanding email-change links.
- Rate limits: login 10 per 15 minutes, register and forgot 5 per hour,
per IP and per address. The bucket count is capped. `trustedProxies`
makes the per-IP limits see real client IPs behind a proxy.
- Server logs carry `#<user id>`, never addresses, tokens or passwords;
mail-provider error texts are redacted before logging.

## Performance

No change to an existing hot path with the feature off. With it on:
- One `users.db` lookup per authenticated request (session by token
hash).
- The admin user table rebuilds its `tbody` on each filter change.
`users.List` caps the result at 1000 rows (`internal/users/users.go`),
which bounds the rebuild.
- `map[string]interface{}` in `openapi.go`: 79 before, 78 after.

## Verification

- `internal/users`, `internal/mailer` and `cmd/server`: `go vet` and `go
test -race` pass locally. 121 new Go tests.
- `cmd/server` with `-tags e2etest`: vet and the e2e hook tests pass.
- `sh test-all.sh` exits 0. `tests/unit/test-user-management-ui.js`: 67
passing (vm, real modules).
- `tests/e2e/test-user-management-e2e.js` (6 steps) passed locally
against an `e2etest` build with the fake mailer and against a
feature-off build. CI builds the `e2etest` binary and runs the suite on
a second server (`deploy.yml`).
- On a staging instance with a real Brevo key: register, activation mail
delivered, activate, admin table, "Refresh status" showing sent,
deferred, delivered, opened and clicked.

## Not in this PR

- Settings sync (#2130), the admin dashboard, approval flows and
notifications (parts B to E of #2128).
- A `requireReadAuth` mode (#1835). Sessions from this PR are what such
a mode would accept.
- Binary size and build time with `modernc.org/sqlite` linked next to
`mattn/go-sqlite3` were not measured. Their driver names do not collide.
#1992 discusses the driver choice.
- No Brevo webhook was configured on staging; delivery status there came
from "Refresh status".

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-07 09:25:29 +02:00
efitenandClaude Opus 5.5 000d9ab030 feat(coverage): RF noise-floor layer on the Mobile RX coverage page (#2113) 
The ingestor already stores the noise floor that CoreDrive RX companions
report with each GPS fix (`client_rf_samples`, opt-in through
`clientRfSamples`, `cmd/ingestor/client_rf_sample.go:62`), but nothing
reads it back. An operator who enables it collects the data and cannot
see it. This adds the read side: `GET /api/rf-noise` and a Signal/Noise
toggle on the Mobile RX coverage page.

## What it does

- **`GET /api/rf-noise?bbox=&z=&days=`** returns a GeoJSON hex grid with
the median, quietest and noisiest noise floor per cell, in the same
shape as `/api/rx-coverage`. It is registered always and 404s unless
`clientRfSamples.enabled` is true, the same pattern as the coverage
routes.
- **Stationary samples are excluded.** A parked companion logs hundreds
of readings at one point, which would otherwise define its cell.
- **Coverage page:** a Signal/Noise toggle, rendered only when
`/api/config/client` reports `clientRfSamples: true`. The noise layer
reuses the coverage colour tokens with the axis inverted, because a
lower dBm is quieter. Tiers are at -115 and -108 dBm.
- **Deep link:** `#/rx-coverage?layer=noise` opens on the noise layer.
- **Empty and failed answers** are labelled on the map ("No RF samples
in this view yet", or a retry hint), so a blank map never reads as
"feature off".

No new configuration key: it reads the existing `clientRfSamples`
section that the ingestor already uses. Default off, so nothing changes
for an instance that has not opted in.

## Where it comes from

Ported from the efiten/CoreScope fork, where it has run on
analyzer.on8ar.eu since September (fork commits `42d09f0e`, `80581c43`,
`cde95078`). The cherry-picks conflicted with upstream's newer
`routes.go`, `types.go` and `rx-coverage.js`, so the final state was
ported by hand. The fork-only `/scopes` route that sat next to it in the
same hunk is deliberately left out.

## Performance

The query is bounded by `sampled_at` (indexed, `idx_crf_prune`) and the
bbox, aggregation is one pass plus a per-cell sort, and the response is
capped at 5000 cells. Measured on analyzer.on8ar.eu, all of Belgium
(`bbox=49.4,2.4,51.6,6.5&z=9`), from a client in Belgium, so network
time included:

| window | samples | cells | response time |
|---|---|---|---|
| 7 days | 10,835 | 241 | 0.37 s |
| 30 days | 43,303 | 429 | 0.39 s |

That table holds 45,270 rows in total. It is only read when someone
opens the noise layer, never on ingest or WebSocket paths.

## Tests

- `cmd/server/rf_noise_test.go`: 8 tests for the aggregation (median and
extremes, stationary exclusion, the cell cap, empty input) and the gate
(404 when off, even with data present). All pass, and the full
`cmd/server` suite passes locally.
- `tests/unit/test-rx-coverage-noise.js` (new, in `test-all.sh`): the
colour axis runs the right way, including both tier bounds and a string
median from the API. Slices the real function out of `rx-coverage.js`
and fails on master's copy, which has no thresholds.
- `tests/e2e/test-rx-coverage-noise-e2e.js` (new, wired in `deploy.yml`
and `scripts/non-unit-tests.json`): no toggle when the flag is off;
Noise fetches `/api/rf-noise`, draws the cells, swaps legend and
subtitle, puts `layer=noise` in the hash; a deep link opens on the noise
layer and an empty answer shows the message. 3/3 locally; against
master's `rx-coverage.js` and `roles.js`, 1/3 (only the "flag off" case
passes, as it should).
- `test-rx-coverage-viewport-e2e.js` still passes. ESLint 8 reports 0
errors on the changed files.

## Not done

- The thresholds (-115 / -108 dBm) are fitted to the fork's own data
(1,241 moving samples at the time) and are constants in
`rx-coverage.js`. Per AGENTS.md rule 8 they belong in the customizer
eventually; not in this PR.
- The E2E suite mocks the API. The real endpoint is covered by the Go
tests and by the measurements above, not by a fixture with seeded
samples.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 22:11:34 +02:00
n30nex 99a326a6b3 test(packets): verify observation hex after rerenders (#2105) 
Red commit: `ca553e4` ([expected assertion
failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37137426691)).

Fixes #2104.

The observation hex test now opens the existing grouped fixture
directly, selects A → B → A with stable-ID locators, and asserts the
selected ID and exact raw bytes after each render. Missing fixture data
fails explicitly. Distinct observation frames are seeded after schema
migration; this test's fixed sleeps and silent skips are removed.

Rebased onto master `415362c` after #2090 merged. Both patches are
unchanged. The previous #1122 layout blocker now passes against the
actual current server and assets.

Validation at `5db8017`:
- All 185 standalone frontend suites passed.
- Core browser: 133 passed, three existing skips; #1122 layout 6/6,
filter 11/11, grouped-collapse 4/4.
- Syntax, YAML, fixture SQL, XSS and lint passed (zero errors, 92
existing warnings).
- Fresh red CI reached the intended detached-observation assertion.
[Green
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37138187496)
passed; three independent automated reviews found no must-fix issues.

Only test code and fixture setup change; no configuration/customizer
implications.

## Preflight overrides
External OpenClaw runner/profile unavailable; repository checks and
actual local Chromium were used.
 2026-10-03 23:48:32 +02:00
Sylvain Rabot a4a447acea fix(packets): pack short columns, one expand arrow, Full Names toggle (#2090) 
The packets table wastes most of its width on short columns, draws two
arrows on grouped rows, and cuts every observer and hop name short. This
fixes all three and adds a way to read the full path when it does not
fit.

## Why the columns were wide

`makeColumnsResizable` measured each column once on load and stored the
widths as **percentages** of the table. Percentages grow with the
screen: on a 2000px display "17s ago" sat in a 180px column. The
measurement also read the virtual-scroll spacer row — a single `colspan`
cell as wide as the table — as column 0, so the expand column came out
160px wide locally and about 950px on a live instance with grouped rows.

Measured on the fixture at 2000px:

| Column | Before | After |
|---|---|---|
| Expand | 162–230px | 32px |
| Time | 111px | 67px |
| Size | 64px | 46px |
| HB | 44px | 29px |
| Scope | 80px | 48px |
| Path / Details | ~180px each | ~650px each |

## The change

**Column packing.** A new `fitColumnsToContent` (`app.js`) sizes every
column except Path and Details to its widest cell, in px; those two
split what is left. Every visible column gets an explicit width and the
table gets their sum: a spacer row keeps a slot for each hidden column,
and leaving any column `auto` hands those slots a share of the slack as
a blank strip at the right edge.

When space is short (detail panel open, narrow window) the widest fixed
columns give way first, then Path and Details shrink to 60px, and only
then does the wrapper scroll. The split itself is a pure function,
`distributeColumnWidths`, with unit tests.

It re-lays out when the container resizes (detail panel), when
TableResponsive hides or reveals columns (a new `table-columns-changed`
event), and after sorting or a column toggle. After each render `grow()`
widens columns for the rows just inserted and never narrows them, so
nothing jitters while scrolling. Drag handles store px; double-clicking
one resets it. Path and Details have no handle.

`makeColumnsResizable`, still used by the nodes, observers and analytics
tables, now skips `colspan` rows when measuring; other rows are measured
as before.

**One expand arrow.** A CSS `::before` "▶" sat next to the SVG caret,
which pointed up when collapsed. The glyph is gone and the caret points
right, then down.

**Full Names.** A toolbar toggle next to Hex Paths shows observer and
hop names untruncated. It is saved per browser; a link can carry
`fullNames=1`, which applies to that page without overwriting the
visitor's saved preference. Paths stay on one line: the virtual scroller
positions rows at one fixed height, so wrapping would make scrolling
jump.

**The "+N" path pill, and a full-path popover.** Hops past the column
edge were meant to go behind a "+N" pill (#1124), but the pill was
appended after the overflowing chips and so sat past the clipped edge:
it never showed. It is now sticky at the right edge, opaque on hover,
and counts hop chips only (it used to count warning icons too). Hovering
or focusing it shows the full path, one hop per line and numbered;
click, Enter or Space pins it until an outside click or Escape, and it
closes when the page changes or its row scrolls away. Clicking the pill
no longer selects the row underneath.

**Two related fixes.**
- Rows drawn before `/api/observers` resolves show raw 64-char pubkeys.
`loadObservers` now re-renders and re-measures, instead of leaving
Observer at the pubkey's width (~600px under Full Names).
- Group child rows indented every cell by 20px, which misaligned them
with the headers and widened every column on expand. Only the Time cell
is indented now.

Anyone who had dragged packet columns gets the new defaults once: the
old percentage key is dropped.

## Performance

`grow()` runs after every `renderVisibleRows`, so it is on the hot path.
Measured in Chromium on the fixture, 300 iterations, three runs:

| Case | Cost |
|---|---|
| Full render, 63 rows measured | +0.88–0.93ms over the forced reflow
alone |
| Scroll step (incremental render) | ~73 `Range` rect reads, only the
inserted rows |
| Large jump / re-render | ~569 rect reads |

Before narrowing `grow()` to inserted rows, one scroll step with 92
rendered rows made 5,472 rect reads. The DOM row count is bounded by the
virtual-scroll window, so the cost does not grow with the packet count.
`distributeColumnWidths` is a binary search over at most ~10 columns.

## Tests

- `tests/e2e/test-packets-compact-columns-e2e.js`, 16 cases: short
columns stay under fixed px bounds at 1920px; Path and Details take the
freed width; hiding both still fills the wrapper; no blank strip and no
horizontal scroll, with and without the detail panel, at 1920, 1100,
1000 and 375px; responsive reveal and re-hide refit; stale percentage
widths dropped; drag follows the pointer and double-click resets;
expanding a group leaves widths alone; Full Names untruncates, keeps
paths on one line, persists and round-trips through the URL; the
Observer column shrinks back when observer names arrive late; the pill
popover lists every hop vertically, closes when the pointer leaves, pins
on click without selecting the row, closes on Escape and on navigation,
and opens on keyboard focus. Registered in `scripts/non-unit-tests.json`
and `deploy.yml`.
- On `master` the suite fails where it should, and restoring each fix's
old behaviour (child-row indent, the late-observer refit) turns its case
red.
- `tests/unit/test-packets.js`: `distributeColumnWidths` (room to spare,
rounding, widest-first capping, floors, dragged columns, the 60px
minimum, overflow), the single chevron and no `::before` glyph, Full
Names on flat and grouped observer cells, and the Full Names CSS.
- `tests/unit/test-xss-escape-sinks.js` now builds the real
`obsCellName` from source: escaping moved into it. Removing its
`escapeHtml` turns both observer-cell tests red.
- `test-issue-1189-composed-cell.js` and
`test-issue-2012-clear-filters-selection.js` evaluate packets.js
fragments with injected helpers and needed `obsCellName` /
`showFullNames` added.
- Existing packets E2E suites pass on this branch and on master against
a CI-prepared fixture: scope column, #1128 (layout and multi-viewport),
#1122, #1188, #1648, table sort, #1692, #1486.
- Unit: 182 of 183 suites pass locally;
`test-issue-1956-release-routing.js` could not create a temp directory
in my sandbox and is unrelated.

## Browser validation

Checked in Chromium against the fixture at 2000, 1920, 1280 (panel open
and closed), 1100 and 375px, and with the seeded grouped row expanded
and collapsed. Screenshots taken at each step; the table fills its
container exactly, grouped rows show one arrow, and the pill popover
lists full names.

## Not done, deliberately

- **No width cap on Observer under Full Names.** A cap brings back the
truncation the toggle removes, and Observer is already the first column
to give way when space is short.
- **Customizer.** The floors (60px flex minimum, 64px shrink floor, 32px
expand, 58px Rpt) are hardcoded, per AGENTS rule 8; exposing them in the
customizer can follow.
- **Wrapping paths.** Rejected because of the fixed row height above;
the pill popover is the way to read a long path.
 2026-10-03 12:15:52 +02:00
n30nex 093e320c2b fix(map): keep Path Inspector candidates accessible and cover route replacement (#2082) 
## Fix

Fixes #2060. Fixes #2081.

Path Inspector route drawing and candidate replacement now have
deterministic browser coverage. An idempotent SQL seed adds four
synthetic repeaters and four edges after fixture migration, without
changing packet ordering. The test uses the real API, normal clicks,
visible Leaflet geometry, and removal of every previous route object.

Restoring coverage exposed a desktop layout bug: drawing the first route
moved the map over the next candidate button. The route sidebar now
participates in the existing flex layout, including resizing and
collapse, while preserving the mobile bottom sheet.

The historical zero-candidate result did not reproduce on the current
baseline. No search thresholds or API behavior changed. No new
dependencies or customizer settings.

## Validation

- Local assertion-red history: `042db2b` requires seeded candidates;
`ca1d22a` exposes the blocked second click. Subsequent commits repair
layout and narrow-window behavior.
- All 183 standalone suites passed. Unchanged-base Go failures: #2083
readiness race (fixed in #2084) and Windows symlink privilege.
- Real Chromium: 11 Path Inspector/layout checks, 33 related map checks,
and core E2E (131 passed, 3 existing fixture skips).
- CSS-variable checks, XSS diff check, inventory, whitespace and PII
checks passed. Seed remains valid after periodic graph refresh.

Local browser validation used Chromium with a 60-second navigation
budget; the OpenClaw profile and external preflight script were
unavailable. Repository checks ran directly. [CI run
36343343569](https://github.com/Kpa-clawbot/CoreScope/actions/runs/36343343569)
records the initial assertion-red commit. Final CI must pass before
merge.
 2026-09-30 11:40:49 +02:00
efitenandClaude Opus 5 9eb3098867 fix(release): publish the release notes as the release body (#2076) 
Three releases went out with an empty release body. Checked with `gh
release view --json body`:

| release | body | assets |
|---|---|---|
| v3.9.1 | 1206 bytes | 2 |
| v3.9.2 | 2768 bytes | 0 |
| v3.10.1 | **0** | 2 |
| v3.11.0 | **0** | 2 |
| v3.12.0 | **0** | 2 |

v3.9.1 and v3.9.2 were written by hand, which this repository then
stopped allowing because published releases are immutable. Nothing
replaced them, so the `docs/release-notes/` convention was never wired
to the release page and three releases shipped with a description of
nothing.

Reported by the fork operator, who went looking on the v3.12.0 page for
the list of fixed issues that older releases carried.

## The cause

`action-gh-release` in this workflow was given `files` and
`fail_on_unmatched_files` and nothing else. No `body`, no `body_path`,
no `generate_release_notes`.

## The change

A step resolves `docs/release-notes/${GITHUB_REF_NAME}.md` and passes it
as `body_path`, with `generate_release_notes: true` so GitHub's pull
request list lands underneath the hand-written notes. A missing notes
file is a warning rather than a failure: the release then still gets the
generated list, which is more than an empty body.

## Also in this PR

The v3.12.0 notes file gains the two sections it should have had:

- **Issues closed**, 13 of them. Collected from each pull request's
`closingIssuesReferences`, not from commit text, which is why 30 pull
requests yield 13 issues.
- **Contributors**, separating pull request authors (@efiten 22,
@liquidraver 3, @A13xB0 2, @n30nex 1, @dborup 1, @anieto 1) from commit
co-authors (@nullrouten, @anieto, SaarMesh-Bot, Openclaw) from issue
reporters (@efiten 5, @n30nex 4, @anieto 2, @liquidraver 1,
@damn-simple-scripts 1). It says outright that 22 of the 30 pull
requests are the interim maintainer's own, which is the shape of a
release cut while the owner is unreachable, not a healthy ratio.

v3.12.0's published body has already been set to exactly this content by
hand, so the release page and the file agree.

## Worth recording

`gh release edit <tag> --notes-file <f>` updates a published release
body even though `action-gh-release` cannot. The immutability that
permanently burns a tag name does not extend to the description, so a
thin release body is recoverable. That was not obvious from the existing
comments, which describe releases as immutable without qualifying which
parts.

## Verification, and its limit

`yaml.safe_load` parses the file, but that proves little: it silently
accepts duplicate mapping keys that GitHub rejects outright, which is
how a previous workflow edit passed a local check and produced no runs
at all. The real validator is this PR's own pipeline, and the change
cannot be exercised end to end until the next tag is pushed.

## Not done

No backfill for v3.10.1 or v3.11.0. Neither has a notes file, and
reconstructing them is the same retrospective work deliberately skipped
for the 3.11.0 changelog entry.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-26 18:45:50 +02:00
efitenandClaude Opus 5 6d5686dbb4 docs: release notes for v3.12.0, and the 3.11.0 entry that was never written (#2075) 
Prepares the v3.12.0 release. No tag is pushed by this PR: the release
procedure requires the tagged commit to carry an `:edge` image whose
revision label matches it, so that check belongs after this merges,
against the merge commit.

## Why minor, not patch

29 commits since v3.11.0: 15 fix, 5 test, 3 perf, **3 feat**, 2 ci, 1
chore. The feats are #2047, #2067 and #2068.

## What the notes lead with

Two changes alter what a running instance does without anyone asking it
to, so they are at the top rather than in a list:

- **#2058**: the first start against a database that has never been
`ANALYZE`d builds planner statistics and stalls ingest while it does.
Measured at 3m43.9s on 9.4 GB, once per database, buffered with nothing
dropped. `db.analysisLimit` set negative skips it.
- **#2035**: `maxMemoryMB` eviction now fires where it previously did
not, because the footprint compared against the limit was undercounted.
An instance that set the limit and never saw eviction will start seeing
it.

## The 3.11.0 gap

`CHANGELOG.md` had no entry for 3.11.0 and `[Unreleased]` was empty, so
52 shipped commits were undocumented. Added as a short entry that says
outright it was written after the fact and has no notes file. The
alternative of reconstructing 52 commits for a superseded version is
error-prone work with little value, and leaving the gap silent is worse
than naming it.

## One fix beyond documentation

`deploy.yml` carried a comment that would mislead the next person
cutting a release. It still said documentation-only commits skip the
workflow "(see the `paths-ignore` above)", which is exactly how v3.10.0
lost its `:edge` image and then its tag name permanently. That filter no
longer exists: the `changes` job forces `code=true` for anything that is
not a pull request (lines 61-73), so every master commit gets an image
and a documentation commit is safe to tag. The history stays in the
comment; the false present tense does not.

## Verified rather than asserted

Both went into the notes as upgrade advice, so both were checked in the
tree:

- `node_declared_regions` is created at boot with `CREATE TABLE IF NOT
EXISTS` (`cmd/ingestor/db.go:435`), added by #2047, which found the
table was read by `region_keys.go` and `config.go` and created by
nothing. So "no manual migration step" is accurate.
- The cgo dependency attributed to #1992 in the 3.11.0 entry is the one
that stops this repo building with `CGO_ENABLED=0` today.

## Not done

- No tag, no release. Next steps, after this merges: confirm the merge
commit's pipeline is green, confirm `:edge`'s revision label is that
commit, then tag `v3.12.0` annotated, push the tag only, dispatch `CI/CD
Pipeline` on the tag ref, and verify the published digests in the
registry rather than in a green job.
- No `docs/release-notes/v3.11.0.md`, deliberately, as above.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-26 12:03:31 +02:00
efitenandClaude Opus 5 25f8426d32 test: make every suite in tests/e2e run, and tell the truth when it does (#2053) 
Closes #2037.

Step 1 (#2045) wired in the nine suites that already passed. This is
steps 2 and 3: the four that ran and failed, and the five that could not
run at all. After this, the count of suites in `tests/e2e` invoked by
nothing goes from 18 to 0.

## Step 2 — the four that ran and failed

Triaged against a fixture server in CI with full output kept, not the
four-line tail the first probe saved.

| suite | verdict |
|---|---|
| `test-packets-scope-column.js` | passes on master today. It failed on
2026-09-18, so something between the two fixed it. Wired in unchanged
rather than investigated. |
| `test-node-reach-e2e.js` | test wrong. It waited for the reach map
whenever any *link* had GPS; `public/node-reach.js` builds the map only
when the *node* has coordinates. Guaranteed 10s timeout on a node with
positioned neighbours and no position of its own. |
| `test-channel-modal-e2e.js` | both failures test-side. The Add
button's visible label was shortened to "+ Add" with the accessible name
moved to `aria-label` (`public/channels.js:746`); and
`.ch-section-mychannels` is conditional on the visitor having added a
channel, which has not happened at that point in the suite. |
| `test-touch-targets.js` | three test-side, two a product finding. |

The three test-side touch-target failures were the harness measuring
controls that are not shown: `.compare-btn` (the CTA was removed in
#1646, and `style.css` says so), `.ch-back-btn` (`display:none` outside
the mobile channels layout), and `.filter-toggle-btn` (`display:none` on
mobile since #1461; the control shown is the navbar mirror, which
`mobile-page-actions.js:70` builds as a `.nav-btn`, so it was already
measured). All three are dropped from the table with the reason recorded
in the file.

The remaining two are **not** a test problem: `.nav-btn` and
`.ch-icon-btn` are each declared twice in `public/style.css`, 48px in
the touch-target block and 44px in their own component rule, and the
later one wins. Rather than lower the blanket or hide the failures, the
suite now has `DEFAULT_MIN = 48` plus a `MIN_OVERRIDES` table holding
those two at their effective 44, so a third selector dropping to 44
still fails the build. The contradiction is **#2052**, with both ways
out costed; the override entries should go when it is settled.

## Step 3 — the five that could not run

None is deleted. I checked each selector and seam against the product
before deciding, and every one still targets a surface that exists and
that nothing else covers.

Four were written against `@playwright/test`, a runner the project
neither installs nor uses anywhere else. Adopting a second runner for
twelve tests costs more than porting them, and the precedent is already
set: `test-path-inspector-coverage-e2e.js` exists, as its own header
says, because `test-path-inspector-e2e.js` could not run. So they are
ported to the plain-node Chromium pattern the other 109 suites use.

- **`test-issue-1522-trace-url-sync-e2e.js`** — the trace hash in the
URL, both directions. `test-e2e-playwright.js` covers that the page
loads and searches; it never looks at the URL, which is the whole of
#1522.
- **`test-marker-outline-weight.js`** — the canvas pulse ring never
thins below 2px. There is no CSS rule to read and axe cannot see inside
a canvas, so sampling the seam is the only way. Added a guard that the
ring was actually visible, so the weight check cannot pass vacuously on
a pulse that never rendered.
- **`test-pr-1490-live-map-gpu-animations-e2e.js`** — the queue drains,
the engine sleeps again, the fading trails stay under the cap of 5, and
the canvas sits on `animationsPane` rather than under the markers.
- **`test-path-inspector-e2e.js`** — reduced to what nothing else
covers: the map side pane, the `/#/traces/<hash>` redirect, the tools
landing. Its standalone-page test duplicated the wired coverage suite
and is dropped. Its "switching candidate clears prior polyline" case
ended after the click with a comment and no assertion, which is the same
green-but-empty problem this issue is about; it now compares path
counts, and skips loudly when the fixture yields too few candidates.

The fifth, **`test-table-sort.js`**, needed `jsdom`, which was declared
nowhere. It is a unit test of `public/table-sort.js` filed under
`tests/e2e`, so: `jsdom` is a devDependency (lockfile updated, `npm ci`
stays consistent), the file moved to `tests/unit/`, and the
`domIntegration` group in `scripts/non-unit-tests.json` is gone with its
only member. It runs 22 tests. 20 passed immediately; 2 had rotted,
because #1648 M2 replaced the up/down glyphs with Phosphor sprites and
the direction moved out of `textContent` into the `<use href>`. Those
two now read the sprite ref and the `aria-sort` value, so they also
guard the accessible announcement.

## Verification

`tests/unit/test-table-sort.js` 22/22 and `test-test-inventory.js` pass
locally; the E2E suites need a fixture server, which I cannot build here
(no cgo toolchain since #1992), so CI is their first run as committed.
The triage above was measured in CI, not assumed.

## Not done

The per-assertion skips named in the second comment on #2037 are
untouched: the two flaky packet-detail cases, the fixture-data ones, and
the two `clientRxCoverage` suites that skip wholesale while reporting
success. Those need a fixture deployment with coverage enabled, which is
its own change. I have not opened it.

Option 2 from the issue, making `test-test-inventory.js` require a
`deploy.yml` line for every `tests/e2e` file, is also not here. It is
the right guard and it is now enforceable, since the list is finally at
zero, but it belongs in its own change where a red build means what it
says.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-22 11:45:11 +02:00
efitenandClaude Opus 5 5c016a210a fix(analytics): show a building state on the distance index's 202, and stop caching it (#2051) 
Closes #1997.

## What was wrong

`/api/analytics/distance` answers `202 {status:"building",
retry_after_seconds:5}` with no `summary` until the lazy index (#1011)
has been built.

1. `renderDistanceTab` read `data.summary.totalHops` straight away. The
TypeError is caught by the tab's own try/catch, so it never reaches
`window.onerror`: it is painted into the tab as `Failed to load distance
analytics: Cannot read properties of undefined (reading 'totalHops')`.
2. `api()` caches any `res.ok` body, and `res.ok` is true for 202, so
the placeholder was stored for the `analyticsRF` TTL. Even a correct
retry read the cached "building" body back. That is the half that made
the broken state outlast the index build.

## What changed

- `public/app.js:173` skips the cache write when `res.status === 202`. A
200 still caches, unchanged.
- `public/analytics.js` renders a building notice and retries itself,
honouring `retry_after_seconds` clamped to [1s, 30s]. The timer is
cleared on tab switch and in `destroy()`, and a new render supersedes a
pending retry, so two renders cannot write into the same tab.

The notice uses `.text-center`/`.text-muted` rather than the `.spinner`
class used at `analytics.js:1683`, because `.spinner` has no CSS
anywhere in the repo and renders nothing.

## Tests

`tests/unit/test-issue-1997-distance-building.js` (7 assertions, wired
into `test-all.sh`) pins the two pure decisions the renderer makes and
`api()`'s refusal to cache a 202 while still caching a 200. Red-run on
the unfixed sources: 6 of 7 fail, and the "a 200 is still cached"
control stays green.

`tests/e2e/test-issue-1997-distance-building-e2e.js` (classified in
`scripts/non-unit-tests.json`, invoked from `deploy.yml` with
`CHROMIUM_REQUIRE=1`) serves both responses by route interception, so it
does not depend on whether the server under test has an index built. It
asserts: the building state appears, the tab does not paint the error
text, a retry arrives with no interaction, the retry replaces the
placeholder once the server answers 200, and no retry fires after
leaving the tab.

Check (2) deliberately asserts on the rendered text and not on
`pageerror`: the TypeError is caught, so a `pageerror` assertion would
pass on the broken build too.

## Verification

No local cgo toolchain here since #1992, so I could not build a server
to run the E2E against. Instead I ran its five steps in Playwright
against a live instance with this branch's `public/app.js` and
`public/analytics.js` injected in place of the deployed ones (both files
are byte-identical between that instance and upstream master, so the
injection is faithful):

| check | deployed build | this branch |
|---|---|---|
| (1) building state shown | fail | pass |
| (2) not rendered as data | fail | pass |
| (3) retried on its own | fail (1 request) | pass (2 requests) |
| (4) real payload after retry | fail | pass |
| (5) no retry after leaving the tab | pass | pass |

(5) passes on the broken build too: it schedules no retry at all, so it
is a control and only means anything together with (3).

The committed E2E suite has not been run as committed. CI is its first
real run.

## Not done

- The server still recomputes on every 202 poll rather than signalling
readiness.
- No other analytics tab was audited for the same assume-a-summary
pattern.
- One pre-existing unit suite (`test-preflight-xss-gate.js`) fails on
this Windows machine with a cp1252 `UnicodeEncodeError` from its Python
helper, on a clean tree as well as with this change. Unrelated, and
green on Linux CI.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-20 17:04:37 +02:00
liquidraver 84eba41592 fix(live): anchor the legend toggle to .live-page so the VCR bar stops eating its clicks (#2049) 
The PACKET TYPES legend could not be dismissed on Live: clicking the palette toggle did nothing, while document.querySelector('#legendToggleBtn').click() from the console worked. That asymmetry was the diagnosis, since a synthesised click skips hit testing.

Residual half of #1833. That fix corrected the button's offset to calc(var(--vcr-bar-height) + 10px) but left the anchor: .legend-toggle-btn was position: fixed, so the offset resolved against the viewport, while the .vcr-bar it clears is absolute inside .live-page, whose height subtracts --bottom-nav-reserve (56px + safe-area at <=768). The two anchors disagreed by exactly the reserve, dropping the button into the bar's band, and .vcr-bar at z-index 1000 against the button's 500 took every real click. The window is 641-768px: below 641 both buttons are display:none, above 768 the reserve is 0, which is why neither the desktop nor the phone layout test saw it.

Both buttons are now position: absolute, resolving against the same containing block as the bar. .feed-show-btn carried the identical defect but missed the bar by 16px because --legend-toggle-stack parks it a row higher, so that change prevents a break rather than repairs one.

Three review rounds, each verified rather than argued:
- The new E2E was classified in non-unit-tests.json but had no deploy.yml line, so it would never have run (#2037). Now wired in, and it covers both buttons.
- The '.feed-show-btn is display:none at <=768' claim was wrong (it is <=640), which the author measured and corrected.
- Its first real execution failed on its own setup: getComputedStyle().getPropertyValue() on an unregistered custom property returns the unevaluated declaration ('calc(56px + 0px)'), so the non-zero check could not work. It now asserts the measured gap instead, and deliberately does not pin 56 because the gap measures 58 on CI.

Final run: 14 passed, 0 failed. The discriminating assertion is (f), each button's offset net of the bar being identical at 720 and 1440 (11px and 123px slack); under the old anchor those differ by the reserve. (g) pins the desktop layout unmoved.

Merged by the interim maintainer without a second human reviewer: CI and the review above are the independent checks.

Fixes #1833
 2026-09-20 14:00:45 +02:00
efiten 29c0a3ba2f ci: name each E2E suite in the log before it runs (#2046) 
The E2E steps invoke 105 suites in a row. GitHub echoes a step's whole script once at the top, so the per-suite output then runs together with nothing between it, and a line like 'node-reach-coverage E2E SKIP (clientRxCoverage disabled on this deployment)' cannot be attributed to its suite without reading the sources and guessing. That is how test-node-reach-coverage-e2e.js came to pass every build while asserting nothing, unnoticed until #2037.

Each invocation now prints '=== E2E SUITE: <file> ===' first, so the log is greppable per suite.

Safe by construction: the banners go to stdout only, never through tee, so e2e-output.txt is byte-identical and scripts/aggregate-e2e-pass.sh sees what it saw before (it keys on digits followed by 'passed', which a banner never matches). The change is mechanical and was checked as such: the diff removes no line and every added line is a banner, 105 for 105 invocations.

Verified on CI run 35347268407: all jobs green, and the banners make attribution work. Using them, exactly one wired suite skips wholesale (test-node-reach-coverage-e2e.js) and three skip individual assertions (test-e2e-playwright.js on a flaky case, test-touch-gestures-coverage-e2e.js and test-issue-1306-collisions-terminology-e2e.js on fixture gaps). That list was guesswork before.

Refs #2037. Merged by the interim maintainer without a second human reviewer: CI is the independent check.
 2026-09-19 11:46:56 +02:00
efiten 20b0ee9742 ci: run the eight tests/e2e suites that had no runner at all (#2045) 
Part of #2037. Of the 113 suites in tests/e2e, 18 were invoked by no deploy.yml line, no script and no workflow: written, classified in scripts/non-unit-tests.json, and never executed.

All 18 were run once on upstream master to find out which still work (CI run 35340835178, each tolerated and timed so one run produced the whole table). Nine passed; eight are wired in here, 45 seconds together: test-analytics-fluid-charts (2s), test-nodes-export-e2e (2s), test-1110-live-filter (3s), test-e2e-1267-mobile-vcr (6s), test-live-dedup (6s), test-issue-1274-legend-coverage (7s), test-issue-1648-m5-icons (9s), test-show-neighbors (10s).

The ninth, test-rx-coverage-mobile-nav-e2e.js, is deliberately left out: it exits 0 with 'SKIP (clientRxCoverage disabled on this deployment)' and coverage is off by default, so it would add the appearance of a guard rather than a guard.

Skip paths were checked rather than assumed: test-issue-1648-m5-icons honours CHROMIUM_REQUIRE and gets the flag, test-nodes-export-e2e's skip branch needs an empty dataset and the fixture holds 182 named positioned nodes of 200, and the other six have no skip path.

Verified on run 35343304933: all jobs green, each of the eight invoked exactly once in the Playwright job, and none of them printed a SKIP.

Left for #2037: the four that run and fail (test-channel-modal-e2e 12/2, test-packets-scope-column 4/3, test-touch-targets 5 assertions, test-node-reach-e2e TimeoutError) and the five that cannot run as node scripts (four import @playwright/test, one jsdom, neither is a dependency).

Merged by the interim maintainer without a second human reviewer: CI is the independent check.
 2026-09-18 14:47:22 +02:00
liquidraver bbf54cfe1b fix(rx-coverage): open at the configured map default, with its own saved viewport (#2033) 
The coverage page opened at a hardcoded [51.0, 4.8] zoom 8 regardless of deployment, ignoring /api/config/map (#2032). It now follows the same precedence as the main map (URL hash, then saved position, then /api/config/map, then [37.6, -122.1] zoom 9) and persists its own position across visits, syncing lat/lon/zoom into the hash so a view is shareable.

The saved position lives under its own key, rx-coverage-view, and the page never reads or writes the main map's map-view: sharing the configured default was the bug, sharing the session position was not. Both suites assert map-view stays untouched after a pan, so reintroducing a shared write fails instead of passing quietly.

Also fixed here: selectedRx is now percent-encoded into the hash, and a generation counter stops a late /api/config/map response or a stale 150ms layout timer from building a map for a page that was already left.

Reviewed twice. Verified by mutation rather than by reading: writing map-view too, ignoring the saved coverage position, and dropping the /api/config/map fetch each make the unit suite exit 1, so it covers the feature, the fix and the rejected alternative. The deploy.yml invocation was confirmed to land inside Run Playwright E2E tests (fail-fast) by parsing the workflow, and CI run 35261689694 is the E2E suite's first real execution: Go prints 'RX coverage viewport regressions OK' and Playwright prints 'RX coverage viewport browser regressions OK'.

Worth recording for the next reviewer: that E2E asserted localStorage.getItem('map-view'), so wiring it into deploy.yml without updating the assertion would have turned the job red on its first ever run. It was registered in scripts/non-unit-tests.json but invoked by nothing, the gap tracked as #2037.

Merged by the interim maintainer without a second human reviewer: CI and the mutation checks above are the independent checks.

Fixes #2032
 2026-09-17 22:02:36 +02:00
Alex B 893773338e chore(tests): move root test-*.js into tests/unit and tests/e2e (#2036) 
Moves 290 root test-*.js into tests/unit (177, listed in test-all.sh) and tests/e2e (113, classified in scripts/non-unit-tests.json), per #1981 and PR-D of #1385. Root goes from 348 entries to 48. test-all.sh and test-fixtures/ stay put. The inventory guard now fails if a test reappears in the root or sits in the wrong folder.

Verified independently of the diff: the invoked sets are unchanged (test-all.sh 177 before and after, deploy.yml 96 before and after, both identical as sets), and a full local run of test-all.sh on master and on the branch produced 4702 output lines each whose only differences are absolute paths, stack-trace line numbers shifted by the REPO_ROOT line, the inventory wording and two perf ratios. The guard was mutation-checked: a test back in the root, a unit suite in tests/e2e, and a suite dropped from test-all.sh each make it exit 1. CI run 35246304316 ran 97 suites from tests/e2e and is green.

Follow-up 9335c51d finished the instruction files: no bare root test command is left in AGENTS.md, the squad charters, .github or docs, and every tests/ path they name resolves.

Merged by the interim maintainer without a second human reviewer: CI and the local runs above are the independent checks.

Known and deliberately out of scope: 18 of the 113 files in tests/e2e are invoked by no runner at all, and one of them cannot run anywhere because it requires jsdom, which is not a declared dependency. Tracked separately.
 2026-09-17 19:06:07 +02:00
efitenandClaude Opus 5 b8c8d98e61 fix(release): keep every platform when re-tagging :edge as a release (#2031) 
## Problem

`crane mutate` works on one image, not on an index. Pointed at the
multi-arch `:edge` tag it silently resolves the default platform, so the
fast path published v3.11.0 as a single amd64 OCI manifest, and `crane
tag` then pointed `v3.11`, `v3` and `latest` at that same manifest.
`docker pull` on arm64 against any of those four tags fails.

Verified in the registry:

| tag | shape | arch |
|---|---|---|
| `v3.9.2`, `v3.10`, `v3.10.1`, `edge` | index, 4 children | multi-arch
|
| `v3.11.0`, `v3.11`, `v3`, `latest` | `oci.image.manifest.v1`, 17
layers | amd64 only, revision `a2ea18f7` |

Earlier releases are indexes, so this only hit v3.11.0. The GitHub
release and both `corescope-decrypt` binaries are unaffected.

## Change

- The fast path now reads the `:edge` manifest, mutates each runnable
platform child by digest (`/app/.image-version` plus the version label,
as #1807 intended) and reassembles an index with `crane index append`. A
single-platform `:edge` still takes the old single mutate path.
- Attestation manifests (`platform.architecture == "unknown"`) are not
carried over: they reference the pre-mutation digests, so copying them
would attest the wrong images.
- A new verification step compares the platform set of `vX.Y.Z`, `vX.Y`,
`vX` and `latest` against `:edge` and fails the run if any of them
differs. A release tag that resolves to one platform is worse than a
slow release, so this should break the build rather than ship.
- Scratch tags (`tmp-vX.Y.Z-linux-amd64`, ...) are deleted best-effort
afterwards; the index references the manifests by digest, so leaving
them behind is only untidy.
- Added `workflow_dispatch` with a `tag` input to republish the images
for an existing release. A dispatched run resolves the tagged commit
itself, because `github.sha` is then the ref the workflow file came
from, and it skips the `deploy.yml` dispatch: that release already
exists and releases here are immutable (the trap from #1955/#1956).

## Tests

None: this repository has no harness that executes workflow files, and
the CI jobs cannot reach a step that pushes to GHCR. What the change is
verified against instead:

- `crane index append` accepts `-m/--manifest` repeated plus `-t/--tag`,
with the base index optional, so building an index from scratch is
supported (crane docs for `index append`).
- The YAML parses and every `run:` block passes `bash -n`.
- The platform comparison was run by hand against the live registry:
`:edge` reports `linux/amd64,linux/arm64` and `v3.11.0` reports
`single`, which is exactly the case the new step must fail on.
- The real test is the dispatch on `v3.11.0` right after merge, which is
also the repair. If the verification step fails there, nothing is
published and the tags stay as they are.

## Not verified

- The scratch-tag delete needs `delete:packages`; `GITHUB_TOKEN` may not
have it. It cannot fail the run.
- Whether GHCR keeps the attestation manifests attached to `:edge`
reachable after the index is rebuilt for a release tag (they stay on
`:edge` itself, which is untouched).

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 11:07:18 +02:00
Sylvain Rabot a2ea18f778 perf(sqlite): swap modernc.org/sqlite for mattn/go-sqlite3, cross-built with zig (#1992) 
Swaps the SQLite driver from `modernc.org/sqlite` (pure Go, SQLite
3.46.0) to `github.com/mattn/go-sqlite3` (cgo, bundled SQLite 3.53.4),
and pays the resulting cross-compilation cost with `zig cc`.

Draft because the riskiest part of this deletes rows — see [Please
review this part first](#please-review-this-part-first) — and because
three things remain unverified at the bottom.

`modernc.org/sqlite` is a transpilation of the C amalgamation. This repo
is read-heavy: `cmd/server` chunk-loads a graph at startup and fans out
neighbour/topology/analytics queries per request, and it pays for that
transpilation on exactly those paths. Head-to-head on the same
120k-transmission / 240k-observation database, running our own hot-path
SQL under both drivers (Apple M4, `-count=5`, medians):

| workload | modernc | mattn | |
|---|---:|---:|---|
| chunk load (`chunked_load.go` v3 join, 20k tx) | 449ms | 196ms |
**2.3×** |
| aggregate scan (240k-row join + `GROUP BY`) | 276ms | 137ms | **2.0×**
|
| 1500 prepared-statement lookups | 512ms | 403ms | **1.3×** |

Allocations fall with it: 1.12M vs 1.64M allocs and 21MB vs 30MB on the
chunk load.

**Superseded by a production run.** @efiten measured both drivers on a
real instance — 11,077,038 observations, 9.7GB database, 4-core arm64 —
as server-only containers against the same live volume, one at a time,
with round 2 reversing the order so the page cache favours the old
driver:

| | audit 7d | audit 24h | background fill (13 chunks) | start →
/api/health |
|---|---:|---:|---:|---:|
| modernc, round 1 | 16.67s | 2.27s | 130.2s | 16.6s |
| mattn, round 1 | 7.87s | 1.34s | 93.8s | 13.5s |
| mattn, round 2 | 8.15s | 1.35s | 96.4s | 13.0s |
| modernc, round 2 | 13.46s | 2.29s | 137.8s | 15.5s |

Warm, the old driver improves to 13.46s on the 7d audit and still loses
by ~1.8×. Chunk load is ~1.4×. `/api/nodes?limit=500` is 0.039s against
0.037s — nothing.

**So the real gain is ~1.4–1.8× on the paths that matter, not 2–2.3×.**
The shape the harness predicted holds — scans and joins gain, small
lookups do not — which is more reassuring than the magnitude would have
been. Quote these numbers.

**The counterweight**, cold and native on that machine: a build goes
from **52s to 163s**. An instance that builds its own image pays that
per deploy.

## The build is cgo now, and one thing about that is a trap

**`CGO_ENABLED=0` still builds.** mattn links a stub, and the binary
dies on its first query with `go-sqlite3 requires cgo to work. This is a
stub`. A green build is not evidence of anything here, which is why
`AGENTS.md` now says so explicitly. `GOOS=linux go build` genuinely
cannot cross-compile any more.

A new root `Makefile` is the entry point. `make crossbuild` uses `zig cc
-target {x86_64,aarch64}-linux-musl` and links static, so each artifact
stays a single self-contained file and the `alpine:3.20` runtime no
longer depends on the base image's libc at all.

`-Wl,-s` is load-bearing: Go's own `-s -w` does not reach the musl
objects zig links in, and without it the server binary is 19.8MB instead
of 12.1MB.

The Dockerfile keeps its single `$BUILDPLATFORM` builder — still no QEMU
for compilation — and gains a checksum-pinned zig plus BuildKit cache
mounts. The mounts are not a nicety: without them an image build
recompiles the amalgamation from cold and takes over half an hour.

## Please review this part first

`internal/dbschema/dedup_index.go` **deletes observation rows**. It is
the one part of this change that can lose data, and it exists because
the migration exposed a real bug rather than causing one.

`stmtInsertObservation` resolves its `ON CONFLICT` against
`idx_observations_dedup`, which `cmd/ingestor/db.go` only ever created
inside the branch that creates the `observations` table for the first
time. Any database whose table predates that branch never got one, so
the UPSERT had no conflict target. modernc failed on the first insert;
mattn fails at `OpenStore`. Same bug, found earlier.

Creating the index unconditionally repairs it — but the index is what
was supposed to prevent duplicates, so a database that never had it can
already hold rows violating it. **`test-fixtures/e2e-fixture.db` in this
repo holds one.** So duplicates are collapsed first. Refusing is not the
safer option: without the index the ingestor cannot prepare its UPSERT,
so it cannot start at all.

Replaying that UPSERT faithfully is subtler than it looks, and a first
cut of this got it wrong twice:

- `COALESCE(excluded.x, x)` means the **incoming** value wins, so down a
group in id order the survivor keeps the **last** non-NULL value. Taking
the first silently discarded newer readings.
- The UPSERT names exactly five columns (`snr`, `rssi`, `score`,
`raw_hex`, `resolved_path`). Every other column must keep the surviving
row's own value; merging those too invents history the ingestor would
never have written.

Merge, delete and `CREATE UNIQUE INDEX` now share one transaction. Split
apart, a writer inserting a duplicate in the gap fails the index
creation while leaving the deletions committed — rows destroyed and no
index to show for it.

Cost, measured on 2.4M synthetic rows holding 5 duplicates: **4.1s**,
holding the write lock throughout, once, at ingestor startup before MQTT
subscribe. Materialising the duplicate-group scan once rather than per
column took that from 9.7s; the pathological case (400k of 600k rows
duplicated) is 5.7s, slightly worse than the 4.2s it was before that
change.

## Four more behavioural differences

Full detail in `docs/sqlite-driver-migration.md`. Briefly:

**Statement preparation is eager.** modernc's `newStmt` stored the SQL
and compiled lazily; mattn calls `sqlite3_prepare_v2` inside `Prepare`,
so SQL naming a missing table fails at *open*. 59 server tests failed on
this alone, all fixtures with partial schemas. `OpenDB` keeps failing
loudly (#1901; `main.go` gates on `dbschema.AssertReady` anyway) and the
fixtures now declare what they are prepared against via
`ensurePreparable`. This also exposed nine `nodes(pubkey …)`
declarations across seven files, where production has only ever had
`public_key` — lazy compilation had hidden the mismatch for as long as
it existed.

**`synchronous` silently dropped FULL → NORMAL.** mattn defaults it to
NORMAL and executes the pragma unconditionally, where SQLite's own
default (what modernc left alone) is FULL. In WAL mode that weakens
durability under power loss. Pinned in `dbschema.WriterDSN`, which both
writers now share — `cmd/migrate` kept a bare path at first and so
quietly wrote at NORMAL, which is what a second copy of a DSN buys you.

**The DSN dialects are mutually invisible.** modernc understood only
`_pragma=name(value)`, mattn only `_`-prefixed parameters, and neither
errors on the other's form — a driver-only rename would have dropped
every pragma in silence. `_journal_mode=WAL` is also gone from the
server's read handle: modernc ignored it, mattn honours it, and setting
`journal_mode` on a read-only connection is a write. Dropping
`_busy_timeout` with it costs nothing, since mattn already defaults to
5000ms — which means the read handle finally *gets* the busy timeout it
had silently lacked.

**`mode=ro` survives for a non-obvious reason.** mattn always passes
`READWRITE|CREATE` and its amalgamation has `SQLITE_USE_URI=0`; what
makes the URI work is its C wrapper ORing `SQLITE_OPEN_URI` in. So the
#1283/#1289 invariant holds with no build flags — but it depends on the
`file:` prefix. `cmd/decrypt` had been building its DSN without one, so
its `mode=ro` had never applied and a missing path was created
read-write. Fixed in passing; never a migration regression.

## What did not change

No modernc-specific API was in use: no `RegisterFunction`, no
`*sqlite.Conn`, no `sqlite/lib` error constants, no `sql.Register`. No
`time.Time` is ever bound as a query argument, so driver time handling
is not in play. Both drivers convert declared
`DATE`/`DATETIME`/`TIMESTAMP` columns to `time.Time`, so
`/api/dropped-packets` keeps emitting `dropped_at` as RFC3339 — an
earlier draft "fixed" that with a `CAST` and would have been the
regression.

## Tests and CI

New regression tests, each written because something got through without
it:

- `TestEnsureObservationsDedupIndexKeepsLatestValues` — the merge
ordering. The original test used complementary NULLs, which passes
whichever direction you pick, which is why the bug survived it.
- `TestCollapseDuplicatesAndIndexIsAtomic` — a failed index creation
must roll the deletions back.
- `TestOpenStorePragmas` / `TestWriterDSNPragmas` — every writer pragma,
read back through the store's own connection. A separate `sqlite3`
session or the startup log line would prove nothing.
- `TestOpenDBRefusesMissingDatabase` — the read-only invariant, which
now rests on a detail of the driver's C wrapper.
- `TestEnsurePreparableMatchesPrepareStatements` — fails when a new
prepared statement outgrows the fixture helper.

CI gains test execution for `cmd/migrate` and `internal/dbschema`, which
had none and both open the database. A PR-time two-arch build plus an
arm64 QEMU smoke gate is new: the GHCR push is push/tag-only, so without
it nothing on a PR would exercise zig, static musl linking or arm64, and
the first signal would arrive on master. `cache-dependency-path` widens
from 2 of the 5 tracked `go.sum` files to all of them.

`make test` passes across all 14 modules, `cmd/server` also under `-race
-count=2` with no failures and no races. `gofmt` and `go vet` clean.
Release-routing and Dockerfile COPY-invariant gates pass.

## Verified by running

- All 8 cross-builds static and correct-architecture; both arches of the
container image built, exported and run under QEMU, serving
`/api/health` and `/api/nodes` against a 2.9M-observation production
snapshot.
- The `migrate` binary repairing that snapshot's duplicate on bare
Alpine.
- `CGO_ENABLED=0` producing a binary that builds and then fails on first
query.

## Not verified

- ~~The 2–2.3× figures come from a standalone harness, not this load
under the old driver.~~ **Closed** by @efiten's production run above,
which also corrected the multiplier.
- SQLite 3.46.0 → 3.53.4 query-planner differences on queries with no
total `ORDER BY`.
- Sustained live ingest through the new writer DSN, and the duplicate
collapse against a database an ingestor is actively writing to. Verified
against a static snapshot only, and the collapse is measured at 4.1s on
2.4M synthetic rows with 5 duplicates — well short of an 11M-row
instance. @efiten has offered a staging instance taking real MQTT
traffic; **this is the item to close before the PR leaves draft.**

An earlier revision of this branch shipped the dedup merge in the wrong
direction with a green test suite, and review then found three more
things in the same file: the repair gated on an error string, a
non-atomic TEMP table drop aimed at the wrong connection, and a deletion
whose only record was a row count. All fixed in ac7e8d38. Passing tests
did not establish safety here, which is why the deletion path wanted a
second pair of eyes rather than a rubber stamp.
 2026-09-16 09:02:13 +02:00
n30nex cd9b4c04d0 test: unify frontend test runs and prevent inventory drift (#1965) 
Make `test-all.sh` the authoritative standalone frontend runner for npm
and CI. Restore stale assertions and reject missing, duplicate, removed
or undocumented inventory entries.

Fixes #1858.

Rebased onto `a2f039d4`. Retains release-routing, map scope-state and
Scope Audit stylesheet tests, adds `test-packets-local-channels.js` to
the sorted runner, and classifies `test-neighbor-map-btn-clip-e2e.js`
under browser. Its separate CI browser step is preserved. Inventory: 280
root suites, 167 standalone, 113 requiring separate setup.

The icon repair fixes two suites red on master:
`test-issue-1648-m2-emoji-scan.js` and
`test-issue-1648-m6-final-sweep.js`. Node/live configured-scope
confirmations now use the existing accessible Phosphor check sprite.
Values, visibility conditions and scanner assertions are preserved.

- Red evidence: `89e45a9` inventory assertions; `4e255df` accessible
confirmation assertion. The latest rebase also reproduced both
unclassified-file failures before adding their entries. This follow-up
only changes runner/classification configuration and counts; no test
files modified.
- Local validation: all 167 standalone suites; 27 M2 browser checks and
16 neighbor geometry checks in Chromium. Syntax, whitespace, PII,
CSS-variable and XSS checks passed.
- Browser coverage includes populated/empty/null configured scopes in
node and live views.
- No new dependencies, requests, application settings or Go changes.
Workflow outside the unit step matches master.
- Windows validation uses process-local UTF-8 settings. Encoding and
node-reach confirmation follow-ups remain separate, as requested.

## Preflight override

External `run-all.sh` is unavailable; applicable repository checks were
run directly.
 2026-09-13 19:19:39 +02:00
nullrouten0 a2f039d4c7 fix(packets): show browser-added channels in the channel filter (#2009) 
Verified against production: ingestor writes enc_<HH> (db.go:2282), store.go:3536 filters on it, live values are uppercase two-digit hex with thousands of packets (enc_28: 2708), /api/channels omits them, and /api/packets?channel=enc_28 returns rows. New test passes at 14 and fails when the dedupe guard is disabled.
 2026-09-12 12:40:44 +02:00
nullrouten0 6d7490e7da fix(tables): stop the trailing action column cropping its own button (#2010) 
Reproduced on a live instance: 41px column for a 53px button at 1440px, 29px at 1024px, max-width:0 and overflow:hidden on the cell. Injecting the .col-action rule took it to 65px with nothing cropped. The new test fails 13 of 16 checks when only the CSS block is removed.
 2026-09-12 12:38:58 +02:00
efiten 51a2a7dd2f fix(scope-audit): ship the stylesheet the page paints its badges with (#2005) 
Fixes #2004. Two review rounds; findings and evidence on the PR. Verified by injecting the stylesheet into a running deployment and reading computed styles before and after: the badges gain background, size, uppercase and padding; at 430px the Config column stays visible; the sorted column keeps its accent. The new test fails on the eight unstyled classes against the pre-fix tree, on .ns-truncated against the first fix, and on a re-added column-hiding rule.
 2026-09-11 15:35:52 +02:00
efiten 296456f9c1 feat(map): colour and filter repeaters by scope-configuration state (#2006) 
Closes #2001. Two review rounds plus a re-review; findings and evidence on the PR. Verified on a deployment against live data: the field over 1653 nodes, marker tints per filter state, the marker title and popup Scope row reaching the DOM, and the colorblind-preset cascade. The audit and the map are held to the same classification by an end-to-end test that fails when either side's wildcard handling drifts.
 2026-09-11 15:07:31 +02:00
efiten 6c92a8b612 test(ingestor): make the suite race-clean, and run the detector when it matters (#1994) 
`go test -race ./...` on `cmd/ingestor` reports **six data races** on
master. None is in production logic. All six come from test helpers that
outlive the test that started them, which is why the detector blames
whichever test happens to be running: two different tests failed on two
consecutive runs of the same code.

## What was racing

**`StartStatsFileWriter` had no way to stop.** Two tests start it at a
50ms interval, and its goroutine then runs for the rest of the process.
It reads the package-level `readProcSelfIOFn` hook, which a later test
replaces to inject a fake, so the write and the read race. The same leak
explains the stray log lines about writing stats into temp directories
that were already cleaned up.

It now returns a stop function that closes the goroutine and waits for
it to exit. Production ignores the return value and runs for the process
lifetime exactly as before; the two tests call it through `t.Cleanup`.

**The migration test read a log buffer while a goroutine wrote to it.**
`log.Logger` serialises its own writes, but `logContains` read
`buf.String()` outside that lock while `RunAsyncMigration` kept logging
after the call that started it had returned. The capture helper now uses
a mutex-protected buffer.

That one is worth calling a real race rather than a test artefact: a
concurrent read during a buffer grow can panic outright with "concurrent
map read and map write"-class behaviour, not merely trip `-race`.

## Measured

`go test -race ./...` on linux/arm64 under go1.27.1: **exit 0, zero
races, 704s**.

## The CI job, and why it is shaped this way

The server has had `-race` since #1208. This closes the same gap for the
ingestor, which carries an `atomic.Pointer` snapshot (the region key set
from #1989) whose safety has been an argument rather than a measurement.

Two deliberate choices, because a check that costs too much gets
switched off:

- **Its own job, not a step inside "Go Build & Test".** Appending
`-race` there puts its ten-odd minutes on the critical path, taking the
pipeline from roughly 20 minutes to roughly 32. As a separate job it
runs beside the E2E job (15-17 minutes) and hides inside that window.
- **Only when `cmd/ingestor/**.go` changed**, decided by the existing
change-scope job, which already gates the heavy jobs on
documentation-only PRs. A frontend or docs PR cannot introduce a data
race in the ingestor. Pushes to master always run it, as they already do
for `code`.

Nothing `needs:` the new job. Adding it to `build-and-publish` would
mean a skipped race job skips everything downstream, which is the
opposite of what a conditional check should do. It reports as its own
check; whether that blocks a merge is a repository setting rather than
workflow logic.

## Scope

Test helpers, one production signature (`StartStatsFileWriter` now
returns a stop function), and the workflow. No change to what the
ingestor does at runtime.
 2026-09-10 22:30:27 +02:00
Anupam MedirattaandClaude Opus 5 de237fc29c fix(qa): bind TEST_PUBKEY as a SQLite parameter instead of interpolating it (#1982) 
Closes #1977. Supersedes #1952. Follow-up filed as #1983.

## What §10.2 did

```bash
q="SELECT COUNT(*) FROM transmissions WHERE from_node = '$TEST_PUBKEY';"
qq=$(printf %q "$q")
if ! count=$(ssh_t "docker exec … sqlite3 … $qq" 2>/dev/null); then
  count=$(ssh_t "sqlite3 … $qq" 2>/dev/null || echo "")
fi
```

The injection is not reachable today — `TEST_PUBKEY` is hex-gated and
the
script `exit 2`s before the SQL is built. The problem is that the SQL
layer's
safety rests entirely on that outer gate rather than on the SQL layer
itself.
#1952 proposed doubling embedded quotes; that is string escaping, not
parameterisation, which is why it was withdrawn in favour of this.

## What this does

Per the four points in the sign-off on #1977:

**1. Bind the value.** A constant `SELECT` and a bound `:pubkey`, fed to
sqlite3 on stdin. The SQL no longer crosses the remote shell as a
command
word, so there is no `printf %q` on the query at all any more.

**Why hex rather than `.parameter set :pk '<value>'`.** Dot-command
arguments
are split on whitespace, so a payload containing a space produces too
many
arguments — and sqlite3 responds by printing the `.parameter` help to
**stdout**, exiting **0**, and leaving `:pk` **unbound**. `COUNT(*)`
then
returns 0, which reads exactly like a passing security fix. `-bail` does
not
catch it. Verified on 3.51.0:

```
$ printf ".parameter set :pk '' OR 1=1 --'\nSELECT COUNT(*) FROM transmissions WHERE from_node = :pk;\n" \
    | sqlite3 -bail ptest.db
.parameter CMD ...       Manage SQL parameter bindings     # <- help, on stdout
   …
0                                                          # <- :pk never bound
$ echo $?
0
```

`.parameter set :pk 1+1` also binds the integer `2` — the value is
evaluated
as an SQL expression and only falls back to a text literal when
evaluation
fails. So interpolating into the `.parameter set` line trades one hazard
for
another.

Hex-encoding removes the quoting layer instead of adding one: the value
is
bound as `cast(x'<hex>' as text)`, so its contribution to the SQL text
is
drawn from the alphabet `[0-9a-f]` only. Nothing to quote, no tokenizer
arity
hazard, and it holds for **arbitrary** input rather than only for
hex-gated
input — which is the point.

Verified against a fixture table holding two rows, one of them
`deadbeef`:

| value | result | exit |
|---|---|---|
| `deadbeef`, bound as `cast(x'6465616462656566' as text)` | `1` | 0 |
| `' OR 1=1 --`, bound the same way | `0` | 0 |
| `' OR 1=1 --`, interpolated the current way | `2` (whole table) | 0 |
| query against a DB with no `transmissions` table, `-bail` | `Parse
error … no such table` on **stderr** | **1** |

**2. Probe the capability, not a version.** `resolve_sqlite_runner`
binds
`corescope-probe-ok` and asserts it comes back — a round trip, not a
bare
`.parameter init`, so the positive control runs against the operator's
actual
binary rather than one we pin. If neither the container nor the host
qualifies,
it fails loudly and names what is needed:

```
  ❌ retain-failed: no sqlite3 able to bind a parameter on the target
     tried: docker exec -i corescope-prod sqlite3, then sqlite3 on runner@example
     need:  the sqlite3 CLI reachable over ssh, supporting '.parameter set'
OCI runtime exec failed: exec: "sqlite3": executable file not found in $PATH
bash: line 1: sqlite3: command not found
```

There is deliberately **no** interpolating fallback. That would leave
the
vulnerable path in place under a nicer name.

**3. The hex gate is kept**, with its comment updated to say why: for
the SQL
layer it is now defence in depth rather than the only guard. Redundant
is not
the same as wrong.

**4. The exit status and stderr survive.** `-batch -bail -init /dev/null
-noheader -list` (stop at the first SQL error; ignore the operator's
`~/.sqliterc`, where a stray `.mode` would make the count unparseable;
stdout
is exactly the number). Query stderr is captured and printed on failure
rather
than sent to `/dev/null`, so a broken query is distinguishable from a
legitimately empty result. Probe stderr is collected too, and printed
only if
*both* probes fail — the container miss is the known-normal case, so
surfacing
it on every run would be noise.

## Also fixed

An existing double-count in §10.2: the `TARGET_DB_PATH unset` branch
incremented `$fails` and then left `count=""`, so the generic branch
incremented it a **second** time for the same failure.
`read_retain_count` now
gives §10.2 exactly one increment point. Opportunistic cleanup in a file
already being touched (AGENTS.md line 318).

## Tests

New `qa/scripts/test-blacklist-sql.sh`, wired into the `go-test` job. 24
assertions, modelled on `scripts/staging/test-disk-monitor.sh`.

Both directions are asserted, because a zero from a command that failed
proves
nothing:

- **Positive control** — `deadbeef` still returns its row (`1`, exit 0),
and so
  does `cafebabe`; an absent pubkey returns `0`.
- **Negative** — `' OR 1=1 --` returns `0` while the table demonstrably
holds
2 rows, and the old interpolated form is asserted to leak all `2`. That
last
  assertion is what makes the `0` above worth something.
- **Error surfacing** — the same SQL against a DB with no
`transmissions` table
  exits non-zero with a message on stderr and nothing on stdout.
- **Alphabet** — `sql_hex_literal` output matches `^x'[0-9a-f]*'$` for
the SQL
payloads, a backslash, `$(id)` / backticks, an embedded newline,
`héllo`, and
a 4096-byte repetitive string. That last one is a regression guard for
`od
  -v`: without the flag `od` collapses repeated identical lines to `*`.
- `run_sqlite` with no resolved runner refuses rather than guessing.

Group 2 skips loudly (rather than silently) if `sqlite3` is not on PATH;
group
1 needs no sqlite3 and always runs.

**Mutation-tested** — each of these breaks the suite, so the assertions
have
teeth:

| mutation | caught by |
|---|---|
| restore full interpolation | `injection payload → 0 rows — expected
'0' got '2'` |
| naive `.parameter set '%s'` | `expected '0' got '.parameter CMD ...'`
|
| drop `od -v` | alphabet failure on `*`, plus `expected '8192' got
'33'` |

Commit 1 is a behaviour-neutral refactor that moves the imperative body
into
`main()` behind a `BASH_SOURCE` guard, so the test can source the script
and
exercise individual helpers. Same idiom as
`scripts/staging/disk-monitor.sh:99`.

## Verification

- `bash qa/scripts/test-blacklist-sql.sh` → 24 passed, 0 failed
- `bash -n` on both scripts
- All three runtime paths exercised end to end with PATH shims for
  `ssh`/`docker`/`sqlite3` against a real fixture DB: success
  (`sqlite3 runner: host`, count 2), query failure (classified message +
`Parse error … no such table`, `fails=1`), and no-capability (the loud
block
  above, both probe stderrs, `fails=1` — not 2)
- The new step lands inside `go-test`, which runs when
`changes.outputs.code ==
  'true'`; `qa/scripts/*.sh` does not match that job's
  `^docs/|[.]md$|^LICENSE$` documentation filter, so it is not skipped

## Deliberately out of scope

- **The `docker exec` branch is dead on current images** → filed as
#1983. The
app container has no `sqlite3` at all: `Dockerfile:15` is pure-Go SQLite
with
  no CGO, and the `apk add` installs only `mosquitto mosquitto-clients
supervisor caddy wget`. So the host "fallback" is the only path that has
ever
executed, silently, because both branches discarded stderr. This change
keeps
both branches and merely makes the outcome visible (`sqlite3 runner: …`
on
  every run).
- **`-readonly` on the target DB.** Tempting, and verified compatible
with
  `.parameter` (the binding table lives in the TEMP database), but a WAL
database needing journal recovery can refuse a read-only open. Adding it
here
risks exactly the "trades an unreachable injection for a script that
does not
  run" outcome flagged in the #1952 thread. Worth its own issue.
- **The other `2>/dev/null` sites** in this file, which also sit
awkwardly with
`qa/README.md`'s "Don't silence stderr". Only the §10.2 lines named in
the
  sign-off are touched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-09-09 15:37:40 +02:00
Sylvain Rabot 3fbff01f64 build: upgrade Go toolchain to 1.27 (#1946) 
## Summary
- Bumps the Go toolchain used to build/test to 1.27:
`golang:1.27-alpine` in `Dockerfile` and `Dockerfile.go`, and
`go-version: '1.27'` in the three `actions/setup-go` steps in
`.github/workflows/deploy.yml`.
- Each module's `go.mod` `go` directive is intentionally left at `1.22`
— no 1.27-only language features are being adopted, and a 1.27 toolchain
builds a `go 1.22`-declared module without issue.

## Test plan
- [x] `go build ./...` + `go vet ./...` for all 13 modules
(`cmd/server`, `cmd/ingestor`, `cmd/migrate`, `cmd/decrypt`, 10
`internal/*` packages) under Go 1.27.0
- [x] `go test ./...` passes for `cmd/server`, `cmd/ingestor`,
`cmd/migrate`, `cmd/decrypt`
- [ ] `docker build` against the new `golang:1.27-alpine` base (Docker
wasn't available in the sandbox this change was prepared in — needs a
check in CI or locally)
 2026-09-09 11:49:33 +02:00
n30nex 2288e28d4e fix: publish release artifacts after successful image retagging (#1964) 
Successful release fast paths publish image tags but never dispatch the
job that creates the GitHub release and decrypt binaries. Dispatch
`deploy.yml` from both image routes. A default-off `images_published`
input skips E2E and image rebuilding only for an already-published tag;
missing or mismatched images keep the complete fallback without
requiring new inputs on older workflow definitions.

Go validation still gates the release binaries, checkout and version
flags retain the tagged source, and the existing release action uploads
both architectures before publication. Missing binary files now fail
publication.

Fixes #1956.

Validation:

- `node test-issue-1956-release-routing.js` executes the actual workflow
shell steps with registry and dispatch commands stubbed. Covers
matching, missing and mismatched images; failed retag and Go validation;
branch/PR boundaries; and both tagged binary commands.
- The original test commit fails because a matching image dispatches
zero artifact workflows; the fix passes the same assertion.
- Existing release workflow Go checks, decrypt/channel tests, YAML
parsing and actionlint pass.
- Both static Linux amd64 and arm64 binaries cross-build with verified
architecture and version metadata.

Actual registry publication and GitHub release creation were not
exercised. Existing immutable releases and old tags that contain older
workflow definitions are outside this fix.

Following #1922, this is a focused release-routing PR. A separate repair
for #1858 rewrites the shared frontend test runner; merging this first
lets that repair retain this regression in its authoritative list.
Please assess current Go and E2E job results separately from
workflow-approval or staging-runner state.
 2026-09-06 21:20:16 +02:00
efitenandClaude Opus 5 a3ee37011a ci: scope docs-only skipping to jobs, so required checks still report (#1955) 
Same change as #1954, opened from a branch on this repository instead of
from a fork. #1954 never received a workflow run: zero runs and zero
check suites for its head commit, and closing and reopening it changed
nothing. A manual `workflow_dispatch` on master ran immediately, so
Actions itself is working; the `pull_request` event from the fork is
what produces nothing. See the note at the end.

Replaces the trigger-level `paths-ignore` from #1949 and #1950. That
approach was wrong, and it is currently blocking #1953 from merging.

## What was wrong

GitHub documents the distinction I had backwards:

> a workflow skipped by path filtering keeps its checks **pending** and
blocks the merge, while a **job** skipped by an `if:` conditional
reports **Success** and does not.

So the filtering has to live on the jobs, not on the trigger.

I compounded it by claiming, in both the commit and the description of
#1949, that master had no required checks: *"verified: the branch
protection endpoint returns 404"*. **That verification was invalid.** A
404 there means the token cannot read protection details, not that none
exist. The branch reports `protected=true`, and #1953 was refused with
`the base branch policy prohibits the merge`.

## What this does

A `🔎 Change scope` job computes whether anything outside `docs/`, `*.md`
and `LICENSE` changed. `go-test`, `e2e-test`, `build-and-publish` and
`release-artifacts` are gated on its output. A documentation-only pull
request skips those jobs, they report Success, and the PR can merge.

**Only pull requests are scoped.** A push or a dispatch always runs the
full pipeline.

That second part is deliberate and it fixes a separate failure.
`release-fast-path.yml` re-tags `:edge` to `:vX.Y.Z` only when the
`:edge` revision label matches the tagged commit. A master commit with
no image breaks tagging, which is what happened to the v3.10.0 tag: the
tagged commit was documentation-only, the fast path could not re-tag, it
fell back to a dispatch, and the dispatch published nothing (#1951).
Master pushes now always produce an image. The cost is running the
pipeline on documentation commits to master; pull requests are where the
queue pressure was.

Two conservative defaults in the scope check: a non-`pull_request` event
and an empty diff both count as code, so an unexpected shape runs
everything rather than silently skipping.

## Note for whoever has repository settings access

Fork pull requests stopped getting workflow runs between 22:36 and
05:40. #1949, #1950 and #1951 all came from the same fork and each got a
run; #1954 got none, with no check suite created at all, which is
different from a skipped run. `repos/.../actions/permissions` returns
403 for a non-admin token, so this could not be confirmed from the API.
If the "Fork pull request workflows from outside collaborators" setting
was tightened, that would explain it, and it would affect every outside
contributor, not just this branch.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-04 09:44:18 +02:00