dandri/meshcore-analyzer
1
0 Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-11-11 11:57:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,126 Commits 85 Branches 42 Tags
417c8dd00a243f29822f34e5c9aaadbd2e522213
Commit Graph
102 Commits
Author SHA1 Message Date
TeTeHackoandClaude Opus 5.5 417c8dd00a feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) (#2300) 
Closes #2270, closes #2271.

The ingestor took only the confirmed region scopes out of an observer's
`/neighbors` report (#1865) and dropped the rest. Nothing could
therefore say when an observer last reported, or whom it heard.

## Ingestor

`observer_neighbor_reports` (created in `dbschema.Apply`) keeps the
latest report of each observer, one row per observer, replaced by every
live report.

The report is unauthenticated, so `neighbors[]` is stored sanitised:

- only 64-hex keys
- `snr` and `heard_secs_ago` finite and in range
- `status` limited to `[a-z_]`
- `scopes` clamped
- at most 500 rows

`total_neighbors` is believed only when it is not below the number of
rows that arrived.

A **retained replay** only fills a missing row, and it is dated by the
report's own `timestamp`. After an ingestor restart, a long-dead
observer therefore does not look as if it just reported.

## API

The table is optional for the server (no `AssertReady` change); without
it the fields are simply absent.

- `GET /api/observers`: `neighbors_report_at` and `neighbors_count`,
from one batch query in the existing cached builder.
- `GET /api/observers/{id}`: also `neighbor_report`, which is
`{received_at, reported_at, neighbor_count, queried_neighbors,
self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr,
heard_secs_ago, status, scopes}]}`. Names and roles come from the node
table in one query.

## Frontend

`public/observer-neighbors.js` is shared by both pages and escapes
everything.

- **#2270, observers list:** a sortable **Neighbors** column, e.g. `3h
ago · 12`, or `—` when the observer never reported. The same value
appears in the slide-over.
- **#2270, observer page:** **Neighbors Report** and **Neighbors** stat
cards.
- **#2271, observer page:** a **Neighbors** table listing each neighbor
by name (a link to the node) or key prefix, with SNR, heard, scope query
status and scopes. The heading says when the list is shorter than the
count, e.g. `(2 of 3 listed)`.

## Tests

- **ingestor:** sanitising, the cap, live replace, the retained rules
(fill only, dated by the report, undated ignored) and the MQTT topic end
to end. The existing #1865 scope tests are unchanged and still pass.
- **server:** the list summary, the detail with names, and both
endpoints without the table.
- **frontend:**
  - unit tests of the helpers
- an XSS case for every rendered field; I checked that it fails when
escaping is removed
- `test-observers-headings` now expects the new column after `Last
Packet`
- **E2E:** `test-observer-neighbors-2270-e2e.js` intercepts the observer
endpoints, since the fixture has no reports. It is listed in
`scripts/non-unit-tests.json` and on shard 2.

Gates that pass:

- `go test -race` in `cmd/server`, `cmd/ingestor` and
`internal/dbschema`
- `sh test-all.sh`
- `eslint@8`, with 0 errors
- the XSS diff gate
- these E2E suites: the new one,
`test-issue-1639-observers-sort-e2e.js`,
`test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js`

Performance: the list adds one primary-key-sized query per cache fill,
and the detail adds two small queries per request.

Also fixes #2330 (folded in from #2337, second commit).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-29 22:36:53 +01:00
n30nexandn30nex 202bafea28 fix(observers): show metadata while analytics loads (#2339) 
Partial follow-up to #1778, limited to observer details; other pages
remain separate.

Observer identity, status and radio information now render when the
basic request finishes. Analytics and clock calibration load into their
own regions, so either can finish first. An analytics failure preserves
usable metadata. Identity failures and obsolete responses cannot
overwrite the current view, and failed chart construction is cleaned up.

The same three requests start together. Generation ownership, request
deduplication, date-window behavior and disposal remain covered. No new
fetches, settings, dependencies or customizer options; secondary replies
preserve the actual metadata DOM node.

[RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426343054/job/121932999182)
proves 14 new assertion failures while all 11 existing unit cases pass.
Local RED browsers reproduce 20 new failures while all 32 old cases
pass. GREEN `52eabad7` passes 221 standalone suites, 28 ownership unit
checks and 56 browser cases across desktop/mobile and both themes; the
parent independently repeated the 56 browser cases.

All three independent review roles passed. Held-response assertions
prove the rendering behavior without claiming a server-latency speedup.
[Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427087948)
passes on this exact head, including all three browser shards, the
repeated slide-over stability gate and the architecture build.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
 2026-10-29 18:34:12 +01:00
n30nexandn30nex 1ed5e4ab3f fix(compare): exclude observer TX-only packets from overlap (#2346) 
Red commit: `39295da8` ([intended CI assertion
failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)).
Green commit: `d1bda496`.

Observer comparison counted an observer's TX-only packets as receptions,
producing false overlap. It now requests `excludeTx=true`: the selected
observer and non-TX direction must match the same observation in memory
and SQLite v2/v3. Normal packet lists and stored TX records remain
unchanged; missing directions retain legacy compatibility.

Refs #667. This completes a comparison slice, not the issue's remaining
observer counters or #671's fleet/SNR feature. Representative display
fields and existing time-filter semantics are unchanged. API/OpenAPI
docs describe the filter and multi-node bypass.

Validation:
- RED: Go tests compiled with 32 intended failures/16 control passes;
unit and browser regressions reproduced false overlap. CI reached the
new unit assertion.
- GREEN: all 48 packet cases pass, plus grouped-cache isolation and
OpenAPI tests. The added cache regression caught a malformed-radio
cache-key collision before correction.
- Full local server suite: 174s; ingestor: 185s; `npm run test:unit`:
pass. Final focused packet/grouped/OpenAPI checks: pass.
- E2E assertion added to
`tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks
pass, including correct 1/2 reception totals and 50% overlap.

Performance: still two capped bulk requests, indexed candidate lookups
retained, one direction check within the existing observation scan/SQL
predicate, no new persistent structures. No configuration/customizer
changes.

Final-head CI passed; independent reviews found no must-fix items.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
 2026-10-29 18:34:06 +01:00
n30nexandn30nex c211ef7643 feat(nodes): show relay activity from certain forwarding evidence (#2336) 
Fixes #2331 with the additive API option.

The Last Relayed row shows green **actively relaying** only when
`last_certain_relay` falls within the existing
`healthThresholds.relayActiveHours` window. Missing, invalid, future or
expired evidence adds no idle/alive claim. Existing `relay_active`,
attributed timestamps and counts retain their meanings.

The optional node list/detail field uses the existing certain-relay
path: an eligible observed frame, matching hop, unique prefix, no
retired namesake and existing signature checks. Originating an advert
does not count. Forwarding another node's advert can count, even with
zero non-advert attributions. Adverts with an unknown origin cannot
establish forwarding.

Evidence uses transmission `first_seen`, not late observation receipt
time, and follows node-membership eviction. Snapshots take one read lock
over requested keys, O(page size), with no packet scan or new database
query. Work per qualifying hop stays bounded. No new dependency or
setting; configuration documentation and OpenAPI explain the
distinction.

The test-first history includes assertion-failing RED commits and GREEN
fixes. The initial RED's nil-return compile stub is explicitly permitted
by `docs/agents/TDD.md`. Local native server/ingestor suites, focused
race checks, 809 frontend helper assertions and 70 browser cases pass.
Browser coverage includes both themes and full/side/tablet detail views.

[Review regression
RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40421464618/job/121916723339)
proves both malformed-origin failures. All three independent review
roles pass at `d1fe79cb`. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40422655728)
passes on that exact commit, including all three browser shards and the
two-architecture image check.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
 2026-10-29 18:33:59 +01:00
efitenandClaude Opus 5.5 15e2d2a4a7 fix(packets): read HB from the row's own raw_hex header (#2351) 
## Problem

The HB column on the packets page can show `4`. Path mode 3 (hash size
4) is reserved: `Packet::isValidPathLen` in `src/Packet.cpp` returns
false for it, and `Dispatcher::checkRecv` drops such a packet. So HB 4
is never a real value.

The three HB cells in `public/packets.js` found `path_len` with
`getPathLenOffset(p.route_type)`, the route type of the
**transmission**. The raw_hex in a child row (and in a live-updated row)
is an **observation's** raw_hex. The packet hash does not cover the
route bits, so the same packet is heard as FLOOD by one observer and as
TRANSPORT_FLOOD by another, and the transmission keeps the route type of
its first observation.

Example from a production DB (transmission 3645334):

| source | raw_hex start | path_len at |
|---|---|---|
| transmission | `01 12 3A646A…` (FLOOD) | byte 1 |
| observation | `00 E2E0 0000 1D 3A646A…` (TRANSPORT_FLOOD) | byte 5 |

Read at the FLOOD offset, the observation gives `0xE2`, `0xE2 >> 6 = 3`,
`+ 1 = 4`. In the last 300,000 observations of that DB, about 4,300 have
a header route type that differs from their transmission's.

## Fix

`pathHashSize(rawHex)` in `public/app.js` already takes the route type
from the header byte of the same raw_hex, and the detail pane uses it.
The three HB cells now call it through a small `hashBytesText` helper
instead of carrying their own copy of the offset logic.

Behaviour change: a raw_hex with no hash size (reserved mode 3, a DIRECT
packet with no hops, missing raw_hex) shows the same placeholder as
TRACE rows instead of a number. Before, missing raw_hex showed `1`.

## Tests

- New `tests/unit/test-hb-column-header-route.js` (7 cases, registered
in `test-all.sh`): transport raw_hex on a FLOOD transmission gives 1,
flood raw_hex with 2-byte hashes on a TRANSPORT_FLOOD transmission gives
2, reserved mode 3 never gives 4, for the helper, the flat row and the
group header. It fails on `upstream/master` with `'4' !== '1'`.
- `sh test-all.sh`: all standalone frontend suites pass.
- Browser: local server on a migrated copy of the E2E fixture, with one
observation rewritten to a TRANSPORT_FLOOD raw_hex. The expanded group
shows HB 2 on all three rows. The old code reads `0xE2` on that child
and shows 4.

## Performance

Same per-cell work as before: one header parse and one byte parse per
rendered row, O(1), no new allocations beyond the existing regex test in
`pathHashSize`. Only visible rows are rendered.

## Not done

- The child-row call site is not hit by a unit test: child rows only
render when the group is in `expandedHashes`, which the test API does
not expose. It is covered by the helper test and the browser check
above.
- The HB sort accessor (`hash_byte_count` / `hash_size`, sort key `hb`
in `packets.js`) is unchanged; it does not read raw_hex.
- `transportBadge(c.route_type)` on child rows still follows the stored
route type, not the observation header. Left out of scope.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-29 06:42:51 +01:00
n30nex 3bf678709f fix(ui): preserve navigation and menu focus during delayed updates (#2333) 
Fixes #2332. Fixes #2335.

Two delayed UI updates could undo user actions. Rebuilding the More menu
removed its focused link; a late packet-list response could replace a
newly requested Nodes URL before `hashchange` ran.

The menu now preserves the selected destination when it stays in More or
moves inline, closes an empty menu, and leaves other controls' focus
alone. Packet URL updates now require the packets route, including its
existing detail/query forms. Observation selection and encoded filters
remain supported.

The changes add one lookup across the existing 12 navigation links and a
URL ownership check. No packet loop, API request, dependency, setting or
customizer change is added.

[Packet
RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409110743/job/121874676123)
fails four wrong-route assertions. [Menu
RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409898375/job/121877663579)
fails the intended focused-destination assertion after the packet
correction removes the competing navigation failure. GREEN is
`57522143`.

Local combined validation passes all 221 standalone frontend suites, 60
navigation browser checks, 17 branding/route checks and 11
filter/deep-link checks. The packet regression controls delivery of a
real API response; substituting the original code still fails it.
Existing exact title assertions remain; diagnostics now report the
actual title. Backend source trees are unchanged from the tested
baseline.

[Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40410894023)
passes on `57522143`, including all three browser shards and both
architecture builds. Three independent source reviews found no blockers.
Final repository checks and after-push browser verification pass. Ready
for maintainer review.
 2026-10-28 21:55:40 +01:00
efitenandClaude Opus 5.5 d29957fbdc feat(observer-logins): lock the shared login out of migrated observers (#2348) 
Follow-up to #2306 / #2319. With personal observer logins, a migrated
observer still is not safe from the shared login: the hook answers
`ignore` for the shared login, so EMQX's built-in database lets it
publish under any observer's topic, including one whose owner has moved
to a personal login. This PR lets an operator close that per migrated
observer, before the shared login is removed altogether.

## Changes

**users.db v8.** `observer_bindings.confirmed_at`: the moment the
owner's personal login first published under that pubkey.
- An auto-bind (the owner's first publish) is confirmed at once.
- An admin assignment stays unconfirmed until the owner's login
publishes under that pubkey; that publish sets it.
- Reassigning to another user or unbinding clears it; reassigning to the
same owner keeps it.
- Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin`
rows stay NULL.

**Config.** `userManagement.observerLogins.sharedUsernames` (default
empty): the broker's shared login names. A name starting with `obs_` is
refused at startup.

**Authz hook.** For a login in `sharedUsernames`, a publish under
`meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is
confirmed, whatever the number of topic segments (the ingestor
attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey).
Logged as `publish denied: user=<name> observer=<8 hex>: confirmed
owner`. Every other request from a shared login keeps `ignore`, so the
built-in database decides as before. Personal logins are unchanged.

An orphaned binding (owner account deleted) does not count as confirmed.

**Admin.** The observers table of the Observer logins tab gets a
Confirmed column; the bindings JSON gets `confirmedAt`.

**Docs.** `docs/observer-mqtt-logins.md`: "Locking the shared login out
of migrated observers" (what it denies, how to see `confirmedAt`, the
risk below). `config.example.json` has the key.

## Risk for operators

An observer that is confirmed but still runs a second uploader on the
shared login loses that second uploader once its name is in
`sharedUsernames`. The deny log line names the observer.

## Tests

- `internal/users`: v7 to v8 migration on a real v7 database, auto
confirm, admin assignment then owner confirm, non-owner and repeated
confirm, reassign clears, orphan reassign.
- `cmd/server`: the decision table (confirmed, unconfirmed, unbound,
non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty
and `client` iata, lowercase key; owner publish confirms, confirm
failure still allows and retries; reassign, unbind and user removal
clear the flag; the index load against a real store; config validation;
the hook and admin endpoints end to end.
- `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an
auto-bound observer.
- Local: `internal/users`, `cmd/server` (two Windows-only tests skipped,
as on master) and `cmd/ingestor` green. The E2E assertion was not run
locally; CI runs it.

## Not in this PR

- Using `confirmed_at` for verified `/neighbors` reports: a separate PR,
built on #2300.
- Removing the shared login: still an operator step after migration.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-28 19:10:41 +01:00
TeTeHackoandClaude Opus 5.5 77a5f14925 feat(observers): radio setting per observation from the observer's radio history (#1869) (#2292) 
Part of #1869.

## Problem

`observers.radio` keeps only the **last** `MHz,kHz,SF,CR` an observer
reported. Once an observer is retuned (say from SF7 to the EU narrow SF8
setting to listen to a neighbouring mesh), every packet it ever heard
reads as heard on the new setting, and there is no way to tell an SF7
packet from an SF8 one.

## What this does

As #1869 proposes, the packet data model is untouched: the setting of an
observation is resolved from its observer and time.

**Ingestor** (all schema in `dbschema.Apply`)

- `observer_radios`: one row per stretch of time an observer, and each
radio of a multi-radio node, stayed on one setting. Every live status
extends the current row and a change starts a new one; retained replays
never extend a row. On the first run it is seeded with every observer's
current radio, which is what the UI already showed for the past.
- Settings are stored canonical through `dbschema.CanonicalRadio`, so
`869.4320068,62.5,7,5` and `869.432,62.50,7,5` are one setting.
- `observation_radios` (`observation_id → rx_radio_id`): multi-radio
openHop nodes tag each packet with `rx_radio_id` and list `radios: [{id,
radio}]` in their status, so their observations resolve per radio.
- The table is sparse: single-radio nodes never send the field, so
`observations` and every query that scans it stay untouched.
  - The row id comes from the observation upsert via `RETURNING id`.
- Rows are pruned with their transmissions, and a purged observer takes
its history along.

**Server** (read-only)

- No `AssertReady` change: an absent table reads as empty and is probed
again later, the same rule as `advert_route_evidence`.
- The history is held in memory. The live poller checks it on every tick
(two primary-key lookups), so a retune or a new `rx_radio_id` is known
before the observation is broadcast. A full reload runs every 10 min.
- `radio` is added to observations, packets, grouped rows (taken from
the header observation) and the live feed.
- `GET /api/packets?radio=MHz,kHz,SF,CR` filters by setting.
- Combined with `observer=`, it means *that* observer heard it on that
setting.
- A radio query is served from memory instead of falling back to SQL,
which would return the window unfiltered.
- `GET /api/radios` (named struct) and `radio_history` on `GET
/api/observers/{id}`.

**Frontend**

- Packet filter fields `radio`, `freq`, `bw`, `sf`, `cr`, e.g. `sf == 8`
or `freq > 869.5 && bw == 62.5`.
- A Radio row in the packet detail and a Radio column in its
observations table (only when the server reports one).
- A radio history table on the observer page once there is more than one
stretch.
- `RfParams.radioText` formats the setting. It escapes all five
characters and returns non-numeric input as it came, escaped.

## Performance

`txMatchesRadio` runs once per observation inside `filterPackets`.

- Each observation costs one map lookup by observer id.
- Observers with a single stretch (nearly all) answer from that lookup
directly.
- Retuned observers add a binary search over their few stretches.
- Multi-radio observers add one lookup by observation id.
- The snapshot is loaded once per filter call.

`BenchmarkTxMatchesRadio`, 30K packets / 150K observations (20
observers, 2 of them retuned):

```
BenchmarkTxMatchesRadio-24    602    2056924 ns/op    0 B/op    0 allocs/op
```

Memory is one entry per stretch plus one per multi-radio observation;
nothing is added per observation otherwise.

## Tests

- **dbschema:** seed runs once, canonical spellings, the tables exist
without `observers.radio`.
- **ingestor:**
  - stretches extend and split; canonical spellings do not split
- two radios of one node hearing one transmission over two paths each
keep their own radio
  - a retained status does not write history
  - prune and purge
  - `rx_radio_id` reaches `PacketData`
- **server:**
- resolution before, between and after stretches, and with an unknown
radio id
  - `observer` + `radio`, and grouped rows carry `radio`
  - incremental refresh, `/api/radios`, `radio_history`, missing tables
  - the benchmark above
- **frontend:** the new filter fields in
`tests/unit/test-packet-filter.js`, and XSS cases through the real
`RfParams.radioText` and the history renderer.
- **E2E:** `tests/e2e/test-radio-setting-1869-e2e.js` checks the grouped
rows' radio, `sf ==` on the packets page, the detail Radio row and the
observer history. It is added to `scripts/non-unit-tests.json` and to
shard 2.

`go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema`
passes, along with `sh test-all.sh`, `eslint@8` (0 errors) and the XSS
diff gate. The new E2E test and `test-e2e-playwright.js` pass against
the fixture.

## Browser validation

This runs on a live instance with six observers, one of them retuned to
869.618/62.5/SF8/CR8:

- The packet detail shows `Radio: 869.432 MHz · SF7 · BW 62.5 kHz · CR
4/5`.
- `sf == 7` shows 106 of 106 grouped rows and `sf == 8` shows 0 of 107;
the SF8 observer has not heard anything yet.
- `/api/radios` lists both settings.
- No page errors.

## Notes

- **The customizer (rule 8)** gets nothing: the only constants are
server-side refresh intervals.
- **Attribution timing:** packets heard between a retune and the
observer's next status message (minutes) are attributed to the previous
setting, because a stretch starts when the status is handled. This is
documented at `RecordObserverRadios`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 21:08:40 +01:00
n30nex 643a53e98d fix(operators): refresh update metadata and wait for readiness (#2285) 
Red commit: `7277ca23` ([failing
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190626951)).

Fixes #2229. Management startup now defines logging helpers before
normalizing CRLF environment files. Updates refresh version, revision
and build time after checking out the target. Deployment verification
and all five Compose healthchecks use `/api/healthz`, so an available
statistics endpoint cannot report an unfinished startup as healthy.

- The actual-script regression passes 15 checks: LF/CRLF/no environment
file, repeat startup, all update targets, failed checkout,
delayed/failed readiness, HTTPS readiness and unchanged informational
statistics.
- RED CI reproduced all 11 intended failures; GREEN `7ff9050b` passes
the full standalone frontend suite and shell syntax checks.
- Browser verified: an isolated Go fixture at http://127.0.0.1:18851
renders Home and returns ready=true from `/api/healthz`.
- Git, Docker and network boundaries are mocked in management tests. No
real container update was performed locally; CI validates the
Compose/build path.
- No new configuration, dependencies, schema changes or
packet-processing work.

Privacy, whitespace and scope checks passed. The external home-directory
preflight bundle is unavailable; applicable checks were performed
locally. Independent review found no outstanding fixes. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191647297)
passed on `7ff9050b`, including all three browser shards and both
architecture builds.
 2026-10-26 15:50:36 +01:00
n30nex b5caea1d5f fix(analytics): render safe, correctly sized battery status icons (#2282) 
Red commit: `4cc2efb3` ([failing
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40189921192)).

Fixes #2232. Battery badges now render their fixed status icon and
append voltage as text, including zero. Chart sizing excludes inline
icons, and status colors use the existing theme variables. Unknown
status values stay muted.

- The original browser regression failed on literal SVG; a second local
run with only the JavaScript fix failed on oversized icons.
- Validation on `eef0872e`: all standalone frontend suites; 47 focused
unit checks; 41 Chromium checks, including both themes, hostile voltage,
accessibility and 16×16 icon bounds.
- Browser verified: http://127.0.0.1:18851 on an isolated Go fixture.
- E2E assertion added: `tests/e2e/test-issue-1648-m2-icons-e2e.js:134`.
- Constant work for one badge; no new requests, dependencies, settings
or schema changes.

Privacy, whitespace, CSS-variable and repository XSS checks passed. The
external home-directory preflight bundle is unavailable; its applicable
checks were performed locally. Independent review found no outstanding
fixes. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190620324)
passed on `eef0872e`, including all three browser shards and both
architecture builds.
 2026-10-26 15:44:23 +01:00
n30nex d07186af43 fix(nodes): preserve names and metadata in preview panels (#2315) 
Fixes #2309 and #2312. Partial follow-up to #1154 (U1 and tablet
timestamp parity).

Node previews omitted Last Relayed, Traffic share and Hash Prefix, while
tablet previews read `advert_name` instead of the API's `name`. Shared
rendering now presents those fields consistently across full detail,
side preview and tablet, with the existing timestamp tooltips.

Relay attribution is labelled neutrally: ambiguous relay evidence no
longer claims a node is alive. Existing certainty/status logic is
unchanged. Measured zero, legacy score fallback, role applicability,
missing values and escaped names/hash tooltips are covered. A resize
regression prevents duplicate score-row IDs across retained desktop and
tablet previews.

Only the selected node's metadata is formatted. Existing request counts
remain five for full/side and two for tablet; no new fetch, cache, hot
loop, setting, customizer value or dependency.

Validation: 806 helper checks, all 218 standalone suites, 46 browser
checks in both themes, eight health checks, and the main browser suite
(149 passes, three existing fixture skips). The unchanged backend tree
passed full server/ingestor suites. Independent reviews completed; the
parent verified the two-line follow-up from its resize check.

Test-only RED `e0f681ae` produced [eight expected assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40232409958/job/121286783705).
Follow-up `2bb1ccd` repairs the help fixture's scroll range without
relaxing its checks; all 13 help checks pass across three font/CPU
configurations. [Final-head CI
passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40234768485),
including all browser shards and architecture builds.
 2026-10-26 15:44:10 +01:00
n30nex 6e1abf0663 ui(nodes): identify path links as example packets (#2296) 
Red commit: `79abbba0` ([regression
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203075654)).

Fixes #2291; addresses U7 of #1154. The two observed-path links now say
View example packet →, accurately describing their existing
sample-packet destination. Recent-advert labels and link behavior remain
unchanged.

- Production diff: two text literals in public/nodes.js.
- Existing browser suite preserves contrast/highlighting assertions and
adds exact labels, canonical destinations and real click-through from
side/full/mobile views.
- RED fails eight label assertions. GREEN passes 15/15 checks normally
and with 4× CPU slowdown; all 218 standalone frontend suites pass.
- Browser verified: http://127.0.0.1:18902.
- E2E assertion added:
tests/e2e/test-issue-1146-path-link-contrast-e2e.js. Initial loads and
theme toggles wait for actual theme-refresh readiness before capture,
fixing a reproduced screenshot-detachment race in the touched harness.
- No runtime logic, requests, settings, dependencies or performance
changes.

Independent reviews and local preflight checks passed. External
home-directory preflight bundle unavailable; applicable checks were
performed locally. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204585697)
passed on `ae849cc6`, including all three browser shards and both
architecture builds.
 2026-10-26 15:43:57 +01:00
n30nex fb101e46e0 fix(nav): fit primary links around late account controls (#2286) 
Red commit: `e5589b4c` ([failing
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191348162)).

Fixes #2216. Navigation measures the available flex strip instead of
guessing component widths. Optional statistics and spacing yield when
primary/active links would overlap More. Account insertion triggers a
layout event immediately, including while authentication is pending or
fails; normal user-change events retain their semantics.

- Regression CI reproduced link overlap across narrow widths and late
account controls. Review added a second RED test for the actual delayed
authentication bootstrap before the three-line correction.
- Validation on `f0ccbf4c`: full frontend suite; 89 user-management
checks; 56 navigation checks; existing priority and clipping suites.
Parent independently reran auth and browser checks.
- Browser verified: http://127.0.0.1:18854; real bootstrap overlap of
14.66px becomes 21.38px clearance. Resizing, account changes, keyboard
More and touch targets remain covered.
- E2E assertion added: `tests/e2e/test-nav-fluid-1055-e2e.js:89`.
- Work is bounded by the fixed navigation link set; events share one
animation frame. No polling, observers, new requests, dependencies or
settings.

Privacy, whitespace, CSS-variable, XSS and ESLint checks passed.
External home-directory preflight bundle unavailable; applicable checks
were performed locally. Independent review verified the bootstrap
correction with no outstanding findings. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40192737545)
passed on `f0ccbf4c`, including all three browser shards and both
architecture builds.
 2026-10-26 15:43:43 +01:00
n30nex c853c64f1f feat(ui): identify current views in browser tab titles (#2287) 
Red commit: `a62753de` ([failing
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191439879)).

Fixes #2259. Browser titles now identify the current page, Analytics
tab, or selected node/packet while retaining the configured site name. A
Reach view reads `Node abababababab | Reach | CoreScope`. Detail
identifiers are limited to 12 characters; account/reset parameters stay
out of titles.

- Branding and page context have separate state, so delayed theme loads
and edits cannot erase the view or accumulate suffixes. Selection/close
paths that use replaceState also update the title.
- Validation: all standalone frontend suites; 29 focused unit checks; 16
browser steps covering routes, history, reload, tabs, real row clicks,
closing, branding and literal route text. An explicit All time selection
keeps the browser test independent of fixture age.
- Browser verified: http://127.0.0.1:18853 on an isolated Go fixture.
- E2E assertion added: `tests/e2e/test-customize-branding-e2e.js:48`.
- Fixed-size work at navigation/selection boundaries; no extra requests,
DOM observers, render-loop work, dependencies or settings.

Privacy, whitespace, XSS and ESLint checks passed. External
home-directory preflight bundle unavailable; applicable checks were
performed locally. Independent review found no outstanding fixes. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40193902834)
passed on `4c037db5`, including all three browser shards and both
architecture builds. The existing logo test now asserts the full
page-and-brand title; its 6/6 checks and the 16/16 title/branding checks
also pass locally.
 2026-10-26 15:43:30 +01:00
TeTeHackoandClaude Opus 5.5 684f89f200 feat(frontend): serve Leaflet, leaflet.heat and Chart.js from vendor/, not unpkg.com (#2299) 
Refs #1554.

`index.html`, `area-map.html` and `geofilter-builder.html` load Leaflet,
leaflet.heat and Chart.js from **unpkg.com** on every page view. That
causes three problems:

- **Privacy:** every visitor sends a request to a third party.
- **Availability:** where unpkg is blocked (script blockers, filtered
networks) or down, the map and every chart are gone. The observer page
then fails with `Error: Chart is not defined`.
- **#1554:** the CSP target there, `script-src 'self'`, blocks these
scripts just like the inline handlers the issue lists, but they are
missing from its inventory.

## Change

- **Vendored libraries:** `public/vendor/leaflet-1.9.4/` (with
`images/`), `vendor/leaflet-heat-0.2.0.js` and
`vendor/chart-4.5.1.umd.min.js`.
- They are the unpkg dist files byte for byte, verified against the SRI
hashes `index.html` already pins.
- The two standalone pages loaded Leaflet without SRI; they now carry
it.
  - This is the same pattern as `vendor/leaflet.markercluster.js`.
- **`.gitattributes`:** `public/vendor/** -text`. Otherwise `* text=auto
eol=lf` rewrites `leaflet.css`, which ships with CRLF, and the browser
refuses it for the integrity mismatch. I hit exactly that while
preparing this.
- **`scripts/instrument-frontend.sh`:** now copies `vendor/`
recursively. The flat `cp public/vendor/*` left the E2E server without
`vendor/leaflet-1.9.4/`, which showed up as `L is not defined`.
- **New test `tests/unit/test-vendored-libs.js`:**
  - no off-site `<script src>` or stylesheet in the served pages
  - every `integrity` attribute matches its vendored file
  - the images Leaflet's stylesheet references exist

  I checked that it fails against the old `index.html`.

## Checks

- `sh test-all.sh` passes.
- These E2E suites pass against the fixture: `test-e2e-playwright.js`,
`test-analytics-fluid-charts.js`, `test-issue-1236-map-mobile-e2e.js`
and `test-issue-1329-map-controls-accordion-e2e.js`.
- On a live instance, a headless browser with `unpkg.com`, `jsdelivr`
and `cdnjs` blocked:
  - the observer page renders all four charts
  - the map loads tiles and markers
  - `typeof L` and `typeof Chart` are defined
  - no CDN request is made and no page error occurs

## Not in this PR

Swagger UI on `/api/docs` (`cmd/server/openapi.go`) still loads
`swagger-ui-dist` from unpkg. I can vendor it in a follow-up if wanted.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 15:43:16 +01:00
n30nex e61ff6e473 fix(observers): keep detail responses tied to the current view (#2323) 
Fixes #2316 and #2320. Related to #1778; progressive loading remains
separate.

Delayed observer responses could replace a newer observer or period,
overwrite its error/status/calibration, and leave detached charts alive.
Each load now captures its observer and days and checks ownership before
success or failure can update the view. Leaving the page invalidates
pending loads, including same-observer revisits.

New visits also reset the request window to the selector's existing
seven-day default. Previously, a new page could display seven days while
requesting the preceding visit's 30-day window.

Production change: 12 added lines and three replacements in one frontend
file. No new requests, caches, settings or dependencies; the ownership
check is O(1). Existing request deduplication, comparison controls,
chart cleanup and current-request errors are preserved.

Validation: 11 unit checks, all 219 standalone suites, 32 controlled
browser cases across desktop/mobile and both themes, comparison 9/9, and
observer-IATA 5/5. Core browser run exited successfully with 149 passes
and three declared fixture skips; existing conditional subchecks remain.
The unchanged backend tree passed both full backend suites. The parent
independently reran the new unit/browser tests.

Test-only RED `a73c227f` produced [nine unit and 28 browser assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40239824064).
Three independent reviews passed. [Final-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40240673369)
passes on `ba52d873` integrated with upstream `11ce1235`, including all
browser shards and architecture builds.
 2026-10-26 15:42:56 +01:00
n30nex 9f6f318857 test(rf-health): wait for airtime badges after grid refresh (#2289) 
Fixes #2288. The RF Health airtime test now waits for the requested
observer's actual airtime badge. Waiting for any tile and then
immediately reading the badge could race with a theme refresh that
clears and rebuilds the grid.

This is a test-only synchronization correction: two lines replaced, no
application changes, and the existing value/warning assertions remain
unchanged. No sleeps, timeout increases, skips or retries were added.

- Existing RED evidence: [upstream-compatible CI
failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190851550/job/121161457287).
The test blob is identical on upstream `cb38ac35` and that PR's head.
- A controlled local reproduction holds the next summary response during
a real theme refresh. The old immediate read fails; the locator read
waits for the correct badge.
- Four local browser runs passed, including an independent parent run.
Wrong-value and missing-warning mutations still fail the original
assertion. Test inventory passed.
- Browser verified: http://127.0.0.1:18851 on an isolated Go fixture.

Privacy and whitespace checks passed. The external home-directory
preflight bundle is unavailable; applicable checks were performed
locally. Independent reviews found no outstanding fixes. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40193032972)
passed on `36d1bf4a`, including all three browser shards and both
architecture builds.
 2026-10-26 15:40:51 +01:00
efitenandClaude Opus 5.5 e149568c6c feat(observer-logins): personal MQTT logins for observers (#2306) 
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.

Closes #2319.

The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.

## Changes in this PR

**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.

**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.

**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.

Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.

**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.

**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.

**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.

**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.

## Enabling it on an instance

Full steps in `docs/observer-mqtt-logins.md`. In short:

1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.

## Who can claim an observer

Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.

What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.

## Performance

`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.

## Verification

- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.

- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.

## Known behaviour

EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.

## Not in this PR

- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.

Note on cb9f4ff3: its message says the scheme problem was reported by an
operator. That report turned out to come from before the broker used the
hook, so it does not show this cause; the change rests on the firmware
source only.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-26 09:50:20 +01:00
n30nex cb38ac356c fix(packets): preserve All time across reloads and links (#2279) 
Desktop All time now survives reloads and shared links, including a
selected older grouped packet and observation. An explicit zero is
retained in storage and the URL; missing/invalid values still use the
default. Mobile requests retain their existing time and row limits.

Fixes #2231.

Validation: RED reproduces the lost zero; all standalone frontend suites
and the unchanged server suite pass. The local main browser suite
(`test-e2e-playwright.js`) passes 149 checks with three explicit
fixture-dependent skips. The gesture suite also preserves the existing
time window when opening a packet trace. The new regression covers
selection, reload, fresh links, older grouped observations and mobile
bounds.

Normalization is constant work; URL updates reuse the existing
serializer and add no requests. No new settings or dependencies. Common
ingestor validation is covered by the separate #2272 test-stability
change.


[Exact-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40160982457)
passed on commit 5278ade6, including all three browser shards.
 2026-10-25 12:11:20 +01:00
n30nex 82aafb53c3 fix(analytics): count roles within selected filters (#2275) 
Analytics Nodes now counts roles from the same complete, filtered node
list used by Network Status. Region/area selections no longer display
the API's global role totals. The global API metadata remains available
to other consumers.

Fixes #2213.

Validation: RED reproduces the mismatching totals; all standalone
frontend suites and the unchanged server suite pass. Browser checks
cover region and area changes and empty results. Regressions also cover
501-node pagination, unknown roles, unchanged metadata and request
counts.

Counting happens inside the existing O(n) status pass, using six bounded
counters and no additional requests. No configuration changes. Common
ingestor validation is covered by the separate #2272 test-stability
change.
 2026-10-25 12:10:36 +01:00
n30nex 7ebbed58dc fix(analytics): anchor resize handles to table headers (#2274) 
Resize handles now use their static table header as the containing
block. Already-positioned headers, including sticky Nodes headers,
retain their existing positioning.

Fixes #2217.

Validation: assertion-failing RED commit followed by the fix; all
standalone frontend suites and the unchanged server suite pass. The
browser suite passes 48 checks covering desktop/mobile bounds, real
dragging, saved widths and sticky headers. The browser regression waits
for the initial theme render before measuring.

The fix adds one positioning check per header during setup, with no new
requests, dependencies or settings. Common ingestor validation is
covered by the separate #2272 test-stability change.
 2026-10-25 12:10:25 +01:00
efitenandClaude Opus 5.5 2bb000f2aa feat(notify): adaptive offline threshold and certain node analytics (#2260) (#2262) 
Part 2 of #2260. **Builds on #2261** (certain liveness): review/merge
that first; until then this diff also shows its commit.

## Adaptive offline threshold

`userManagement.notifications.adaptiveOffline` — **off by default**:

```json
"notifications": {
  "enabled": true,
  "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 }
}
```

The fixed `infraSilentHours` (72 h) treats every repeater the same. With
this on, a watched node's `node.offline` window is learned from its own
**certain** traffic (the #2261 rule):

1. Certain packets per hour over the week **before its last certain
packet**. Ending there, not at now, keeps an outage from dragging the
rate down until the fixed window returns and the node reads "back
online".
2. Empty runs are classified: ≤ 6 h and recurring on another day at the
same hours = normal silence (kept; the window is never shorter than the
longest + 1 h); any other run = outage, left out of the learning so it
neither stretches the window nor blocks learning.
3. λ = 10th percentile of the remaining hourly counts, so quiet nights
don't alert.
4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson,
ln 10⁶ ≈ 13.8).
5. Clamped to [`floorMinutes`, the role's silent window].

Kept on the fixed window: λ < 1/hour, under 24 h of history in memory,
or a 10th-percentile hour heard by fewer than `minObservers` observers
(one observer dropping out must not read as the node going offline).
Silence is measured up to the newest packet in the store, so a stalling
feed stops every node's clock. A node stored offline comes back only on
evidence from the moment it went offline on. The notifier caches each
learned window for 10 minutes and drops unwatched nodes.

For the #2260 repeater (~840 packets/day) the mail would have gone out
about 30 minutes into the outage, instead of never (or 72 h with #2261
alone).

## Certain node analytics

- Each `activityTimeline` bucket carries `certain`: the part that can
only be this node (`certainFor`, the packet-level form of the
`certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other
endpoints are unchanged.
- Availability and longest silence use certain buckets only; the longest
silence now includes one **still running** (an outage in progress is the
silence worth seeing).
- The chart stacks certain (`--accent`) and uncertain (`--text-muted`) —
existing theme variables, so no new customizer entry. Descriptions
updated.

## Performance

`CertainActivity` walks a week of packets per watched node, at most once
per node per 10 minutes, with the read lock taken **per node**:

```
BenchmarkCertainActivity-16   12.1 ms/op   (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold)
BenchmarkNoteCertainRelays-16  121 ns/op   0 allocs
```

Node analytics adds one `certainFor` per packet of the requested node (a
prefix-map lookup, or the cached decoded JSON for its own packets).

## Tests

- `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet
nights set the rate; > 10 % empty hours, short history, too few
observers → fixed.
- `CertainActivity` ends at the last certain packet and counts only
certain packets — verified red by mutating the window to end at now.
- Notifier end to end: learned window passed → bad; inside → good; too
few observers / setting off → fixed 72 h; cache refresh and prune.
- `certainFor`, `longestSilence` (incl. running silence), config
defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket.
- Frontend: `NodeAnalyticsTimeline.series` (new
`tests/unit/test-node-analytics-certain-timeline.js`).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork.

## Validation

Server on the freshened e2e fixture: analytics buckets carry `certain`
(e.g. 89 of 90 packets certain for one repeater). **No browser
screenshot**: Playwright isn't installed on this machine.

## Later (rule 8)

`floorMinutes` / `minObservers` are server config, not theme settings;
an admin UI for notification settings would be the place, not the
customizer.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-22 21:20:42 +02:00
efitenandClaude Opus 5.5 a08f2f4e7c fix(liveness): count only evidence that can be the node alone (#2260) (#2261) 
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold
follows in a separate PR).

## Problem

A repeater silent since 2026-10-18 showed **Active — Last relayed 4m
ago** and **Last Heard 36m**, and its offline mail never fired. Every
hit after that date was the 1-byte hop `67`, shared by nine nodes:

- the live resolver's affinity tiebreak (`resolvePathForObs`) put those
transmissions in `byNode`, which fed `stats.lastHeard` and
`LastHeardMap`;
- `last_relayed` folds in the 1-byte prefix bucket on purpose (#662),
and drove the status in `roles.js` and the notify offline check.

## Fix

The store keeps `certainHeard` per node: the newest transmission that
can only be that node.

| Evidence | Counts? |
|---|---|
| Own packet (`pubKey`, `srcPubKey`) | ✅ |
| Flood relay hop whose prefix, at the packet's path hash size, matches
exactly one path-capable node | ✅ |
| 1-byte hop shared with other repeaters | ❌ |
| Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌
|
| TRACE (path bytes are SNR) | ❌ |
| Destination (`destPubKey`) | ❌ |

A multibyte repeater legitimately appears as a 1-byte hop (the
originator picks the hash size, `Mesh::routeRecvPacket` appends at that
size), so only uniqueness decides. The rule is applied in every load
path through `indexResolvedPathHops` and the new
`indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live
ingest, late observations.

Liveness reads only that:

- `/api/nodes` and the node detail: `last_heard` is the newer of
`last_seen` and the certain last heard. Before, it was a plain copy of
`last_seen`.
- health / bulk-health `stats.lastHeard`.
- `LastHeardMap` → notify offline check. The separate relay input
(`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed.
- `roles.js getEffectiveHeardMs`: newest known timestamp, without
`last_relayed`. The side pane no longer says "Last relayed"; the Last
Relayed row stays (analytics) with an updated tooltip.
- New status **`uncertain`** (repeaters and rooms): nothing certain
within the silent window, but the shared prefix (`last_relayed`) still
in paths. Grey badge with explanation, an *Uncertain* filter on the
nodes list and the map, dimmed like stale. The offline mail for such a
node says only its shared prefix was still seen.
- A prefix also held by a path-capable node in `inactive_nodes` is not
unique; adverts with `signatureValid: false` are no evidence;
`/api/nodes/search` reports the same `last_heard`.

Attribution for analytics (`byNode`, packet counts, `last_relayed`,
relay counters, usefulness) is unchanged.

## Performance

Hot path: `noteCertainRelays` runs per observation on ingest under the
write lock: two hex bytes of header parsing plus one prefix-map lookup
per resolved hop.

```
BenchmarkNoteCertainRelays-16   10307640   116.2 ns/op   0 B/op   0 allocs/op   (8-hop 2-byte path, 2,000 repeaters)
BenchmarkLastHeardMap-16            1902   0.77 ms/op                            (5,000 nodes; was 3.7 ms scanning byNode)
```

Memory: one string per node ever certainly heard, dropped together with
the node's `byNode` list on eviction.

## Tests

- `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk,
LoadChunked — each with and without resolved_path — live ingest, late
observations) feeds `certainHeard`; each of the 8 call sites verified
red with a nil prefix map.
- `certain_heard_test.go`: hop width per route/payload type; retired
namesake prefix; failed signature; uncertain offline mail; shared vs
unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load
path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`,
detail and health agree.
- Regression for #2260: last certain packet on day X, 20 shared-prefix
hops after it → `LastHeardMap` stays at X and `node.offline` goes bad
after 72 h. Verified red with the uniqueness check mutated out.
- Notify: `TestEvaluateCertainHeard` replaces the relay-input test.
- Frontend: `last_relayed` alone no longer keeps a repeater or room
active; newest timestamp wins (an old `_liveSeen` can't hide a fresh
certain relay).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork with 457 extra commits.

## Validation

Ran the server on the freshened e2e fixture: list, detail and health
report the same last heard; e.g. a repeater whose `last_relayed` is 3
min newer than anything certain keeps its certain time. **No browser
screenshot**: Playwright isn't installed on this machine; CI's E2E
covers the pages.

## Not changed

- The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a
shared hop only when exactly one candidate is adjacent in the neighbor
graph; it did not fire in this case. Left as is.
- The `lastHeard=` list filter still filters `last_seen` in SQL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-22 19:11:30 +02:00
AdamandClaude Opus 5.5 dbcdca9dc5 feat(region-filter): honor ?region= in the URL on every page (#2255) (#2256) 
Fixes #2255.

Only the packets page read `?region=` from the hash, with its own
one-off code. Live, nodes, observers, channels, analytics and scope
audit ignored it, so a link like this opened with whatever region the
visitor last picked:

```
https://<instance>/#/live?region=SJC
```

We link into our instance from a community directory (one link per
region), and wanted those links to land on the live map or node list
already filtered, not only on packets.

## What changes

**`public/region-filter.js`**
- `?region=` (comma-separated, trimmed) overrides the saved selection
and is saved like a click in the control, the same as packets did
through `setSelected()`.
- **Fresh load:** it's applied when the script runs, before any page
reads the selection. This matters for live, which fetches nodes before
it calls `RegionFilter.init()`.
- **In-app navigation:** `init()` applies it before its `await`, so a
page that loads after `init()` uses it. If it changed the selection,
listeners are told once regions load, so a page that already fetched
(live) reloads instead of showing the old region's data under the new
region's label.
- Unknown codes are dropped by the existing cleanup in `fetchRegions()`.
No param, or an empty one, leaves the saved selection alone.

**`public/packets.js`**
- Drops its own `_pendingUrlRegion` handling; `RegionFilter` covers it.
- Removes its region and area listeners in `destroy()`. They were
registered without a handle and outlived the page: after visiting
packets, a region change on any other page reloaded packets and
`replaceState`'d the URL to `#/packets?...` while you were on another
page. That already happened with the dropdown on master; the in-app
notify above would make it happen on every region link.

## Tests

`tests/unit/test-region-filter-url-param.js`, 14 assertions:
- URL region overrides the saved selection; comma lists, whitespace,
unknown codes, empty param, no param, no hash
- fresh load: selection is set before `init()` is called, and `init()`
doesn't notify
- in-app navigation: a changed region notifies listeners once, the same
region doesn't
- selection is saved to `localStorage`
- packets keeps handles for both listeners and removes them in
`destroy()`

With `region-filter.js` from master, 5 of the first 8 assertions fail.

How I checked:
- `test-all.sh` suites all pass except
`test-issue-1956-release-routing.js` and `test-preflight-xss-gate.js`.
Both also fail on unmodified master on macOS: they shell out to bash-4
features (`mapfile`, and a syntax error in `check-xss-sinks.sh` under
bash 3.2).
- `eslint@8.57.1` on the two files: 0 errors, 3 warnings, the same as
master.
- Browser: served `public/` locally with `/api` proxied to a live
instance and checked the requests each page made:
- fresh load of `#/nodes?region=AUS`, `#/live?region=DFW`,
`#/packets?region=HOU`: the first request carries the URL's region and
the dropdown shows it
- packets → `#/live?region=DFW`: live fetches with the old region, then
reloads with DFW; the dropdown shows DFW
- packets → `#/nodes?region=AUS`: no packets requests, and the URL stays
on `#/nodes`

I couldn't run the Playwright E2E suite locally (no Go toolchain on this
machine).

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-22 19:11:16 +02:00
efitenandClaude Opus 5.5 0e3ce48cff feat(nodes): flag nodes whose LoRa settings differ from nearby nodes (#1268) (#2210) 
Refs #1268. #2209 is merged as `cab4f71c`, and master is merged into
this branch (`47ce8264`), so the diff against master is this PR's own
change: `ea4d1130` plus the fix in `5cbbe5d5`.

## What it does

Two nodes hear each other only when SF and bandwidth match and their
frequencies are close enough. `/api/rf-params` now also returns
`deviations`: nodes whose setting differs from a majority of the nodes
around them on the same band.

- **Peers:** nodes with GPS and RF params within
`rfParams.compareRadiusKm` (default 50) whose frequency is within 30
MHz.
- **Same channel:** frequencies within a quarter of the bandwidth count
as one channel (15.625 kHz at BW 62.5). The SX1262 tolerates a
transmitter/receiver offset of up to 25% of BW without sensitivity loss,
for SF5 to SF12 and all bandwidths. Source: the `FERR_L` row in the
[S62F module
datasheet](https://lora-alliance.org/wp-content/uploads/2020/07/S62F_SPEC_Ver.B.pdf),
an SX1262 module; I did not find the figure in a Semtech document
itself. That table also lists -50/-100/-200 ppm for SF12/SF11/SF10,
which at 869 MHz is wider than BW/4 at 62.5 and 125 kHz, so BW/4 is the
tighter bound there.
- **Flagged when:** a setting the node cannot hear is shared by more
than half of the group, the node itself counted, with at least 3 peers
and no other incompatible setting at the same support. So 5 against 5
flags nobody, and 2 of 4 does not count as a majority. Each deviation
carries the majority, how many peers are compatible with it, the peer
count, and which of freq/sf/bw differ.
- **Coding rate is not compared.** RadioLib starts SX126x radios with an
explicit LoRa header (`SX126x::begin` sets
`RADIOLIB_SX126X_LORA_HEADER_EXPLICIT` at `6d89348`, the commit the
firmware pins in `platformio.ini`), and nothing in `firmware/src`
switches to implicit. The header carries the CR, so a CR mismatch still
decodes. I checked SX126x only, not SX127x or LR11x0.

UI:
- Nodes list: a badge such as `LoRa SF7` next to the name of a deviating
node (the majority in its tooltip), and a `LoRa: differs from nearby
(N)` filter kept in the URL as `?lora=differs`.
- Node detail: one line, for example "Differs from 12 of 14 nearby nodes
on this band, which use 869.618 MHz · SF8 · BW 62.5 kHz".
- Badge and filter only exist while the server has map data.

## Why same-band only

On our instance (1,110 nodes with params and GPS, measured with the
first version of the rule), a plain neighbourhood majority flagged
exactly 15 nodes, all of them a deliberate 433 MHz network in Limburg
next to the 868 MHz one (names like `NL-ZAY-RP01 | 433 X 868`). The
same-band comparison flags none there. On our network this PR shows
nothing today. The flags are for instances where a node really is set to
a different SF, bandwidth or channel than the nodes around it.

## Cost

- Points are bucketed on a radius-sized grid (rows of radius/110.57
degrees, longitude columns widened by 1/cos(lat)) and compared with a
squared equirectangular distance.
- `BenchmarkRfDeviations1k` (1,100 nodes over Belgium and the southern
Netherlands): 4.5 ms. `BenchmarkRfDeviations10k` (10,000 nodes in the
same box): 388 ms, 1.3 MB. Both on the first version.
- After `5cbbe5d5`, on a slower machine, 3 runs each: 1k 4.4 to 9.5 ms,
10k 821 to 852 ms, 1.8 MB. On that machine the first version measured
7.6 to 9.3 ms and 819 to 861 ms, so the majority step adds no measurable
time. It is one pass over the distinct settings around a node, and those
are few (one per preset in use).
- Inside the radius the work is close to O(n²): 10,000 nodes in one
dense box means about 2,000 neighbours per node, and a request after the
5 minute TTL waits for that computation.
- The result is cached per map snapshot for 5 minutes; concurrent
requests wait for one computation.
- On staging `/api/perf` reports `/api/rf-params` at 9.9 ms average, 26
ms max (8 requests).

## Tests

- Go: deviation cases (SF, freq within band, BW, CR ignored, other band
ignored, no GPS, no params, far away), split vote and too few peers,
radius, a peer in another longitude column at 60°N, the cache (same
snapshot within TTL, new snapshot, after TTL), the route with four
inserted nodes, plus the two benchmarks.
- Go, added in `5cbbe5d5`: a 5/5 split and 2 of 4 flag nobody; a 1 and a
15 kHz offset are not deviations while 32 kHz is, and offset peers add
up into one majority; the BW/4 boundary; peers two grid columns away at
60°N; a peer at the north edge of the radius. Each fails on the first
version. The two-column test also fails with a ±1 column search, which
the existing longitude test did not catch.
- Full `cmd/server` suite and `test-all.sh` pass locally on `5cbbe5d5`.
- Frontend unit: 6 new assertions for `deviationBadge` and the detail
line (13 in the file).
- E2E: 4 new checks: badge only on the deviating node, filter count and
result, `?lora=differs` written and restored, detail line. The disabled
check now also asserts no filter and no badges. 8/8 locally.

## Validation

On staging with live map data, with the first version: 1,115 nodes
served, `deviations: {}`, and the filter shows `LoRa: differs from
nearby (0)` with no badges. Because no real deviation exists there, the
badge, filter and detail line were checked by serving one simulated
deviation through Playwright route interception against the staging
page.

## Not done

- No map marker for deviating nodes.
- `compareRadiusKm` is config-only; the customizer is a later step. The
other values are hardcoded and belong in that step too: at least 3
peers, the 30 MHz band, the more-than-half rule, BW/4 and the 5 minute
TTL.
- Not re-run on staging after `5cbbe5d5`, and the E2E suite was not
re-run locally. The fix changes only the server's choice of nodes, not
the UI.
- The detail line says the peers "use" the majority setting. With the
BW/4 tolerance, some of them may be a few kHz off it.
- Companions and room servers are compared too, not only repeaters.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 15:18:14 +02:00
efitenandClaude Opus 5.5 dc98886d92 feat(nodes): name and show repeaters known only from the name they reported (#2238) (#2239) 
Part of #2238. Path hops through such a repeater (`/api/nodes`,
`/api/resolve-hops`, the frontend hop resolver) still show a bare
prefix; that part of the issue stays open.

## Summary

A repeater that never sends an advert can still tell a nearby companion
its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and
publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`.
This PR stores those answers, resolves names with them, and shows these
repeaters on the map, in the nodes list and in search, marked as
self-reported.

### Ingestor: store the names
- `367ecde1` adds a new `names` subtopic.
  - It is gated on `clientNodeNames.enabled`, default off.
- Answers go into a new table, `client_node_names`: one row per
observation, `UNIQUE(target, rx_pubkey, observed_at)`.
- **Validation.**
  - `target` must be 64 hex.
- The name is trimmed and rejected when it is empty, over 31 runes
(firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8,
or contains a control character.
- **Retention.** `retention.clientNodeNamesDays` (0 = keep).
`config.example.json` sets 30, like the other client tables: a companion
only asks a repeater for its name when CoreScope has none (`nameMissing`
in CoreDrive's `src/names.js`), so pruning is also how a renamed
repeater gets its new name.
- **Compatibility.** An ingestor without this change drops `/names`
silently, so CoreDrive v1.20.0 works against either version.

### Server: resolve by the reported name
- `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and
`client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys`
(the coverage tooltips) both use it.
- **Which name wins.** The advert name, otherwise the newest reported
name by `observed_at`. A prefix that matches different pubkeys across
the two tables stays ambiguous.
- **Only repeaters heard directly.** A reported name counts only once
its target has at least one `discover` or `advert` reception in
`client_receptions`, keyed by the full pubkey, or by its 8-byte prefix
when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names`
starts with that prefix (`clientNameHeard`). A name for a pubkey nobody
heard adds no candidate, so it cannot make a real node's prefix
ambiguous. Without the uniqueness check, a made-up pubkey that shares a
heard repeater's first 8 bytes borrowed that repeater's receptions and
passed (fixed in `4c569f91`).
- **Filters.** The blacklist and the hidden name prefixes apply to both
sources.
- **Missing table.** A database without the table resolves from `nodes`
alone, and the server checks for the table again until the ingestor has
created it.
- **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?,
name?, ambiguous}`.

### Server: estimated position
- `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`.
It lists every repeater in `client_node_names` that never sent an advert
(no row in `nodes` or `inactive_nodes`) and was heard directly at least
once (see above). A node with an advert but no position stays out: it is
already in `/api/nodes` under its advert name.
- **Gated** like the other endpoints on `client_receptions`: 404 when
`clientRxCoverage.enabled` is off, because the counts and the estimate
come from companion GPS fixes.
- **Which receptions count.** Only direct ones:
- `discover` and `advert` rows, keyed by the full pubkey, or by its
8-byte prefix when no other known pubkey shares it;
  - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone.

  `geo` rows are left out.
- **How the estimate is made.**
- Receptions within 100 m form one spot, with the best SNR of that spot.
  - An estimate needs at least 3 spots, at least 200 m apart.
- The position is the SNR-weighted centroid (`10^(snr/20)`). The radius
is the distance to the farthest spot, at least 500 m.
- The response is cached for 5 minutes, and the compute time is logged.

### Map
- `5fcde554` adds `public/map-estimated.js`, which draws each estimate
on its own layer:
  - a hollow, dashed "≈" marker and a dashed uncertainty circle;
- a popup with an "Estimated position" badge, the method, the caveat
that one-sided driving skews the estimate, and a link to the coverage
map;
- a Display toggle, "Estimated positions": on by default, kept in
localStorage, and also hidden by the repeater role filter and in route
view.

### Nodes page and search
- `12e6b345` adds a "Self-reported repeaters" section under the nodes
table, on the All and Repeaters tabs, filtered by the search box.
- **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a
self-reported detail view replaces "Node not found":
  - who reported the name;
  - the receptions;
  - the estimate, or why there is none;
  - links to the map and the coverage map.

  An unknown key still shows "Node not found".
- **Header search** adds matching self-reported repeaters, tagged
"self-reported".
- **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as
they are. These repeaters have no packets, so the packet filters, the
live view and the channel sender lookup must not match them.

### Docs
- `03580437` corrects the `/regions` section of
`docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the
switch, but the code uses `clientRegions.enabled`.
- The new subtopic, endpoint and config keys are documented in the same
file.

## Trust

Any companion the broker lets publish can report a name for any
repeater, and the newest report wins. This is the same model as
`/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to
linked companions.

A report for a repeater nobody heard directly is stored but not used: it
is not listed and adds no prefix candidate. A companion can still report
a wrong name for a repeater that was heard. The same companion can also
publish a fake `discover` reception for a made-up pubkey, so the "heard
directly" check filters junk, not a malicious companion;
`requireLinkedCompanion` is the control for that (its gate runs before
the `/names` handler too).

## Performance

- **`lookupPrefixes`** adds one term per key on `client_node_names`.
- It matches the prefix as a range, `target >= ? AND target < ?` (upper
bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive
column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX
idx_cnn_target (target>? AND target<?)`, and the reception check is
`SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`.
- It is still one query per chunk of 200 keys: 400 compound terms, under
SQLite's default limit of 500.
  - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks.
- **`/api/nodes/reported`** makes three kinds of query:
  - one grouped query on `client_node_names`;
  - one `lookupPrefixes` call for the short hashes;
- one `IN` query per 400 keys on `client_receptions`, served by
`idx_client_recept_heard_geo`.

The spread check is O(spots²) per repeater, with at most one spot per
100 m visited. The result is cached for 5 minutes; concurrent misses
wait for one computation, and the response is written after the lock is
released.
- **The map does not wait for it.** It draws its markers first and adds
the estimates when the list arrives. The map and the nodes page share
one cached fetch (`ReportedNodes.load()`).
- **Not benchmarked.** On the instance where this runs,
`client_node_names` holds one row today. I expect at most tens of such
repeaters per instance, but have not measured that.

## Notes

- **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2
occurrences: the handler's `msg` parameter and its `gps` field. That is
the decoded MQTT message, passed in the same shape as
`handleClientPacket` and `handleClientRegions` already take it. No API
response or internal structure uses it.
- **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min
TTL. They should move to the customizer later (rule 8).
- **Map filters.** The estimates follow the "Estimated positions" toggle
and the repeater role filter, not the area, region or last-heard
filters. The toggle's tooltip says so.
- **Name characters.** Names are rejected for control characters but not
for Unicode format characters (bidi overrides, zero-width joiners),
because emoji sequences in node names use the zero-width joiner. All
output goes through `escapeHtml`.
- **Not done:** the home-page search box (`home.js`, `/api/nodes/search`
with a Claim button) does not list these repeaters. Claiming a node
without adverts would have nothing to show.

## Tests

On this branch (based on `550829af`):
- `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and
`cmd/ingestor`.
- **New Go tests:**
  - handler and validation cases;
  - idempotent redelivery;
  - newest `observed_at` wins;
  - prune;
  - the gate on and off, without falling through to the observer path;
- client-only node, nameless `nodes` row, advert name wins, ambiguity
across the two sources;
  - hidden and blacklisted nodes;
  - absent-then-created table;
  - spot merging, threshold, weighted centroid and radius floor;
- `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an
advert position and a missing table;
- an advert without a position, a never-heard name, a name heard only in
`rxlog` paths, and the `clientRxCoverage` gate (404). These fail on
`12e6b345` and pass at head.
- a made-up pubkey sharing a heard repeater's 8-byte prefix, and two
reported targets sharing one (`4c569f91`). Both fail without that
commit.
- **New frontend unit suites:** `tests/unit/test-map-estimated.js` and
`tests/unit/test-reported-nodes.js` pass. Both load the real
`escapeHtml` from `app.js`.
- **`test-all.sh`** passes, all suites (with Python available for the
XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the
four changed frontend files.
- **E2E**, against a static server with a mocked API:
- `test-map-estimated-e2e.js`: 6/6, including the repeater role filter
and route view hiding the estimates. Mutating either hide makes its step
fail.
  - `test-reported-nodes-e2e.js`: 6/6.
  - `test-issue-1150-404-state-e2e.js`: 5/5.
- **Real data, API only.** On the fork's staging, `/api/nodes/reported`
lists one repeater: 5 spots, 17 receptions, an estimate with a radius of
2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays
listed under the stricter rules. Not re-checked after `4c569f91`.
- **Not done:** a check of that repeater in a browser.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 15:17:58 +02:00
nullrouten0nullrouten0Claude Opus 5.5
 		9534e1c25e fix(reach): retry a busy (429) reach report in the reach page (#2250) 
Follow-up to #2127: @dborup's point 2 there, and @efiten's non-blocking
note on the approval ("worth checking the node page's reach panel shows
a retry hint on 429 rather than a generic error").

## What changes

- **The reach page retries a 429 itself.** It shows "Reach is busy
computing other reports, retrying in N s", waits `Retry-After` seconds,
and tries again.
- When the header is missing or is not a number of seconds (an
HTTP-date, 0, a negative number), it waits 2 s. It never waits more than
30 s.
- After 5 retries it shows the old error ("Failed to load reach: API 429
…").
- Leaving the page stops the retries, both while it waits and while a
request is still in flight. Picking another window while a request is in
flight also drops that request's retries.
- **`api()` does not retry 429.** Login, signup, activation, email
change, companion linking, settings PUT and channel proposals also
answer 429 from their own limits, and those must not auto-retry. `api()`
now puts `status` and `retryAfter` on the errors it throws, so a caller
can decide; the message text is unchanged. A unit test pins that `api()`
makes exactly one request on a 429.
- **Fix in `api()`:** the `_inflight` cleanup chain
(`promise.finally(...)`) rejected unhandled on every failed call, so
each API error also logged "Uncaught (in promise)". That would have
added up to 5 more per busy reach load. The chain now catches; callers
still get the rejection through the returned promise.

This works with the server as it is on master (instant 429,
`Retry-After: 5`), and with the bounded wait in #2249 (429 after 3 s,
`Retry-After: 2`). Neither PR depends on the other.

## What I checked

- **Suites.** `sh test-all.sh` passes, the XSS sink gate passes, and
eslint has 0 errors and no new warnings.
- **Unit tests** (`test-frontend-helpers.js`):
- `reachRetryDelayMs`: Retry-After values; fallback on missing, zero,
negative or HTTP-date values; the 30 s cap; giving up after 5 retries;
no retry on 400, 404, 500, 503 or network errors.
- `api()`: a 429 error carries `status` and `retryAfter`, makes exactly
one request, and leaves no unhandled rejection.
- The block sits above the file's `Promise.allSettled(pendingTests)`
summary. Async tests registered below that line are never awaited, so
they cannot fail the run; nothing below it is async today.
- **E2E** (`test-node-reach-e2e.js`, already in CI suite 2): reach
requests are answered through `page.route`, so the server does not have
to be made busy. It checks:
- two 429s, then the real report: the hint shows "retrying in 1 s", the
retries wait (at least 1.8 s between the first and third request), and
no error is shown;
  - always 429: 6 requests (1 + 5 retries), then the error;
  - leaving while it waits: no more requests;
- leaving while the first request is still in flight: no hint written,
no retry;
  - a 500 with Retry-After: one request, no retry.

On master's frontend the new section fails at the first busy hint.
Locally against the freshened `e2e-fixture.db`, the whole file takes
about 17 s (2 runs).
- **Mutations.** I wrote 15 from the code, and the tests caught all 15.
They include: `api()` drops `status` or `retryAfter`, `api()` retries
429 itself, retry any error, retry limit off by one, fallback or cap
changed, Retry-After 0 accepted, either "left the page" check removed,
the wait not honoured, the hint showing ms, the retry count ignored, no
error after giving up, and the cleanup chain unhandled again. Each
mutation was checked against the file the server actually served before
the tests ran.
- **Real browser** (Chromium via Playwright, on a copy of a
10.5M-observation database):
- Current master's server, 6 tabs opened at once on uncached 14-day
reports: 4 got 429 with `Retry-After: 5` and showed the hint, and all 6
reports rendered within 11.5 s (one tab retried twice). Without this
change those 4 tabs show "Failed to load reach: API 429".
- With #2249's server: 4 of 6 got 429 with `Retry-After: 2`, and all
rendered within 8.1 s.
- The only console errors were Chrome's own "Failed to load resource"
lines for the 429s; there was no "Uncaught (in promise)".

## Not checked

- **Mobile layout.** The hint reuses the existing `.nq-load` style, so I
did not take a screenshot at phone width.
- **Other browsers.** Firefox and Safari were not tried; the code is
plain `setTimeout` / `await`.
- **Window switches.** While the hint is showing, the time-range buttons
are not on screen; the same is true of the existing error and loading
states. Picking a new window means waiting, or going back and opening
reach again.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: nullrouten0 <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-21 11:58:24 +02:00
efitenandClaude Opus 5.5 d43f3c3777 feat(theme): ?theme=dark|light URL parameter for one page load (#2236) (#2251) 
Fixes #2236.

A display that can only be given a URL (digital signage, an iframe
without cookies or login) had no way to choose dark or light. This adds
`theme=dark` or `theme=light` to the hash query, next to `embed=1`:

```
https://<instance>/#/map?embed=1&theme=dark
```

## What changes

**`public/index.html`, the pre-paint script**
- Reads `theme` from the hash query with `URLSearchParams`, as
`shouldEmbedRoute` does for `embed`. Only `dark` and `light` count;
anything else is ignored.
- A valid value wins over the saved choice and the OS preference, and is
set before first paint, so there is no light flash.
- The value is handed to `app.js` as `window.__themeOverride`.
- When `localStorage` throws (a sandboxed iframe), the URL value is
still applied.

**`public/app.js`, the dark-mode setup**
- On load, a URL value is applied through `applyTheme(theme, false)`,
which does not write `meshcore-theme`. The saved choice, and the account
settings that `settings-sync.js` syncs from that key, stay as they were.
- Theme changes from other tabs or from settings sync (the `storage`
listener) are ignored while the URL value holds. Otherwise a logged-in
display would flip within a minute.
- A click on the dark-mode toggle ends the URL value and is saved as
before.

It works on every page, not only on the two `embed=1` pages. The value
is read once per page load, so later hash changes (navigation, map
viewport) keep it.

## Tests

| File | Assertions | Covers |
|---|---|---|
| `tests/unit/test-theme-url-param-2236.js` | 8 | Runs the real
pre-paint script from `index.html`: `dark` and `light` win over a saved
choice and the OS, nothing is written to `localStorage`, any page and
position in the query, an unknown value, no param, `localStorage`
throwing |
| `tests/e2e/test-theme-url-param-2236-e2e.js` | 6 steps |
`#/map?embed=1&theme=dark` over a saved "light" (dark, embed, saved
still "light"), a reload without the param, `theme=light` over a saved
"dark", a storage event while the override holds, the toggle ending it,
`theme=blue` |

How I checked them:
- With `index.html` and `app.js` from master, 6 of the 8 unit assertions
fail, and the E2E fails at step 1 ("expected dark + embed + saved
light").
- With only the `app.js` change reverted, the E2E fails at step 1. With
only the `if (themeOverride) return;` line in the `storage` listener
removed, it fails at step 4.
- `sh test-all.sh` passes, except `test-issue-1956-release-routing.js`,
which I skipped: it needs the workflow's `GITHUB_REF_NAME` and does not
run in local git-bash. `npx eslint public/app.js` reports 0 errors (18
warnings, the same count as master).
- E2E on a fixture seeded as in CI:

  | Suite | Result |
  |---|---|
  | `test-e2e-playwright.js` | 148 passed, 3 skipped |
  | `test-customize-theme-e2e.js` | 8/8 |
  | the new #2236 suite | 6/6 |

- `test-logo-theme-e2e.js` fails locally with "wordmark `<text>`
overflows SVG viewBox", with the same numbers on master. It measures
font metrics, which differ on this Windows machine.

The new E2E is registered in `scripts/non-unit-tests.json` and
`deploy.yml` (shard 2, next to the logo theme suites); the unit test is
in `test-all.sh`.

## Not done

- No screenshot: the E2E checks `data-theme` in headless Chromium, not
how the page looks.
- `live.js` Matrix mode forces dark and, when switched off, saves the
theme it restores. With a URL value active, turning Matrix off saves
that value as the stored choice. I left this as is, because a display
that only shows a URL does not toggle Matrix mode.
- Only `theme=`; the `?darkMode=1` form from the issue text is not
supported, so there is one form to document.
- I placed the E2E in shard 2 without measuring suite times.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 11:40:25 +02:00
efitenandClaude Opus 5.5 3960fadd96 fix(pages): drop late responses that belong to an earlier page visit (#2218, #2230, #2200) (#2248) 
Fixes #2218, fixes #2230, fixes #2200.

Node analytics, the account page and analytics await their data, then
write into the shared `#app` container. A response that lands after the
user has navigated away replaces the new page. In #2218's reproduction,
the Nodes list, with 200 rows, became an Analytics heading.

## The shared part

`public/app.js` keeps one view generation, `viewGeneration()`:
- `navigate()` bumps it before the old page's `destroy()`, so every page
init counts as a new visit, including a switch to another node of the
same page type.
- A page captures the value at init, and after each `await` drops the
result when the value has moved on.

`.eslintrc.json` declares it as a global.

## Per page

**#2218, `node-analytics.js`**
- `loadAnalytics` and `loadBatteryChart` check the generation on both
success and error. A page-local `loadSeq` makes a time-range click a new
load on the same visit, so an older range can't land over a newer one.
- `NodeHopAnalytics.load` already checked `container.isConnected`, so it
is unchanged.

**#2230, `account.js`**
- The render after `CSAuth.ready()` runs only while the generation is
unchanged.
- `destroy()` was empty. It now clears `shown.app` and `shown.userId`.

**#2200, `analytics.js`**
- A page-local `_loadSeq` sits next to the generation. Results and
errors of an older window/region selection or an earlier visit are
dropped, so a late failure of an old load no longer shows an error over
newer results.
- The bulk load renders only the six tabs drawn from its data. A tab
that loads its own data (Prefix Tool, Nodes, Roles, ...) is no longer
rebuilt or replaced by the bulk result: a deep-linked one renders at
init, and a window, region or area change re-renders it directly.
- `renderTab` guards the six tabs drawn from the bulk data (overview,
rf, topology, channels, hashsizes, collisions). Until the data arrives
they show "Loading analytics…", or the current load's error. Overview no
longer dereferences an undefined `rf`.
- Three related fixes in the same lines:
  - The error message is now escaped with `esc()`; it was inserted raw.
- `destroy()` removes the RegionFilter and AreaFilter listeners. They
piled up on every visit and called `loadAnalytics` from other pages.
  - `destroy()` resets the stored error.

## Tests

| File | Assertions | Covers |
|---|---|---|
| `tests/unit/test-issue-2218-node-analytics-stale-render.js` | 12 |
Loads the real router: the generation rises on every init, including
another node. Late success and error after leaving, a stale visit of the
same node, an older range landing last, late battery success and error.
The current response, battery and error still render |
| `tests/unit/test-issue-2230-account-stale-render.js` | 6 | Leaving
before ready (login view and "not enabled"), re-entering before ready,
the current render, an auth change after `destroy()` |
| `tests/unit/test-issue-2200-analytics-stale-load.js` | 19 | An older
selection landing last, a late rejection of an older selection, leave
and re-enter (success and error), the current error rendered escaped and
kept on a tab click, Overview and RF clicked before the data |
| `tests/e2e/test-issue-2218-node-analytics-stale-e2e.js` | 4 steps |
Holds `/api/nodes/*/analytics` with `page.route`, returns to `#/nodes`,
then releases the response as a 200 and as a 500. The 200 rows stay and
no Analytics heading appears; a normal visit still renders |

How I checked them:
- With the production files reverted and the tests kept, 7 of 12, 4 of 6
and 7 of 12 unit assertions fail. With the fix, all pass.
- The E2E fails on the unfixed frontend with "Nodes rows 200 before
release, 0 after".
- Single mutations, each reverted:
- removing the success check in `loadAnalytics` (analytics) fails 2
assertions;
  - removing the check after `CSAuth.ready()` fails 3.

Other test changes:
- `test-analytics-theme-refresh-before-data.js` and
`test-user-management-ui.js` get a `viewGeneration: () => 0` stub.
- The new E2E is registered in `scripts/non-unit-tests.json` and
`deploy.yml` (shard 2, next to the #748 suite). The unit tests are in
`test-all.sh`.
- The broad suite's "Node side panel Details link navigates" test
selected an anchor that now opens Analytics, as #2218 notes. It now
clicks `#nodesRight .node-detail-btn`.

Results:
- `sh test-all.sh` passes, and `npx eslint public/*.js` reports 0
errors.
- E2E on a fixture seeded as in CI:

  | Suite | Result |
  |---|---|
  | `test-e2e-playwright.js` | 148 passed, 3 skipped |
  | `test-user-management-e2e.js` | 28/28 |
  | the new #2218 suite | 4/4 |
  | `test-issue-748-node-analytics-coverage-e2e.js` | 2/2 |
  | `test-analytics-fluid-charts.js` | 8/8 |
  | `test-issue-1657-analytics-channels-group-sprites-e2e.js` | 5/5 |

## Not done

- Two tabs that each fetch their own data can still overwrite each other
after a fast tab switch between them. #2200 does not ask for that.
- `nodes.js` `navigateToNode` calls its own `init` without the router,
so it does not bump the generation. None of the three pages fixed here
depends on that.
- I placed the E2E in shard 2 without measuring suite times, which the
`deploy.yml` comment asks for.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 10:39:15 +02:00
CoderNemesisandClaude Opus 5.5 e55c95db24 1-Byte Hop Path Inference for Packet and Map Analysis (#2245) 
## Summary

On meshes with 1-byte path hashes, most hop prefixes are shared by
several repeaters. The packet detail pane currently resolves each hop on
its own and stores the result by prefix. Two things go wrong as a
result:

- A prefix that appears twice in one path resolves to the same node both
times.
- Nothing checks that the chosen hops form a path a packet could
actually take.

This PR adds `HopResolver.inferPath()`, which infers the most likely
node behind every hop of **one observation's path**, scoring the path as
a whole and anchoring it on the observer that heard it. The packet
detail pane and the map route view now use it, so both show the same
nodes for the selected observation (`?obs=`).

## The problem, on a real packet

Packet `57da37d1636f895a`, observation `1388126`: a 46-hop flood with
1-byte hashes. The server's `resolved_path` resolved 1 hop of 46, so the
client resolver did the rest. Eight prefixes appear twice in the path.

| | Current resolver | `inferPath()` |
|---|---|---|
| Hops longer than 120 km | 13 (21 longer than 60 km, two over 950 km) |
0 |
| Distance from the last hop to the observer | 59 km | 0.1 km |
| Hops reported as unlisted (no listed node fits) | 0, every hop guessed
| 13 |
| Agreement on the shared path prefix across the packet's 23
observations | — | 97.8% |

The current resolver picks badly for three reasons:

1. **Results are keyed by prefix.** Hop 0 and hop 45 share a prefix, so
both show the node near the start of the path, 59 km from the observer.
2. **Picks are greedy.** Each hop looks only at its nearest resolved
neighbours, so nothing penalises a 1,100 km jump.
3. **The detail pane read the wrong `resolved_path`.** It used the
packet row's, not the selected observation's, and when the lengths
matched it never fell back for the hops that were `null`.

## Firmware basis

The model relies on two facts from `firmware/src/Mesh.cpp`:

- **The last hop is the one the observer heard.**
`Mesh::routeRecvPacket` appends the relaying node's own hash to the end
of a flood path.
- **A node appears at most once in a flood path.** A node forwards a
given packet once: `_tables->wasSeen()` / `markSeen()`.

## How the model works

Each hop is either one of the **located** repeaters or room servers
sharing its prefix, or **unlisted**: a node missing from the node list,
or one without GPS. With 1-byte prefixes, unlisted is common, and naming
it beats drawing a 300 km hop.

| Term | Cost |
|---|---|
| A hop between two located nodes | distance / 50 km; past 100 km per
hop, the cost rises 4× faster |
| A hop with a neighbour-graph link | the distance cost × 0.5 |
| An unlisted hop | 1.6, flat (about one 80 km hop) |
| The last hop to the observer | the same distance cost; falls back to
the observer's IATA region centre (with 50 km slack) when the observer
reports no GPS or 0,0 |
| The first hop to the sender | the same distance cost, when the
sender's position is known (e.g. an ADVERT) |

- **Search:** dynamic programming over (hop, candidate) finds the
cheapest sequence. When the best path uses a node twice, the node is
banned from whichever position costs less to give up.
- **Confidence:** each hop's confidence comes from the min-marginal
costs of its alternatives, put through a softmax. A node the path
already uses at another hop is not counted as an alternative.
- **Server hint:** the observation's own `resolved_path` restricts a hop
to the named node, but that hop can still be reported unlisted if the
node doesn't fit.

The tuning values live in `PATH_MODEL` in `hop-resolver.js`, with a TODO
to expose them in the customizer (AGENTS rule 8).

## What changes in the UI

**Packet detail pane** (`packets.js`):

- The Path row, the cross-observer aggregate row and the byte breakdown
render hops **by position**, from the inferred entries.
- The detail pane no longer writes into the list's shared per-prefix
cache (`hopNameCache`), so it can't change what the packets list shows.

**Hop display** (`hop-display.js`):

| Hop state | Display |
|---|---|
| Confident (≥ 70%) | The node's name; the tooltip says `NN% likely` and
how many nodes share the prefix |
| A guess (< 70%) | Dashed underline, plus a candidate badge when there
are alternatives |
| Unlisted | The hex prefix in muted text; the badge lists the rejected
candidates |
| The only known node, without GPS | The name, with `(no GPS)` |

The candidate popover lists each candidate with its likelihood, or `hop
N` when the path uses that node elsewhere. The "Path symbols" legend
gains an entry for unlisted hops.

**Map route view** (`map.js`, `route-view.js`):

- The `#/map?packet=…&obs=…` deep link infers the chosen observation's
hops with the same shared helper, so the map and the detail pane agree.
- Unlisted hops render as `unlisted` placeholders rather than as the
nearest node that happens to share the prefix.
- When inferred hops are available, the `/api/resolve-hops` request is
skipped.

**Shared helpers** (`packet-helpers.js`):

- `inferObservationPath(obs, observerPos, originPos)`
- `ensureHopAffinity()`: loads the neighbour graph once per page, cached
by `api()` for 5 minutes.
- `routeHopFromInferred(hop, entry, role)`

**Cleanup:** removed dead hop resolution from the "View route on map"
button handler. It computed hops, sender and recipient, then navigated
by URL and discarded them.

**Unchanged:** the packets list, live view, node pages and
`HopResolver.resolve()`.

## Performance

AGENTS rule 0 asks for proof of performance claims; the figures follow.

- **Complexity:** O(hops × 8 × candidates²). Hops more than 8 positions
apart aren't compared directly; a longer run of unlisted hops costs a
flat amount, tracked with a running minimum. Hop-to-hop costs are
computed once and reused by every pass.
- **Where it runs:** once per detail render (plus once for the aggregate
row) and once per map deep link. Nothing runs in the list, live feed or
WebSocket paths.
- **Measured** (Node 22, warm JIT):

| Case | Time |
|---|---|
| 64 hops × 16 candidates, synthetic worst case | 9–14 ms median |
| Real 39–48 hop paths, 1,153 nodes (23 observations) | 1–12 ms each |

- **Enforced:** `test-hop-path-inference.js` asserts a median under 40
ms (target 20 ms) for the 64 × 16 case. It times the module outside
Node's `vm` sandbox, because the sandbox makes global lookups about 10×
slower than a browser.
- **Network:** one extra bulk request per page life
(`/api/analytics/neighbor-graph`, cached server-side, TTL 5 min). No
per-hop or per-packet requests.

## Tests

**New:**

- `tests/unit/test-hop-path-inference.js` (37 assertions). The model
itself:
  - a repeated prefix resolves per position, and no node appears twice;
- the observer anchors the last hop, with the IATA fallback when the
observer has no position or reports 0,0;
  - a far-away lone candidate is reported unlisted;
  - a neighbour-graph link breaks a tie;
  - the server hint and the origin anchor are applied;
  - a node without GPS is handled;
  - output shape, and the performance gate.
- `tests/unit/test-hop-path-detail.js` (32 assertions). Rendering and
wiring:
  - inferred-hop rendering and the popover;
  - positional `renderPath`, including with the 1-byte hop filter on;
- `renderDetail` infers the shown observation from its own
`resolved_path` and leaves the list cache alone;
  - the neighbour graph is fetched once;
  - the shared map helpers, and source checks on the map wiring.
- `tests/e2e/test-hop-path-inference-e2e.js` (6 steps, shard 1). Runs on
fixture packet `57224a4a01c9c682`, observation 147, a 22-hop path that
repeats one prefix three times:
  - every hop is inferred;
  - the repeats resolve per position;
  - the popover shows likelihoods;
  - the map's route list matches the detail pane hop for hop.

**Updated, because they pinned the old behaviour:**

- `test-issue-1418-deeplink-hops-channels.js`: it tested a copy of the
old map priority logic. It now tests the real `inferObservationPath()`,
including `resolved_path` as a hint.
- `test-issue-2098-last-hop-region.js`: the detail pane no longer writes
the prefix cache. The test now reads the region-anchored last hop from
the rendered Path row.
- `test-issue-2097-hop-ambiguity-badge.js`: the source check allows the
new `entries` option while still asserting that the detail pane never
uses the summarised form.
- `.eslintrc.json`: declares the three new shared helpers as globals.

**Results on this branch:**

- `sh test-all.sh`: all standalone frontend suites pass.
- `npx eslint public/*.js`: 0 errors. All 85 warnings are pre-existing;
none come from code this PR adds.
- The "Run Playwright E2E tests (fail-fast)" step was run locally with
its script taken verbatim from `deploy.yml` (`bash -eo pipefail`,
fixture prepared as in CI):
  - shard 1: 27 of 27 passed;
  - shard 2: 47 of 47 passed;
  - shard 3: 50 of 50 passed.

Note for reviewers: `test-issue-1692-packets-init-parallel-e2e.js` only
passes while the freshened fixture's newest packets are less than 15
minutes old. The test sets the time window in `localStorage` after the
page has already read it. CI meets that condition because shards start
right after freshening. Unmodified `master` fails the same way when run
late.

**Browser validation:** the local server ran on the fixture DB. Checked
in the browser:

- the Path row, the candidate popover and the byte breakdown;
- the map route view, which matched the detail pane hop for hop with
unlisted placeholders;
- the console showed no errors.

## Known limitations and follow-ups

- **TRACE packets are not modelled yet.** A TRACE's path is its
*planned* route. The header path byte counts the hops actually forwarded
(repeaters append SNR, not their hash, `Mesh.cpp`), and a trace may
revisit a node on its way back (see the comment in
`Packet::calculatePacketHash`). A follow-up will add a `heardAfter` /
`allowRepeats` option, so a trace is anchored after the hops it actually
completed and may repeat nodes.
- **Customizer (AGENTS rule 8):** the `PATH_MODEL` tuning values are
hardcoded for now. The unlisted cost, typical and maximum hop length,
and the confidence threshold should become customizer settings.
- **Server-side `resolved_path`:** the ingestor writes a hop as resolved
whenever its prefix is unique in the node table, even when that node is
hundreds of km from the neighbouring hops. `inferPath()` treats these as
hints and can overrule them, but the stored data is still misleading.
- **The first hop without a sender position is often a close call.** On
encrypted channel messages, hop 0 is constrained only by what follows
it, so 55/45-style splits between a long chain of known nodes and
"unlisted" are expected. The badge and the percentage show this rather
than hide it.

## Screenshots

This is what a normal corescope instance would show for a 1 byte path.
<img width="318" height="338" alt="image"
src="https://github.com/user-attachments/assets/4056bc27-0b37-4107-9b25-31ac0f7864ec"
/>

<img width="430" height="601" alt="image"
src="https://github.com/user-attachments/assets/01da5ee8-babf-4e41-a9ab-836b5d640549"
/>

Now the same packet but with the logic to infer each hope along the path
different nodes are chosen as they're ranked and selected based on
proximity to the next hop,

<img width="436" height="646" alt="image"
src="https://github.com/user-attachments/assets/bcca73b8-daf0-4031-8ea1-8c96d716c8b3"
/>

<img width="436" height="646" alt="image"
src="https://github.com/user-attachments/assets/fc926d9e-158c-48ce-bee6-93086afb0775"
/>

The higher the confidence level then there's no need to show the
exclamation mark to show that multiple nodes share the same prefix but
it does continue to share that

Examples of it currently running can be found at,
https://corescope.baymc.org - modified instance

https://analyzer.comchan.net - unmodified instance
https://analyzer-stage.comchan.net - modified instance

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-19 20:00:07 +02:00
efitenandClaude Opus 5.5 2d09335cfb fix(packets): keep the node filter pick on a slow click or late search (#2243) (#2244) 
Fixes #2243.

## Problem

The packets node filter selected a suggestion on `click`, while the
input's `blur` handler hides the dropdown 200 ms after focus leaves. A
press held longer than that, or a search response that rebuilt the list
between mousedown and mouseup, lost the pick: the typed text stayed and
no `node=` filter was applied.

## Change

- `public/packets.js`: suggestions select on `mousedown` with
`preventDefault`, so the input keeps focus and the pick happens before
any rebuild. `live.js` already does this for its node filter.
- `public/packets.js`: a sequence counter drops search responses for an
older query, bumped on every search and on a pick, so a late response
can't replace or reopen the list.
- `tests/e2e/test-packets-node-filter-e2e.js` (new), registered in
`scripts/non-unit-tests.json` and on shard 1 in `deploy.yml`.

## Tests

The new E2E has three cases: a normal click, a 400 ms press, and a
search response released mid-click.

| Code | Normal click | 400 ms press | Response mid-click |
|---|---|---|---|
| master | pass | fail | fail |
| this PR without the sequence guard | pass | pass | fail (list reopens)
|
| this PR | pass | pass | pass |

Run locally against the freshened e2e fixture. `sh test-all.sh` passes
(locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and
`PYTHONUTF8=1`, unrelated to this change).

## Not done

- Full `test-e2e-playwright.js` not run locally.
- Could not reproduce on the reporter's instance with the browser tool,
whose clicks are shorter than 200 ms. The 400 ms Playwright press
reproduces it on the fixture.
- `live.js` and `packets.js` still carry two separate node-autocomplete
implementations. Merging them is out of scope here.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 21:30:57 +02:00
efitenandClaude Opus 5.5 a910a3e480 fix(packets): re-fetch when the observer filter changes (#2241) (#2242) 
Fixes #2241.

## Problem

In grouped mode, ticking an observer on `#/packets` updated the URL and
the trigger label but left the table unchanged until a reload. Since
#1821 `applyObserverFilter` leaves grouped rows to the server's EXISTS
filter, while the observer menu's change handler still only called
`renderTableRows()`, which re-renders the rows already loaded without a
new `/api/packets` request.

## Change

- `public/packets.js`: the observer menu change handler calls
`loadPackets()` instead of `renderTableRows()`, so the request carries
the new `observer=` set.
- `tests/unit/test-observer-menu-interactions.js`: two tests asserting
the handler re-fetches, for ticking one observer and for "All
Observers". Both failed before the change.

## Verification

- `sh test-all.sh`: all suites pass (locally
`test-preflight-xss-gate.js` needs a real `python3` on PATH and
`PYTHONUTF8=1`, unrelated to this change).
- Browser, local server on the freshened e2e fixture: 55 rows; one
observer ticked gives 2 rows, all from that observer; back to "All
Observers" gives 53 rows. No reload in between.
- Reproduced the original bug on an operator's v3.14.0 instance before
the change.

## Not done

- Playwright E2E suite not run locally.
- Perf: one `/api/packets` request per checkbox toggle, the same request
a reload or the node filter already makes. No per-item calls.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 21:07:27 +02:00
Alex BandClaude Opus 5.5 8b5ca5735c fix(analytics): Packets / Hour readable on long histories (#2193) (#2194) 
Fixes #2193.

## Problem
On an instance that keeps months of packets, the Overview tab's Packets
/ Hour chart becomes unreadable: 2,932 hourly bars in 720 units of plot,
a date label every day, and the hours with no packets (which the server
leaves out) squeezed off the axis.

## Before and after
Both use the real `/api/analytics/rf` answer from the instance in the
issue (2,933 hourly buckets over 124 days), served to the page in place
of the fixture's own.

**Before** (master, page on "All data"):


![before](https://raw.githubusercontent.com/A13xB0/CoreScope/e3cdf135/2026-10/pph-2193/before-desk.png)

**After**, same data:


![after](https://raw.githubusercontent.com/A13xB0/CoreScope/7862abd8/2026-10/pph-2193/after-desk.png)

Phone: <img
src="https://raw.githubusercontent.com/A13xB0/CoreScope/7862abd8/2026-10/pph-2193/after-phone.png"
width="300">

**A short store** (an instance holding 7 days in memory, plus a few old
packets a day for weeks before that because they were heard again). The
chart starts where the real data does (this shot predates the "since
MM-DD" note now in the title when that crop applies):

![short
store](https://raw.githubusercontent.com/A13xB0/CoreScope/7862abd8/2026-10/pph-2193/short-store.png)

## Change
All in `public/analytics.js`; no API change.
- **`packetsPerHourCard()`** replaces the inline block:
- rolls the hours up to hourly, 6-hourly, daily or weekly bars, at most
180, and titles the card to match (Packets / Day);
  - counts missing hours as zero;
  - puts about 10 labels on round UTC times;
- **under "All data" only**, starts on the first UTC day with at least
2% of a typical recent day (the median of the seven days before the last
one), at its first hour with a tenth of a typical hour, and says "since
MM-DD" in the title when that cuts anything. Scanning forward, an outage
after the start is never taken for it. In a picked window (1h to 30d)
the server has already cut the data, so nothing is cropped and a quieter
early stretch stays visible.
- **Theme refresh before the data** (separate commit; this also happens
on master without the new chart): every load fires `theme-refresh` about
300 ms after the theme resolves, and the page re-rendered the current
tab on it. When the analytics answers came later (a busy or
just-restarted server), the Overview was drawn from an empty object and
threw `Cannot read properties of undefined (reading
'totalTransmissions')`. It now skips that refresh until the data has
landed; `loadAnalytics()` renders the tab with the current theme when it
does.

The span is still picked by the page's time window.

## Tests
- `tests/unit/test-issue-2193-packets-per-hour.js` (in `test-all.sh`,
red then green): 122 days gives 123 daily bars with 4 to 12 labels; 7
days gives 168 hourly bars; missing hours are drawn as zero; the chart
starts at the data, not at the stragglers, and not after an outage; in a
picked window nothing is cropped (90 days with 30 quiet ones gives 91
bars); a cropped chart says "since MM-DD" and an uncropped one does not;
no data says so.
- `tests/unit/test-analytics-theme-refresh-before-data.js` (in
`test-all.sh`): drives `init()` with answers that never land, fires the
`theme-refresh` handler, and checks that nothing throws and the tab is
not re-rendered. Without the guard it fails with the production error.
- `sh test-all.sh` passes; the #1925 neighbour-graph theme-refresh test
and the analytics chart test still pass; the screenshots show no page
errors and no sideways scroll at 1280 px or 390 px.

## Later
- The 2% floor, the tenth-of-an-hour start and the 180-bar cap are
hardcoded. Per AGENTS.md rule 8 they can move to the customizer in a
later step.
- Edge bars are partial periods (data starting at 17h gives a 7-hour
first day; the last bar is still filling), and weekly bars align on
epoch weeks (Thursday). Both are left as they are here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 11:46:28 +02:00
hermes-gadget fdcec91578 fix(route-view): "N hops" counts only true hops, not payload endpoints (#2240) 
Fixes #2206

## What

Follow-up to #2207, as sketched in its review: the route meta (desktop
header + mobile summary) printed `positions.length` as "N hops". For a
DM, `positions` includes the payload-decoded sender and recipient rows —
endpoints, not hops — so the count was inflated. Live on
corescope.ukmesh.com right now, the reported 6-relay chain reads **"8
hops · 913 km"** with "from payload" chips on rows 1 and 8.

## Change

- `public/route-view.js` — new `countHops(positions)` helper: counts
rows **without** `_fromPayload` (both builders set the flag since
#2207). The desktop header and the mobile summary both render it, so
they can't disagree. Only the displayed figure changes; the row-count
`total` still drives distances, stripe colours and row slicing.
- `tests/unit/test-issue-2206-hop-count.js` — regression test (listed in
`test-all.sh`): extracts `countHops()` from the shipping file and
exercises it (6-relay DM → 6; payload-only → 0; plain chains and
non-payload origin/dest unchanged), and pins both meta strings to the
helper so a regression back to `total` fails.

The count intentionally counts relay rows (per the review note), not
edges: a 6-relay DM reads "6 hops".

Frontend-only, static assets, no build step, no new dependencies. Cache
buster untouched (AGENTS.md rule 3).

## Perf

One `.filter()` over a ≤ ~10-entry array, once per sidebar build; no
hot-path impact.

## Testing

- `node tests/unit/test-issue-2206-hop-count.js` — 11 passed, 0 failed
- `sh test-all.sh` — full unit suite green, exit 0
- Live cross-check against the deployed instances (pre-fix logic):
`https://corescope.ukmesh.com/#/map?packet=44259e35b7c826a8&obs=97234`
shows 8 rows, meta "8 hops · 913 km", payload chips on rows 1 and 8 →
this change makes that same chain read "6 hops · 913 km". The new count
ships with the next build that carries it; happy to re-check on a
deployment once one does.
 2026-10-17 10:39:08 +02:00
nullrouten0nullroutenClaude Sonnet 5.5
 		127dd511a7 fix(health): judge favorites and home status cards by the node's role (#2215) 
## The problem

A starred repeater can show two different health statuses. The My
Repeaters tab (#1761) and `getNodeStatus` judge it by the infra
thresholds (degraded after 24 h, silent after 72 h). The favorites
dropdown and the home page cards judge every node by the node thresholds
(degraded after 1 h, silent after 24 h), because they never look at the
node's role.

| repeater last heard | favorites / home cards (before) | My Repeaters
tab | after this PR |
|---|---|---|---|
| 5 h ago | degraded | healthy | healthy |
| 30 h ago | silent | degraded | degraded |
| 80 h ago | silent | silent | silent |

If you set `healthThresholds` in `config.json`, every view follows your
values.

## The change

Three call sites now use `getHealthThresholds(role)`, the helper the My
Repeaters tab already uses:

- `renderFavDropdown` in `app.js`
- the "My Nodes" grid card in `home.js`
- the hero search-result card in `home.js`

The search card's fallback text, "Not heard in 24+ hours", was only true
for non-infra nodes, so it now reads "Not heard recently".

## Scope

- I kept the existing three-state healthy / degraded / silent logic and
changed only which thresholds apply.
- Not included: counting a repeater that is relaying traffic as alive
(#1598, which `getNodeStatus` does). That is a bigger behavior change
and would be its own PR.

## Risk

Low. It is 9 lines in two files, and the helper and thresholds already
exist and are used elsewhere.

## Tests

New `tests/unit/test-health-card-role-thresholds.js`, added to
`test-all.sh`. It checks that infra roles get looser thresholds than
nodes, and that each of the three call sites goes through
`getHealthThresholds(role)` with no node-only comparison left. On
current `master` 5 of its 7 assertions fail; with this change all pass.

`sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning from an
a11y self-test, which it also does on `master`.

## Update after review (commit `0f2376df`)

The text above is the description efiten reviewed on 2026-10-15. This
section describes the code as it is now.

### The change

- New `getHealthStatus(role, heard, now)` in `roles.js`. It returns
`healthy`, `degraded` or `silent` by the role's thresholds, or `null`
when there is no usable timestamp. It only applies the cut-offs; each
caller picks the timestamp.
- All six places that compared an age against `degradedMs` / `silentMs`
now call it:
  - the favorites dropdown (`app.js`);
  - the "My Nodes" card and the home health card (`home.js`);
  - the analytics network status and My Repeaters (`analytics.js`);
  - the live node panel (`live.js`).

Each keeps its own labels and timestamp, so the last three show the same
as before.

`analytics.js` and `live.js` no longer guard with `typeof
getHealthThresholds`. `roles.js` loads before both, and both already
used it without a guard elsewhere: the old network status called
`getHealthThresholds` directly, and `live.js` reads `ROLE_COLORS`.
- The favorites dropdown and both home cards now fall back to the node's
`last_seen` when the packet store has no packet for it. My Repeaters
(`last_heard || last_seen`) and the node page (`nodes.js:729`) already
do this. Without it, with a packet store window under 72 h, a repeater
could be silent on the cards and degraded on My Repeaters. The "ago"
text uses the same timestamp.

With the default thresholds, the fallback changes a status only when the
packet store holds less than 72 h of packets. `retentionHours` unset
(unlimited) or at the example's 168 h holds more than that. Otherwise a
node with no packet in the window is silent either way, and the views
now show how long ago it was last seen instead of "never" or nothing.
- With no timestamp at all, the home health card said "Not heard in 24+
hours". It now says "No packets in the current window".

### Why a new helper and not `getNodeStatus`

`getNodeStatus` has two states (active / stale) and counts a repeater
that relays traffic as alive (#1598). Using it here would change more
than the thresholds. `getHealthStatus` keeps these views' three states.

### Scope

- Not included: counting a repeater that relays traffic as alive (#1598)
in these views. That is a bigger behavior change and would be its own
PR.
- The live node panel still judges by `last_seen` alone, and the node
page stays relay-aware. Both now use the shared thresholds; aligning
their timestamps would also be its own change.

### Tests

- `tests/unit/test-health-status.js` runs `roles.js` in `vm`. It covers:
- repeater, room, companion, sensor, an unknown role and a missing role,
at ages either side of 1 h, 24 h and 72 h;
  - ISO and epoch input;
  - null, empty and invalid timestamps;
  - a `healthThresholds` override through `/api/config/client`.

One source guard remains: no age-vs-threshold comparison outside
`roles.js`.
- `tests/unit/test-my-repeaters-dashboard.js` now uses the real helper
instead of a stub, which gave repeaters the node thresholds. It also
checks that a repeater heard 30 h ago is Degraded.
- New `tests/e2e/test-health-status-e2e.js` mocks
`/api/nodes/<pk>/health` for six nodes. It checks five views: the
favorites dropdown, the "My Nodes" card, the home health card, the
analytics network status and the live node panel. It fails on `master`
and on this PR's first commit. I put it on shard 2, which was the
shortest shard in the last CI run.
- Mutation check: I made 33 deliberate breaks. In the helper they cover
each comparison and boundary. At each of the six call sites they cover
the timestamp field, the role argument and at least one label mapping.
Each one makes the suite fail.
- I checked a `config.json` with `infraDegradedHours: 2` and
`infraSilentHours: 6` in Chromium. Repeaters heard 3 h and 7 h ago show
Degraded and Silent.
- `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning, which
comes from a deliberate fixture in the a11y self-test.

### Cost

The analytics network status loops over every node. In headless
Chromium, 30,000 synthetic nodes take 9.0 to 9.2 ms before this change
and 9.1 ms after (median of 21 runs, on one machine).

### Risk

Low. The change is frontend only, with no API, config or data change.
Rolling back means reverting the commits.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
 2026-10-17 09:38:11 +02:00
hermes-gadgetandhermes-gadget 550829af6e fix(route-view): badge payload-decoded sender/recipient rows in the sequence renderer (#2207) 
Refs #2206

## What

DM packets on the route view showed **sender** and **recipient** rows —
decoded from the packet payload (`destHash`/`srcHash`, one byte each)
rather than from the outer path bytes — as ordinary hops, without the
"from payload" badge, whenever the chain renders through the
multi-path/sequence builder. Only the single-path builder set
`_fromPayload`, and the isolate/sequence list renderer never drew the
chip.

Effects: payload endpoints are indistinguishable from observed hops, and
the displayed hop count is inflated (a 6-relay DM renders 8 rows, with
the last "hop" being a single-byte recipient prefix resolved to
whichever node shares it).

## Change

- `public/map.js` — set `_fromPayload: true` on the sequence builder's
origin (sender) and destination (recipient) entries (3 + 2 sites).
- `public/route-view.js` — render the `mc-rt-status-chip
mc-rt-status-payload` "from payload" chip in the isolate/sequence list,
matching the canonical renderer. The chip markup now lives in one
`payloadChip(p)` helper called by both row builders (it was duplicated
between them, so they could drift apart).
- `tests/unit/test-issue-2206-payload-badge.js` — regression test
(listed in `test-all.sh`): extracts `payloadChip()` and `rowFor()` from
the shipping `route-view.js` and exercises them in a vm sandbox, and
pins every sender/recipient entry in `drawPacketRouteMulti` to
`_fromPayload: true`.

Frontend-only, static assets, no build step, no new dependencies. Per
AGENTS.md rule 3 the cache buster in `public/index.html` is
intentionally untouched — `__BUST__` is replaced automatically at server
startup.

## Perf

Two ternaries + one string concat per rendered row in a display-only
list; no impact on hot paths.

## Testing

- `node tests/unit/test-issue-2206-payload-badge.js` — 22 passed, 0
failed (new regression test)
- `sh test-all.sh` — full unit suite green, exit 0 (201 suites)
- Browser validation, re-run against the live deployments (which carry
the equivalent patch): on
https://corescope.nemesh.uk/#/map?packet=44259e35b7c826a8&obs=97234 the
canonical chain marks rows 1 and 5 "from payload"; isolating the 6-hop
path (`B4CE24→…→8C197F`) renders the 8-row chain with rows 1 and 8
marked. The refactor keeps the renderer output identical (pinned by the
new test). The identical patched assets have been running on two
independent deployments since 2026-10-13.

## Notes

The underlying ambiguity (a 1-byte prefix matched against many nodes,
resolved to the most-recently-seen candidate) is what the badge surfaces
to the user; changing the resolution strategy itself is out of scope for
this PR. The `N hops` header still counts every row — counting only
non-payload rows would change every DM header, so it stays a follow-up;
hence `Refs #2206`.

---------

Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com>
 2026-10-16 10:54:36 +02:00
efitenandClaude Opus 5.5 927a8eec8a fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211) 
Refs #748 (M3a: correct numbers within the in-memory window; the SQL
history in M3b is a separate plan, see the end).

## Problem

The node analytics range picker offers 24h, 7d, 30d and All (365d), but
the store only holds its in-memory window (`retentionHours`, then
eviction). `GetNodeAnalytics` still divided availability and packets/day
by the requested range. Measured on our live instance (14-day window),
repeater BE-BRE-ON8AR:

| Range | Availability | Packets/day |
|---|---|---|
| 7d | 88.7% | 91 |
| 30d | 42.6% | 265 |
| All | 3.5% | 22 |

Nothing on the page said that the data stopped at the edge of the
window.

## What changes

- `coverageStart` returns the edge of the in-memory window: the latest
of `oldestLoaded` (background-load progress), `now - retentionHours`,
and a new `evictedThrough` that eviction sets to the newest `FirstSeen`
it removed, so memory-pressure eviction inside the retention window
moves the edge too.
- `s.packets[0]` is not that edge, and the first version of this PR used
it. Chunks are selected by `last_seen`, so a transmission first heard
weeks ago and heard again recently sits at the head with its old
`FirstSeen`. On staging (7-day window) it returned 21/07.
- When the edge lies more than a minute after the requested start, node
analytics counts only transmissions from the edge, computes availability
and packets/day over that span, and returns `timeRange.dataFrom` and
`timeRange.partial`.
- The minute of slack keeps a range equal to the retention window from
reading as partial. Without it every 7d request on staging (168h) was
partial, because the two edges came from separate clock reads.
- The analytics page shows "Data from 2026-10-07: the server keeps about
7 days of packets in memory. Availability and packets/day cover that
period, not the full 365 days." when partial.
- **UTC fix.** The window bounds of node analytics and node hop
analytics are formatted in UTC. They are compared as strings with UTC
`FirstSeen` values, so on a non-UTC host the 7d window lost its oldest
hours (2 at +02:00). Docker images run in UTC, which is why this did not
show in production.
- `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the
edge is kept.

## Tests

- Go, `node_analytics_coverage_test.go`:
  - full coverage within memory;
  - 30d over 3 days in memory;
  - empty store;
- a re-heard 85-day-old transmission at the head of `s.packets` (edge
stays at retention, packet not counted, no bucket before `dataFrom`);
  - memory-eviction edge;
  - eviction records `evictedThrough`;
  - retention 30 s shorter than the range is not partial, while 30d is;
  - window bounds under a +02:00 `time.Local`.
- Two tests were checked to fail on the old code: the local-zone test
(166 of 168 packets, a `+02:00` bound) and the retention test without
the slack (`dataFrom 09:57:08` vs `from 09:56:38`).
- Full `cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master). One run of five reported `FAIL` and four reruns were green; I
did not capture which test failed.
- Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js`
(5 assertions on the real `node-analytics.js` in `vm`), in
`test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js`
(registered in `deploy.yml` and `scripts/non-unit-tests.json`), against
the real fixture response with no interception: 7d is not partial and
shows no notice; All is partial and the notice names the real start
date. 2/2 locally.

## Validation

Staging (retentionHours 168), same repeater, after the background load
completed:

| Range | Before | After |
|---|---|---|
| 7d | 82.7% | 83.3%, not partial |
| 30d | 19.3% | 83.3%, partial from 2026-10-07 |
| All | 1.6% | 83.3%, partial from 2026-10-07 |

In the browser the notice appears under All and not under 7d.

## Not done

- M3b, older history from SQLite. In the database only the sender is
indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay
hops only in `observations.resolved_path` (filled for about 27% of
observations on our live instance, not indexed). A fallback today would
show only a node's own packets for older days, a different meaning than
the in-memory part. Doing it properly needs a per-node index written by
the ingestor (pubkey, transmission, role); that is a separate plan.
- The other analytics endpoints with a range (observer analytics, RF)
were not checked for the same dilution.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 09:27:21 +02:00
efitenandClaude Opus 5.5 cab4f71c64 feat(nodes): LoRa radio settings per node from the MeshCore map (#1268) (#2209) 
Fixes #1268.

Frequency, spreading factor, bandwidth and coding rate are per-device
settings that never travel in a packet. The public MeshCore map is the
only source. This PR adds them to the node detail views, opt-in, and
answers the open policy questions from the triage comments.

## Policy decisions

| Question (triage) | Answer in this PR |
|---|---|
| Opt-out for self-hosted instances | Opt-**in**: `rfParams.nodesUrl`
empty (default) = nothing fetched, `/api/rf-params` returns `enabled:
false`. Same model as `knownChannelsUrl`. |
| 33 MB every 6 h, thundering herd | Random 0-60 s delay before the
first fetch, then every `refreshHours` (default 6). The map sends gzip
(17 MB on the wire). It sends no ETag or Last-Modified, so conditional
requests are not possible. |
| Size cap | 128 MB on the decoded body (it is 51.5 MB today). Exceeding
it fails the fetch instead of parsing a truncated body. |
| Field shapes at the parse boundary | 64-hex key; 100-1100 MHz; SF
5-12; bandwidth from the LoRa set (7.8 to 500 kHz); CR 5-8. Invalid rows
are dropped. A body that is not a well-formed array fails the fetch. |
| Keep stale cache on anomaly | Any failure keeps the previous snapshot;
a failed presets fetch keeps the previous presets. `fetched_at` is
returned so an outage is visible rather than hidden. |
| Freshness | Only adverts within `maxAgeDays` (default 7) are kept at
fetch time and re-checked at serve time. |
| TLS pinning | Not done: Go's certificate validation plus the field
validation above. |

## What changes

- `cmd/server/rf_params_cache.go`: stream decoder, validation, cap
reader, preset parser (presets with identical values are merged, titles
joined), background refresh. In memory only, no DB writes, so the
read-only server invariant holds.
- `GET /api/rf-params`: `{enabled, fetched_at, source, presets, nodes}`;
`nodes` holds only nodes this instance knows, keyed by lower-case
pubkey. `/api/nodes` is unchanged, so its payload does not grow for
every consumer and the 500-row paging is unaffected.
- Node detail (side panel and full page): `LoRa` row with the four
values, the preset name when all four match one, and the age of the map
advert. Shown only when the feature is on and the server has fetched the
map at least once.
- No column in the Nodes table, on purpose: on our network such a column
showed one preset title in 814 of 1,366 rows and nothing in 443. Showing
only the nodes that differ from their region is the follow-up PR.
- `config.example.json` documents the block with the two URLs to use.

## Cost

- Measured on the real map payload (51.5 MB, 66,336 nodes): parse 0.52
s, 15 MB allocated in total, 26,815 nodes kept (advert within 7 days).
Runs in a background goroutine every 6 h.
- The endpoint walks the cached node list once with map lookups:
O(nodes).
- Frontend: one bulk fetch of `/api/rf-params` when a node detail opens
(client TTL 5 min), not one per node.

## Tests

- Go (`rf_params_cache_test.go`): parse with every rejection reason,
malformed bodies (not an array, truncated, wrong type, empty), preset
merge and invalid presets, cap reader (one byte over refused, exactly at
the cap accepted), fail-soft against an `httptest` upstream (503, HTML
body, presets-only failure), config defaults, route (disabled shape,
only known nodes, entry aged past max age dropped at serve time). Full
`cmd/server` suite passes locally (skipping
`TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean
master).
- Frontend unit: `tests/unit/test-issue-1268-rf-params.js` (7 assertions
on the real file in `vm`), in `test-all.sh`; the full suite passes.
- E2E: `tests/e2e/test-issue-1268-rf-params-e2e.js` (registered in
`deploy.yml` and `scripts/non-unit-tests.json`). It checks the real
disabled state on the fixture server (no detail row), then serves
`/api/rf-params` through `page.route`: no table column, full detail page
with preset, side panel with raw values, and the hint for a node that is
not on the map. 4/4 locally, plus the existing node table and node
detail E2E files.

## Validation

On a staging instance with `rfParams` set to the map and preset URLs:

- Log: `[rf-params] 26821 nodes within max age, 20 presets`, about 6 s
after start.
- `/api/rf-params` returned 1,115 of our nodes: 988 on
869.618/SF8/BW62.5/CR8, 111 on the same with CR5, 15 on 433.65 MHz, 1 on
SF7.
- In the browser the detail row shows in the side panel and on the full
page, and the Nodes table keeps its original columns.

## Not done

- Highlighting nodes whose settings differ from their region's majority
(a badge in the Nodes list plus a filter) is the follow-up (separate
PR). In that follow-up a CR-only difference stays informational until it
is confirmed whether MeshCore radios use the explicit LoRa header (then
CR travels in the header and a mismatch still decodes).
- `maxAgeDays` is config-only; the customizer is a later step.
- No data is written to the DB, so the RF parameters are gone after a
restart until the next fetch (at most 60 s plus the fetch time).

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 09:26:45 +02:00
efitenandClaude Opus 5.5 1ae4c74913 feat(observers): rolling 1 h TX airtime and a duty-cycle warning (#661) (#2208) 
Fixes #661 (the narrowed scope from the 2026-10-14 comment).

## What changes

- `GET /api/observers/metrics/summary` adds `tx_airtime_pct_1h` and
`rx_airtime_pct_1h` per observer: the airtime share over the hour before
now, whatever `window` is, from the cumulative `tx_air_secs` /
`rx_air_secs` counters in `observer_metrics`.
- A counter that goes down marks a reboot; only the stretch after the
last reset counts.
- Under 15 minutes of samples (one sample, or a reboot late in the hour)
returns `null`.
  - Clamped to 0..100, rounded to 2 decimals.
- The response also carries `duty_cycle_warn_pct`, from the new config
key `dutyCycleWarnPct` (default 10, the EU 868 MHz g3 sub-band limit).
Added to `config.example.json`.
- RF Health tiles show `TX 1h: x%`; the observer list gets a sortable
numeric `TX 1h` column. Both use the shared `public/observer-airtime.js`
helper and flag values at or over the threshold with `.airtime-over`
(`var(--status-red-text)`).
- The handler now returns a named `MetricsSummaryResponse` struct
instead of `map[string]interface{}` (one occurrence fewer).

## Why the warning matters

Repeater and room-server firmware default to `airtime_factor = 1.0`
(`examples/simple_repeater/MyMesh.cpp:890`), which `Dispatcher.cpp:26`
turns into a 50% TX budget per 1 h window. The device therefore does not
keep itself under the EU 10% limit; this view is where an operator can
see it.

## Cost

- Server: one extra query per summary request, `SELECT observer_id,
timestamp, tx_air_secs, rx_air_secs FROM observer_metrics WHERE
timestamp >= now-1h`, on the existing `idx_observer_metrics_timestamp`,
then one O(rows) pass. On our instance that is 35 observers x about 12
samples (5 min interval).
- Observer list: one extra bulk call `summary?window=1h` per load
(client TTL 30 s, failure ignored), not a call per observer.

## Tests

- Go: `TestRollingAirtimePct` (steady, reboot mid-hour, single sample,
span under 15 min, reboot leaving under 15 min, nil counters, clamp,
rounding), `TestGetMetricsSummaryAirtime1h` (samples older than 1 h
ignored inside a 24 h window, single-sample observer gets null),
`TestMetricsSummaryDutyCycleWarnPct` (default 10, config override,
`observers` is `[]` not `null`). Full `cmd/server` suite passes locally
(skipping `TestSaveGeoFilterPreservesFileMode`, which fails on Windows
on clean master too).
- Frontend unit: `tests/unit/test-issue-661-observer-airtime.js` (8
assertions, real file in `vm`), added to `test-all.sh`.
`test-observers-headings.js` updated for the new column. Full
`test-all.sh` passes locally.
- E2E: `tests/e2e/test-issue-661-airtime-e2e.js`, registered in
`deploy.yml` and `scripts/non-unit-tests.json`. The fixture DB has no
`observer_metrics` table, so the summary is served through `page.route`
with known values for the fixture's own observer ids. It checks the
column header, over/under/unknown cells, sort order and the RF Health
tile. 4/4 locally. It caught a real bug during development:
`loadRFHealthData` declares a local `const window`, so the tile calls
the helper as the global `ObserverAirtime`.

## Validation

Deployed to a staging instance with live MQTT data (35 observers with
metrics) and checked in the browser:

- The observer list shows the column for 42 rows (33 with a value, 9
without enough samples). Sorting puts the highest share (5.3%) first.
- RF Health shows `TX 1h` on all 35 tiles. With a fourth stat the line
wrapped inside items on narrow tiles; the stats row now wraps between
items only.
- No observer was over 10% at the time, so the red state was checked by
rendering one cell client-side at 12.5% (computed colour `rgb(185, 28,
28)` in light mode), and by the E2E test.

## Not done

- The threshold is config-only; exposing it in the customizer is a
follow-up.
- No airtime comparison across observers (that is #671) and no
per-packet airtime (Relay Airtime Share covers it).
- Dark mode was not checked in the browser; `--status-red-text` is
defined for both themes.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-14 21:07:20 +02:00
efitenandClaude Opus 5.5 fbeb9e4d71 ci: frontend coverage from every E2E suite, merged across the shards (#2204) 
Fixes #2203.

## Problem

The frontend coverage badge (35.16% on run 38413268001) came from two
sources on shard 2: one dump part-way through `test-e2e-playwright.js`
and the `collect-frontend-coverage.js` tour. The other E2E suites run
against the instrumented build but never save `window.__coverage__`,
including the suites written for single pages (`path-inspector.js`
4.72%, `touch-gestures.js` 14% on that run).

## Change

1. `9d3bfad5` `test(e2e): red:` `tests/unit/test-e2e-coverage.js`, with
fake pages, contexts and browsers (no Chromium needed). On the
placeholder module the first assertion fails.
2. `c00bbb12` `feat(e2e):` the collector.
- `scripts/e2e-coverage-hook.js` is a preload (`node --require`) that
wraps `chromium.launch` from `playwright`. `playwright` re-exports
`playwright-core`, so a suite's own `const { chromium } =
require('playwright')` gets the same, wrapped object.
- `scripts/e2e-coverage.js` reads `window.__coverage__` before every
call that replaces or ends a page's document: `goto`, `reload`,
`setContent`, `goBack`, `goForward`, `close`, and context and browser
`close`. Pages from `newPage`, `newContext` and the context `page` event
are all tracked.
- Each read zeroes the page's counters, so a document read twice is not
counted twice. Maps travel once per file; later reads carry counts only.
- The page returns a JSON string. Returning the object cost about 10 s
per suite, because Playwright serialises it node by node.
- A read is given up after 5 s, so a busy page cannot hold up a suite's
cleanup.
- On exit the hook writes one file per suite to `.nyc_output/`, or
nothing without coverage. A failed write is logged and never changes the
exit code.
3. `64487c4c` `ci:` the workflow.
- On master pushes the E2E run step sets `NODE_OPTIONS=--require
./scripts/e2e-coverage-hook.js`. On PRs it is empty, so PR runs are
unchanged.
- Each shard runs `nyc merge` (one file per suite is about 6 MB, mostly
the same maps) and uploads `coverage-shard.json` with its results
instead of `.badges/`.
- The `e2e-test` job merges the three shards and makes the frontend
badge (it used to come from shard 2 alone).
- The merge, setup-node and badge steps are `continue-on-error`. When
`nyc report` gives no statement percentage the step warns and writes no
badge, so a registry hiccup cannot publish 0%.
- `test-e2e-playwright.js` no longer writes its own `e2e-coverage.json`:
the hook covers that suite completely, including the tests after the old
dump. `collect-frontend-coverage.js` stays as an extra source.

## Performance

Measured locally against an instrumented server (Windows, Chrome):

| Suite | Without hook | With hook |
|---|---|---|
| `test-path-inspector-coverage-e2e.js` | 1.9 s | 2.2 s |
| `test-touch-gestures-coverage-e2e.js` | 17 s | 16 s |
| `test-e2e-playwright.js` (stops at the same local-fixture failure in
both runs) | 52 s | 52 s |
| `browser.close()` on a page in a 30 s busy loop | 153 ms | 5.2 s |

The hook only loads on master pushes.

## Testing

- `tests/unit/test-e2e-coverage.js` covers:
  - reads before each document-replacing call and before close;
  - pages opened by the page itself;
  - context and browser close;
  - summing without double counts;
  - a failed read, and a read that never answers;
  - the file writer, including a failing write.
- Nine deliberate breaks of the library each turn it red:
  - dropping the counter reset;
- dropping the `page` event, the `goto` read, the `goForward` read, the
`close` read or the browser `close` read;
  - removing the timeout or the write guard, or writing an empty store.
- The test also fails if it stops before its end. Without that check, a
read that never returns would let node exit 0 half-way.
- Real suites with the hook, measured on their own (the badge run had
all suites):

| File | Badge run 38413268001 | Own suite with the hook |
|---|---|---|
| `path-inspector.js` | 4.72% | 52.75% |
| `touch-gestures.js` | 14% | 72.96% |

- Exit codes and pass counts were unchanged with the hook for these
suites (the first three were run before the 5 s read timeout was added):
  - `test-slideover-1056-e2e.js` (6 contexts);
  - `test-a11y-axe-1668.js` (`@axe-core/playwright`);
  - `test-issue-1150-404-state-e2e.js`;
  - the three above.
- The badge step, run locally against two merged shard files:
  - It gives the same per-file numbers as `nyc report` on the inputs.
- With no shard coverage, or with `nyc report` failing, it warns and
writes no badge.
- `sh test-all.sh` passes locally, apart from
`test-issue-1956-release-routing.js`, which needs the workflow's
`GITHUB_REF_NAME` and was skipped. The test inventory classifies 321
suites. `actionlint` 1.7.7 reports only the self-hosted runner label, as
on master.

**Not tested:** the shard and badge steps in CI. They only run on a
master push, so the first master run after merge is the real test. I do
not know what the merged number will be. I will compare the badge with
the merged report in that run's `e2e-test` log.

**Not counted:** a full page load the suite does not start through one
of the wrapped calls, such as a link that leaves the SPA or
`location.reload()` from page code.

## Not in this PR

- The `tests/unit/` suites, which load frontend files through
`vm.createContext` without instrumentation (noted in #2203).
- Coverage on PR runs. The overhead above is small, so this could follow
if wanted.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-13 22:04:03 +02:00
SaarMesh-BotSaarMesh-BotClaude
 		45680a22cf fix(#2191): apply the channel filter to live WebSocket packets (#2192) 
Fixes #2191.

## Problem

On the Packets page the channel filter only applies to the initial load
(`/api/packets?channel=` → server-side `transmissions.channel_hash`
match). The WebSocket handler re-checks time window, type, observer,
region and node, but not `filters.channel`, and the channel filter is
not re-applied at render time. With a channel selected, every new live
packet is appended regardless of its channel.

## Fix

- New pure helper `packetMatchesChannel(p, channel)` in
`public/packets.js`, exported as `window._packetsMatchesChannelForTest`
(same pattern as `_packetsBuildChannelOptionsForTest`).
- It derives the value the server filters on from `decoded_json`,
because WS packets don't carry `channel_hash`:
  - `channel` when the server decrypted the `GRP_TXT`,
  - otherwise `'enc_' + channelHashHex.toUpperCase()`.
- Only `payload_type === 5` can match, mirroring the server (the column
is only set for `GRP_TXT`). If a packet ever does carry `channel_hash`,
that wins.
- The WS filter calls it: `if (filters.channel &&
!packetMatchesChannel(p, filters.channel)) return false;`

Checked the derivation against a production DB (live.saarmesh.de): it
equals `channel_hash` for all 3,193 `GRP_TXT` rows of the last two days,
0 mismatches.

## Performance

WS hot path, but the new check runs only while a channel filter is
active, and for non-`GRP_TXT` packets it returns before any parsing. For
`GRP_TXT` it is one `JSON.parse` of the already-received `decoded_json`
string, O(1) per packet. WS messages are already debounced/coalesced by
`debouncedOnWS`, so at typical rates (a few packets per second) this is
negligible. No new state, no allocations beyond the parse, no API calls.

## Tests

New `tests/unit/test-packets-live-channel-filter.js` (registered in
`test-all.sh`), 10 cases:

- no filter accepts everything
- decrypted name match / mismatch
- non-`GRP_TXT` rejected while a channel filter is active
- `enc_<HH>` match incl. hex case normalisation and `enc_00`
- `channel_hash` passthrough takes precedence
- already-parsed `decoded_json` object
- malformed / missing `decoded_json` doesn't throw and doesn't match
- source check that the WS filter block actually applies the helper

Without the `packets.js` change the new test fails; with it all pass.
Existing `test-packets.js` (117), `test-packets-local-channels.js` (14),
`test-packet-filter.js` (99) and `test-frontend-helpers.js` still pass.

## Browser validation

The same change is deployed on live.saarmesh.de (hot-patched
`packets.js`, server restarted so `__BUST__` refreshed; the served file
contains the fix). I have not taken a screenshot of the filtered live
table myself, so treat the in-browser check as pending until a reviewer
or operator confirms.

No config or customizer implications.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01HJcbiRjQC5cSmydVFXbPbR

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-10-12 15:25:59 +02:00
efitenandClaude Opus 5.5 60af9bd469 fix(hops): narrow only the directly heard hop to the observer region (#2098) (#2188) 
Fixes #2098.

## Problem

`HopResolver.resolve()` narrowed every hop of a path to the observer's
IATA region (300 km around the code's coordinates). Only the last hop
was heard by the observer directly; every earlier hop can be anywhere
the mesh reaches, so the region says nothing about it.

This stayed invisible on deployments whose observer codes were missing
from `/api/iata-coords`: there `nodeInRegion()` returns `null` and
nothing is filtered. A Belgian deployment's 42 observers use 10 codes
and none was in the table. Filling the table alone would have switched
on a filter that is wrong for every hop but the last.

## Why only the last hop

#2098 lists three options and argues against simply repopulating the
table, because the filter it would switch on is not sound. This PR does
fill the table, but only after limiting the filter to the one hop where
it is sound:

- In a MeshCore path the last hop is the repeater the observer received
the packet from, so it is within radio range of the observer. Measured
below: p99 30 km for directly heard multi-byte hops.
- An earlier hop says nothing about the observer's position. On the
Belgian deployment nearly every observer hears nearly every flood, so a
region filter on earlier hops would drop the right candidate whenever
the packet started outside the observer's region.

This was decided while preparing the PR, not on the issue thread, so the
reasoning is recorded here.

## Change

- **`public/hop-resolver.js`:** `resolve()` takes an optional
`opts.directHops`, the hops the observer heard directly. Without it, the
input is one path and its last hop is the direct one. The region check
runs only on direct hops. An earlier hop gets `regional: false` and,
with an observer, `regionSkipped: true`. `globalFallback` now means
checked against the region with no candidate in it, so it is only set on
a direct hop.
- **`public/hop-display.js`:** the badge counts every candidate for
`globalFallback` or `regionSkipped`. The dashed underline and the
popover's "(global fallback)" stay for `globalFallback` only, so an
earlier hop is not labelled as matched outside the region when no region
check ran on it.
- **`public/packets.js`:** `resolveHopsForPackets()` resolves the last
hop of each path apart from the earlier hops. `hopCacheKey(h,
observerId, direct)` keeps the two in separate cache entries, because
the same prefix can be the last hop in one packet and a relayed hop in
another. `renderPath()` reads the direct entry for the path's last hop,
and the detail-pane resolve writes it. The region-narrowed entry never
goes to the bare (observer-less) key, in the list or in the detail pane.
The detail pane resolves and caches the shown observation's path with
that observation's observer (`effectivePkt.observer_id`), the observer
`renderPath` already used; before, it resolved with the packet row's
observer, so when the first observation came from another observer the
last hop missed its direct entry and showed unfiltered. The route button
uses the same observer.
- **`cmd/server/routes.go`:** `iataCoords` gains 104 entries from
OurAirports `airports.csv` (public domain): every IATA code in Belgium,
the Netherlands and Luxembourg, and the airports with scheduled service
in Germany and France. AMS, CDG, FRA and MUC were already present and
keep their values.

## Measured before building

On 5000 packets from the Belgian deployment (3661 with a path):

- 1-byte last hops: 1212, all ambiguous, 1204 of them without a server
`resolved_path`, so they go through the client resolver.
- Distance from the observer to a directly heard node, on 1892 unique
multi-byte last hops: p95 25 km, p99 30 km, max 82 km.
- With this change, on 906 1-byte last hops from observers whose code is
now in the table: the badge counts 6.1 candidates on average instead of
6.6. The displayed name changes for 52, all from observers without their
own lat/lon, where distance to the region now orders the candidates
instead of list order. For observers with a position, no displayed name
changes, because the #2097 observer anchor already picked the nearest
candidate.

## Tests

- `tests/unit/test-issue-2098-last-hop-region.js` (new, in
`test-all.sh`, 25 assertions):
- HopResolver: the last hop of a path is narrowed and an earlier hop is
not; an earlier hop keeps every candidate and reports `regionSkipped`,
not `globalFallback`; a direct hop with no candidate in region reports
`globalFallback`; `directHops` overrides the position; a one-hop path is
narrowed.
- HopDisplay (real file): an earlier hop keeps a badge with every
candidate but no fallback class or popover text; a direct hop outside
the region keeps both.
- packets.js (real file, real resolver and display): with one observer,
a prefix that is the last hop of one packet and an earlier hop of
another renders narrowed in the first and with a badge in the second;
after the detail pane resolves a path, the observer-less render does not
show the narrowed last hop; when the shown observation comes from
another observer than the packet row, its last hop is narrowed to the
region of the observer that heard it.
- `TestIataCoordsIncludesBelgianObserverCodes`: ANR, BRU, GNE, KJK, LGG,
OBL, OST and MST with their OurAirports coordinates.
- Red commits first (`26000a22`, `bb91ff9d`, `189f55d7`, failing on
assertions), then green. Mutation check on the packets.js tests:
inverting `isDirect` fails 3 assertions, swapping the `directHops`
argument fails 4.
- `test-issue-2097-hop-ambiguity-badge.js`: two structural regexes now
allow the new `direct` argument; every assertion still holds.
- All 198 `node tests/unit/*` suites in `test-all.sh` pass. `cd
cmd/server && go test ./...` passes (with
`TestSaveGeoFilterPreservesFileMode` skipped, a Windows-only failure on
master too); `go vet` clean.

## Browser validation

Staging (this branch on top of the deployment's current build), headless
Chrome on `#/packets/<hash>` for packets whose path ends in a 1-byte
hop: the last hop's popover lists only candidates within the region with
their distances (for example `7E`: 3 candidates at 46, 207 and 296 km),
earlier hops list every candidate (`1D`: 13). The same prefix `7E`
showed 3 in region as the last hop of one path and 4 as a relayed hop of
another. No page errors. `/api/iata-coords` returns 158 codes.

## Browser validation of the follow-up

Staging again (this branch at `4e129b49` on top of the deployment's
build), headless Chrome:

- Packets list, 41 paths of 2+ hops: the global-fallback class appears
on 2 last hops and on 0 earlier hops.
- Detail pane, 5 packets whose path ends in a 1-byte hop (1 to 13 hops
shown): no earlier hop carries the class or "(global fallback)" in its
popover, and every ambiguous earlier hop keeps its badge. Every last hop
was checked against the region: 4 show "N candidates in region", 1 shows
"(global fallback)" with the class.
- Before the observer fix, 2 of 5 detail panes showed a last hop with
neither label, because the shown observation came from another observer
than the packet row (`b66a8f4c`: row from one observer with 13 hops,
shown observation from another with 9).
- No page errors.

## Not done

- The 300 km radius is unchanged. The measured direct-hearing distances
suggest it is wide, but a smaller radius would also have to hold for
less dense networks, and I have no data for those.
- `live.js` passes no observer, so the region filter does not run there,
before or after this change. `map.js` passes the observer of the chosen
path in path order, so once the resolver knows the observer regions it
narrows that path's last hop only.
- The "Cross-observer aggregate" line in the detail pane renders the
longest path with the packet's own observer, so its last hop is narrowed
to that observer's region even if another observer reported that path.
The per-observer keying there predates this PR (#2097).
- Two of the Belgian deployment's codes, HEP and NRW, are not airport
codes and are not in OurAirports; those observers stay without a region
filter.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 15:24:13 +02:00
Alex BandClaude Opus 5.5 955386b63f test(map): settle the router's focus frame before the teardown snapshot (#2179) 
## Summary

Fixes an intermittent failure in
`tests/e2e/test-map-nodes-pagination-e2e.js`, the "late response after
map teardown" cases added for #2030 in #2048:

```
✗ late /observers response after map teardown: old response must not change the destination page
```

It failed on #2164 ([run
38080812509](https://github.com/Kpa-clawbot/CoreScope/actions/runs/38080812509),
shard 2/3), a Go-only change, and passed on the rerun.

## Cause

`checkMapTeardown` compares `#app`'s HTML before and after releasing the
old map's pending response. When it failed, the only difference was on
the destination page's heading:

```
before: <div class="tools-landing"><h2>Tools</h2>…
after:  <div class="tools-landing"><h2 tabindex="-1">Tools</h2>…
```

That attribute comes from the router, not the old response. Since #630
(`app.js`, "SPA focus management") it focuses the new page's first
heading in a `requestAnimationFrame` after the page renders. The
before-snapshot was taken as soon as `#leaflet-map` detached, sometimes
ahead of that frame.

## Fix

The test now waits two frames before the before-snapshot, the same
settle it already does before the after-snapshot. The router's frame is
queued by then, so it always lands first. The change is test-only: no
app code changes, and every assertion stays as it was.

## Tests

Run locally with the e2e fixture (`tools/freshen-fixture.sh`, migrated)
served by this branch's server, headless Chromium:

| | runs | failed |
|---|---|---|
| test on `master` (`352a0ccd`) | 20 | 2 |
| this branch | 50 | 0 |

- **Before:** the two failures on `master` were "old response must not
change the destination page" (teardown, `/config/regions` and `nodes`),
plus one 30 s timeout in an "after map replacement" case in the same
runs.
- **After:** none of these failures or the timeout came back in 50 runs.

## Checks

- **Lint:** ESLint passes with the repo config and with Node globals.
SonarQube rules on the file report nothing on the added lines; its four
existing minor findings are on lines this PR doesn't touch.
- **Repo rules:** one logical change, explicit `git add`, no private
data, no new dependencies. This is a test-only fix, so there's no
red/green pair: the "red" is the intermittent failure above, measured as
2 in 20.

Related: #2030, #2048 (where the test was added).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-11 13:28:25 +02:00
Alex BandClaude Opus 5.5 a4ee3a7328 feat(scope-audit): observer region filter for both views (#2171) 
## Summary

Builds on the transport view from #2167 and #2169 (both merged); rebased
onto `master` (`352a0ccd`), so the diff is this PR's six commits only.

The Scope Audit was the one analysis page without the region selector.
This adds the shared region filter to it, for both the declared and the
transport view. `?region=SFO,SJC` means what it means on every other
endpoint: count only forwarding **heard by observers in those regions**,
not repeaters located there.

- **API** (`GET /api/scope-audit`, both modes):
- The region codes are resolved to their observers first (a handful of
rows), then the audit's hop scan gets `AND o.observer_idx IN (…)` on v3,
or `o.observer_id IN (…)` on v2. The 3.5M-row hop scan never joins
`observers`.
- `All` means no filter (shared `normalizeRegionCodes`). Codes with no
observer match nothing, instead of silently widening to every observer.
- Responses echo the normalised codes as `region`. The declared and
transport caches and their singleflight keys now include the region, so
each region set is cached separately with the existing TTLs.
- Declared-region verification (`regionEvidence`) is deliberately left
unfiltered: it corroborates what a repeater declares from its own
traffic, which doesn't depend on where a hop was heard.
- **Page:** the shared `RegionFilter` sits with the view and window
controls; a change reloads the view, and the selection persists like
every other page. The transport view's text box is relabelled **Scope**
(it already sends `?scope=` since the rename in #2167), so the page no
longer has two controls called Region.
- **Docs:** `docs/api-spec.md` (parameter, both response shapes) and the
OpenAPI parameters for the route (`mode`, `scope` and `region` were all
missing).

Partial fix for #2142 (its region filter for rollout tracking), together
with #2167 and #2169.

## Performance

The unfiltered path must not get slower: it's the same query with an
empty clause. `BenchmarkComputeScopeTransport` (from #2167, 1,200
repeaters), the PR it stacks on vs this branch, `-count 8`:

| | before | after | |
|---|---|---|---|
| sec/op | 220.3 ms | 218.0 ms | ~ (p=0.065) |
| B/op | 43.77 MiB | 43.70 MiB | ~ |
| allocs/op | 833.5k | 833.5k | ~ |

**Filtered requests, measured on staging** (a copy of a production
database): the first version drove `o.observer_idx IN (…)` from
`idx_observations_observer_idx`, reading every observation the region's
observers ever made. Fixed in `80414242` with a unary `+` (the same
treatment as #2166), so the scan starts from the `first_seen` window and
filters per row. Red test `67e063b1` asserts the query plan against a
fixture with the real indexes.

| scan, one region over 24h (13 observers) | plan | time | hop rows |
|---|---|---|---|
| `o.observer_idx IN (…)` | observer index | 13.44 s | 102,114 |
| `+o.observer_idx IN (…)` | `first_seen` window | **0.25 s** | 102,114
|
| unfiltered | `first_seen` window | 0.35 s | 225,983 |

Cold end-to-end requests on staging after the fix: 24h declared,
unfiltered 0.84 s · one region 0.48 s (was 16.7 s) · two regions 0.22 s
· transport, one region 0.42 s · transport, six regions 0.75 s · 7d, one
region 3.46 s. 24h transport, live data: 383 repeaters unfiltered, 204
heard by one country's observers.

## Tests

- Red `629e4f36` (Go): extends the audit fixture with an `observers`
table and `observations.observer_idx`. Covers the declared view ignoring
a scope heard only by another region's observer, the transport view
listing only repeaters heard in the region, `region=All`, per-region
caching, a region with no observers, and the echoed codes. Five fail on
assertions; `region=All` passes because the stub ignores the filter,
which is the behaviour it pins.
- Green `6f1b1e1c`: the implementation, plus a v2-schema case
(`observer_id`), so both branches of the clause are covered.
- Red `b4933d68` / green `08ecc172` (frontend): `apiPath` carries the
selection in both views and adds nothing when none is selected. The
region control on the page is a net-new UI surface.
- `go test` (cmd/server) passes. Frontend suites: 193 of 194. The one
failure, `test-issue-1956-release-routing.js`, needs CI's
`GITHUB_REF_NAME` and fails the same way on `master` locally.

## Screenshot

The same transport view with the shared region filter set to **SJC**:
only forwarding heard by San Jose observers counts, so 7 repeaters are
listed instead of 44, with their carried scopes counted from those
observers alone. Taken against `test-fixtures/e2e-fixture.db` with
fictional scopes.

![Scope Audit with the observer region filter on
SJC](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/scope-audit-region-filter.png)

## Browser validation

The branch's server was run locally with 14 configured regions (dropdown
layout), checked in headless Chromium at 1280, 900, 700 and 390 px
(touch):

| check | 1280 | 900 | 700 | 390 touch |
|---|---|---|---|---|
| ticking SJC re-requests with `&region=SJC` | yes | yes | yes | yes |
| menu fully on screen | yes | yes | yes | yes |
| page wider than viewport | no | no | no | no |
| page errors | none | none | none | none |

Found and fixed this way: the filter is the right-most control on
desktop, and the menu (which opens rightwards) ran 66 px off the page.
It now opens leftwards above 640 px. On a phone the controls wrap, the
filter starts a line, and the default applies.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **S3776 on `ScopeAuditForwardingHeardIn`:** complexity 55. The
function it generalises, `ScopeAuditForwarding`, is 54 on `master`; the
region clause adds one point.
- **S3776 on `computeScopeAudit`:** complexity 65, unchanged from
`master`.
- Both pre-date this PR. Splitting them is better as a refactor PR of
its own than mixed into a feature.
- The page code's findings were fixed in the transport-view PR it stacks
on.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-11 11:17:12 +02:00
efitenandClaude Opus 5.5 55dd28164d test(e2e): assert the configured-scope row after the theme-refresh re-render (#2180) 
## What

`test-issue-1648-m2-icons-e2e.js` now asserts the configured-scope row
only after the page's `theme-refresh` re-render has fired and the detail
has rendered again.

## Why

On 2026-10-10 this assertion failed CI on four PRs (#2169, #2171, #2154,
#2164), each time at the first live-map case: `confirmed scope row must
be visible, including an empty scope`. A re-run of the failed jobs
passed every time.

The code path:
- `customize-v2.js:814` dispatches `theme-changed` once
`/api/config/theme` resolves, on every page load.
- `app.js:1309-1315` turns that into `theme-refresh` 300 ms later.
- `live.js:4713` re-runs `showNodeDetail(activeNodeDetailKey)` on
`theme-refresh`, and `showNodeDetail` first sets `#nodeDetailContent` to
"Loading…" (`live.js:2598`). This is the only caller besides the marker
click (`live.js:3090`).
- The test waited for `#nodeDetailContent table` and then called
`row.isVisible()`, which does not wait. A refresh landing between the
two fails the check.

The same race could let the null-scope case pass on the "Loading…"
panel, because `row.count() === 0` holds there too.

The fix uses the same `addInitScript` flag pattern as
`test-e2e-playwright.js:83`.

## Testing

- Locally against a server on the E2E fixture: 5 of 5 runs pass, 27 of
27 checks each.
- **Not reproduced locally.** The unchanged test also passed every local
run: 5 plain, 3 with `/api/config/theme` delayed by 1 s, 5 with 8x CPU
throttling. So the reasoning above comes from the code path and the four
CI failures, not from a local red-then-green.
- Test-only change, no product code touched.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-11 10:58:20 +02:00
Alex BandClaude Opus 5.5 352a0ccd19 feat(region-filter): configurable region quick picks (#2170) 
## Summary

**Independent of every other open PR.** Choosing a pick saves the
selection, redraws the control and notifies the page's listeners itself,
exactly as a click in the control does, so it re-queries straight away
whether or not #2168 (`setSelected` notifies) is merged; with both in,
nothing fires twice.

Region quick picks are named groups of region (IATA) codes, offered as
one-tap choices inside the region filter, e.g. "Northern California" for
a dozen airport codes instead of ticking each box. One deployment has
been injecting these into the page through an nginx `sub_filter` and a
script that wraps `window.RegionFilter`; this makes them a configured
feature.

```jsonc
"regionQuickPicks": [
  { "name": "Northern California", "description": "Bay Area and Sacramento observers.", "regions": ["SJC", "SFO", "OAK", "SMF"] }
]
```

- **Server:** `GET /api/config/region-quick-picks` returns the groups
cleaned up: names and codes trimmed, codes upper-cased and de-duplicated
in their configured order, groups without a name or codes dropped, at
most 20 groups of 200 codes, always a list. `/api/config/regions` is
unchanged.
- **Region filter:** loads the groups with the regions and offers every
group with at least one observer today, inside the region control:
- dropdown layout: rows of the same kind as **All** (checkbox and bold
name), directly under All with a divider before the single regions; the
row is ticked while that pick is the selection, ticking it selects the
pick and closes the menu, unticking goes back to all;
- pill layout: the picks lead the same bar, set off from the single
regions by a thin divider.
- A tap selects the group's codes that have an observer and re-queries
straight away.
- The selector then reads the group's name ("South Bay ▾") instead of "2
Regions", and the button shows as pressed. Selecting the same codes by
hand does the same.
  - Tapping the active group again goes back to all regions.
- A failed fetch, or no configured groups, leaves the filter exactly as
before.
- **Mobile:** the rows take the menu's 44 px row height under `(pointer:
coarse)`; the pill-bar picks reuse `.region-pill` and its touch target.
Inside `.filter-bar` (Packets) the existing touch rule for inputs made
every region checkbox a 44 px square; the checkbox is now 20 px and the
44 px row is the tap target.
- **Semantics:** like every region filter, a group means "heard by
observers in these regions", not "located there"; `config.example.json`
says so.

No existing issue covers this.

## Screenshot

Nodes page with two picks configured (example config below), after
tapping **South Bay**: the pick and its codes (SJC, MRY) are selected,
and the list re-queried for them. Taken against
`test-fixtures/e2e-fixture.db`.

![Region quick picks on the Nodes
page](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/region-quick-picks.png)

## Config example

```json
"regionQuickPicks": [
  { "name": "South Bay", "description": "San Jose and Monterey observers", "regions": ["SJC", "MRY"] },
  { "name": "East and North Bay", "description": "Oakland and San Francisco observers", "regions": ["OAK", "SFO"] }
]
```

Each pick needs a `name` and at least one code; `description` is
optional (shown as the tooltip). Codes are trimmed, upper-cased and
de-duplicated; a pick with no code that has an observer today is not
shown. At most 20 picks of 200 codes.

## Browser validation

The branch's server was run locally against a migrated copy of
`test-fixtures/e2e-fixture.db` (observers in MRY, OAK, SFO, SJC) with
three groups configured: "South Bay" (SJC, MRY, SCZ), "East and North
Bay" (OAK, SFO, sts) and "Nowhere yet" (XXX). Checked in headless
Chromium:

| check | desktop 1280 px | phone 390 px (touch) |
|---|---|---|
| groups shown | South Bay, East and North Bay ("Nowhere yet" hidden) |
same |
| tap South Bay selects | SJC, MRY (SCZ has no observer) | same |
| next `/api/packets` request | `region=SJC,MRY` | `region=SJC,MRY` |
| selector label | "South Bay ▾" | "South Bay ▾" |
| second tap | back to all regions | same |
| button height | 24 px | 44 px |
| page wider than viewport | no | no |
| page errors | none | none |

Checked on Packets (dropdown layout) and Nodes (pill layout). Three
layout problems found this way on the phone were fixed before opening: a
sideways-scrolling row clipped at the screen edge, the selector squeezed
into a narrow column (the filter group is a non-wrapping flex row), and
the two wrapped rows spread apart on Nodes (`align-content`).

**Revision 1:** the first version put the picks in a separate row beside
the selector; they moved inside the dropdown. Re-checked with a
14-region list and three picks on Packets: desktop 1280 px dark and
phone 390 px light both keep the menu on screen (menu 340 px wide, page
not wider than the viewport), the pick closes the menu and the trigger
names the pick.

**Revision 2:** picks are checkbox rows like **All** instead of chips.
Re-checked on Packets at 1280 px dark and 390 px light: rows render bold
under All, the divider separates the single regions, ticking a pick
ticks its codes and names the trigger, no sideways scroll.

Unrelated and not changed: on Nodes at 390 px the region row (existing
pills included) starts flush with the screen edge.

## Tests

- Red commit `81ea67d6`: the config type, a stub normaliser and endpoint
returning nothing, and tests.
- Go: cleaning (trim, upper-case, de-duplicate, drop incomplete groups),
caps, nil config, and the endpoint's JSON.
- Frontend, `tests/e2e/test-region-quick-picks.js` (moved from
`tests/unit/` in `fced69a7`; it needs jsdom, so it runs in the E2E job
like `test-table-sort.js`): the real `region-filter.js` in **jsdom**
against faked config endpoints. Nine cases: rendering and hiding groups
with no observer, selecting and re-querying, the selector label,
toggling back, a hand-made selection marked active, the pill layout, no
groups, a failed fetch, HTML escaping. Seven fail on the stub, all on
assertions.
- Green commit `0e33210e`: the implementation, styles,
`config.example.json` (`regionQuickPicks` +
`_comment_regionQuickPicks`), `docs/api-spec.md`, and the OpenAPI route
list (`TestOpenAPICompleteness` requires it). The jsdom test also had a
cross-realm comparison bug fixed in this commit (`Array.from` on arrays
coming out of the jsdom window).
- `make test`: 16 of 16 modules pass. Frontend suites: 195 of 196; the
one failure, `test-issue-1956-release-routing.js`, needs CI's
`GITHUB_REF_NAME` and fails the same way on `master` locally.

- Red commit `f991060d` (revision): picks live inside the dropdown menu
before the checkboxes, choosing one closes the menu, and in the pill
layout they share the region bar; two fail on assertions against the old
layout. Green commit `1f720e35`. Frontend 195 of 196 (same
environment-only failure).

- Red `134f5256` / green `da0adb50` (second revision): picks are
checkbox rows between All and the single regions with a divider, no
chips in the menu; ticking and unticking a pick row. The green commit
also repairs `style.css`: `1f720e35` had prepended its block to the top
of the file (its edit matched an earlier `.region-pill {`) and left the
previous rules in place. The file now differs from `master` by 13 lines.

## Review notes

- Groups are offered only when at least one of their codes appears in
`/api/config/regions`, and only those codes are selected. The stored
selection is pruned to known codes on load anyway, so selecting absent
codes would not survive a reload.
- Customizer (AGENTS.md rule 8): groups are operator configuration; a
customizer editor could come later.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `b84ace4e`:
- `normalizeQuickPickCodes` split out of the config normaliser (S3776,
17 > 15);
  - a comment on why a failed quick-pick fetch is ignored (S2486);
  - `for…of` in `activePick` (S4138);
- in the tests: `node:` specifiers, `includes()`, and an explicit string
comparator for sorted selections (S7772, S7765, S2871).
- **Kept on purpose:**
- S2486 is still reported on that `catch`. Swallowing the error is the
intended behaviour: the filter works as before, just without picks.
  - S1523 on the `vm` test loader, as above.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 21:23:05 +02:00
Alex BandClaude Opus 5.5 99efaed33d fix(region-filter): setSelected notifies change listeners on a real change (#2168) 
## Summary

`RegionFilter.setSelected()` saved the selection and redrew the control
but never ran the `onChange` listeners; only a click (`toggleRegion`)
did. A programmatic selection therefore updated the dropdown while the
page kept showing data for the old selection until the next click.

Affected today: the Packets page applies a `?region=` URL parameter
through `setSelected`. It registers its listener afterwards, so its
first load is fine, but any other page's listener that is still
registered sees nothing. And anything that offers one-click region
groups has to wrap `setSelected` to make it re-query (an nginx-injected
overlay on one deployment does exactly that). This PR stands alone;
#2170 (configurable quick picks) does not need it.

`setSelected` now compares the new selection with the current one
(order-insensitive; empty and null both mean "all") and, only when they
differ, runs the listeners after saving and re-rendering, exactly as a
click does. Re-applying the current selection stays silent, so a caller
restoring state does not trigger a reload.

No existing issue covers this.

## Tests

- Red commit `75569b78`: `tests/unit/test-region-filter-setselected.js`
loads the real `public/region-filter.js` in a `vm` sandbox. Two cases
fail on `master` (a programmatic selection notifies once with the new
codes; a clear notifies once with `null`). Two pass on both commits on
purpose: re-applying the current selection in another order, and
clearing an empty selection, must not notify.
- Green commit `8a7ddfe1`: the fix.
- Listed in `test-all.sh` in sorted order; `test-test-inventory.js`
passes. `test-1108-region-hide-nodes.js` (the other region-filter suite)
passes. Frontend suites: 194 of 195; the one failure,
`test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and
fails the same way on `master` locally.

## Review notes

- `packets.js` "clear all filters" calls `setSelected([])` and then
reloads itself. When a region was selected, that path now also triggers
the page's region listener, so the packets list is requested twice. It
is a cosmetic double fetch on an explicit user action; left as is to
keep this PR to the component.
- No performance claim; this is a correctness fix.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `9e732706`: `node:` specifiers (S7772).
- **Kept on purpose:** S1523 on `vm.runInContext`, which loads the
repo's own `public/region-filter.js` as the other frontend unit tests
do.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 21:22:54 +02:00
Alex BandClaude Opus 5.5 deb55dc82e feat(scope-audit): transport view on the Scope Audit page (#2142, page) (#2169) 
> **Requires #2167 to be merged first.** This page calls the transport
view API that #2167 adds (`/api/scope-audit?mode=transport`), so it
cannot be split from it; until #2167 merges this PR also shows its
commits.

## Summary

Page half of #2142. **Stacked on #2167** (the API); review that first.
#2171 adds the observer region filter on top. Partial fix for #2142
until all three are in.

Adds a **Declared / Transport** toggle to `#/scope-audit`. The transport
view calls `?mode=transport` and lists every repeater seen forwarding in
the window:

- **Seen carrying:** one chip per region scope with its packet count
(first/last seen in the tooltip);
- **Declared:** the repeater's declared regions coloured by whether they
were seen carried (the declared view's own chips), or *not asked* when
no declared-regions answer exists. Never asked is worded differently
from declaring nothing;
- **Other traffic:** unscoped, unmatched and ambiguous-hop counts.

A **Region** box (suggestions from the scopes seen) adds a *Carries X*
column, lists the repeaters not yet carrying it first, and summarises
carriers against active non-carriers: the rollout report the issue asks
for. Window, view and region are deep-linked
(`#/scope-audit?window=7d&mode=transport&scope=be`) and survive a
reload. The page says plainly that *transported* means seen carrying,
not configured, and that over a short window a quiet region looks the
same as a missing one. The intro text changes with the view.

## Screenshot

The transport view with scope `be` typed in: every repeater seen
forwarding in the last 24 h, whether it carries `be` (non-carriers
first), the scopes it was seen carrying with counts, and its declared
regions or *not asked*. Taken against `test-fixtures/e2e-fixture.db`
with fictional `be` / `be-van` / `nl` scopes.

![Scope Audit transport
view](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/scope-audit-transport-view.png)

## Browser validation (staging, real data)

Headless Chromium against a staging copy of a production database with
its live feed:

- `#/scope-audit?window=24h`: the declared view, 16 rows, as before.
- Clicking **Transport**: 385 rows; the hash gains `mode=transport`.
- Typing a scope as `#XX-YY` in capitals: normalised to `xx-yy`; the
hash gains it (now `scope=xx-yy`); summary "385 repeaters seen
forwarding in the last 24h · 16 with a declared-regions answer · 51
carried xx-yy · 334 active but not seen carrying it"; the first rows are
the non-carriers.
- Reloading that hash restores view and region. Switching back to
Declared hides the Region box and restores the declared intro. No page
errors.
- Checked at 1280 px and 390 px wide.

Two layout bugs found this way were fixed before opening: the region box
inherited `width: 100%` from `.nodes-search`, and `display: flex`
overrode the `hidden` attribute on the region bar.

## Tests

`tests/unit/test-frontend-helpers.js` gains 10 cases on rendered markup
and state, using the page's exported internals like the existing
scope-audit tests: hash and API paths (and that a region never leaks
into the declared view's link), region normalisation (`#`, capitals,
empty), *not asked* vs declared chips (with the `*` wildcard),
carried-scope chips with counts and the filtered region marked, the
carries column only with a region, HTML escaping of names, the summary
counts, region suggestions, and the per-view intro. 768 passing in that
file.

Net-new UI surface, so per TDD.md the tests land in this PR rather than
as a preceding red commit. Frontend suites on this branch: 193 of 194;
the one failure, `test-issue-1956-release-routing.js`, needs CI's
`GITHUB_REF_NAME` and fails identically on `master` locally (#2149 and
#2160 address that).

## Review notes

- Colours only through existing CSS variables (no hex literals); the
filtered region is marked with the link colour, not a new status colour.
- The window note's wording is in the same spirit as the declared view's
`windowHonestyNote`.
- No API calls per row: one request per view/window/region, cached
client-side for 30 s like the declared view.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `df53e770`:
- one `modeQuery` helper for the page hash and the API path (S3358; the
two copies were identical);
  - the carries-scope cell built before the row (S3358);
- `localeCompare` for the region suggestions (S2871), `for…of` and
`.dataset` (S4138, S7761), optional chaining (S6582);
  - object spread in the helper tests (S6661).
- Optional chaining and spread are already used throughout `public/`. No
behaviour change.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 21:22:37 +02:00
Alex BandClaude Opus 5.5 323b8040d4 perf(frontend): throttle nav stats to one refresh per 15 s, none in hidden tabs (#2162) 
## Summary

The nav bar's packet / node / observer counts were refreshed three ways
at once: a 15 s timer, every batch of live WebSocket packets
(`debouncedOnWS`, 250 ms), and the `/stats` client cache being cleared
every 5 s while packets arrive. None of them checked whether the tab was
visible. So every open tab, including hidden ones, asked `/api/stats`
every few seconds for as long as it stayed open.

On one production instance's nginx log for a single day, `/api/stats`
was **135,654 of 187,411** CoreScope API requests (72%), about 20,000 a
day for each browser that kept a tab open. Each response is cheap thanks
to the 10 s server-side cache (#1910), so this is about request volume,
bandwidth and log noise, not server CPU.

This PR adds `createVisibleThrottle()` to `app.js` and routes all three
triggers through one throttled refresh:

- at most one `/api/stats` request per 15 s from a visible tab (the old
timer's own period);
- none from a hidden tab;
- one immediate refresh when a hidden tab becomes visible again.

No existing issue covers this.

## Performance

Measured with headless Chromium, counting `/api/stats` requests from a
real tab: 60 s visible, then 60 s with the page hidden
(`document.hidden` and a `visibilitychange` event), then back to
visible.

| `/api/stats` requests | production (current release) | staging,
`master` | staging, this PR |
|---|---|---|---|
| visible tab, per minute | 9 | 4 | 4 (at +10, +25, +40, +55 s) |
| hidden tab, per minute | 6 | 5 | **0** |
| on return to visible | n/a | n/a | 1, immediately |

The same check confirmed the nav bar still renders its counts (`… pkts ·
… nodes · … obs`).

## Tests

- Red commit `2c81a3e4`: `tests/unit/test-nav-stats-throttle.js` loads
the real `public/app.js` in a `vm` context. `createVisibleThrottle`
lands as a pass-through stub, so the test fails on its first assertion
(40 refreshes where 1 is expected).
- Green commit `f0ba92ab`: the throttle, and the nav stats timer, WS
hook and a new `visibilitychange` listener calling one throttled
trigger. All 6 cases pass: one refresh per burst, again after the
interval, at most 241 an hour, none while hidden, one on return, and a
source check that `updateNavStats` is wrapped and not called by the
timer directly.
- Added to `test-all.sh` in sorted order; `test-test-inventory.js`
passes.
- Frontend suites: 194 of 195 pass. The one failure,
`test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and
fails the same way on `master` locally.

## Review notes

- `home.js` keeps its own one-shot `/stats` load on the home page; only
the nav bar's polling changed.
- A tab opened in the background loads its counts when it is first
shown.
- The 15 s interval is a constant at the call site. Could be exposed in
the customizer later (AGENTS.md rule 8); not added here.
- No Playwright E2E added: the change is timing behaviour, covered by
the unit test against the real file and by the browser measurement
above. Happy to add an E2E if wanted.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `9eb94ea1`: `node:` specifiers (S7772).
- **Kept on purpose:**
- S7773 wants `Number.isFinite`/`Number.parseInt` for the `isFinite` and
`parseInt` handed to the `vm` sandbox, but `app.js` calls the global
functions, so those are the ones it needs.
- S1523 flags `vm.runInContext`, which is how the frontend unit tests
load the repo's own `public/app.js`.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 21:21:30 +02:00
n30nex a5b60ca594 test: make local validation independent of runner environment (#2160) 
Fixes #2159. Related to #1995.

Standalone release tests failed when `GITHUB_REF_NAME` was absent and
could use an unrelated runner tag when it was present. Supply the
simulated tag explicitly; exercise the real release-notes step with both
missing and unrelated host values, including present and absent notes
files.

Two filesystem fixtures now report unsupported Windows capabilities
explicitly: POSIX permission bits, and symlink creation denied with
error 1314. Unix mode checks and all symlink safety assertions remain
intact; other symlink setup errors still fail. Production code and
workflows are unchanged. This is 29 added lines across three test files.

Validation: reproduced all three failures before fixing them; focused
regressions, the full standalone frontend suite, and native Go
server/ingestor suites pass. The POSIX permission and unprivileged
Windows symlink fixtures are explicitly skipped on this host and remain
enabled in Linux CI. The browser smoke suite passed 148 checks, with
three fixture-dependent skips and two pre-existing disabled cases. No
configuration or customizer changes.
 2026-10-10 21:20:40 +02:00