142 Commits
Author SHA1 Message Date
AdamandAdam c83499763d Allow mobile clients to manage repeater claims and details (#2429)
Draft for feedback: a working version of native repeater claims and
edits from mobile clients, built on the repeater info feature from
#2415. Four open questions are at the end.

## Situation

Repeater info (#2415) can only be claimed and edited from a browser
session. Device tokens from `POST /api/auth/device-token` are limited to
`rx` (CoreDrive RX: settings and companion linking), and `GET
/api/repeaters/{pubkey}/info` treats a bearer token as anonymous. A
mobile client can show a repeater's details but can't offer "claim" or
"edit" natively.

`PUT /api/repeaters/{pubkey}/info` is also a full replace with no
precondition. Two people editing from the same copy silently overwrite
each other. That's rare in the browser, but likelier from an app that
keeps a cached copy.

## Change

Three commits, one per piece.

**1. Device-token scopes for the repeater routes** (`8aba3715`)

- `POST /api/auth/device-token` takes an optional `scopes` list. Omitted
(or `null`) still means `["rx"]`, so CoreDrive RX and every existing
token keep exactly their current permissions. An empty list or an
unknown scope answers 400, checked before the password.
- New scopes:
- `repeater:claim`: `POST` and `DELETE /api/repeaters/{pubkey}/claim`.
These already only add or remove the caller's own claim. Several
claimers per repeater, idempotent `POST`, unchanged.
- `repeater:edit`: `PUT /api/repeaters/{pubkey}/info`, still subject to
`CanEditRepeater`.
- Every device token may use `/api/auth/me` and `/api/auth/logout`, so a
repeater-only token can check and end its own login.
- `GET /api/repeaters/{pubkey}/info` with a token holding either
repeater scope returns `canEdit` and `claimedByMe`. An unknown or
expired token answers 401 rather than passing as anonymous, so a client
learns its login is gone. A valid token without a repeater scope still
gets the public view. A cookie still wins over a bearer header, as
elsewhere.
- `GET /api/account/repeaters` (My repeaters, from #2430) also takes a
token holding either repeater scope, so an app gets the account's claims
in one request. It can't build that list from the public catalog, where
claimers are display names, and reading repeaters one at a time would
miss a claimed repeater that has left the node list. An `rx`-only token
still answers 403.
- Scope checks are method-aware (`bearerScopesFor(method, path)`). The
repeater paths are added to the bearer CORS paths.
- Same handlers, validation, rate limit (`repeater`, 120/h per user and
per IP), audit actions and admin claim removal as the browser. Token
writes add `via: device` to the `repeater.claim`, `repeater.edit` and
`repeater.unclaim` rows, and the `user.login` row records the granted
scopes. The admin route keeps refusing bearer tokens.

**2. Conditional edits with a revision** (`9f9bf47b`)

- `users.db` schema v10 adds `repeater_info.revision`. Existing rows
start at 1; a repeater with no saved details is at 0. Claims don't
change it.
- `GET` (single and list) and `PUT` responses include `revision`.
- `PUT` accepts an optional `baseRevision`. The check is part of the
write statement, so of two saves from the same base exactly one lands:
- `baseRevision: 0` (first save): `INSERT … ON CONFLICT(pubkey) DO
NOTHING`.
- `baseRevision: n`: `UPDATE … SET revision = revision + 1 … WHERE
pubkey = ? AND revision = ?`.
- No match answers `409` with the stored details
(`canEdit`/`claimedByMe` included) and writes nothing. That's the same
shape `PUT /api/account/settings` uses for its conflicts.
  - A stale base is refused even when the body would change nothing.
- Permission is checked first, so a 409 never shows details to someone
who can't edit.
- Without `baseRevision` the save is unconditional and bumps the
revision, so `public/repeater-info.js` works unchanged.
- `/api/config/client` adds `userManagement.repeaterTokens: true`, so a
client can tell this server from one without the scopes or
`baseRevision` (older servers answer the device-token request with the
SPA page, as noted for `companionLinking`).

**Discovery and conflict recovery, for a client:**

1. Check `userManagement.repeaterTokens` in `/api/config/client`.
Without it, keep the claim and edit actions hidden.
2. Log in with `scopes: ["repeater:claim", "repeater:edit"]` (or just
what the client offers). The response echoes the granted `scopes`.
3. Read `GET /api/repeaters/{pubkey}/info` with the token. Show
Claim/Unclaim from `claimedByMe` and Edit from `canEdit`, and keep
`revision`.
4. Save with `PUT …/info` and `baseRevision` set to that revision. On
200, keep the new `revision` from the response. On 409, the body is the
current record: show or merge it, then retry with its `revision`. On
401, log in again. On 403, the account can no longer edit (for example
someone claimed it), so refresh from `canEdit`.

**3. Account page device list** (`d28374aa`)

My account, Devices labelled every device token "CoreDrive RX".
`/api/account/sessions` now includes a device token's `scopes`. The page
keeps the CoreDrive RX label for `rx` tokens (and rows without scopes).
It shows other tokens under the device name the app sent, with a line
such as "can claim repeaters and edit repeater details".

Docs: `openapi.go`, `docs/api-spec.md` and the Repeater info section of
`docs/user-guide/accounts.md`.

## Questions

1. **One scope or two?** This uses separate `repeater:claim` and
`repeater:edit`, so a client can ask for less, and a token that only
edits can't put the user's name on a repeater publicly. One `repeater`
scope would be simpler if you'd rather not split it.
2. **Editing unclaimed repeaters from mobile.** This keeps web parity:
any active user may edit a repeater nobody has claimed, and once it has
claimers only they (and admins) can. The alternative is to require a
claim before a token may edit. That's stricter, but then app and browser
rules differ.
3. **Operator switch.** There's no new setting: the scopes are available
whenever `userManagement` is on, and the user opts in when the app asks
for them at login. Would you want an explicit opt-in such as
`userManagement.mobileRepeaterWrites`?
4. **Stale-edit contract.** Does `revision` plus `baseRevision` in the
body, with 409 returning the current record, fit? I matched account
settings rather than `ETag`/`If-Match`, to keep one pattern in the API.
The browser editor doesn't send `baseRevision` yet. Making it send one
would be a small follow-up, if you want browser saves protected too.

## Tests

- `cmd/server/repeater_device_token_test.go`:
- default and requested scopes, rejected scopes, and the method-aware
route table;
- per-scope allowed and denied routes, logout with a repeater-only
token;
  - claim and unclaim of own claim only, alongside another claimer;
- edit permission (unclaimed repeater, claimer, non-claimer, admin) and
a disabled account;
  - the admin route refusing tokens;
  - authenticated, public and expired-token reads;
- `via: device` audit rows, cookie writes still needing CSRF, and
session scopes.
- My repeaters with a claim or edit token (own claims only), and 403 for
an `rx`-only token.
- `cmd/server/repeater_revision_test.go`:
- revision on reads, a first edit with no saved row, and a stale base
returning 409 with the current record and no audit row;
  - unconditional browser saves, validation and permission order;
- two concurrent saves from the same revision (one 200, one 409), and
the client config flag.
- `internal/users/repeater_revision_test.go`:
  - revision counting and conditional first and later saves;
  - an 8-way race for both cases (`-race`);
  - the v9 to v10 upgrade.
- `tests/unit/test-user-management-ui.js`: app token labels and scope
line.

Run locally:

- `make test`: all 16 modules OK.
- `sh test-all.sh`: all suites pass, run in `node:24-bookworm`. On
macOS, `test-manage-workflow.js` fails 2 cases on unchanged master
because of the BSD tools.
- `npx eslint public/*.js`: 0 errors, the same 87 warnings as master.
- `node tests/e2e/test-lint-network-calls.js`
- `tests/e2e/test-user-management-e2e.js` against local e2etest servers
with the CI config: 44/44.
- Manually against a local instance:
  - in the browser: claim, edit, unclaim, and the Devices list;
  - with curl: the token flow, including a 409 after a browser save.

Not run: the full Playwright shard matrix from CI.

---------

Co-authored-by: Adam <196335+anieto@users.noreply.github.com>
2026-11-21 22:27:48 +01:00
ZachandClaude Opus 5.5 1a6322a707 feat(map): more tile providers (topo, satellite, gray, vector), hillshade overlay and custom tile servers (#2434)
Closes #2428.

Adds free and open map providers (topographic, satellite, gray and
vector basemaps), a hillshade overlay, and support for operators' own
tile servers. Everything is additive and opt-in: Carto stays the
default, every new provider group is off until the operator turns it on,
and no existing config key or provider was removed. An upgrade with an
untouched config changes nothing in the picker.

> **Dependency:** #2426 has merged. The branch is rebased onto current
`master` (`6c78a1d2`) with no conflicts.

## What's new

- **Provider registry** (`public/map-tile-providers.js`). New groups are
**opt-in**: each is off until `map.tiles.providers.<group>.enabled` is
`true` (the boolean), so licences and volunteer servers are the
operator's call:
- **EOX**: Sentinel-2 satellite (2025 and 2016), Terrain Light, and a
place-labels overlay. EOX 2025 imagery is CC BY-NC-SA 4.0
(non-commercial); a commercial deployment should leave `eox` off or
license it.
- **OSM France**: Humanitarian, CyclOSM, OSM Gray and OSM Gray Dark
(volunteer-run, fair use).
- **OpenFreeMap** and **VersaTiles** vector styles: Liberty, Bright,
Positron, Dark, Fiord, Colorful, Muted and the Gray variants.
- **Hillshade** (below) is opt-in the same way. **OpenTopoMap**, the OSM
community server, Stamen and USGS keep their existing opt-in.
- **Custom tile servers**: `map.tiles.custom` adds raster (XYZ), vector
(MapLibre style URL) and raster-overlay entries as `custom-<id>`. They
can be used in `darkDefault` / `lightDefault`. Invalid entries are
skipped with a console warning.
- **Hillshade overlay** (`public/map-hillshade.js`): shaded in the
browser from Terrarium elevation tiles.
`map.tiles.providers.hillshade.url` can point at a mirror.
- **Vector basemaps** (`public/map-vector.js`) use a vendored MapLibre
GL 5.24.0 (CSP build) and `@maplibre/maplibre-gl-leaflet` 0.1.4.
- They are lazy-loaded only when a vector map is picked (about 380 KB
gzipped).
  - The load times out after 15 s.
  - Without WebGL the map falls back to the theme's default raster map.
- **One picker on Map, Live and Repeater Planner**: Auto, Street,
Topographic, Satellite, plus an Overlays group.
- **A map picked there is saved** (`localStorage` `mc-tile-pinned`,
untrusted: honoured only while registered). Every picker reopens on it
after a reload or on another page, until the visitor picks **Auto**.
- The other maps (home mini-map, node detail, node reach, analytics)
follow the visitor's choice through the shared builder: the saved pick
when it suits the current theme (satellite always does), otherwise the
theme's map.
- Those small maps never load MapLibre, so a vector choice shows the
theme's default raster map there.
- **Security contract**: the `CONTRACT` block at the top of
`map-tile-providers.js` (S1 to S7) covers one shared tile-URL validator
(`https?://` or a same-origin path, no credentials in the authority,
whitelisted placeholders), attribution HTML-escaped once, picker labels
set with `textContent`, explicit MapLibre options, pinned vendored
versions, CORS-safe hillshade decoding, and treating stored picks as
untrusted input. `map.tiles` is served publicly in `/api/config/client`,
so keys put there are public (the docs warn about this).

## Fixes included

The race stress suite written for this PR found teardown races when
leaving map pages mid-load:
- a lazy MapLibre load finishing after navigation;
- a hillshade tile decoding after `remove()`;
- a provider-change event arriving during teardown.

It also found a pre-existing document-listener leak in `AreaFilter` /
`RegionFilter`. All of these are fixed in `e6709e74`, with regression
tests in `tests/e2e/test-map-teardown-races.js`.

The first CI run then caught a bug in that fix (`7f8d792c`).
`MC_safeRemoveMap` called `map.off()` before `map.remove()`, so the
map's `unload` event reached no one. Leaflet relies on that event to
detach controls and clean up the zoom-animation proxy and BoxZoom, and
`test-issue-1206` watches it to confirm node maps are disposed. `off()`
now runs after `remove()`, and a new regression test checks that
`unload` fires, controls detach and no listener is left.

A later stress run (a new seed) found a real two-tab bug, fixed in
`0506fc1d`. When another tab's overlay pick is already in localStorage
but its `storage` event hasn't arrived, a layer-picker click here
triggered a sync that added that overlay while Leaflet was still
applying the click. Leaflet doesn't redraw the list for changes it sees
mid-click, so the overlay's checkbox stayed unticked, and the next click
on any overlay removed it and wrote the removal back to every tab. The
overlay set now defers a sync until the click has finished. New E2E
check `4b` in `test-map-providers-e2e.js` plants another tab's pick and
clicks twice: one write per click, the planted overlay stays ticked. It
fails on the previous code.

## Docs

- `docs/deployment.md`, "Map Tile Providers" section, rewritten. It
covers every provider and its licence, turning groups on and off, custom
servers, vector maps, hillshade, self-hosting, and what is public.
- User guide:
  - `map.md` gets a "Map style and overlays" section;
  - `live.md` notes the picker;
  - `configuration.md` gets a "Map tiles" section;
  - the FAQ gets a new entry on satellite, terrain and gray maps.
- `CHANGELOG.md`: an Unreleased entry.

## Test data

Two datasets were used, so every number below says which one it came
from.

**1. Large synthetic dataset** (perf A/B and screenshots). It is
generated, contains no real mesh data, and is not checked in.

- **Generator:** `synthgen` 1.0.0, seed 1, "full" scale. Built from
upstream `be16173e` and retimed so its 18-day window (2026-10-31 to
2026-11-18) ends at the time of the run.
- **Database:** SQLite, 3.94 GB (586 MB as zstd), 11.6 M rows in all.

| Table | Rows |
|---|---|
| Nodes | 18,102 (12,606 companions, 3,652 repeaters, 953 rooms, 891
sensors) |
| Nodes with a location | 13,452, spread worldwide (latitude −37° to
60°, longitude −123° to 175°) |
| Observers | 1,175, in 42 regions on six continents |
| Transmissions | 759,133 |
| Observations | 10,569,668 |
| Neighbour edges | 42,806 |
| Advert route evidence | 212,142 |

During the perf runs, the server's packet store was pinned to the newest
48 h with a 1.5 GB memory cap. Each server held about 65 k transmissions
and 0.9 M observations in memory (baseline 65,647 / 906,467; candidate
64,862 / 896,248), plus all 18,102 nodes (about 17,650 active) and 1,175
observers. The tracked store was about 450 MB.

**2. The repo's CI fixture** (`test-fixtures/e2e-fixture.db`, 704 KB).
All E2E suites and the race stress gates ran against it, freshened the
way CI does: 204 nodes, 31 observers, 500 transmissions, 502
observations. A second copy with the Repeater Planner enabled was used
for the planner-on stress runs.

**Perf environment:**
- Headless Chromium 158 via Playwright 1.58.2, at 1280×800 with device
scale factor 1.
- 9 measured runs per scenario plus 1 warm-up, on each of `master` and
this branch.
- Host: Linux, AMD Ryzen 7 3700X (16 threads), 63 GB RAM.
- **Tiles were stubbed.** The harness answers every tile, elevation and
style request locally with the same stub on both sides (`--mode stub`),
so perf timings involve no network. Only the screenshot gallery uses
real tiles.

## Screenshots

These were taken from the synthetic dataset with real tiles in a real
browser (headless Chromium, 1280×800). Click a thumbnail for the full
size. The images are hosted on a separate assets branch, so they are not
in this diff.

#### Street

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-light.png"
width="260" alt="Street: Carto (default) (light)"></a><br>Carto
(default) (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-dark.png"
width="260" alt="Street: Carto (default) (dark)"></a><br>Carto (default)
(dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-osmfr-hot-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-osmfr-hot-light.png"
width="260" alt="Street: OSM France Humanitarian (light)"></a><br>OSM
France Humanitarian (light) |

#### Gray

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-light.png"
width="260" alt="Gray: OSM France (light)"></a><br>OSM France (light) |
<a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-dark.png"
width="260" alt="Gray: OSM France (dark)"></a><br>OSM France (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-light.png"
width="260" alt="Gray: VersaTiles (light)"></a><br>VersaTiles (light) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-dark.png"
width="260" alt="Gray: VersaTiles (dark)"></a><br>VersaTiles (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-light.png"
width="260" alt="Gray: OpenFreeMap Positron (light)"></a><br>OpenFreeMap
Positron (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-dark.png"
width="260" alt="Gray: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap Dark
(dark) |

#### Topographic

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-cyclosm-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-cyclosm-light.png"
width="260" alt="Topo: CyclOSM (light)"></a><br>CyclOSM (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-eox-terrain-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-eox-terrain-light.png"
width="260" alt="Topo: EOX Terrain Light (light)"></a><br>EOX Terrain
Light (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-opentopomap-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-opentopomap-light.png"
width="260" alt="Topo: OpenTopoMap (opt-in) (light)"></a><br>OpenTopoMap
(opt-in) (light) |

#### Satellite

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-light.png"
width="260" alt="Satellite: Sentinel-2 2025 (light)"></a><br>Sentinel-2
2025 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-light.png"
width="260" alt="Satellite: Sentinel-2 2025 with labels
(light)"></a><br>Sentinel-2 2025 with labels (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-dark.png"
width="260" alt="Satellite: Sentinel-2 2025 (dark)"></a><br>Sentinel-2
2025 (dark) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-dark.png"
width="260" alt="Satellite: Sentinel-2 2025 with labels
(dark)"></a><br>Sentinel-2 2025 with labels (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-light.png"
width="260" alt="Satellite: Sentinel-2 2016 (light)"></a><br>Sentinel-2
2016 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-labels-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-labels-light.png"
width="260" alt="Satellite: Sentinel-2 2016 with labels
(light)"></a><br>Sentinel-2 2016 with labels (light) |

#### Vector (MapLibre)

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-liberty-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-liberty-light.png"
width="260" alt="Vector: OpenFreeMap Liberty
(light)"></a><br>OpenFreeMap Liberty (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-dark-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-dark-dark.png"
width="260" alt="Vector: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap
Dark (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-light.png"
width="260" alt="Vector: VersaTiles Colorful (light)"></a><br>VersaTiles
Colorful (light) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-dark.png"
width="260" alt="Vector: VersaTiles Colorful Dark
(dark)"></a><br>VersaTiles Colorful Dark (dark) | | |

#### Hillshade overlay

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-light.png"
width="260" alt="Overlay: hillshade over Sentinel-2
(light)"></a><br>hillshade over Sentinel-2 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-light.png"
width="260" alt="Overlay: hillshade over Carto
(light)"></a><br>hillshade over Carto (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-dark.png"
width="260" alt="Overlay: hillshade over Sentinel-2
(dark)"></a><br>hillshade over Sentinel-2 (dark) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-dark.png"
width="260" alt="Overlay: hillshade over Carto (dark)"></a><br>hillshade
over Carto (dark) | | |

#### Repeater Planner

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-light.png"
width="260" alt="Planner: Sentinel-2 + hillshade
(light)"></a><br>Sentinel-2 + hillshade (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-dark.png"
width="260" alt="Planner: Sentinel-2 + hillshade
(dark)"></a><br>Sentinel-2 + hillshade (dark) |

#### Layer picker

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-light.png"
width="260" alt="Layer picker open (light)"></a><br>Layer picker open
(light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-dark.png"
width="260" alt="Layer picker open (dark)"></a><br>Layer picker open
(dark) |

#### Custom tile server

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-map.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-map.png"
width="260" alt="Custom raster provider on the map"></a><br>Custom
raster provider on the map | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-picker.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-picker.png"
width="260" alt="Custom raster provider in the picker"></a><br>Custom
raster provider in the picker |

## Testing

The full suite ran twice, each time on a fresh clone:

| | Pass 1 | Pass 2 |
|---|---|---|
| Commit (before rebase) | `8a56e841` | `9355169b` |
| `test-all.sh` (unit) | pass | pass |
| `npm run lint` | 0 errors (87 warnings, all pre-existing) | 0 errors
(87 warnings, all pre-existing) |
| Go `cmd/server` / `cmd/ingestor` | ok / ok | ok / ok |
| E2E, map-related files (CI fixture) | 44/44 | 44/44 + planner-on run |
| `test-e2e-playwright.js` | 149 passed, 3 skipped | 149 passed, 3
skipped |
| Race stress gate (seeds) | 101, 202: all pass | 303, 404: all pass |
| Stress with planner enabled | pass | pass (seed 505) |
| Gallery | 32/32 | 32/32 |

The rebased equivalents of the pass 1 and pass 2 commits are `fde2552c`
and `e4d5c447`.

**After the first CI run.** The two passes covered only the map-related
E2E files, and CI exposed two problems:

1. `test-issue-1206` failed 7 checks. This was the `unload` bug
described under "Fixes included".
2. The JS unit gate runs without `node_modules`, so the two new jsdom
suites crashed `test-all.sh`. Locally, jsdom was reachable from a parent
checkout, which hid this.

The second problem is fixed in `57720bf3`, following the
`domIntegration` convention in `scripts/non-unit-tests.json`.
`test-map-teardown-races.js` moved to `tests/e2e/`, and the three jsdom
zoom-limit tests split out into
`tests/e2e/test-map-vector-leaflet-zoom.js`, so 30 + 3 = 33 tests are
kept. Both run from E2E shard 2 and take a few seconds. The heavy suites
(race stress, perf, gallery) stay local-only and are not run in GitHub
Actions.

These checks ran on the fixed, rebased tree (`57720bf3`):

| Check | Result |
|---|---|
| `test-all.sh` from a copy with no `node_modules` (as CI runs it) |
pass |
| `test-test-inventory.js` | 399 suites classified |
| `npm run lint` | 0 errors |
| Go `cmd/server` / `cmd/ingestor` | ok / ok |
| `test-issue-1206-resize-observer-leak-e2e.js` | 13/13 (7 were failing)
|
| `test-e2e-playwright.js` | 149 passed, 3 skipped |
| `test-map-providers-e2e.js` / harness E2E | 17/17, 16/16 |
| `test-map-teardown-races.js` / `test-map-vector-leaflet-zoom.js` |
19/19, 3/3 |
| Race stress (3 iterations) | all scenarios pass |

The commits since pass 2 are `76c67184` (docs only), `7f8d792c` and
`57720bf3`.

**GitHub Actions on `0506fc1d` (rebased onto `master`, with the two-tab
fix): every check passed** (13 passed, 4 skipped as usual for a PR).
That covers Go build and test, all three E2E shards, the server race
detector and both image builds. The moved suites ran in shard 2
(`test-map-teardown-races.js` 19/19, `test-map-vector-leaflet-zoom.js`
3/3), and the unit gate ran `test-map-vector.js` (30/30).

**Race stress suite** (`tests/e2e/test-map-race-stress-e2e.js`). Each
gate run does 20 iterations of scenarios a to e and 50 cycles of f. The
scenarios cover:
- rapid provider switching;
- theme flips during a vector load;
- navigating away mid-load (including a 15 s MapLibre timeout);
- repeated map-page navigation with heap checks;
- overlay toggles;
- cross-page picker persistence.

Results for each run:
- Listener counts stay flat: at most 3 `mc-tile-provider-changed`
listeners and 5 `storage` listeners.
- At most one live WebGL context at a time.
- Heap growth in the second half of scenario d was 0.25 to 0.45 MB
across the four gate runs.
- Upstream `master` grows about 0.39 MB per map navigation loop in the
same scenario; this branch levels off.

## Maintainer review (addressed)

1. **Licence default.** All five new groups (`eox`, `osmfr`,
`openfreemap`, `versatiles`, `hillshade`) are opt-in: off unless
`enabled: true` (`"true"`, `1` or `{}` don't count). With an untouched
config the picker lists the same 6 base maps as before. Docs,
`config.example.json` (now `enabled: false` for each) and the changelog
say so, and each group's terms are listed before you turn it on.
2. **Saved pick.** The on-map picker now saves the pick (option 1 of
your two), as the docs promised. Behaviour, and what an other-theme pick
does:
- Every picker opens on the pinned map, on any page and after a reload,
until **Auto** is picked. A pin is shown whatever the theme, as an
explicit pick already was.
- Maps without a picker show it when it fits the current theme (`type`
is the theme or `any`), otherwise the theme's map. So a light pin isn't
drawn on a dark node map; satellite always is.
- A stored pin whose group is turned off reads as Auto, and comes back
if the group is turned on again; only picking Auto clears it.
- On a map with a picker, Auto always means the theme's map: the pin is
only the explicit layer, never loaded into the hidden Auto layer. A
choice in the customizer, and Reset All, clear the pin. A failed write
(quota full) never brings back an older stored pin.
- E2E `7a` to `7c` in `test-map-providers-e2e.js`: pick, reload, the
radio and tiles are the pick, the node detail map draws it, and a light
pin in dark is kept but not drawn. Auto clears the key; junk in it gives
Auto; an untouched config lists none of the new groups. 7a and 7b fail
with the save line removed. Unit tests are in `test-map-tile-pin.js`
(21).
3. **CI.** `test-map-providers-e2e.js` (24 checks, including `4b`, the
two-tab regression check) and `test-map-providers-harness-e2e.js` (16)
now run in E2E shard 3 against the fixture server: every tile host is
stubbed.

Non-blocking:
- `getTileUrl()` now returns the raster fallback for a vector pick
(`MC_rasterTileUrl`). There's a test in `test-tile-resolver.js` that
fails on the old code.
- The `map-vector.js` header and the binding's vendor README list the
Leaflet and binding internals to recheck on an upgrade.
- `docs/deployment.md` says SRI makes a JS-rewriting proxy drop vector
maps to raster, so serve `public/vendor/` unmodified.
- `scripts/map-perf/README.md` says where `SEED_DB` and `SHIFT_SH` come
from.
- The maps with `attributionControl: false` are left as they are. That
predates this PR, and with every new group opt-in, an upgrade adds no
uncredited imagery. Follow-up for all of them: #2441.
- The OpenFreeMap credit line is `OpenFreeMap © OpenMapTiles Data from
OpenStreetMap` (`2424f3c3`), the line OpenFreeMap gives for clients that
don't add it automatically.

Verified after these changes:
- `test-all.sh` and lint pass, with 0 errors.
- Both provider E2E suites pass: 24/24 and 16/16.
- Race stress, seed 979: scenarios a, b, d and e pass. c (15/15) and f
(37/37) pass after updating their model of the pick: they expected Auto
after a return, which was the old unsaved behaviour.
- The stress suite now enables the five groups itself, so it keeps
exercising them.

## Performance

A/B test: `master` against this branch on the large synthetic dataset,
with 9 runs per scenario and stubbed tiles (see Test data;
`scripts/map-perf/`). The full HTML report is not attached; ask if you
want it.

- **No budget failures** in either pass. The `raster-no-maplibre` budget
holds: no raster scenario loads MapLibre.
- **DOMContentLoaded is unchanged on every page**, even with the two new
scripts (`map-hillshade.js` and `map-vector.js`, about 635 lines in all;
MapLibre itself loads only on demand).
- **Switching providers 50 times (raster only), median ms per switch:**
  - dark: 146 → 108.5;
  - light: 208 → 208.5.
  - Heap growth during the run: 0.3 to 0.4% on both sides.
- **Flagged and checked:**
- Live and Analytics `loadEventMs` are flagged in some runs. This metric
has two modes on **both** branches: `load` fires either before Leaflet
adds its first tile `<img>`s (about 150 ms) or after they finish (about
400 ms). A direct probe of `master` alone gave 428, 404 and 124 ms. On
these scenarios, map creation, first tile, all tiles, first marker and
wall time are within a few ms on both sides.
- Carto and Positron `allTilesSinceMapMs` are 4 to 6 ms slower, just
above `master`'s p95. Tiles are stubbed locally, so this is not network
noise. The cause is not isolated. Both providers now go through the
shared layer builder, and on the same scenarios first tile, all tiles
and wall time are within 10 ms of `master` (Carto Voyager +2 to +3 ms;
Positron Dark +6 to +9 ms).
- The Analytics map shows one long task of about 50 ms against a
`master` median of 0, but `master`'s p95 is 52 to 84 ms. This is
disclosed in case it shows up for you.
- Other desktop apps were running during the perf runs, so expect some
noise.

## Commits

1. `c6bf12ed` feat(map): tile provider registry with custom servers,
gray, topo and satellite maps, and overlays
2. `9237230b` feat(map): client-side hillshade overlay
3. `781e41b4` feat(map): vector basemaps via vendored MapLibre GL 5.24.0
4. `dae36d2e` refactor(map): every map surface uses the shared basemap
builder
5. `3d4afcdc` fix(map): teardown races and listener leaks when leaving
map pages
6. `b798b2fe` test(server): pin map.tiles passthrough in
/api/config/client
7. `171b1bc4` test(map): E2E, race stress suite, perf harness and
screenshot gallery
8. `61f73875` docs(map): providers, custom tile servers, overlays and
self-hosting
9. `135a9b12` test(map): gallery shows opt-in OpenTopoMap and retries
slow tile servers
10. `3833572b` test(map): harness waits for vector basemaps; stress
suite fixes from test pass 1
11. `3e543cad` test(map-perf): redact tile-URL credentials; compare
switch50 like with like
12. `d5006111` docs(map): user guide for map styles, overlays and
map.tiles
13. `90ee1d5d` fix(map): MC_safeRemoveMap lets unload listeners run
before dropping them
14. `5fc57b5c` test(map): run the jsdom map suites in the E2E job, not
the unit gate
15. `bd36b634` fix(map): another tab's overlay pick that lands mid-click
is no longer removed
16. `38df65d2` fix(map): new tile groups are opt-in; a map picked in the
picker is saved
17. `eafbcac9` ci(e2e): run the map provider E2E suites in shard 3
18. `65b223ec` fix(map): getTileUrl() never returns a vector style url;
review notes
19. `f3402c5d` test(map): race stress enables the opt-in groups and
expects the saved pick
20. `e92b8631` fix(map): Auto on a picker map ignores the saved pick;
customizer clears it

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 22:26:46 +01:00
AdamandAdam 01d2bbbbfd fix(account): keep Unclaim and Unlink disabled until the list reloads (#2440)
Follow-up to #2430, covering the two points from @efiten's re-review.

- **Unclaim and Unlink stay disabled until the list reloads.** Both
click handlers now return the list reload from the DELETE's `.then`
(`public/account.js`), so the button is re-enabled only after the
redrawn list has replaced it. Before, a click between the DELETE's
answer and the list GET's answer sent a second DELETE.
- **The dropped-from-node-list note shows in every state.**
`repeaterHealth` adds "no longer in the node list" whenever `r.inactive`
is set, so an inactive repeater that was heard recently or whose prefix
still relays is marked too, not only a silent one.

Tests (`tests/unit/test-user-management-ui.js`, 103/103):
- For both Unclaim and Unlink, the button stays disabled while the list
GET after the DELETE is pending, and a click in that window sends no
second DELETE.
- An inactive repeater carries the note when active, uncertain, stale
and never heard, and none of those rows links to a node page.

`tests/e2e/test-lint-network-calls.js` 47/47.

Co-authored-by: Adam <196335+anieto@users.noreply.github.com>
2026-11-21 22:26:28 +01:00
AdamandAdam 49dd4ee773 List my claimed repeaters on My account (#2430)
Follow-up to #2415, see #2414.

## Situation

A user can claim repeaters, but no page lists the ones they claimed.
Claims show on each node page ("Maintained by"), in the admin user
detail and in the "Download my data" export. A maintainer of several
repeaters has to remember them and open each node page to see whether
one went quiet.

## Change

**`GET /api/account/repeaters`** (browser session; device tokens answer
403) returns the caller's claims. For each one:

- `pubkey` and `name`.
- `known`: false once the node left the analyzer database. The claim
stays.
- `role` and `claimedAt`.
- `lastHeard`: the newer of the last advert and the last transmission
certainly from the node, the same sources as the node page's Last Heard.
- `lastRelayed`: when its hop prefix last relayed. Only used to mark a
silent repeater as uncertain, since the prefix can be shared.
- `status`: the saved repeater status, or null.
- `comaintainers`: the other active claimers' display names.

The list is sorted by name, with unnamed repeaters after, by pubkey.
It's built on the existing `RepeaterClaimsByUser`, `NotifyNodes` and
`LastHeardMap`.

**My account, My repeaters**, a new section after Companions:

- Silent repeaters come first. Health uses `getNodeStatus`, so the
threshold matches the node pages and map (`infraSilentMs`):
  - green "heard 2h ago";
  - yellow "last heard …, its prefix still relays";
  - red "silent, last heard 184d ago" or "never heard";
  - grey "no longer in the database".
- Each row links to the node page and shows the saved status ("no
details yet" without one), the claim date and "also maintained by …".
- *Unclaim* asks for confirmation, then uses the existing `DELETE
/api/repeaters/{pubkey}/claim`.
- The empty state says how to claim.
- `#/account?section=repeaters` deep-links to the section, like
`section=notifications`.

Editing stays on the node page, so there's only one editor. The admin
user detail and the export are unchanged.

Docs: `openapi.go` and the Repeater info section of
`docs/user-guide/accounts.md`.

## Notes

- A mobile client will want the same list. #2429 proposes device-token
scopes for repeater claims. If it lands, letting a `repeater:claim`
token read this route is a one-line addition to the bearer route table,
and I'd rather do that there than here.
- The two PRs touch nearby lines in `openapi.go`, `repeater_handlers.go`
(route registration) and `public/account.js`. Whichever merges second
needs a small rebase.

## Tests

- `cmd/server/my_repeaters_test.go`:
  - empty list;
  - ordering (named first, case-insensitive);
  - unheard, heard and departed nodes;
  - saved status;
  - co-maintainers, excluding the caller and disabled users;
  - another user's view;
  - unclaim removes the row;
  - anonymous gets 401 and a device token 403;
  - no analyzer DB.
- `tests/unit/test-user-management-ui.js`: health sort and chips,
escaping, the short-pubkey fallback, status and co-maintainer lines,
Unclaim on every row, and the empty state.
- `tests/e2e/test-user-management-e2e.js`:
  - the claimed repeater appears with its status and node link;
  - axe on the section;
- Unclaim: dismissing the confirm keeps the claim, accepting empties the
list.

Run locally:

- `make test`: all 16 modules OK.
- `sh test-all.sh` and `node tests/e2e/test-lint-network-calls.js`:
pass, run in `node:24-bookworm`.
- `npx eslint public/*.js`: 0 errors, the same 87 warnings as master.
- `test-user-management-e2e.js` against local e2etest servers with the
CI config: 46/46.
- Manually in Chrome, light and dark: heard, silent and co-maintained
rows, and the deep link.

Not run: the full Playwright shard matrix from CI.

---------

Co-authored-by: Adam <196335+anieto@users.noreply.github.com>
2026-11-20 16:25:51 +01:00
efiten ca74bc12f9 fix(repeater-info): keep an open edit form on screen through a re-render (#2437)
## What

On the full node page, an open repeater-info edit form now stays on
screen when the page re-renders (`theme-refresh`). Before, the fresh
section stayed empty until `GET /api/repeaters/<pk>/info` answered.

## Why

`loadFullNode` (`public/nodes.js`) replaces the page body and calls
`CSRepeaterInfo.mount` on a fresh element. `mount` kept the typed values
in its `drafts` map, but only showed the form again after the GET, so
for one round trip the form was gone.

That made `tests/e2e/test-user-management-e2e.js` flaky. Every page load
fires one `theme-refresh` about 300 ms after `/api/config/theme`
resolves. On CI that refresh sometimes landed between filling the form
and the `page.evaluate` that marks the notes field, which then threw
`Cannot read properties of null (reading 'dataset')`. The three later
repeater-info steps failed as a consequence.

Since 2026-11-10 this happened in 2 of about 150 CI runs: run
44506140120 (#2435) and run 44187421862 (a docs-only branch).

Reproduced locally with `/info` delayed 600 ms and a `theme-refresh`
after filling the form: the notes field was missing during the refresh
on the old code, and present on this branch (3 of 3 runs).

## Changes

`public/repeater-info.js`:
- `mount` shows an open draft at once, in the same task as the
re-render. The GET only replaces it when the user lost edit rights.
- Cancel before the GET answers no longer throws on a null `info`; the
section stays hidden until the view can render.
- A failed GET leaves an open form in place.
- A save on the early form is not overwritten by the older GET answer.
- A section the page already replaced ignores its late GET, so it cannot
delete the draft its successor owns.

`tests/e2e/test-user-management-e2e.js:745`: the side-pane step
collapsed whitespace with `/s+/`; now `/\s+/`.

## Tests

- `tests/unit/test-repeater-info.js`: 6 new cases, 27 pass. All 6 fail
on the old `repeater-info.js`.
- `tests/e2e/test-user-management-e2e.js`: 44/44 locally against an
`e2etest` build, run twice.
- `test-all.sh`: green locally, except the three suites that also fail
on master on Windows (`test-manage-workflow.js`,
`test-preflight-xss-gate.js`, `test-dev-static.js`).

## Not done

- The side pane (`#node-repeater-info-pane`) still has a short gap. On
`theme-refresh`, `selectNode` shows "Loading…" before `fetchNodeDetail`
and only then mounts. The draft comes back, but the form is off screen
during that fetch. Fixing it means changing the pane render path in
`nodes.js`, which is outside this PR.
- Focus and caret are still lost when the form is rebuilt. That was
already the case before this PR.
2026-11-19 18:33:14 +01:00
efiten 87b70562fd fix(node-analytics): show silent hours in the activity timeline (#2435)
## Problem

The node analytics activity timeline only shows the hours that hold
packets. `GetNodeAnalytics` sends sparse hourly buckets
(`cmd/server/store.go:10328`) and `buildActivityChart` used them as
category labels, so a silent stretch takes no room on the axis. An 11 h
outage of one repeater collapsed into two neighbouring bars, and two
repeaters that went down 12 h apart looked as if they dropped at the
same moment.

## Change

- `public/node-analytics.js`: `fillTimeline(tl, timeRange)` places every
hour from the start of the covered window (`timeRange.dataFrom` when
partial, else `timeRange.from`) to `timeRange.to` on the axis. Silent
hours get `{count: 0, certain: 0}`, so the certain/uncertain split from
#2260 stacks them as empty. Buckets the server sent are kept as they
are, also one outside the window. A missing or malformed `timeRange`
leaves the timeline unchanged.
- Above 8 days of covered window the hours are summed per UTC day (count
and certain), and the bar is labelled with its date plus "UTC". 337
hourly bars (14 days in memory) are about 1.5 px wide on a 700 px
canvas, so a quiet node would look empty. 7d stays hourly (169 bars).
- Buckets from older ingestors (`"2006-01-02 15:04:05"`, space instead
of T) are normalised so they land on their own hour.
- Exported as `NodeAnalyticsTimeline.fill` next to `series`.

Server stats are not touched: availability counts the hours with certain
packets (`store.go:10540`) and the longest silence is the gap between
them (`store.go:10549`), both correct on the sparse list. Sending zero
buckets from the server would break both.

## Scale

One bar per hour up to 8 days covered (25 for 24h, 169 for 7d), one per
day above it (31 for 30d, 366 for a full year). The fill is one pass
over the hours of the window: 8761 iterations for 365 days.

## Tests

- New `tests/unit/test-node-analytics-timeline-gaps.js` (13 cases, real
`node-analytics.js` in a vm), listed in `test-all.sh`: gaps filled with
zeros, window edges rounded to the hour, partial window starts at
`dataFrom`, empty node gives a full row of zeros, filled buckets go
through `series` as 0/0, bad `timeRange` returns the input,
out-of-window bucket kept, end on the hour included, 7d stays hourly, >8
days sums per day, 365 days gives 366 buckets, legacy space-format
bucket lands on its hour.
- Every suite in `test-all.sh` run one by one: all pass except
`test-manage-workflow.js` and `test-preflight-xss-gate.js`, which fail
the same way on this checkout without the change (Windows line endings /
console encoding).

## Browser check

Local server on the e2e fixture with one transmission moved 6 h back:
the 24h chart shows 10 hourly bars, five of them empty between the lone
early packet and the recent ones. Before the change the same data drew 4
adjacent bars. With the same transmission moved 10 days back, 30d shows
11 day bars labelled `2026-11-09 UTC` to `2026-11-19 UTC`, the lone
early packet visible at the left.

Not done: no E2E test added, no check on a mobile viewport.
2026-11-19 11:56:52 +01:00
efiten b2c8693cbc ci(e2e): run the axe gate's viewport/theme pairs concurrently (#2433)
Refs #2424. After #2432 the E2E shards are the longest path of a PR run:
9m34s, 11m00s and 10m51s on its run 44394972332.

## Where the E2E time goes

From the timestamped logs of that run: 1,655 s of suites over three
shards, plus about 1 min of setup per runner.

- **`test-a11y-axe-1668.js`: 359 s, 22% of the total, alone on shard
1.** It checks 104 cells (26 routes × 2 themes × 2 viewports) one after
another, and each cell waits a fixed 1.5 s for the SPA to render.
- **Next largest:** `test-e2e-playwright.js` 153 s,
`test-issue-1147-section-order` 89 s, `test-observer-detail-ownership`
88 s, and the slide-over flake gate about 100 s.
- **117 suites under 10 s each:** 409 s together, mostly the start of a
node process and a browser.

## What changes

1. **The axe gate runs its four (viewport, theme) pairs at the same
time.**
- Each pair already had its own browser context; the route loop moved
unchanged into `runCombo()`, and `main()` awaits the four with
`Promise.all`.
- Each pair buffers its log lines, which print afterwards in the old
order, viewport headers included.
- The 1.5 s render wait is unchanged, so axe sees the same page state;
the waits only overlap.
2. **Four suites move to shard 1, which gains about 240 s:**
- from shard 2: `test-issue-1147-section-order` (89 s) and
`test-health-status` (29 s);
- from shard 3: `test-observer-detail-ownership` (88 s) and
`test-issue-1281-location-row` (25 s).

None of them uses the shard-specific servers (`:13582` on shard 3,
`:8788`/`:13591` on shard 1). Expected, with the axe gate scaled by the
local ratio: about 479 / 468 / 461 s of tests per shard, down from 490 /
586 / 574.

## Checked locally

Windows, Chrome, the e2e fixture migrated, server on `:13581`:

- **Same output, three times faster:**

  | version | time | result |
  |---|---|---|
  | upstream/master | 282 s | 104 cells, net=0 |
  | this branch | 92 s | 104 cells, net=0 |

The per-cell lines (`✅/❌`, viewport headers, `SUMMARY`) are identical in
both runs (`diff` of the grep).
- **The failure path still works.** Against a port with no server
(`AXE_ROUTES_ONLY=/packets`), all four cells report `❌ net=1`, four
screenshots are written, and the exit code is 1.
- **Unit tests pass:** `test-test-inventory.js`, the a11y unit tests
that load this file (selftest, routes coverage), and
`test-issue-1956-release-routing.js`.

## Not checked

- **The CI time.** It comes from this PR's run. Four pairs at once on a
4-core runner may speed up less than on my laptop; I'll post the shard
times.
- **The moved suites, run locally.** Their code is unchanged and they
use the shared server on `:13581`. On shard 1 they run after the
split-frontend suite instead of their old neighbours.
- **The other suites themselves.** `test-e2e-playwright.js` holds 54
`waitForTimeout` calls. Replacing fixed waits with readiness checks
would change what each suite waits for, so that is not in this PR.
2026-11-18 23:11:35 +01:00
efiten f4d15f8456 ci: run the server race detector in four shards beside go-test (#2432)
Refs #2424. The Go Build & Test job is the longest path of the pipeline:
23.5 min on #2410's last run, against 10 to 13 min for the E2E shards.
Of that, the server step (`go test -race -coverprofile`) took 21 min.

## Why it is slow

- The server package has 2,364 tests and none call `t.Parallel()`, so
they run one after another in one process.
- Natively, without `-race`, the whole package takes 101 s locally
(Windows, `go test -json`). Under `-race` with coverage on the hosted
runner it takes about 18 min.
- No single test explains it: 2,151 of the tests take under 0.1 s each.
The slowest one, `TestViewshedMatchesReference`, is trimmed separately
in #2431.

## What changes

- **`go-test`**: the server suite runs with coverage only, no `-race`.
Coverage badges and the e2etest step are unchanged.
- **New `server-race-shard` matrix, 4 runners.**
- Each shard builds the race test binary once (`go test -race -c`) and
lists the tests with `-test.list`, keeping `Test`, `Example` and `Fuzz`
names, which is what `go test` runs by default.
  - It runs every fourth name with `-test.run "^(A|B|...)$"`.
- Every test still runs under `-race`, in exactly one shard. A shard
with no tests fails.
- **New `server-race` gate**, built like `e2e-test`: it requires every
shard. `build-and-publish` requires the gate.
- **Tag runs**: the shards skip on a tag run with `images_published`,
the same condition as the E2E shards. The `:edge` commit was already
checked.
- **`tests/unit/test-issue-1956-release-routing.js`** routes both new
jobs. It asserts that a failed shard fails the gate rather than skipping
it, and that a failed shard or gate fails `build-and-publish`.

## Checked locally

Without `-race`, Windows, test binary from this branch:

- The four shard lists hold 591 tests each. Their union is the full list
of 2,364 names, with no duplicates.
- Each shard ran on its own and passed: 16 s, 25 s, 39 s and 33 s. This
shows no test depends on state left behind by a test in another shard,
at least without `-race`.
- Shard 2 run with `-v` reported exactly 591 top-level results.
- Each `-test.run` pattern is about 21 KB, well under Linux's 128 KB
per-argument limit.
- The routing test passes.

## Not checked

- The hosted wall time. The shard times will come from this PR's own
run. Each shard compiles the race build of cgo SQLite itself, because
the go-build cache that `go-test` saves holds non-race objects. I'll
post the measured times in a comment.
- Whether branch protection requires specific check names. The API
answers 404 for me. If "✅ Go Build & Test" is required, "🏁 Race detector
(server)" should be added next to it. Otherwise a race failure turns the
run red but does not block a merge.
2026-11-18 23:05:55 +01:00
0af70cf2c1 fix(build): pin musl thread stacks at 8 MiB and check the artifacts (#2410)
## Summary

Pins musl thread stacks at 8 MiB on every link, and fixes a crash in
**native Alpine builds** (`make build`, `make test`, or `go test` on an
Alpine host).

**The released images are not affected.** An earlier version of this
description said public endpoints could crash the shipped image. That
was wrong, and review caught it: see "Who is affected".

## Root cause

- musl sizes every non-main thread's stack from the ELF `PT_GNU_STACK`
size, with a floor of 128 KiB and a cap of 8 MiB (glibc defaults to 8
MiB).
- cgo calls run on the calling thread's stack, and SQLite's
`multiSelect` recurses about 850 bytes per `UNION ALL` term.
- `lookupPrefixes` (up to 400 terms) and `sharedEightBytePrefixes` (up
to 450 terms) overflow a 128 KiB stack and SIGSEGV the whole process.

## Who is affected

| Link | `PT_GNU_STACK` without the flag | Thread stack | Wide-UNION
tests |
|---|---|---|---|
| Alpine gcc + GNU ld (`make build` / `make test` / `go test` on Alpine)
| `0` | 128 KiB | **SIGSEGV** |
| zig 0.16.0, x86_64-linux-musl (Dockerfile, `crossbuild`, release job)
| `0x1000000` (16 MiB) | 8 MiB | pass 3/3 |
| zig 0.16.0, aarch64-linux-musl | `0x1000000` (16 MiB) | 8 MiB | pass
3/3, run on an arm64 node |

So for the released images the flag pins a value that zig's default
already provides. It stops being a property of a linker default, and a
build check now fails if it is lost. For native Alpine links it is the
fix: with the flag those tests pass.

Control: a zig arm64 link with `-Wl,-z,stack-size=131072` gets 128 KiB
threads and the same tests SIGSEGV in cgo on the arm64 node.

## Fix

- Every musl link passes `-Wl,-z,stack-size=8388608`: the `Dockerfile`,
the `Makefile` (`build` and `test` on Linux, and `crossbuild`) and the
release job in `deploy.yml`.
- This reserves address space only. Memory is committed as pages are
touched.
- **The artifacts are checked.** The Dockerfile's builder stage and the
release job's "Verify release binaries" step read `PT_GNU_STACK` with
`readelf` and fail unless it is exactly 8 MiB. Verified: the builder
stage passes for `linux/amd64` and `linux/arm64`, and fails with
`--build-arg STACK_LDFLAG=` (`/corescope-server: PT_GNU_STACK is not 8
MiB`, memsz `0x1000000`).

## Testing

- `TestMuslThreadStackWideUnionQueries`
(`cmd/server/musl_thread_stack_linux_test.go`, Linux only) runs both
queries off the main thread and asserts exact results, so a failed chunk
cannot pass as "nothing matched". Under `golang:1.27-alpine` with gcc:
SIGSEGV without the flag, pass with it. On CI's glibc runners it passes
either way, which is why the artifact checks above exist.
- `tests/unit/test-issue-1956-release-routing.js`: the release verify
step must check both binaries, and a `0x000000` or `0x1000000` header
must fail the release.
- `tests/unit/test-dockerfile-buildplatform.js`: all three `go build`
lines carry `STACK_LDFLAG`, and the `readelf` check covers all three
binaries.
- `sh test-all.sh` passes. `go vet` passes for `cmd/server` on Linux
with gofmt clean.
- Not verified: a Windows `go vet`. The fix there is the
`_linux_test.go` file name; no Windows machine was available to run it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## Merge order

These four PRs were split out of #2396 and are **independent — they can
merge in any order.** Verified: every pair is conflict-free, and all
four merged together pass `go vet`, `go test ./...` (all 16 modules) and
`sh test-all.sh`. (That combined run predates the review fixes on #2409
and #2410.)

Suggested order, none of it urgent:

1. **#2408** — never lose rows during startup / a failed poll.
2. **#2407** — explicit `observers.idx` primary key.
3. **#2409** — content-hash migration moved into the ingestor.
4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not
affected; it fixes native Alpine builds.

After these land, #2396 (Postgres) will be rebased onto master.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-11-18 18:35:41 +01:00
n30nexandn30nex 17c1e3d1b3 test(live): wait for drag controls to finish initializing (#2423)
Partial fix for #1995. The Live page displays its panel headers before
it finishes loading map configuration and registering the drag handlers.
The browser test could drag the feed during that interval and then fail
its position, storage and reload checks. Its short-click check could
also pass before a handler saw the gesture.

After each reload, the suite waits for the visible header and the
current page’s drag manager. It hovers the header, presses the mouse,
checks that the intended panel accepted the press, and measures movement
from the actual press coordinates. This removes three fixed waits.

TDD exemption: pure test-harness repair. The only altered file is
`tests/e2e/test-drag-manager-e2e.js`. All seven scenarios and eleven
original assertions remain. The additional ownership check prevents a
click on the map canvas from satisfying the legend’s dead-zone test.

Browser verified: `http://localhost:15027/#/live`. Delaying the actual
map configuration response by 2.5 seconds reproduced three assertion
failures on upstream. The final candidate passed 7/7 under normal and
delayed loading. Blocking a genuine legend press made the old check pass
incorrectly. The strengthened check failed precisely that scenario, with
the other six passing.

The earlier #2419 CI failure showed the same three symptoms. That run
did not record initialization timing.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-18 18:16:18 +01:00
n30nexandn30nex be16173ef2 feat(nodes): link one-byte repeaters to Prefix Tool (#2412)
Red commit: `e27f81f705bf757da25c9fcd5858a6c73fcf54ed` ([CI
run](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44061866573)).

Known one-byte repeaters now offer **Inspect in Prefix Tool** in the
shared Hash Prefix metadata. The link carries the full public key so the
existing checker can exclude the selected repeater and automatically
show its matches. Full detail, wide preview, tablet and mobile use the
same helper. Unknown sizes, other roles and incomplete or invalid keys
receive no action; existing prefix and variation text remain unchanged.

Partial fix for #1154 U11. This adds navigation only; inline collision
warnings/counts and the broader tracker remain open. Prefix Tool retains
its existing filtered, capped population. No configuration or customizer
changes.

- GREEN: `f7a67b0d5ed7764a2400d091cee2dd1c35536e9b`; RED tests
unchanged.
- Helpers: 814/814; standalone frontend: all 244 suites; lint: zero
errors.
- E2E assertion added:
`tests/e2e/test-issue-1147-section-order-e2e.js:227`. Full suite: 90/90,
including both themes, responsive layouts, click/Enter navigation,
prefill, results and scope checks. Additional real-data click-through
passed without API stubs.
- Two initial document-loading timeouts passed on the unchanged
sequential rerun. Standalone validation used private Windows path/Python
adapters; repository infrastructure is unchanged.
- Performance: bounded key validation and one anchor; no node-render
requests, scans or new retained state. Existing theme link color
applies.

Parent independently repeated 814 helper checks and the full 90-case
browser suite, and inspected mobile/tablet screenshots. [Exact-head
GREEN
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44066398243)
and three independent reviews pass with no remaining findings.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-17 23:32:20 +01:00
n30nexandn30nex 71d84ee432 fix(nodes): group exact counts in detail views (#2400)
Node details display exact packet, observation and relay counts using
browser-locale grouping: for example, `12278×` becomes `12,278×`. This
covers full details, wide previews and tablet metadata. Existing numeric
validation, zero handling and raw numeric observer sorting are
preserved.

Partial implementation of #1154 U5. Directory rows, path summaries and
analytics-wide formatting remain outside this slice. No compact
rounding, settings, dependencies or API changes.

Validation:

- Tests-only RED `c9902a165` builds and [fails the intended grouping
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43828088770/job/133166683510);
GREEN `dbb8f37c0` supplies the production change.
- All 230 standalone frontend suites pass, including 812 helper
assertions. Two focused browser suites pass 89 cases across
English/German, desktop/tablet/mobile, both themes, numeric sorting,
edge values and layout bounds.
- Follow-ups align an older help fixture with grouped text and wait for
completed initial theme loading before sorting. Every original
grouping/sort assertion remains intact. Final source still fails four
grouping assertions when served the upstream node renderer.
- General browser suite: 149 passed, three existing fixture skips;
optional/disabled checks remain outside coverage. Parent independently
verified all 13 help scenarios and the seven affected location cases,
including a deliberately delayed real theme response.
- Three independent reviews completed; parent verified the corrections.
[Final CI
passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43838164835)
on `c611492c6`.

Constant formatting work per displayed count; existing loops remain
linear, without new requests or retained collections. No speedup claim
or customizer change.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-17 23:32:09 +01:00
n30nexandn30nex 7850d77a80 fix(nodes): retain timestamp context in status explanations (#2403)
Red commit: `2e65630193311d5b9b13a0ca50355b346cc7a56a` ([failing
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43893211034)).

Partial fix for #1154 U10. Active and Uncertain status explanations now
use the shared timestamp renderer in full detail and wide preview.
Relative dates expose exact-time tooltips; future dates retain their
clock-warning icon. Existing timestamp preferences apply.

- Keeps certain activity separate from attributed relay activity,
preserving status decisions and wording.
- Tests cover one Active timestamp, two Uncertain timestamps, missing
certain activity, and future dates. Helpers pass 813/813; the full
health-status browser suite passes.
- Browser acceptance passed 128 combinations across desktop full/wide,
390px mobile, 800px tablet navigation, relative/absolute mode, UTC/local
time, and both themes. No horizontal overflow or browser errors; 14
screenshots captured.
- Constant work for at most two timestamps per explanation; no added
requests, scans, caches, or retained collections. No new settings or
colors.

All 236 standalone frontend suites pass using process-local Windows
file-URL/Python launcher adapters; repository tests are unchanged.
ESLint reports zero errors. The broader U10 audit and other #1154 items
remain open.

Parent independently reran the helper and affected browser suites and
checked desktop/mobile screenshots. [Exact-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43898474610)
and three independent reviews pass, with no remaining findings. The
separate compact-preview date-selection discrepancy is tracked in #2405.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-17 23:31:57 +01:00
n30nexandn30nex a2417b25a3 test: normalize static helper import as a file URL (#2404)
Partial fix for #1995.

On native Windows, this suite aborts before its five checks because the
absolute module path is treated as an unsupported `f:` URL. Use Node's
built-in `pathToFileURL(...).href` for the dynamic import.

TDD exemption: pure test-harness refactor/repair. Application code is
unchanged. Per-altered-test justification for
`tests/unit/test-dev-static.js`: import-path normalization is the sole
changed fixture statement, plus the required built-in import. All five
existing test bodies, assertions and teardown are byte-identical. The
original loading failure is not claimed as assertion RED.

Validation: Windows x64, Node v22.18.0.

- Untouched suite: `ERR_UNSUPPORTED_ESM_URL_SCHEME`, exit 1 before
assertions.
- Updated suite: 5 passed, 0 failed; also 5/5 from a directory
containing spaces, `#` and `%`.
- Byte comparison, `node --check tests/unit/test-dev-static.js` and `git
diff --check` pass.

Parent independently passed the five native checks and all 236
standalone frontend suites without an ESM adapter. [Exact-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43899294545),
including all three browser shards, and three independent reviews pass.
Runner reconciliation, browser waits, coverage and the rest of #1995
remain open.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-17 23:31:08 +01:00
11eb72f578 feat(planner): Coverage tab, reach of a candidate repeater site (#2406)
The repeater planner (#2391) gets a fourth tab, **Coverage**: put a
marker where you are thinking of placing a repeater, set its antenna
height, and see on the map who would hear it. It answers the siting
question "what does a repeater reach if we put it here", with two
layers: handheld users and other repeaters.

## What it shows

- **Users layer:** the chance that a handheld at 1.5 m hears the site,
in 4 bands (10-25%, 25-50%, 50-75%, 75% and up; below 10% is not
painted). The chance comes from a distance curve `P(d) = 1 / (1 +
exp((ln d - ln d50) / slope))`, fitted on drive-test receptions of 41
Belgian repeaters (d50 2.7 km, slope 0.30). Terrain only lowers a cell
by 2 bands where a 1.5 m receiver sits deeper than 1 first-Fresnel
radius under the terrain (`cmd/server/planner_viewshed.go:19-21`).
Antenna height does not change the curve: in the drive tests, handheld
reach barely changed between 1, 7 and 15 m. The panel says so in one
line.
- **Repeaters layer:** first-Fresnel clearance from the site to every
cell, for a receiver at "Repeaters at" height (default
`planner.existingAntennaM`). 3 classes: line of sight (clearance 0% or
more), slightly blocked (-20% to 0%), blocked (not painted). Every known
repeater within range gets its link computed with the existing planner
link code, a dot in its class colour, a line for line of sight and
slightly blocked, and an entry in a list.
- **Compute button:** changing site, height or range moves the marker
and updates the URL hash right away, and dims an existing result with an
"Outdated" note. Only "Compute coverage" starts a job. A shared link
that carries a site computes once on open. The planner has 1 heavy-job
slot shared with Area and allows 6 jobs per minute per IP, so
recomputing on every change would burn both.

State is in the hash:
`#/tools/planner?mode=coverage&site=51.06,4.41&h=10&rx=5&range=30&layer=repeaters`.

This builds on the repeater planner @zfouts wrote in #2391: the terrain
tiles, the first-Fresnel analysis, the link code, the job protocol and
the Area module pattern are all reused here. Thanks for that work, it
made this tab a small addition instead of a new subsystem.

## Why the Repeaters classes are not the route LOS verdicts

The route verdicts (clear 100%+, clear60 60%+, marginal 0%+, blocked)
almost never show on Shuttle Radar Topography Mission (SRTM) terrain at
mast heights. At 10 m / 5 m, 30 km around Mechelen, 109 of 70,980 cells
reach clear60 or clear. SRTM includes trees and roofs, so the line
between 2 masts rarely clears 60% of the Fresnel zone.

I checked the clearance against one Belgian repeater (28 m, receivers at
5 m) and the repeaters live data shows it actually hears:

| Distance | Heard, clearance >= 0 | Heard, clearance < 0 |
|---|---|---|
| 0-10 km | 9 of 11 | none in range |
| 10-20 km | 3 of 4 | 1 of 8 |
| 20-30 km | 0 of 6 | 3 of 35 |
| 30-60 km | 1 of 2 | 0 of 84 |

Every link heard within 16 km had a clearance between 2% and 54%, so
"marginal" is where working links live. The map classes follow that
(`repeaterClass`, `cmd/server/planner_viewshed.go:75-89`). Chain, Relay
and Area keep the shared verdicts; `losVerdict` is unchanged. This rests
on 17 heard links from 1 repeater, which is why the -20% band is narrow
and the panel says line of sight stops being enough beyond about 20 km.

## Server

- `GET /api/planner/viewshed?site=&h=&rx=&range=`
(`cmd/server/planner_api.go:603`, route `cmd/server/routes.go:393`).
Same job protocol, result cache, rate limits and opt-in as the other
planner endpoints: 404 while `planner.enabled` is false.
- One byte per cell: low nibble Users class 1-5, high nibble Repeaters
class 1-3, 0 for no data. The grid is about 300 x 300 at every range:
0.2 km cells up to 30 km, `range/150` above (`planner_viewshed.go:47`).
- Links skip rooms, stale repeaters and a repeater within 100 m of the
site (that is the site itself). They carry no `elev_m` profile.
- `truncated` is true only when the 300-node cap may have dropped a
repeater within range (`viewshedTruncated`, `planner_api.go:694`).
- Tile cap: `Viewshed` in the planner tile limits, sized from the
largest offered range (60 km).
- New config: `planner.userReach.{d50Km,slope}` with a `_comment` in
`config.example.json`. Bounds 0.2-50 km and 0.05-2.
- No writes, no new `map[string]interface{}`; response types
`PlannerViewshedResponse/Params/Link` in `planner_types.go`.

## Frontend

- `public/planner-coverage.js` + `.css`, mounted like `planner-area.js`.
`planner.js` gets a `MODULES` table (`public/planner.js:31`) so Area and
Coverage share one enter/leave path.
- Canvas raster in an `L.imageOverlay`; the layer switch repaints
locally without a new request. Requests go through `fetchJob` / `CSApi`.
- Colours are CSS variables `--pl-cov-user-2..5` and `--pl-cov-rep-1..3`
in `:root` and both dark blocks of `style.css`; the Repeaters ones point
at `--status-*`, so a theme that changes those changes the map.

## Performance

The job is opt-in and runs in the existing planner job slot, never in a
hot path. Measured on real SRTM tiles, local build, 304 Belgian
repeaters:

| Site | Range | Mast | Job time |
|---|---|---|---|
| Maasvallei (hilly) | 30 km | 10 m | 2.07 s |
| Maasvallei | 30 km | 30 m | 2.57 s |
| Maasvallei | 60 km | 10 m | 3.98 s |
| Mechelen (flat) | 30 km | 10 m | 2.61 s |

On a staging instance (ARM64, cold tile cache) a 30 km job took 6.8 s
including the tile downloads. Response size at 60 km with 115 links:
239,942 bytes with link elevation profiles, 149,984 without (what this
PR ships). `BenchmarkViewshedMaxLeg` covers the 60 km worst case on a
synthetic terrain view.

## Verification

- Go: 26 viewshed tests and benchmarks in `planner_viewshed_test.go`,
`planner_api_test.go`, `planner_config_test.go` and
`planner_tiles_test.go`, among them: flat terrain equals the plain curve
and `analyzeLOS`, a ridge shadows only its far side, missing terrain is
"no data", the `truncated` rule (proven to fail on the old rule), links
without `elev_m`, the repeater at the site left out. `go test ./...` in
`cmd/server` passes.
- Frontend: `tests/unit/test-planner-coverage.js` (23 tests, in
`test-all.sh`): grid decode, class-to-colour mapping, unpainted classes,
hash round trip, no fetch on parameter change, 1 fetch per click,
deep-link autorun once, stale responses ignored, link classes at the
boundaries, escaped repeater names.
- E2E: 5 Coverage cases in `tests/e2e/test-planner-e2e.js` (stubbed
viewshed): click and compute, layer switch without a request, outdated
then recompute, deep link, leaving the tab removes the overlay. Planner
E2E 25/25 locally.
- eslint 0 errors. Browser: light and dark theme, Users and Repeaters on
flat, hilly and coastal sites, deep links, no console errors; also on a
staging instance with live data.

## Not in this PR

- The `--pl-cov-*` colours are not in the customizer's `THEME_CSS_MAP`
yet. They follow `--status-*` for the Repeaters layer; the Users colours
need their own entries (AGENTS.md rule 8, later milestone).
- In Flanders the Users layer is close to a circle: 0.3% of cells differ
from the plain curve around Mechelen, 42% in the Maasvallei, 64% in the
Ardennes. The terrain correction is untested in hilly areas, the drive
tests had no repeater there.
- The Repeaters calibration uses 1 repeater. Other operators with drive
tests or neighbour data can tighten it.
- A link's class comes from `clearance_pct`, rounded to 0.1%, while the
grid uses the raw clearance: within 0.05% of a bound a link can land 1
class off the cell under it. List, line and dot always agree with each
other.
- An OS dark-mode switch without the theme toggle does not repaint the
overlay until the next compute, the same as the planner's tile layer.

---------

Co-authored-by: zfouts <zachary.fouts@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 23:19:49 +01:00
efiten 63d8ab23ed fix(repeater-info): an open edit form survives a node page re-render (#2417)
Follow-up to #2415, see #2414.
2026-11-17 13:21:30 +01:00
efiten af732d18aa fix(repeater-info): log bulk read errors, cap long notes (#2416)
Follow-up to #2415, see #2414.
2026-11-17 12:56:27 +01:00
efiten 0cf22cdd79 feat: repeater info, claims and community-maintained details (#2415)
Closes #2414
2026-11-17 11:40:05 +01:00
4ebb5ccc11 feat: radio line-of-sight on packet routes + repeater planner (#2391)
Both features are OPT-IN ("los" and "planner" config blocks, default
off); with them off nothing is fetched and payloads are unchanged.

Route view
- Multi-path picker discoverability: on-map "N hops shown · tap to see
all" badge, picker pulse (reduced-motion safe), one-time tip, row hover
arrow; "no GPS" hops counted apart from "unresolved".
- Line of sight for every leg of a packet route: clear / marginal /
blocked chips (first-Fresnel clearance, k=4/3), obstruction markers on
the map, docked terrain-profile panel with antenna/frequency inputs and
a &los= deep link. Backed by GET /api/elevation/profile, an Open-Meteo
proxy that only accepts known node positions, with weighted budgets,
back-off and a per-IP limit.

Repeater planner (#/tools/planner)
- Chain: LOS on every leg between any number of points or nodes.
- Relay: best single relay for A→B among known repeaters and open
terrain.
- Area: suggested new repeater sites that best join the existing
repeater backbone in a box up to 100×100 km, picked greedily to merge
clusters, then remove single points of failure, then reach unlinked
repeaters.
- Terrain: SRTM 1° tiles from AWS Open Data, in memory only (~90 m,
bounded LRU); the server stays read-only. Exact-length gunzip, fetch
budget with back-off, missing terrain reported rather than read as sea
level, fuzzed pixel maths.
- GET /api/planner/{profile,relay,coverage}: quantised cache keys, 202 +
progress then cached 200, separate compute slots, per-IP and global
limits, failed jobs remembered, unpolled jobs cancelled. Nodes pass the
same blacklist / hidden-name / geo_filter rules as /api/nodes. The Go
LOS maths matches public/los.js on shared test vectors.

Also: the per-IP client limiter is extracted from elevation.go into
client_limit.go (behaviour-preserving) so both features share it.

Perf (real SRTM): coverage cold 1.5-5.6 s (tile download), warm 10 ms;
worst real compute 545 ms; peak RSS 91 MiB; live /api/nodes and
/api/packets p95 at most 1.6x while a job computes (GOMAXPROCS=2).

Tests: Go suites for elevation, terrain (fuzz), LOS vectors, graph
(brute-force property test), search and planner API; frontend unit
suites in test-all.sh; Playwright e2e for route LOS and the planner.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-11-16 16:15:36 +01:00
ZachandClaude Opus 5.5 138473d8c2 test(tile-resolver): load CSApi into the sandbox before roles.js (#2402)
## Problem

`tests/unit/test-tile-resolver.js` fails 0/26 on master with `CSApi is
not defined`, and master's CI on `e7fd48a0` is red because of it
([run](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43880608436)).

Two PRs that each passed CI on their own met on master:
- #2386 added this suite, which runs `public/roles.js` in a `vm`
sandbox.
- #2383 made `roles.js` call `CSApi.fetch('/api/config/client')` as it
loads.

The sandbox never had `CSApi`.

## Fix

The suite now loads `deployment.js` and `api-base.js` into the sandbox
first, through `tests/unit/api-base-sandbox.js`, as the other suites
that load `roles.js` already do. That's two lines in a test file, with
no app code changes.

## Verification

- `node tests/unit/test-tile-resolver.js`: 26 passed, 0 failed (it was
0/26).
- `sh test-all.sh` on this branch: exits 0.

## Related

#2391 needs this. It now includes #2383 from master, so its CI hits the
same failure. Once this lands, I'll merge master into #2391 again.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:59:55 +01:00
Paul PicazoandClaude Opus 5.5 e7fd48a054 feat(frontend): serve the UI from a web container or a static host, three deployment modes (#2383)
## What this does

The frontend half of serving the UI apart from the server. The server
half was #2362. It gives CoreScope three deployment modes. The server
container is the same image in all three, and it still serves
everything, the classic UI included.

| Mode | The UI is served by | API | Server CORS setting | Logins |
|---|---|---|---|---|
| All-in-one container (today) | the server container | same origin |
none | yes |
| Split containers (new) | a web container that forwards the API | same
origin | none | yes |
| Static host or CDN (new) | Cloudflare Pages, a CDN or any web server |
another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI |

### Commit 1: the UI as static files, for any host

- **One place for API URLs:** `public/deployment.js` says where the API
is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket
URL. A lint rule keeps raw `/api` fetches out of the other files. The
embedded UI keeps `apiBase: ''`, so it behaves as before, except that
its WebSocket now uses `/ws` instead of `/`.
- **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`,
which is `public/` byte for byte except for the cache buster in
`index.html` and `deployment.js`.
- It is `dist/web` rather than `dist/` because the Go binaries are built
into `dist/` and `make clean` removes that folder.
- `API_BASE=same-origin` is for a host that forwards `/api` and `/ws`
itself, as the web container does.
- **Anonymous build (`AUTH=false`):** for another origin. It has no
account features: login, account, admin and other writes link to the
same page on the classic UI. It sends no cookies, and its three JSON
POSTs go as `text/plain`, so there is no preflight.
- **Error notices:** a blocked API answer or an HTML answer each show a
notice.
- **Channel badge:** `beta`, `next` and `dev` builds show a badge that
opens a pre-filled issue, plus an "Open in classic" link.
- **Runbook:** `deploy/static/README.md` covers Cloudflare Pages,
previews, server CORS and rollback.

### Commit 2: the web container and the three modes

- **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It
packages the UI with `build.sh` when the image is built, serves it, and
forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`).
`SITE_ADDRESS` set to a domain name turns on automatic HTTPS.
- **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules
as `deploy/static/_headers`, and compresses responses with zstd or gzip.
- **`docker-compose.split.yml`:** the stock server image (its own Caddy
off) plus the web container:
`CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f
docker-compose.split.yml up -d --build`.
- `CORESCOPE_IMAGE` is required and must match the checkout's version,
so a release server is never paired with a newer UI by default.
- The server's built-in broker, which accepts anonymous clients, is
published on `127.0.0.1` unless `MQTT_BIND` says otherwise.
- **Docs:** `docs/deployment.md` has a new "Deployment modes" section,
and `DEPLOY.md` points to it.

### Keeping new code on `CSApi`

- **Lint rule:** `.eslintrc.json` now rejects every direct network call
in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`,
`sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in
`href`, `src` and `action` (assigned, set with `setAttribute` or written
into markup), `window.open`, `location.assign`, `location.replace` and
`location =`. `api-base.js` is the one file exempt.
- **HTML pages:** `eslint public/*.js` never sees the inline scripts and
markup of `public/*.html`, so `test-lint-network-calls.js` applies the
same rule to them, at their page lines. `area-map.html`'s two fetches
carry an `eslint-disable` with the reason: its Server field can name
another server.
- **Why it changed:** the earlier rule only caught a literal `/api…`
passed straight to `fetch`.
- **Check:** CI already fails on lint errors, and
`tests/e2e/test-lint-network-calls.js` pins exactly what the rule
reports: 25 bad statements reported, 15 good ones not, `api-base.js`
exempt.
- **AGENTS.md:** a new "API calls" section covers:
  - which helper to use for each need;
  - the lint rule;
  - how anonymous builds gate login-only features;
- loading `api-base.js` in vm and jsdom tests (the region quick-picks
suite missed that, and failed shard 1 on the first CI run).

### Review follow-ups

- **Account tokens:** the channel badge's pre-filled issue drops the
query on `#/account/…` routes, so mail-link tokens never reach a GitHub
URL. "Open in classic" keeps the full route, since the classic UI needs
the token.
- **HTML answers:** on a packaged UI (static host, web container),
`api()` rejects an HTML answer before status and retry handling. An HTML
404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server
returned an unexpected response" after one request. It used to give a
generic error, or six "warming up" retries for a 503. JSON errors and
the server's JSON 503 warm-up behave as before. The embedded UI keeps
its old handling (`CSApi.embedded`).
- **Logins:** `docs/deployment.md`, `Dockerfile.web` and
`docker-compose.split.yml` say `userManagement.publicBaseUrl` must be
the web container's public origin.

### Audit fixes

- **No broken logins build:** `build.sh` refuses `AUTH=true` with a
cross-origin `API_BASE`. The server allows no credentialed CORS, so
every API request of such a build was blocked; in Chromium all 12 were,
and the pages stayed empty.
- **No accidental deletes:** `build.sh` refuses an `OUT` that is neither
empty nor a previous build. It used to wipe whatever folder or file
`OUT` named.
- **Lint:** the rule now also reports `/api/` paths in `href`, `src` and
`action` attributes in single-quoted strings and template literals.
`tests/e2e/test-lint-network-calls.js` pins exactly what it reports.
- **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published
on localhost only.
- **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since
v3.16.0 and v3.15.0 predate the static build. It also names the server
version the static build needs: before #2362 the warm-up headers stay
hidden and wildcard previews are refused.

### Audit notes, fixed

- **Cache busting on every page:** the tool pages (`area-map.html`,
`geofilter-builder.html`) load their scripts with `?v=__BUST__`. The
server and `build.sh` replace it in every top-level page, and `_headers`
and the Caddyfile make every page revalidate. Behind Caddy, a returning
visitor could otherwise get a new tool page with an old `api-base.js`.
- **HTTP/3:** the split compose publishes 443 over UDP too, for the
HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with
Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`.
- **WebSocket on `/`:** a Go test pins that an upgrade on `/` still
works, for UIs cached from before `/ws`.

### Second review

- **Trusted proxies:** the docs said to trust the compose subnet, and
also to publish port 3000 if wanted. Together they let a client spoof
its address: the subnet holds the gateway, where a client reaching a
published 3000 through Docker's proxy arrives from. The compose file now
fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web
container's address (`172.30.83.200`), and the docs say to trust that
address alone.
- **Classic UI unchanged:** the HTML-answer notice above applies to
packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as
before, and an unknown `/api` path fails to parse as before, with no
notice.
- **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md
says what the rule cannot see: a URL assembled elsewhere and passed in a
variable.
- **AGENTS.md:** `build.sh` is described as packaging (a copy,
`deployment.js` and `__BUST__`, never a code transform), and rule 3 says
`__BUST__` is filled in every top-level page.
- **CI builds the web image:** the image-check job (no publish) builds
`Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`,
the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It
also validates the split compose file.
- **Cross-origin logins:** a hand-edited `deployment.js` with `auth:
true` and another origin now logs a warning naming the setting, instead
of failing with CORS errors alone.

## Upgrade notes

- **WebSocket path:** the UI's live feed connects to `/ws` instead of
`/`. The server has answered `/ws` since #1793 and still upgrades `/`,
so browsers holding the previous UI keep working.
- A reverse proxy that forwards WebSocket upgrades only for an exact `/`
must forward `/ws` too.
- The nginx example in `docs/deployment.md` (`location /`) already
covers it.
- **Static UI from another origin:**
- The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for
the warm-up banner and wildcard preview origins.
- `build.sh` refuses `AUTH=true` there, since the server allows no
credentialed CORS.
- **Split containers:**
  - `CORESCOPE_IMAGE` is required.
- With user accounts on, `userManagement.publicBaseUrl` must be the web
container's public origin.
- The built-in MQTT broker is published on `127.0.0.1` unless
`MQTT_BIND` says otherwise.
- The compose network is `172.30.83.0/24`, with the web container at
`.200`. If Docker reports that the pool overlaps another network, set
`SPLIT_NET`.

## Testing

- `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with
new tests for busting every page and the WebSocket upgrade on `/`.
- `sh test-all.sh` passes. New unit suites:
- `test-static-build.js`: the build and its input checks, `same-origin`
included, cross-origin logins refused, and an unrelated `OUT` left
intact.
- `test-web-container.js`: the Dockerfile, Caddyfile and compose file
agree on packaging, forwarded paths, cache rules and the server port;
the server image has no default and MQTT stays on localhost.
  - The `api-base`, anonymous mode, badge and `dev-static` suites.
- `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes
47/47, the four pages included; it needs eslint, so CI runs it on E2E
shard 1.
- **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes
11/11. That is the static build on one origin against an API-only server
on another, which CI runs on E2E shard 1.
- **Split containers mode:**
- The image builds, and `caddy validate` passes (Caddy 2.11.4). The new
CI step passes locally, and fails when the Caddyfile stops forwarding
`/api`.
- Trusted proxies, with the stack up: the web container gets
`172.30.83.200`. With only that address trusted, a spoofed
`X-Forwarded-For` sent straight to a published 3000 is ignored, and one
sent through Caddy is replaced with the client's address. With the
subnet trusted, as the docs used to say, the spoofed address was
believed: a deny entry for it rejected the upgrade.
- With `docker-compose.split.yml` on the E2E fixture, seeded the way CI
seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both
through the web container and against the server directly.
  - Headers, checked with curl:
    - `no-cache` on `/`, `/index.html` and `/deployment.js`;
    - zstd on static assets;
    - `/api/*` proxied (`Via: 1.1 Caddy`);
    - a `101` WebSocket upgrade on `/ws`.
  - In Chromium:
- every API request and the WebSocket stay on the web container's
origin;
    - there are no page errors and no horizontal overflow at 390 px;
    - no badge shows on the `static` channel.
- Logins through the web container: with user management on and
`userManagement.publicBaseUrl` set to the web container's origin, a
Chromium run passed every step, all on the web container's origin with
no page errors:
- registration, and activation from the mail link (which uses
`publicBaseUrl`);
    - the session surviving a reload;
    - a CSRF-protected `PATCH /api/account`;
    - logout, then login with the form.
- The same login with the backend's origin as `Origin` is refused with
403. The server checks `Origin` against `publicBaseUrl`, not `Host`,
which is why the docs ask for it.

## Performance

There are no hot-path changes. API calls go through `CSApi.url()`, a
string concatenation. In the web container, Caddy serves files from disk
and compresses them; API responses pass through with one proxy hop, the
same as the all-in-one image's built-in Caddy.

## Notes

- **Per-IP limits:** behind the web container, as behind the all-in-one
image's Caddy, requests reach the server from the proxy's address. The
per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`,
the web container's fixed address. The docs say so.
- **`depends_on`:** the web container starts with the server, not after
its healthcheck, like the all-in-one image's Caddy. Gating on health
would make every `up` wait for the first check (30 s) and fail outright
when a large database loads for longer than the healthcheck allows;
until the server listens, Caddy answers `/api` with 502.
- **Not in this PR:**
- publishing a `corescope-web` image (CI builds and smokes it, without
publishing);
- Cross-origin login—including a static frontend and API on sibling
custom domains—is outside this PR. In the deployments supported by this
PR, account support requires the frontend and API to share one browser
origin, either served together or through a reverse proxy.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:56:09 +01:00
0e23c24847 fix(scope-stats): report total and split unscoped by route (#2364) (#2368)
Fixes #2364.

## Problem

Since #1838, `GetScopeStats` folds every non-transport transmission
(FLOOD 1, DIRECT 2) into `summary.unscoped`, while `transportTotal` and
`scoped` still cover transport routes (0, 3) only. The documented
`transportTotal = scoped + unscoped` no longer holds, and a client
computing an unscoped share against `transportTotal` gets >100% (the
reporter shows `transportTotal 7620`, `unscoped 14320`).

## Fix — additive, no existing field changes value

`summary` gains:

| field | meaning |
|---|---|
| `total` | all observed transmissions, routes 0–3 (= `scoped +
unscoped`) |
| `unscopedTransport` | route 0/3 with Code1 = 0000 |
| `unscopedFlood` | route 1 — unscoped by protocol |
| `unscopedDirect` | route 2 — unscoped by protocol |

Invariants (now asserted in `TestGetScopeStats`):

```
transportTotal = scoped + unscopedTransport
unscoped       = unscopedTransport + unscopedFlood + unscopedDirect
total          = scoped + unscoped
```

The non-transport count query keeps its single scan; it just splits
route 1 vs 2 with `CASE`.

On the design question in the issue: rather than changing what
`unscoped` means (the Scopes tab and #1842 rely on it), the split lets a
rollout view use `unscopedFlood` directly: repeaters apply region
rejection to flood routes only (`isRouteFlood()` in
`examples/simple_repeater/MyMesh.cpp`), so DIRECT traffic is not subject
to flood-region filtering. `docs/api-spec.md` now says so, spells out
the route mapping, and notes that `timeSeries` is transport-only (its
`unscoped` matches `unscopedTransport`).

## Frontend

Scopes tab uses `summary.total` when present and shows `N transport, N
flood, N direct` on the Unscoped card; falls back unchanged for older
servers. The cards are built by `scopeSummaryCardsHtml(summary)`
(exposed as `window._analyticsScopeSummaryCardsHtml`).

## Tested

- `go vet ./...` (cmd/server) clean, `gofmt -l` clean
- `go test -run ScopeStats ./...` — pass (incl. extended
`TestGetScopeStats` and `/api/scope-stats` route test)
- `node --check public/analytics.js`;
`tests/unit/test-issue-1375-scope-stats-fetch.js`,
`test-issue-1979-scope-adverts-by-role.js` pass
- new `tests/unit/test-issue-2364-scope-summary-cards.js` (vm sandbox,
11 assertions: shares of `total`, route split, fallback for servers
without `total`/split, partial split, empty window); full `test-all.sh`
green

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-11-15 12:24:27 +01:00
TrixandOpenCode a0b4e6cf5f fix(nodes): light-mode map follows tile picks, key and theme changes (#1470/#1420) (#2386)
fix(nodes): light-mode map follows tile picks, key and theme changes
(#1470/#1420)

## Problem

The node-detail maps (`#/nodes/<pubkey>` full page + side pane — and
their
reusers `node-reach-map.js`, `rx-coverage.js`) pick a different basemap
in
**light mode** than the rest of the app:

- hardcode Carto Positron: the user's *Light Mode Provider* customizer
pick
  and the operator's `map.tiles.lightDefault` are ignored
- render Carto's `API KEY REQUIRED` watermark when `/api/config/client`
lands after the map is created — the tile requests stay keyless until a
  remount (they come back HTTP 200, so nothing surfaces as an error)
- never react to a theme flip or provider change while the page is open

Dark mode is unaffected, which is why it only shows up in light mode.

Reproduce on `master`: open `#/nodes/<id>` in light mode with
`mc-light-tile-provider = carto-voyager` — the inset requests
`light_all`
while the main map requests `rastertiles/voyager`.

## Root cause

`_applyTilesToNodeMap` (`public/nodes.js`) asked `window.getTileUrl()`
for a
single URL string, and roles.js's light branch returned the frozen
`TILE_LIGHT` scalar — the dark branch was the only one that consulted
the
runtime registry (`public/map-tile-providers.js`), which is what owns
localStorage picks, server defaults, the Carto key, and per-provider
attribution/maxZoom/refUrl/invertFilter. map.js and live.js keep the
only
other registry-aware resolvers of the same logic.

## What this PR does

- adds one resolver, `MC_resolveTile(isDark)`, in map-tile-providers.js,
  returning the full descriptor
  `{ type, id, url, attribution, maxZoom, refUrl, invertFilter }`:
  - `url` is always a resolved string — function-typed providers are
    invoked at call time (contract of #1614)
  - one precedence rule everywhere: explicit operator overrides
    (`cfg.tiles.dark|light`, `map.tiles.darkUrl|lightUrl`, tracked via
    `MC_LEGACY_TILE_OVERRIDE`) > user/server registry pick >
    `TILE_DARK`/`TILE_LIGHT` constants — **this is a deliberate behavior
    change on map.js/live.js, see below**
- fully lazy: registry rebuilds / pick changes are picked up by the next
    resolve, so config landing after page load fixes itself
- roles.js `getTileUrl()` becomes a thin wrapper over it; the public API
and
the #1614 sandbox contract are unchanged; `getActiveTileProvider()`
keeps
  its documented null-in-light contract
- `nodes.js _applyTilesToNodeMap` resolves through the descriptor and
stays
reactive: one shared `mc-tile-provider-changed` listener + one
`data-theme`
MutationObserver re-sync every registered map in place — old layers
removed
before the new ones are added, never stacked; maps whose container left
the
  DOM are pruned at sync time, so SPA navigation cannot leak listeners
- map.js `_resolveTileUrl` / live.js `_liveResolveTile` delegate to the
  shared resolver (their legacy bodies remain only as sandbox fallbacks)
- provider `maxZoom` replaces the hardcoded 18/19 caps (OpenTopoMap 17
and
  USGS 16 used to 404 above their max zoom)
- analytics.js subpath minimap gets provider `maxZoom` as well

## Behavior change (deliberate): operator URL overrides now win on every
surface

On `master`, the two tile-selection mechanisms conflicted and each
surface
resolved the conflict differently:

- `map.js` / `live.js`: user/server **registry pick won** — an
operator's
  `tiles.dark|light` / `map.tiles.darkUrl|lightUrl` was effectively dead
  config on those pages
- the legacy roles.js path (nodes page, reach map, rx-coverage,
analytics
  minimap): **operator override won**

This PR unifies on **operator-override-wins**, because that is the only
reading that gives live operator config a consistent meaning:

- these keys are still applied today. The deprecation warning says they
  would "be ignored in v3.5.0", and v3.5.0 shipped 2026-04-08 — the
instance line is now at v3.16.0 and the keys still take effect, so they
  are live configuration, not dead letters. An explicit operator pin
  should not be silently defeated by a per-user pick on some pages and
honored on others; whichever way the mechanism eventually goes, it must
  behave identically everywhere until it is actually removed.
- user choice is not reduced in the general case: picks remain
  authoritative for every provider the registry offers, and overrides
  only exist where an operator explicitly wrote a literal URL.

If maintainers would rather have pick-wins (operator URL acts only as a
fallback default), flipping the precedence in `MC_resolveTile` is a
small,
well-tested change — say so in review and it will be reworked.

Related cleanup, out of scope here: the stale "ignored in v3.5.0"
deprecation warning in `cmd/server/config.go` should either be honored
or
reworded now that the project is past v3.16.0.

## No change without operator overrides or picks

## Verification

Unit (all deterministic, `vm`-sandbox based):

- new `tests/unit/test-tile-resolver.js` — 22 cases: light/dark pick
honored,
operator `lightDefault`, disabled-provider fallback, override
precedence,
lazy function-URLs, key-after-async-config, in-place swap without
stacking,
  detached-map prune, multi-map single-listener sync, provider maxZoom;
  listed in `test-all.sh`
- existing suites green: `test-issue-1420-tile-providers` (47),
`test-issue-1470-node-tile-helper` (6),
`test-issue-1614-tile-url-function`
  (3), `test-frontend-helpers` (79), `test-settings-sync`,
  `test-packet-filter` (105), `test-aging`
- every one of the 227 suites listed in `test-all.sh`, run individually:
  all green except `test-manage-workflow` (2 failures that are local-env
only — CRLF bootstrap artifact — and pass in CI; `sh test-all.sh` aborts
  there locally under `set -e`, which is why the suites were run one by
  one this time; the scripted pipeline runs fully in CI)
- `eslint` on all six touched files: 0 errors

Browser (headless Chromium against this branch, `/api/config/client`
delayed
~800 ms to force the config race):

- light + `/#/nodes/<key>`: 12 keyless tile requests before config, 12
keyed
  re-requests after — no reload, no stacked layer
- light pick `carto-voyager`: inset requests `rastertiles/voyager` from
the
  start (parity with the main map)
- theme flip on an open node page: `dark_all` swaps in in place
- dark defaults unchanged (`dark_all`, keyed after config)

The Playwright E2E suite was not run locally — no Go toolchain on the
workstation to serve the backend — CI will cover it.

## Scope / follow-ups (intentionally not here)

- `home.js` mini-map (OSM only) and the geo-filter mini-maps in the
customizers are untouched on purpose: they are outside this bug's path,
  and migrating them to `MC_resolveTile` later is mechanical

## Performance

Not a hot path: one registry lookup + string build per map creation and
per
provider/theme-change event (each is rare); no per-packet or per-render
work;
the listener footprint is O(1) app-wide (single shared listener pair + a
pruned registry), so there is no growth across SPA navigations.

Co-authored-by: OpenCode <git-llm@trix.moe>
Co-authored-by: OpenCode:GLM-5.3-Flash <git-llm@trix.moe>
Assisted-by: OpenCode:GLM-5.3-Flash eslint

---------

Co-authored-by: OpenCode <git-llm@trix.moe>
2026-11-11 18:22:42 +01:00
efiten 873e8b0e16 feat(rx-coverage): QR code and link to the operator's CoreDrive RX deployment (#2389)
## What

An operator who runs a CoreDrive RX deployment (the web and Android
companion app that feeds `clientRxCoverage`) can now point visitors of
the Coverage page at it.

- New optional config key `clientRxCoverage.appUrl`.
`/api/config/client` sends it as `clientRxAppUrl`, only while
`clientRxCoverage.enabled` is true and the value is an absolute `https`
URL with a host name. Any other value is left out and logged once at
startup with a `[config]` line.
- `#/rx-coverage` shows a panel under the subtitle when the field is
present: a QR code of exactly that URL (vendored `qrcode.js`, same call
as the node detail QR), the URL as a link, and one line on scanning it
in the app or opening it on a phone. Without the field there is no
panel. The page still needs no login.
- The subtitle named the app "CoreScope-RX" and linked to
`github.com/efiten/corescope-rx`, the app's old repository name. It now
says "CoreDrive RX" and links to `github.com/efiten/coredrive-rx`, or
carries no link when the panel already shows the deployment.

## Why

The CoreDrive RX Android app is not tied to one server: on first start
it asks for the address of a deployment. Scanning a QR code on the
Coverage page is the shortest way for a visitor to get there, including
people without an account.

Disclosure: CoreDrive RX (`efiten/coredrive-rx`) is my own project.

## Changes

- `cmd/server`: `config.go` (`validClientRxAppURL`, startup warning),
`types.go`, `routes.go`, `main.go`, `openapi.go`.
- `public/rx-coverage.js` (`appPanelHtml`, `appLinkHtml`),
`public/roles.js` (`window.MC_CLIENT_RX_APP_URL`),
`public/node-reach-coverage.css` (existing CSS variables only).
- Docs: `config.example.json`, `docs/client-rx-coverage.md`,
`docs/api-spec.md`.

## Tests

- 3 new Go tests in `cmd/server/client_rx_app_url_test.go`: URL
validation over 15 inputs, the endpoint field in 6 config states, the
startup warning. Full `go test ./cmd/server` passes.
- `tests/unit/test-rx-coverage-app-qr.js` (real `rx-coverage.js` and
`qrcode.js` in a vm): panel present and absent, QR content, HTML
escaping, subtitle link in both cases, old names gone. Added to
`test-all.sh`. All 6 `tests/unit/test-rx-coverage-*.js` pass.

## Not done

- No E2E test.
- Browser-checked on a local server with and without `appUrl` in the
light theme only. The QR keeps black modules on white in both themes,
unlike the node QR, which is inverted in dark mode; dark mode itself was
not checked in a browser.
- `cmd/ingestor` is unchanged; it does not need the URL.
- The same change runs on analyzer.on8ar.eu (a fork build) since
2026-11-10.
2026-11-11 00:25:32 +01:00
n30nexandn30nex 4140028165 fix(ui): keep slide-over close controls above navigation (#2304)
Fixes #2302.

Navigation covered tablet slide-over close controls and intercepted
pointer input. Two CSS substitutions put the panel/backdrop on the
existing modal layers. The regression uses actual close/backdrop clicks
and taps, retaining the existing keyboard, focus, scroll and resize
assertions.

Refreshed at `cd834802` by normally merging current master `d1bf4ca0`,
with zero conflicts. The exact three-file feature patch and original
RED/GREEN commits are preserved. Both feature tests are unchanged by
this refresh, including the supported 180-minute fixture window.

[Original RED
job](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212233915/job/121225789154):
40 passes, 17 pointer assertion failures, zero skips. The job completed
with failure before its aggregate run was cancelled. Merged #2289
resolves the old unrelated RF test failure.

Validation: four 57-case slide-over runs; three focus/scroll cases; 20
supplemental cases; 224 frontend suites; related navigation/gesture
checks; browser smoke with 149 passes/three existing fixture skips.
Parent rerun passes 57+3 after an initial-response timeout; tests were
unchanged. [Fresh
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42324795657)
passes, including the repeat gate.

Independent reviews found no introduced must-fixes; Kent Beck Gate:
PASS. Existing observer-focus and packet-navigation problems are
separately tracked in #2379 and #2380; runtime JavaScript is unchanged.

Browser verified: disposable Go fixture at `http://127.0.0.1:14964`.
E2E assertion added: `tests/e2e/test-slideover-1056-e2e.js`.

No new settings, dependencies, requests or hot-loop work.

## Preflight overrides

- check-red-commit: use the completed RED job’s 17 assertion failures;
its aggregate run was subsequently cancelled.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-11 00:07:56 +01:00
n30nexandn30nex 9e78e5c98d fix(live): show the newest certain node activity (#2283)
Fixes #2233.

Live node details use the newest certain activity from the node and
health responses for both status and Last Activity. Repeaters and rooms
with only recent shared-prefix relay evidence remain Uncertain.
Stale/missing values and a failed health request retain conservative
behavior.

Refreshed at `4d645dbc` by normally merging current master `d1bf4ca0`,
with zero conflicts. The three-file feature patch and original RED/GREEN
history are unchanged. [Original RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190088413)
remains; the preserved unit regression reaches 15 passes/eight assertion
failures locally.

Validation: all 224 frontend suites; 23 health-unit cases; eight health
browser groups including 12 Live cases; four RF airtime groups; and 48
Live cases across desktop/phone and light/dark. Parent reran the unit
and health browser suites. Backend trees are byte-identical to the
already-qualified master; no new native Go result is claimed.

The old unrelated RF test failure was reproduced during an actual theme
refresh. Upstream #2289’s locator waits correctly while retaining its
value/warning assertions; it is included in this refresh.

Browser verified: disposable Go fixture at `http://127.0.0.1:14963`.
Readable screenshots use the existing hidden-legend setting; the default
desktop legend overlap is unchanged.
E2E assertion added: `tests/e2e/test-health-status-e2e.js:259`.

Fixed-size work for one opened panel; no extra requests, dependencies or
settings. [Fresh
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42323634522)
passes. Independent correctness, UI and test-quality reviews found no
introduced must-fixes; Kent Beck Gate: PASS.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-11 00:07:42 +01:00
DSMoyleandClaude 9ddb29c946 fix/channels-mobile-header-overflow (#2382)
Fixes #2381. Drops the 56px header cap on mobile; title + Add on row 1,
region pills on a sideways-scrolling row 2. See the issue for details

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 00:00:39 +01:00
n30nexandn30nex afbae53b71 ui(nodes): add direct map and packet navigation (#2295)
Fixes #2290; addresses U6 of #1154.

Full node details, the side preview and tablet slide-over offer Open in
Map and View packets through existing deep links. A shared renderer
validates coordinates, including zero, and encodes identifiers without
extra fetches. Closing a preview preserves newly selected destinations
and titles; manual close returns to Nodes. Tablet targets stay separate,
and modified clicks retain native behavior.

Refreshed at `538cb169` by normally merging current master `d1bf4ca0`,
with zero conflicts. The original five commits and exact seven-file
feature patch are preserved. The [original RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203064817)
remains; the preserved RED also reproduces 39 assertions locally.
Upstream #2333 now supplies the correction for the previous unrelated
navigation-focus CI failure.

Current validation: all 225 frontend suites; 52 focused unit checks; 31
shortcut browser cases in each theme; 24 additional title/native-click
cases; navigation 60/60, branding 17/17 and slide-over 27/27. Parent
independently reran 52 unit and 31 browser cases. Native server and
ingestor suites pass. General browser smoke reports 149 passes and three
existing fixture skips. Unchanged serial retries resolved two documented
local test-server/navigation setup failures.

Browser verified: disposable local Go fixture at
`http://127.0.0.1:14954`.
E2E assertion added: `tests/e2e/test-node-quick-actions-e2e.js`.

Constant work for one detail view; no new requests, dependencies or
settings. [Final refreshed
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42318272198)
passes. Independent adversarial, UI and test-quality reviews found no
must-fixes; Kent Beck Gate: PASS.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-11 00:00:26 +01:00
n30nexandn30nex a59b42d8c2 fix(observers): keep charts and controls within narrow pages (#2344)
Fixes #2341.

Observer chart grids fit narrow screens, the doughnut stays square while
resizing, and empty canvases/long comparison choices no longer widen the
page. Radio formatting, requests and chart calculations are unchanged.

Refreshed at `f2ad231e` with a conflict-free normal merge of current
master `1577b57d`. Merged #2343 delayed-focus coverage and #2374 shared
payload labels are preserved. In response to review, the layout test now
checks one column at 760px and two at 900px, safely away from the exact
breakpoint. All original geometry, reload, disposal and request-count
assertions remain.

Original RED proof is retained: [four chart-width
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42011115036/job/127190202787)
and [four empty-data/control
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42016791855/job/127209299782).

Current validation: all 226 frontend suites and 68 ownership browser
cases pass. Parent independently reran four chart cases covering 32
layouts. Restoring current-base production through browser interception
makes all eight layout cases fail their overflow assertions; tests
remain sensitive to the bug.

Browser verified: disposable current-base Go fixture at
`http://127.0.0.1:14973`, desktop/mobile and both themes.
E2E assertion added: `tests/e2e/test-observer-detail-ownership-e2e.js`.

No new settings, dependencies, API calls or resize code. [Current
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42803853923)
and independent adversarial, visualization and Kent/TDD review pass at
`f2ad231e`.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-10 23:56:11 +01:00
n30nexandn30nex c03147cf0f fix(ui): share timestamp cells and retain future-clock warnings (#2387)
Partial fix for #1154 U10; follow-up to the maintainer's markup/warning
note on #2377.

Peer Interactions Last Contact now retains the same future-clock warning
as Nodes and Packets. One shared HTML renderer in app.js owns the
escaped timestamp, tooltip and existing warning markup; the three
surfaces delegate to it. Existing wrapper names/fallbacks, timestamp
preferences, placeholders and date algorithms are preserved. No new
settings, dependencies, requests or render triggers; work stays constant
per cell.

Tests-only RED `28e09e7c` reached [three unit assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42800749635/job/129777366878)
and [six browser missing-warning
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42800749635/job/129777366837).
GREEN `27f9fa49` leaves those tests unchanged and passes 115 unit
assertions, all 226 frontend suites and 10 browser cases. Parent
independently reran the unit and browser checks. Twenty-two
Nodes/Packets outputs match base exactly across 11 input/preference
cases.

Browser verified: disposable Go fixture at `http://127.0.0.1:14975`,
desktop/phone with UTC/local and relative/absolute preferences. Visible
warning/tooltip checks pass. The fixture's missing battery table and
existing final-row bottom-navigation overlap are unchanged limitations.
E2E assertion added:
`tests/e2e/test-issue-2218-node-analytics-stale-e2e.js`.

No production use of a test-only export, and no broader timestamp audit
claimed. [Current
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42804968476)
and independent adversarial, visualization and Kent/TDD review pass at
`27f9fa49`.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-10 23:55:59 +01:00
n30nexandn30nex 899a3ad0b1 fix(map): normalize saved viewport longitudes (#2378)
Partial fix for #2375. Restoring a saved longitude such as -352.7
previously put markers one world-width off-screen while tiles looked
correct. Saved positions now use Leaflet’s native wrapping and the
existing viewport validator; movement saves canonical coordinates.
Invalid saved values fall back to configured defaults. Valid URL
viewports and node highlighting keep precedence.

This repairs saved state, not live dragging across world copies. No new
settings, dependencies, requests, marker loops or render triggers; added
work is constant per initialization/existing moveend.

The reported-node URL diagnosis is separate: no-slash returns JSON when
enabled and intentionally 404 when disabled; trailing-slash returns
HTML. See #2372. The endpoint/gate remain unchanged; #2375 stays open
for maintainer reconciliation.

RED `cb453c0d` produced [14 passes and 10 intended CI
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42317069830/job/128232241526).
Production fix `e6b70dfd` passed CI, 224 frontend suites, 24 browser
cases, six estimated-marker and three mobile checks. Marker placement
changes from x=262754/offscreen to x=610/visible in the same fixture.

Review follow-up `99a4746e` changes only test data: configured and saved
centers now differ, while invalid storage independently expects the
default. Parent rerun passes 24/24; a browser-only mutation discarding
saved coordinates now fails seven cases. All previous assertions and
production files are retained. The single review finding is resolved.

Browser verified: disposable Go fixture at `http://127.0.0.1:14956`,
real Leaflet rendering.
E2E assertion added: `tests/e2e/test-map-nodes-pagination-e2e.js:129`.

Final follow-up CI is blocked by the unchanged observer disposal
assertion (#2342; fix #2343). A maintainer must rerun the workflow; this
remains draft.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:40:40 +01:00
n30nexandn30nex 8420a7ac0f feat(map): identify historical foreign-advert flags (#2371)
Partial implementation of #730 M2; the issue remains open.

The map marks nodes whose stored `foreign` Boolean is true in dot and
hash-label modes. Titles, accessible names and popups explain that an
advert was flagged outside the configured area at ingestion. This
historical flag does not establish current location, bridging, an
observer or a recent count. Analytics/history remain separate.

The warning uses existing theme tokens and stays clear of nearby markers
and phone controls. Footprint preparation adds O(n) work and numeric
metadata to the existing marker list (around 2K nodes), with one
typography read per applicable render and no per-marker DOM measurement
or requests. Existing bounded-offset placement retains its worst-case
quadratic search; this is not a speedup claim. No new settings or render
triggers.

RED: `4ad11148` produced [44 missing-warning
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42012004213/job/127193151694).
Follow-up `c4a157de` produced [11 warning-occlusion
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42023035345/job/127230364384).

GREEN `ad465e32`: parent browser check 66/66, 224 frontend suites, 38
existing map/route checks and CSS audit. Incorrect-copy controls also
prove the tests reject a current-location inference.

Browser verified: local Go fixture, desktop/phone, both themes. E2E
assertion added: `tests/e2e/test-map-foreign-advert-e2e.js` covers
keyboard access, missing/false flags, no-GPS, refresh, clustering,
overlays, zoom and resize.

## Preflight overrides

- check-red-commit: initial RED includes the new browser regression plus
necessary CI/inventory registration, with no production changes; CI
reached the intended assertions.

[Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42026358900)
passes. Three independent reviews were consolidated; both findings are
fixed and parent-verified. Kent Beck Gate: PASS.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:13:33 +01:00
CoderNemesis 65990873ad feat(map): deep-link the route path picker by observer or All view (#2376)
## Summary

The map route deep link (`#/map?packet=<hash>[&obs=<id>]`) always opened
in the canonical-path view. Two optional query params now choose the
path picker's starting view, so a shared link shows exactly what the
sender meant.

| Param | Effect |
|---|---|
| `paths=all` | Opens the picker's **All** view (every unique path), the
same as clicking **All** |
| `observer=<pubkey>` | Full public key or a hex prefix of 8+
characters, case-insensitive. Picks that observer's **earliest**
observation (by timestamp, then id), makes it the canonical route and
isolates its path row, the same as clicking the row. "Back to packet"
then links to that observation. |

Examples:

```
#/map?packet=974ad4ecefbe3761&paths=all
#/map?packet=974ad4ecefbe3761&observer=a1b2c3d4
```

When params are combined, `obs=` wins over `observer=`, and `observer=`
wins over `paths=all`. An unknown or malformed observer falls back to
the current default view and logs a console warning.

## Changes

- `public/route-view-utils.js`: two new pure helpers.
- `resolveDeepLinkObservation(observations, params)` picks the
observation and the initial view.
- `pathGroupKey(rawHops, filterHops)` is the path picker's grouping key.
It was written out twice in `route-view.js` and now lives in one place,
which keeps the deep link and the picker rows in sync.
- `public/map.js`: `loadRouteFromDeepLink` uses the helper and passes
`initialPathView` through `drawPacketRouteMulti` to the renderer.
- `public/route-view.js`: the picker count and rows use `pathGroupKey`.
`sidebar._applyPathView` applies the view through the existing All and
row click handlers, in the frame after the route's initial fit.

## Performance

This doesn't touch a hot path. The work is one pass over a single
packet's observations (tens at most) plus a scan of the picker rows, and
it runs once per deep-link load. It adds no API calls.

## Config / customizer

No new configurable values.

## Tests

- New unit suite `tests/unit/test-map-deeplink-path-params.js` (29
checks). It covers prefix vs full key, case, short/non-hex keys,
earliest-observation choice with tie-break, the precedence rules,
`pathGroupKey`, and the source wiring.
- `tests/unit/test-issue-1633-hide-1byte-hops.js` replayed a copy of the
key logic. It now tests the real `pathGroupKey`.
- New Playwright suite `tests/e2e/test-map-deeplink-path-params-e2e.js`
(5 checks) on the seeded #1486 packet. It's registered in
`scripts/non-unit-tests.json` and CI shard 3. With the `route-view.js`
change removed, 3 of its 5 checks fail.
- Local runs against the CI-prepared fixture: `test-all.sh` passes.
Route-map a11y E2E 20/20, map mobile E2E 3/3, `test-e2e-playwright.js`
149/152 (3 skipped by the fixture, 0 failed).

## Browser validation

Checked in a browser against a local server, using a scratch copy of the
fixture that gives the seeded packet multi-hop paths through real GPS
nodes. One observer had two observations, and the API listed the later
one first.
- `observer=<prefix>` picked that observer's earliest observation and
drew only its path, with numbered markers and the map fitted to it.
- `paths=all` drew the same picture as clicking **All** by hand.
2026-11-08 17:13:22 +01:00
Marek WojtaszekandClaude Opus 5.5 3100d7a4c9 fix(ui): route remaining payload label surfaces through payload-labels.js (#2373) (#2374)
Fixes #2373

## Summary

Finishes the #1799 / #1804 migration: every frontend surface that names
a payload type now reads it from `public/payload-labels.js`. Node
Analytics, Observer detail and Observer comparison showed GRP_DATA,
MULTIPART and RAW_CUSTOM as `Type 6` / `Type 10` / `Type 15`. The map
route sidebar had RAW_CUSTOM on id 12; the firmware uses `0x0F`.

## Changes

- **`public/payload-labels.js`**: new
`PayloadLabels.api.displayName(id)` returns the canonical short label,
or `Type N` for ids the firmware does not define (e.g. a corrupted
header byte stays visible as `Type 13`). It lives on `.api` only; the
legacy root aliases are not extended.
- **`public/node-analytics.js`, `public/observer-detail.js`,
`public/compare.js`**: drop the local `PAYLOAD_LABELS` maps and use
`displayName()`.
- **`public/nodes.js`**: drop the local `PAYLOAD_TYPES` map; Recent
Adverts uses `shortById()` and keeps its `Packet` fallback.
- **`public/app.js`**: drop `PAYLOAD_TYPES`; `payloadTypeName()` uses
`shortById()` and keeps its `UNKNOWN` fallback.
- **`public/map.js`**: drop `PAYLOAD_TYPE_MAP`; `inferredType` uses
`enumNameById()`, which fixes RAW_CUSTOM (15 → `RAW_CUSTOM` instead of
`OTHER`).
- **`public/packets.js`**: `typeName()` is now `displayName` (same
output, one implementation).

Consumers resolve labels at render time rather than at file load, so the
many unit sandboxes that load these files without `payload-labels.js`
keep working. `index.html` already loads `payload-labels.js` before all
of them.

## Tests

- **New `tests/unit/test-payload-labels-surfaces.js`** (listed in
`test-all.sh`):
- drift guard: fails if any `public/*.js` other than `payload-labels.js`
pairs a numeric id with a short label or firmware enum name (it caught
the `map.js` copy, which was not on the original list);
- `displayName()` for every enum, string ids, and the `Type N` fallback;
- loads the real `node-analytics.js` in a vm sandbox and asserts the
doughnut labels for ids 6, 10, 15 and 13.
  - Before the fix: 11 failures. After: 82 passed.
- **`tests/e2e/test-issue-1799-label-vocab-e2e.js`**: two new steps
intercept `/api/nodes/:pk/analytics` and `/api/observers/:id/analytics`
(the fixture has no GRP_DATA in the store), inject ids 6, 10, 15 and 13,
and read the labels from `Chart.getChart()`. Against upstream `master`:
11 passed, 2 failed (`["Type 6","Type 10","Type 15","Type 13"]`). With
this branch: 13 passed.
- **Sandbox updates**: `test-compare-overlap.js` and
`test-observer-detail-ownership.js` now load `payload-labels.js` before
the page script, as `index.html` does.
`test-issue-1418-raw-hex-extraction.js` asserted the hand-rolled
`PAYLOAD_TYPE_MAP` literal; it now asserts that `inferredType` uses the
canonical map and that no local map exists.
- Unit suites, each `test-all.sh` entry run on its own: 223 pass on this
branch vs 222 on `master` (+1 new suite). `test-manage-workflow.js` and
`test-preflight-xss-gate.js` fail identically on `master` in my Windows
environment (CRLF in the shell fixture, no `python`); they are unrelated
to this change.
- E2E, run against a local server on the CI fixture (freshened, seeded
and migrated as in `deploy.yml`): `test-issue-1799-label-vocab-e2e.js`,
`test-charts-fluid-1058-e2e.js`, `test-issue-1147-section-order-e2e.js`,
`test-issue-1640-compare-discovery-e2e.js`,
`test-issue-1648-m2/m3-icons-e2e.js` and
`test-observer-detail-ownership-e2e.js` all pass.
`test-e2e-playwright.js` fails one step (`Version info lives on Perf
dashboard`, `#navStats` wait timeout) with the same result on `master`
locally.

## Performance

No hot-path change. Every lookup is still one object-key read per
packet: the packets-table sort accessor calls `displayName(t)` instead
of `TYPE_NAMES[t] ?? ...`. No new loops, requests or allocations.

## Browser validation

Checked against the local fixture server, with the analytics responses
patched in the page to contain GRP_DATA:

- Node Analytics → Packet Types legend: `Group Data, Advert, Channel
Msg, Direct Msg, Multipart, Raw Custom, Type 13`
- Observer detail → doughnut and the first Recent Packets row read
`Group Data`
- Observer comparison → packet table rows read `Group Data`
- Map route deep link (`#/map?packet=…`) renders the route panel; no new
console errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 17:13:10 +01:00
n30nexandn30nex 6618820836 fix(nodes): show timestamp context for Last Contact (#2377)
Partial fix for #1154 U10. Peer Interactions Last Contact now shows
timestamp context through the existing shared formatter: relative text
with an absolute tooltip, or the reverse when absolute display is
selected. Existing timezone, precision and custom-format preferences
apply; missing/invalid values remain a dash. Other U10 surfaces and
tracking items stay open.

The production change only formats the existing cell. Work stays O(P)
over the existing maximum 20 peers; no requests, settings, dependencies,
caches or render triggers are added. Tooltip and text are HTML-escaped.

RED `c47cce20` reached [11 unit assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42316165311/job/128229216777)
and [six browser missing-tooltip
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42316165311/job/128229216764).
GREEN `2c29eccb` keeps those tests unchanged and passes 82 unit
assertions, 10 focused browser cases (parent rerun), and all 224
frontend suites. Existing browser smoke reports 149 passes and three
fixture skips. One separate navigation timeout passed on an unchanged
serial retry.

Browser verified: disposable Go fixture at `http://127.0.0.1:14953`,
desktop/phone; tests retain the real API response and supply
deterministic peer contacts. Its unrelated missing battery-metrics table
is a fixture limitation.
E2E assertion added:
`tests/e2e/test-issue-2218-node-analytics-stale-e2e.js:123`.

[Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42317025356)
passes. Independent adversarial, UI and test-quality reviews found no
must-fixes; Kent Beck Gate: PASS.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:13:00 +01:00
n30nexandn30nex 0e2ed89829 fix(packets): explain observer-report coverage (#2370)
Red commit: `e86294b6` ([four intended CI assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42009863423/job/127185896565)).

Partial fix for #667, Part B. The Packets page now explains that its
counts come from observer reports received by this instance—a partial
view, rather than network-wide totals or device firmware counters. The
note remains visible on phones and describes the table for assistive
technology.

This does not change TX/RX storage, counts, filtering or grouping, and
leaves the rest of #667 open. It invents no contributing-observer number
and makes no RX-only claim. No new settings, dependencies, requests or
processing work.

Validation on `08036329`: parent-run browser regression 10/10; required
frontend unit suites 932 assertions. The RED run completes six existing
cases and fails the four new missing-context assertions.

Browser verified: disposable local fixture at `http://127.0.0.1:14943`
(desktop, phone, populated/empty filters, grouping and refresh). E2E
assertion added: `tests/e2e/test-issue-1122-packets-filter-ux-e2e.js`.

Updated CI passes; independent adversarial, UI and test-quality reviews
found no must-fix issues.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:12:49 +01:00
n30nexandn30nex c4dc46f349 test(e2e): wait for comparison focus before disposal assertions (#2343)
Red commit: `eedadc43` ([four intended CI assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40586787653/job/122480320591)).

Fixes #2342. The disposal test could capture comparison results before
the router's accessibility-focus frame added the heading tabindex, then
mistake that legitimate setup change for a late observer response.

Wait for the actual tabindex in attached state before capturing the
baseline, including when the mobile heading is hidden. A controlled
late-focus variant proves the regression. Full HTML equality, chart
cleanup, route and control assertions remain intact; no production
behavior or settings change.

Current head `cb537ebe` includes current master while preserving the
original RED/GREEN commits and newer progressive-render coverage. Local
validation: ownership browser suite 60/60 (independently rerun by
parent), navigation 60/60, branding 17/17 and all 224 standalone
frontend suites.

Browser verified: real local Go fixture server at
`http://127.0.0.1:14944`. E2E assertion added:
`tests/e2e/test-observer-detail-ownership-e2e.js`.

Updated CI passes; independent adversarial, UI and test-quality reviews
found no must-fix issues.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:12:03 +01:00
JadeandClaude d1bf4ca042 fix(docker): build natively on arm64 hosts (#2359)
Building the image natively on an arm64 machine (a Raspberry Pi 4 or 5
on 64-bit OS, Apple silicon) fails at the first `RUN` of the build
stage:

```
=> ERROR [builder  2/40] RUN apk add --no-cache curl xz && ...
0.841 exec /bin/sh: exec format error
```

**Cause:** the Dockerfile declares `ARG BUILDPLATFORM=linux/amd64`. A
default on that ARG overrides the value BuildKit sets for the machine
doing the build, so an arm64 host pulls the amd64 `golang` image and
can't run it. CI didn't catch it because its arm64 image is cross-built
on an amd64 runner, where `linux/amd64` was correct anyway.

**Fix:** declare `ARG BUILDPLATFORM` without a default. BuildKit, the
default builder since Docker 23 and what buildx uses, fills it in on
every host. amd64 builds and the CI cross-build are unchanged.

**Workaround on current master:** `docker build --build-arg
BUILDPLATFORM=linux/arm64 .`

**Test:** `tests/unit/test-dockerfile-buildplatform.js` fails on the old
Dockerfile and passes on this one. It checks three things:
- `BUILDPLATFORM` has no default.
- The Go builder stays pinned to `$BUILDPLATFORM`.
- No stage hard-codes a platform.

It's wired into `test-all.sh`.

Found while building the image on a Raspberry Pi running Raspberry Pi OS
Trixie (arm64) with Docker 29.


---

Co-authored-by: Claude <noreply@anthropic.com>
2026-10-31 12:13:29 +01:00
n30nexandn30nex d083f9ade9 test: cover partial observer chart construction failures (#2354)
Partial follow-up for #1778; addresses the chart-cleanup test gap
identified in the review of #2339.

The observer detail unit harness now throws during the second or fourth
chart constructor. Both cases require existing chart instances to be
destroyed, metadata/calibration to remain visible, the error to be
rendered as text, and a later time-window retry to recover without extra
requests.

Production cleanup already behaves correctly; this PR adds coverage
only. The parent-run ownership suite passes 30/30. Removing the actual
cleanup produces exactly the two intended assertion failures. The
remaining unit sources and runner match the current-master baseline
whose 222 frontend suites passed; only this updated unit file differs
within that test set.

TDD scope: no production change or new behavior. The negative control
removes existing cleanup to prove the new assertions detect its loss. No
configuration, dependencies or runtime performance changes. Exact-head
CI passed, including the browser shards. Independent adversarial,
correctness and TDD reviews found no must-fix items.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-31 12:00:06 +01:00
efitenandClaude Opus 5.5 bb3c8cc001 fix(rx-coverage): My coverage and the observer filter on the noise layer (#2326)
"My coverage" and a selected observer only narrowed the signal layer on
the coverage page. On the noise layer the toggle stayed pressed, but
`/api/rf-noise` was called without `mine` or `rx` and returned every
companion's samples.

## Changes

- **Backend:** `/api/rf-noise` now accepts `rx` and `mine=1`, with the
same session rules as `/api/rx-coverage`: 401 without a session, 403 for
a device token, 404 with user management off. The `rx_pubkey` filter
moved into 1 helper, `appendRxFilter` (`cmd/server/rx_dashboard.go`),
which both queries use. Both parameters are in the OpenAPI entry.
- **Frontend:** `public/rx-coverage.js` fetches both layers through
`fetchLayer`. It builds the URL with the time window, observer and
toggle, and judges the reply with the existing `coverageReply`. The
request sequence (`signalSeq`, now `drawSeq`) covers both layers, so the
noise layer also drops a superseded reply. Before, a slow unfiltered
noise reply could overwrite a newer filtered one.

## Performance

The noise query gets 1 extra `WHERE` clause on `rx_pubkey`. There are no
new requests: the page still makes 1 request per redraw.

## Verification

- New Go tests in `cmd/server/rf_noise_mine_test.go`: `rx` narrows the
result (case-insensitive), `mine=1` returns 401 without a session,
returns empty with nothing linked, returns the linked companion's cells,
returns 403 for a device token, and returns 404 with user management
off. Both failed before the fix.
- `tests/unit/test-rx-coverage-mine.js` is updated for the new
`coverageUrl(path, ...)` signature and checks that both layers fetch
through `fetchLayer`.
- New step in `tests/e2e/test-rx-coverage-noise-e2e.js`: a
`?layer=noise&rx=...` link sends `rx`, and clicking "My coverage"
refetches with `mine=1`. It fails against the current master frontend
and passes with this change.
- Local runs (Windows): `go test` and `go vet` in `cmd/server`,
`test-all.sh`, and the noise, gaps and viewport E2E in Chrome. All
green. I did not run the full Playwright suite locally.

## Not in this PR

- The "nothing received" gap cells still skip `mine=1`, unchanged: the
track query cannot narrow to a set of companions yet.
- The time window (`days`) already worked on the noise layer, so it is
not touched.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:39:02 +01:00
TeTeHackoandClaude Opus 5.5 417c8dd00a feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) (#2300)
Closes #2270, closes #2271.

The ingestor took only the confirmed region scopes out of an observer's
`/neighbors` report (#1865) and dropped the rest. Nothing could
therefore say when an observer last reported, or whom it heard.

## Ingestor

`observer_neighbor_reports` (created in `dbschema.Apply`) keeps the
latest report of each observer, one row per observer, replaced by every
live report.

The report is unauthenticated, so `neighbors[]` is stored sanitised:

- only 64-hex keys
- `snr` and `heard_secs_ago` finite and in range
- `status` limited to `[a-z_]`
- `scopes` clamped
- at most 500 rows

`total_neighbors` is believed only when it is not below the number of
rows that arrived.

A **retained replay** only fills a missing row, and it is dated by the
report's own `timestamp`. After an ingestor restart, a long-dead
observer therefore does not look as if it just reported.

## API

The table is optional for the server (no `AssertReady` change); without
it the fields are simply absent.

- `GET /api/observers`: `neighbors_report_at` and `neighbors_count`,
from one batch query in the existing cached builder.
- `GET /api/observers/{id}`: also `neighbor_report`, which is
`{received_at, reported_at, neighbor_count, queried_neighbors,
self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr,
heard_secs_ago, status, scopes}]}`. Names and roles come from the node
table in one query.

## Frontend

`public/observer-neighbors.js` is shared by both pages and escapes
everything.

- **#2270, observers list:** a sortable **Neighbors** column, e.g. `3h
ago · 12`, or `—` when the observer never reported. The same value
appears in the slide-over.
- **#2270, observer page:** **Neighbors Report** and **Neighbors** stat
cards.
- **#2271, observer page:** a **Neighbors** table listing each neighbor
by name (a link to the node) or key prefix, with SNR, heard, scope query
status and scopes. The heading says when the list is shorter than the
count, e.g. `(2 of 3 listed)`.

## Tests

- **ingestor:** sanitising, the cap, live replace, the retained rules
(fill only, dated by the report, undated ignored) and the MQTT topic end
to end. The existing #1865 scope tests are unchanged and still pass.
- **server:** the list summary, the detail with names, and both
endpoints without the table.
- **frontend:**
  - unit tests of the helpers
- an XSS case for every rendered field; I checked that it fails when
escaping is removed
- `test-observers-headings` now expects the new column after `Last
Packet`
- **E2E:** `test-observer-neighbors-2270-e2e.js` intercepts the observer
endpoints, since the fixture has no reports. It is listed in
`scripts/non-unit-tests.json` and on shard 2.

Gates that pass:

- `go test -race` in `cmd/server`, `cmd/ingestor` and
`internal/dbschema`
- `sh test-all.sh`
- `eslint@8`, with 0 errors
- the XSS diff gate
- these E2E suites: the new one,
`test-issue-1639-observers-sort-e2e.js`,
`test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js`

Performance: the list adds one primary-key-sized query per cache fill,
and the detail adds two small queries per request.

Also fixes #2330 (folded in from #2337, second commit).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:36:53 +01:00
n30nexandn30nex 202bafea28 fix(observers): show metadata while analytics loads (#2339)
Partial follow-up to #1778, limited to observer details; other pages
remain separate.

Observer identity, status and radio information now render when the
basic request finishes. Analytics and clock calibration load into their
own regions, so either can finish first. An analytics failure preserves
usable metadata. Identity failures and obsolete responses cannot
overwrite the current view, and failed chart construction is cleaned up.

The same three requests start together. Generation ownership, request
deduplication, date-window behavior and disposal remain covered. No new
fetches, settings, dependencies or customizer options; secondary replies
preserve the actual metadata DOM node.

[RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426343054/job/121932999182)
proves 14 new assertion failures while all 11 existing unit cases pass.
Local RED browsers reproduce 20 new failures while all 32 old cases
pass. GREEN `52eabad7` passes 221 standalone suites, 28 ownership unit
checks and 56 browser cases across desktop/mobile and both themes; the
parent independently repeated the 56 browser cases.

All three independent review roles passed. Held-response assertions
prove the rendering behavior without claiming a server-latency speedup.
[Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427087948)
passes on this exact head, including all three browser shards, the
repeated slide-over stability gate and the architecture build.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:34:12 +01:00
n30nexandn30nex 1ed5e4ab3f fix(compare): exclude observer TX-only packets from overlap (#2346)
Red commit: `39295da8` ([intended CI assertion
failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)).
Green commit: `d1bda496`.

Observer comparison counted an observer's TX-only packets as receptions,
producing false overlap. It now requests `excludeTx=true`: the selected
observer and non-TX direction must match the same observation in memory
and SQLite v2/v3. Normal packet lists and stored TX records remain
unchanged; missing directions retain legacy compatibility.

Refs #667. This completes a comparison slice, not the issue's remaining
observer counters or #671's fleet/SNR feature. Representative display
fields and existing time-filter semantics are unchanged. API/OpenAPI
docs describe the filter and multi-node bypass.

Validation:
- RED: Go tests compiled with 32 intended failures/16 control passes;
unit and browser regressions reproduced false overlap. CI reached the
new unit assertion.
- GREEN: all 48 packet cases pass, plus grouped-cache isolation and
OpenAPI tests. The added cache regression caught a malformed-radio
cache-key collision before correction.
- Full local server suite: 174s; ingestor: 185s; `npm run test:unit`:
pass. Final focused packet/grouped/OpenAPI checks: pass.
- E2E assertion added to
`tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks
pass, including correct 1/2 reception totals and 50% overlap.

Performance: still two capped bulk requests, indexed candidate lookups
retained, one direction check within the existing observation scan/SQL
predicate, no new persistent structures. No configuration/customizer
changes.

Final-head CI passed; independent reviews found no must-fix items.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:34:06 +01:00
n30nexandn30nex c211ef7643 feat(nodes): show relay activity from certain forwarding evidence (#2336)
Fixes #2331 with the additive API option.

The Last Relayed row shows green **actively relaying** only when
`last_certain_relay` falls within the existing
`healthThresholds.relayActiveHours` window. Missing, invalid, future or
expired evidence adds no idle/alive claim. Existing `relay_active`,
attributed timestamps and counts retain their meanings.

The optional node list/detail field uses the existing certain-relay
path: an eligible observed frame, matching hop, unique prefix, no
retired namesake and existing signature checks. Originating an advert
does not count. Forwarding another node's advert can count, even with
zero non-advert attributions. Adverts with an unknown origin cannot
establish forwarding.

Evidence uses transmission `first_seen`, not late observation receipt
time, and follows node-membership eviction. Snapshots take one read lock
over requested keys, O(page size), with no packet scan or new database
query. Work per qualifying hop stays bounded. No new dependency or
setting; configuration documentation and OpenAPI explain the
distinction.

The test-first history includes assertion-failing RED commits and GREEN
fixes. The initial RED's nil-return compile stub is explicitly permitted
by `docs/agents/TDD.md`. Local native server/ingestor suites, focused
race checks, 809 frontend helper assertions and 70 browser cases pass.
Browser coverage includes both themes and full/side/tablet detail views.

[Review regression
RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40421464618/job/121916723339)
proves both malformed-origin failures. All three independent review
roles pass at `d1fe79cb`. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40422655728)
passes on that exact commit, including all three browser shards and the
two-architecture image check.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:33:59 +01:00
efitenandClaude Opus 5.5 15e2d2a4a7 fix(packets): read HB from the row's own raw_hex header (#2351)
## Problem

The HB column on the packets page can show `4`. Path mode 3 (hash size
4) is reserved: `Packet::isValidPathLen` in `src/Packet.cpp` returns
false for it, and `Dispatcher::checkRecv` drops such a packet. So HB 4
is never a real value.

The three HB cells in `public/packets.js` found `path_len` with
`getPathLenOffset(p.route_type)`, the route type of the
**transmission**. The raw_hex in a child row (and in a live-updated row)
is an **observation's** raw_hex. The packet hash does not cover the
route bits, so the same packet is heard as FLOOD by one observer and as
TRANSPORT_FLOOD by another, and the transmission keeps the route type of
its first observation.

Example from a production DB (transmission 3645334):

| source | raw_hex start | path_len at |
|---|---|---|
| transmission | `01 12 3A646A…` (FLOOD) | byte 1 |
| observation | `00 E2E0 0000 1D 3A646A…` (TRANSPORT_FLOOD) | byte 5 |

Read at the FLOOD offset, the observation gives `0xE2`, `0xE2 >> 6 = 3`,
`+ 1 = 4`. In the last 300,000 observations of that DB, about 4,300 have
a header route type that differs from their transmission's.

## Fix

`pathHashSize(rawHex)` in `public/app.js` already takes the route type
from the header byte of the same raw_hex, and the detail pane uses it.
The three HB cells now call it through a small `hashBytesText` helper
instead of carrying their own copy of the offset logic.

Behaviour change: a raw_hex with no hash size (reserved mode 3, a DIRECT
packet with no hops, missing raw_hex) shows the same placeholder as
TRACE rows instead of a number. Before, missing raw_hex showed `1`.

## Tests

- New `tests/unit/test-hb-column-header-route.js` (7 cases, registered
in `test-all.sh`): transport raw_hex on a FLOOD transmission gives 1,
flood raw_hex with 2-byte hashes on a TRANSPORT_FLOOD transmission gives
2, reserved mode 3 never gives 4, for the helper, the flat row and the
group header. It fails on `upstream/master` with `'4' !== '1'`.
- `sh test-all.sh`: all standalone frontend suites pass.
- Browser: local server on a migrated copy of the E2E fixture, with one
observation rewritten to a TRANSPORT_FLOOD raw_hex. The expanded group
shows HB 2 on all three rows. The old code reads `0xE2` on that child
and shows 4.

## Performance

Same per-cell work as before: one header parse and one byte parse per
rendered row, O(1), no new allocations beyond the existing regex test in
`pathHashSize`. Only visible rows are rendered.

## Not done

- The child-row call site is not hit by a unit test: child rows only
render when the group is in `expandedHashes`, which the test API does
not expose. It is covered by the helper test and the browser check
above.
- The HB sort accessor (`hash_byte_count` / `hash_size`, sort key `hb`
in `packets.js`) is unchanged; it does not read raw_hex.
- `transportBadge(c.route_type)` on child rows still follows the stored
route type, not the observation header. Left out of scope.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 06:42:51 +01:00
n30nex 3bf678709f fix(ui): preserve navigation and menu focus during delayed updates (#2333)
Fixes #2332. Fixes #2335.

Two delayed UI updates could undo user actions. Rebuilding the More menu
removed its focused link; a late packet-list response could replace a
newly requested Nodes URL before `hashchange` ran.

The menu now preserves the selected destination when it stays in More or
moves inline, closes an empty menu, and leaves other controls' focus
alone. Packet URL updates now require the packets route, including its
existing detail/query forms. Observation selection and encoded filters
remain supported.

The changes add one lookup across the existing 12 navigation links and a
URL ownership check. No packet loop, API request, dependency, setting or
customizer change is added.

[Packet
RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409110743/job/121874676123)
fails four wrong-route assertions. [Menu
RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409898375/job/121877663579)
fails the intended focused-destination assertion after the packet
correction removes the competing navigation failure. GREEN is
`57522143`.

Local combined validation passes all 221 standalone frontend suites, 60
navigation browser checks, 17 branding/route checks and 11
filter/deep-link checks. The packet regression controls delivery of a
real API response; substituting the original code still fails it.
Existing exact title assertions remain; diagnostics now report the
actual title. Backend source trees are unchanged from the tested
baseline.

[Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40410894023)
passes on `57522143`, including all three browser shards and both
architecture builds. Three independent source reviews found no blockers.
Final repository checks and after-push browser verification pass. Ready
for maintainer review.
2026-10-28 21:55:40 +01:00
efitenandClaude Opus 5.5 d29957fbdc feat(observer-logins): lock the shared login out of migrated observers (#2348)
Follow-up to #2306 / #2319. With personal observer logins, a migrated
observer still is not safe from the shared login: the hook answers
`ignore` for the shared login, so EMQX's built-in database lets it
publish under any observer's topic, including one whose owner has moved
to a personal login. This PR lets an operator close that per migrated
observer, before the shared login is removed altogether.

## Changes

**users.db v8.** `observer_bindings.confirmed_at`: the moment the
owner's personal login first published under that pubkey.
- An auto-bind (the owner's first publish) is confirmed at once.
- An admin assignment stays unconfirmed until the owner's login
publishes under that pubkey; that publish sets it.
- Reassigning to another user or unbinding clears it; reassigning to the
same owner keeps it.
- Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin`
rows stay NULL.

**Config.** `userManagement.observerLogins.sharedUsernames` (default
empty): the broker's shared login names. A name starting with `obs_` is
refused at startup.

**Authz hook.** For a login in `sharedUsernames`, a publish under
`meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is
confirmed, whatever the number of topic segments (the ingestor
attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey).
Logged as `publish denied: user=<name> observer=<8 hex>: confirmed
owner`. Every other request from a shared login keeps `ignore`, so the
built-in database decides as before. Personal logins are unchanged.

An orphaned binding (owner account deleted) does not count as confirmed.

**Admin.** The observers table of the Observer logins tab gets a
Confirmed column; the bindings JSON gets `confirmedAt`.

**Docs.** `docs/observer-mqtt-logins.md`: "Locking the shared login out
of migrated observers" (what it denies, how to see `confirmedAt`, the
risk below). `config.example.json` has the key.

## Risk for operators

An observer that is confirmed but still runs a second uploader on the
shared login loses that second uploader once its name is in
`sharedUsernames`. The deny log line names the observer.

## Tests

- `internal/users`: v7 to v8 migration on a real v7 database, auto
confirm, admin assignment then owner confirm, non-owner and repeated
confirm, reassign clears, orphan reassign.
- `cmd/server`: the decision table (confirmed, unconfirmed, unbound,
non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty
and `client` iata, lowercase key; owner publish confirms, confirm
failure still allows and retries; reassign, unbind and user removal
clear the flag; the index load against a real store; config validation;
the hook and admin endpoints end to end.
- `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an
auto-bound observer.
- Local: `internal/users`, `cmd/server` (two Windows-only tests skipped,
as on master) and `cmd/ingestor` green. The E2E assertion was not run
locally; CI runs it.

## Not in this PR

- Using `confirmed_at` for verified `/neighbors` reports: a separate PR,
built on #2300.
- Removing the shared login: still an operator step after migration.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:10:41 +01:00
TeTeHackoandClaude Opus 5.5 77a5f14925 feat(observers): radio setting per observation from the observer's radio history (#1869) (#2292)
Part of #1869.

## Problem

`observers.radio` keeps only the **last** `MHz,kHz,SF,CR` an observer
reported. Once an observer is retuned (say from SF7 to the EU narrow SF8
setting to listen to a neighbouring mesh), every packet it ever heard
reads as heard on the new setting, and there is no way to tell an SF7
packet from an SF8 one.

## What this does

As #1869 proposes, the packet data model is untouched: the setting of an
observation is resolved from its observer and time.

**Ingestor** (all schema in `dbschema.Apply`)

- `observer_radios`: one row per stretch of time an observer, and each
radio of a multi-radio node, stayed on one setting. Every live status
extends the current row and a change starts a new one; retained replays
never extend a row. On the first run it is seeded with every observer's
current radio, which is what the UI already showed for the past.
- Settings are stored canonical through `dbschema.CanonicalRadio`, so
`869.4320068,62.5,7,5` and `869.432,62.50,7,5` are one setting.
- `observation_radios` (`observation_id → rx_radio_id`): multi-radio
openHop nodes tag each packet with `rx_radio_id` and list `radios: [{id,
radio}]` in their status, so their observations resolve per radio.
- The table is sparse: single-radio nodes never send the field, so
`observations` and every query that scans it stay untouched.
  - The row id comes from the observation upsert via `RETURNING id`.
- Rows are pruned with their transmissions, and a purged observer takes
its history along.

**Server** (read-only)

- No `AssertReady` change: an absent table reads as empty and is probed
again later, the same rule as `advert_route_evidence`.
- The history is held in memory. The live poller checks it on every tick
(two primary-key lookups), so a retune or a new `rx_radio_id` is known
before the observation is broadcast. A full reload runs every 10 min.
- `radio` is added to observations, packets, grouped rows (taken from
the header observation) and the live feed.
- `GET /api/packets?radio=MHz,kHz,SF,CR` filters by setting.
- Combined with `observer=`, it means *that* observer heard it on that
setting.
- A radio query is served from memory instead of falling back to SQL,
which would return the window unfiltered.
- `GET /api/radios` (named struct) and `radio_history` on `GET
/api/observers/{id}`.

**Frontend**

- Packet filter fields `radio`, `freq`, `bw`, `sf`, `cr`, e.g. `sf == 8`
or `freq > 869.5 && bw == 62.5`.
- A Radio row in the packet detail and a Radio column in its
observations table (only when the server reports one).
- A radio history table on the observer page once there is more than one
stretch.
- `RfParams.radioText` formats the setting. It escapes all five
characters and returns non-numeric input as it came, escaped.

## Performance

`txMatchesRadio` runs once per observation inside `filterPackets`.

- Each observation costs one map lookup by observer id.
- Observers with a single stretch (nearly all) answer from that lookup
directly.
- Retuned observers add a binary search over their few stretches.
- Multi-radio observers add one lookup by observation id.
- The snapshot is loaded once per filter call.

`BenchmarkTxMatchesRadio`, 30K packets / 150K observations (20
observers, 2 of them retuned):

```
BenchmarkTxMatchesRadio-24    602    2056924 ns/op    0 B/op    0 allocs/op
```

Memory is one entry per stretch plus one per multi-radio observation;
nothing is added per observation otherwise.

## Tests

- **dbschema:** seed runs once, canonical spellings, the tables exist
without `observers.radio`.
- **ingestor:**
  - stretches extend and split; canonical spellings do not split
- two radios of one node hearing one transmission over two paths each
keep their own radio
  - a retained status does not write history
  - prune and purge
  - `rx_radio_id` reaches `PacketData`
- **server:**
- resolution before, between and after stretches, and with an unknown
radio id
  - `observer` + `radio`, and grouped rows carry `radio`
  - incremental refresh, `/api/radios`, `radio_history`, missing tables
  - the benchmark above
- **frontend:** the new filter fields in
`tests/unit/test-packet-filter.js`, and XSS cases through the real
`RfParams.radioText` and the history renderer.
- **E2E:** `tests/e2e/test-radio-setting-1869-e2e.js` checks the grouped
rows' radio, `sf ==` on the packets page, the detail Radio row and the
observer history. It is added to `scripts/non-unit-tests.json` and to
shard 2.

`go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema`
passes, along with `sh test-all.sh`, `eslint@8` (0 errors) and the XSS
diff gate. The new E2E test and `test-e2e-playwright.js` pass against
the fixture.

## Browser validation

This runs on a live instance with six observers, one of them retuned to
869.618/62.5/SF8/CR8:

- The packet detail shows `Radio: 869.432 MHz · SF7 · BW 62.5 kHz · CR
4/5`.
- `sf == 7` shows 106 of 106 grouped rows and `sf == 8` shows 0 of 107;
the SF8 observer has not heard anything yet.
- `/api/radios` lists both settings.
- No page errors.

## Notes

- **The customizer (rule 8)** gets nothing: the only constants are
server-side refresh intervals.
- **Attribution timing:** packets heard between a retune and the
observer's next status message (minutes) are attributed to the previous
setting, because a stretch starts when the status is handled. This is
documented at `RecordObserverRadios`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 21:08:40 +01:00
n30nex 643a53e98d fix(operators): refresh update metadata and wait for readiness (#2285)
Red commit: `7277ca23` ([failing
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190626951)).

Fixes #2229. Management startup now defines logging helpers before
normalizing CRLF environment files. Updates refresh version, revision
and build time after checking out the target. Deployment verification
and all five Compose healthchecks use `/api/healthz`, so an available
statistics endpoint cannot report an unfinished startup as healthy.

- The actual-script regression passes 15 checks: LF/CRLF/no environment
file, repeat startup, all update targets, failed checkout,
delayed/failed readiness, HTTPS readiness and unchanged informational
statistics.
- RED CI reproduced all 11 intended failures; GREEN `7ff9050b` passes
the full standalone frontend suite and shell syntax checks.
- Browser verified: an isolated Go fixture at http://127.0.0.1:18851
renders Home and returns ready=true from `/api/healthz`.
- Git, Docker and network boundaries are mocked in management tests. No
real container update was performed locally; CI validates the
Compose/build path.
- No new configuration, dependencies, schema changes or
packet-processing work.

Privacy, whitespace and scope checks passed. The external home-directory
preflight bundle is unavailable; applicable checks were performed
locally. Independent review found no outstanding fixes. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191647297)
passed on `7ff9050b`, including all three browser shards and both
architecture builds.
2026-10-26 15:50:36 +01:00
n30nex b5caea1d5f fix(analytics): render safe, correctly sized battery status icons (#2282)
Red commit: `4cc2efb3` ([failing
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40189921192)).

Fixes #2232. Battery badges now render their fixed status icon and
append voltage as text, including zero. Chart sizing excludes inline
icons, and status colors use the existing theme variables. Unknown
status values stay muted.

- The original browser regression failed on literal SVG; a second local
run with only the JavaScript fix failed on oversized icons.
- Validation on `eef0872e`: all standalone frontend suites; 47 focused
unit checks; 41 Chromium checks, including both themes, hostile voltage,
accessibility and 16×16 icon bounds.
- Browser verified: http://127.0.0.1:18851 on an isolated Go fixture.
- E2E assertion added: `tests/e2e/test-issue-1648-m2-icons-e2e.js:134`.
- Constant work for one badge; no new requests, dependencies, settings
or schema changes.

Privacy, whitespace, CSS-variable and repository XSS checks passed. The
external home-directory preflight bundle is unavailable; its applicable
checks were performed locally. Independent review found no outstanding
fixes. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190620324)
passed on `eef0872e`, including all three browser shards and both
architecture builds.
2026-10-26 15:44:23 +01:00