155 Commits
Author SHA1 Message Date
AdamandAdam c83499763d Allow mobile clients to manage repeater claims and details (#2429)
Draft for feedback: a working version of native repeater claims and
edits from mobile clients, built on the repeater info feature from
#2415. Four open questions are at the end.

## Situation

Repeater info (#2415) can only be claimed and edited from a browser
session. Device tokens from `POST /api/auth/device-token` are limited to
`rx` (CoreDrive RX: settings and companion linking), and `GET
/api/repeaters/{pubkey}/info` treats a bearer token as anonymous. A
mobile client can show a repeater's details but can't offer "claim" or
"edit" natively.

`PUT /api/repeaters/{pubkey}/info` is also a full replace with no
precondition. Two people editing from the same copy silently overwrite
each other. That's rare in the browser, but likelier from an app that
keeps a cached copy.

## Change

Three commits, one per piece.

**1. Device-token scopes for the repeater routes** (`8aba3715`)

- `POST /api/auth/device-token` takes an optional `scopes` list. Omitted
(or `null`) still means `["rx"]`, so CoreDrive RX and every existing
token keep exactly their current permissions. An empty list or an
unknown scope answers 400, checked before the password.
- New scopes:
- `repeater:claim`: `POST` and `DELETE /api/repeaters/{pubkey}/claim`.
These already only add or remove the caller's own claim. Several
claimers per repeater, idempotent `POST`, unchanged.
- `repeater:edit`: `PUT /api/repeaters/{pubkey}/info`, still subject to
`CanEditRepeater`.
- Every device token may use `/api/auth/me` and `/api/auth/logout`, so a
repeater-only token can check and end its own login.
- `GET /api/repeaters/{pubkey}/info` with a token holding either
repeater scope returns `canEdit` and `claimedByMe`. An unknown or
expired token answers 401 rather than passing as anonymous, so a client
learns its login is gone. A valid token without a repeater scope still
gets the public view. A cookie still wins over a bearer header, as
elsewhere.
- `GET /api/account/repeaters` (My repeaters, from #2430) also takes a
token holding either repeater scope, so an app gets the account's claims
in one request. It can't build that list from the public catalog, where
claimers are display names, and reading repeaters one at a time would
miss a claimed repeater that has left the node list. An `rx`-only token
still answers 403.
- Scope checks are method-aware (`bearerScopesFor(method, path)`). The
repeater paths are added to the bearer CORS paths.
- Same handlers, validation, rate limit (`repeater`, 120/h per user and
per IP), audit actions and admin claim removal as the browser. Token
writes add `via: device` to the `repeater.claim`, `repeater.edit` and
`repeater.unclaim` rows, and the `user.login` row records the granted
scopes. The admin route keeps refusing bearer tokens.

**2. Conditional edits with a revision** (`9f9bf47b`)

- `users.db` schema v10 adds `repeater_info.revision`. Existing rows
start at 1; a repeater with no saved details is at 0. Claims don't
change it.
- `GET` (single and list) and `PUT` responses include `revision`.
- `PUT` accepts an optional `baseRevision`. The check is part of the
write statement, so of two saves from the same base exactly one lands:
- `baseRevision: 0` (first save): `INSERT … ON CONFLICT(pubkey) DO
NOTHING`.
- `baseRevision: n`: `UPDATE … SET revision = revision + 1 … WHERE
pubkey = ? AND revision = ?`.
- No match answers `409` with the stored details
(`canEdit`/`claimedByMe` included) and writes nothing. That's the same
shape `PUT /api/account/settings` uses for its conflicts.
  - A stale base is refused even when the body would change nothing.
- Permission is checked first, so a 409 never shows details to someone
who can't edit.
- Without `baseRevision` the save is unconditional and bumps the
revision, so `public/repeater-info.js` works unchanged.
- `/api/config/client` adds `userManagement.repeaterTokens: true`, so a
client can tell this server from one without the scopes or
`baseRevision` (older servers answer the device-token request with the
SPA page, as noted for `companionLinking`).

**Discovery and conflict recovery, for a client:**

1. Check `userManagement.repeaterTokens` in `/api/config/client`.
Without it, keep the claim and edit actions hidden.
2. Log in with `scopes: ["repeater:claim", "repeater:edit"]` (or just
what the client offers). The response echoes the granted `scopes`.
3. Read `GET /api/repeaters/{pubkey}/info` with the token. Show
Claim/Unclaim from `claimedByMe` and Edit from `canEdit`, and keep
`revision`.
4. Save with `PUT …/info` and `baseRevision` set to that revision. On
200, keep the new `revision` from the response. On 409, the body is the
current record: show or merge it, then retry with its `revision`. On
401, log in again. On 403, the account can no longer edit (for example
someone claimed it), so refresh from `canEdit`.

**3. Account page device list** (`d28374aa`)

My account, Devices labelled every device token "CoreDrive RX".
`/api/account/sessions` now includes a device token's `scopes`. The page
keeps the CoreDrive RX label for `rx` tokens (and rows without scopes).
It shows other tokens under the device name the app sent, with a line
such as "can claim repeaters and edit repeater details".

Docs: `openapi.go`, `docs/api-spec.md` and the Repeater info section of
`docs/user-guide/accounts.md`.

## Questions

1. **One scope or two?** This uses separate `repeater:claim` and
`repeater:edit`, so a client can ask for less, and a token that only
edits can't put the user's name on a repeater publicly. One `repeater`
scope would be simpler if you'd rather not split it.
2. **Editing unclaimed repeaters from mobile.** This keeps web parity:
any active user may edit a repeater nobody has claimed, and once it has
claimers only they (and admins) can. The alternative is to require a
claim before a token may edit. That's stricter, but then app and browser
rules differ.
3. **Operator switch.** There's no new setting: the scopes are available
whenever `userManagement` is on, and the user opts in when the app asks
for them at login. Would you want an explicit opt-in such as
`userManagement.mobileRepeaterWrites`?
4. **Stale-edit contract.** Does `revision` plus `baseRevision` in the
body, with 409 returning the current record, fit? I matched account
settings rather than `ETag`/`If-Match`, to keep one pattern in the API.
The browser editor doesn't send `baseRevision` yet. Making it send one
would be a small follow-up, if you want browser saves protected too.

## Tests

- `cmd/server/repeater_device_token_test.go`:
- default and requested scopes, rejected scopes, and the method-aware
route table;
- per-scope allowed and denied routes, logout with a repeater-only
token;
  - claim and unclaim of own claim only, alongside another claimer;
- edit permission (unclaimed repeater, claimer, non-claimer, admin) and
a disabled account;
  - the admin route refusing tokens;
  - authenticated, public and expired-token reads;
- `via: device` audit rows, cookie writes still needing CSRF, and
session scopes.
- My repeaters with a claim or edit token (own claims only), and 403 for
an `rx`-only token.
- `cmd/server/repeater_revision_test.go`:
- revision on reads, a first edit with no saved row, and a stale base
returning 409 with the current record and no audit row;
  - unconditional browser saves, validation and permission order;
- two concurrent saves from the same revision (one 200, one 409), and
the client config flag.
- `internal/users/repeater_revision_test.go`:
  - revision counting and conditional first and later saves;
  - an 8-way race for both cases (`-race`);
  - the v9 to v10 upgrade.
- `tests/unit/test-user-management-ui.js`: app token labels and scope
line.

Run locally:

- `make test`: all 16 modules OK.
- `sh test-all.sh`: all suites pass, run in `node:24-bookworm`. On
macOS, `test-manage-workflow.js` fails 2 cases on unchanged master
because of the BSD tools.
- `npx eslint public/*.js`: 0 errors, the same 87 warnings as master.
- `node tests/e2e/test-lint-network-calls.js`
- `tests/e2e/test-user-management-e2e.js` against local e2etest servers
with the CI config: 44/44.
- Manually against a local instance:
  - in the browser: claim, edit, unclaim, and the Devices list;
  - with curl: the token flow, including a 409 after a browser save.

Not run: the full Playwright shard matrix from CI.

---------

Co-authored-by: Adam <196335+anieto@users.noreply.github.com>
2026-11-21 22:27:48 +01:00
ZachandClaude Opus 5.5 1a6322a707 feat(map): more tile providers (topo, satellite, gray, vector), hillshade overlay and custom tile servers (#2434)
Closes #2428.

Adds free and open map providers (topographic, satellite, gray and
vector basemaps), a hillshade overlay, and support for operators' own
tile servers. Everything is additive and opt-in: Carto stays the
default, every new provider group is off until the operator turns it on,
and no existing config key or provider was removed. An upgrade with an
untouched config changes nothing in the picker.

> **Dependency:** #2426 has merged. The branch is rebased onto current
`master` (`6c78a1d2`) with no conflicts.

## What's new

- **Provider registry** (`public/map-tile-providers.js`). New groups are
**opt-in**: each is off until `map.tiles.providers.<group>.enabled` is
`true` (the boolean), so licences and volunteer servers are the
operator's call:
- **EOX**: Sentinel-2 satellite (2025 and 2016), Terrain Light, and a
place-labels overlay. EOX 2025 imagery is CC BY-NC-SA 4.0
(non-commercial); a commercial deployment should leave `eox` off or
license it.
- **OSM France**: Humanitarian, CyclOSM, OSM Gray and OSM Gray Dark
(volunteer-run, fair use).
- **OpenFreeMap** and **VersaTiles** vector styles: Liberty, Bright,
Positron, Dark, Fiord, Colorful, Muted and the Gray variants.
- **Hillshade** (below) is opt-in the same way. **OpenTopoMap**, the OSM
community server, Stamen and USGS keep their existing opt-in.
- **Custom tile servers**: `map.tiles.custom` adds raster (XYZ), vector
(MapLibre style URL) and raster-overlay entries as `custom-<id>`. They
can be used in `darkDefault` / `lightDefault`. Invalid entries are
skipped with a console warning.
- **Hillshade overlay** (`public/map-hillshade.js`): shaded in the
browser from Terrarium elevation tiles.
`map.tiles.providers.hillshade.url` can point at a mirror.
- **Vector basemaps** (`public/map-vector.js`) use a vendored MapLibre
GL 5.24.0 (CSP build) and `@maplibre/maplibre-gl-leaflet` 0.1.4.
- They are lazy-loaded only when a vector map is picked (about 380 KB
gzipped).
  - The load times out after 15 s.
  - Without WebGL the map falls back to the theme's default raster map.
- **One picker on Map, Live and Repeater Planner**: Auto, Street,
Topographic, Satellite, plus an Overlays group.
- **A map picked there is saved** (`localStorage` `mc-tile-pinned`,
untrusted: honoured only while registered). Every picker reopens on it
after a reload or on another page, until the visitor picks **Auto**.
- The other maps (home mini-map, node detail, node reach, analytics)
follow the visitor's choice through the shared builder: the saved pick
when it suits the current theme (satellite always does), otherwise the
theme's map.
- Those small maps never load MapLibre, so a vector choice shows the
theme's default raster map there.
- **Security contract**: the `CONTRACT` block at the top of
`map-tile-providers.js` (S1 to S7) covers one shared tile-URL validator
(`https?://` or a same-origin path, no credentials in the authority,
whitelisted placeholders), attribution HTML-escaped once, picker labels
set with `textContent`, explicit MapLibre options, pinned vendored
versions, CORS-safe hillshade decoding, and treating stored picks as
untrusted input. `map.tiles` is served publicly in `/api/config/client`,
so keys put there are public (the docs warn about this).

## Fixes included

The race stress suite written for this PR found teardown races when
leaving map pages mid-load:
- a lazy MapLibre load finishing after navigation;
- a hillshade tile decoding after `remove()`;
- a provider-change event arriving during teardown.

It also found a pre-existing document-listener leak in `AreaFilter` /
`RegionFilter`. All of these are fixed in `e6709e74`, with regression
tests in `tests/e2e/test-map-teardown-races.js`.

The first CI run then caught a bug in that fix (`7f8d792c`).
`MC_safeRemoveMap` called `map.off()` before `map.remove()`, so the
map's `unload` event reached no one. Leaflet relies on that event to
detach controls and clean up the zoom-animation proxy and BoxZoom, and
`test-issue-1206` watches it to confirm node maps are disposed. `off()`
now runs after `remove()`, and a new regression test checks that
`unload` fires, controls detach and no listener is left.

A later stress run (a new seed) found a real two-tab bug, fixed in
`0506fc1d`. When another tab's overlay pick is already in localStorage
but its `storage` event hasn't arrived, a layer-picker click here
triggered a sync that added that overlay while Leaflet was still
applying the click. Leaflet doesn't redraw the list for changes it sees
mid-click, so the overlay's checkbox stayed unticked, and the next click
on any overlay removed it and wrote the removal back to every tab. The
overlay set now defers a sync until the click has finished. New E2E
check `4b` in `test-map-providers-e2e.js` plants another tab's pick and
clicks twice: one write per click, the planted overlay stays ticked. It
fails on the previous code.

## Docs

- `docs/deployment.md`, "Map Tile Providers" section, rewritten. It
covers every provider and its licence, turning groups on and off, custom
servers, vector maps, hillshade, self-hosting, and what is public.
- User guide:
  - `map.md` gets a "Map style and overlays" section;
  - `live.md` notes the picker;
  - `configuration.md` gets a "Map tiles" section;
  - the FAQ gets a new entry on satellite, terrain and gray maps.
- `CHANGELOG.md`: an Unreleased entry.

## Test data

Two datasets were used, so every number below says which one it came
from.

**1. Large synthetic dataset** (perf A/B and screenshots). It is
generated, contains no real mesh data, and is not checked in.

- **Generator:** `synthgen` 1.0.0, seed 1, "full" scale. Built from
upstream `be16173e` and retimed so its 18-day window (2026-10-31 to
2026-11-18) ends at the time of the run.
- **Database:** SQLite, 3.94 GB (586 MB as zstd), 11.6 M rows in all.

| Table | Rows |
|---|---|
| Nodes | 18,102 (12,606 companions, 3,652 repeaters, 953 rooms, 891
sensors) |
| Nodes with a location | 13,452, spread worldwide (latitude −37° to
60°, longitude −123° to 175°) |
| Observers | 1,175, in 42 regions on six continents |
| Transmissions | 759,133 |
| Observations | 10,569,668 |
| Neighbour edges | 42,806 |
| Advert route evidence | 212,142 |

During the perf runs, the server's packet store was pinned to the newest
48 h with a 1.5 GB memory cap. Each server held about 65 k transmissions
and 0.9 M observations in memory (baseline 65,647 / 906,467; candidate
64,862 / 896,248), plus all 18,102 nodes (about 17,650 active) and 1,175
observers. The tracked store was about 450 MB.

**2. The repo's CI fixture** (`test-fixtures/e2e-fixture.db`, 704 KB).
All E2E suites and the race stress gates ran against it, freshened the
way CI does: 204 nodes, 31 observers, 500 transmissions, 502
observations. A second copy with the Repeater Planner enabled was used
for the planner-on stress runs.

**Perf environment:**
- Headless Chromium 158 via Playwright 1.58.2, at 1280×800 with device
scale factor 1.
- 9 measured runs per scenario plus 1 warm-up, on each of `master` and
this branch.
- Host: Linux, AMD Ryzen 7 3700X (16 threads), 63 GB RAM.
- **Tiles were stubbed.** The harness answers every tile, elevation and
style request locally with the same stub on both sides (`--mode stub`),
so perf timings involve no network. Only the screenshot gallery uses
real tiles.

## Screenshots

These were taken from the synthetic dataset with real tiles in a real
browser (headless Chromium, 1280×800). Click a thumbnail for the full
size. The images are hosted on a separate assets branch, so they are not
in this diff.

#### Street

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-light.png"
width="260" alt="Street: Carto (default) (light)"></a><br>Carto
(default) (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-dark.png"
width="260" alt="Street: Carto (default) (dark)"></a><br>Carto (default)
(dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-osmfr-hot-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-osmfr-hot-light.png"
width="260" alt="Street: OSM France Humanitarian (light)"></a><br>OSM
France Humanitarian (light) |

#### Gray

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-light.png"
width="260" alt="Gray: OSM France (light)"></a><br>OSM France (light) |
<a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-dark.png"
width="260" alt="Gray: OSM France (dark)"></a><br>OSM France (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-light.png"
width="260" alt="Gray: VersaTiles (light)"></a><br>VersaTiles (light) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-dark.png"
width="260" alt="Gray: VersaTiles (dark)"></a><br>VersaTiles (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-light.png"
width="260" alt="Gray: OpenFreeMap Positron (light)"></a><br>OpenFreeMap
Positron (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-dark.png"
width="260" alt="Gray: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap Dark
(dark) |

#### Topographic

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-cyclosm-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-cyclosm-light.png"
width="260" alt="Topo: CyclOSM (light)"></a><br>CyclOSM (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-eox-terrain-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-eox-terrain-light.png"
width="260" alt="Topo: EOX Terrain Light (light)"></a><br>EOX Terrain
Light (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-opentopomap-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-opentopomap-light.png"
width="260" alt="Topo: OpenTopoMap (opt-in) (light)"></a><br>OpenTopoMap
(opt-in) (light) |

#### Satellite

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-light.png"
width="260" alt="Satellite: Sentinel-2 2025 (light)"></a><br>Sentinel-2
2025 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-light.png"
width="260" alt="Satellite: Sentinel-2 2025 with labels
(light)"></a><br>Sentinel-2 2025 with labels (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-dark.png"
width="260" alt="Satellite: Sentinel-2 2025 (dark)"></a><br>Sentinel-2
2025 (dark) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-dark.png"
width="260" alt="Satellite: Sentinel-2 2025 with labels
(dark)"></a><br>Sentinel-2 2025 with labels (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-light.png"
width="260" alt="Satellite: Sentinel-2 2016 (light)"></a><br>Sentinel-2
2016 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-labels-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-labels-light.png"
width="260" alt="Satellite: Sentinel-2 2016 with labels
(light)"></a><br>Sentinel-2 2016 with labels (light) |

#### Vector (MapLibre)

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-liberty-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-liberty-light.png"
width="260" alt="Vector: OpenFreeMap Liberty
(light)"></a><br>OpenFreeMap Liberty (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-dark-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-dark-dark.png"
width="260" alt="Vector: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap
Dark (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-light.png"
width="260" alt="Vector: VersaTiles Colorful (light)"></a><br>VersaTiles
Colorful (light) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-dark.png"
width="260" alt="Vector: VersaTiles Colorful Dark
(dark)"></a><br>VersaTiles Colorful Dark (dark) | | |

#### Hillshade overlay

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-light.png"
width="260" alt="Overlay: hillshade over Sentinel-2
(light)"></a><br>hillshade over Sentinel-2 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-light.png"
width="260" alt="Overlay: hillshade over Carto
(light)"></a><br>hillshade over Carto (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-dark.png"
width="260" alt="Overlay: hillshade over Sentinel-2
(dark)"></a><br>hillshade over Sentinel-2 (dark) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-dark.png"
width="260" alt="Overlay: hillshade over Carto (dark)"></a><br>hillshade
over Carto (dark) | | |

#### Repeater Planner

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-light.png"
width="260" alt="Planner: Sentinel-2 + hillshade
(light)"></a><br>Sentinel-2 + hillshade (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-dark.png"
width="260" alt="Planner: Sentinel-2 + hillshade
(dark)"></a><br>Sentinel-2 + hillshade (dark) |

#### Layer picker

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-light.png"
width="260" alt="Layer picker open (light)"></a><br>Layer picker open
(light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-dark.png"
width="260" alt="Layer picker open (dark)"></a><br>Layer picker open
(dark) |

#### Custom tile server

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-map.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-map.png"
width="260" alt="Custom raster provider on the map"></a><br>Custom
raster provider on the map | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-picker.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-picker.png"
width="260" alt="Custom raster provider in the picker"></a><br>Custom
raster provider in the picker |

## Testing

The full suite ran twice, each time on a fresh clone:

| | Pass 1 | Pass 2 |
|---|---|---|
| Commit (before rebase) | `8a56e841` | `9355169b` |
| `test-all.sh` (unit) | pass | pass |
| `npm run lint` | 0 errors (87 warnings, all pre-existing) | 0 errors
(87 warnings, all pre-existing) |
| Go `cmd/server` / `cmd/ingestor` | ok / ok | ok / ok |
| E2E, map-related files (CI fixture) | 44/44 | 44/44 + planner-on run |
| `test-e2e-playwright.js` | 149 passed, 3 skipped | 149 passed, 3
skipped |
| Race stress gate (seeds) | 101, 202: all pass | 303, 404: all pass |
| Stress with planner enabled | pass | pass (seed 505) |
| Gallery | 32/32 | 32/32 |

The rebased equivalents of the pass 1 and pass 2 commits are `fde2552c`
and `e4d5c447`.

**After the first CI run.** The two passes covered only the map-related
E2E files, and CI exposed two problems:

1. `test-issue-1206` failed 7 checks. This was the `unload` bug
described under "Fixes included".
2. The JS unit gate runs without `node_modules`, so the two new jsdom
suites crashed `test-all.sh`. Locally, jsdom was reachable from a parent
checkout, which hid this.

The second problem is fixed in `57720bf3`, following the
`domIntegration` convention in `scripts/non-unit-tests.json`.
`test-map-teardown-races.js` moved to `tests/e2e/`, and the three jsdom
zoom-limit tests split out into
`tests/e2e/test-map-vector-leaflet-zoom.js`, so 30 + 3 = 33 tests are
kept. Both run from E2E shard 2 and take a few seconds. The heavy suites
(race stress, perf, gallery) stay local-only and are not run in GitHub
Actions.

These checks ran on the fixed, rebased tree (`57720bf3`):

| Check | Result |
|---|---|
| `test-all.sh` from a copy with no `node_modules` (as CI runs it) |
pass |
| `test-test-inventory.js` | 399 suites classified |
| `npm run lint` | 0 errors |
| Go `cmd/server` / `cmd/ingestor` | ok / ok |
| `test-issue-1206-resize-observer-leak-e2e.js` | 13/13 (7 were failing)
|
| `test-e2e-playwright.js` | 149 passed, 3 skipped |
| `test-map-providers-e2e.js` / harness E2E | 17/17, 16/16 |
| `test-map-teardown-races.js` / `test-map-vector-leaflet-zoom.js` |
19/19, 3/3 |
| Race stress (3 iterations) | all scenarios pass |

The commits since pass 2 are `76c67184` (docs only), `7f8d792c` and
`57720bf3`.

**GitHub Actions on `0506fc1d` (rebased onto `master`, with the two-tab
fix): every check passed** (13 passed, 4 skipped as usual for a PR).
That covers Go build and test, all three E2E shards, the server race
detector and both image builds. The moved suites ran in shard 2
(`test-map-teardown-races.js` 19/19, `test-map-vector-leaflet-zoom.js`
3/3), and the unit gate ran `test-map-vector.js` (30/30).

**Race stress suite** (`tests/e2e/test-map-race-stress-e2e.js`). Each
gate run does 20 iterations of scenarios a to e and 50 cycles of f. The
scenarios cover:
- rapid provider switching;
- theme flips during a vector load;
- navigating away mid-load (including a 15 s MapLibre timeout);
- repeated map-page navigation with heap checks;
- overlay toggles;
- cross-page picker persistence.

Results for each run:
- Listener counts stay flat: at most 3 `mc-tile-provider-changed`
listeners and 5 `storage` listeners.
- At most one live WebGL context at a time.
- Heap growth in the second half of scenario d was 0.25 to 0.45 MB
across the four gate runs.
- Upstream `master` grows about 0.39 MB per map navigation loop in the
same scenario; this branch levels off.

## Maintainer review (addressed)

1. **Licence default.** All five new groups (`eox`, `osmfr`,
`openfreemap`, `versatiles`, `hillshade`) are opt-in: off unless
`enabled: true` (`"true"`, `1` or `{}` don't count). With an untouched
config the picker lists the same 6 base maps as before. Docs,
`config.example.json` (now `enabled: false` for each) and the changelog
say so, and each group's terms are listed before you turn it on.
2. **Saved pick.** The on-map picker now saves the pick (option 1 of
your two), as the docs promised. Behaviour, and what an other-theme pick
does:
- Every picker opens on the pinned map, on any page and after a reload,
until **Auto** is picked. A pin is shown whatever the theme, as an
explicit pick already was.
- Maps without a picker show it when it fits the current theme (`type`
is the theme or `any`), otherwise the theme's map. So a light pin isn't
drawn on a dark node map; satellite always is.
- A stored pin whose group is turned off reads as Auto, and comes back
if the group is turned on again; only picking Auto clears it.
- On a map with a picker, Auto always means the theme's map: the pin is
only the explicit layer, never loaded into the hidden Auto layer. A
choice in the customizer, and Reset All, clear the pin. A failed write
(quota full) never brings back an older stored pin.
- E2E `7a` to `7c` in `test-map-providers-e2e.js`: pick, reload, the
radio and tiles are the pick, the node detail map draws it, and a light
pin in dark is kept but not drawn. Auto clears the key; junk in it gives
Auto; an untouched config lists none of the new groups. 7a and 7b fail
with the save line removed. Unit tests are in `test-map-tile-pin.js`
(21).
3. **CI.** `test-map-providers-e2e.js` (24 checks, including `4b`, the
two-tab regression check) and `test-map-providers-harness-e2e.js` (16)
now run in E2E shard 3 against the fixture server: every tile host is
stubbed.

Non-blocking:
- `getTileUrl()` now returns the raster fallback for a vector pick
(`MC_rasterTileUrl`). There's a test in `test-tile-resolver.js` that
fails on the old code.
- The `map-vector.js` header and the binding's vendor README list the
Leaflet and binding internals to recheck on an upgrade.
- `docs/deployment.md` says SRI makes a JS-rewriting proxy drop vector
maps to raster, so serve `public/vendor/` unmodified.
- `scripts/map-perf/README.md` says where `SEED_DB` and `SHIFT_SH` come
from.
- The maps with `attributionControl: false` are left as they are. That
predates this PR, and with every new group opt-in, an upgrade adds no
uncredited imagery. Follow-up for all of them: #2441.
- The OpenFreeMap credit line is `OpenFreeMap © OpenMapTiles Data from
OpenStreetMap` (`2424f3c3`), the line OpenFreeMap gives for clients that
don't add it automatically.

Verified after these changes:
- `test-all.sh` and lint pass, with 0 errors.
- Both provider E2E suites pass: 24/24 and 16/16.
- Race stress, seed 979: scenarios a, b, d and e pass. c (15/15) and f
(37/37) pass after updating their model of the pick: they expected Auto
after a return, which was the old unsaved behaviour.
- The stress suite now enables the five groups itself, so it keeps
exercising them.

## Performance

A/B test: `master` against this branch on the large synthetic dataset,
with 9 runs per scenario and stubbed tiles (see Test data;
`scripts/map-perf/`). The full HTML report is not attached; ask if you
want it.

- **No budget failures** in either pass. The `raster-no-maplibre` budget
holds: no raster scenario loads MapLibre.
- **DOMContentLoaded is unchanged on every page**, even with the two new
scripts (`map-hillshade.js` and `map-vector.js`, about 635 lines in all;
MapLibre itself loads only on demand).
- **Switching providers 50 times (raster only), median ms per switch:**
  - dark: 146 → 108.5;
  - light: 208 → 208.5.
  - Heap growth during the run: 0.3 to 0.4% on both sides.
- **Flagged and checked:**
- Live and Analytics `loadEventMs` are flagged in some runs. This metric
has two modes on **both** branches: `load` fires either before Leaflet
adds its first tile `<img>`s (about 150 ms) or after they finish (about
400 ms). A direct probe of `master` alone gave 428, 404 and 124 ms. On
these scenarios, map creation, first tile, all tiles, first marker and
wall time are within a few ms on both sides.
- Carto and Positron `allTilesSinceMapMs` are 4 to 6 ms slower, just
above `master`'s p95. Tiles are stubbed locally, so this is not network
noise. The cause is not isolated. Both providers now go through the
shared layer builder, and on the same scenarios first tile, all tiles
and wall time are within 10 ms of `master` (Carto Voyager +2 to +3 ms;
Positron Dark +6 to +9 ms).
- The Analytics map shows one long task of about 50 ms against a
`master` median of 0, but `master`'s p95 is 52 to 84 ms. This is
disclosed in case it shows up for you.
- Other desktop apps were running during the perf runs, so expect some
noise.

## Commits

1. `c6bf12ed` feat(map): tile provider registry with custom servers,
gray, topo and satellite maps, and overlays
2. `9237230b` feat(map): client-side hillshade overlay
3. `781e41b4` feat(map): vector basemaps via vendored MapLibre GL 5.24.0
4. `dae36d2e` refactor(map): every map surface uses the shared basemap
builder
5. `3d4afcdc` fix(map): teardown races and listener leaks when leaving
map pages
6. `b798b2fe` test(server): pin map.tiles passthrough in
/api/config/client
7. `171b1bc4` test(map): E2E, race stress suite, perf harness and
screenshot gallery
8. `61f73875` docs(map): providers, custom tile servers, overlays and
self-hosting
9. `135a9b12` test(map): gallery shows opt-in OpenTopoMap and retries
slow tile servers
10. `3833572b` test(map): harness waits for vector basemaps; stress
suite fixes from test pass 1
11. `3e543cad` test(map-perf): redact tile-URL credentials; compare
switch50 like with like
12. `d5006111` docs(map): user guide for map styles, overlays and
map.tiles
13. `90ee1d5d` fix(map): MC_safeRemoveMap lets unload listeners run
before dropping them
14. `5fc57b5c` test(map): run the jsdom map suites in the E2E job, not
the unit gate
15. `bd36b634` fix(map): another tab's overlay pick that lands mid-click
is no longer removed
16. `38df65d2` fix(map): new tile groups are opt-in; a map picked in the
picker is saved
17. `eafbcac9` ci(e2e): run the map provider E2E suites in shard 3
18. `65b223ec` fix(map): getTileUrl() never returns a vector style url;
review notes
19. `f3402c5d` test(map): race stress enables the opt-in groups and
expects the saved pick
20. `e92b8631` fix(map): Auto on a picker map ignores the saved pick;
customizer clears it

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 22:26:46 +01:00
AdamandAdam 49dd4ee773 List my claimed repeaters on My account (#2430)
Follow-up to #2415, see #2414.

## Situation

A user can claim repeaters, but no page lists the ones they claimed.
Claims show on each node page ("Maintained by"), in the admin user
detail and in the "Download my data" export. A maintainer of several
repeaters has to remember them and open each node page to see whether
one went quiet.

## Change

**`GET /api/account/repeaters`** (browser session; device tokens answer
403) returns the caller's claims. For each one:

- `pubkey` and `name`.
- `known`: false once the node left the analyzer database. The claim
stays.
- `role` and `claimedAt`.
- `lastHeard`: the newer of the last advert and the last transmission
certainly from the node, the same sources as the node page's Last Heard.
- `lastRelayed`: when its hop prefix last relayed. Only used to mark a
silent repeater as uncertain, since the prefix can be shared.
- `status`: the saved repeater status, or null.
- `comaintainers`: the other active claimers' display names.

The list is sorted by name, with unnamed repeaters after, by pubkey.
It's built on the existing `RepeaterClaimsByUser`, `NotifyNodes` and
`LastHeardMap`.

**My account, My repeaters**, a new section after Companions:

- Silent repeaters come first. Health uses `getNodeStatus`, so the
threshold matches the node pages and map (`infraSilentMs`):
  - green "heard 2h ago";
  - yellow "last heard …, its prefix still relays";
  - red "silent, last heard 184d ago" or "never heard";
  - grey "no longer in the database".
- Each row links to the node page and shows the saved status ("no
details yet" without one), the claim date and "also maintained by …".
- *Unclaim* asks for confirmation, then uses the existing `DELETE
/api/repeaters/{pubkey}/claim`.
- The empty state says how to claim.
- `#/account?section=repeaters` deep-links to the section, like
`section=notifications`.

Editing stays on the node page, so there's only one editor. The admin
user detail and the export are unchanged.

Docs: `openapi.go` and the Repeater info section of
`docs/user-guide/accounts.md`.

## Notes

- A mobile client will want the same list. #2429 proposes device-token
scopes for repeater claims. If it lands, letting a `repeater:claim`
token read this route is a one-line addition to the bearer route table,
and I'd rather do that there than here.
- The two PRs touch nearby lines in `openapi.go`, `repeater_handlers.go`
(route registration) and `public/account.js`. Whichever merges second
needs a small rebase.

## Tests

- `cmd/server/my_repeaters_test.go`:
  - empty list;
  - ordering (named first, case-insensitive);
  - unheard, heard and departed nodes;
  - saved status;
  - co-maintainers, excluding the caller and disabled users;
  - another user's view;
  - unclaim removes the row;
  - anonymous gets 401 and a device token 403;
  - no analyzer DB.
- `tests/unit/test-user-management-ui.js`: health sort and chips,
escaping, the short-pubkey fallback, status and co-maintainer lines,
Unclaim on every row, and the empty state.
- `tests/e2e/test-user-management-e2e.js`:
  - the claimed repeater appears with its status and node link;
  - axe on the section;
- Unclaim: dismissing the confirm keeps the claim, accepting empties the
list.

Run locally:

- `make test`: all 16 modules OK.
- `sh test-all.sh` and `node tests/e2e/test-lint-network-calls.js`:
pass, run in `node:24-bookworm`.
- `npx eslint public/*.js`: 0 errors, the same 87 warnings as master.
- `test-user-management-e2e.js` against local e2etest servers with the
CI config: 46/46.
- Manually in Chrome, light and dark: heard, silent and co-maintained
rows, and the deep link.

Not run: the full Playwright shard matrix from CI.

---------

Co-authored-by: Adam <196335+anieto@users.noreply.github.com>
2026-11-20 16:25:51 +01:00
11eb72f578 feat(planner): Coverage tab, reach of a candidate repeater site (#2406)
The repeater planner (#2391) gets a fourth tab, **Coverage**: put a
marker where you are thinking of placing a repeater, set its antenna
height, and see on the map who would hear it. It answers the siting
question "what does a repeater reach if we put it here", with two
layers: handheld users and other repeaters.

## What it shows

- **Users layer:** the chance that a handheld at 1.5 m hears the site,
in 4 bands (10-25%, 25-50%, 50-75%, 75% and up; below 10% is not
painted). The chance comes from a distance curve `P(d) = 1 / (1 +
exp((ln d - ln d50) / slope))`, fitted on drive-test receptions of 41
Belgian repeaters (d50 2.7 km, slope 0.30). Terrain only lowers a cell
by 2 bands where a 1.5 m receiver sits deeper than 1 first-Fresnel
radius under the terrain (`cmd/server/planner_viewshed.go:19-21`).
Antenna height does not change the curve: in the drive tests, handheld
reach barely changed between 1, 7 and 15 m. The panel says so in one
line.
- **Repeaters layer:** first-Fresnel clearance from the site to every
cell, for a receiver at "Repeaters at" height (default
`planner.existingAntennaM`). 3 classes: line of sight (clearance 0% or
more), slightly blocked (-20% to 0%), blocked (not painted). Every known
repeater within range gets its link computed with the existing planner
link code, a dot in its class colour, a line for line of sight and
slightly blocked, and an entry in a list.
- **Compute button:** changing site, height or range moves the marker
and updates the URL hash right away, and dims an existing result with an
"Outdated" note. Only "Compute coverage" starts a job. A shared link
that carries a site computes once on open. The planner has 1 heavy-job
slot shared with Area and allows 6 jobs per minute per IP, so
recomputing on every change would burn both.

State is in the hash:
`#/tools/planner?mode=coverage&site=51.06,4.41&h=10&rx=5&range=30&layer=repeaters`.

This builds on the repeater planner @zfouts wrote in #2391: the terrain
tiles, the first-Fresnel analysis, the link code, the job protocol and
the Area module pattern are all reused here. Thanks for that work, it
made this tab a small addition instead of a new subsystem.

## Why the Repeaters classes are not the route LOS verdicts

The route verdicts (clear 100%+, clear60 60%+, marginal 0%+, blocked)
almost never show on Shuttle Radar Topography Mission (SRTM) terrain at
mast heights. At 10 m / 5 m, 30 km around Mechelen, 109 of 70,980 cells
reach clear60 or clear. SRTM includes trees and roofs, so the line
between 2 masts rarely clears 60% of the Fresnel zone.

I checked the clearance against one Belgian repeater (28 m, receivers at
5 m) and the repeaters live data shows it actually hears:

| Distance | Heard, clearance >= 0 | Heard, clearance < 0 |
|---|---|---|
| 0-10 km | 9 of 11 | none in range |
| 10-20 km | 3 of 4 | 1 of 8 |
| 20-30 km | 0 of 6 | 3 of 35 |
| 30-60 km | 1 of 2 | 0 of 84 |

Every link heard within 16 km had a clearance between 2% and 54%, so
"marginal" is where working links live. The map classes follow that
(`repeaterClass`, `cmd/server/planner_viewshed.go:75-89`). Chain, Relay
and Area keep the shared verdicts; `losVerdict` is unchanged. This rests
on 17 heard links from 1 repeater, which is why the -20% band is narrow
and the panel says line of sight stops being enough beyond about 20 km.

## Server

- `GET /api/planner/viewshed?site=&h=&rx=&range=`
(`cmd/server/planner_api.go:603`, route `cmd/server/routes.go:393`).
Same job protocol, result cache, rate limits and opt-in as the other
planner endpoints: 404 while `planner.enabled` is false.
- One byte per cell: low nibble Users class 1-5, high nibble Repeaters
class 1-3, 0 for no data. The grid is about 300 x 300 at every range:
0.2 km cells up to 30 km, `range/150` above (`planner_viewshed.go:47`).
- Links skip rooms, stale repeaters and a repeater within 100 m of the
site (that is the site itself). They carry no `elev_m` profile.
- `truncated` is true only when the 300-node cap may have dropped a
repeater within range (`viewshedTruncated`, `planner_api.go:694`).
- Tile cap: `Viewshed` in the planner tile limits, sized from the
largest offered range (60 km).
- New config: `planner.userReach.{d50Km,slope}` with a `_comment` in
`config.example.json`. Bounds 0.2-50 km and 0.05-2.
- No writes, no new `map[string]interface{}`; response types
`PlannerViewshedResponse/Params/Link` in `planner_types.go`.

## Frontend

- `public/planner-coverage.js` + `.css`, mounted like `planner-area.js`.
`planner.js` gets a `MODULES` table (`public/planner.js:31`) so Area and
Coverage share one enter/leave path.
- Canvas raster in an `L.imageOverlay`; the layer switch repaints
locally without a new request. Requests go through `fetchJob` / `CSApi`.
- Colours are CSS variables `--pl-cov-user-2..5` and `--pl-cov-rep-1..3`
in `:root` and both dark blocks of `style.css`; the Repeaters ones point
at `--status-*`, so a theme that changes those changes the map.

## Performance

The job is opt-in and runs in the existing planner job slot, never in a
hot path. Measured on real SRTM tiles, local build, 304 Belgian
repeaters:

| Site | Range | Mast | Job time |
|---|---|---|---|
| Maasvallei (hilly) | 30 km | 10 m | 2.07 s |
| Maasvallei | 30 km | 30 m | 2.57 s |
| Maasvallei | 60 km | 10 m | 3.98 s |
| Mechelen (flat) | 30 km | 10 m | 2.61 s |

On a staging instance (ARM64, cold tile cache) a 30 km job took 6.8 s
including the tile downloads. Response size at 60 km with 115 links:
239,942 bytes with link elevation profiles, 149,984 without (what this
PR ships). `BenchmarkViewshedMaxLeg` covers the 60 km worst case on a
synthetic terrain view.

## Verification

- Go: 26 viewshed tests and benchmarks in `planner_viewshed_test.go`,
`planner_api_test.go`, `planner_config_test.go` and
`planner_tiles_test.go`, among them: flat terrain equals the plain curve
and `analyzeLOS`, a ridge shadows only its far side, missing terrain is
"no data", the `truncated` rule (proven to fail on the old rule), links
without `elev_m`, the repeater at the site left out. `go test ./...` in
`cmd/server` passes.
- Frontend: `tests/unit/test-planner-coverage.js` (23 tests, in
`test-all.sh`): grid decode, class-to-colour mapping, unpainted classes,
hash round trip, no fetch on parameter change, 1 fetch per click,
deep-link autorun once, stale responses ignored, link classes at the
boundaries, escaped repeater names.
- E2E: 5 Coverage cases in `tests/e2e/test-planner-e2e.js` (stubbed
viewshed): click and compute, layer switch without a request, outdated
then recompute, deep link, leaving the tab removes the overlay. Planner
E2E 25/25 locally.
- eslint 0 errors. Browser: light and dark theme, Users and Repeaters on
flat, hilly and coastal sites, deep links, no console errors; also on a
staging instance with live data.

## Not in this PR

- The `--pl-cov-*` colours are not in the customizer's `THEME_CSS_MAP`
yet. They follow `--status-*` for the Repeaters layer; the Users colours
need their own entries (AGENTS.md rule 8, later milestone).
- In Flanders the Users layer is close to a circle: 0.3% of cells differ
from the plain curve around Mechelen, 42% in the Maasvallei, 64% in the
Ardennes. The terrain correction is untested in hilly areas, the drive
tests had no repeater there.
- The Repeaters calibration uses 1 repeater. Other operators with drive
tests or neighbour data can tighten it.
- A link's class comes from `clearance_pct`, rounded to 0.1%, while the
grid uses the raw clearance: within 0.05% of a bound a link can land 1
class off the cell under it. List, line and dot always agree with each
other.
- An OS dark-mode switch without the theme toggle does not repaint the
overlay until the next compute, the same as the planner's tile layer.

---------

Co-authored-by: zfouts <zachary.fouts@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 23:19:49 +01:00
efiten 12f9c5c686 docs(release): v3.18.0 release notes and changelog entry (#2420)
Release notes and changelog entry for v3.18.0, to be tagged on this PR's
merge commit.

- `docs/release-notes/v3.18.0.md`: the notes, in the same shape as
v3.17.0. The CI/CD Pipeline uses this file as the GitHub release body.
- `CHANGELOG.md`: a 3.18.0 entry under `[Unreleased]`.

11 commits since v3.17.0: 5 fix, 3 feat, 1 test, 1 refactor, 1 docs.
"Read this before upgrading" covers the `users.db` schema v9 migration
with user accounts on (#2415), the WebSocket path `/ws` (#2383) and
config tile URLs winning on the Map and Live pages (#2386).

Documentation only, no code change.
2026-11-17 16:42:05 +01:00
efiten 0cf22cdd79 feat: repeater info, claims and community-maintained details (#2415)
Closes #2414
2026-11-17 11:40:05 +01:00
4ebb5ccc11 feat: radio line-of-sight on packet routes + repeater planner (#2391)
Both features are OPT-IN ("los" and "planner" config blocks, default
off); with them off nothing is fetched and payloads are unchanged.

Route view
- Multi-path picker discoverability: on-map "N hops shown · tap to see
all" badge, picker pulse (reduced-motion safe), one-time tip, row hover
arrow; "no GPS" hops counted apart from "unresolved".
- Line of sight for every leg of a packet route: clear / marginal /
blocked chips (first-Fresnel clearance, k=4/3), obstruction markers on
the map, docked terrain-profile panel with antenna/frequency inputs and
a &los= deep link. Backed by GET /api/elevation/profile, an Open-Meteo
proxy that only accepts known node positions, with weighted budgets,
back-off and a per-IP limit.

Repeater planner (#/tools/planner)
- Chain: LOS on every leg between any number of points or nodes.
- Relay: best single relay for A→B among known repeaters and open
terrain.
- Area: suggested new repeater sites that best join the existing
repeater backbone in a box up to 100×100 km, picked greedily to merge
clusters, then remove single points of failure, then reach unlinked
repeaters.
- Terrain: SRTM 1° tiles from AWS Open Data, in memory only (~90 m,
bounded LRU); the server stays read-only. Exact-length gunzip, fetch
budget with back-off, missing terrain reported rather than read as sea
level, fuzzed pixel maths.
- GET /api/planner/{profile,relay,coverage}: quantised cache keys, 202 +
progress then cached 200, separate compute slots, per-IP and global
limits, failed jobs remembered, unpolled jobs cancelled. Nodes pass the
same blacklist / hidden-name / geo_filter rules as /api/nodes. The Go
LOS maths matches public/los.js on shared test vectors.

Also: the per-IP client limiter is extracted from elevation.go into
client_limit.go (behaviour-preserving) so both features share it.

Perf (real SRTM): coverage cold 1.5-5.6 s (tile download), warm 10 ms;
worst real compute 545 ms; peak RSS 91 MiB; live /api/nodes and
/api/packets p95 at most 1.6x while a job computes (GOMAXPROCS=2).

Tests: Go suites for elevation, terrain (fuzz), LOS vectors, graph
(brute-force property test), search and planner API; frontend unit
suites in test-all.sh; Playwright e2e for route LOS and the planner.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-11-16 16:15:36 +01:00
Paul PicazoandClaude Opus 5.5 e7fd48a054 feat(frontend): serve the UI from a web container or a static host, three deployment modes (#2383)
## What this does

The frontend half of serving the UI apart from the server. The server
half was #2362. It gives CoreScope three deployment modes. The server
container is the same image in all three, and it still serves
everything, the classic UI included.

| Mode | The UI is served by | API | Server CORS setting | Logins |
|---|---|---|---|---|
| All-in-one container (today) | the server container | same origin |
none | yes |
| Split containers (new) | a web container that forwards the API | same
origin | none | yes |
| Static host or CDN (new) | Cloudflare Pages, a CDN or any web server |
another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI |

### Commit 1: the UI as static files, for any host

- **One place for API URLs:** `public/deployment.js` says where the API
is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket
URL. A lint rule keeps raw `/api` fetches out of the other files. The
embedded UI keeps `apiBase: ''`, so it behaves as before, except that
its WebSocket now uses `/ws` instead of `/`.
- **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`,
which is `public/` byte for byte except for the cache buster in
`index.html` and `deployment.js`.
- It is `dist/web` rather than `dist/` because the Go binaries are built
into `dist/` and `make clean` removes that folder.
- `API_BASE=same-origin` is for a host that forwards `/api` and `/ws`
itself, as the web container does.
- **Anonymous build (`AUTH=false`):** for another origin. It has no
account features: login, account, admin and other writes link to the
same page on the classic UI. It sends no cookies, and its three JSON
POSTs go as `text/plain`, so there is no preflight.
- **Error notices:** a blocked API answer or an HTML answer each show a
notice.
- **Channel badge:** `beta`, `next` and `dev` builds show a badge that
opens a pre-filled issue, plus an "Open in classic" link.
- **Runbook:** `deploy/static/README.md` covers Cloudflare Pages,
previews, server CORS and rollback.

### Commit 2: the web container and the three modes

- **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It
packages the UI with `build.sh` when the image is built, serves it, and
forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`).
`SITE_ADDRESS` set to a domain name turns on automatic HTTPS.
- **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules
as `deploy/static/_headers`, and compresses responses with zstd or gzip.
- **`docker-compose.split.yml`:** the stock server image (its own Caddy
off) plus the web container:
`CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f
docker-compose.split.yml up -d --build`.
- `CORESCOPE_IMAGE` is required and must match the checkout's version,
so a release server is never paired with a newer UI by default.
- The server's built-in broker, which accepts anonymous clients, is
published on `127.0.0.1` unless `MQTT_BIND` says otherwise.
- **Docs:** `docs/deployment.md` has a new "Deployment modes" section,
and `DEPLOY.md` points to it.

### Keeping new code on `CSApi`

- **Lint rule:** `.eslintrc.json` now rejects every direct network call
in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`,
`sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in
`href`, `src` and `action` (assigned, set with `setAttribute` or written
into markup), `window.open`, `location.assign`, `location.replace` and
`location =`. `api-base.js` is the one file exempt.
- **HTML pages:** `eslint public/*.js` never sees the inline scripts and
markup of `public/*.html`, so `test-lint-network-calls.js` applies the
same rule to them, at their page lines. `area-map.html`'s two fetches
carry an `eslint-disable` with the reason: its Server field can name
another server.
- **Why it changed:** the earlier rule only caught a literal `/api…`
passed straight to `fetch`.
- **Check:** CI already fails on lint errors, and
`tests/e2e/test-lint-network-calls.js` pins exactly what the rule
reports: 25 bad statements reported, 15 good ones not, `api-base.js`
exempt.
- **AGENTS.md:** a new "API calls" section covers:
  - which helper to use for each need;
  - the lint rule;
  - how anonymous builds gate login-only features;
- loading `api-base.js` in vm and jsdom tests (the region quick-picks
suite missed that, and failed shard 1 on the first CI run).

### Review follow-ups

- **Account tokens:** the channel badge's pre-filled issue drops the
query on `#/account/…` routes, so mail-link tokens never reach a GitHub
URL. "Open in classic" keeps the full route, since the classic UI needs
the token.
- **HTML answers:** on a packaged UI (static host, web container),
`api()` rejects an HTML answer before status and retry handling. An HTML
404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server
returned an unexpected response" after one request. It used to give a
generic error, or six "warming up" retries for a 503. JSON errors and
the server's JSON 503 warm-up behave as before. The embedded UI keeps
its old handling (`CSApi.embedded`).
- **Logins:** `docs/deployment.md`, `Dockerfile.web` and
`docker-compose.split.yml` say `userManagement.publicBaseUrl` must be
the web container's public origin.

### Audit fixes

- **No broken logins build:** `build.sh` refuses `AUTH=true` with a
cross-origin `API_BASE`. The server allows no credentialed CORS, so
every API request of such a build was blocked; in Chromium all 12 were,
and the pages stayed empty.
- **No accidental deletes:** `build.sh` refuses an `OUT` that is neither
empty nor a previous build. It used to wipe whatever folder or file
`OUT` named.
- **Lint:** the rule now also reports `/api/` paths in `href`, `src` and
`action` attributes in single-quoted strings and template literals.
`tests/e2e/test-lint-network-calls.js` pins exactly what it reports.
- **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published
on localhost only.
- **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since
v3.16.0 and v3.15.0 predate the static build. It also names the server
version the static build needs: before #2362 the warm-up headers stay
hidden and wildcard previews are refused.

### Audit notes, fixed

- **Cache busting on every page:** the tool pages (`area-map.html`,
`geofilter-builder.html`) load their scripts with `?v=__BUST__`. The
server and `build.sh` replace it in every top-level page, and `_headers`
and the Caddyfile make every page revalidate. Behind Caddy, a returning
visitor could otherwise get a new tool page with an old `api-base.js`.
- **HTTP/3:** the split compose publishes 443 over UDP too, for the
HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with
Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`.
- **WebSocket on `/`:** a Go test pins that an upgrade on `/` still
works, for UIs cached from before `/ws`.

### Second review

- **Trusted proxies:** the docs said to trust the compose subnet, and
also to publish port 3000 if wanted. Together they let a client spoof
its address: the subnet holds the gateway, where a client reaching a
published 3000 through Docker's proxy arrives from. The compose file now
fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web
container's address (`172.30.83.200`), and the docs say to trust that
address alone.
- **Classic UI unchanged:** the HTML-answer notice above applies to
packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as
before, and an unknown `/api` path fails to parse as before, with no
notice.
- **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md
says what the rule cannot see: a URL assembled elsewhere and passed in a
variable.
- **AGENTS.md:** `build.sh` is described as packaging (a copy,
`deployment.js` and `__BUST__`, never a code transform), and rule 3 says
`__BUST__` is filled in every top-level page.
- **CI builds the web image:** the image-check job (no publish) builds
`Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`,
the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It
also validates the split compose file.
- **Cross-origin logins:** a hand-edited `deployment.js` with `auth:
true` and another origin now logs a warning naming the setting, instead
of failing with CORS errors alone.

## Upgrade notes

- **WebSocket path:** the UI's live feed connects to `/ws` instead of
`/`. The server has answered `/ws` since #1793 and still upgrades `/`,
so browsers holding the previous UI keep working.
- A reverse proxy that forwards WebSocket upgrades only for an exact `/`
must forward `/ws` too.
- The nginx example in `docs/deployment.md` (`location /`) already
covers it.
- **Static UI from another origin:**
- The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for
the warm-up banner and wildcard preview origins.
- `build.sh` refuses `AUTH=true` there, since the server allows no
credentialed CORS.
- **Split containers:**
  - `CORESCOPE_IMAGE` is required.
- With user accounts on, `userManagement.publicBaseUrl` must be the web
container's public origin.
- The built-in MQTT broker is published on `127.0.0.1` unless
`MQTT_BIND` says otherwise.
- The compose network is `172.30.83.0/24`, with the web container at
`.200`. If Docker reports that the pool overlaps another network, set
`SPLIT_NET`.

## Testing

- `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with
new tests for busting every page and the WebSocket upgrade on `/`.
- `sh test-all.sh` passes. New unit suites:
- `test-static-build.js`: the build and its input checks, `same-origin`
included, cross-origin logins refused, and an unrelated `OUT` left
intact.
- `test-web-container.js`: the Dockerfile, Caddyfile and compose file
agree on packaging, forwarded paths, cache rules and the server port;
the server image has no default and MQTT stays on localhost.
  - The `api-base`, anonymous mode, badge and `dev-static` suites.
- `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes
47/47, the four pages included; it needs eslint, so CI runs it on E2E
shard 1.
- **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes
11/11. That is the static build on one origin against an API-only server
on another, which CI runs on E2E shard 1.
- **Split containers mode:**
- The image builds, and `caddy validate` passes (Caddy 2.11.4). The new
CI step passes locally, and fails when the Caddyfile stops forwarding
`/api`.
- Trusted proxies, with the stack up: the web container gets
`172.30.83.200`. With only that address trusted, a spoofed
`X-Forwarded-For` sent straight to a published 3000 is ignored, and one
sent through Caddy is replaced with the client's address. With the
subnet trusted, as the docs used to say, the spoofed address was
believed: a deny entry for it rejected the upgrade.
- With `docker-compose.split.yml` on the E2E fixture, seeded the way CI
seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both
through the web container and against the server directly.
  - Headers, checked with curl:
    - `no-cache` on `/`, `/index.html` and `/deployment.js`;
    - zstd on static assets;
    - `/api/*` proxied (`Via: 1.1 Caddy`);
    - a `101` WebSocket upgrade on `/ws`.
  - In Chromium:
- every API request and the WebSocket stay on the web container's
origin;
    - there are no page errors and no horizontal overflow at 390 px;
    - no badge shows on the `static` channel.
- Logins through the web container: with user management on and
`userManagement.publicBaseUrl` set to the web container's origin, a
Chromium run passed every step, all on the web container's origin with
no page errors:
- registration, and activation from the mail link (which uses
`publicBaseUrl`);
    - the session surviving a reload;
    - a CSRF-protected `PATCH /api/account`;
    - logout, then login with the form.
- The same login with the backend's origin as `Origin` is refused with
403. The server checks `Origin` against `publicBaseUrl`, not `Host`,
which is why the docs ask for it.

## Performance

There are no hot-path changes. API calls go through `CSApi.url()`, a
string concatenation. In the web container, Caddy serves files from disk
and compresses them; API responses pass through with one proxy hop, the
same as the all-in-one image's built-in Caddy.

## Notes

- **Per-IP limits:** behind the web container, as behind the all-in-one
image's Caddy, requests reach the server from the proxy's address. The
per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`,
the web container's fixed address. The docs say so.
- **`depends_on`:** the web container starts with the server, not after
its healthcheck, like the all-in-one image's Caddy. Gating on health
would make every `up` wait for the first check (30 s) and fail outright
when a large database loads for longer than the healthcheck allows;
until the server listens, Caddy answers `/api` with 502.
- **Not in this PR:**
- publishing a `corescope-web` image (CI builds and smokes it, without
publishing);
- Cross-origin login—including a static frontend and API on sibling
custom domains—is outside this PR. In the deployments supported by this
PR, account support requires the frontend and API to share one browser
origin, either served together or through a reverse proxy.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:56:09 +01:00
0e23c24847 fix(scope-stats): report total and split unscoped by route (#2364) (#2368)
Fixes #2364.

## Problem

Since #1838, `GetScopeStats` folds every non-transport transmission
(FLOOD 1, DIRECT 2) into `summary.unscoped`, while `transportTotal` and
`scoped` still cover transport routes (0, 3) only. The documented
`transportTotal = scoped + unscoped` no longer holds, and a client
computing an unscoped share against `transportTotal` gets >100% (the
reporter shows `transportTotal 7620`, `unscoped 14320`).

## Fix — additive, no existing field changes value

`summary` gains:

| field | meaning |
|---|---|
| `total` | all observed transmissions, routes 0–3 (= `scoped +
unscoped`) |
| `unscopedTransport` | route 0/3 with Code1 = 0000 |
| `unscopedFlood` | route 1 — unscoped by protocol |
| `unscopedDirect` | route 2 — unscoped by protocol |

Invariants (now asserted in `TestGetScopeStats`):

```
transportTotal = scoped + unscopedTransport
unscoped       = unscopedTransport + unscopedFlood + unscopedDirect
total          = scoped + unscoped
```

The non-transport count query keeps its single scan; it just splits
route 1 vs 2 with `CASE`.

On the design question in the issue: rather than changing what
`unscoped` means (the Scopes tab and #1842 rely on it), the split lets a
rollout view use `unscopedFlood` directly: repeaters apply region
rejection to flood routes only (`isRouteFlood()` in
`examples/simple_repeater/MyMesh.cpp`), so DIRECT traffic is not subject
to flood-region filtering. `docs/api-spec.md` now says so, spells out
the route mapping, and notes that `timeSeries` is transport-only (its
`unscoped` matches `unscopedTransport`).

## Frontend

Scopes tab uses `summary.total` when present and shows `N transport, N
flood, N direct` on the Unscoped card; falls back unchanged for older
servers. The cards are built by `scopeSummaryCardsHtml(summary)`
(exposed as `window._analyticsScopeSummaryCardsHtml`).

## Tested

- `go vet ./...` (cmd/server) clean, `gofmt -l` clean
- `go test -run ScopeStats ./...` — pass (incl. extended
`TestGetScopeStats` and `/api/scope-stats` route test)
- `node --check public/analytics.js`;
`tests/unit/test-issue-1375-scope-stats-fetch.js`,
`test-issue-1979-scope-adverts-by-role.js` pass
- new `tests/unit/test-issue-2364-scope-summary-cards.js` (vm sandbox,
11 assertions: shares of `total`, route split, fallback for servers
without `total`/split, partial split, empty window); full `test-all.sh`
green

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-11-15 12:24:27 +01:00
n30nexandn30nex c2e770ef65 docs: archive superseded repeater liveness plans (#2392)
Partial fix for #1385 (PR-E).

Move the two April repeater-liveness documents from `docs/superpowers/`
into existing `docs/archive/` with `git mv`. Preserve their dated
filenames, content and headings; add historical notices clarifying that
the proposed `relaying` status/signature is outdated and linking current
guidance.

Validation: exact commit blobs preserve 41,483 original bytes and 24
headings after removing only notices; four local links resolve; no
tracked old-path references; whitespace and PII checks pass.

TDD exemption: pure documentation, no behavior change. No runtime,
configuration, customizer, test, hierarchy or ignore-rule changes.
Application suites were not rerun. Both published Markdown pages and
their guidance links render correctly. [Docs-only
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43199869942)
and three independent reviews pass. The umbrella issue stays open
pending maintainer reconciliation.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-15 12:23:50 +01:00
efiten 6dd62711bc docs(release): v3.17.0 release notes and changelog entry (#2390)
Release notes for v3.17.0: `docs/release-notes/v3.17.0.md` (used as the
GitHub release body by `deploy.yml`) and the `CHANGELOG.md` entry.

24 commits since v3.16.0, counted with #2389, which is merged before
this PR. Issues closed come from each PR's `closingIssuesReferences`.

Docs only. After merge: tag `v3.17.0` on the merge commit once master CI
is green and the `:edge` image carries that commit.
2026-11-11 00:26:33 +01:00
efiten 873e8b0e16 feat(rx-coverage): QR code and link to the operator's CoreDrive RX deployment (#2389)
## What

An operator who runs a CoreDrive RX deployment (the web and Android
companion app that feeds `clientRxCoverage`) can now point visitors of
the Coverage page at it.

- New optional config key `clientRxCoverage.appUrl`.
`/api/config/client` sends it as `clientRxAppUrl`, only while
`clientRxCoverage.enabled` is true and the value is an absolute `https`
URL with a host name. Any other value is left out and logged once at
startup with a `[config]` line.
- `#/rx-coverage` shows a panel under the subtitle when the field is
present: a QR code of exactly that URL (vendored `qrcode.js`, same call
as the node detail QR), the URL as a link, and one line on scanning it
in the app or opening it on a phone. Without the field there is no
panel. The page still needs no login.
- The subtitle named the app "CoreScope-RX" and linked to
`github.com/efiten/corescope-rx`, the app's old repository name. It now
says "CoreDrive RX" and links to `github.com/efiten/coredrive-rx`, or
carries no link when the panel already shows the deployment.

## Why

The CoreDrive RX Android app is not tied to one server: on first start
it asks for the address of a deployment. Scanning a QR code on the
Coverage page is the shortest way for a visitor to get there, including
people without an account.

Disclosure: CoreDrive RX (`efiten/coredrive-rx`) is my own project.

## Changes

- `cmd/server`: `config.go` (`validClientRxAppURL`, startup warning),
`types.go`, `routes.go`, `main.go`, `openapi.go`.
- `public/rx-coverage.js` (`appPanelHtml`, `appLinkHtml`),
`public/roles.js` (`window.MC_CLIENT_RX_APP_URL`),
`public/node-reach-coverage.css` (existing CSS variables only).
- Docs: `config.example.json`, `docs/client-rx-coverage.md`,
`docs/api-spec.md`.

## Tests

- 3 new Go tests in `cmd/server/client_rx_app_url_test.go`: URL
validation over 15 inputs, the endpoint field in 6 config states, the
startup warning. Full `go test ./cmd/server` passes.
- `tests/unit/test-rx-coverage-app-qr.js` (real `rx-coverage.js` and
`qrcode.js` in a vm): panel present and absent, QR content, HTML
escaping, subtitle link in both cases, old names gone. Added to
`test-all.sh`. All 6 `tests/unit/test-rx-coverage-*.js` pass.

## Not done

- No E2E test.
- Browser-checked on a local server with and without `appUrl` in the
light theme only. The QR keeps black modules on white in both themes,
unlike the node QR, which is inverted in dark mode; dark mode itself was
not checked in a browser.
- `cmd/ingestor` is unchanged; it does not need the URL.
- The same change runs on analyzer.on8ar.eu (a fork build) since
2026-11-10.
2026-11-11 00:25:32 +01:00
efitenandClaude Opus 5.5 fad303a4f4 docs: split the Docker minimum into running and building the image (#2361)
`docs/deployment.md` listed Docker 20.10+ as the minimum, without
separating running the published image from building it. The Dockerfile
needs BuildKit (`# syntax=docker/dockerfile:1` on line 1, `RUN --mount`
from line 71), and BuildKit is the default builder only from Docker
23.0. On 20.10 to 22.x the legacy builder rejects `RUN --mount` unless
`DOCKER_BUILDKIT=1` is set. Found while reviewing #2359.

Changes:
- `docs/deployment.md`: the requirements table gets two Docker rows,
20.10+ to run the published image, and 23.0+ (or 20.10+ with
`DOCKER_BUILDKIT=1`) to build it, plus one sentence on why.
- `docs/go-migration.md`: that guide builds the image, so its
prerequisite becomes 23.0+ (or 20.10+ with `DOCKER_BUILDKIT=1`).

Not verified: I did not run a build on Docker 20.10 or 22.x; the
legacy-builder behaviour comes from Docker's documentation, not from a
test here. Podman and buildah are not covered.

Docs only.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 00:00:52 +01:00
efitenandClaude Opus 5.5 1577b57d67 docs(accounts): approved channels are backfilled since #2133 (#2385)
`docs/user-guide/accounts.md` said that messages received before a
channel proposal is approved stay encrypted. Since #2133 that is no
longer the case: when the key set grows, the ingestor re-decodes the
stored rows it could not decrypt (`cmd/ingestor/key_backfill.go`), and
the server tails `tx_rewrite_feed`, so no restart is needed.

Checked on an instance running #2133: approving four hashtag channels
triggered one pass (`[key-backfill] channel: 4 of 410 key(s) not yet
applied to history`, then `pass complete, 801 row(s) rewritten`, 13 s),
and the 801 rows are exactly the stored messages of those four channels,
the oldest from three months before the approval.

Docs only, one sentence. Not done: I did not check the other docs pages
for the same claim beyond a grep for "stay encrypted" and "received
before" in `docs/` and `README.md`, which found only this one.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 22:31:07 +01:00
199df07c21 docs(proto): correct route_type values to match the decoder (#2363) (#2367)
Fixes #2363.

The published contract described `route_type` as `0=DIRECT, 1=FLOOD,
2=reserved, 3=TRANSPORT`. Both decoders (`cmd/ingestor/decoder.go`,
`cmd/server/decoder.go` — `routeTypeNames`) and MeshCore's
`docs/packet_format.md` use:

```
0 = TRANSPORT_FLOOD, 1 = FLOOD, 2 = DIRECT, 3 = TRANSPORT_DIRECT
```

Updated:
- `proto/decoded.proto` — `DecodedHeader.route_type` and the
`route_type_name` value list
- `proto/packet.proto` — `Transmission.route_type` (same stale comment)
- `docs/api-spec.md` — Packet Object `route_type` line (same stale
comment)

Comment/doc-only; no generated code in-tree, no behaviour change.

Co-Authored-By: Claude <noreply@anthropic.com>

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-11-08 17:12:37 +01:00
n30nexandn30nex b3a9f2a9e9 docs: make scheduled backups portable and bounded (#2353)
Fixes #2352. Follow-up to #2345.

The older deployment guide recommended copying a live telemetry
database. It now points to the consistent-snapshot procedure. The
scheduled-download example uses a trailing-X temporary filename, bounds
connection and total download time, and explains the weak-key denylist.

Validation: the parent reran the extracted command against eight
disposable HTTP cases: valid SQLite, unauthorized, server error, empty
content, HTML, corrupt SQLite, truncated transfer and timeout. Failed
downloads removed partial files and preserved existing completed
backups; the timeout exited 28. Shell syntax and diff checks passed. GNU
tools were exercised; BSD/macOS was not run natively.

TDD exemption: pure documentation, no application or repository test
files changed. No runtime/browser behaviour, dependencies or
configuration changed. Independent reviews are complete; the one
timeout-wording finding was addressed in 33d629b5 without changing the
validated command. Exact-head CI passed, including the browser shards.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-31 11:59:39 +01:00
efitenandClaude Opus 5.5 faf6f792ae docs(release): v3.16.0 release notes and changelog entry (#2355)
Release notes for v3.16.0 (`docs/release-notes/v3.16.0.md`) and the
matching `CHANGELOG.md` entry. The tag pipeline uses this file as the
GitHub release body.

Range `v3.15.0..af2e96d9`, 47 commits: 23 fix, 9 feat, 6 docs, 2 test, 2
perf, 2 ci, 1 ui, 1 deps, 1 chore.

"Read this before upgrading" covers three changes:
- relay hops refresh Last Heard only with a unique prefix and only on
flood packets (#2334, #2308, #2298, #2336);
- `users.db` schema v6 to v8 with user accounts on (#2306, #2348);
- Go 1.26 for source builds and Alpine 3.24 in the image (#2297).

Not verified: the mosquitto, Caddy and supervisor versions are taken
from #2297's local build, not from a pull of the published image.
Performance numbers are copied from the PRs, not re-measured.

Docs only, no code change.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 23:11:30 +01:00
af2e96d92a fix(reach): queue a cold request up to 3 s for a build slot before answering 429 (#2249)
Follow-up to #2127, from @dborup's review there. The limiter, most of
its tests and the docs text are dborup's (dborup/CoreScope@195ec4c3),
adapted to master; the commit carries `Co-authored-by: dborup`.

## What changes

`GET /api/nodes/{pubkey}/reach` still runs at most 2 uncached reports at
once. What changes is the next uncached request:

- **Before:** 429 at once, `Retry-After: 5`.
- **After:** it waits up to 3 s for a slot. It gets 429 with
`Retry-After: 2` and a JSON body only if no slot frees in that time.

Unchanged: cached answers never wait, and concurrent requests for the
same node and `days` share one computation and one slot.

One addition beyond dborup's version: the shared computation runs on the
context of the request that started it. If that request disconnects,
while queued or mid-scan, the other requests sharing it that are still
connected now get 429 instead of 500. The 3 s wait made that case more
likely.

## Why

dborup's three points on #2127:

1. An instant 429 fails two tabs, or a quick switch between 7/14/30
days, when the database is not under real pressure.
2. The page cannot recover from it, because `api()` retries only 503.
The panel retry is a separate PR so that this one stays server-only.
3. No test covered the promised property: a cached request still answers
while every slot is held.

Waiting parks a goroutine, not a database connection, so the pool is
protected exactly as before.

## What I checked

- **Tests.** `make test` (all 16 modules), `make vet`, `gofmt`, `sh
test-all.sh` and eslint (0 errors) all pass. The reach tests also ran 10
times under `-race`.
- **New tests.** 14 tests in `node_reach_cold_limit_test.go`: dborup's
10 plus 4 new. `TestNodeReachConcurrentBuildCap` now drives the new
limiter through the production router. Point 3 is
`TestNodeReach_WarmHitSkipsColdLimit`. The 4 new tests are:
  - a queued request is served when a slot frees;
  - a cache filled just before the flight skips the limiter;
- coalesced waiters get 429 when the starter disconnects, queued or
mid-scan (two tests).
- **Mutations.** I wrote 21 from the code before running the tests, and
the tests caught all 21: instant reject, cap +1, release that leaks,
slot taken before the in-flight recheck, busy → 500, Retry-After header
or value, status 503, dropped context mapping, and others.
- **Real binary.** Master `2d09335c` and this branch, run one after the
other on a copy of a 10.5M-observation, 11 GB database. Each burst fires
its reach requests at once, while `/api/nodes?limit=50` and
`/api/packets?limit=50` are polled every 200 ms.

| Burst (distinct uncached keys) | master | this branch |
|---|---|---|
| 3 keys, `days=7`, warm page cache (2 runs) | 2 × 200, 1 × 429 after
0.01 s | 3 × 200; the third answered at 4.1–4.3 s |
| 6 keys, three of them `days=14` (2 runs) | 2 × 200, 4 × 429 after ≤
0.01 s | 2 × 200, 4 × 429 after 3.0 s |
| 24 keys, cold page cache | 2 × 200, 22 × 429 | 2 × 200, 22 × 429 after
3.00 s |
| `/api/nodes` during every burst | median 1 ms, max 53 ms | median 1
ms, max 66 ms |

## Not checked

- **Slow scans.** On a database this size, uncached scans took from
about 9 s to over 60 s, so the 3 s wait only helps when page cache is
warm or the database is smaller. Otherwise the result is the same 429, 3
s later. A 30-day uncached scan hit the 60 s `WriteTimeout` on both
builds; that is the scan #2214's opt-in hop index addresses.
- **Pool contention under a live store.** On this copy the in-memory
store loaded 0 packets, so `/api/nodes` latency does not show contention
under a live load.
- **Platforms.** One machine (linux/amd64). No browser check: the server
change has no UI, and the panel retry comes with its own PR.

**Base.** Rebased on master `7c00a429`, which includes #2214; every test
and mutation above was re-run on the rebased commit. The real-binary
runs used `2d09335c`, just before #2214, on a database without the hop
index. That is the default after #2214 too (`hopIndexDays` unset means
off). With the index on, scans are shorter, so more bursts fit in the 3
s wait; I did not measure that.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


---

## Update after review (commits `c65f4d3b`, `1d2cd99d`, `04e49854`)

efiten asked how many requests may wait at once. With only the 3 s
timeout, the number of waiting distinct keys grew with the request rate.
n30nex wrote a cap (nullrouten0/Baymc-CoreScope#1, tracked in #2264).
Its two commits are now on this branch unchanged, as a fast-forward.

- At most 8 distinct cold requests wait for a slot. Any further one gets
429 with `Retry-After: 2` at once, before a timer is set.
- A waiter gives its place back on every exit: it gets a slot,
disconnects or times out.
- Cached answers and same-key sharing are unchanged. Requests for a key
that is already queued or building wait on that one place and are not
counted. `04e49854` corrects the code comment that said the ceiling was
independent of request rate.

**What I checked on `04e49854`:**
- The two new tests fail on `28da016f` and pass now.
- 12 mutations of the cap, all caught: cap 7 and cap 9; the place never
returned, returned before waiting, returned only on promotion, leaked on
cancel, leaked on timeout; a full queue that blocks, returns the wrong
error or skips the counter; a timer armed before the check; a queue
sized by the slot count.
- `make test`, `make vet`, `make fmt-check`, `sh test-all.sh` and eslint
8.57.1 (0 errors) pass, and the reach tests pass 5 times under `-race`.
On a local merge with current master `660cbbf5`, which includes #2277,
the reach tests pass 3 times under `-race` and the `cmd/server` package
passes.
- Real binary: both heads, alternately, on a copy of a 2.5 GB soak
database, with 24 distinct uncached `days=7` requests fired at once, two
runs each:

| | `28da016f` | `04e49854` |
|---|---|---|
| 200 | 2, 2 | 2, 3 |
| 429 at once | 0, 0 | 14, 14 |
| 429 after 3 s | 22, 22 | 8, 7 |

**Not checked:** a sustained flood, a burst of same-key requests against
the cap, `/api/nodes` latency during these bursts, and platforms other
than linux/amd64. The real-binary table further up was measured before
the cap.

---------

Co-authored-by: nullrouten0 <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: dborup <kontakt@meshview.dk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: n30nex <blushellneon@outlook.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-10-29 22:50:59 +01:00
n30nexandn30nex 1ed5e4ab3f fix(compare): exclude observer TX-only packets from overlap (#2346)
Red commit: `39295da8` ([intended CI assertion
failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)).
Green commit: `d1bda496`.

Observer comparison counted an observer's TX-only packets as receptions,
producing false overlap. It now requests `excludeTx=true`: the selected
observer and non-TX direction must match the same observation in memory
and SQLite v2/v3. Normal packet lists and stored TX records remain
unchanged; missing directions retain legacy compatibility.

Refs #667. This completes a comparison slice, not the issue's remaining
observer counters or #671's fleet/SNR feature. Representative display
fields and existing time-filter semantics are unchanged. API/OpenAPI
docs describe the filter and multi-node bypass.

Validation:
- RED: Go tests compiled with 32 intended failures/16 control passes;
unit and browser regressions reproduced false overlap. CI reached the
new unit assertion.
- GREEN: all 48 packet cases pass, plus grouped-cache isolation and
OpenAPI tests. The added cache regression caught a malformed-radio
cache-key collision before correction.
- Full local server suite: 174s; ingestor: 185s; `npm run test:unit`:
pass. Final focused packet/grouped/OpenAPI checks: pass.
- E2E assertion added to
`tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks
pass, including correct 1/2 reception totals and 50% overlap.

Performance: still two capped bulk requests, indexed candidate lookups
retained, one direction check within the existing observation scan/SQL
predicate, no new persistent structures. No configuration/customizer
changes.

Final-head CI passed; independent reviews found no must-fix items.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:34:06 +01:00
n30nexandn30nex 8437d22356 docs: use consistent snapshots for backup and restore (#2345)
The deployment guide copied a running `meshcore.db`, which can omit
committed WAL data or produce an inconsistent backup. It now uses the
existing authenticated `/api/backup` snapshot, checks the download
before publishing it, and explains how to restore with all database
users stopped while preserving the old database and sidecars for
rollback.

Fixes #703, following the [maintainer's remaining documentation
scope](https://github.com/Kpa-clawbot/CoreScope/issues/703#issuecomment-6163256620).

- Protects API credentials in a header file and keeps incomplete
downloads out of the completed backup set.
- Links the separate account-backup guide; configuration and secrets
remain separate backup responsibilities.
- Replaces unsafe live-copy scheduling instructions with the same
checked download procedure.

Validation:
- Exact shell example with real curl/SQLite: valid download, HTTP
401/500, empty, HTML, corrupt database, and truncated transfer. Failures
preserved the previous backup and published no new `.db`.
- Existing `TestBackup*` passed. Chromium downloaded an authenticated
snapshot from a disposable instance; a row committed only in its
nonempty WAL survived and `quick_check` returned `ok`.
- Disposable stopped-file restore and rollback recovered the original
WAL-only row and preserved the original files byte-for-byte.
- `git diff --check` and privacy preflight passed.

TDD exemption: pure docs; no application behavior changed and no test
files modified. No new config/customizer settings. Live operator
services were not used for validation.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:33:14 +01:00
efitenandClaude Opus 5.5 b65412ab83 feat(observer-logins): turn on neighbour reports in the firmware block of the credentials mail (#2349)
The firmware block of the observer credentials mail now ends with `set
mqtt.neighbors on`, so firmware observers send the same `/neighbors`
report that the OpenHop block already turns on ("Publish Neighbours",
#2347). The ingestor reads that report for region scopes (#1865).

## Changes

- `cmd/server/observer_mail.go`: one extra line in the firmware block,
and one sentence in its note. The sentence says the command sends the
neighbour table and region scopes once a day, and that older firmware
does not know it and should skip it.
- `cmd/server/observer_mail_test.go`: the full firmware block assertion
includes the new line, and the slot test checks the line and the note.
- `docs/observer-mqtt-logins.md`: the mail section describes the new
line.

## Tests

- `go test ./...` in `cmd/server`: ok. Skipped
`TestSaveGeoFilterPreservesFileMode`, which needs Unix file modes; this
run was on Windows.
- `TestObserverCredentialsMail*` failed before the change, on the
missing line, and passes after it.

## Not checked

- The command name and the 24-hour default come from #1865. I did not
read the observer firmware source.
- The wording of the error older firmware gives for an unknown command
is not quoted in the mail. The note only says to skip the line.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 06:25:42 +01:00
efitenandClaude Opus 5.5 cadc666bd9 feat(observer-logins): OpenHop block in the credentials mail (#2347)
The observer credentials mail (#2306) had blocks for meshcoretomqtt, the
Gessaman/Dutch-MeshCore firmware, Home Assistant and plain values. An
operator running OpenHop (repeater software with its own MQTT uploader)
could not tell which of these matched its broker form, and picked the
format by trial.

## Changes

- `cmd/server/observer_mail.go`: a fifth block, "4. OpenHop (repeater
software)", before the plain values. It uses the field and option labels
of the OpenHop broker form:
- `Format: Letsmesh`, `Transport: TCP` or `Websockets` (from
`observerLogins.transport`), host, port, username, password, `TLS:
on/off`, `Use JWT Auth: off`, `Retain Status: on`, `Publish Neighbours:
on`.
- The note says Meshcoretomqtt works too, but the format labelled
"openHop" (value `mqtt`) does not: it publishes to
`meshcore/repeater/<name>`, which a personal login may not use.
- Publish Neighbours is on because the ingestor reads the `neighbors`
topic for region scopes (#1865).
- `openapi.go` and `docs/observer-mqtt-logins.md`: "four formats"
becomes five.

## Sources

Read from OpenHop, not tested on a device by me:
- format family and topics:
`openhop_repeater/repeater/data_acquisition/mqtt_handler.py`
(`MC2MQTT_FORMATS = ("meshcoretomqtt", "letsmesh", "waev")`, the `mqtt`
format's `meshcore/repeater/<node_name>` base topic, transport values
`tcp`/`websockets`);
- labels: the built web UI (`Configuration-*.js`: "Add Broker", option
labels `Letsmesh`, `Meshcoretomqtt`, `openHop`, `TCP`, `Websockets`,
"Retain Status", "Publish Neighbours", "Use JWT Auth").

An operator on our instance set it up from a screenshot of that form
with format Letsmesh, TCP, Retain Status and Publish Neighbours on; the
observer publishes, and its `neighbors` report set `configured_scope` on
its node.

## Tests

- `TestObserverCredentialsMailOpenHopBlock`: the block text for TCP
without TLS and for Websockets with TLS, the "openHop" warning, HTML
escaping, and the block order.
- `TestObserverCredentialsMailBlockTitles`: five titles in order.
- `cmd/server` suite green locally (the two Windows-only failures
skipped, as on master).

## Not in this PR

- The websocket path: OpenHop uses paho's default, the doc already tells
operators to match the listener path.
- Anything with the other fields of the `neighbors` report (SNR,
heard_secs_ago, default_scope): we are measuring how many observers send
it first.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:31:43 +01:00
efitenandClaude Opus 5.5 d29957fbdc feat(observer-logins): lock the shared login out of migrated observers (#2348)
Follow-up to #2306 / #2319. With personal observer logins, a migrated
observer still is not safe from the shared login: the hook answers
`ignore` for the shared login, so EMQX's built-in database lets it
publish under any observer's topic, including one whose owner has moved
to a personal login. This PR lets an operator close that per migrated
observer, before the shared login is removed altogether.

## Changes

**users.db v8.** `observer_bindings.confirmed_at`: the moment the
owner's personal login first published under that pubkey.
- An auto-bind (the owner's first publish) is confirmed at once.
- An admin assignment stays unconfirmed until the owner's login
publishes under that pubkey; that publish sets it.
- Reassigning to another user or unbinding clears it; reassigning to the
same owner keeps it.
- Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin`
rows stay NULL.

**Config.** `userManagement.observerLogins.sharedUsernames` (default
empty): the broker's shared login names. A name starting with `obs_` is
refused at startup.

**Authz hook.** For a login in `sharedUsernames`, a publish under
`meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is
confirmed, whatever the number of topic segments (the ingestor
attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey).
Logged as `publish denied: user=<name> observer=<8 hex>: confirmed
owner`. Every other request from a shared login keeps `ignore`, so the
built-in database decides as before. Personal logins are unchanged.

An orphaned binding (owner account deleted) does not count as confirmed.

**Admin.** The observers table of the Observer logins tab gets a
Confirmed column; the bindings JSON gets `confirmedAt`.

**Docs.** `docs/observer-mqtt-logins.md`: "Locking the shared login out
of migrated observers" (what it denies, how to see `confirmedAt`, the
risk below). `config.example.json` has the key.

## Risk for operators

An observer that is confirmed but still runs a second uploader on the
shared login loses that second uploader once its name is in
`sharedUsernames`. The deny log line names the observer.

## Tests

- `internal/users`: v7 to v8 migration on a real v7 database, auto
confirm, admin assignment then owner confirm, non-owner and repeated
confirm, reassign clears, orphan reassign.
- `cmd/server`: the decision table (confirmed, unconfirmed, unbound,
non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty
and `client` iata, lowercase key; owner publish confirms, confirm
failure still allows and retries; reassign, unbind and user removal
clear the flag; the index load against a real store; config validation;
the hook and admin endpoints end to end.
- `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an
auto-bound observer.
- Local: `internal/users`, `cmd/server` (two Windows-only tests skipped,
as on master) and `cmd/ingestor` green. The E2E assertion was not run
locally; CI runs it.

## Not in this PR

- Using `confirmed_at` for verified `/neighbors` reports: a separate PR,
built on #2300.
- Removing the shared login: still an operator step after migration.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:10:41 +01:00
n30nex 410db29fe6 chore: remove unused legacy Dockerfiles (#2314)
Fixes #2310. Partial follow-up to #1385 (PR-C).

Remove unused `Dockerfile.go` and `Dockerfile.node`, which describe
retired alternate builds. Current workflows, Makefile, Compose variants
and operator scripts already select the canonical `Dockerfile`.

Remove the obsolete formatter exclusions and label the old Node-to-Go
migration guide historical, linking current deployment instructions.
Historical release prose remains untouched. The canonical image, runtime
files, dependencies and Compose behavior are unchanged.

Validation: audited all 1,501 tracked files; old and new formatter
inputs are exactly the same 633 real Go files. Both formatter commands,
YAML/shell/workflow-reference checks, existing workflow/routing tests,
COPY gate and test inventory pass. All 218 standalone frontend suites
pass. The unchanged backend tree passed full server/ingestor suites, and
the unchanged app passed 12 browser smoke cases.

TDD exemption: behavior-preserving cleanup/refactor plus documentation;
no test files modified. `make` and `actionlint` were unavailable
locally; equivalent formatter and focused workflow checks ran directly.

[CI passed on this
head](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40230911566),
including canonical container builds and all browser shards. Three
independent reviews passed. The published historical guide and its
current-deployment links render correctly in Chromium. No
settings/customizer changes.
2026-10-26 15:41:34 +01:00
n30nex 23f938d1d4 docs: consolidate historical release notes (#2313)
Fixes #2311. Partial follow-up to #1385 (PR-B).

Five historical release notes lived outside `docs/release-notes/`. Move
v2.6.0, v3.0.0, v3.1.0, v3.4.0 and v3.6.0 into that directory using its
existing version naming convention.

Release content is preserved except for concise historical notices and
four relative-link repairs. The v3.0.0 notice identifies its obsolete
Node rollback instructions; current setup links point to README and
DEPLOY. No new documentation hierarchy or index is introduced.

Validation: all five moves are one-to-one; every document matches its
original bytes after only the listed edits; all 14 local Markdown links
resolve; no old-path references remain. Existing external links are
unchanged. All 218 standalone frontend suites pass. The unchanged
backend tree passed full server/ingestor suites, and the unchanged app
passed 12 browser smoke cases.

All five published documents were rendered in Chromium; historical
notices and repaired link targets were verified. Three independent
reviews passed.

TDD exemption: pure documentation; no application, test, CI,
configuration or deployment behavior changed. No settings/customizer
implications. [CI
passed](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40230718673);
its docs-only classification correctly skipped application jobs. The
full-suite results above are local validation.
2026-10-26 15:40:38 +01:00
n30nex 60615784d9 docs: archive historical planning documents (#2321)
Fixes #2318. Partial follow-up to #1385 (PR-A).

Move eight historical planning documents from the root into
`docs/archive/`, keeping their original filenames and headings. Each
gets a brief archived-plan notice linking current README and deployment
guidance. Historical proposals, implementation claims and code examples
remain intact.

Validation: eight one-to-one renames, byte-preserved original content,
16 valid added links, and no inbound or executable references requiring
updates. Seven already-absent historical source paths remain plain text
under the archive notice; no replacement links were invented. All other
tracked files are unchanged.

All 218 standalone suites passed locally. The unchanged backend tree
passed full server/ingestor tests, and the unchanged app passed 12
browser smoke cases. TDD exemption: pure documentation; no test,
application, configuration or deployment change. No customizer
implications.

All eight published documents and their guidance links were verified in
Chromium. Three independent reviews passed. [CI
passed](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40238488019);
its docs-only classification correctly skipped application jobs. The
full-suite results above are local validation. This only addresses PR-A;
other cleanup remains tracked separately.
2026-10-26 15:40:24 +01:00
efitenandClaude Opus 5.5 e149568c6c feat(observer-logins): personal MQTT logins for observers (#2306)
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.

Closes #2319.

The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.

## Changes in this PR

**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.

**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.

**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.

Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.

**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.

**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.

**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.

**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.

## Enabling it on an instance

Full steps in `docs/observer-mqtt-logins.md`. In short:

1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.

## Who can claim an observer

Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.

What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.

## Performance

`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.

## Verification

- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.

- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.

## Known behaviour

EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.

## Not in this PR

- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.

Note on cb9f4ff3: its message says the scheme problem was reported by an
operator. That report turned out to come from before the broker used the
hook, so it does not show this cause; the change rests on the firmware
source only.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 09:50:20 +01:00
efitenandClaude Opus 5.5 7fa86fe1c1 docs(release-notes): v3.15.0 (#2263)
Release notes for v3.15.0: `docs/release-notes/v3.15.0.md` and the
`CHANGELOG.md` entry. The release workflow uses the notes file as the
GitHub release body.

## What is in it

- 77 commits since v3.14.0 (76 merged pull requests plus this one): 27
fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs, 1 chore. Nineteen feats, so
minor.
- "Read this before upgrading" first. It covers:
  - the liveness change (#2261, #2262);
- the `users.db` v6 migration and what it means for a rollback (#2174);
- the backfill at every ingestor start (#2133) and the 24 h
planner-statistics refresh (#2158);
- the caches that trade freshness for latency (#2165, #2222, #2147,
#2156);
- changes in hop names and region placement (#2188, #2190, #2234, #2245,
#2220);
  - the role thresholds on favorites (#2215);
  - the geo-filter save on a single-file mount (#2137);
  - the `/api/stats` polling in hidden tabs (#2162).
- Features, fixes, performance, then tests/CI/docs.
- 29 issues closed, taken from each pull request's
`closingIssuesReferences`.
- Contributors: pull request authors and issue reporters with counts.
The commit co-author trailers only name the pull request's own author or
tooling.

## How it was checked

- I read every pull request body. Every number in the notes comes from a
pull request body.
- I checked the config keys and defaults against `config.example.json`
on `2bb000f2`.
- I checked the UNASKED and opt-in claims for #2125, #2133, #2235 and
#2239 against the source.
- For the other pull requests I relied on their bodies and did not open
the diffs.

## Not covered

- The uncertain count is a single count on one staging instance on the
day of the change (1,682 repeaters and rooms). It is not a before/after
comparison.
- #2133 gives no measured cost for its backfill on a large database, and
the notes say so.

## After merge

Tag `v3.15.0` on the merge commit once its `:edge` image exists. Then
check in the registry that `v3.15.0`, `v3.15`, `v3` and `latest` share
one digest, and that the release has the notes as its body and both
`corescope-decrypt` binaries.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 21:39:49 +02:00
efitenandClaude Opus 5.5 2bb000f2aa feat(notify): adaptive offline threshold and certain node analytics (#2260) (#2262)
Part 2 of #2260. **Builds on #2261** (certain liveness): review/merge
that first; until then this diff also shows its commit.

## Adaptive offline threshold

`userManagement.notifications.adaptiveOffline` — **off by default**:

```json
"notifications": {
  "enabled": true,
  "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 }
}
```

The fixed `infraSilentHours` (72 h) treats every repeater the same. With
this on, a watched node's `node.offline` window is learned from its own
**certain** traffic (the #2261 rule):

1. Certain packets per hour over the week **before its last certain
packet**. Ending there, not at now, keeps an outage from dragging the
rate down until the fixed window returns and the node reads "back
online".
2. Empty runs are classified: ≤ 6 h and recurring on another day at the
same hours = normal silence (kept; the window is never shorter than the
longest + 1 h); any other run = outage, left out of the learning so it
neither stretches the window nor blocks learning.
3. λ = 10th percentile of the remaining hourly counts, so quiet nights
don't alert.
4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson,
ln 10⁶ ≈ 13.8).
5. Clamped to [`floorMinutes`, the role's silent window].

Kept on the fixed window: λ < 1/hour, under 24 h of history in memory,
or a 10th-percentile hour heard by fewer than `minObservers` observers
(one observer dropping out must not read as the node going offline).
Silence is measured up to the newest packet in the store, so a stalling
feed stops every node's clock. A node stored offline comes back only on
evidence from the moment it went offline on. The notifier caches each
learned window for 10 minutes and drops unwatched nodes.

For the #2260 repeater (~840 packets/day) the mail would have gone out
about 30 minutes into the outage, instead of never (or 72 h with #2261
alone).

## Certain node analytics

- Each `activityTimeline` bucket carries `certain`: the part that can
only be this node (`certainFor`, the packet-level form of the
`certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other
endpoints are unchanged.
- Availability and longest silence use certain buckets only; the longest
silence now includes one **still running** (an outage in progress is the
silence worth seeing).
- The chart stacks certain (`--accent`) and uncertain (`--text-muted`) —
existing theme variables, so no new customizer entry. Descriptions
updated.

## Performance

`CertainActivity` walks a week of packets per watched node, at most once
per node per 10 minutes, with the read lock taken **per node**:

```
BenchmarkCertainActivity-16   12.1 ms/op   (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold)
BenchmarkNoteCertainRelays-16  121 ns/op   0 allocs
```

Node analytics adds one `certainFor` per packet of the requested node (a
prefix-map lookup, or the cached decoded JSON for its own packets).

## Tests

- `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet
nights set the rate; > 10 % empty hours, short history, too few
observers → fixed.
- `CertainActivity` ends at the last certain packet and counts only
certain packets — verified red by mutating the window to end at now.
- Notifier end to end: learned window passed → bad; inside → good; too
few observers / setting off → fixed 72 h; cache refresh and prune.
- `certainFor`, `longestSilence` (incl. running silence), config
defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket.
- Frontend: `NodeAnalyticsTimeline.series` (new
`tests/unit/test-node-analytics-certain-timeline.js`).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork.

## Validation

Server on the freshened e2e fixture: analytics buckets carry `certain`
(e.g. 89 of 90 packets certain for one repeater). **No browser
screenshot**: Playwright isn't installed on this machine.

## Later (rule 8)

`floorMinutes` / `minObservers` are server config, not theme settings;
an admin UI for notification settings would be the place, not the
customizer.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 21:20:42 +02:00
efitenandClaude Opus 5.5 a08f2f4e7c fix(liveness): count only evidence that can be the node alone (#2260) (#2261)
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold
follows in a separate PR).

## Problem

A repeater silent since 2026-10-18 showed **Active — Last relayed 4m
ago** and **Last Heard 36m**, and its offline mail never fired. Every
hit after that date was the 1-byte hop `67`, shared by nine nodes:

- the live resolver's affinity tiebreak (`resolvePathForObs`) put those
transmissions in `byNode`, which fed `stats.lastHeard` and
`LastHeardMap`;
- `last_relayed` folds in the 1-byte prefix bucket on purpose (#662),
and drove the status in `roles.js` and the notify offline check.

## Fix

The store keeps `certainHeard` per node: the newest transmission that
can only be that node.

| Evidence | Counts? |
|---|---|
| Own packet (`pubKey`, `srcPubKey`) | ✅ |
| Flood relay hop whose prefix, at the packet's path hash size, matches
exactly one path-capable node | ✅ |
| 1-byte hop shared with other repeaters | ❌ |
| Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌
|
| TRACE (path bytes are SNR) | ❌ |
| Destination (`destPubKey`) | ❌ |

A multibyte repeater legitimately appears as a 1-byte hop (the
originator picks the hash size, `Mesh::routeRecvPacket` appends at that
size), so only uniqueness decides. The rule is applied in every load
path through `indexResolvedPathHops` and the new
`indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live
ingest, late observations.

Liveness reads only that:

- `/api/nodes` and the node detail: `last_heard` is the newer of
`last_seen` and the certain last heard. Before, it was a plain copy of
`last_seen`.
- health / bulk-health `stats.lastHeard`.
- `LastHeardMap` → notify offline check. The separate relay input
(`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed.
- `roles.js getEffectiveHeardMs`: newest known timestamp, without
`last_relayed`. The side pane no longer says "Last relayed"; the Last
Relayed row stays (analytics) with an updated tooltip.
- New status **`uncertain`** (repeaters and rooms): nothing certain
within the silent window, but the shared prefix (`last_relayed`) still
in paths. Grey badge with explanation, an *Uncertain* filter on the
nodes list and the map, dimmed like stale. The offline mail for such a
node says only its shared prefix was still seen.
- A prefix also held by a path-capable node in `inactive_nodes` is not
unique; adverts with `signatureValid: false` are no evidence;
`/api/nodes/search` reports the same `last_heard`.

Attribution for analytics (`byNode`, packet counts, `last_relayed`,
relay counters, usefulness) is unchanged.

## Performance

Hot path: `noteCertainRelays` runs per observation on ingest under the
write lock: two hex bytes of header parsing plus one prefix-map lookup
per resolved hop.

```
BenchmarkNoteCertainRelays-16   10307640   116.2 ns/op   0 B/op   0 allocs/op   (8-hop 2-byte path, 2,000 repeaters)
BenchmarkLastHeardMap-16            1902   0.77 ms/op                            (5,000 nodes; was 3.7 ms scanning byNode)
```

Memory: one string per node ever certainly heard, dropped together with
the node's `byNode` list on eviction.

## Tests

- `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk,
LoadChunked — each with and without resolved_path — live ingest, late
observations) feeds `certainHeard`; each of the 8 call sites verified
red with a nil prefix map.
- `certain_heard_test.go`: hop width per route/payload type; retired
namesake prefix; failed signature; uncertain offline mail; shared vs
unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load
path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`,
detail and health agree.
- Regression for #2260: last certain packet on day X, 20 shared-prefix
hops after it → `LastHeardMap` stays at X and `node.offline` goes bad
after 72 h. Verified red with the uniqueness check mutated out.
- Notify: `TestEvaluateCertainHeard` replaces the relay-input test.
- Frontend: `last_relayed` alone no longer keeps a repeater or room
active; newest timestamp wins (an old `_liveSeen` can't hide a fresh
certain relay).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork with 457 extra commits.

## Validation

Ran the server on the freshened e2e fixture: list, detail and health
report the same last heard; e.g. a repeater whose `last_relayed` is 3
min newer than anything certain keeps its certain time. **No browser
screenshot**: Playwright isn't installed on this machine; CI's E2E
covers the pages.

## Not changed

- The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a
shared hop only when exactly one candidate is adjacent in the neighbor
graph; it did not fire in this case. Left as is.
- The `lastHeard=` list filter still filters `last_seen` in SQL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 19:11:30 +02:00
efitenandClaude Opus 5.5 dc98886d92 feat(nodes): name and show repeaters known only from the name they reported (#2238) (#2239)
Part of #2238. Path hops through such a repeater (`/api/nodes`,
`/api/resolve-hops`, the frontend hop resolver) still show a bare
prefix; that part of the issue stays open.

## Summary

A repeater that never sends an advert can still tell a nearby companion
its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and
publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`.
This PR stores those answers, resolves names with them, and shows these
repeaters on the map, in the nodes list and in search, marked as
self-reported.

### Ingestor: store the names
- `367ecde1` adds a new `names` subtopic.
  - It is gated on `clientNodeNames.enabled`, default off.
- Answers go into a new table, `client_node_names`: one row per
observation, `UNIQUE(target, rx_pubkey, observed_at)`.
- **Validation.**
  - `target` must be 64 hex.
- The name is trimmed and rejected when it is empty, over 31 runes
(firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8,
or contains a control character.
- **Retention.** `retention.clientNodeNamesDays` (0 = keep).
`config.example.json` sets 30, like the other client tables: a companion
only asks a repeater for its name when CoreScope has none (`nameMissing`
in CoreDrive's `src/names.js`), so pruning is also how a renamed
repeater gets its new name.
- **Compatibility.** An ingestor without this change drops `/names`
silently, so CoreDrive v1.20.0 works against either version.

### Server: resolve by the reported name
- `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and
`client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys`
(the coverage tooltips) both use it.
- **Which name wins.** The advert name, otherwise the newest reported
name by `observed_at`. A prefix that matches different pubkeys across
the two tables stays ambiguous.
- **Only repeaters heard directly.** A reported name counts only once
its target has at least one `discover` or `advert` reception in
`client_receptions`, keyed by the full pubkey, or by its 8-byte prefix
when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names`
starts with that prefix (`clientNameHeard`). A name for a pubkey nobody
heard adds no candidate, so it cannot make a real node's prefix
ambiguous. Without the uniqueness check, a made-up pubkey that shares a
heard repeater's first 8 bytes borrowed that repeater's receptions and
passed (fixed in `4c569f91`).
- **Filters.** The blacklist and the hidden name prefixes apply to both
sources.
- **Missing table.** A database without the table resolves from `nodes`
alone, and the server checks for the table again until the ingestor has
created it.
- **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?,
name?, ambiguous}`.

### Server: estimated position
- `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`.
It lists every repeater in `client_node_names` that never sent an advert
(no row in `nodes` or `inactive_nodes`) and was heard directly at least
once (see above). A node with an advert but no position stays out: it is
already in `/api/nodes` under its advert name.
- **Gated** like the other endpoints on `client_receptions`: 404 when
`clientRxCoverage.enabled` is off, because the counts and the estimate
come from companion GPS fixes.
- **Which receptions count.** Only direct ones:
- `discover` and `advert` rows, keyed by the full pubkey, or by its
8-byte prefix when no other known pubkey shares it;
  - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone.

  `geo` rows are left out.
- **How the estimate is made.**
- Receptions within 100 m form one spot, with the best SNR of that spot.
  - An estimate needs at least 3 spots, at least 200 m apart.
- The position is the SNR-weighted centroid (`10^(snr/20)`). The radius
is the distance to the farthest spot, at least 500 m.
- The response is cached for 5 minutes, and the compute time is logged.

### Map
- `5fcde554` adds `public/map-estimated.js`, which draws each estimate
on its own layer:
  - a hollow, dashed "≈" marker and a dashed uncertainty circle;
- a popup with an "Estimated position" badge, the method, the caveat
that one-sided driving skews the estimate, and a link to the coverage
map;
- a Display toggle, "Estimated positions": on by default, kept in
localStorage, and also hidden by the repeater role filter and in route
view.

### Nodes page and search
- `12e6b345` adds a "Self-reported repeaters" section under the nodes
table, on the All and Repeaters tabs, filtered by the search box.
- **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a
self-reported detail view replaces "Node not found":
  - who reported the name;
  - the receptions;
  - the estimate, or why there is none;
  - links to the map and the coverage map.

  An unknown key still shows "Node not found".
- **Header search** adds matching self-reported repeaters, tagged
"self-reported".
- **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as
they are. These repeaters have no packets, so the packet filters, the
live view and the channel sender lookup must not match them.

### Docs
- `03580437` corrects the `/regions` section of
`docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the
switch, but the code uses `clientRegions.enabled`.
- The new subtopic, endpoint and config keys are documented in the same
file.

## Trust

Any companion the broker lets publish can report a name for any
repeater, and the newest report wins. This is the same model as
`/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to
linked companions.

A report for a repeater nobody heard directly is stored but not used: it
is not listed and adds no prefix candidate. A companion can still report
a wrong name for a repeater that was heard. The same companion can also
publish a fake `discover` reception for a made-up pubkey, so the "heard
directly" check filters junk, not a malicious companion;
`requireLinkedCompanion` is the control for that (its gate runs before
the `/names` handler too).

## Performance

- **`lookupPrefixes`** adds one term per key on `client_node_names`.
- It matches the prefix as a range, `target >= ? AND target < ?` (upper
bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive
column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX
idx_cnn_target (target>? AND target<?)`, and the reception check is
`SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`.
- It is still one query per chunk of 200 keys: 400 compound terms, under
SQLite's default limit of 500.
  - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks.
- **`/api/nodes/reported`** makes three kinds of query:
  - one grouped query on `client_node_names`;
  - one `lookupPrefixes` call for the short hashes;
- one `IN` query per 400 keys on `client_receptions`, served by
`idx_client_recept_heard_geo`.

The spread check is O(spots²) per repeater, with at most one spot per
100 m visited. The result is cached for 5 minutes; concurrent misses
wait for one computation, and the response is written after the lock is
released.
- **The map does not wait for it.** It draws its markers first and adds
the estimates when the list arrives. The map and the nodes page share
one cached fetch (`ReportedNodes.load()`).
- **Not benchmarked.** On the instance where this runs,
`client_node_names` holds one row today. I expect at most tens of such
repeaters per instance, but have not measured that.

## Notes

- **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2
occurrences: the handler's `msg` parameter and its `gps` field. That is
the decoded MQTT message, passed in the same shape as
`handleClientPacket` and `handleClientRegions` already take it. No API
response or internal structure uses it.
- **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min
TTL. They should move to the customizer later (rule 8).
- **Map filters.** The estimates follow the "Estimated positions" toggle
and the repeater role filter, not the area, region or last-heard
filters. The toggle's tooltip says so.
- **Name characters.** Names are rejected for control characters but not
for Unicode format characters (bidi overrides, zero-width joiners),
because emoji sequences in node names use the zero-width joiner. All
output goes through `escapeHtml`.
- **Not done:** the home-page search box (`home.js`, `/api/nodes/search`
with a Claim button) does not list these repeaters. Claiming a node
without adverts would have nothing to show.

## Tests

On this branch (based on `550829af`):
- `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and
`cmd/ingestor`.
- **New Go tests:**
  - handler and validation cases;
  - idempotent redelivery;
  - newest `observed_at` wins;
  - prune;
  - the gate on and off, without falling through to the observer path;
- client-only node, nameless `nodes` row, advert name wins, ambiguity
across the two sources;
  - hidden and blacklisted nodes;
  - absent-then-created table;
  - spot merging, threshold, weighted centroid and radius floor;
- `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an
advert position and a missing table;
- an advert without a position, a never-heard name, a name heard only in
`rxlog` paths, and the `clientRxCoverage` gate (404). These fail on
`12e6b345` and pass at head.
- a made-up pubkey sharing a heard repeater's 8-byte prefix, and two
reported targets sharing one (`4c569f91`). Both fail without that
commit.
- **New frontend unit suites:** `tests/unit/test-map-estimated.js` and
`tests/unit/test-reported-nodes.js` pass. Both load the real
`escapeHtml` from `app.js`.
- **`test-all.sh`** passes, all suites (with Python available for the
XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the
four changed frontend files.
- **E2E**, against a static server with a mocked API:
- `test-map-estimated-e2e.js`: 6/6, including the repeater role filter
and route view hiding the estimates. Mutating either hide makes its step
fail.
  - `test-reported-nodes-e2e.js`: 6/6.
  - `test-issue-1150-404-state-e2e.js`: 5/5.
- **Real data, API only.** On the fork's staging, `/api/nodes/reported`
lists one repeater: 5 spots, 17 receptions, an estimate with a radius of
2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays
listed under the stricter rules. Not re-checked after `4c569f91`.
- **Not done:** a check of that repeater in a browser.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 15:17:58 +02:00
n30nex 93f9fc37e0 perf(ingestor): yield between committed retention batches (#2153)
Fixes #2151. Related to #2146.

Yield to runnable goroutines after each committed full retention batch,
once the writer lock is released. This gives the serial ingest consumer
another opportunity to submit its next write. Batch size, cutoff and
deletion semantics stay unchanged; dense fanout within one transaction
remains unbounded.

The branch is rebased onto upstream
`b9593f71745b856651cfd6d6c80bb1f3f091e13e`. The duplicate test commit
`99c2722d` is dropped because #2160 is already present upstream as
`a5b60ca5`. The runtime and regression-test patch is unchanged from the
previously reviewed `b70a9f6b`.

The raw JSON profile is omitted from this branch. The [Markdown
report](https://github.com/n30nex/CoreScope/blob/307432370f8a7d9d623223beac1ba15fdd2d6546/docs/performance/sqlite-retention-b5-2026-10-09.md)
links to the [immutable original
profile](https://github.com/n30nex/CoreScope/blob/323b18d4221e83c9827b4722145e2a9d5b9d9599/docs/performance/sqlite-retention-b5-2026-10-09.json);
the linked profile was fetched and verified against its original Git
blob.

Performance scope:

- The historical five-pair Linux comparison remains pinned to `9dbc287`
versus `b70a9f6`, with 2,048,000 observations, 50 ingest events/s and 20
HTTP requests/s.
- Additional-observation scheduled-to-complete p95/p99 improved by
median paired ratios of 0.9503/0.9191, while service-time p99 rose 29.1%
and cleanup took 0.5% longer. HTTP results were mixed, and multi-second
ingestion queues remained.
- Those figures do not measure this rebased candidate against current
master. The current benchmark run is an execution smoke check, not a new
performance comparison; the independent reviewer's results are not
pooled into the historical study.
- Added runtime work is one scheduler yield per full 250-transmission
batch, with constant extra state and no additional SQL scans or per-row
work. No config or customizer values change.

Local validation on Windows with Go 1.27.2, CGO enabled, and package
parallelism limited to one:

- Focused retention regressions passed.
- Full ingestor suite passed.
- Focused retention regressions under the race detector passed.
- Both retained benchmarks passed one iteration of every fixture
(`typical16`, `dense256`, `singleHot`).
- The full ingestor race suite passed in 264.31 seconds with
`GOMAXPROCS=4`. An initial run during competing test jobs reached its
600-second limit; after those jobs finished, the retry passed with the
same 600-second limit and all assertions intact.

Post-push Chromium validation passed 148 checks with three explicit
fixture-dependent skips. The server and frontend source are identical to
upstream `b9593f71`; this browser pass checks the existing application
journeys and does not measure retention performance. [Fresh
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/39433150610)
passed on `30743237`, including the full Go job, ingestion race
detector, all three browser shards and the two-architecture build.
2026-10-21 09:40:22 +02:00
a164a44e7f fix(docker): run the Go services as an unprivileged user (#2125)
Runs `corescope-server` and `corescope-ingestor` unprivileged, opt-in
with `RUN_AS_UID`. A code-execution bug in either one then no longer
lands as root next to the mounted data and `config.json`. `supervisord`
stays root, for mosquitto and caddy.

| `RUN_AS_UID` | Go services run as |
|---|---|
| unset (default) | root, as before. Upgrades change nothing |
| set | `RUN_AS_UID` / `RUN_AS_GID` |

With `RUN_AS_UID` set:
- Each start hands the data directory to that uid (`chown -h`), except
`.env` and anything mounted into it on its own. Failures are logged.
- `.env` is read only if it is root-owned and only root can write it.
`RUN_AS_UID` and `RUN_AS_GID` are never taken from it.
- `/app` stays root's. **Depends on #2137.**
- A uid with no account in the image gets one at start; `caddy` and
`mosquitto` are reused. A uid that can't be provided fails startup
instead of falling back to root.

The logic is in `docker/select-service-user.sh`, tested by
`docker/test-select-service-user.sh`, which needs no Docker or root and
runs in CI. See also "Running the Go services unprivileged" in
`docs/deployment.md`.

Verified in the built image (master + this + #2137) across 14 cases.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Mythos 5.1 <noreply@anthropic.com>
Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com>
2026-10-15 09:49:29 +02:00
CoderNemesisandCoderNemesis d9342bbebd (feat): UserManagement, Add Support for Postal Email Delivery Provider (#2154)
This PR extends email support to add another email delivery provider
called Postal for user management. It is a, open source, self hosted
email delivery and reputation management service.

It adds 2 new additional configuration settings to the
userManagement.mail object which are "postalBaseUrl" and "postalApiKey".
It also adds "postal" as a supported value for the "provider" setting.
All existing functionality including email delivery notification is
supported by Postal.

Included test cases to support the new functionality and all pipelines
pass.

---------

Co-authored-by: CoderNemesis <rfedor@comchan.net>
2026-10-11 11:28:20 +02:00
Alex BandClaude Opus 5.5 a4ee3a7328 feat(scope-audit): observer region filter for both views (#2171)
## Summary

Builds on the transport view from #2167 and #2169 (both merged); rebased
onto `master` (`352a0ccd`), so the diff is this PR's six commits only.

The Scope Audit was the one analysis page without the region selector.
This adds the shared region filter to it, for both the declared and the
transport view. `?region=SFO,SJC` means what it means on every other
endpoint: count only forwarding **heard by observers in those regions**,
not repeaters located there.

- **API** (`GET /api/scope-audit`, both modes):
- The region codes are resolved to their observers first (a handful of
rows), then the audit's hop scan gets `AND o.observer_idx IN (…)` on v3,
or `o.observer_id IN (…)` on v2. The 3.5M-row hop scan never joins
`observers`.
- `All` means no filter (shared `normalizeRegionCodes`). Codes with no
observer match nothing, instead of silently widening to every observer.
- Responses echo the normalised codes as `region`. The declared and
transport caches and their singleflight keys now include the region, so
each region set is cached separately with the existing TTLs.
- Declared-region verification (`regionEvidence`) is deliberately left
unfiltered: it corroborates what a repeater declares from its own
traffic, which doesn't depend on where a hop was heard.
- **Page:** the shared `RegionFilter` sits with the view and window
controls; a change reloads the view, and the selection persists like
every other page. The transport view's text box is relabelled **Scope**
(it already sends `?scope=` since the rename in #2167), so the page no
longer has two controls called Region.
- **Docs:** `docs/api-spec.md` (parameter, both response shapes) and the
OpenAPI parameters for the route (`mode`, `scope` and `region` were all
missing).

Partial fix for #2142 (its region filter for rollout tracking), together
with #2167 and #2169.

## Performance

The unfiltered path must not get slower: it's the same query with an
empty clause. `BenchmarkComputeScopeTransport` (from #2167, 1,200
repeaters), the PR it stacks on vs this branch, `-count 8`:

| | before | after | |
|---|---|---|---|
| sec/op | 220.3 ms | 218.0 ms | ~ (p=0.065) |
| B/op | 43.77 MiB | 43.70 MiB | ~ |
| allocs/op | 833.5k | 833.5k | ~ |

**Filtered requests, measured on staging** (a copy of a production
database): the first version drove `o.observer_idx IN (…)` from
`idx_observations_observer_idx`, reading every observation the region's
observers ever made. Fixed in `80414242` with a unary `+` (the same
treatment as #2166), so the scan starts from the `first_seen` window and
filters per row. Red test `67e063b1` asserts the query plan against a
fixture with the real indexes.

| scan, one region over 24h (13 observers) | plan | time | hop rows |
|---|---|---|---|
| `o.observer_idx IN (…)` | observer index | 13.44 s | 102,114 |
| `+o.observer_idx IN (…)` | `first_seen` window | **0.25 s** | 102,114
|
| unfiltered | `first_seen` window | 0.35 s | 225,983 |

Cold end-to-end requests on staging after the fix: 24h declared,
unfiltered 0.84 s · one region 0.48 s (was 16.7 s) · two regions 0.22 s
· transport, one region 0.42 s · transport, six regions 0.75 s · 7d, one
region 3.46 s. 24h transport, live data: 383 repeaters unfiltered, 204
heard by one country's observers.

## Tests

- Red `629e4f36` (Go): extends the audit fixture with an `observers`
table and `observations.observer_idx`. Covers the declared view ignoring
a scope heard only by another region's observer, the transport view
listing only repeaters heard in the region, `region=All`, per-region
caching, a region with no observers, and the echoed codes. Five fail on
assertions; `region=All` passes because the stub ignores the filter,
which is the behaviour it pins.
- Green `6f1b1e1c`: the implementation, plus a v2-schema case
(`observer_id`), so both branches of the clause are covered.
- Red `b4933d68` / green `08ecc172` (frontend): `apiPath` carries the
selection in both views and adds nothing when none is selected. The
region control on the page is a net-new UI surface.
- `go test` (cmd/server) passes. Frontend suites: 193 of 194. The one
failure, `test-issue-1956-release-routing.js`, needs CI's
`GITHUB_REF_NAME` and fails the same way on `master` locally.

## Screenshot

The same transport view with the shared region filter set to **SJC**:
only forwarding heard by San Jose observers counts, so 7 repeaters are
listed instead of 44, with their carried scopes counted from those
observers alone. Taken against `test-fixtures/e2e-fixture.db` with
fictional scopes.

![Scope Audit with the observer region filter on
SJC](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/scope-audit-region-filter.png)

## Browser validation

The branch's server was run locally with 14 configured regions (dropdown
layout), checked in headless Chromium at 1280, 900, 700 and 390 px
(touch):

| check | 1280 | 900 | 700 | 390 touch |
|---|---|---|---|---|
| ticking SJC re-requests with `&region=SJC` | yes | yes | yes | yes |
| menu fully on screen | yes | yes | yes | yes |
| page wider than viewport | no | no | no | no |
| page errors | none | none | none | none |

Found and fixed this way: the filter is the right-most control on
desktop, and the menu (which opens rightwards) ran 66 px off the page.
It now opens leftwards above 640 px. On a phone the controls wrap, the
filter starts a line, and the default applies.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **S3776 on `ScopeAuditForwardingHeardIn`:** complexity 55. The
function it generalises, `ScopeAuditForwarding`, is 54 on `master`; the
region clause adds one point.
- **S3776 on `computeScopeAudit`:** complexity 65, unchanged from
`master`.
- Both pre-date this PR. Splitting them is better as a refactor PR of
its own than mixed into a feature.
- The page code's findings were fixed in the transport-view PR it stacks
on.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:17:12 +02:00
Alex BandClaude Opus 5.5 352a0ccd19 feat(region-filter): configurable region quick picks (#2170)
## Summary

**Independent of every other open PR.** Choosing a pick saves the
selection, redraws the control and notifies the page's listeners itself,
exactly as a click in the control does, so it re-queries straight away
whether or not #2168 (`setSelected` notifies) is merged; with both in,
nothing fires twice.

Region quick picks are named groups of region (IATA) codes, offered as
one-tap choices inside the region filter, e.g. "Northern California" for
a dozen airport codes instead of ticking each box. One deployment has
been injecting these into the page through an nginx `sub_filter` and a
script that wraps `window.RegionFilter`; this makes them a configured
feature.

```jsonc
"regionQuickPicks": [
  { "name": "Northern California", "description": "Bay Area and Sacramento observers.", "regions": ["SJC", "SFO", "OAK", "SMF"] }
]
```

- **Server:** `GET /api/config/region-quick-picks` returns the groups
cleaned up: names and codes trimmed, codes upper-cased and de-duplicated
in their configured order, groups without a name or codes dropped, at
most 20 groups of 200 codes, always a list. `/api/config/regions` is
unchanged.
- **Region filter:** loads the groups with the regions and offers every
group with at least one observer today, inside the region control:
- dropdown layout: rows of the same kind as **All** (checkbox and bold
name), directly under All with a divider before the single regions; the
row is ticked while that pick is the selection, ticking it selects the
pick and closes the menu, unticking goes back to all;
- pill layout: the picks lead the same bar, set off from the single
regions by a thin divider.
- A tap selects the group's codes that have an observer and re-queries
straight away.
- The selector then reads the group's name ("South Bay ▾") instead of "2
Regions", and the button shows as pressed. Selecting the same codes by
hand does the same.
  - Tapping the active group again goes back to all regions.
- A failed fetch, or no configured groups, leaves the filter exactly as
before.
- **Mobile:** the rows take the menu's 44 px row height under `(pointer:
coarse)`; the pill-bar picks reuse `.region-pill` and its touch target.
Inside `.filter-bar` (Packets) the existing touch rule for inputs made
every region checkbox a 44 px square; the checkbox is now 20 px and the
44 px row is the tap target.
- **Semantics:** like every region filter, a group means "heard by
observers in these regions", not "located there"; `config.example.json`
says so.

No existing issue covers this.

## Screenshot

Nodes page with two picks configured (example config below), after
tapping **South Bay**: the pick and its codes (SJC, MRY) are selected,
and the list re-queried for them. Taken against
`test-fixtures/e2e-fixture.db`.

![Region quick picks on the Nodes
page](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/region-quick-picks.png)

## Config example

```json
"regionQuickPicks": [
  { "name": "South Bay", "description": "San Jose and Monterey observers", "regions": ["SJC", "MRY"] },
  { "name": "East and North Bay", "description": "Oakland and San Francisco observers", "regions": ["OAK", "SFO"] }
]
```

Each pick needs a `name` and at least one code; `description` is
optional (shown as the tooltip). Codes are trimmed, upper-cased and
de-duplicated; a pick with no code that has an observer today is not
shown. At most 20 picks of 200 codes.

## Browser validation

The branch's server was run locally against a migrated copy of
`test-fixtures/e2e-fixture.db` (observers in MRY, OAK, SFO, SJC) with
three groups configured: "South Bay" (SJC, MRY, SCZ), "East and North
Bay" (OAK, SFO, sts) and "Nowhere yet" (XXX). Checked in headless
Chromium:

| check | desktop 1280 px | phone 390 px (touch) |
|---|---|---|
| groups shown | South Bay, East and North Bay ("Nowhere yet" hidden) |
same |
| tap South Bay selects | SJC, MRY (SCZ has no observer) | same |
| next `/api/packets` request | `region=SJC,MRY` | `region=SJC,MRY` |
| selector label | "South Bay ▾" | "South Bay ▾" |
| second tap | back to all regions | same |
| button height | 24 px | 44 px |
| page wider than viewport | no | no |
| page errors | none | none |

Checked on Packets (dropdown layout) and Nodes (pill layout). Three
layout problems found this way on the phone were fixed before opening: a
sideways-scrolling row clipped at the screen edge, the selector squeezed
into a narrow column (the filter group is a non-wrapping flex row), and
the two wrapped rows spread apart on Nodes (`align-content`).

**Revision 1:** the first version put the picks in a separate row beside
the selector; they moved inside the dropdown. Re-checked with a
14-region list and three picks on Packets: desktop 1280 px dark and
phone 390 px light both keep the menu on screen (menu 340 px wide, page
not wider than the viewport), the pick closes the menu and the trigger
names the pick.

**Revision 2:** picks are checkbox rows like **All** instead of chips.
Re-checked on Packets at 1280 px dark and 390 px light: rows render bold
under All, the divider separates the single regions, ticking a pick
ticks its codes and names the trigger, no sideways scroll.

Unrelated and not changed: on Nodes at 390 px the region row (existing
pills included) starts flush with the screen edge.

## Tests

- Red commit `81ea67d6`: the config type, a stub normaliser and endpoint
returning nothing, and tests.
- Go: cleaning (trim, upper-case, de-duplicate, drop incomplete groups),
caps, nil config, and the endpoint's JSON.
- Frontend, `tests/e2e/test-region-quick-picks.js` (moved from
`tests/unit/` in `fced69a7`; it needs jsdom, so it runs in the E2E job
like `test-table-sort.js`): the real `region-filter.js` in **jsdom**
against faked config endpoints. Nine cases: rendering and hiding groups
with no observer, selecting and re-querying, the selector label,
toggling back, a hand-made selection marked active, the pill layout, no
groups, a failed fetch, HTML escaping. Seven fail on the stub, all on
assertions.
- Green commit `0e33210e`: the implementation, styles,
`config.example.json` (`regionQuickPicks` +
`_comment_regionQuickPicks`), `docs/api-spec.md`, and the OpenAPI route
list (`TestOpenAPICompleteness` requires it). The jsdom test also had a
cross-realm comparison bug fixed in this commit (`Array.from` on arrays
coming out of the jsdom window).
- `make test`: 16 of 16 modules pass. Frontend suites: 195 of 196; the
one failure, `test-issue-1956-release-routing.js`, needs CI's
`GITHUB_REF_NAME` and fails the same way on `master` locally.

- Red commit `f991060d` (revision): picks live inside the dropdown menu
before the checkboxes, choosing one closes the menu, and in the pill
layout they share the region bar; two fail on assertions against the old
layout. Green commit `1f720e35`. Frontend 195 of 196 (same
environment-only failure).

- Red `134f5256` / green `da0adb50` (second revision): picks are
checkbox rows between All and the single regions with a divider, no
chips in the menu; ticking and unticking a pick row. The green commit
also repairs `style.css`: `1f720e35` had prepended its block to the top
of the file (its edit matched an earlier `.region-pill {`) and left the
previous rules in place. The file now differs from `master` by 13 lines.

## Review notes

- Groups are offered only when at least one of their codes appears in
`/api/config/regions`, and only those codes are selected. The stored
selection is pruned to known codes on load anyway, so selecting absent
codes would not survive a reload.
- Customizer (AGENTS.md rule 8): groups are operator configuration; a
customizer editor could come later.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `b84ace4e`:
- `normalizeQuickPickCodes` split out of the config normaliser (S3776,
17 > 15);
  - a comment on why a failed quick-pick fetch is ignored (S2486);
  - `for…of` in `activePick` (S4138);
- in the tests: `node:` specifiers, `includes()`, and an explicit string
comparator for sorted selections (S7772, S7765, S2871).
- **Kept on purpose:**
- S2486 is still reported on that `catch`. Swallowing the error is the
intended behaviour: the filter works as before, just without picks.
  - S1523 on the `vm` test loader, as above.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:23:05 +02:00
Alex BandClaude Opus 5.5 696906315a feat(scope-audit): transport view of every repeater seen forwarding (#2142, API) (#2167)
## Summary

Backend half of #2142 (the page is the stacked PR #2169).

`GET /api/scope-audit` lists only repeaters that have answered a
declared-regions request: 17 of almost 200 nodes on the instance the
request came from. This adds a transport view, `GET
/api/scope-audit?mode=transport&window=…[&scope=X]`, with one row per
repeater **seen forwarding** in the window:

- `transported`: the named region scopes it was seen carrying, each with
packets and first/last seen (most packets first), plus unscoped,
unmatched and ambiguous-hop counts;
- `asked`, `declaredRegions`, `notObserved`, `configState`, `declaredAt`
where a declared answer exists; `asked: false` with null declared fields
where it does not ("not asked" is not "declares nothing");
- with `scope=X` (spelled `be`, `#be` or `BE`): `carriesScope` on every
row and `carrying` / `notCarrying` counts, which is the rollout report
the issue asks for.

`mode=declared` (or no `mode`) is unchanged. An unknown `mode` is a 400.

Partial fix for #2142: this PR provides the API; the page is #2169 and
the observer region filter #2171. The issue should only be considered
done once all three are in.

## How it follows the issue's notes

- **The scan does not change.** It is the same single
`ScopeAuditForwarding` pass, with every repeater and room in `nodes`
(plus any declared target) as a target instead of only the declared
ones. Same attribution rules: FLOOD-family route types only,
`minForwarderHopHexLen` (no 1-byte hops), and a hop whose prefix matches
more than one repeater is credited to none of them.
- **Identity lookup with 1,000+ pubkeys:** `scopeAuditNodeIdentities`
runs one `IN` query; exercised at 1,200 keys in the benchmark below.
- **Row count and response size:** measured below.
- **Cache:** the transport view is cached per window with the declared
view's TTLs (30 s; 5 min for 7d), under the same singleflight group with
a `transport|` key. The scope filter is applied to a copy of the cached
result, so one scan serves every scope. (`scope=`, not `region=`: across
the API `region=` is the observer IATA filter.)
- **Wording:** "transported" is documented as *seen carrying*, not
configured for; the declared side stays separate.

## Performance

`BenchmarkComputeScopeTransport`: 1,200 repeaters (#1975 quotes an
instance with 1,179), 30k flood transmissions in the last 24h on 4-hop
paths of 3-byte prefixes, a third region-scoped:

```
BenchmarkComputeScopeTransport-12   5   219253727 ns/op   425.4 response-KB   1200 rows
```

219 ms per compute, served from cache for 30 s (24h) or 5 min (7d); a
425 KB uncompressed response for 1,200 rows. This is new code with no
"before"; the declared view's numbers are untouched (same scan, same
cache).

### On real data (a staging copy of a production database and its live
feed)

| request | rows | result | time |
|---|---|---|---|
| declared view, 24h | 16 | (unchanged) | |
| `mode=transport`, 24h | **385** repeaters seen forwarding, 16 of them
with a declared answer | | 1.5 s cold |
| `mode=transport&scope=<scope A>`, 7d | 664 | **51 carrying it, 613
not** | 14.4 s cold |
| `mode=transport&scope=%23<SCOPE B>` (`#` and capitals), 7d, same
window now cached | 664 | 237 carrying, 427 not | 0.35 s |

The 7d cold compute is the same full-window hop scan the declared view
already pays for 7d (its own TTL comment measures 16.7 s on a
live-shaped database); like the declared view it is then cached for 5
minutes, and every scope filter is served from that one scan.
- Commit `ceaeb56b` renamed the filter from `region=` to `scope=` after
opening; the real-data figures above were taken before the rename and
are otherwise the same code.

## Tests

- Red commit `2a9b1286`: response types and a `mode=transport` stub that
returns no rows, plus tests. Four fail on the stub:
- repeaters that never answered are listed with `asked: false` and null
declared fields (and the answered one with its declared list and
`notObserved`);
- `scope=fr` / `%23fr` / `FR` marks carriers and non-carriers and counts
them; no scope means no `carriesScope` or counts;
  - an unknown mode is a 400;
- the transport and declared views are cached separately (requesting one
does not serve the other).
- Also added, passing on the stub by construction: a repeater with no
traffic in the window is not listed; the node blacklist applies.
- Green commit `60c669f4`: the implementation, `docs/api-spec.md` (new
`mode` and `scope` parameters, a Transport view section with the
response shape), and the benchmark.
- `make test`: 16 of 16 modules pass, including every existing
scope-audit test.

## Review notes

- Hidden-name and blacklist filtering use the same rules as the declared
view.
- `newTestStoreWithDB` now takes `testing.TB` so the benchmark can use
it (callers unchanged).
- No new `map[string]interface{}`; the response is a named struct.
- No config added.

## SonarLint

SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.

- **Fixed** in `060668b6`: S3776 (cognitive complexity 41 > 15) in the
new `computeScopeTransport`. It is split into target, visibility, row
and sort helpers; the response is the same and the transport tests are
unchanged.

## Checked against the repo rules

- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:22:20 +02:00
efitenandClaude Opus 5.5 0737f10a5d feat(notify): optional external node alerts feed and the node.external event (#2177) (#2178)
Closes #2177.

## What

Node notifications can mail alerts computed outside CoreScope. With
`userManagement.notifications.externalAlerts` configured, users can
choose a new event, `node.external`, which is off by default and only
offered when configured:

```json
"notifications": {
  "enabled": true,
  "externalAlerts": { "url": "http://coverage-job/alerts.json", "maxAgeHours": 48, "label": "Coverage" }
}
```

The feed:

```json
{ "generatedAt": "2026-10-11T01:35:12Z",
  "alerts": [ { "pubkey": "<64 hex>", "key": "SE", "text": "never heard to the south-east", "url": "https://example.org/coverage" } ] }
```

Mail lines: `<name>: Coverage: never heard to the south-east, <time>`
when an alert appears, `<name>: Coverage resolved (SE), <time>` when it
is gone from a fresh feed.

## How

- `cmd/server/notify_external.go`: feed types, `parseExternalFeed` (RFC
3339 `generatedAt`, age check, per-alert validation),
`fetchExternalFeed` (200 only, 1 MB cap, 10 s timeout).
- `notify_eval.go`: `evalExternal`. Subjects are `<pubkey>/<key>`; a
`<pubkey>/*` row is the per-node baseline, so the first fresh feed after
watching stores the current alerts without a mail (like `foreign.new`),
and later alerts mail at once. `External == nil` (not configured, nobody
chose it, or stale) leaves every `node.external` state as it is.
- `notify_service.go`: the feed is fetched only when configured and at
least one user chose the event. Each new reason the feed is unusable is
logged once, and so is the recovery. Mail text and link: the alert's
`url`, otherwise the node page.
- `notify_handlers.go`: `availableEvents` and the new `externalLabel`
include the event only when configured; PUT answers 400 for
`node.external` otherwise.
- `internal/users/notify.go`: event constant, `OptionalNodeNotifyEvents`
(canonical order: node, optional, admin), `RemoveWatch` also deletes the
node's `<pubkey>/...` rows.
- Account page label, OpenAPI descriptions, `config.example.json`
comment, `docs/user-guide/accounts.md` section.

Decisions on the three open questions in #2177: the generic name
`node.external`; the resolved mail names the key instead of repeating
the text, so there is **no `users.db` schema change** (a bump would also
make older binaries refuse the file); one feed.

## Invariants

- Read/write separation (#1283): the server only reads the feed over
HTTP. State rows go to `users.db` through `internal/users`, like the
other events.
- A dead, garbled or old feed never mails "resolved":
`TestEvaluateExternalStaleFeedChangesNothing` and
`TestNotifierMailsExternalAlertsAndSkipsAStaleFeed` (HTTP 502 and a 72 h
old feed in a row, then a fresh empty feed resolves).

## Performance

The evaluator adds one pass over the stored states (already read every
tick) plus, per user with the event on and per watched node, the node's
current alerts. Bounded by `maxWatchesPerUser` (50) times the alerts per
node. The feed is fetched once per tick (default 5 minutes) only when
someone chose the event, capped at 1 MB. No packet-store lock is taken
for it. Nothing changes for instances without `externalAlerts`.

## Tests

- `cmd/server`: `go test` green locally (Windows;
`TestSaveGeoFilterPreservesFileMode` skipped as it needs Unix mode
bits). New: `notify_external_test.go` (feed parsing and limits, HTTP
fetch 200/404/oversized, config resolution, evaluator
baseline/new/resolved/returning/stale/not chosen/notifications off,
notifier end to end with a fake feed, account routes).
- `internal/users`: green, 2 new tests (optional event and canonical
order, `RemoveWatch` deletes external subjects only for that node).
- Frontend: `test-all.sh` green, 1 new test in
`test-notifications-ui.js` (label from `externalLabel`, escaped).
- `scripts/check-xss-sinks.sh --file public/notifications.js`: clean.

## Not done

- No E2E test: the account page change is one label, covered by the unit
test.
- Not run against a live instance yet. The first feed in use is the
coverage job of on8ar.eu (`alerts.json`, 29 alerts at the moment).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:54:35 +02:00
efitenandClaude Opus 5.5 c2a590b741 feat(companions): linked-only ingest, account Companions and My coverage (#2176)
Part 3 of 3 for #2173 (companion linking): the optional linked-only
ingest filter, the account page and the "My coverage" toggle, plus the
docs. Builds on #2174 and #2175 (both merged); `upstream/master` is
merged in, so the diff shows only this part.

## Changes in this PR

**Ingestor: linked-only ingest** (`cmd/ingestor/linked_companions.go`)
- With `clientRxCoverage.requireLinkedCompanion` on (default off) and
user management enabled, every `meshcore/client/<pubkey>/...` message
whose pubkey is not in `companion_links` is dropped before any client
handler runs, and counted (`client_unlinked_dropped` in the stats file).
- The ingestor opens `users.db` read-only (`mode=ro`, raw SQL, no
`internal/users` import), like the approved-channels reader, so the
#1283 write invariant holds. `TestLinkedCompanionSetIsReadOnly` pins it.
- The set refreshes every 60 s. A miss re-reads at most once per 5 s, so
a companion linked a moment ago is accepted within seconds while a flood
of unknown pubkeys costs one read per 5 s.
- It passes everything until its first successful read, and when
`users.db` has no `companion_links` table yet (a server older than
this). After the first read it keeps the last set on a read error. It is
a filter, not a security boundary: anyone with the shared broker
credential can still publish under a linked pubkey.

**Frontend**
- Account page: device tokens show under Devices as "CoreDrive RX" with
their label; a new Companions section lists linked companions (name,
short key, linked since, last seen) with an Unlink button.
- Coverage page (`#/rx-coverage`): a "My coverage" toggle for logged-in
users (`mine=1`). Only the newest coverage reply is drawn, and a refused
`mine=1` turns the toggle off. Under "My coverage" the page does not ask
for the "nothing received" cells from #2172, and their legend entry is
hidden.

**Docs**: user guide (accounts: devices, companions),
`docs/client-rx-coverage.md` (linking, linked-only ingest, the CORS
exception), `config.example.json`, and the spec status.

## Performance

The filter runs on every client message. Benchmarks with 1000 linked
companions (12-core dev box, `go test -bench`, 3 runs each):

| Path | Time | Allocations |
|---|---|---|
| linked pubkey (lock-free map lookup) | 54 ns/op | 0 |
| linked pubkey, parallel | 11-14 ns/op | 0 |
| unlinked pubkey inside the 5 s miss gap | 85-95 ns/op | 0 |
| one `users.db` re-read | 0.38 ms | 3030 |

The unlinked benchmark asserts that no extra `users.db` read happens. A
re-read happens at most once per 5 s on a miss and once a minute from
the refresh loop.

## Verification

- `cmd/ingestor`: 15 new tests (filter off changes nothing, linked
passes, unlinked dropped before every handler, miss refresh accepts a
new link, the 5 s cap holds under a flood, unlink honoured after the
periodic refresh, missing table and missing file, fail-closed only after
the first read, read-only open, startup, stats counter) and 4
benchmarks. Full ingestor suite green.
- Frontend unit tests for the account page (device tokens, companions
list escaping, empty state, Unlink confirm and cancel, load errors) and
`tests/unit/test-rx-coverage-mine.js`.
- E2E `tests/e2e/test-user-management-e2e.js` gains a companion step
(device token under Devices, a companion linked with a real Ed25519
signature under Companions, Unlink). Run locally against the e2etest
build: 28/28. The coverage suites (`test-rx-coverage-gaps-e2e.js`,
`-noise-`, `-viewport-`) still pass.
- Frontend unit suite (`test-all.sh`) green.

## Not in this PR

- Narrowing the "nothing received" cells to the caller's companions
under "My coverage" (the track query takes one companion today).
- Per-user broker credentials or ACLs, which would turn the filter into
an access control (non-goal in the spec).

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 18:38:27 +02:00
efitenandClaude Opus 5.5 b2cda21a27 feat(server): companion linking routes, device tokens and bearer auth (#2175)
Part 2 of 3 for #2173 (companion linking): the server routes, bearer
auth and the link handshake. Builds on #2174 (merged); `upstream/master`
is merged in, so the diff shows only this part. Everything here is
behind `userManagement.enabled` and answers 404 with it off.

## Changes in this PR

**Device tokens and bearer auth**
- `POST /api/auth/device-token`: email + password returns a bearer
token, stored as a `kind = 'device'` session with a label. Same failure
answers and the same rate limit as login.
- Server-enforced scope for a bearer token: `/api/auth/me`,
`/api/auth/logout`, `/api/account/companions*`, `/api/account/settings`.
Everything else answers 403.
- A bearer header only counts when there is no session cookie, so an
auth proxy that adds its own `Authorization` header does not hide a
logged-in browser. Cookies never accept a device token.
- The sessions list on the account API carries `kind` and `label`;
logout with a bearer token revokes that device.

**Linking a companion**
- `POST /api/account/companions/challenge` `{pubkey}` returns
`{challenge, expiresAt, host}`: 32 random bytes, single use, 5 minutes,
bound to the caller and the pubkey.
- `POST /api/account/companions` `{pubkey, challenge, signature, name}`:
the signature is the companion's Ed25519 signature over
`corescope-link:<host>:<challenge>`
(`internal/sigvalidate.VerifyMessage`). The challenge is consumed in
every case.
- A companion linked to another account moves to the caller (the newest
proof wins). Both users get a `companion.transfer` audit row and the
previous owner a mail. That mail is a security notice: it is sent
whatever the node notification settings are.
- `GET /api/account/companions` (with `lastSeenAt` from
`client_receptions`) and `DELETE /api/account/companions/{pubkey}`.
- Linking does not touch the synced `meshcore-my-nodes`: a companion is
not a node to monitor.
- Admin user detail lists linked companions; the account data export
includes companions and device sessions.

**Coverage and config**
- `GET /api/rx-coverage?mine=1`: only coverage of the caller's linked
companions (401 when logged out, 403 for a device token). With `gaps=1`
(#2172) the reply carries no `gaps` member under `mine=1`, because the
track query takes one companion and cannot narrow to a set yet
(`TestRxCoverageMineOmitsGaps`).
- `clientRxCoverage.requireLinkedCompanion` (default off) is read and
reported in `/api/config/client`, which also announces
`userManagement.companionLinking`, so CoreDrive RX shows its login only
where linking exists. The ingestor side is part 3.
- CORS: for origins in `corsAllowedOrigins`, the device-token routes
alone also allow POST/PUT/DELETE and the `Authorization` and
`Content-Type` headers. No credentialed CORS. Documented in
`config.example.json`.
- OpenAPI and `docs/api-spec.md` cover the new routes.

## Verification

- 26 new tests in `cmd/server` and `internal/sigvalidate`, among them:
`TestCompanionLinkFlow`, `TestCompanionLinkRejections` (signature by
another key, malformed signature, challenge bound to another pubkey,
unknown challenge; reuse and expiry are covered by the store tests in
#2174), `TestCompanionTransfer`,
`TestCompanionTransferMailIgnoresNotificationSettings`,
`TestBearerScopedRoutes`, `TestBearerIgnoredWithCookie`,
`TestDeviceTokenSharesLoginRateLimit`, `TestCORSBearerRoutes`,
`TestCompanionRoutesAbsentWhenOff`, `TestRxCoverageMine`,
`TestRxCoverageMineOmitsGaps` (fails without the `mine == nil` guard),
`TestVerifyMessage`.
- `cmd/server`, `internal/sigvalidate` and `internal/users`: `go vet`
and `go test ./...` green, `gofmt` clean.
- The linking flow runs on analyzer.on8ar.eu since 9 October 2026 with
CoreDrive RX v1.19.0 (a real companion linked through the signed
challenge).

## Not in this PR

- The ingestor's linked-only filter, the account page (Devices,
Companions), the "My coverage" toggle and the user docs: part 3.
- Per-user broker credentials and per-user API keys (non-goals in the
spec).

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 18:20:49 +02:00
efitenandClaude Opus 5.5 9897d52cf6 feat(users): companion linking store (schema v6, device sessions, link challenges) (#2174)
Part 1 of 3 for #2173 (companion linking). This PR adds only the
`users.db` side: the schema and store methods the server (part 2) and
ingestor (part 3) build on. No route, no UI and no behaviour change for
a running server.

## Changes in this PR

- **Design spec** `docs/specs/2026-10-08-companion-linking-design.md`,
including the optional linked-only ingest.
- **Schema v6** (`internal/users/schema.go`):
- `sessions` gains `kind` (`'web'` or `'device'`), `label` and `scopes`
(`''` = full, for web sessions).
- `companion_links (pubkey PRIMARY KEY, user_id, name, linked_at)`,
cascading on user delete.
- `link_challenges`: single-use challenges bound to a user and a pubkey,
with an expiry.
- **Device sessions** (`sessions.go`): create a device session with a
label and scopes, list sessions with kind and label.
- **Challenges** (`challenges.go`): issue and consume a challenge
exactly once; consumed, expired, or bound to another user or pubkey all
fail.
- **Companion links** (`companions.go`): link, list, unlink. Linking a
pubkey that belongs to another user moves it and reports the previous
owner (transfer detection), so the server can audit and mail.
- **Janitor**: the existing janitor also prunes expired challenges
(`cmd/server/auth_service.go`, 3 lines).

## Upgrade note

The store migrates `users.db` to v6 on start. A binary that only knows
v5 refuses to open a v6 file (`users: database schema version 6 is newer
than this binary supports (5)`), so rolling back after this lands needs
the pre-upgrade `users.db` backup.

## Verification

- `internal/users`: new tests for the v6 migration from v5, device
sessions, challenges (single use, expiry, binding) and companion links
(transfer, cascade on user delete). `go vet` and `go test ./...` green.
- `cmd/server` full suite green against the v6 store.

## Not in this PR

- Routes, bearer auth and the link handshake: part 2.
- Ingestor filter, account page and "My coverage": part 3.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 18:08:51 +02:00
efitenandClaude Opus 5.5 9dbc287579 docs: release notes for v3.14.0 (#2143)
Release notes and the CHANGELOG entry for v3.14.0. The release workflow
reads `docs/release-notes/v3.14.0.md` from the tagged commit as the
release body, so this lands before the tag.

23 commits since v3.13.1: 13 fix, 8 feat, 1 ci, and this docs commit.
The headline is optional user accounts (#2128). "Read this before
upgrading" covers the new limits on unauthenticated endpoints, the
`traffic_share_score` drop (#2117), the region filter window (#2114) and
the config file mode (#2126).

After merge: tag this commit as `v3.14.0` once its `:edge` image is
built, then dispatch the CI/CD pipeline on the tag.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:19:04 +02:00
efitenandClaude Opus 5.5 2456762179 feat: account data export and users.db backup (#2141)
A small follow-up to parts A to E of #2128: users can download all data
the instance holds about them, and the server backs up `users.db`.

Stacked on #2140. Review the commits after that branch.

## The situation

- An instance with accounts holds personal data, but a user has no way
to get a copy (GDPR articles 15 and 20). Self-delete already exists.
- `users.db` is the only file the server writes, and nothing backs it
up: `GET /api/backup` snapshots the analyzer database only.

## What this PR adds

**Data export.** `GET /api/account/export` and a "Download my data"
button on the account page give one JSON file with:
- the profile, including activation and a pending email change;
- sessions (times and user agent);
- synced settings, own proposals, notification prefs and watches;
- audit rows where the user is actor or target, with other accounts as
an id only;
- the mail log.

Password hash, session and link tokens and the unsubscribe token are
left out: they are credentials. Each export is audited as `user.export`.

**users.db backup**
- Daily `VACUUM INTO` snapshots through `internal/users`, on by default
when user management is on. They go to `backups/` next to `users.db`,
the 7 newest are kept, the directory is 0700 and the files 0600.
Rotation only touches `users-<timestamp>.db` files and never the
snapshot it just wrote. Config: `userManagement.backup {enabled, dir,
keep}`.
- `GET /api/admin/users-backup`: an admin downloads a fresh snapshot to
keep off the server; audited as `user.backup`.
- `docs/user-guide/accounts.md` gains a restore procedure. It covers
what a restore brings back (accounts deleted after the snapshot, old
passwords, revoked sessions) and how to clean that up afterwards.

Spec:
[`docs/specs/2026-10-08-account-export-and-users-backup-design.md`](https://github.com/efiten/CoreScope/blob/feat/account-export-backup/docs/specs/2026-10-08-account-export-and-users-backup-design.md).

## Verification

- `internal/users` with `-race` and the `cmd/server` suite pass locally,
30 new Go tests. The file-mode tests skip on Windows.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E: 27 of 27 steps locally.
- On our production instance since 8 October 2026; the first snapshot
there was 143 kB.

## Not in this PR

- Importing an export into another instance.
- Off-site upload of the snapshots. The admin download covers that by
hand.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 22:16:49 +02:00
efitenandClaude Opus 5.5 de364a8bb6 feat: mail notifications for watched nodes (user management part E) (#2140)
Part E of #2128: a logged-in user watches nodes and gets one mail when a
watched node goes offline, comes back, or reports a low battery. Admins
can add instance events: a new foreign node, and an observer going
offline or back. This covers the per-user mail part of #775.

Stacked on #2139 (part D, which builds on #2138). Review the commits
after that branch.

## The situation

A sysop learns that a repeater went silent, or that its battery is
running down, only by opening CoreScope. #775 asks for notifications.
Accounts (part A) now give a verified address to mail and a mailer with
delivery status.

## What this PR adds

**Events.** They use the instance thresholds, so a mail does not
disagree with the node page:
- `node.offline`: silent for the role's `healthThresholds` window. Last
heard comes from the packet store, and repeaters and rooms count relayed
traffic (#1598).
- `node.battery`: advert telemetry below `batteryThresholds.lowMv`,
recovered at `lowMv + 100`.
- Admins only: `foreign.new` (once per node) and `observer.offline`.

**Store.** Schema v5 with `notification_prefs`, `notification_watches`
and `notification_state`.

**Server** (only with `userManagement.notifications.enabled`)
- A notifier next to the janitor, every `intervalMinutes` (5). The
evaluator is a pure function of watches, prefs, stored states and node
snapshots.
- No burst after a restart or a new watch: a subject's first evaluation
stores its state without mailing. The loop waits for the startup load.
While ingest is stale (newest packet older than 30 minutes) the offline
checks pause, and after recovery they wait one silent window.
- One mail per user per check. Limits: 20 per user and 100 per instance
per rolling 24 hours, 50 watches per user. A change over a limit is
recorded and never mailed later.
- Every mail has a one-click unsubscribe (`List-Unsubscribe` and
`List-Unsubscribe-Post`). The GET only redirects to a confirm page, so
link scanners cannot unsubscribe anyone. Node names are cleaned of
control and bidi characters before they go into a mail.
- Routes: `GET`/`PUT /api/account/notifications`, `PUT`/`DELETE
/api/account/notifications/watches/{pubkey}`, `POST
/api/account/notifications/watch-my-nodes` (copies the synced "my nodes"
list), `GET`/`POST /api/notifications/unsubscribe`. `mailer.Message`
gains `Headers`.

**Frontend.** A "Notify me" toggle on the node pages, a Notifications
section on the account page (`#/account?section=notifications`), an
unsubscribe page, and the notification mail count on the admin overview.

Spec:
[`docs/specs/2026-10-07-node-notifications-design.md`](https://github.com/efiten/CoreScope/blob/feat/notifications/docs/specs/2026-10-07-node-notifications-design.md).

## Performance

Each check reads watches, prefs and states from `users.db` (one query
each), the watched nodes from the analyzer DB in chunks of 500, and
last-heard times from the packet store.

| Measurement | Result |
|---|---|
| One check on our staging instance, 130,000 packets in memory, 1
watcher | 6.9 ms, of which 0.93 ms holding the store read lock |
| Evaluator benchmark, 100 users with 50 watches each, 2,000 nodes |
1.75 ms |

No work is added to ingest, broadcast or any request path.

## Verification

- `internal/users` and `internal/mailer` with `-race`; the `cmd/server`
suite plus `-race` on the notifier tests; 74 new Go tests. The same
Windows-only failure as noted in #2138 applies.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E: 26 of 26 steps locally. With notifications off
the new steps fail, so they do test the flag.
- On our staging and production instance since 7 October 2026.

## Not in this PR

- Other channels (Discord, Telegram, webhooks). Detection is separate
from delivery, so one can be added.
- The topology, RF and anomaly alerts of #775.
- If the whole server starts after a feed outage that already ended, the
grace window is unknown and a watcher can get one wrong offline mail.
The user guide says so.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 22:02:39 +02:00
efitenandClaude Opus 5.5 abbe6846ff feat: propose and approve hashtag channels (user management part D) (#2139)
Part D of #2128: logged-in users propose a hashtag channel, an admin
approves, rejects or later revokes it, and the ingestor decrypts
approved channels without a restart. It is the first consumer of a
generic proposal table, and answers the request in #2092.

Stacked on #2138 (part C). Review the commits after that branch; GitHub
shows C's commits here until it is merged.

## The situation

- An instance decrypts only the hashtag channels listed in
`hashChannels`. Users who want a city or club channel shown have to ask
the operator, who edits `config.json` and restarts the ingestor.
- #2092 asked for a suggest-and-approve flow; a downstream fork
(dborup/CoreScope#99) runs one with anonymous suggestions and a file
queue.

## What this PR adds

**Name rules (`internal/channel`).** `ValidateHashtagName`, mirrored in
the browser:
- at most 31 UTF-8 bytes including `#` (firmware
`ChannelDetails.name[32]`), case preserved, `Public` refused in any
case;
- control, bidi, separator and format characters refused except ZWJ,
plus invisible fillers (`Other_Default_Ignorable_Code_Point`, U+2800,
non-ASCII spaces).

Go and JS give the same verdict for every code point.

**Store.** Schema v4, one generic `proposals` table (kind, subject,
status pending/approved/rejected/revoked, proposer, reviewer, note). One
row per (kind, subject); limits are checked in the same transaction as
the change.

**Server** (only with `userManagement.channelProposals.enabled`)
- `POST /api/proposals`, `GET /api/account/proposals`, `GET
/api/admin/proposals`, `POST
/api/admin/proposals/{id}/approve|reject|revoke`.
- 400 for an invalid name. 409 for a duplicate, a rejected name, or a
name already in `hashChannels`/`channelKeys` ("this channel is already
decrypted on this instance"). 429 over a limit (`maxPending` 100,
`maxApproved` 128, `perUserPerDay` 5).
- `GET /api/channels` gains `approvedChannels`, so approved channels are
listed before they have traffic. With the feature off the response is
byte-identical.

**Ingestor.** Opens `users.db` with `mode=ro` every 60 seconds,
re-validates the names, and swaps its key map through an
`atomic.Pointer`. Configured channels always win, and a failed read
keeps the last good set. The ingestor never writes `users.db`; AGENTS.md
now says so.

**Frontend.** "Propose for everyone" in the add-channel dialog, an
"approved" marker in the channel list, "My proposals" on the account
page, and an admin "Proposals" tab with a warning on approve. The packet
detail pane now escapes the channel name: until now only the operator
chose those names.

Spec:
[`docs/specs/2026-10-07-channel-proposals-design.md`](https://github.com/efiten/CoreScope/blob/feat/channel-proposals/docs/specs/2026-10-07-channel-proposals-design.md).

## Performance

Every key is tried on each GRP_TXT that no key opens (`decodeGrpTxt`).
Benchmark of that worst case:

| Keys | Time per packet |
|---|---|
| 320 configured | 204 to 216 µs |
| 320 configured + 128 approved | 272 to 320 µs |

That is about +33%, linear in the key count and capped by `maxApproved`.
With the feature off the decoder gets the identical map and no ticker
runs.

## Verification

- `internal/channel`, `internal/users` (with `-race`), `cmd/server` and
`cmd/ingestor` pass locally, 53 new Go tests. The ingestor's
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced` needs symlink rights on
Windows and fails on `master` too.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E: 21 of 21 steps locally (propose, approve, the
channel appears, revoke removes it).
- On our staging and production instance since 7 October 2026.

## Not in this PR

- Private (PSK) channels: their keys stay in the browser (#725).
- Anonymous proposals.
- Names that need ZWNJ (some Persian and Urdu spellings) and subdivision
flags are refused, because ZWNJ and tag characters are format
characters. Allowing them later only widens the rule, so no approved
name gets stranded.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 21:51:42 +02:00
efitenandClaude Opus 5.5 18a634c115 feat: admin dashboard with overview and audit log (user management part C) (#2138)
Part C of #2128: an admin area with an overview, the user table and a
global audit log. Off by default like the rest of user management: with
`userManagement` off nothing changes, and non-admins get no new UI.

This is the first of 4 stacked PRs (C, D, E, then a small export/backup
follow-up). Each later one contains this branch; review them in order.

## The situation

- An admin could manage users one by one in `#/admin/users`, but nothing
showed whether the instance needs attention: accounts stuck in
activation, bouncing mail, someone guessing a password, an MQTT source
that dropped.
- The audit log existed (`internal/users/audit.go`) but could only be
read per user, and logins were not recorded.

## What this PR adds

**Store (`internal/users`)**
- Schema v3: an index on `audit_log(at)`.
- `AuditList` with filters (action or group prefix like `user.login.*`,
user as actor or target, period) and keyset pagination; `PruneAudit`;
`Stats` for the user figures.

**Server**
- `GET /api/admin/stats` (typed struct): accounts by status, admins,
registrations and active users over 7 and 30 days, logins and failed
logins in 24 hours, mail by final status, and the attention items
computed server-side.
- `GET /api/admin/audit`: filtered, newest first, `next` cursor.
- `GET /api/admin/users` gains `bouncing=1`.
- Logins are audited as `user.login` and `user.login.failed` (reason
`wrong_password`, `pending` or `disabled`). An unknown address writes no
row. The writes are asynchronous, so the login response does not wait on
`users.db`. Login rows are pruned after 90 days.

**Frontend**
- `#/admin?tab=overview|users|audit`: `admin.js` (tab shell),
`admin-overview.js` ("Needs attention", Users card, System card from the
existing health, MQTT and observer endpoints), `admin-audit.js` (filters
in the URL, "Load more"). `admin-users.js` becomes the Users tab; the
old `#/admin/users` link rewrites to it.
- `/api/healthz` is read on open and on Refresh only, not on the
60-second timer, because it walks every packet under a read lock.

Spec:
[`docs/specs/2026-10-07-admin-dashboard-design.md`](https://github.com/efiten/CoreScope/blob/feat/admin-dashboard/docs/specs/2026-10-07-admin-dashboard-design.md).

## Verification

- `internal/users` and `cmd/server`: `go vet` and `go test` pass
locally, 26 new Go tests. One upstream test,
`TestSaveGeoFilterPreservesFileMode`, also fails on Windows on plain
`master` (file modes) and is unrelated.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E with the `e2etest` build: 13 of 13 steps locally.
- Running on our staging and production instance since 7 October 2026.

## Not in this PR

- Restricting existing pages (perf, MQTT status) to logged-in users or
admins. The admin area adds no access mechanism of its own, so that
stays a later router rule plus endpoint check.
- Server-enforced customizer tab restrictions (#1508).
- The 24-hour, 5-attempt and 10-minute attention thresholds are
constants for now (AGENTS.md rule 8: customizer later).

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 21:38:30 +02:00
efitenandClaude Opus 5.5 f3ae8ac25a feat: sync a logged-in user's settings across devices (part B) (#2130)
Part B of #2128: a logged-in user's settings follow them across devices.
Log in on a phone and your own nodes, favorites, customizer and filters
are there; a change on one device reaches the others within a minute or
when you return to the tab.

**This PR builds on #2129.** Until that one is merged, the diff here
includes it. The commits for this part start at `docs(specs): settings
sync for optional user management (sub-project B)`.

## The situation

Everything a visitor sets up lives in one browser's `localStorage`
(about 100 keys in `public/`). A second device or a cleared cache starts
from zero (#895).

## What this PR adds

**Storage.** `users.db` schema v2: one JSON document per user in
`user_settings`, with a revision number and a generation id. A write
succeeds only when the client's revision and generation match the stored
ones, so two devices cannot overwrite each other silently.

**Server.** `GET`, `PUT` and `DELETE /api/account/settings`, behind the
same session and CSRF checks as the account routes.
- The server owns the list of synced keys (61 keys,
[`settings_allowlist.go`](https://github.com/efiten/CoreScope/blob/feat/settings-sync/cmd/server/settings_allowlist.go))
and sends it to the client, so the two cannot drift.
- A hard denylist, checked first, refuses `meshcore-api-key`, every
`corescope_channel_*` key and `live-channel-colors` (#725). The colour
map is keyed by channel hash, and for a user-added channel that hash is
`user:<name>`, which would expose hashtag channel names.
- Documents are capped at 256 KiB, measured like `JSON.stringify`. PUT
is limited to 60 requests per hour per user. A stale revision gets 409
with the current document.

**Client**
([`settings-sync.js`](https://github.com/efiten/CoreScope/blob/feat/settings-sync/public/settings-sync.js)).
Inert unless the feature is on and someone is logged in.
- It wraps `localStorage.setItem` and `removeItem` for allowlisted keys
only and pushes 2 seconds after the last change.
- It pulls on login, page load, tab focus and every 60 seconds while the
tab is visible.
- **Merge:** three-way, against a per-device baseline that belongs to
one user and one document generation. Lists (own nodes, favorites, saved
filters) merge per item, so an item added anywhere is kept and an item
removed on one device does not come back from another. Single values:
the profile wins unless only this device changed it.
- Remote changes are written without a push, theme and colour-blind
preset are re-applied, and the current page re-renders (skipped on
account pages and while the geofilter editor is open).

**UI.**
- Logout asks: keep my settings on this device (default), remove them
from this device, or cancel. Channel keys are never removed: no copy
exists anywhere else.
- The account page gets a "Settings sync" section: last synced time,
"Sync now", what is and is not synced, and "Delete synced settings from
my account".

## Not synced

Layout and device state (panel and column widths, collapsed panels, map
positions, geofilter drafts), channel data (#725), the API key, and all
`sessionStorage`. The full list is in the
[spec](https://github.com/efiten/CoreScope/blob/feat/settings-sync/docs/specs/2026-10-06-user-settings-sync-design.md).

## Performance

- One GET per page load, tab focus and minute while visible; one
debounced PUT per burst of changes.
- The `setItem` wrapper costs one Set lookup per write for non-synced
keys. A synced write reads one small revision key, not the stored
document.
- The server reads or writes one row per request.

## Verification

- `internal/users` and `cmd/server`: `go vet` and `go test` pass locally
(22 new Go tests), including a test that every allowlisted key still
occurs in `public/`, and denylist tests.
- `tests/unit/test-settings-sync.js`: 79 passing (vm, real module). The
cases cover the merge table, two tabs sharing one storage, stale answers
after a push, delete while a push is in flight, and logout while the
final push fails.
- `sh test-all.sh` exits 0.
- `tests/e2e/test-user-management-e2e.js` (10 steps, 4 of them new)
passed locally with two browser contexts as two devices: a favorite and
the packet time window travel from device 1 to device 2, a removal does
not come back, and "remove from this device" clears the synced keys
while a channel key stays.
- Checked by hand on a staging instance with a desktop and a phone on
one account.

## Not in this PR

- On a shared browser where the previous user chose "keep", the next
user's first login merges those settings into their own account. The
user guide says to choose "remove" on shared computers.
- Saved filter expressions are synced as typed, including any channel
names written in them. The guide says so.
- Realtime push between devices; the minute pull is the sync interval.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 11:20:44 +02:00
efitenandClaude Opus 5.5 d232072f85 feat: optional user accounts (part A: foundation) (#2129)
Part A of #2128: optional, off-by-default user accounts. With the
feature off nothing changes; with it on, visitors can register and log
in, and admins manage users and use the operator actions without the API
key.

PR #2130 (settings sync) builds on this one. The two are meant to be
merged together.

## The situation

- Operator actions (geofilter save and prune, backup, perf reset) need
the shared `apiKey`. There is no per-person right.
- Nothing in CoreScope knows who a visitor is, so the requests in #2128
that need that (#1835, #2092, #1508, #730) have nothing to build on.

## What this PR adds

**Two new Go modules**
- `internal/users`: a separate `users.db` (SQLite through
`modernc.org/sqlite`) with users, sessions, single-use tokens, an audit
log and a mail log. Passwords use argon2id.
- `internal/mailer`: a `Mailer` interface with a Brevo client (send,
delivery events, webhook parsing) and an in-memory fake for tests.

**Server (`cmd/server`)**, active only with `userManagement.enabled`
- 24 routes, all documented in OpenAPI under the `users` tag
([`auth_routes.go`](https://github.com/efiten/CoreScope/blob/feat/user-management/cmd/server/auth_routes.go)):
  - auth: register, activate, login, logout, me, forgot, reset;
- account: profile, password, email change with confirmation, sessions,
self-delete;
- admin: list, detail, disable, enable, delete, role, resend activation,
manual activation, mail status refresh;
  - a Brevo webhook, registered only when `mail.webhookSecret` is set.
- `requireAdmin` replaces `requireAPIKey` at the 7 operator call sites:
the API key **or** an admin session. With the feature off it is the old
API-key gate (`TestRequireAdminWithoutUserManagementIsAPIKeyGate`).
- `/api/config/client` gets `userManagement: {enabled: true}` only when
the service started; with the feature off the response is
byte-identical.

**Frontend**
- `auth.js` (header account control, request helper that adds the CSRF
header), `account.js` (login, register, activate, forgot, reset, confirm
email, my account), `admin-users.js` (`#/admin/users`, deep-linked
filters), `account.css` (theme tokens only).
- On phones the top-bar control is hidden, so a conditional entry goes
into the bottom-nav "More" sheet and the nav drawer.
- The customizer geofilter tab and the Perf "Reset stats" button use the
admin session when there is one.

**Config.** A `userManagement` block (`config.example.json`,
[`docs/user-guide/accounts.md`](https://github.com/efiten/CoreScope/blob/feat/user-management/docs/user-guide/accounts.md)).
The Brevo key can come from `CORESCOPE_BREVO_API_KEY`. The server
refuses to start when the block is enabled but incomplete.

## Security choices

- Session cookie `cs_session`: HttpOnly, SameSite=Lax, Secure when
`publicBaseUrl` is https. Every cookie-authenticated state change needs
the `X-CS-CSRF` header and a matching Origin.
- Activation needs the token **and** the account password. Without the
password, an attacker who keeps re-registering a known address could get
the owner to activate an account that carries the attacker's password.
- Register, forgot and email change answer identically for known and
unknown addresses. A password reset ends all sessions, a password change
ends all other sessions, and both end outstanding email-change links.
- Rate limits: login 10 per 15 minutes, register and forgot 5 per hour,
per IP and per address. The bucket count is capped. `trustedProxies`
makes the per-IP limits see real client IPs behind a proxy.
- Server logs carry `#<user id>`, never addresses, tokens or passwords;
mail-provider error texts are redacted before logging.

## Performance

No change to an existing hot path with the feature off. With it on:
- One `users.db` lookup per authenticated request (session by token
hash).
- The admin user table rebuilds its `tbody` on each filter change.
`users.List` caps the result at 1000 rows (`internal/users/users.go`),
which bounds the rebuild.
- `map[string]interface{}` in `openapi.go`: 79 before, 78 after.

## Verification

- `internal/users`, `internal/mailer` and `cmd/server`: `go vet` and `go
test -race` pass locally. 121 new Go tests.
- `cmd/server` with `-tags e2etest`: vet and the e2e hook tests pass.
- `sh test-all.sh` exits 0. `tests/unit/test-user-management-ui.js`: 67
passing (vm, real modules).
- `tests/e2e/test-user-management-e2e.js` (6 steps) passed locally
against an `e2etest` build with the fake mailer and against a
feature-off build. CI builds the `e2etest` binary and runs the suite on
a second server (`deploy.yml`).
- On a staging instance with a real Brevo key: register, activation mail
delivered, activate, admin table, "Refresh status" showing sent,
deferred, delivered, opened and clicked.

## Not in this PR

- Settings sync (#2130), the admin dashboard, approval flows and
notifications (parts B to E of #2128).
- A `requireReadAuth` mode (#1835). Sessions from this PR are what such
a mode would accept.
- Binary size and build time with `modernc.org/sqlite` linked next to
`mattn/go-sqlite3` were not measured. Their driver names do not collide.
#1992 discusses the driver choice.
- No Brevo webhook was configured on staging; delivery status there came
from "Refresh status".

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 09:25:29 +02:00
efitenandClaude Opus 5.5 000d9ab030 feat(coverage): RF noise-floor layer on the Mobile RX coverage page (#2113)
The ingestor already stores the noise floor that CoreDrive RX companions
report with each GPS fix (`client_rf_samples`, opt-in through
`clientRfSamples`, `cmd/ingestor/client_rf_sample.go:62`), but nothing
reads it back. An operator who enables it collects the data and cannot
see it. This adds the read side: `GET /api/rf-noise` and a Signal/Noise
toggle on the Mobile RX coverage page.

## What it does

- **`GET /api/rf-noise?bbox=&z=&days=`** returns a GeoJSON hex grid with
the median, quietest and noisiest noise floor per cell, in the same
shape as `/api/rx-coverage`. It is registered always and 404s unless
`clientRfSamples.enabled` is true, the same pattern as the coverage
routes.
- **Stationary samples are excluded.** A parked companion logs hundreds
of readings at one point, which would otherwise define its cell.
- **Coverage page:** a Signal/Noise toggle, rendered only when
`/api/config/client` reports `clientRfSamples: true`. The noise layer
reuses the coverage colour tokens with the axis inverted, because a
lower dBm is quieter. Tiers are at -115 and -108 dBm.
- **Deep link:** `#/rx-coverage?layer=noise` opens on the noise layer.
- **Empty and failed answers** are labelled on the map ("No RF samples
in this view yet", or a retry hint), so a blank map never reads as
"feature off".

No new configuration key: it reads the existing `clientRfSamples`
section that the ingestor already uses. Default off, so nothing changes
for an instance that has not opted in.

## Where it comes from

Ported from the efiten/CoreScope fork, where it has run on
analyzer.on8ar.eu since September (fork commits `42d09f0e`, `80581c43`,
`cde95078`). The cherry-picks conflicted with upstream's newer
`routes.go`, `types.go` and `rx-coverage.js`, so the final state was
ported by hand. The fork-only `/scopes` route that sat next to it in the
same hunk is deliberately left out.

## Performance

The query is bounded by `sampled_at` (indexed, `idx_crf_prune`) and the
bbox, aggregation is one pass plus a per-cell sort, and the response is
capped at 5000 cells. Measured on analyzer.on8ar.eu, all of Belgium
(`bbox=49.4,2.4,51.6,6.5&z=9`), from a client in Belgium, so network
time included:

| window | samples | cells | response time |
|---|---|---|---|
| 7 days | 10,835 | 241 | 0.37 s |
| 30 days | 43,303 | 429 | 0.39 s |

That table holds 45,270 rows in total. It is only read when someone
opens the noise layer, never on ingest or WebSocket paths.

## Tests

- `cmd/server/rf_noise_test.go`: 8 tests for the aggregation (median and
extremes, stationary exclusion, the cell cap, empty input) and the gate
(404 when off, even with data present). All pass, and the full
`cmd/server` suite passes locally.
- `tests/unit/test-rx-coverage-noise.js` (new, in `test-all.sh`): the
colour axis runs the right way, including both tier bounds and a string
median from the API. Slices the real function out of `rx-coverage.js`
and fails on master's copy, which has no thresholds.
- `tests/e2e/test-rx-coverage-noise-e2e.js` (new, wired in `deploy.yml`
and `scripts/non-unit-tests.json`): no toggle when the flag is off;
Noise fetches `/api/rf-noise`, draws the cells, swaps legend and
subtitle, puts `layer=noise` in the hash; a deep link opens on the noise
layer and an empty answer shows the message. 3/3 locally; against
master's `rx-coverage.js` and `roles.js`, 1/3 (only the "flag off" case
passes, as it should).
- `test-rx-coverage-viewport-e2e.js` still passes. ESLint 8 reports 0
errors on the changed files.

## Not done

- The thresholds (-115 / -108 dBm) are fitted to the fork's own data
(1,241 moving samples at the time) and are constants in
`rx-coverage.js`. Per AGENTS.md rule 8 they belong in the customizer
eventually; not in this PR.
- The E2E suite mocks the API. The real endpoint is covered by the Go
tests and by the measurements above, not by a fixture with seeded
samples.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:11:34 +02:00
efitenandClaude Opus 5.5 9c3d76e14f docs(release): notes and changelog for v3.13.1 (#2112)
Release notes and CHANGELOG entry for v3.13.1, a patch release that
ships #2111 (node-discover replies count as coverage).

Once merged, the tag goes on the merge commit so `deploy.yml` picks up
`docs/release-notes/v3.13.1.md` as the release body.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 17:22:56 +02:00