599 Commits
Author SHA1 Message Date
AdamandAdam c83499763d Allow mobile clients to manage repeater claims and details (#2429)
Draft for feedback: a working version of native repeater claims and
edits from mobile clients, built on the repeater info feature from
#2415. Four open questions are at the end.

## Situation

Repeater info (#2415) can only be claimed and edited from a browser
session. Device tokens from `POST /api/auth/device-token` are limited to
`rx` (CoreDrive RX: settings and companion linking), and `GET
/api/repeaters/{pubkey}/info` treats a bearer token as anonymous. A
mobile client can show a repeater's details but can't offer "claim" or
"edit" natively.

`PUT /api/repeaters/{pubkey}/info` is also a full replace with no
precondition. Two people editing from the same copy silently overwrite
each other. That's rare in the browser, but likelier from an app that
keeps a cached copy.

## Change

Three commits, one per piece.

**1. Device-token scopes for the repeater routes** (`8aba3715`)

- `POST /api/auth/device-token` takes an optional `scopes` list. Omitted
(or `null`) still means `["rx"]`, so CoreDrive RX and every existing
token keep exactly their current permissions. An empty list or an
unknown scope answers 400, checked before the password.
- New scopes:
- `repeater:claim`: `POST` and `DELETE /api/repeaters/{pubkey}/claim`.
These already only add or remove the caller's own claim. Several
claimers per repeater, idempotent `POST`, unchanged.
- `repeater:edit`: `PUT /api/repeaters/{pubkey}/info`, still subject to
`CanEditRepeater`.
- Every device token may use `/api/auth/me` and `/api/auth/logout`, so a
repeater-only token can check and end its own login.
- `GET /api/repeaters/{pubkey}/info` with a token holding either
repeater scope returns `canEdit` and `claimedByMe`. An unknown or
expired token answers 401 rather than passing as anonymous, so a client
learns its login is gone. A valid token without a repeater scope still
gets the public view. A cookie still wins over a bearer header, as
elsewhere.
- `GET /api/account/repeaters` (My repeaters, from #2430) also takes a
token holding either repeater scope, so an app gets the account's claims
in one request. It can't build that list from the public catalog, where
claimers are display names, and reading repeaters one at a time would
miss a claimed repeater that has left the node list. An `rx`-only token
still answers 403.
- Scope checks are method-aware (`bearerScopesFor(method, path)`). The
repeater paths are added to the bearer CORS paths.
- Same handlers, validation, rate limit (`repeater`, 120/h per user and
per IP), audit actions and admin claim removal as the browser. Token
writes add `via: device` to the `repeater.claim`, `repeater.edit` and
`repeater.unclaim` rows, and the `user.login` row records the granted
scopes. The admin route keeps refusing bearer tokens.

**2. Conditional edits with a revision** (`9f9bf47b`)

- `users.db` schema v10 adds `repeater_info.revision`. Existing rows
start at 1; a repeater with no saved details is at 0. Claims don't
change it.
- `GET` (single and list) and `PUT` responses include `revision`.
- `PUT` accepts an optional `baseRevision`. The check is part of the
write statement, so of two saves from the same base exactly one lands:
- `baseRevision: 0` (first save): `INSERT … ON CONFLICT(pubkey) DO
NOTHING`.
- `baseRevision: n`: `UPDATE … SET revision = revision + 1 … WHERE
pubkey = ? AND revision = ?`.
- No match answers `409` with the stored details
(`canEdit`/`claimedByMe` included) and writes nothing. That's the same
shape `PUT /api/account/settings` uses for its conflicts.
  - A stale base is refused even when the body would change nothing.
- Permission is checked first, so a 409 never shows details to someone
who can't edit.
- Without `baseRevision` the save is unconditional and bumps the
revision, so `public/repeater-info.js` works unchanged.
- `/api/config/client` adds `userManagement.repeaterTokens: true`, so a
client can tell this server from one without the scopes or
`baseRevision` (older servers answer the device-token request with the
SPA page, as noted for `companionLinking`).

**Discovery and conflict recovery, for a client:**

1. Check `userManagement.repeaterTokens` in `/api/config/client`.
Without it, keep the claim and edit actions hidden.
2. Log in with `scopes: ["repeater:claim", "repeater:edit"]` (or just
what the client offers). The response echoes the granted `scopes`.
3. Read `GET /api/repeaters/{pubkey}/info` with the token. Show
Claim/Unclaim from `claimedByMe` and Edit from `canEdit`, and keep
`revision`.
4. Save with `PUT …/info` and `baseRevision` set to that revision. On
200, keep the new `revision` from the response. On 409, the body is the
current record: show or merge it, then retry with its `revision`. On
401, log in again. On 403, the account can no longer edit (for example
someone claimed it), so refresh from `canEdit`.

**3. Account page device list** (`d28374aa`)

My account, Devices labelled every device token "CoreDrive RX".
`/api/account/sessions` now includes a device token's `scopes`. The page
keeps the CoreDrive RX label for `rx` tokens (and rows without scopes).
It shows other tokens under the device name the app sent, with a line
such as "can claim repeaters and edit repeater details".

Docs: `openapi.go`, `docs/api-spec.md` and the Repeater info section of
`docs/user-guide/accounts.md`.

## Questions

1. **One scope or two?** This uses separate `repeater:claim` and
`repeater:edit`, so a client can ask for less, and a token that only
edits can't put the user's name on a repeater publicly. One `repeater`
scope would be simpler if you'd rather not split it.
2. **Editing unclaimed repeaters from mobile.** This keeps web parity:
any active user may edit a repeater nobody has claimed, and once it has
claimers only they (and admins) can. The alternative is to require a
claim before a token may edit. That's stricter, but then app and browser
rules differ.
3. **Operator switch.** There's no new setting: the scopes are available
whenever `userManagement` is on, and the user opts in when the app asks
for them at login. Would you want an explicit opt-in such as
`userManagement.mobileRepeaterWrites`?
4. **Stale-edit contract.** Does `revision` plus `baseRevision` in the
body, with 409 returning the current record, fit? I matched account
settings rather than `ETag`/`If-Match`, to keep one pattern in the API.
The browser editor doesn't send `baseRevision` yet. Making it send one
would be a small follow-up, if you want browser saves protected too.

## Tests

- `cmd/server/repeater_device_token_test.go`:
- default and requested scopes, rejected scopes, and the method-aware
route table;
- per-scope allowed and denied routes, logout with a repeater-only
token;
  - claim and unclaim of own claim only, alongside another claimer;
- edit permission (unclaimed repeater, claimer, non-claimer, admin) and
a disabled account;
  - the admin route refusing tokens;
  - authenticated, public and expired-token reads;
- `via: device` audit rows, cookie writes still needing CSRF, and
session scopes.
- My repeaters with a claim or edit token (own claims only), and 403 for
an `rx`-only token.
- `cmd/server/repeater_revision_test.go`:
- revision on reads, a first edit with no saved row, and a stale base
returning 409 with the current record and no audit row;
  - unconditional browser saves, validation and permission order;
- two concurrent saves from the same revision (one 200, one 409), and
the client config flag.
- `internal/users/repeater_revision_test.go`:
  - revision counting and conditional first and later saves;
  - an 8-way race for both cases (`-race`);
  - the v9 to v10 upgrade.
- `tests/unit/test-user-management-ui.js`: app token labels and scope
line.

Run locally:

- `make test`: all 16 modules OK.
- `sh test-all.sh`: all suites pass, run in `node:24-bookworm`. On
macOS, `test-manage-workflow.js` fails 2 cases on unchanged master
because of the BSD tools.
- `npx eslint public/*.js`: 0 errors, the same 87 warnings as master.
- `node tests/e2e/test-lint-network-calls.js`
- `tests/e2e/test-user-management-e2e.js` against local e2etest servers
with the CI config: 44/44.
- Manually against a local instance:
  - in the browser: claim, edit, unclaim, and the Devices list;
  - with curl: the token flow, including a 409 after a browser save.

Not run: the full Playwright shard matrix from CI.

---------

Co-authored-by: Adam <196335+anieto@users.noreply.github.com>
2026-11-21 22:27:48 +01:00
ZachandClaude Opus 5.5 900bffb782 refactor(schema): give observers an explicit idx primary key (#2407)
## Summary

`observations.observer_idx` points at the **implicit rowid** of
`observers`, whose declared key is `id TEXT`. Implicit rowids are not
stable:

- a dump-and-reload (`sqlite3 .dump`, `iterdump`) renumbers them, which
silently re-links observations to the wrong observer;
- PostgreSQL has no rowid at all.

This change makes the key explicit. Split out of #2396 at review's
request, so SQLite installs get it independently of the Postgres work.

## What changes

- **One-time rebuild, in the ingestor.** On first start, `observers` is
rebuilt as `idx INTEGER PRIMARY KEY AUTOINCREMENT`, copying every rowid
into `idx` in one transaction. It fails closed on unknown columns or
NULL ids.
- **Rollback-safe.** In SQLite an `INTEGER PRIMARY KEY` *is* the rowid,
so `rowid == idx` forever. Older binaries that still join on
`observers.rowid` keep working, so both upgrade and rollback are safe.
- **No inherited orphans.** `AUTOINCREMENT` is seeded past
`MAX(observations.observer_idx)`, so a new observer can never inherit a
deleted observer's leftover rows.
- Fresh installs create the new shape directly, and `AssertReady`
requires `observers.idx` as the primary key (a plain column named `idx`
does not pass).
- **Observer writes update first.** With `AUTOINCREMENT`, SQLite
advances `sqlite_sequence` for every INSERT it attempts, even one `ON
CONFLICT` turns into an update. The per-packet observer upsert and the
clock-skew upsert now run an `UPDATE` and fall back to the unchanged
`INSERT ... ON CONFLICT` only for a new observer. So the sequence moves
only when an observer is added, and the next new observer gets `max(idx)
+ 1`.
- `packets_v` now has a single definition
(`dbschema.EnsurePacketsView`).
- Every server and ingestor query, the fixture capture script and the
test schemas use `idx`. Guard tests fail if `observers.rowid` comes
back.

## Operator impact

Every SQLite install gets this on its next ingestor start, with no
config change. The rebuild is one transaction over `observers`, which
holds one row per observer (typically hundreds), so it's quick.

**When the rebuild refuses.** It refuses rather than lose data in two
cases, and then the ingestor stops on every start until the cause is
fixed:

- `observers` has a column CoreScope doesn't know (one added locally).
The rebuild would drop it. Save its data, then drop or rename it: `ALTER
TABLE observers DROP COLUMN <name>`.
- A row has a NULL `id` (the legacy `TEXT PRIMARY KEY` allows one). Find
it with `SELECT rowid, * FROM observers WHERE id IS NULL`, then delete
or fix it.

The error message names the column or the row count and these steps.
Starting the previous version also works: nothing was changed, and
`packets_v` stays readable for external SQLite readers in the meantime.

**A refusal stops the server too, not only ingest.** The server checks
the schema at startup (`AssertReady`) and exits when `observers.idx` is
not the primary key, so the whole site is down until the cause above is
fixed or the previous version is started again. Check for a locally
added column or a NULL `id` before upgrading if you are unsure.

**Ingest cost.** Measured with `BenchmarkUpsertObserverAtExisting`
(`cmd/ingestor`): 20000 `UpsertObserverAt` calls over 500 existing
observers, on a store opened by `OpenStore` with its own pragmas (WAL,
`synchronous=FULL`), Ryzen 3700X in Docker. A plain upsert on the new
table writes 5.23 WAL frames per call; update-first writes 4.23, one
frame less. The pre-PR table was not measured. Time per call is 676 to
726 µs before and 665 to 668 µs after; it is bound by fsync, so the
difference is within noise.

## Testing

- `internal/dbschema/observers_idx_test.go`: the rebuild keeps every
observation linked to the same observer, refuses an unknown column or a
NULL id, never reuses an `idx` (including one only orphaned observations
still reference), is idempotent, and handles the full legacy table.
- `TestObserversIdxPreservesLinksOnE2EFixture`: on a copy of
`test-fixtures/e2e-fixture.db`, 499/499 observations keep the same
observer.
- Refusals and rollback: a refused rebuild leaves the table, its rows,
`foreign_keys` and a readable `packets_v` as they were; a failure after
the old table is dropped rolls all of it back, leaves no
`observers_new`, and the next start completes the rebuild
(`TestObserversIdxRollsBackMidRebuild`,
`TestPacketsViewFollowsObserversShape`).
- Guards: `TestNoObserversRowidInSource` (ingestor) and
`TestNoObserversRowidInServerSources` fail if a query joins on
`observers.rowid` again. `TestFreshObserversMatchesRebuiltShape` pins
that a fresh install and a rebuilt one end up identical.
- `TestUpsertObserverAtLeavesSequenceAlone` and
`TestRecordNaiveSkewLeavesSequenceAlone`: 1000 calls on an existing
observer leave the sequence alone (on the previous head both moved it 1
-> 1001), and a new observer then gets `max(idx) + 1`.
`TestUpdateObserverMatchesUpsert` and `TestUpdateNaiveSkewMatchesUpsert`
check that the update path writes the same row as the upsert (COALESCE
of empty fields, `last_seen` never going back, the `can_relay`
tri-state, the 24 h skew counter).
- `TestAssertReadyRequiresObserversIdxPrimaryKey`: the rebuilt shape
passes; a plain `idx` column or none fails.
`TestOpenStoreRefusesUnconvertibleObservers`: through `OpenStore`, an
unknown column or a NULL id fails the open and leaves the table, its
rows and `packets_v` as they were.
- `go vet` and `go test` pass on all modules (`scripts/allmod.sh`), with
gofmt clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)



---

## Merge order

These four PRs were split out of #2396 and are **independent — they can
merge in any order.** Verified: every pair is conflict-free, and all
four merged together pass `go vet`, `go test ./...` (all 16 modules) and
`sh test-all.sh`.

Suggested order, none of it urgent:

1. **#2408** — never lose rows during startup / a failed poll.
2. **#2407** — explicit `observers.idx` primary key.
3. **#2409** — content-hash migration moved into the ingestor.
4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not
affected (zig already links them with 8 MiB thread stacks); it fixes
native Alpine builds.

After these land, #2396 (Postgres) will be rebased onto master.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 22:27:09 +01:00
ZachandClaude Opus 5.5 1a6322a707 feat(map): more tile providers (topo, satellite, gray, vector), hillshade overlay and custom tile servers (#2434)
Closes #2428.

Adds free and open map providers (topographic, satellite, gray and
vector basemaps), a hillshade overlay, and support for operators' own
tile servers. Everything is additive and opt-in: Carto stays the
default, every new provider group is off until the operator turns it on,
and no existing config key or provider was removed. An upgrade with an
untouched config changes nothing in the picker.

> **Dependency:** #2426 has merged. The branch is rebased onto current
`master` (`6c78a1d2`) with no conflicts.

## What's new

- **Provider registry** (`public/map-tile-providers.js`). New groups are
**opt-in**: each is off until `map.tiles.providers.<group>.enabled` is
`true` (the boolean), so licences and volunteer servers are the
operator's call:
- **EOX**: Sentinel-2 satellite (2025 and 2016), Terrain Light, and a
place-labels overlay. EOX 2025 imagery is CC BY-NC-SA 4.0
(non-commercial); a commercial deployment should leave `eox` off or
license it.
- **OSM France**: Humanitarian, CyclOSM, OSM Gray and OSM Gray Dark
(volunteer-run, fair use).
- **OpenFreeMap** and **VersaTiles** vector styles: Liberty, Bright,
Positron, Dark, Fiord, Colorful, Muted and the Gray variants.
- **Hillshade** (below) is opt-in the same way. **OpenTopoMap**, the OSM
community server, Stamen and USGS keep their existing opt-in.
- **Custom tile servers**: `map.tiles.custom` adds raster (XYZ), vector
(MapLibre style URL) and raster-overlay entries as `custom-<id>`. They
can be used in `darkDefault` / `lightDefault`. Invalid entries are
skipped with a console warning.
- **Hillshade overlay** (`public/map-hillshade.js`): shaded in the
browser from Terrarium elevation tiles.
`map.tiles.providers.hillshade.url` can point at a mirror.
- **Vector basemaps** (`public/map-vector.js`) use a vendored MapLibre
GL 5.24.0 (CSP build) and `@maplibre/maplibre-gl-leaflet` 0.1.4.
- They are lazy-loaded only when a vector map is picked (about 380 KB
gzipped).
  - The load times out after 15 s.
  - Without WebGL the map falls back to the theme's default raster map.
- **One picker on Map, Live and Repeater Planner**: Auto, Street,
Topographic, Satellite, plus an Overlays group.
- **A map picked there is saved** (`localStorage` `mc-tile-pinned`,
untrusted: honoured only while registered). Every picker reopens on it
after a reload or on another page, until the visitor picks **Auto**.
- The other maps (home mini-map, node detail, node reach, analytics)
follow the visitor's choice through the shared builder: the saved pick
when it suits the current theme (satellite always does), otherwise the
theme's map.
- Those small maps never load MapLibre, so a vector choice shows the
theme's default raster map there.
- **Security contract**: the `CONTRACT` block at the top of
`map-tile-providers.js` (S1 to S7) covers one shared tile-URL validator
(`https?://` or a same-origin path, no credentials in the authority,
whitelisted placeholders), attribution HTML-escaped once, picker labels
set with `textContent`, explicit MapLibre options, pinned vendored
versions, CORS-safe hillshade decoding, and treating stored picks as
untrusted input. `map.tiles` is served publicly in `/api/config/client`,
so keys put there are public (the docs warn about this).

## Fixes included

The race stress suite written for this PR found teardown races when
leaving map pages mid-load:
- a lazy MapLibre load finishing after navigation;
- a hillshade tile decoding after `remove()`;
- a provider-change event arriving during teardown.

It also found a pre-existing document-listener leak in `AreaFilter` /
`RegionFilter`. All of these are fixed in `e6709e74`, with regression
tests in `tests/e2e/test-map-teardown-races.js`.

The first CI run then caught a bug in that fix (`7f8d792c`).
`MC_safeRemoveMap` called `map.off()` before `map.remove()`, so the
map's `unload` event reached no one. Leaflet relies on that event to
detach controls and clean up the zoom-animation proxy and BoxZoom, and
`test-issue-1206` watches it to confirm node maps are disposed. `off()`
now runs after `remove()`, and a new regression test checks that
`unload` fires, controls detach and no listener is left.

A later stress run (a new seed) found a real two-tab bug, fixed in
`0506fc1d`. When another tab's overlay pick is already in localStorage
but its `storage` event hasn't arrived, a layer-picker click here
triggered a sync that added that overlay while Leaflet was still
applying the click. Leaflet doesn't redraw the list for changes it sees
mid-click, so the overlay's checkbox stayed unticked, and the next click
on any overlay removed it and wrote the removal back to every tab. The
overlay set now defers a sync until the click has finished. New E2E
check `4b` in `test-map-providers-e2e.js` plants another tab's pick and
clicks twice: one write per click, the planted overlay stays ticked. It
fails on the previous code.

## Docs

- `docs/deployment.md`, "Map Tile Providers" section, rewritten. It
covers every provider and its licence, turning groups on and off, custom
servers, vector maps, hillshade, self-hosting, and what is public.
- User guide:
  - `map.md` gets a "Map style and overlays" section;
  - `live.md` notes the picker;
  - `configuration.md` gets a "Map tiles" section;
  - the FAQ gets a new entry on satellite, terrain and gray maps.
- `CHANGELOG.md`: an Unreleased entry.

## Test data

Two datasets were used, so every number below says which one it came
from.

**1. Large synthetic dataset** (perf A/B and screenshots). It is
generated, contains no real mesh data, and is not checked in.

- **Generator:** `synthgen` 1.0.0, seed 1, "full" scale. Built from
upstream `be16173e` and retimed so its 18-day window (2026-10-31 to
2026-11-18) ends at the time of the run.
- **Database:** SQLite, 3.94 GB (586 MB as zstd), 11.6 M rows in all.

| Table | Rows |
|---|---|
| Nodes | 18,102 (12,606 companions, 3,652 repeaters, 953 rooms, 891
sensors) |
| Nodes with a location | 13,452, spread worldwide (latitude −37° to
60°, longitude −123° to 175°) |
| Observers | 1,175, in 42 regions on six continents |
| Transmissions | 759,133 |
| Observations | 10,569,668 |
| Neighbour edges | 42,806 |
| Advert route evidence | 212,142 |

During the perf runs, the server's packet store was pinned to the newest
48 h with a 1.5 GB memory cap. Each server held about 65 k transmissions
and 0.9 M observations in memory (baseline 65,647 / 906,467; candidate
64,862 / 896,248), plus all 18,102 nodes (about 17,650 active) and 1,175
observers. The tracked store was about 450 MB.

**2. The repo's CI fixture** (`test-fixtures/e2e-fixture.db`, 704 KB).
All E2E suites and the race stress gates ran against it, freshened the
way CI does: 204 nodes, 31 observers, 500 transmissions, 502
observations. A second copy with the Repeater Planner enabled was used
for the planner-on stress runs.

**Perf environment:**
- Headless Chromium 158 via Playwright 1.58.2, at 1280×800 with device
scale factor 1.
- 9 measured runs per scenario plus 1 warm-up, on each of `master` and
this branch.
- Host: Linux, AMD Ryzen 7 3700X (16 threads), 63 GB RAM.
- **Tiles were stubbed.** The harness answers every tile, elevation and
style request locally with the same stub on both sides (`--mode stub`),
so perf timings involve no network. Only the screenshot gallery uses
real tiles.

## Screenshots

These were taken from the synthetic dataset with real tiles in a real
browser (headless Chromium, 1280×800). Click a thumbnail for the full
size. The images are hosted on a separate assets branch, so they are not
in this diff.

#### Street

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-light.png"
width="260" alt="Street: Carto (default) (light)"></a><br>Carto
(default) (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-dark.png"
width="260" alt="Street: Carto (default) (dark)"></a><br>Carto (default)
(dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-osmfr-hot-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-osmfr-hot-light.png"
width="260" alt="Street: OSM France Humanitarian (light)"></a><br>OSM
France Humanitarian (light) |

#### Gray

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-light.png"
width="260" alt="Gray: OSM France (light)"></a><br>OSM France (light) |
<a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-dark.png"
width="260" alt="Gray: OSM France (dark)"></a><br>OSM France (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-light.png"
width="260" alt="Gray: VersaTiles (light)"></a><br>VersaTiles (light) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-dark.png"
width="260" alt="Gray: VersaTiles (dark)"></a><br>VersaTiles (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-light.png"
width="260" alt="Gray: OpenFreeMap Positron (light)"></a><br>OpenFreeMap
Positron (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-dark.png"
width="260" alt="Gray: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap Dark
(dark) |

#### Topographic

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-cyclosm-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-cyclosm-light.png"
width="260" alt="Topo: CyclOSM (light)"></a><br>CyclOSM (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-eox-terrain-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-eox-terrain-light.png"
width="260" alt="Topo: EOX Terrain Light (light)"></a><br>EOX Terrain
Light (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-opentopomap-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-opentopomap-light.png"
width="260" alt="Topo: OpenTopoMap (opt-in) (light)"></a><br>OpenTopoMap
(opt-in) (light) |

#### Satellite

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-light.png"
width="260" alt="Satellite: Sentinel-2 2025 (light)"></a><br>Sentinel-2
2025 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-light.png"
width="260" alt="Satellite: Sentinel-2 2025 with labels
(light)"></a><br>Sentinel-2 2025 with labels (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-dark.png"
width="260" alt="Satellite: Sentinel-2 2025 (dark)"></a><br>Sentinel-2
2025 (dark) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-dark.png"
width="260" alt="Satellite: Sentinel-2 2025 with labels
(dark)"></a><br>Sentinel-2 2025 with labels (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-light.png"
width="260" alt="Satellite: Sentinel-2 2016 (light)"></a><br>Sentinel-2
2016 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-labels-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-labels-light.png"
width="260" alt="Satellite: Sentinel-2 2016 with labels
(light)"></a><br>Sentinel-2 2016 with labels (light) |

#### Vector (MapLibre)

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-liberty-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-liberty-light.png"
width="260" alt="Vector: OpenFreeMap Liberty
(light)"></a><br>OpenFreeMap Liberty (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-dark-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-dark-dark.png"
width="260" alt="Vector: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap
Dark (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-light.png"
width="260" alt="Vector: VersaTiles Colorful (light)"></a><br>VersaTiles
Colorful (light) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-dark.png"
width="260" alt="Vector: VersaTiles Colorful Dark
(dark)"></a><br>VersaTiles Colorful Dark (dark) | | |

#### Hillshade overlay

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-light.png"
width="260" alt="Overlay: hillshade over Sentinel-2
(light)"></a><br>hillshade over Sentinel-2 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-light.png"
width="260" alt="Overlay: hillshade over Carto
(light)"></a><br>hillshade over Carto (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-dark.png"
width="260" alt="Overlay: hillshade over Sentinel-2
(dark)"></a><br>hillshade over Sentinel-2 (dark) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-dark.png"
width="260" alt="Overlay: hillshade over Carto (dark)"></a><br>hillshade
over Carto (dark) | | |

#### Repeater Planner

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-light.png"
width="260" alt="Planner: Sentinel-2 + hillshade
(light)"></a><br>Sentinel-2 + hillshade (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-dark.png"
width="260" alt="Planner: Sentinel-2 + hillshade
(dark)"></a><br>Sentinel-2 + hillshade (dark) |

#### Layer picker

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-light.png"
width="260" alt="Layer picker open (light)"></a><br>Layer picker open
(light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-dark.png"
width="260" alt="Layer picker open (dark)"></a><br>Layer picker open
(dark) |

#### Custom tile server

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-map.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-map.png"
width="260" alt="Custom raster provider on the map"></a><br>Custom
raster provider on the map | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-picker.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-picker.png"
width="260" alt="Custom raster provider in the picker"></a><br>Custom
raster provider in the picker |

## Testing

The full suite ran twice, each time on a fresh clone:

| | Pass 1 | Pass 2 |
|---|---|---|
| Commit (before rebase) | `8a56e841` | `9355169b` |
| `test-all.sh` (unit) | pass | pass |
| `npm run lint` | 0 errors (87 warnings, all pre-existing) | 0 errors
(87 warnings, all pre-existing) |
| Go `cmd/server` / `cmd/ingestor` | ok / ok | ok / ok |
| E2E, map-related files (CI fixture) | 44/44 | 44/44 + planner-on run |
| `test-e2e-playwright.js` | 149 passed, 3 skipped | 149 passed, 3
skipped |
| Race stress gate (seeds) | 101, 202: all pass | 303, 404: all pass |
| Stress with planner enabled | pass | pass (seed 505) |
| Gallery | 32/32 | 32/32 |

The rebased equivalents of the pass 1 and pass 2 commits are `fde2552c`
and `e4d5c447`.

**After the first CI run.** The two passes covered only the map-related
E2E files, and CI exposed two problems:

1. `test-issue-1206` failed 7 checks. This was the `unload` bug
described under "Fixes included".
2. The JS unit gate runs without `node_modules`, so the two new jsdom
suites crashed `test-all.sh`. Locally, jsdom was reachable from a parent
checkout, which hid this.

The second problem is fixed in `57720bf3`, following the
`domIntegration` convention in `scripts/non-unit-tests.json`.
`test-map-teardown-races.js` moved to `tests/e2e/`, and the three jsdom
zoom-limit tests split out into
`tests/e2e/test-map-vector-leaflet-zoom.js`, so 30 + 3 = 33 tests are
kept. Both run from E2E shard 2 and take a few seconds. The heavy suites
(race stress, perf, gallery) stay local-only and are not run in GitHub
Actions.

These checks ran on the fixed, rebased tree (`57720bf3`):

| Check | Result |
|---|---|
| `test-all.sh` from a copy with no `node_modules` (as CI runs it) |
pass |
| `test-test-inventory.js` | 399 suites classified |
| `npm run lint` | 0 errors |
| Go `cmd/server` / `cmd/ingestor` | ok / ok |
| `test-issue-1206-resize-observer-leak-e2e.js` | 13/13 (7 were failing)
|
| `test-e2e-playwright.js` | 149 passed, 3 skipped |
| `test-map-providers-e2e.js` / harness E2E | 17/17, 16/16 |
| `test-map-teardown-races.js` / `test-map-vector-leaflet-zoom.js` |
19/19, 3/3 |
| Race stress (3 iterations) | all scenarios pass |

The commits since pass 2 are `76c67184` (docs only), `7f8d792c` and
`57720bf3`.

**GitHub Actions on `0506fc1d` (rebased onto `master`, with the two-tab
fix): every check passed** (13 passed, 4 skipped as usual for a PR).
That covers Go build and test, all three E2E shards, the server race
detector and both image builds. The moved suites ran in shard 2
(`test-map-teardown-races.js` 19/19, `test-map-vector-leaflet-zoom.js`
3/3), and the unit gate ran `test-map-vector.js` (30/30).

**Race stress suite** (`tests/e2e/test-map-race-stress-e2e.js`). Each
gate run does 20 iterations of scenarios a to e and 50 cycles of f. The
scenarios cover:
- rapid provider switching;
- theme flips during a vector load;
- navigating away mid-load (including a 15 s MapLibre timeout);
- repeated map-page navigation with heap checks;
- overlay toggles;
- cross-page picker persistence.

Results for each run:
- Listener counts stay flat: at most 3 `mc-tile-provider-changed`
listeners and 5 `storage` listeners.
- At most one live WebGL context at a time.
- Heap growth in the second half of scenario d was 0.25 to 0.45 MB
across the four gate runs.
- Upstream `master` grows about 0.39 MB per map navigation loop in the
same scenario; this branch levels off.

## Maintainer review (addressed)

1. **Licence default.** All five new groups (`eox`, `osmfr`,
`openfreemap`, `versatiles`, `hillshade`) are opt-in: off unless
`enabled: true` (`"true"`, `1` or `{}` don't count). With an untouched
config the picker lists the same 6 base maps as before. Docs,
`config.example.json` (now `enabled: false` for each) and the changelog
say so, and each group's terms are listed before you turn it on.
2. **Saved pick.** The on-map picker now saves the pick (option 1 of
your two), as the docs promised. Behaviour, and what an other-theme pick
does:
- Every picker opens on the pinned map, on any page and after a reload,
until **Auto** is picked. A pin is shown whatever the theme, as an
explicit pick already was.
- Maps without a picker show it when it fits the current theme (`type`
is the theme or `any`), otherwise the theme's map. So a light pin isn't
drawn on a dark node map; satellite always is.
- A stored pin whose group is turned off reads as Auto, and comes back
if the group is turned on again; only picking Auto clears it.
- On a map with a picker, Auto always means the theme's map: the pin is
only the explicit layer, never loaded into the hidden Auto layer. A
choice in the customizer, and Reset All, clear the pin. A failed write
(quota full) never brings back an older stored pin.
- E2E `7a` to `7c` in `test-map-providers-e2e.js`: pick, reload, the
radio and tiles are the pick, the node detail map draws it, and a light
pin in dark is kept but not drawn. Auto clears the key; junk in it gives
Auto; an untouched config lists none of the new groups. 7a and 7b fail
with the save line removed. Unit tests are in `test-map-tile-pin.js`
(21).
3. **CI.** `test-map-providers-e2e.js` (24 checks, including `4b`, the
two-tab regression check) and `test-map-providers-harness-e2e.js` (16)
now run in E2E shard 3 against the fixture server: every tile host is
stubbed.

Non-blocking:
- `getTileUrl()` now returns the raster fallback for a vector pick
(`MC_rasterTileUrl`). There's a test in `test-tile-resolver.js` that
fails on the old code.
- The `map-vector.js` header and the binding's vendor README list the
Leaflet and binding internals to recheck on an upgrade.
- `docs/deployment.md` says SRI makes a JS-rewriting proxy drop vector
maps to raster, so serve `public/vendor/` unmodified.
- `scripts/map-perf/README.md` says where `SEED_DB` and `SHIFT_SH` come
from.
- The maps with `attributionControl: false` are left as they are. That
predates this PR, and with every new group opt-in, an upgrade adds no
uncredited imagery. Follow-up for all of them: #2441.
- The OpenFreeMap credit line is `OpenFreeMap © OpenMapTiles Data from
OpenStreetMap` (`2424f3c3`), the line OpenFreeMap gives for clients that
don't add it automatically.

Verified after these changes:
- `test-all.sh` and lint pass, with 0 errors.
- Both provider E2E suites pass: 24/24 and 16/16.
- Race stress, seed 979: scenarios a, b, d and e pass. c (15/15) and f
(37/37) pass after updating their model of the pick: they expected Auto
after a return, which was the old unsaved behaviour.
- The stress suite now enables the five groups itself, so it keeps
exercising them.

## Performance

A/B test: `master` against this branch on the large synthetic dataset,
with 9 runs per scenario and stubbed tiles (see Test data;
`scripts/map-perf/`). The full HTML report is not attached; ask if you
want it.

- **No budget failures** in either pass. The `raster-no-maplibre` budget
holds: no raster scenario loads MapLibre.
- **DOMContentLoaded is unchanged on every page**, even with the two new
scripts (`map-hillshade.js` and `map-vector.js`, about 635 lines in all;
MapLibre itself loads only on demand).
- **Switching providers 50 times (raster only), median ms per switch:**
  - dark: 146 → 108.5;
  - light: 208 → 208.5.
  - Heap growth during the run: 0.3 to 0.4% on both sides.
- **Flagged and checked:**
- Live and Analytics `loadEventMs` are flagged in some runs. This metric
has two modes on **both** branches: `load` fires either before Leaflet
adds its first tile `<img>`s (about 150 ms) or after they finish (about
400 ms). A direct probe of `master` alone gave 428, 404 and 124 ms. On
these scenarios, map creation, first tile, all tiles, first marker and
wall time are within a few ms on both sides.
- Carto and Positron `allTilesSinceMapMs` are 4 to 6 ms slower, just
above `master`'s p95. Tiles are stubbed locally, so this is not network
noise. The cause is not isolated. Both providers now go through the
shared layer builder, and on the same scenarios first tile, all tiles
and wall time are within 10 ms of `master` (Carto Voyager +2 to +3 ms;
Positron Dark +6 to +9 ms).
- The Analytics map shows one long task of about 50 ms against a
`master` median of 0, but `master`'s p95 is 52 to 84 ms. This is
disclosed in case it shows up for you.
- Other desktop apps were running during the perf runs, so expect some
noise.

## Commits

1. `c6bf12ed` feat(map): tile provider registry with custom servers,
gray, topo and satellite maps, and overlays
2. `9237230b` feat(map): client-side hillshade overlay
3. `781e41b4` feat(map): vector basemaps via vendored MapLibre GL 5.24.0
4. `dae36d2e` refactor(map): every map surface uses the shared basemap
builder
5. `3d4afcdc` fix(map): teardown races and listener leaks when leaving
map pages
6. `b798b2fe` test(server): pin map.tiles passthrough in
/api/config/client
7. `171b1bc4` test(map): E2E, race stress suite, perf harness and
screenshot gallery
8. `61f73875` docs(map): providers, custom tile servers, overlays and
self-hosting
9. `135a9b12` test(map): gallery shows opt-in OpenTopoMap and retries
slow tile servers
10. `3833572b` test(map): harness waits for vector basemaps; stress
suite fixes from test pass 1
11. `3e543cad` test(map-perf): redact tile-URL credentials; compare
switch50 like with like
12. `d5006111` docs(map): user guide for map styles, overlays and
map.tiles
13. `90ee1d5d` fix(map): MC_safeRemoveMap lets unload listeners run
before dropping them
14. `5fc57b5c` test(map): run the jsdom map suites in the E2E job, not
the unit gate
15. `bd36b634` fix(map): another tab's overlay pick that lands mid-click
is no longer removed
16. `38df65d2` fix(map): new tile groups are opt-in; a map picked in the
picker is saved
17. `eafbcac9` ci(e2e): run the map provider E2E suites in shard 3
18. `65b223ec` fix(map): getTileUrl() never returns a vector style url;
review notes
19. `f3402c5d` test(map): race stress enables the opt-in groups and
expects the saved pick
20. `e92b8631` fix(map): Auto on a picker map ignores the saved pick;
customizer clears it

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 22:26:46 +01:00
ZachandClaude Opus 5.5 9a57b2fca6 test(server): set the local zone in a child process, not via time.Local (#2438)
## Summary

`TestNodeAnalyticsWindowIgnoresLocalZone` set the zone by assigning the
global `time.Local`. Every `time.Now()` reads that variable, so if a
timer left over from an earlier test fires during this test, the race
detector reports a data race. It counts as a failure of this test,
whichever test left the timer.

It showed up as an intermittent failure of the server race detector
(shard 3/4) on a PR run (#2436). Three local runs of the same shard
passed. The code under test is fine; only the test was racy.

## Change

The test now re-runs itself in a child process with `TZ=Etc/GMT-2`
(UTC+2) and writes nothing global:
- **Embedded zone data:** `time/tzdata` is embedded in the test binary,
so the zone resolves without the host's zoneinfo.
- **Zone check:** the child first checks that its local offset really is
+02:00.
- **Test name:** the child's `-test.run` pattern is built from
`t.Name()`.
- **Vacuous pass guarded:** the parent requires `--- PASS: <name>` in
the child's output, so a rename can't make it pass without running.
- **Windows:** the test is skipped there before the re-exec. Go on
Windows takes the local zone from the OS and ignores `TZ`, so the child
cannot be put in UTC+2.
- **Bounded child:** the child runs under a 60 s context, so a hung
child fails with its output.

This follows the existing re-exec pattern in
`distance_background_fill_test.go`. `TestMain` is just
`os.Exit(m.Run())`, so the child runs only this test. It works under the
sharded `-test.run` in the race job, and in coverage builds.

## Testing

- `go test -race -run TestNodeAnalyticsWindowIgnoresLocalZone`: pass.
gofmt and vet are clean.
- **Still guards the original bug:** with `time.Now().UTC()` changed
back to `time.Now()` in `GetNodeAnalytics`, it fails as before. The 7d
window holds 166 of 168 packets, and `From` is `...+02:00`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 22:26:05 +01:00
AdamandAdam 49dd4ee773 List my claimed repeaters on My account (#2430)
Follow-up to #2415, see #2414.

## Situation

A user can claim repeaters, but no page lists the ones they claimed.
Claims show on each node page ("Maintained by"), in the admin user
detail and in the "Download my data" export. A maintainer of several
repeaters has to remember them and open each node page to see whether
one went quiet.

## Change

**`GET /api/account/repeaters`** (browser session; device tokens answer
403) returns the caller's claims. For each one:

- `pubkey` and `name`.
- `known`: false once the node left the analyzer database. The claim
stays.
- `role` and `claimedAt`.
- `lastHeard`: the newer of the last advert and the last transmission
certainly from the node, the same sources as the node page's Last Heard.
- `lastRelayed`: when its hop prefix last relayed. Only used to mark a
silent repeater as uncertain, since the prefix can be shared.
- `status`: the saved repeater status, or null.
- `comaintainers`: the other active claimers' display names.

The list is sorted by name, with unnamed repeaters after, by pubkey.
It's built on the existing `RepeaterClaimsByUser`, `NotifyNodes` and
`LastHeardMap`.

**My account, My repeaters**, a new section after Companions:

- Silent repeaters come first. Health uses `getNodeStatus`, so the
threshold matches the node pages and map (`infraSilentMs`):
  - green "heard 2h ago";
  - yellow "last heard …, its prefix still relays";
  - red "silent, last heard 184d ago" or "never heard";
  - grey "no longer in the database".
- Each row links to the node page and shows the saved status ("no
details yet" without one), the claim date and "also maintained by …".
- *Unclaim* asks for confirmation, then uses the existing `DELETE
/api/repeaters/{pubkey}/claim`.
- The empty state says how to claim.
- `#/account?section=repeaters` deep-links to the section, like
`section=notifications`.

Editing stays on the node page, so there's only one editor. The admin
user detail and the export are unchanged.

Docs: `openapi.go` and the Repeater info section of
`docs/user-guide/accounts.md`.

## Notes

- A mobile client will want the same list. #2429 proposes device-token
scopes for repeater claims. If it lands, letting a `repeater:claim`
token read this route is a one-line addition to the bearer route table,
and I'd rather do that there than here.
- The two PRs touch nearby lines in `openapi.go`, `repeater_handlers.go`
(route registration) and `public/account.js`. Whichever merges second
needs a small rebase.

## Tests

- `cmd/server/my_repeaters_test.go`:
  - empty list;
  - ordering (named first, case-insensitive);
  - unheard, heard and departed nodes;
  - saved status;
  - co-maintainers, excluding the caller and disabled users;
  - another user's view;
  - unclaim removes the row;
  - anonymous gets 401 and a device token 403;
  - no analyzer DB.
- `tests/unit/test-user-management-ui.js`: health sort and chips,
escaping, the short-pubkey fallback, status and co-maintainer lines,
Unclaim on every row, and the empty state.
- `tests/e2e/test-user-management-e2e.js`:
  - the claimed repeater appears with its status and node link;
  - axe on the section;
- Unclaim: dismissing the confirm keeps the claim, accepting empties the
list.

Run locally:

- `make test`: all 16 modules OK.
- `sh test-all.sh` and `node tests/e2e/test-lint-network-calls.js`:
pass, run in `node:24-bookworm`.
- `npx eslint public/*.js`: 0 errors, the same 87 warnings as master.
- `test-user-management-e2e.js` against local e2etest servers with the
CI config: 46/46.
- Manually in Chrome, light and dark: heard, silent and co-maintained
rows, and the deep link.

Not run: the full Playwright shard matrix from CI.

---------

Co-authored-by: Adam <196335+anieto@users.noreply.github.com>
2026-11-20 16:25:51 +01:00
6c78a1d2c1 fix(ingestor): detect rejected SUBACK and retry subscription (#2413)
## Problem

`paho.mqtt.golang` does not surface a SUBACK failure return code
(`0x80`) via `token.Error()`. The subscribe token completes
"successfully", so the client stays connected **with zero
subscriptions**. The ingestor logs `subscribed to meshcore/#` and then
silently ingests nothing until it is restarted.

### Observed in production (live.saarmesh.de)

1. EMQX was restarted. CoreScope reconnected about 15 s later.
2. EMQX accepts CONNECT before its authorization sources are loaded.
With `authorization.no_match = deny`, the SUBSCRIBE sent from
`OnConnect` was rejected.
3. The ingestor logged `subscribed to meshcore/#`, but `emqx ctl clients
list` showed `subscriptions=0` for the CoreScope client.
4. The liveness watchdog reported `NEVER received a message …`. By
design (#1335) it does not force a reconnect for `NeverReceived`, so
ingestion stayed dead (about 40 min until a manual container restart).

## Fix

- `subscribeOnce`: bounds the wait with a timeout and treats a SUBACK
return code `>= 0x80` for the topic as an error. It matches on the
`Result() map[string]byte` method that `*mqtt.SubscribeToken`
implements, so it can be tested.
- `subscribeWithRetry`: on failure, retries in the background with
exponential backoff (2 s → 60 s) while the client stays connected. The
loop ends on disconnect, and the next `OnConnect` starts a fresh
attempt. `OnConnect` itself is never blocked by the retries.
- `OnConnect` now calls `subscribeWithRetry` per topic instead of the
previous unbounded `token.Wait()` + `token.Error()` check.

## Tests

New `subscribe_retry_test.go` with a fake client/token covers:
- granted SUBACK → success
- rejected SUBACK (`0x80`) → error
- token error → error
- rejection ×2 then grant → recovers, and retries stop after success
- success → exactly one SUBSCRIBE
- disconnect during retry → loop stops

`go vet` is clean, `gofmt` is clean, and the full `cmd/ingestor` test
suite passes. The same change, backported onto v3.17.0, is running on
live.saarmesh.de and subscribes cleanly after a restart.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01XfZdhsbrRHCYG2ksxRUo6B

---------

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 21:16:58 +01:00
efiten 58c937c462 test(planner): trim the 30 km viewshed equivalence cases (#2431)
Refs #2424. Second step after #2426: `TestViewshedMatchesReference` is
the slowest test in the server package.

## What changes

Only `cmd/server/planner_viewshed_test.go`.

1. **30 km equivalence cases.** Of the three 30 km terrains (fake, flat,
ridge) only the ridge case stays. Up to 30 km the cell size is the 0.2
km floor (`viewshedCellKm` = `max(0.2, r/150)`,
`planner_viewshed.go:50`), so the 30 km cases add a bigger grid and
longer rays (more samples per profile, `planProfileSamples`), not a
different cell size. The ridge case keeps one full-size grid in the
regular suite, with level ground west of the ridge and shadow east of
it.
2. **Missing-tile case.** The old case could not reach its missing tile.
The tile `{31,-98}` starts at lat 31; the 30 km bbox around lat 30.5,
expanded by 1 km, ends near lat 30.78. So it was a second copy of the 30
km fake case. It now uses a site at lat 30.95 (0.05 degrees, about 5.6
km, south of the tile edge) with a 12 km range.
3. **New guard `TestViewshedMissingTileCaseReachesTheTile`.** It checks
that the case touches the missing tile (`MissingTiles()` = 1 of at least
2), that every in-range cell north of the edge is NoData, and that cells
south of it have data. With the old site (lat 30.5) it fails:
`MissingTiles() = 0 of 1, want 1 of at least 2`.

## Measured

Native, Windows, Go 1.26.1, `go test -count=1 -run
'^TestViewshedMatchesReference$' -v .` with a temporary timer per case:

| case | before | after |
|---|---|---|
| fake/30km | 3.49 s | removed |
| flat/30km | 3.07 s | removed |
| ridge/30km | 2.78 s | kept |
| missing-tile | 4.05 s (30 km) | 12 km |
| whole test | 17.0 s | 6.2 s |

Before this change the test was 16.1 of 101 s for the whole server
package in a local `-json` run (16%). `go test -count=1 -run
'TestViewshed|TestPlanner|TestClearance|TestCoverage' .` passes; `gofmt`
and `go vet` are clean.

## Not done

- Not measured under `-race` and coverage, the way CI runs it. The CI
time comes from the hosted run of this PR.
- No equivalence case above 30 km, where the cell grows past 0.2 km. The
old suite had none either; `TestViewshedCellKm` covers the formula and
`BenchmarkViewshedWorkCount` the 60 km counts.
- The CI job split (running `-race` in shards next to the coverage run)
is a separate PR.
2026-11-18 23:06:06 +01:00
0af70cf2c1 fix(build): pin musl thread stacks at 8 MiB and check the artifacts (#2410)
## Summary

Pins musl thread stacks at 8 MiB on every link, and fixes a crash in
**native Alpine builds** (`make build`, `make test`, or `go test` on an
Alpine host).

**The released images are not affected.** An earlier version of this
description said public endpoints could crash the shipped image. That
was wrong, and review caught it: see "Who is affected".

## Root cause

- musl sizes every non-main thread's stack from the ELF `PT_GNU_STACK`
size, with a floor of 128 KiB and a cap of 8 MiB (glibc defaults to 8
MiB).
- cgo calls run on the calling thread's stack, and SQLite's
`multiSelect` recurses about 850 bytes per `UNION ALL` term.
- `lookupPrefixes` (up to 400 terms) and `sharedEightBytePrefixes` (up
to 450 terms) overflow a 128 KiB stack and SIGSEGV the whole process.

## Who is affected

| Link | `PT_GNU_STACK` without the flag | Thread stack | Wide-UNION
tests |
|---|---|---|---|
| Alpine gcc + GNU ld (`make build` / `make test` / `go test` on Alpine)
| `0` | 128 KiB | **SIGSEGV** |
| zig 0.16.0, x86_64-linux-musl (Dockerfile, `crossbuild`, release job)
| `0x1000000` (16 MiB) | 8 MiB | pass 3/3 |
| zig 0.16.0, aarch64-linux-musl | `0x1000000` (16 MiB) | 8 MiB | pass
3/3, run on an arm64 node |

So for the released images the flag pins a value that zig's default
already provides. It stops being a property of a linker default, and a
build check now fails if it is lost. For native Alpine links it is the
fix: with the flag those tests pass.

Control: a zig arm64 link with `-Wl,-z,stack-size=131072` gets 128 KiB
threads and the same tests SIGSEGV in cgo on the arm64 node.

## Fix

- Every musl link passes `-Wl,-z,stack-size=8388608`: the `Dockerfile`,
the `Makefile` (`build` and `test` on Linux, and `crossbuild`) and the
release job in `deploy.yml`.
- This reserves address space only. Memory is committed as pages are
touched.
- **The artifacts are checked.** The Dockerfile's builder stage and the
release job's "Verify release binaries" step read `PT_GNU_STACK` with
`readelf` and fail unless it is exactly 8 MiB. Verified: the builder
stage passes for `linux/amd64` and `linux/arm64`, and fails with
`--build-arg STACK_LDFLAG=` (`/corescope-server: PT_GNU_STACK is not 8
MiB`, memsz `0x1000000`).

## Testing

- `TestMuslThreadStackWideUnionQueries`
(`cmd/server/musl_thread_stack_linux_test.go`, Linux only) runs both
queries off the main thread and asserts exact results, so a failed chunk
cannot pass as "nothing matched". Under `golang:1.27-alpine` with gcc:
SIGSEGV without the flag, pass with it. On CI's glibc runners it passes
either way, which is why the artifact checks above exist.
- `tests/unit/test-issue-1956-release-routing.js`: the release verify
step must check both binaries, and a `0x000000` or `0x1000000` header
must fail the release.
- `tests/unit/test-dockerfile-buildplatform.js`: all three `go build`
lines carry `STACK_LDFLAG`, and the `readelf` check covers all three
binaries.
- `sh test-all.sh` passes. `go vet` passes for `cmd/server` on Linux
with gofmt clean.
- Not verified: a Windows `go vet`. The fix there is the
`_linux_test.go` file name; no Windows machine was available to run it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## Merge order

These four PRs were split out of #2396 and are **independent — they can
merge in any order.** Verified: every pair is conflict-free, and all
four merged together pass `go vet`, `go test ./...` (all 16 modules) and
`sh test-all.sh`. (That combined run predates the review fixes on #2409
and #2410.)

Suggested order, none of it urgent:

1. **#2408** — never lose rows during startup / a failed poll.
2. **#2407** — explicit `observers.idx` primary key.
3. **#2409** — content-hash migration moved into the ingestor.
4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not
affected; it fixes native Alpine builds.

After these land, #2396 (Postgres) will be rebased onto master.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-11-18 18:35:41 +01:00
ZachandClaude Opus 5.5 7ae6645337 fix(#1856): run the content-hash migration in the ingestor (#2409)
## Summary

Fixes #1856. The content-hash migration from #787 has done nothing since
#1283. It ran in the server on the read-only analyzer handle, so every
batch failed. Before #1958 it also reported `hashMigrationComplete:
true` anyway. It also meant the server issued `UPDATE` and `DELETE`
against a database it must never write.

The migration now runs in the ingestor as the async migration
`content_hash_payload_bits_v1`. Split out of #2396 at review's request.

## Who is affected

Only installs holding transmissions from **before #787 (April 18)** that
did not run a server between #787 and #1283 (April 18 to May 18), the
only month the old server-side pass could commit. `retention.packetDays`
defaults to keep-forever, so long-lived installs can qualify. For
everyone else this is a one-time scan that changes nothing.

## What changes

- Rows present at boot are rescanned by id in batches of 5000, read
outside the writer lock. Stale rows commit 250 per transaction (the
retention bound), so ingest waits for one chunk at most, not a whole
batch.
- After each scan batch that changed rows, a progress line is logged.
- A stale hash is rewritten in place.
- A row whose new hash already exists is **merged** into that survivor:
- its observations move, except any that would violate the dedup index;
  - the survivor's `first_seen` and `last_seen` widen to cover both;
- the duplicate is deleted with the same child cleanup retention uses
(now one shared helper).
- Collisions are found by lookup, not by catching a constraint error.
- The server copy is deleted. `/api/stats.hashMigrationComplete` keeps
its shape and now reports the ingestor's `_async_migrations` status.
- `readonly_invariant_test.go` now fails on any `Begin` or
`INSERT`/`UPDATE`/`DELETE` against the analyzer DB from `cmd/server`.

## Known limitation: a running server does not follow the pass

A server that loaded these rows before the pass keeps the old hashes,
the merged duplicates, and survivors without their moved observations
until it restarts. That's deliberate:
- `tx_rewrite_feed` cannot carry a merge. Its rows cascade away with the
deleted duplicate, and the server's tail refreshes only `decoded_json`
and `scope_name`.
- Removing a transmission from every in-memory index is hot-path code
that a one-time pass over old rows does not justify.

Instead, when the pass changed anything, its summary log line tells the
operator to restart servers. The reasoning is recorded at the top of
`cmd/ingestor/hash_migrate.go`. (Raised by a reviewer in #2396.)

## Testing

- `cmd/ingestor/hash_migrate_test.go`: in-place rewrite; stale rows of
one scan batch commit in write-sized chunks
(`TestMigrateContentHashesCommitsInWriteChunks`); merge of a colliding
pair (observation move, dedup-index conflicts, span widening, child
cleanup); a stale pair split across batches; no-op; stops on cancel;
reports a write failure; scheduled on open; and
`TestContentHashMigrationSummary` (the restart hint, and silence when
nothing changed).
- `cmd/server/hash_migration_status_test.go`: `hashMigrationComplete`
follows the ingestor's migration row.
- `cmd/server/readonly_invariant_test.go`: no write path to the analyzer
DB from the server.
- Lock hold, measured on 100k stale transmissions (half merging, 5
observations each) with a concurrent writer probing the lock:

  | write batch | hold p50 | hold max | probe max wait | total |
  |---|---|---|---|---|
  | 5000 (before) | 1269 ms | 1356 ms | 1351 ms | 25.3 s |
  | 250 (now) | 63 ms | 108 ms | 158 ms | 27.5 s |

The same run at 10× the size (1M stale transmissions, 5M observations):

  | write batch | hold p50 | hold max | probe max wait | total |
  |---|---|---|---|---|
  | 5000 (before) | 1344 ms | 1808 ms | 1802 ms | 4 m 33 s |
  | 250 (now) | 66 ms | 132 ms | 199 ms | 4 m 38 s |
- Removed `cmd/server/hash_migrate_test.go` and
`hash_migrate_eviction_test.go` (#2327) along with the server code they
covered.
- `go vet` and `go test` pass on all modules, with gofmt clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


---

## Merge order

These four PRs were split out of #2396 and are **independent — they can
merge in any order.** Verified: every pair is conflict-free, and all
four merged together pass `go vet`, `go test ./...` (all 16 modules) and
`sh test-all.sh`.

Suggested order, none of it urgent:

1. **#2408** — never lose rows during startup / a failed poll.
2. **#2407** — explicit `observers.idx` primary key.
3. **#2409** — content-hash migration moved into the ingestor.
4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not
affected (zig already links them with 8 MiB thread stacks); it fixes
native Alpine builds.

After these land, #2396 (Postgres) will be rebased onto master.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:16:29 +01:00
n30nexandn30nex cfb38e76a7 test(planner): benchmark full-range viewshed work counts (#2426)
Fixes #2424. The server’s default race-and-coverage suite ran full 30 km
and 60 km terrain diagnostics on every run. Three workflows reached
their 20-minute package deadline while the work-count test was running.

The ordinary test now exercises a 5 km grid with 2,012 classified cells
and checks the actual lookup reduction. Excluding the initial site
lookup, the single terrain walk must use at most half the reference
lookups. The missing-site test still allocates and checks the 60 km
grid. Full-size diagnostic counts are available from `cmd/server`:

```sh
go test -run '^$' -bench '^BenchmarkViewshedWorkCount$' -benchtime=1x
```

TDD exemption: pure test-harness refactor. Only
`planner_viewshed_test.go` changes. The benchmark preserves the original
grid dimensions, classified cells and lookup counts at both distances.
The existing reference-equivalence, parallel-worker, cancellation and
race checks remain in the regular suite.

Validation on Windows, Go 1.27.2, `GOMAXPROCS=2`:

- Default work-count test, native: 17.60 s before, 0.04 s after. With
race detection and atomic coverage, the original exceeded the 210 s
diagnostic limit. The candidate took 0.63 s.
- Independent benchmark reproduced 31,756,821 reference / 15,878,411
single-pass lookups at 30 km, and 62,381,281 / 31,190,641 at 60 km.
- Deliberate duplicate traversal and oversized-grid controls both failed
their intended assertions. These timings measure the test workload.
- The [hosted Go
job](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44199958643/job/134451261110)
passed the full server race-and-coverage suite in 1,125.875 s under its
existing 1,200 s deadline. All browser shards passed.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-18 18:15:58 +01:00
ZachandClaude Opus 5.5 2ecc3d1e72 fix(server): never lose rows committed during startup or a failed poll (#2408)
## Summary

Two pre-existing ways a running server **permanently** missed data. The
database was correct; the server's in-memory store was not, and stayed
wrong until the next restart.

Found while comparing several server replicas side by side in #2396: two
replicas that restarted mid-ingest each lacked one transmission and its
12 observations. Split out of #2396 at review's request, because it
affects every SQLite install.

## The bugs

1. **Startup window.** The poller took its starting cursors from a fresh
`MAX(id)` *after* the startup load finished. A row committed between the
load's last chunk snapshot and that query ended up in neither the store
nor any later poll. Its later observations were then dropped as orphans.
2. **Failed poll.** A failed observation read still advanced the cursor
to its upper bound, skipping up to 500 observations for good.

## The fix

- `NewPacketStore` records the transmission and observation high-water
marks **before** the load, as the advert-evidence and rewrite feeds
already did.
  - The loaded max transmission id stays as a floor.
- The loaded max observation id no longer is one: a chunk-1 transmission
can gain an observation whose id is *below* one that chunk 2 loaded.
  - Rows read twice are deduplicated and broadcast at most once.
- A failed read keeps the cursor and retries on the next tick, logging
once per failure streak.
- Comments and a test pin why an observation of a transmission whose
chunk has not merged yet cannot be skipped: the poller's evidence
refresh waits on that chunk's lock.

## Performance

- **Startup:** two extra indexed `MAX(id)` queries, which are
primary-key lookups.
- **Poll loop:** unchanged complexity. The only new per-tick work is a
dedupe check on rows read twice, a map lookup by id (O(1)). Re-reads
happen only for rows in the startup window, which is bounded by what was
committed during the load, and on a retry after a failed read.

## Testing

`cmd/server/startup_poll_gap_test.go` seeds the ingestor's own DDL plus
`dbschema.Apply`:

| Test | What it pins |
|------|--------------|
| `TestStartupLoadPollerGap_SQLite` | A transmission and observations
committed between the load's snapshot and the poller's start reach the
store |
| `TestStartupLoadPollerGapMultiChunk_SQLite` | Same, across 3 chunks,
including an observation id below the max loaded one |
| `TestStartupObsDuringChunkMerge_SQLite` | An observation of a
transmission whose chunk has not merged yet is not skipped |
| `TestPollerRetriesFailedRead_SQLite` | A failed observation read does
not advance the cursor |

**Red/green:** against master's `store.go`, `websocket.go`,
`chunked_load.go` and `db.go`, the first, second and fourth tests fail
with the rows missing, e.g.:

```
tx 3, committed between the load's snapshot and the poller's cursor init, is missing from the store (lost forever)
obs 3, committed while the observation read failed, never reached the store: the cursor passed it
```

The third depends on test hooks this change adds, so it cannot run
against the old code. `go vet` and `go test` pass on all modules, with
gofmt clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


---

## Merge order

These four PRs were split out of #2396 and are **independent — they can
merge in any order.** Verified: every pair is conflict-free, and all
four merged together pass `go vet`, `go test ./...` (all 16 modules) and
`sh test-all.sh`.

Recommended **priority** (not a dependency):

1. **#2410** — musl 8 MiB thread stacks. A public, unauthenticated GET
can SIGSEGV the whole server on the shipped image, so merge this first.
2. **#2408** — never lose rows during startup / a failed poll.
3. **#2407** — explicit `observers.idx` primary key.
4. **#2409** — content-hash migration moved into the ingestor.

After these land, #2396 (Postgres) will be rebased onto master.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 23:31:45 +01:00
n30nexandn30nex 11d25f9f7d docs(api): describe configured areas and geometry (#2401)
Partial fix for #1670.

Document the two public area configuration endpoints with reusable array
response schemas. The descriptions distinguish labeled summaries from
geometry entries, specify latitude/longitude order, and explain omitted
fields and preserved zero bounds. Known route-description gaps decrease
from 15 to 13.

This changes generated API metadata only. Handler behavior,
authentication, configuration and packet processing are unchanged; no
new customizer settings or dependencies.

Validation:

- Tests-only RED `8f27f8ff0` builds and [fails the intended
documentation
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43828097551/job/133166654496).
- GREEN `5966b8021` passes the byte-unchanged tests; [final CI
passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43830879045).
- Handler fixtures cover absent/empty config, labeled/unlabeled and
underscore keys, whitespace labels, normal/short/empty polygons, partial
bounds and zero values without assuming entry order.
- All 26 focused checks pass on final GREEN. Full native server/ingestor
suites passed at `e19c49e26` (158.088s/160.279s); the final amendment
only clarifies four descriptions, and final CI includes both suites.
- Parent browser validation rendered both operations in Swagger UI and
executed their actual public GETs, checking coordinate shape, omissions
and zero values.
- Three independent reviews completed with no remaining findings; parent
read every report.

The remaining 13 route-description gaps and broader parameter/schema
work stay open under #1670.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-17 23:31:33 +01:00
n30nexandn30nex 86448911a1 docs(api): document known channels catalogue (#2411)
Partial fix for #1670.

Document the existing public `GET /api/known-channels` endpoint with a
concrete response schema and its optional catalogue-region filter. The
descriptions cover missing-cache sentinels, omitted fields, verbatim
upstream strings, unspecified entry order, and the actual 30/300-second
cache lifetimes. Catalogue entries do not establish observed traffic or
successful decryption. Known route-description gaps decrease from 15 to
14 on this base.

This changes generated API metadata only. Handler, parser, cache,
authentication and configuration behavior are unchanged. No new
dependencies or customizer settings.

Validation:

- Tests-only RED `fadc0fb5e` compiles and [fails the
missing-documentation/schema
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44058546803/job/133954154417);
all ten registered-handler cases already pass.
- GREEN `5ed158aab` passes the byte-unchanged tests. Fixtures cover
nil/unpopulated/empty caches, case and whitespace handling, arbitrary
and unknown regions, raw metadata, omissions, public access and headers.
- All 21 focused tests pass. Full native server and ingestor suites pass
in 287.788s and 245.003s respectively.
- Parent verified Swagger rendering and five real public queries against
a cached catalogue while its source was offline, plus the disabled-cache
response. [Exact-head GREEN
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44065924428)
and three independent reviews pass with no remaining findings.

The remaining endpoints and broader parameter/schema work stay open
under #1670. This change is independent of the area-configuration
documentation in #2401.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-17 23:31:21 +01:00
11eb72f578 feat(planner): Coverage tab, reach of a candidate repeater site (#2406)
The repeater planner (#2391) gets a fourth tab, **Coverage**: put a
marker where you are thinking of placing a repeater, set its antenna
height, and see on the map who would hear it. It answers the siting
question "what does a repeater reach if we put it here", with two
layers: handheld users and other repeaters.

## What it shows

- **Users layer:** the chance that a handheld at 1.5 m hears the site,
in 4 bands (10-25%, 25-50%, 50-75%, 75% and up; below 10% is not
painted). The chance comes from a distance curve `P(d) = 1 / (1 +
exp((ln d - ln d50) / slope))`, fitted on drive-test receptions of 41
Belgian repeaters (d50 2.7 km, slope 0.30). Terrain only lowers a cell
by 2 bands where a 1.5 m receiver sits deeper than 1 first-Fresnel
radius under the terrain (`cmd/server/planner_viewshed.go:19-21`).
Antenna height does not change the curve: in the drive tests, handheld
reach barely changed between 1, 7 and 15 m. The panel says so in one
line.
- **Repeaters layer:** first-Fresnel clearance from the site to every
cell, for a receiver at "Repeaters at" height (default
`planner.existingAntennaM`). 3 classes: line of sight (clearance 0% or
more), slightly blocked (-20% to 0%), blocked (not painted). Every known
repeater within range gets its link computed with the existing planner
link code, a dot in its class colour, a line for line of sight and
slightly blocked, and an entry in a list.
- **Compute button:** changing site, height or range moves the marker
and updates the URL hash right away, and dims an existing result with an
"Outdated" note. Only "Compute coverage" starts a job. A shared link
that carries a site computes once on open. The planner has 1 heavy-job
slot shared with Area and allows 6 jobs per minute per IP, so
recomputing on every change would burn both.

State is in the hash:
`#/tools/planner?mode=coverage&site=51.06,4.41&h=10&rx=5&range=30&layer=repeaters`.

This builds on the repeater planner @zfouts wrote in #2391: the terrain
tiles, the first-Fresnel analysis, the link code, the job protocol and
the Area module pattern are all reused here. Thanks for that work, it
made this tab a small addition instead of a new subsystem.

## Why the Repeaters classes are not the route LOS verdicts

The route verdicts (clear 100%+, clear60 60%+, marginal 0%+, blocked)
almost never show on Shuttle Radar Topography Mission (SRTM) terrain at
mast heights. At 10 m / 5 m, 30 km around Mechelen, 109 of 70,980 cells
reach clear60 or clear. SRTM includes trees and roofs, so the line
between 2 masts rarely clears 60% of the Fresnel zone.

I checked the clearance against one Belgian repeater (28 m, receivers at
5 m) and the repeaters live data shows it actually hears:

| Distance | Heard, clearance >= 0 | Heard, clearance < 0 |
|---|---|---|
| 0-10 km | 9 of 11 | none in range |
| 10-20 km | 3 of 4 | 1 of 8 |
| 20-30 km | 0 of 6 | 3 of 35 |
| 30-60 km | 1 of 2 | 0 of 84 |

Every link heard within 16 km had a clearance between 2% and 54%, so
"marginal" is where working links live. The map classes follow that
(`repeaterClass`, `cmd/server/planner_viewshed.go:75-89`). Chain, Relay
and Area keep the shared verdicts; `losVerdict` is unchanged. This rests
on 17 heard links from 1 repeater, which is why the -20% band is narrow
and the panel says line of sight stops being enough beyond about 20 km.

## Server

- `GET /api/planner/viewshed?site=&h=&rx=&range=`
(`cmd/server/planner_api.go:603`, route `cmd/server/routes.go:393`).
Same job protocol, result cache, rate limits and opt-in as the other
planner endpoints: 404 while `planner.enabled` is false.
- One byte per cell: low nibble Users class 1-5, high nibble Repeaters
class 1-3, 0 for no data. The grid is about 300 x 300 at every range:
0.2 km cells up to 30 km, `range/150` above (`planner_viewshed.go:47`).
- Links skip rooms, stale repeaters and a repeater within 100 m of the
site (that is the site itself). They carry no `elev_m` profile.
- `truncated` is true only when the 300-node cap may have dropped a
repeater within range (`viewshedTruncated`, `planner_api.go:694`).
- Tile cap: `Viewshed` in the planner tile limits, sized from the
largest offered range (60 km).
- New config: `planner.userReach.{d50Km,slope}` with a `_comment` in
`config.example.json`. Bounds 0.2-50 km and 0.05-2.
- No writes, no new `map[string]interface{}`; response types
`PlannerViewshedResponse/Params/Link` in `planner_types.go`.

## Frontend

- `public/planner-coverage.js` + `.css`, mounted like `planner-area.js`.
`planner.js` gets a `MODULES` table (`public/planner.js:31`) so Area and
Coverage share one enter/leave path.
- Canvas raster in an `L.imageOverlay`; the layer switch repaints
locally without a new request. Requests go through `fetchJob` / `CSApi`.
- Colours are CSS variables `--pl-cov-user-2..5` and `--pl-cov-rep-1..3`
in `:root` and both dark blocks of `style.css`; the Repeaters ones point
at `--status-*`, so a theme that changes those changes the map.

## Performance

The job is opt-in and runs in the existing planner job slot, never in a
hot path. Measured on real SRTM tiles, local build, 304 Belgian
repeaters:

| Site | Range | Mast | Job time |
|---|---|---|---|
| Maasvallei (hilly) | 30 km | 10 m | 2.07 s |
| Maasvallei | 30 km | 30 m | 2.57 s |
| Maasvallei | 60 km | 10 m | 3.98 s |
| Mechelen (flat) | 30 km | 10 m | 2.61 s |

On a staging instance (ARM64, cold tile cache) a 30 km job took 6.8 s
including the tile downloads. Response size at 60 km with 115 links:
239,942 bytes with link elevation profiles, 149,984 without (what this
PR ships). `BenchmarkViewshedMaxLeg` covers the 60 km worst case on a
synthetic terrain view.

## Verification

- Go: 26 viewshed tests and benchmarks in `planner_viewshed_test.go`,
`planner_api_test.go`, `planner_config_test.go` and
`planner_tiles_test.go`, among them: flat terrain equals the plain curve
and `analyzeLOS`, a ridge shadows only its far side, missing terrain is
"no data", the `truncated` rule (proven to fail on the old rule), links
without `elev_m`, the repeater at the site left out. `go test ./...` in
`cmd/server` passes.
- Frontend: `tests/unit/test-planner-coverage.js` (23 tests, in
`test-all.sh`): grid decode, class-to-colour mapping, unpainted classes,
hash round trip, no fetch on parameter change, 1 fetch per click,
deep-link autorun once, stale responses ignored, link classes at the
boundaries, escaped repeater names.
- E2E: 5 Coverage cases in `tests/e2e/test-planner-e2e.js` (stubbed
viewshed): click and compute, layer switch without a request, outdated
then recompute, deep link, leaving the tab removes the overlay. Planner
E2E 25/25 locally.
- eslint 0 errors. Browser: light and dark theme, Users and Repeaters on
flat, hilly and coastal sites, deep links, no console errors; also on a
staging instance with live data.

## Not in this PR

- The `--pl-cov-*` colours are not in the customizer's `THEME_CSS_MAP`
yet. They follow `--status-*` for the Repeaters layer; the Users colours
need their own entries (AGENTS.md rule 8, later milestone).
- In Flanders the Users layer is close to a circle: 0.3% of cells differ
from the plain curve around Mechelen, 42% in the Maasvallei, 64% in the
Ardennes. The terrain correction is untested in hilly areas, the drive
tests had no repeater there.
- The Repeaters calibration uses 1 repeater. Other operators with drive
tests or neighbour data can tighten it.
- A link's class comes from `clearance_pct`, rounded to 0.1%, while the
grid uses the raw clearance: within 0.05% of a bound a link can land 1
class off the cell under it. List, line and dot always agree with each
other.
- An OS dark-mode switch without the theme toggle does not repaint the
overlay until the next compute, the same as the planner's tile layer.

---------

Co-authored-by: zfouts <zachary.fouts@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 23:19:49 +01:00
efiten af732d18aa fix(repeater-info): log bulk read errors, cap long notes (#2416)
Follow-up to #2415, see #2414.
2026-11-17 12:56:27 +01:00
efiten 0cf22cdd79 feat: repeater info, claims and community-maintained details (#2415)
Closes #2414
2026-11-17 11:40:05 +01:00
4ebb5ccc11 feat: radio line-of-sight on packet routes + repeater planner (#2391)
Both features are OPT-IN ("los" and "planner" config blocks, default
off); with them off nothing is fetched and payloads are unchanged.

Route view
- Multi-path picker discoverability: on-map "N hops shown · tap to see
all" badge, picker pulse (reduced-motion safe), one-time tip, row hover
arrow; "no GPS" hops counted apart from "unresolved".
- Line of sight for every leg of a packet route: clear / marginal /
blocked chips (first-Fresnel clearance, k=4/3), obstruction markers on
the map, docked terrain-profile panel with antenna/frequency inputs and
a &los= deep link. Backed by GET /api/elevation/profile, an Open-Meteo
proxy that only accepts known node positions, with weighted budgets,
back-off and a per-IP limit.

Repeater planner (#/tools/planner)
- Chain: LOS on every leg between any number of points or nodes.
- Relay: best single relay for A→B among known repeaters and open
terrain.
- Area: suggested new repeater sites that best join the existing
repeater backbone in a box up to 100×100 km, picked greedily to merge
clusters, then remove single points of failure, then reach unlinked
repeaters.
- Terrain: SRTM 1° tiles from AWS Open Data, in memory only (~90 m,
bounded LRU); the server stays read-only. Exact-length gunzip, fetch
budget with back-off, missing terrain reported rather than read as sea
level, fuzzed pixel maths.
- GET /api/planner/{profile,relay,coverage}: quantised cache keys, 202 +
progress then cached 200, separate compute slots, per-IP and global
limits, failed jobs remembered, unpolled jobs cancelled. Nodes pass the
same blacklist / hidden-name / geo_filter rules as /api/nodes. The Go
LOS maths matches public/los.js on shared test vectors.

Also: the per-IP client limiter is extracted from elevation.go into
client_limit.go (behaviour-preserving) so both features share it.

Perf (real SRTM): coverage cold 1.5-5.6 s (tile download), warm 10 ms;
worst real compute 545 ms; peak RSS 91 MiB; live /api/nodes and
/api/packets p95 at most 1.6x while a job computes (GOMAXPROCS=2).

Tests: Go suites for elevation, terrain (fuzz), LOS vectors, graph
(brute-force property test), search and planner API; frontend unit
suites in test-all.sh; Playwright e2e for route LOS and the planner.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-11-16 16:15:36 +01:00
Paul PicazoandClaude Opus 5.5 e7fd48a054 feat(frontend): serve the UI from a web container or a static host, three deployment modes (#2383)
## What this does

The frontend half of serving the UI apart from the server. The server
half was #2362. It gives CoreScope three deployment modes. The server
container is the same image in all three, and it still serves
everything, the classic UI included.

| Mode | The UI is served by | API | Server CORS setting | Logins |
|---|---|---|---|---|
| All-in-one container (today) | the server container | same origin |
none | yes |
| Split containers (new) | a web container that forwards the API | same
origin | none | yes |
| Static host or CDN (new) | Cloudflare Pages, a CDN or any web server |
another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI |

### Commit 1: the UI as static files, for any host

- **One place for API URLs:** `public/deployment.js` says where the API
is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket
URL. A lint rule keeps raw `/api` fetches out of the other files. The
embedded UI keeps `apiBase: ''`, so it behaves as before, except that
its WebSocket now uses `/ws` instead of `/`.
- **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`,
which is `public/` byte for byte except for the cache buster in
`index.html` and `deployment.js`.
- It is `dist/web` rather than `dist/` because the Go binaries are built
into `dist/` and `make clean` removes that folder.
- `API_BASE=same-origin` is for a host that forwards `/api` and `/ws`
itself, as the web container does.
- **Anonymous build (`AUTH=false`):** for another origin. It has no
account features: login, account, admin and other writes link to the
same page on the classic UI. It sends no cookies, and its three JSON
POSTs go as `text/plain`, so there is no preflight.
- **Error notices:** a blocked API answer or an HTML answer each show a
notice.
- **Channel badge:** `beta`, `next` and `dev` builds show a badge that
opens a pre-filled issue, plus an "Open in classic" link.
- **Runbook:** `deploy/static/README.md` covers Cloudflare Pages,
previews, server CORS and rollback.

### Commit 2: the web container and the three modes

- **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It
packages the UI with `build.sh` when the image is built, serves it, and
forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`).
`SITE_ADDRESS` set to a domain name turns on automatic HTTPS.
- **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules
as `deploy/static/_headers`, and compresses responses with zstd or gzip.
- **`docker-compose.split.yml`:** the stock server image (its own Caddy
off) plus the web container:
`CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f
docker-compose.split.yml up -d --build`.
- `CORESCOPE_IMAGE` is required and must match the checkout's version,
so a release server is never paired with a newer UI by default.
- The server's built-in broker, which accepts anonymous clients, is
published on `127.0.0.1` unless `MQTT_BIND` says otherwise.
- **Docs:** `docs/deployment.md` has a new "Deployment modes" section,
and `DEPLOY.md` points to it.

### Keeping new code on `CSApi`

- **Lint rule:** `.eslintrc.json` now rejects every direct network call
in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`,
`sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in
`href`, `src` and `action` (assigned, set with `setAttribute` or written
into markup), `window.open`, `location.assign`, `location.replace` and
`location =`. `api-base.js` is the one file exempt.
- **HTML pages:** `eslint public/*.js` never sees the inline scripts and
markup of `public/*.html`, so `test-lint-network-calls.js` applies the
same rule to them, at their page lines. `area-map.html`'s two fetches
carry an `eslint-disable` with the reason: its Server field can name
another server.
- **Why it changed:** the earlier rule only caught a literal `/api…`
passed straight to `fetch`.
- **Check:** CI already fails on lint errors, and
`tests/e2e/test-lint-network-calls.js` pins exactly what the rule
reports: 25 bad statements reported, 15 good ones not, `api-base.js`
exempt.
- **AGENTS.md:** a new "API calls" section covers:
  - which helper to use for each need;
  - the lint rule;
  - how anonymous builds gate login-only features;
- loading `api-base.js` in vm and jsdom tests (the region quick-picks
suite missed that, and failed shard 1 on the first CI run).

### Review follow-ups

- **Account tokens:** the channel badge's pre-filled issue drops the
query on `#/account/…` routes, so mail-link tokens never reach a GitHub
URL. "Open in classic" keeps the full route, since the classic UI needs
the token.
- **HTML answers:** on a packaged UI (static host, web container),
`api()` rejects an HTML answer before status and retry handling. An HTML
404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server
returned an unexpected response" after one request. It used to give a
generic error, or six "warming up" retries for a 503. JSON errors and
the server's JSON 503 warm-up behave as before. The embedded UI keeps
its old handling (`CSApi.embedded`).
- **Logins:** `docs/deployment.md`, `Dockerfile.web` and
`docker-compose.split.yml` say `userManagement.publicBaseUrl` must be
the web container's public origin.

### Audit fixes

- **No broken logins build:** `build.sh` refuses `AUTH=true` with a
cross-origin `API_BASE`. The server allows no credentialed CORS, so
every API request of such a build was blocked; in Chromium all 12 were,
and the pages stayed empty.
- **No accidental deletes:** `build.sh` refuses an `OUT` that is neither
empty nor a previous build. It used to wipe whatever folder or file
`OUT` named.
- **Lint:** the rule now also reports `/api/` paths in `href`, `src` and
`action` attributes in single-quoted strings and template literals.
`tests/e2e/test-lint-network-calls.js` pins exactly what it reports.
- **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published
on localhost only.
- **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since
v3.16.0 and v3.15.0 predate the static build. It also names the server
version the static build needs: before #2362 the warm-up headers stay
hidden and wildcard previews are refused.

### Audit notes, fixed

- **Cache busting on every page:** the tool pages (`area-map.html`,
`geofilter-builder.html`) load their scripts with `?v=__BUST__`. The
server and `build.sh` replace it in every top-level page, and `_headers`
and the Caddyfile make every page revalidate. Behind Caddy, a returning
visitor could otherwise get a new tool page with an old `api-base.js`.
- **HTTP/3:** the split compose publishes 443 over UDP too, for the
HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with
Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`.
- **WebSocket on `/`:** a Go test pins that an upgrade on `/` still
works, for UIs cached from before `/ws`.

### Second review

- **Trusted proxies:** the docs said to trust the compose subnet, and
also to publish port 3000 if wanted. Together they let a client spoof
its address: the subnet holds the gateway, where a client reaching a
published 3000 through Docker's proxy arrives from. The compose file now
fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web
container's address (`172.30.83.200`), and the docs say to trust that
address alone.
- **Classic UI unchanged:** the HTML-answer notice above applies to
packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as
before, and an unknown `/api` path fails to parse as before, with no
notice.
- **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md
says what the rule cannot see: a URL assembled elsewhere and passed in a
variable.
- **AGENTS.md:** `build.sh` is described as packaging (a copy,
`deployment.js` and `__BUST__`, never a code transform), and rule 3 says
`__BUST__` is filled in every top-level page.
- **CI builds the web image:** the image-check job (no publish) builds
`Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`,
the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It
also validates the split compose file.
- **Cross-origin logins:** a hand-edited `deployment.js` with `auth:
true` and another origin now logs a warning naming the setting, instead
of failing with CORS errors alone.

## Upgrade notes

- **WebSocket path:** the UI's live feed connects to `/ws` instead of
`/`. The server has answered `/ws` since #1793 and still upgrades `/`,
so browsers holding the previous UI keep working.
- A reverse proxy that forwards WebSocket upgrades only for an exact `/`
must forward `/ws` too.
- The nginx example in `docs/deployment.md` (`location /`) already
covers it.
- **Static UI from another origin:**
- The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for
the warm-up banner and wildcard preview origins.
- `build.sh` refuses `AUTH=true` there, since the server allows no
credentialed CORS.
- **Split containers:**
  - `CORESCOPE_IMAGE` is required.
- With user accounts on, `userManagement.publicBaseUrl` must be the web
container's public origin.
- The built-in MQTT broker is published on `127.0.0.1` unless
`MQTT_BIND` says otherwise.
- The compose network is `172.30.83.0/24`, with the web container at
`.200`. If Docker reports that the pool overlaps another network, set
`SPLIT_NET`.

## Testing

- `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with
new tests for busting every page and the WebSocket upgrade on `/`.
- `sh test-all.sh` passes. New unit suites:
- `test-static-build.js`: the build and its input checks, `same-origin`
included, cross-origin logins refused, and an unrelated `OUT` left
intact.
- `test-web-container.js`: the Dockerfile, Caddyfile and compose file
agree on packaging, forwarded paths, cache rules and the server port;
the server image has no default and MQTT stays on localhost.
  - The `api-base`, anonymous mode, badge and `dev-static` suites.
- `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes
47/47, the four pages included; it needs eslint, so CI runs it on E2E
shard 1.
- **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes
11/11. That is the static build on one origin against an API-only server
on another, which CI runs on E2E shard 1.
- **Split containers mode:**
- The image builds, and `caddy validate` passes (Caddy 2.11.4). The new
CI step passes locally, and fails when the Caddyfile stops forwarding
`/api`.
- Trusted proxies, with the stack up: the web container gets
`172.30.83.200`. With only that address trusted, a spoofed
`X-Forwarded-For` sent straight to a published 3000 is ignored, and one
sent through Caddy is replaced with the client's address. With the
subnet trusted, as the docs used to say, the spoofed address was
believed: a deny entry for it rejected the upgrade.
- With `docker-compose.split.yml` on the E2E fixture, seeded the way CI
seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both
through the web container and against the server directly.
  - Headers, checked with curl:
    - `no-cache` on `/`, `/index.html` and `/deployment.js`;
    - zstd on static assets;
    - `/api/*` proxied (`Via: 1.1 Caddy`);
    - a `101` WebSocket upgrade on `/ws`.
  - In Chromium:
- every API request and the WebSocket stay on the web container's
origin;
    - there are no page errors and no horizontal overflow at 390 px;
    - no badge shows on the `static` channel.
- Logins through the web container: with user management on and
`userManagement.publicBaseUrl` set to the web container's origin, a
Chromium run passed every step, all on the web container's origin with
no page errors:
- registration, and activation from the mail link (which uses
`publicBaseUrl`);
    - the session surviving a reload;
    - a CSRF-protected `PATCH /api/account`;
    - logout, then login with the form.
- The same login with the backend's origin as `Origin` is refused with
403. The server checks `Origin` against `publicBaseUrl`, not `Host`,
which is why the docs ask for it.

## Performance

There are no hot-path changes. API calls go through `CSApi.url()`, a
string concatenation. In the web container, Caddy serves files from disk
and compresses them; API responses pass through with one proxy hop, the
same as the all-in-one image's built-in Caddy.

## Notes

- **Per-IP limits:** behind the web container, as behind the all-in-one
image's Caddy, requests reach the server from the proxy's address. The
per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`,
the web container's fixed address. The docs say so.
- **`depends_on`:** the web container starts with the server, not after
its healthcheck, like the all-in-one image's Caddy. Gating on health
would make every `up` wait for the first check (30 s) and fail outright
when a large database loads for longer than the healthcheck allows;
until the server listens, Caddy answers `/api` with 502.
- **Not in this PR:**
- publishing a `corescope-web` image (CI builds and smokes it, without
publishing);
- Cross-origin login—including a static frontend and API on sibling
custom domains—is outside this PR. In the deployments supported by this
PR, account support requires the frontend and API to share one browser
origin, either served together or through a reverse proxy.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:56:09 +01:00
n30nexandn30nex d80ce90976 fix(store): release evicted packet references from secondary indexes (#2327)
Addresses #2324 for upstream SQLite; PostgreSQL qualification remains
separate.

Eviction releases actual node memberships, pointer tails and cache roots
without SQL reconstruction. Borrowed distance snapshots and bounded
hash-migration ownership remain protected.

Review correction `0b1049b5` skips each distance-array copy when no
record is evicted from that array. When affected, it clones once and
filters the suffix beginning at the first match, preserving borrowed
snapshots and the surviving prefix. Work remains O(distance records); no
new locks, reader bookkeeping, settings or general speedup claim.

Tests-only `16509577` reaches [three real assertion failures in
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43200112374/job/131121073845).
GREEN leaves those tests unchanged. Cases cover
neither/hops-only/paths-only impact, repeated removals, live prefixes,
tail removal and snapshot contents. Parent independently reproduced RED
and reran GREEN.

Full native server/ingestor suites pass. Targeted checks: 167 passed;
race: 165 passed, with two existing timing skips that pass in the
ordinary run. Current master `a0b4e6cf` is normally merged; original
RED/GREEN history is preserved.

Historical November 10 measurements: synthetic retained heap 98.00→27.14
MiB with equal live data; path-update cost at 30k hops/10k paths
0.591→3.675ms and 2,824→5,368,584 B/op. The older ~30ms extra eviction
lock time at 240k transmissions is unremeasured. Large-window
qualification/optimization: #2393. Invalidated migration progress:
#2394; no SQL migration redesign here.

Browser verified: populated matching revision (500 transmissions/502
observations), 3/3 interactions. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43202356164)
and three independent reviews pass. No settings changes.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-15 12:24:43 +01:00
0e23c24847 fix(scope-stats): report total and split unscoped by route (#2364) (#2368)
Fixes #2364.

## Problem

Since #1838, `GetScopeStats` folds every non-transport transmission
(FLOOD 1, DIRECT 2) into `summary.unscoped`, while `transportTotal` and
`scoped` still cover transport routes (0, 3) only. The documented
`transportTotal = scoped + unscoped` no longer holds, and a client
computing an unscoped share against `transportTotal` gets >100% (the
reporter shows `transportTotal 7620`, `unscoped 14320`).

## Fix — additive, no existing field changes value

`summary` gains:

| field | meaning |
|---|---|
| `total` | all observed transmissions, routes 0–3 (= `scoped +
unscoped`) |
| `unscopedTransport` | route 0/3 with Code1 = 0000 |
| `unscopedFlood` | route 1 — unscoped by protocol |
| `unscopedDirect` | route 2 — unscoped by protocol |

Invariants (now asserted in `TestGetScopeStats`):

```
transportTotal = scoped + unscopedTransport
unscoped       = unscopedTransport + unscopedFlood + unscopedDirect
total          = scoped + unscoped
```

The non-transport count query keeps its single scan; it just splits
route 1 vs 2 with `CASE`.

On the design question in the issue: rather than changing what
`unscoped` means (the Scopes tab and #1842 rely on it), the split lets a
rollout view use `unscopedFlood` directly: repeaters apply region
rejection to flood routes only (`isRouteFlood()` in
`examples/simple_repeater/MyMesh.cpp`), so DIRECT traffic is not subject
to flood-region filtering. `docs/api-spec.md` now says so, spells out
the route mapping, and notes that `timeSeries` is transport-only (its
`unscoped` matches `unscopedTransport`).

## Frontend

Scopes tab uses `summary.total` when present and shows `N transport, N
flood, N direct` on the Unscoped card; falls back unchanged for older
servers. The cards are built by `scopeSummaryCardsHtml(summary)`
(exposed as `window._analyticsScopeSummaryCardsHtml`).

## Tested

- `go vet ./...` (cmd/server) clean, `gofmt -l` clean
- `go test -run ScopeStats ./...` — pass (incl. extended
`TestGetScopeStats` and `/api/scope-stats` route test)
- `node --check public/analytics.js`;
`tests/unit/test-issue-1375-scope-stats-fetch.js`,
`test-issue-1979-scope-adverts-by-role.js` pass
- new `tests/unit/test-issue-2364-scope-summary-cards.js` (vm sandbox,
11 assertions: shares of `total`, route split, fallback for servers
without `total`/split, partial split, empty window); full `test-all.sh`
green

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-11-15 12:24:27 +01:00
efiten 873e8b0e16 feat(rx-coverage): QR code and link to the operator's CoreDrive RX deployment (#2389)
## What

An operator who runs a CoreDrive RX deployment (the web and Android
companion app that feeds `clientRxCoverage`) can now point visitors of
the Coverage page at it.

- New optional config key `clientRxCoverage.appUrl`.
`/api/config/client` sends it as `clientRxAppUrl`, only while
`clientRxCoverage.enabled` is true and the value is an absolute `https`
URL with a host name. Any other value is left out and logged once at
startup with a `[config]` line.
- `#/rx-coverage` shows a panel under the subtitle when the field is
present: a QR code of exactly that URL (vendored `qrcode.js`, same call
as the node detail QR), the URL as a link, and one line on scanning it
in the app or opening it on a phone. Without the field there is no
panel. The page still needs no login.
- The subtitle named the app "CoreScope-RX" and linked to
`github.com/efiten/corescope-rx`, the app's old repository name. It now
says "CoreDrive RX" and links to `github.com/efiten/coredrive-rx`, or
carries no link when the panel already shows the deployment.

## Why

The CoreDrive RX Android app is not tied to one server: on first start
it asks for the address of a deployment. Scanning a QR code on the
Coverage page is the shortest way for a visitor to get there, including
people without an account.

Disclosure: CoreDrive RX (`efiten/coredrive-rx`) is my own project.

## Changes

- `cmd/server`: `config.go` (`validClientRxAppURL`, startup warning),
`types.go`, `routes.go`, `main.go`, `openapi.go`.
- `public/rx-coverage.js` (`appPanelHtml`, `appLinkHtml`),
`public/roles.js` (`window.MC_CLIENT_RX_APP_URL`),
`public/node-reach-coverage.css` (existing CSS variables only).
- Docs: `config.example.json`, `docs/client-rx-coverage.md`,
`docs/api-spec.md`.

## Tests

- 3 new Go tests in `cmd/server/client_rx_app_url_test.go`: URL
validation over 15 inputs, the endpoint field in 6 config states, the
startup warning. Full `go test ./cmd/server` passes.
- `tests/unit/test-rx-coverage-app-qr.js` (real `rx-coverage.js` and
`qrcode.js` in a vm): panel present and absent, QR content, HTML
escaping, subtitle link in both cases, old names gone. Added to
`test-all.sh`. All 6 `tests/unit/test-rx-coverage-*.js` pass.

## Not done

- No E2E test.
- Browser-checked on a local server with and without `appUrl` in the
light theme only. The QR keeps black modules on white in both themes,
unlike the node QR, which is inverted in dark mode; dark mode itself was
not checked in a browser.
- `cmd/ingestor` is unchanged; it does not need the URL.
- The same change runs on analyzer.on8ar.eu (a fork build) since
2026-11-10.
2026-11-11 00:25:32 +01:00
Paul PicazoandClaude Opus 5.5 d66a219bcb feat(cors): let a UI on another origin connect, static host previews included (#2362)
## What this does

Server changes so the UI can be served from another origin, such as a
Cloudflare Pages project and its previews, against an existing backend.
It ships before the frontend, which is parked on
[`backup/static-frontend-1a`](https://github.com/ppicazo/CoreScope/tree/backup/static-frontend-1a)
and will come as its own PR.

- **Exposed headers:** allowed origins get
`Access-Control-Expose-Headers: Retry-After, X-CoreScope-Load-Status`.
The UI reads both, and cross-origin they are `null` without this.
- **Wildcard origins:** an entry may be `scheme://*.host[:port]`, such
as `https://*.myproject.pages.dev`. It matches exactly one leftmost
label (each deploy and branch preview) for both CORS and `/ws`, but not
`myproject.pages.dev` itself. The part after `*.` needs at least three
labels, so `*.pages.dev` is refused. Refused or malformed wildcards are
dropped at startup with a log line. That floor is a sanity check, not a
Public Suffix List lookup: `https://*.s3.amazonaws.com` passes and would
admit every S3 website endpoint, so a wildcard belongs only over a host
the operator controls.
- **Case:** exact entries now match case-insensitively for CORS (was
`==`), as they already did for `/ws`.
- **Test only:** pins that the three anonymous POSTs (`/api/decode`,
`/api/packets/observations`, `/api/paths/inspect`) accept a `text/plain`
JSON body, which keeps them free of a preflight.

Credentials stay off, so nothing a listed origin sends rides on cookies.
Beyond GET, a listed origin, and so every deployment under a wildcard
entry, can:

- call those three anonymous POSTs as simple requests. That predates
this PR; they compute an answer and write nothing.
`Access-Control-Allow-Methods: GET, HEAD, OPTIONS` only governs
preflighted requests.
- with `userManagement.enabled`, use the existing bearer tier for
companion linking (`/api/auth/device-token`, `/api/auth/me`,
`/api/auth/logout`, `/api/account/settings`, `/api/account/companions*`)
with POST, PUT and DELETE. Those routes need a device token in
`Authorization`, not a cookie, but a preview deployment gets them too.
On such a server, list a wildcard only if previews may use them.

## Configuring

```yaml
- CORS_ALLOWED_ORIGINS=https://existing-embed.example,https://myproject.pages.dev,https://*.myproject.pages.dev
```

The env var replaces `corsAllowedOrigins`, so keep the origins already
listed there.

## Testing

- `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`. The
new tests were written first and failed first.
- In Chromium, the static build from the backup branch, served on
`pr-1.proj.preview.localhost`, passed its split E2E (9/9, WebSocket
included) against this server.
- From the same setup, the project origin connected, a two-level
subdomain was blocked, and `X-CoreScope-Load-Status` read `ready` where
master gives `null`.

## Performance

These run per request in the CORS middleware and the `/ws` handshake,
and they allocate nothing. Same-origin requests skip them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 17:46:17 +01:00
n30nexandn30nex 9341ec744b fix(observers): honour node visibility in neighbour reports (#2357)
Fixes #2356.

Observer neighbour reports now apply the existing node blacklist and
hidden-name-prefix rules before returning displayed identities. Names
are already batch-loaded, so the fix filters the request-owned slice
without another SQL query. It preserves reported totals, timestamps,
stored data, public/unknown neighbours and the existing reporter-only
observer-blacklist semantics. An all-hidden list remains `[]`.

The work is bounded by the existing 500-entry ingestion cap plus the
existing visibility checks; no new configuration, frontend change or
database write is introduced. The filter reuses the existing report
slice.

Validation:

- [RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40971452075/job/123789013428)
at `7341b334` compiled and reached the four expected visibility
assertions.
- GREEN `b38e95e6`: focused native checks passed in 6.607 s; parent
rerun passed in 1.865 s.
- Full native server passed in 189.743 s; parent ingestor suite passed
in 399.859 s. Existing neighbour frontend checks passed 7/7.
- Tests use the actual registered router, direct-node 404 controls,
casing/shared-name/unknown-node cases, metadata preservation and an
unchanged stored report.

Local standalone browser replay is unavailable because the
fixture-server launch was blocked. Fresh GREEN CI passed, including
server race checks and browser shards. Independent adversarial,
security-boundary and TDD reviews found no must-fix items. No
live-instance incident or deployment is claimed.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-31 12:01:19 +01:00
n30nexandn30nex f8c3b32633 docs(api): describe readiness responses and ingestor liveness (#2358)
Partial follow-up for #1670.

Documents public `GET /api/healthz`: 503/loading, current in-memory
counts, the fixed legacy backfill compatibility object, and optional
ingestor liveness timestamps. Receipt and handler-entry clocks have
different reset semantics; handler entry does not prove a completed
database write. The legacy object does not track real migration
progress.

The 200 response has a typed schema, including its dynamic source-name
map. Only `/api/healthz` leaves the known-gap list (16 → 15). Handlers,
configuration and frontend behavior are unchanged; formal non-200
response-schema work remains separate.

Validation:

- The new documentation assertion failed before metadata was added;
actual handler-response checks passed.
- Full native server/ingestor suites passed in 257.438 s / 452.613 s.
- After review corrections, focused checks passed in 2.757 s and the
parent rerun in 2.097 s, including exact compatibility values and
retained-receipt/zero-handler timestamps.
- Independent adversarial, correctness and TDD reviews completed; the
parent verified both semantic corrections against actual call sites.
- [Exact-head CI
passed](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40979351261),
including browser shards.

TDD exemption: documentation/API metadata only. Local standalone browser
smoke was blocked; no local-browser or deployment claim is made.
Production untyped-map count is unchanged.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-31 11:59:20 +01:00
af2e96d92a fix(reach): queue a cold request up to 3 s for a build slot before answering 429 (#2249)
Follow-up to #2127, from @dborup's review there. The limiter, most of
its tests and the docs text are dborup's (dborup/CoreScope@195ec4c3),
adapted to master; the commit carries `Co-authored-by: dborup`.

## What changes

`GET /api/nodes/{pubkey}/reach` still runs at most 2 uncached reports at
once. What changes is the next uncached request:

- **Before:** 429 at once, `Retry-After: 5`.
- **After:** it waits up to 3 s for a slot. It gets 429 with
`Retry-After: 2` and a JSON body only if no slot frees in that time.

Unchanged: cached answers never wait, and concurrent requests for the
same node and `days` share one computation and one slot.

One addition beyond dborup's version: the shared computation runs on the
context of the request that started it. If that request disconnects,
while queued or mid-scan, the other requests sharing it that are still
connected now get 429 instead of 500. The 3 s wait made that case more
likely.

## Why

dborup's three points on #2127:

1. An instant 429 fails two tabs, or a quick switch between 7/14/30
days, when the database is not under real pressure.
2. The page cannot recover from it, because `api()` retries only 503.
The panel retry is a separate PR so that this one stays server-only.
3. No test covered the promised property: a cached request still answers
while every slot is held.

Waiting parks a goroutine, not a database connection, so the pool is
protected exactly as before.

## What I checked

- **Tests.** `make test` (all 16 modules), `make vet`, `gofmt`, `sh
test-all.sh` and eslint (0 errors) all pass. The reach tests also ran 10
times under `-race`.
- **New tests.** 14 tests in `node_reach_cold_limit_test.go`: dborup's
10 plus 4 new. `TestNodeReachConcurrentBuildCap` now drives the new
limiter through the production router. Point 3 is
`TestNodeReach_WarmHitSkipsColdLimit`. The 4 new tests are:
  - a queued request is served when a slot frees;
  - a cache filled just before the flight skips the limiter;
- coalesced waiters get 429 when the starter disconnects, queued or
mid-scan (two tests).
- **Mutations.** I wrote 21 from the code before running the tests, and
the tests caught all 21: instant reject, cap +1, release that leaks,
slot taken before the in-flight recheck, busy → 500, Retry-After header
or value, status 503, dropped context mapping, and others.
- **Real binary.** Master `2d09335c` and this branch, run one after the
other on a copy of a 10.5M-observation, 11 GB database. Each burst fires
its reach requests at once, while `/api/nodes?limit=50` and
`/api/packets?limit=50` are polled every 200 ms.

| Burst (distinct uncached keys) | master | this branch |
|---|---|---|
| 3 keys, `days=7`, warm page cache (2 runs) | 2 × 200, 1 × 429 after
0.01 s | 3 × 200; the third answered at 4.1–4.3 s |
| 6 keys, three of them `days=14` (2 runs) | 2 × 200, 4 × 429 after ≤
0.01 s | 2 × 200, 4 × 429 after 3.0 s |
| 24 keys, cold page cache | 2 × 200, 22 × 429 | 2 × 200, 22 × 429 after
3.00 s |
| `/api/nodes` during every burst | median 1 ms, max 53 ms | median 1
ms, max 66 ms |

## Not checked

- **Slow scans.** On a database this size, uncached scans took from
about 9 s to over 60 s, so the 3 s wait only helps when page cache is
warm or the database is smaller. Otherwise the result is the same 429, 3
s later. A 30-day uncached scan hit the 60 s `WriteTimeout` on both
builds; that is the scan #2214's opt-in hop index addresses.
- **Pool contention under a live store.** On this copy the in-memory
store loaded 0 packets, so `/api/nodes` latency does not show contention
under a live load.
- **Platforms.** One machine (linux/amd64). No browser check: the server
change has no UI, and the panel retry comes with its own PR.

**Base.** Rebased on master `7c00a429`, which includes #2214; every test
and mutation above was re-run on the rebased commit. The real-binary
runs used `2d09335c`, just before #2214, on a database without the hop
index. That is the default after #2214 too (`hopIndexDays` unset means
off). With the index on, scans are shorter, so more bursts fit in the 3
s wait; I did not measure that.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


---

## Update after review (commits `c65f4d3b`, `1d2cd99d`, `04e49854`)

efiten asked how many requests may wait at once. With only the 3 s
timeout, the number of waiting distinct keys grew with the request rate.
n30nex wrote a cap (nullrouten0/Baymc-CoreScope#1, tracked in #2264).
Its two commits are now on this branch unchanged, as a fast-forward.

- At most 8 distinct cold requests wait for a slot. Any further one gets
429 with `Retry-After: 2` at once, before a timer is set.
- A waiter gives its place back on every exit: it gets a slot,
disconnects or times out.
- Cached answers and same-key sharing are unchanged. Requests for a key
that is already queued or building wait on that one place and are not
counted. `04e49854` corrects the code comment that said the ceiling was
independent of request rate.

**What I checked on `04e49854`:**
- The two new tests fail on `28da016f` and pass now.
- 12 mutations of the cap, all caught: cap 7 and cap 9; the place never
returned, returned before waiting, returned only on promotion, leaked on
cancel, leaked on timeout; a full queue that blocks, returns the wrong
error or skips the counter; a timer armed before the check; a queue
sized by the slot count.
- `make test`, `make vet`, `make fmt-check`, `sh test-all.sh` and eslint
8.57.1 (0 errors) pass, and the reach tests pass 5 times under `-race`.
On a local merge with current master `660cbbf5`, which includes #2277,
the reach tests pass 3 times under `-race` and the `cmd/server` package
passes.
- Real binary: both heads, alternately, on a copy of a 2.5 GB soak
database, with 24 distinct uncached `days=7` requests fired at once, two
runs each:

| | `28da016f` | `04e49854` |
|---|---|---|
| 200 | 2, 2 | 2, 3 |
| 429 at once | 0, 0 | 14, 14 |
| 429 after 3 s | 22, 22 | 8, 7 |

**Not checked:** a sustained flood, a burst of same-key requests against
the cap, `/api/nodes` latency during these bursts, and platforms other
than linux/amd64. The real-binary table further up was measured before
the cap.

---------

Co-authored-by: nullrouten0 <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: dborup <kontakt@meshview.dk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: n30nex <blushellneon@outlook.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-10-29 22:50:59 +01:00
efitenandClaude Opus 5.5 bb3c8cc001 fix(rx-coverage): My coverage and the observer filter on the noise layer (#2326)
"My coverage" and a selected observer only narrowed the signal layer on
the coverage page. On the noise layer the toggle stayed pressed, but
`/api/rf-noise` was called without `mine` or `rx` and returned every
companion's samples.

## Changes

- **Backend:** `/api/rf-noise` now accepts `rx` and `mine=1`, with the
same session rules as `/api/rx-coverage`: 401 without a session, 403 for
a device token, 404 with user management off. The `rx_pubkey` filter
moved into 1 helper, `appendRxFilter` (`cmd/server/rx_dashboard.go`),
which both queries use. Both parameters are in the OpenAPI entry.
- **Frontend:** `public/rx-coverage.js` fetches both layers through
`fetchLayer`. It builds the URL with the time window, observer and
toggle, and judges the reply with the existing `coverageReply`. The
request sequence (`signalSeq`, now `drawSeq`) covers both layers, so the
noise layer also drops a superseded reply. Before, a slow unfiltered
noise reply could overwrite a newer filtered one.

## Performance

The noise query gets 1 extra `WHERE` clause on `rx_pubkey`. There are no
new requests: the page still makes 1 request per redraw.

## Verification

- New Go tests in `cmd/server/rf_noise_mine_test.go`: `rx` narrows the
result (case-insensitive), `mine=1` returns 401 without a session,
returns empty with nothing linked, returns the linked companion's cells,
returns 403 for a device token, and returns 404 with user management
off. Both failed before the fix.
- `tests/unit/test-rx-coverage-mine.js` is updated for the new
`coverageUrl(path, ...)` signature and checks that both layers fetch
through `fetchLayer`.
- New step in `tests/e2e/test-rx-coverage-noise-e2e.js`: a
`?layer=noise&rx=...` link sends `rx`, and clicking "My coverage"
refetches with `mine=1`. It fails against the current master frontend
and passes with this change.
- Local runs (Windows): `go test` and `go vet` in `cmd/server`,
`test-all.sh`, and the noise, gaps and viewport E2E in Chrome. All
green. I did not run the full Playwright suite locally.

## Not in this PR

- The "nothing received" gap cells still skip `mine=1`, unchanged: the
track query cannot narrow to a set of companions yet.
- The time window (`days`) already worked on the noise layer, so it is
not touched.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:39:02 +01:00
TeTeHackoandClaude Opus 5.5 417c8dd00a feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) (#2300)
Closes #2270, closes #2271.

The ingestor took only the confirmed region scopes out of an observer's
`/neighbors` report (#1865) and dropped the rest. Nothing could
therefore say when an observer last reported, or whom it heard.

## Ingestor

`observer_neighbor_reports` (created in `dbschema.Apply`) keeps the
latest report of each observer, one row per observer, replaced by every
live report.

The report is unauthenticated, so `neighbors[]` is stored sanitised:

- only 64-hex keys
- `snr` and `heard_secs_ago` finite and in range
- `status` limited to `[a-z_]`
- `scopes` clamped
- at most 500 rows

`total_neighbors` is believed only when it is not below the number of
rows that arrived.

A **retained replay** only fills a missing row, and it is dated by the
report's own `timestamp`. After an ingestor restart, a long-dead
observer therefore does not look as if it just reported.

## API

The table is optional for the server (no `AssertReady` change); without
it the fields are simply absent.

- `GET /api/observers`: `neighbors_report_at` and `neighbors_count`,
from one batch query in the existing cached builder.
- `GET /api/observers/{id}`: also `neighbor_report`, which is
`{received_at, reported_at, neighbor_count, queried_neighbors,
self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr,
heard_secs_ago, status, scopes}]}`. Names and roles come from the node
table in one query.

## Frontend

`public/observer-neighbors.js` is shared by both pages and escapes
everything.

- **#2270, observers list:** a sortable **Neighbors** column, e.g. `3h
ago · 12`, or `—` when the observer never reported. The same value
appears in the slide-over.
- **#2270, observer page:** **Neighbors Report** and **Neighbors** stat
cards.
- **#2271, observer page:** a **Neighbors** table listing each neighbor
by name (a link to the node) or key prefix, with SNR, heard, scope query
status and scopes. The heading says when the list is shorter than the
count, e.g. `(2 of 3 listed)`.

## Tests

- **ingestor:** sanitising, the cap, live replace, the retained rules
(fill only, dated by the report, undated ignored) and the MQTT topic end
to end. The existing #1865 scope tests are unchanged and still pass.
- **server:** the list summary, the detail with names, and both
endpoints without the table.
- **frontend:**
  - unit tests of the helpers
- an XSS case for every rendered field; I checked that it fails when
escaping is removed
- `test-observers-headings` now expects the new column after `Last
Packet`
- **E2E:** `test-observer-neighbors-2270-e2e.js` intercepts the observer
endpoints, since the fixture has no reports. It is listed in
`scripts/non-unit-tests.json` and on shard 2.

Gates that pass:

- `go test -race` in `cmd/server`, `cmd/ingestor` and
`internal/dbschema`
- `sh test-all.sh`
- `eslint@8`, with 0 errors
- the XSS diff gate
- these E2E suites: the new one,
`test-issue-1639-observers-sort-e2e.js`,
`test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js`

Performance: the list adds one primary-key-sized query per cache fill,
and the detail adds two small queries per request.

Also fixes #2330 (folded in from #2337, second commit).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:36:53 +01:00
n30nexandn30nex 1ed5e4ab3f fix(compare): exclude observer TX-only packets from overlap (#2346)
Red commit: `39295da8` ([intended CI assertion
failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)).
Green commit: `d1bda496`.

Observer comparison counted an observer's TX-only packets as receptions,
producing false overlap. It now requests `excludeTx=true`: the selected
observer and non-TX direction must match the same observation in memory
and SQLite v2/v3. Normal packet lists and stored TX records remain
unchanged; missing directions retain legacy compatibility.

Refs #667. This completes a comparison slice, not the issue's remaining
observer counters or #671's fleet/SNR feature. Representative display
fields and existing time-filter semantics are unchanged. API/OpenAPI
docs describe the filter and multi-node bypass.

Validation:
- RED: Go tests compiled with 32 intended failures/16 control passes;
unit and browser regressions reproduced false overlap. CI reached the
new unit assertion.
- GREEN: all 48 packet cases pass, plus grouped-cache isolation and
OpenAPI tests. The added cache regression caught a malformed-radio
cache-key collision before correction.
- Full local server suite: 174s; ingestor: 185s; `npm run test:unit`:
pass. Final focused packet/grouped/OpenAPI checks: pass.
- E2E assertion added to
`tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks
pass, including correct 1/2 reception totals and 50% overlap.

Performance: still two capped bulk requests, indexed candidate lookups
retained, one direction check within the existing observation scan/SQL
predicate, no new persistent structures. No configuration/customizer
changes.

Final-head CI passed; independent reviews found no must-fix items.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:34:06 +01:00
n30nexandn30nex c211ef7643 feat(nodes): show relay activity from certain forwarding evidence (#2336)
Fixes #2331 with the additive API option.

The Last Relayed row shows green **actively relaying** only when
`last_certain_relay` falls within the existing
`healthThresholds.relayActiveHours` window. Missing, invalid, future or
expired evidence adds no idle/alive claim. Existing `relay_active`,
attributed timestamps and counts retain their meanings.

The optional node list/detail field uses the existing certain-relay
path: an eligible observed frame, matching hop, unique prefix, no
retired namesake and existing signature checks. Originating an advert
does not count. Forwarding another node's advert can count, even with
zero non-advert attributions. Adverts with an unknown origin cannot
establish forwarding.

Evidence uses transmission `first_seen`, not late observation receipt
time, and follows node-membership eviction. Snapshots take one read lock
over requested keys, O(page size), with no packet scan or new database
query. Work per qualifying hop stays bounded. No new dependency or
setting; configuration documentation and OpenAPI explain the
distinction.

The test-first history includes assertion-failing RED commits and GREEN
fixes. The initial RED's nil-return compile stub is explicitly permitted
by `docs/agents/TDD.md`. Local native server/ingestor suites, focused
race checks, 809 frontend helper assertions and 70 browser cases pass.
Browser coverage includes both themes and full/side/tablet detail views.

[Review regression
RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40421464618/job/121916723339)
proves both malformed-origin failures. All three independent review
roles pass at `d1fe79cb`. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40422655728)
passes on that exact commit, including all three browser shards and the
two-architecture image check.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:33:59 +01:00
n30nex 1e1bfa0255 fix(ingestor): require unique hop prefixes for relay liveness (#2334)
Fixes #2329 (option 1: unique-prefix liveness).

The ingestor could select a likely relay through neighbour adjacency and
refresh its `last_seen` even when its prefix matched two nodes.
Contextual resolution remains available in stored `resolved_path`;
liveness now uses the existing unique-prefix resolver, sharing the
parsed path and prefix-index snapshot. The current flood-frame
eligibility gate remains unchanged. Retired-prefix catalogue parity with
the server is separate.

The native SQLite regression exercises real `InsertTransmission`: the
displayed path still selects the adjacent candidate, neither colliding
relay is touched, and the unique preceding hop is refreshed. Existing
direct/TRACE/flood checks remain intact. [RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40407478987/job/121868990704)
fails the timestamp/touch-count assertions; [GREEN race
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409860054/job/121877671472)
passes at `77be50e8`.

Full local server and ingestor suites, repeated focused race checks and
all 221 frontend suites pass. Existing test assertions are unchanged;
one misleading resolver comment is corrected. Frame/path assumptions
were checked against MeshCore firmware revision `81dbae59`.

Performance: one additional indexed lookup per hop, O(h), with no node
scan or extra database query. No schema, dependency, configuration or
customizer change. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409860054)
passes on `77be50e8`, including the race detector, all browser shards
and architecture builds. Three independent reviews found no blockers;
final repository checks pass. Ready for maintainer review.
2026-10-29 18:33:53 +01:00
n30nex f9b7c66a8b fix(ingestor): open buffered writes before startup packet retention (#2328)
Partial fix for #2325; leave the issue open for the remaining startup
work.

MQTT subscribed early but its buffer could not drain until the entire
packet-retention backlog finished. Required startup work and the
migration barrier still finish first; ingestion now opens before the
initial packet sweep, so buffered writes can use the gaps between the
existing 250-transmission transactions.

The sweep remains synchronous with its count/error logging. No new
worker, shutdown owner, setting, retention window or batch size. The
readiness log reflects the actual gate opening. Earlier non-packet
maintenance, initial vacuum ordering and diagnostics publication during
maintenance remain unchanged.

Native SQLite tests exercise blocked migrations, inter-batch writes,
FIFO/drop accounting, stopped buffers, callback counts, rollback/retry
and main-function wiring. Removing the migration wait fails the
regression. Existing tests are unchanged; GREEN strengthens the new RED
test with a blocked-gate assertion and early-failure cleanup.

[RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40308382930)
fails the intended assertions. [Final-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40311222867)
passes on `f8a4d175`, including Go/JS tests, race checks, all browser
shards and architecture builds. Full local ingestor tests, repeated
focused race checks and local browser/API smoke checks pass. Three
independent reviews found no blockers in this partial scope.

This removes the whole-sweep startup gate; it does not bound an
individual deletion transaction or claim a measured live latency
improvement.
2026-10-29 18:33:47 +01:00
n30nexandn30nex ef17d78909 docs(api): document node resolution and battery history (#2338)
Partial follow-up to #1670; the broader documentation issue stays open.

Documents the existing node-prefix resolution and node battery-history
GET APIs: parameters, normalization and lookup rules, defaults,
ambiguity/visibility, ordered samples, nullable latest values and error
media types. The generator now has concrete success schemas for both.
Error statuses remain documented in prose under its existing convention.

Removes exactly these two entries from the known-gap list: 18 → 16, with
115 of 131 routes described. No handler, query, configuration or
dependency changes.

[RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426035729/job/121931981058)
reaches the missing-documentation assertions. GREEN `521fb3ef` passes
both full native CGO suites and focused OpenAPI/real-handler contract
tests. Parent browser acceptance passed 28 checks through the rendered
Swagger UI and actual endpoints, including defaults, errors and
empty/null history.

All three independent review roles passed. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427081025)
passes on this exact head, including all browser shards and the
two-architecture image check.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:33:41 +01:00
efitenandClaude Opus 5.5 b65412ab83 feat(observer-logins): turn on neighbour reports in the firmware block of the credentials mail (#2349)
The firmware block of the observer credentials mail now ends with `set
mqtt.neighbors on`, so firmware observers send the same `/neighbors`
report that the OpenHop block already turns on ("Publish Neighbours",
#2347). The ingestor reads that report for region scopes (#1865).

## Changes

- `cmd/server/observer_mail.go`: one extra line in the firmware block,
and one sentence in its note. The sentence says the command sends the
neighbour table and region scopes once a day, and that older firmware
does not know it and should skip it.
- `cmd/server/observer_mail_test.go`: the full firmware block assertion
includes the new line, and the slot test checks the line and the note.
- `docs/observer-mqtt-logins.md`: the mail section describes the new
line.

## Tests

- `go test ./...` in `cmd/server`: ok. Skipped
`TestSaveGeoFilterPreservesFileMode`, which needs Unix file modes; this
run was on Windows.
- `TestObserverCredentialsMail*` failed before the change, on the
missing line, and passes after it.

## Not checked

- The command name and the 24-hour default come from #1865. I did not
read the observer firmware source.
- The wording of the error older firmware gives for an unknown command
is not quoted in the mail. The note only says to skip the line.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 06:25:42 +01:00
efitenandClaude Opus 5.5 cadc666bd9 feat(observer-logins): OpenHop block in the credentials mail (#2347)
The observer credentials mail (#2306) had blocks for meshcoretomqtt, the
Gessaman/Dutch-MeshCore firmware, Home Assistant and plain values. An
operator running OpenHop (repeater software with its own MQTT uploader)
could not tell which of these matched its broker form, and picked the
format by trial.

## Changes

- `cmd/server/observer_mail.go`: a fifth block, "4. OpenHop (repeater
software)", before the plain values. It uses the field and option labels
of the OpenHop broker form:
- `Format: Letsmesh`, `Transport: TCP` or `Websockets` (from
`observerLogins.transport`), host, port, username, password, `TLS:
on/off`, `Use JWT Auth: off`, `Retain Status: on`, `Publish Neighbours:
on`.
- The note says Meshcoretomqtt works too, but the format labelled
"openHop" (value `mqtt`) does not: it publishes to
`meshcore/repeater/<name>`, which a personal login may not use.
- Publish Neighbours is on because the ingestor reads the `neighbors`
topic for region scopes (#1865).
- `openapi.go` and `docs/observer-mqtt-logins.md`: "four formats"
becomes five.

## Sources

Read from OpenHop, not tested on a device by me:
- format family and topics:
`openhop_repeater/repeater/data_acquisition/mqtt_handler.py`
(`MC2MQTT_FORMATS = ("meshcoretomqtt", "letsmesh", "waev")`, the `mqtt`
format's `meshcore/repeater/<node_name>` base topic, transport values
`tcp`/`websockets`);
- labels: the built web UI (`Configuration-*.js`: "Add Broker", option
labels `Letsmesh`, `Meshcoretomqtt`, `openHop`, `TCP`, `Websockets`,
"Retain Status", "Publish Neighbours", "Use JWT Auth").

An operator on our instance set it up from a screenshot of that form
with format Letsmesh, TCP, Retain Status and Publish Neighbours on; the
observer publishes, and its `neighbors` report set `configured_scope` on
its node.

## Tests

- `TestObserverCredentialsMailOpenHopBlock`: the block text for TCP
without TLS and for Websockets with TLS, the "openHop" warning, HTML
escaping, and the block order.
- `TestObserverCredentialsMailBlockTitles`: five titles in order.
- `cmd/server` suite green locally (the two Windows-only failures
skipped, as on master).

## Not in this PR

- The websocket path: OpenHop uses paho's default, the doc already tells
operators to match the listener path.
- Anything with the other fields of the `neighbors` report (SNR,
heard_secs_ago, default_scope): we are measuring how many observers send
it first.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:31:43 +01:00
efitenandClaude Opus 5.5 d29957fbdc feat(observer-logins): lock the shared login out of migrated observers (#2348)
Follow-up to #2306 / #2319. With personal observer logins, a migrated
observer still is not safe from the shared login: the hook answers
`ignore` for the shared login, so EMQX's built-in database lets it
publish under any observer's topic, including one whose owner has moved
to a personal login. This PR lets an operator close that per migrated
observer, before the shared login is removed altogether.

## Changes

**users.db v8.** `observer_bindings.confirmed_at`: the moment the
owner's personal login first published under that pubkey.
- An auto-bind (the owner's first publish) is confirmed at once.
- An admin assignment stays unconfirmed until the owner's login
publishes under that pubkey; that publish sets it.
- Reassigning to another user or unbinding clears it; reassigning to the
same owner keeps it.
- Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin`
rows stay NULL.

**Config.** `userManagement.observerLogins.sharedUsernames` (default
empty): the broker's shared login names. A name starting with `obs_` is
refused at startup.

**Authz hook.** For a login in `sharedUsernames`, a publish under
`meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is
confirmed, whatever the number of topic segments (the ingestor
attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey).
Logged as `publish denied: user=<name> observer=<8 hex>: confirmed
owner`. Every other request from a shared login keeps `ignore`, so the
built-in database decides as before. Personal logins are unchanged.

An orphaned binding (owner account deleted) does not count as confirmed.

**Admin.** The observers table of the Observer logins tab gets a
Confirmed column; the bindings JSON gets `confirmedAt`.

**Docs.** `docs/observer-mqtt-logins.md`: "Locking the shared login out
of migrated observers" (what it denies, how to see `confirmedAt`, the
risk below). `config.example.json` has the key.

## Risk for operators

An observer that is confirmed but still runs a second uploader on the
shared login loses that second uploader once its name is in
`sharedUsernames`. The deny log line names the observer.

## Tests

- `internal/users`: v7 to v8 migration on a real v7 database, auto
confirm, admin assignment then owner confirm, non-owner and repeated
confirm, reassign clears, orphan reassign.
- `cmd/server`: the decision table (confirmed, unconfirmed, unbound,
non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty
and `client` iata, lowercase key; owner publish confirms, confirm
failure still allows and retries; reassign, unbind and user removal
clear the flag; the index load against a real store; config validation;
the hook and admin endpoints end to end.
- `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an
auto-bound observer.
- Local: `internal/users`, `cmd/server` (two Windows-only tests skipped,
as on master) and `cmd/ingestor` green. The E2E assertion was not run
locally; CI runs it.

## Not in this PR

- Using `confirmed_at` for verified `/neighbors` reports: a separate PR,
built on #2300.
- Removing the shared login: still an operator step after migration.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:10:41 +01:00
TeTeHackoandClaude Opus 5.5 660cbbf5ff deps: golang.org/x/net v0.61.0, x/crypto v0.58.0, paho.mqtt.golang v1.5.1; runtime alpine:3.24 (#2297)
A Trivy scan of the published image found **19 HIGH and 20 MEDIUM** in
Go dependencies, plus a runtime base past its support window. There is
no Dependabot here, so this collects the bumps in one PR.

| dependency | from | to | why |
|---|---|---|---|
| `golang.org/x/net` (ingestor) | v0.27.0 | v0.61.0 | CVE-2024-45338 and
later advisories, fixed up to v0.60.0 |
| `golang.org/x/crypto` (server, `internal/users`) | v0.31.0 | v0.58.0 |
CVE-2025-22869, CVE-2025-47913 and later; `x/sys` follows to v0.49.0 |
| `github.com/eclipse/paho.mqtt.golang` | v1.5.0 | v1.5.1 |
CVE-2025-10543, an integer overflow in the UTF-8 string encoding |
| runtime image | `alpine:3.20` | `alpine:3.24` | 3.20 is out of
support; its musl 1.2.5-r3 carries CVE-2026-107891 |

The Alpine change covers `Dockerfile`, `Dockerfile.go`, the arm64 smoke
step in `deploy.yml` and the comment in
`docker/test-select-service-user.sh`.

**What changes besides the versions:**

- The new `x/` modules declare `go 1.26.0`, so the `go` directive of
`cmd/server`, `cmd/ingestor` and `internal/users` moves up from 1.22.
The builder image and CI already run 1.27.
- `x/sync` follows paho to v0.17.0.

**What is left:** only `GO-2026-5932`. It flags `x/crypto/openpgp` as
unmaintained; nothing here imports that package and no fix exists.

## Checks

- Every module passes `go mod tidy`, `go build` and `go vet`.
- `go test -race` passes in `internal/users`, `cmd/ingestor` and
`cmd/server`.
- A local image build on alpine 3.24.3:
  - ships mosquitto 2.1.2, caddy v2.11.4 and supervisor 4.3.0
- keeps the caddy and mosquitto accounts at the same uid:gid (100:101,
101:102), so `test-select-service-user.sh` still describes the image
(116 assertions pass)
- comes up healthy, ingests an MQTT status through mosquitto and serves
the frontend

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 21:09:01 +01:00
TeTeHackoandClaude Opus 5.5 77a5f14925 feat(observers): radio setting per observation from the observer's radio history (#1869) (#2292)
Part of #1869.

## Problem

`observers.radio` keeps only the **last** `MHz,kHz,SF,CR` an observer
reported. Once an observer is retuned (say from SF7 to the EU narrow SF8
setting to listen to a neighbouring mesh), every packet it ever heard
reads as heard on the new setting, and there is no way to tell an SF7
packet from an SF8 one.

## What this does

As #1869 proposes, the packet data model is untouched: the setting of an
observation is resolved from its observer and time.

**Ingestor** (all schema in `dbschema.Apply`)

- `observer_radios`: one row per stretch of time an observer, and each
radio of a multi-radio node, stayed on one setting. Every live status
extends the current row and a change starts a new one; retained replays
never extend a row. On the first run it is seeded with every observer's
current radio, which is what the UI already showed for the past.
- Settings are stored canonical through `dbschema.CanonicalRadio`, so
`869.4320068,62.5,7,5` and `869.432,62.50,7,5` are one setting.
- `observation_radios` (`observation_id → rx_radio_id`): multi-radio
openHop nodes tag each packet with `rx_radio_id` and list `radios: [{id,
radio}]` in their status, so their observations resolve per radio.
- The table is sparse: single-radio nodes never send the field, so
`observations` and every query that scans it stay untouched.
  - The row id comes from the observation upsert via `RETURNING id`.
- Rows are pruned with their transmissions, and a purged observer takes
its history along.

**Server** (read-only)

- No `AssertReady` change: an absent table reads as empty and is probed
again later, the same rule as `advert_route_evidence`.
- The history is held in memory. The live poller checks it on every tick
(two primary-key lookups), so a retune or a new `rx_radio_id` is known
before the observation is broadcast. A full reload runs every 10 min.
- `radio` is added to observations, packets, grouped rows (taken from
the header observation) and the live feed.
- `GET /api/packets?radio=MHz,kHz,SF,CR` filters by setting.
- Combined with `observer=`, it means *that* observer heard it on that
setting.
- A radio query is served from memory instead of falling back to SQL,
which would return the window unfiltered.
- `GET /api/radios` (named struct) and `radio_history` on `GET
/api/observers/{id}`.

**Frontend**

- Packet filter fields `radio`, `freq`, `bw`, `sf`, `cr`, e.g. `sf == 8`
or `freq > 869.5 && bw == 62.5`.
- A Radio row in the packet detail and a Radio column in its
observations table (only when the server reports one).
- A radio history table on the observer page once there is more than one
stretch.
- `RfParams.radioText` formats the setting. It escapes all five
characters and returns non-numeric input as it came, escaped.

## Performance

`txMatchesRadio` runs once per observation inside `filterPackets`.

- Each observation costs one map lookup by observer id.
- Observers with a single stretch (nearly all) answer from that lookup
directly.
- Retuned observers add a binary search over their few stretches.
- Multi-radio observers add one lookup by observation id.
- The snapshot is loaded once per filter call.

`BenchmarkTxMatchesRadio`, 30K packets / 150K observations (20
observers, 2 of them retuned):

```
BenchmarkTxMatchesRadio-24    602    2056924 ns/op    0 B/op    0 allocs/op
```

Memory is one entry per stretch plus one per multi-radio observation;
nothing is added per observation otherwise.

## Tests

- **dbschema:** seed runs once, canonical spellings, the tables exist
without `observers.radio`.
- **ingestor:**
  - stretches extend and split; canonical spellings do not split
- two radios of one node hearing one transmission over two paths each
keep their own radio
  - a retained status does not write history
  - prune and purge
  - `rx_radio_id` reaches `PacketData`
- **server:**
- resolution before, between and after stretches, and with an unknown
radio id
  - `observer` + `radio`, and grouped rows carry `radio`
  - incremental refresh, `/api/radios`, `radio_history`, missing tables
  - the benchmark above
- **frontend:** the new filter fields in
`tests/unit/test-packet-filter.js`, and XSS cases through the real
`RfParams.radioText` and the history renderer.
- **E2E:** `tests/e2e/test-radio-setting-1869-e2e.js` checks the grouped
rows' radio, `sf ==` on the packets page, the detail Radio row and the
observer history. It is added to `scripts/non-unit-tests.json` and to
shard 2.

`go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema`
passes, along with `sh test-all.sh`, `eslint@8` (0 errors) and the XSS
diff gate. The new E2E test and `test-e2e-playwright.js` pass against
the fixture.

## Browser validation

This runs on a live instance with six observers, one of them retuned to
869.618/62.5/SF8/CR8:

- The packet detail shows `Radio: 869.432 MHz · SF7 · BW 62.5 kHz · CR
4/5`.
- `sf == 7` shows 106 of 106 grouped rows and `sf == 8` shows 0 of 107;
the SF8 observer has not heard anything yet.
- `/api/radios` lists both settings.
- No page errors.

## Notes

- **The customizer (rule 8)** gets nothing: the only constants are
server-side refresh intervals.
- **Attribution timing:** packets heard between a retune and the
observer's next status message (minutes) are attributed to the previous
setting, because a stretch starts when the status is handled. This is
documented at `RecordObserverRadios`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 21:08:40 +01:00
n30nex 1dfcd0f5d3 fix(analytics): preserve distance invalidation during an active build (#2284)
Red commit: `38c0abe1` ([failing
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190273036)).

Fixes #2281. Background-fill completion could reset an active
`sync.Once`, causing a fatal mutex unlock or letting an older distance
build mark an incomplete index ready. Build admission now uses the
existing mutex, and a generation check prevents stale completion from
consuming invalidation. The loader also releases the store lock before
acquiring the distance gate.

- Deterministic regression drives the actual loader, recomputer and HTTP
handler; the old runtime crash is contained in a child process.
- Two cold observations stay below the normal 5% rebuild threshold. Ten
concurrent callers must share one rebuild, and the final response must
contain the cold path.
- Validation on `b8a2333b`: full server suite, focused distance suite,
ten repeated race runs plus a parent race rerun. Shared-base ingestor
suite passed.
- Browser verified: http://127.0.0.1:18855 returned 202 with
Retry-After, then 200 with 20 paths and 20 hops.
- Constant-time admission/invalidation and one fixed-size counter;
existing computation costs remain unchanged. No schema, settings or
API-shape changes.

Privacy, whitespace and scope checks passed. The external home-directory
preflight bundle is unavailable; applicable checks were performed
locally. Independent review found no outstanding fixes. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191246966)
passed on `b8a2333b`, including all three browser shards and both
architecture builds.
2026-10-26 15:42:42 +01:00
n30nex f66cd320da perf(graph): reuse indexed shortest-path workspaces (#2293)
Red commit: `8b3cffd0` ([regression
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203043161)).

Fixes #2226. Bridge and coverage scoring now convert keys once and reuse
indexed distance/predecessor workspaces and at most one heap item per
vertex. Existing algorithms, normalization, snapshot publication and
synchronous startup behavior remain intact.

Five paired runs, deterministic 2,000-node/10,000-edge fixture, Go
1.27.2 on Windows/amd64, GOMAXPROCS=2:

| Calculation | Median time | Allocated bytes/run |
|---|---:|---:|
| Bridge | 8.237s → 0.819s | 2,083,295,328 → 2,517,664 |
| Coverage | 4.900s → 0.694s | 1,508,440,736 → 2,209,984 |

The desktop was shared; baseline bridge runs ranged 7.81–17.56s. These
are algorithm/allocation measurements, not startup/API latency or
retained memory. Rebuilding committed `5f34b5bf` produced the exact
measured binary hash.

- Independent path oracle, edge/epsilon/tie/disconnected cases,
allocation regression, race checks and full server/frontend suites pass.
- Browser verified: isolated native servers at http://127.0.0.1:18903
and http://127.0.0.1:18904 return identical nonzero graph scores.
- O(V+E) bounded workspace; O(V(V+E) log V) calculation. No persistent
cache, schema, dependency or configuration change.

Independent reviews and local preflight checks passed. External
home-directory preflight bundle unavailable; applicable checks were
performed locally. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204557795)
passed on `5f34b5bf`, including all three browser shards and both
architecture builds.
2026-10-26 15:42:29 +01:00
n30nex 63d63ea90a perf(healthz): avoid scanning retained packets after startup (#2305)
Fixes #2301.

Readiness probes currently scan every retained transmission while
holding the packet-store read lock. Once `StartupLoadDone` closes, this
reads the maintained observation counter beside the transmission count
under that same lock. Count extraction becomes O(1), with no new cache,
query or allocation.

Startup retains the original scan: the background loader publishes
indexes before its packet slice, so switching on readiness or
`LoadComplete` would be premature. Response fields, readiness status,
ingestion and configuration remain unchanged.

Whole-handler benchmark medians, microseconds per probe:

| Retained transmissions | Upstream | Candidate |
| ---: | ---: | ---: |
| 1,000 | 72.14 | 83.84 |
| 30,000 | 320.85 | 65.74 |
| 100,000 | 1,228.50 | 81.97 |

Four alternating pairs, 10,000 iterations/size, real SQLite-loaded
fixtures with three observations per transmission on average; Go 1.27.2,
Windows/amd64, two CPUs. All samples retained; shared-host variability
means no small-cache or production-latency improvement is claimed.
Allocations remain 36/probe. The included benchmark supports
reproduction against unchanged upstream production plus the identical
harness.

TDD exemption: pure performance refactor; **all pre-existing test files
are byte-unchanged**. Added characterization tests pass before and
after, covering loaders, deduplication, live updates, eviction, partial
startup failures, concurrent snapshots and liveness. Independent review
prompted test-only follow-up `2a0083b1`, which detects premature counter
use; production remains identical to measured `1b8e6680`.

Full server, race, ingestor and post-push browser/API checks pass
locally. [Final-head CI
passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40213482779),
including all browser shards and architecture builds. No customizer
changes.
2026-10-26 15:42:15 +01:00
n30nex abad8a26b3 fix(liveness): require evidence from each observed relay frame (#2298)
Red commit: `46e3b082` ([regression
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203680455)).

Fixes #2266 and #2267. Certain relay activity now requires an eligible
path from the observation that supplied it. A representative
transmission header, destination attribution or catalogue uniqueness
alone cannot supply that evidence. Later valid FLOOD evidence survives
observer/path deduplication.

The fix preserves valid own packets and excludes failed signatures,
planned DIRECT/TRACE paths, ambiguous/retired prefixes and missing or
mismatched headers. Firmware reference: meshcore-dev/MeshCore
`81dbae59`.

- One eligibility byte replaces an unused raw-frame field. StoreObs
remains 152 bytes in the protected 160-byte allocation class; no wire
frames are retained. Existing dedup maps gain one pointer per slot only
at 64+ observations.
- Processing is linear in retained paths, bounded by 64 wire path bytes.
No new SQL, schema, configuration or maps.
- Tests cover five load/ingest paths, late duplicates,
analytics/adaptive thresholds, positive controls and concurrent readers.
Review correction `4ad6c5ad` preserves lazy decoded caches while
retaining escaped/legacy signature rejection.
- Full server suite, focused/race checks and frontend checks pass.
Browser verified on http://127.0.0.1:18905: both upstream false-positive
cases disappear and a genuine later FLOOD is recognized.

Independent review findings are addressed. Local preflight checks
passed; external home-directory bundle unavailable. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40205015381)
passed on `4ad6c5ad`, including all three browser shards and both
architecture builds.
2026-10-26 15:42:01 +01:00
AdamandClaude Opus 5.5 d777f4b7ce fix(ingestor): only flood relay hops refresh nodes.last_seen (#2307) (#2308)
Fixes #2307.

The ingestor refreshed `nodes.last_seen` for every resolved hop in a
path. For DIRECT routes those hops are still to come, and for TRACE
`path_json` is the route the trace was asked to take, so tracing or
messaging a switched-off repeater kept it looking recently heard.

## What changes

**`cmd/ingestor/db.go`**
- New `pathHopsTransmitted(data)`: true only when
`PathBytesAreHops(payloadType)` and the route is FLOOD or
TRANSPORT_FLOOD. It's the same rule as the server's `relayHopHexLen` in
`certain_heard.go`.
- `InsertTransmission` calls `touchRelayNodesLocked` only when that
holds. `observations.resolved_path` is still written for every
observation, so analytics are unchanged.
- The route and payload type come from the observation's own decoded
header (`PacketData`), not the transmission's first frame (see #2266).

## Perf

There's one integer comparison per observation, before the existing
touch. The touch now skips DIRECT and TRACE observations, so it does
fewer `UPDATE nodes` calls, never more.

## Tests

`cmd/ingestor/relay_touch_test.go`:
`TestInsertTransmission_TouchesOnlyFloodRelays` goes through
`InsertTransmission` with a real store and prefix index, using a node
whose 2-byte prefix is unique:

- FLOOD and TRANSPORT_FLOOD TXT_MSG: `last_seen` advances to the rx time
- DIRECT TXT_MSG and DIRECT TRACE: `last_seen` is unchanged

With `db.go` from master, the DIRECT and TRACE cases fail.

How I checked:
- `go vet ./...` and `go test ./...` in `cmd/ingestor` pass, and `gofmt
-l` is clean (`golang:1.27` container, since there's no local Go
toolchain).
- No server, frontend or schema changes, so I didn't run the browser and
E2E steps.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 15:41:48 +01:00
n30nex 3a85a98cc5 docs(api): describe the supported packet-list query parameters (#2294)
Red commit: `cee2a4bc` ([regression
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203052113)).

Partial follow-up for #1670. GET /api/packets now documents all 16
supported query parameters, including hash, route, node(s), region,
channel, area and expand. It removes unsupported sort/timeRange/search
entries, corrects numeric types, and describes actual defaults, limits
and memory/SQL/multi-node differences.

This changes only packet-route metadata. Runtime API behavior and the
schema generator stay unchanged.

- Served-spec regressions fail on upstream while actual-route controls
pass.
- Focused OpenAPI/SwaggerUI tests, full server suite and full standalone
frontend suite pass on `d8014193`.
- Browser verified: http://127.0.0.1:18906/api/spec exposes the 16
supported parameters and integer types; the running packet route
confirms default limit 50 and offset 0.
- No new requests, dependencies, settings, schema or untyped-map
occurrences.

The umbrella remains open: current route coverage stays 98/122 with 24
allowlisted gaps. Other endpoint parameters/schemas and a generic
parameter-presence gate are outside this PR.

Independent reviews and local preflight checks passed. External
home-directory preflight bundle unavailable; applicable checks were
performed locally. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204567683)
passed on `d8014193`, including all three browser shards and both
architecture builds.
2026-10-26 15:41:20 +01:00
n30nex 041daecd1e docs(api): describe performance and clock diagnostics (#2303)
Partial follow-up to #1670.

Six existing read-only diagnostics endpoints lacked usable OpenAPI
descriptions: process I/O, SQLite metrics, ingestor write counters, and
fleet/node/observer clock skew. This documents their parameters,
empty/error behavior and response root shapes. The known-gap list falls
from 24 to 18.

Descriptions preserve current behavior, including unknown/empty area
fallback, the PacketStore cache ratio, stale write-counter snapshots,
and the single-node 404. Schemas remain minimal; the existing generator
still declares only the generic 200 response, with 404 explained in
prose.

No runtime handler, authentication, configuration or customizer changes.
No added generic response maps.

Validation: regression tests exercise the served spec and real router,
including missing/malformed data, area selection and fallback, and
calibration responses. Full server and ingestor tests pass locally;
post-push Chromium verifies the served spec, six routes and Swagger
rendering. Three independent reviews passed.

Test-only RED `218a3bf3` produced the [six expected documentation
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212159661/job/121225534416).
GREEN `1e2d8c08` adds the documentation and [passes
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212678062),
including all browser shards and architecture builds.
2026-10-26 15:41:05 +01:00
efitenandClaude Opus 5.5 e149568c6c feat(observer-logins): personal MQTT logins for observers (#2306)
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.

Closes #2319.

The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.

## Changes in this PR

**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.

**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.

**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.

Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.

**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.

**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.

**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.

**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.

## Enabling it on an instance

Full steps in `docs/observer-mqtt-logins.md`. In short:

1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.

## Who can claim an observer

Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.

What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.

## Performance

`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.

## Verification

- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.

- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.

## Known behaviour

EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.

## Not in this PR

- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.

Note on cb9f4ff3: its message says the scheme problem was reported by an
operator. That report turned out to come from before the broker used the
hook, so it does not show this cause; the change rests on the firmware
source only.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 09:50:20 +01:00
n30nex b3367f5aea fix(analytics): keep distance indexing lazy during startup (#2278)
Chunked startup now leaves the distance index lazy, as the legacy loader
already does. Requests wait for the initial chunks to finish, then
trigger one build and publish the refreshed result. A terminal
initial-load failure returns a clear error instead of retrying forever;
a completed hot window remains usable if later background filling fails.

Fixes #2227.

Validation: RED reproduces eager and premature builds plus the
terminal-error retry loop. Full server and unchanged frontend suites
pass, with focused race checks. The browser/API check observed 202 then
200, nonempty distances, and exactly one build.

Performance evidence is the test-enforced absence of eager/duplicate
distance construction, with required path indexes retained. No timing
speedup is claimed. No new settings. Common ingestor validation is
covered by the separate #2272 test-stability change.
2026-10-25 12:11:09 +01:00
n30nex 77c0392947 fix(reach): avoid caching interrupted query results (#2277)
Reach now propagates failed scope lookups, rejects cancelled results
before caching, and retains the prior complete degree snapshot when
scanning or row iteration fails. Missing rows and optional scope columns
still produce the legitimate empty result.

Fixes #2258.

Validation: RED reproduces partial degree snapshots and cached success
after failed/cancelled follow-up queries. The full server and unchanged
frontend suites pass; focused race and browser Reach checks pass.
Handler regressions verify that failed builds do not replace caches and
that a later request rebuilds successfully.

The shared-build cancellation policy and query count are unchanged. This
is independent of #2249's admission/queue work. Common ingestor
validation is covered by the separate #2272 test-stability change.
2026-10-25 12:10:58 +01:00
n30nex c085ca8cba fix(rf-params): respect node visibility in responses (#2276)
The RF endpoint now applies the current blacklist and hidden-name policy
to both exposed node metadata and deviations, including after cache
warm-up. Unhiding a node restores its data without waiting for cache
expiry.

Fixes #2268.

The shared RF snapshot and peer calculation remain unchanged; filtering
copies only the response data. This adds a linear node pass and bounded
deviation copies, with no additional database calls or cache
invalidation. The feature remains opt-in.

Validation: RED reproduces both response leaks. The full server and
standalone frontend suites pass, as do focused race tests and the RF
browser suite. Regressions cover warm/cold caches, policy updates,
unhide, visible controls and default-off behavior. Common ingestor
validation is covered by the separate #2272 test-stability change.
2026-10-25 12:10:47 +01:00
n30nex 12e0a40a82 test: make hop coverage clock checks deterministic (#2273)
Three hop-coverage tests sampled the clock before database setup, then
allowed only five seconds of drift. Slow setup made otherwise correct
coverage checks fail. They now use the existing clock hook, restore it
after fixture workers finish, and assert exact equality. Production code
is unchanged.

Fixes #2272.

Validation: the original full run reproduced all three failures; the
corrected full ingestor suite passes. Focused race checks pass, and
row/deletion/ordering assertions remain intact. This is a test-only
stabilization of existing failing assertions, not a new runtime feature.
2026-10-25 12:10:14 +01:00
efitenandClaude Opus 5.5 2bb000f2aa feat(notify): adaptive offline threshold and certain node analytics (#2260) (#2262)
Part 2 of #2260. **Builds on #2261** (certain liveness): review/merge
that first; until then this diff also shows its commit.

## Adaptive offline threshold

`userManagement.notifications.adaptiveOffline` — **off by default**:

```json
"notifications": {
  "enabled": true,
  "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 }
}
```

The fixed `infraSilentHours` (72 h) treats every repeater the same. With
this on, a watched node's `node.offline` window is learned from its own
**certain** traffic (the #2261 rule):

1. Certain packets per hour over the week **before its last certain
packet**. Ending there, not at now, keeps an outage from dragging the
rate down until the fixed window returns and the node reads "back
online".
2. Empty runs are classified: ≤ 6 h and recurring on another day at the
same hours = normal silence (kept; the window is never shorter than the
longest + 1 h); any other run = outage, left out of the learning so it
neither stretches the window nor blocks learning.
3. λ = 10th percentile of the remaining hourly counts, so quiet nights
don't alert.
4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson,
ln 10⁶ ≈ 13.8).
5. Clamped to [`floorMinutes`, the role's silent window].

Kept on the fixed window: λ < 1/hour, under 24 h of history in memory,
or a 10th-percentile hour heard by fewer than `minObservers` observers
(one observer dropping out must not read as the node going offline).
Silence is measured up to the newest packet in the store, so a stalling
feed stops every node's clock. A node stored offline comes back only on
evidence from the moment it went offline on. The notifier caches each
learned window for 10 minutes and drops unwatched nodes.

For the #2260 repeater (~840 packets/day) the mail would have gone out
about 30 minutes into the outage, instead of never (or 72 h with #2261
alone).

## Certain node analytics

- Each `activityTimeline` bucket carries `certain`: the part that can
only be this node (`certainFor`, the packet-level form of the
`certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other
endpoints are unchanged.
- Availability and longest silence use certain buckets only; the longest
silence now includes one **still running** (an outage in progress is the
silence worth seeing).
- The chart stacks certain (`--accent`) and uncertain (`--text-muted`) —
existing theme variables, so no new customizer entry. Descriptions
updated.

## Performance

`CertainActivity` walks a week of packets per watched node, at most once
per node per 10 minutes, with the read lock taken **per node**:

```
BenchmarkCertainActivity-16   12.1 ms/op   (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold)
BenchmarkNoteCertainRelays-16  121 ns/op   0 allocs
```

Node analytics adds one `certainFor` per packet of the requested node (a
prefix-map lookup, or the cached decoded JSON for its own packets).

## Tests

- `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet
nights set the rate; > 10 % empty hours, short history, too few
observers → fixed.
- `CertainActivity` ends at the last certain packet and counts only
certain packets — verified red by mutating the window to end at now.
- Notifier end to end: learned window passed → bad; inside → good; too
few observers / setting off → fixed 72 h; cache refresh and prune.
- `certainFor`, `longestSilence` (incl. running silence), config
defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket.
- Frontend: `NodeAnalyticsTimeline.series` (new
`tests/unit/test-node-analytics-certain-timeline.js`).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork.

## Validation

Server on the freshened e2e fixture: analytics buckets carry `certain`
(e.g. 89 of 90 packets certain for one repeater). **No browser
screenshot**: Playwright isn't installed on this machine.

## Later (rule 8)

`floorMinutes` / `minObservers` are server config, not theme settings;
an admin UI for notification settings would be the place, not the
customizer.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 21:20:42 +02:00