26 Commits
Author SHA1 Message Date
AdamandAdam c83499763d Allow mobile clients to manage repeater claims and details (#2429)
Draft for feedback: a working version of native repeater claims and
edits from mobile clients, built on the repeater info feature from
#2415. Four open questions are at the end.

## Situation

Repeater info (#2415) can only be claimed and edited from a browser
session. Device tokens from `POST /api/auth/device-token` are limited to
`rx` (CoreDrive RX: settings and companion linking), and `GET
/api/repeaters/{pubkey}/info` treats a bearer token as anonymous. A
mobile client can show a repeater's details but can't offer "claim" or
"edit" natively.

`PUT /api/repeaters/{pubkey}/info` is also a full replace with no
precondition. Two people editing from the same copy silently overwrite
each other. That's rare in the browser, but likelier from an app that
keeps a cached copy.

## Change

Three commits, one per piece.

**1. Device-token scopes for the repeater routes** (`8aba3715`)

- `POST /api/auth/device-token` takes an optional `scopes` list. Omitted
(or `null`) still means `["rx"]`, so CoreDrive RX and every existing
token keep exactly their current permissions. An empty list or an
unknown scope answers 400, checked before the password.
- New scopes:
- `repeater:claim`: `POST` and `DELETE /api/repeaters/{pubkey}/claim`.
These already only add or remove the caller's own claim. Several
claimers per repeater, idempotent `POST`, unchanged.
- `repeater:edit`: `PUT /api/repeaters/{pubkey}/info`, still subject to
`CanEditRepeater`.
- Every device token may use `/api/auth/me` and `/api/auth/logout`, so a
repeater-only token can check and end its own login.
- `GET /api/repeaters/{pubkey}/info` with a token holding either
repeater scope returns `canEdit` and `claimedByMe`. An unknown or
expired token answers 401 rather than passing as anonymous, so a client
learns its login is gone. A valid token without a repeater scope still
gets the public view. A cookie still wins over a bearer header, as
elsewhere.
- `GET /api/account/repeaters` (My repeaters, from #2430) also takes a
token holding either repeater scope, so an app gets the account's claims
in one request. It can't build that list from the public catalog, where
claimers are display names, and reading repeaters one at a time would
miss a claimed repeater that has left the node list. An `rx`-only token
still answers 403.
- Scope checks are method-aware (`bearerScopesFor(method, path)`). The
repeater paths are added to the bearer CORS paths.
- Same handlers, validation, rate limit (`repeater`, 120/h per user and
per IP), audit actions and admin claim removal as the browser. Token
writes add `via: device` to the `repeater.claim`, `repeater.edit` and
`repeater.unclaim` rows, and the `user.login` row records the granted
scopes. The admin route keeps refusing bearer tokens.

**2. Conditional edits with a revision** (`9f9bf47b`)

- `users.db` schema v10 adds `repeater_info.revision`. Existing rows
start at 1; a repeater with no saved details is at 0. Claims don't
change it.
- `GET` (single and list) and `PUT` responses include `revision`.
- `PUT` accepts an optional `baseRevision`. The check is part of the
write statement, so of two saves from the same base exactly one lands:
- `baseRevision: 0` (first save): `INSERT … ON CONFLICT(pubkey) DO
NOTHING`.
- `baseRevision: n`: `UPDATE … SET revision = revision + 1 … WHERE
pubkey = ? AND revision = ?`.
- No match answers `409` with the stored details
(`canEdit`/`claimedByMe` included) and writes nothing. That's the same
shape `PUT /api/account/settings` uses for its conflicts.
  - A stale base is refused even when the body would change nothing.
- Permission is checked first, so a 409 never shows details to someone
who can't edit.
- Without `baseRevision` the save is unconditional and bumps the
revision, so `public/repeater-info.js` works unchanged.
- `/api/config/client` adds `userManagement.repeaterTokens: true`, so a
client can tell this server from one without the scopes or
`baseRevision` (older servers answer the device-token request with the
SPA page, as noted for `companionLinking`).

**Discovery and conflict recovery, for a client:**

1. Check `userManagement.repeaterTokens` in `/api/config/client`.
Without it, keep the claim and edit actions hidden.
2. Log in with `scopes: ["repeater:claim", "repeater:edit"]` (or just
what the client offers). The response echoes the granted `scopes`.
3. Read `GET /api/repeaters/{pubkey}/info` with the token. Show
Claim/Unclaim from `claimedByMe` and Edit from `canEdit`, and keep
`revision`.
4. Save with `PUT …/info` and `baseRevision` set to that revision. On
200, keep the new `revision` from the response. On 409, the body is the
current record: show or merge it, then retry with its `revision`. On
401, log in again. On 403, the account can no longer edit (for example
someone claimed it), so refresh from `canEdit`.

**3. Account page device list** (`d28374aa`)

My account, Devices labelled every device token "CoreDrive RX".
`/api/account/sessions` now includes a device token's `scopes`. The page
keeps the CoreDrive RX label for `rx` tokens (and rows without scopes).
It shows other tokens under the device name the app sent, with a line
such as "can claim repeaters and edit repeater details".

Docs: `openapi.go`, `docs/api-spec.md` and the Repeater info section of
`docs/user-guide/accounts.md`.

## Questions

1. **One scope or two?** This uses separate `repeater:claim` and
`repeater:edit`, so a client can ask for less, and a token that only
edits can't put the user's name on a repeater publicly. One `repeater`
scope would be simpler if you'd rather not split it.
2. **Editing unclaimed repeaters from mobile.** This keeps web parity:
any active user may edit a repeater nobody has claimed, and once it has
claimers only they (and admins) can. The alternative is to require a
claim before a token may edit. That's stricter, but then app and browser
rules differ.
3. **Operator switch.** There's no new setting: the scopes are available
whenever `userManagement` is on, and the user opts in when the app asks
for them at login. Would you want an explicit opt-in such as
`userManagement.mobileRepeaterWrites`?
4. **Stale-edit contract.** Does `revision` plus `baseRevision` in the
body, with 409 returning the current record, fit? I matched account
settings rather than `ETag`/`If-Match`, to keep one pattern in the API.
The browser editor doesn't send `baseRevision` yet. Making it send one
would be a small follow-up, if you want browser saves protected too.

## Tests

- `cmd/server/repeater_device_token_test.go`:
- default and requested scopes, rejected scopes, and the method-aware
route table;
- per-scope allowed and denied routes, logout with a repeater-only
token;
  - claim and unclaim of own claim only, alongside another claimer;
- edit permission (unclaimed repeater, claimer, non-claimer, admin) and
a disabled account;
  - the admin route refusing tokens;
  - authenticated, public and expired-token reads;
- `via: device` audit rows, cookie writes still needing CSRF, and
session scopes.
- My repeaters with a claim or edit token (own claims only), and 403 for
an `rx`-only token.
- `cmd/server/repeater_revision_test.go`:
- revision on reads, a first edit with no saved row, and a stale base
returning 409 with the current record and no audit row;
  - unconditional browser saves, validation and permission order;
- two concurrent saves from the same revision (one 200, one 409), and
the client config flag.
- `internal/users/repeater_revision_test.go`:
  - revision counting and conditional first and later saves;
  - an 8-way race for both cases (`-race`);
  - the v9 to v10 upgrade.
- `tests/unit/test-user-management-ui.js`: app token labels and scope
line.

Run locally:

- `make test`: all 16 modules OK.
- `sh test-all.sh`: all suites pass, run in `node:24-bookworm`. On
macOS, `test-manage-workflow.js` fails 2 cases on unchanged master
because of the BSD tools.
- `npx eslint public/*.js`: 0 errors, the same 87 warnings as master.
- `node tests/e2e/test-lint-network-calls.js`
- `tests/e2e/test-user-management-e2e.js` against local e2etest servers
with the CI config: 44/44.
- Manually against a local instance:
  - in the browser: claim, edit, unclaim, and the Devices list;
  - with curl: the token flow, including a 409 after a browser save.

Not run: the full Playwright shard matrix from CI.

---------

Co-authored-by: Adam <196335+anieto@users.noreply.github.com>
2026-11-21 22:27:48 +01:00
ZachandClaude Opus 5.5 1a6322a707 feat(map): more tile providers (topo, satellite, gray, vector), hillshade overlay and custom tile servers (#2434)
Closes #2428.

Adds free and open map providers (topographic, satellite, gray and
vector basemaps), a hillshade overlay, and support for operators' own
tile servers. Everything is additive and opt-in: Carto stays the
default, every new provider group is off until the operator turns it on,
and no existing config key or provider was removed. An upgrade with an
untouched config changes nothing in the picker.

> **Dependency:** #2426 has merged. The branch is rebased onto current
`master` (`6c78a1d2`) with no conflicts.

## What's new

- **Provider registry** (`public/map-tile-providers.js`). New groups are
**opt-in**: each is off until `map.tiles.providers.<group>.enabled` is
`true` (the boolean), so licences and volunteer servers are the
operator's call:
- **EOX**: Sentinel-2 satellite (2025 and 2016), Terrain Light, and a
place-labels overlay. EOX 2025 imagery is CC BY-NC-SA 4.0
(non-commercial); a commercial deployment should leave `eox` off or
license it.
- **OSM France**: Humanitarian, CyclOSM, OSM Gray and OSM Gray Dark
(volunteer-run, fair use).
- **OpenFreeMap** and **VersaTiles** vector styles: Liberty, Bright,
Positron, Dark, Fiord, Colorful, Muted and the Gray variants.
- **Hillshade** (below) is opt-in the same way. **OpenTopoMap**, the OSM
community server, Stamen and USGS keep their existing opt-in.
- **Custom tile servers**: `map.tiles.custom` adds raster (XYZ), vector
(MapLibre style URL) and raster-overlay entries as `custom-<id>`. They
can be used in `darkDefault` / `lightDefault`. Invalid entries are
skipped with a console warning.
- **Hillshade overlay** (`public/map-hillshade.js`): shaded in the
browser from Terrarium elevation tiles.
`map.tiles.providers.hillshade.url` can point at a mirror.
- **Vector basemaps** (`public/map-vector.js`) use a vendored MapLibre
GL 5.24.0 (CSP build) and `@maplibre/maplibre-gl-leaflet` 0.1.4.
- They are lazy-loaded only when a vector map is picked (about 380 KB
gzipped).
  - The load times out after 15 s.
  - Without WebGL the map falls back to the theme's default raster map.
- **One picker on Map, Live and Repeater Planner**: Auto, Street,
Topographic, Satellite, plus an Overlays group.
- **A map picked there is saved** (`localStorage` `mc-tile-pinned`,
untrusted: honoured only while registered). Every picker reopens on it
after a reload or on another page, until the visitor picks **Auto**.
- The other maps (home mini-map, node detail, node reach, analytics)
follow the visitor's choice through the shared builder: the saved pick
when it suits the current theme (satellite always does), otherwise the
theme's map.
- Those small maps never load MapLibre, so a vector choice shows the
theme's default raster map there.
- **Security contract**: the `CONTRACT` block at the top of
`map-tile-providers.js` (S1 to S7) covers one shared tile-URL validator
(`https?://` or a same-origin path, no credentials in the authority,
whitelisted placeholders), attribution HTML-escaped once, picker labels
set with `textContent`, explicit MapLibre options, pinned vendored
versions, CORS-safe hillshade decoding, and treating stored picks as
untrusted input. `map.tiles` is served publicly in `/api/config/client`,
so keys put there are public (the docs warn about this).

## Fixes included

The race stress suite written for this PR found teardown races when
leaving map pages mid-load:
- a lazy MapLibre load finishing after navigation;
- a hillshade tile decoding after `remove()`;
- a provider-change event arriving during teardown.

It also found a pre-existing document-listener leak in `AreaFilter` /
`RegionFilter`. All of these are fixed in `e6709e74`, with regression
tests in `tests/e2e/test-map-teardown-races.js`.

The first CI run then caught a bug in that fix (`7f8d792c`).
`MC_safeRemoveMap` called `map.off()` before `map.remove()`, so the
map's `unload` event reached no one. Leaflet relies on that event to
detach controls and clean up the zoom-animation proxy and BoxZoom, and
`test-issue-1206` watches it to confirm node maps are disposed. `off()`
now runs after `remove()`, and a new regression test checks that
`unload` fires, controls detach and no listener is left.

A later stress run (a new seed) found a real two-tab bug, fixed in
`0506fc1d`. When another tab's overlay pick is already in localStorage
but its `storage` event hasn't arrived, a layer-picker click here
triggered a sync that added that overlay while Leaflet was still
applying the click. Leaflet doesn't redraw the list for changes it sees
mid-click, so the overlay's checkbox stayed unticked, and the next click
on any overlay removed it and wrote the removal back to every tab. The
overlay set now defers a sync until the click has finished. New E2E
check `4b` in `test-map-providers-e2e.js` plants another tab's pick and
clicks twice: one write per click, the planted overlay stays ticked. It
fails on the previous code.

## Docs

- `docs/deployment.md`, "Map Tile Providers" section, rewritten. It
covers every provider and its licence, turning groups on and off, custom
servers, vector maps, hillshade, self-hosting, and what is public.
- User guide:
  - `map.md` gets a "Map style and overlays" section;
  - `live.md` notes the picker;
  - `configuration.md` gets a "Map tiles" section;
  - the FAQ gets a new entry on satellite, terrain and gray maps.
- `CHANGELOG.md`: an Unreleased entry.

## Test data

Two datasets were used, so every number below says which one it came
from.

**1. Large synthetic dataset** (perf A/B and screenshots). It is
generated, contains no real mesh data, and is not checked in.

- **Generator:** `synthgen` 1.0.0, seed 1, "full" scale. Built from
upstream `be16173e` and retimed so its 18-day window (2026-10-31 to
2026-11-18) ends at the time of the run.
- **Database:** SQLite, 3.94 GB (586 MB as zstd), 11.6 M rows in all.

| Table | Rows |
|---|---|
| Nodes | 18,102 (12,606 companions, 3,652 repeaters, 953 rooms, 891
sensors) |
| Nodes with a location | 13,452, spread worldwide (latitude −37° to
60°, longitude −123° to 175°) |
| Observers | 1,175, in 42 regions on six continents |
| Transmissions | 759,133 |
| Observations | 10,569,668 |
| Neighbour edges | 42,806 |
| Advert route evidence | 212,142 |

During the perf runs, the server's packet store was pinned to the newest
48 h with a 1.5 GB memory cap. Each server held about 65 k transmissions
and 0.9 M observations in memory (baseline 65,647 / 906,467; candidate
64,862 / 896,248), plus all 18,102 nodes (about 17,650 active) and 1,175
observers. The tracked store was about 450 MB.

**2. The repo's CI fixture** (`test-fixtures/e2e-fixture.db`, 704 KB).
All E2E suites and the race stress gates ran against it, freshened the
way CI does: 204 nodes, 31 observers, 500 transmissions, 502
observations. A second copy with the Repeater Planner enabled was used
for the planner-on stress runs.

**Perf environment:**
- Headless Chromium 158 via Playwright 1.58.2, at 1280×800 with device
scale factor 1.
- 9 measured runs per scenario plus 1 warm-up, on each of `master` and
this branch.
- Host: Linux, AMD Ryzen 7 3700X (16 threads), 63 GB RAM.
- **Tiles were stubbed.** The harness answers every tile, elevation and
style request locally with the same stub on both sides (`--mode stub`),
so perf timings involve no network. Only the screenshot gallery uses
real tiles.

## Screenshots

These were taken from the synthetic dataset with real tiles in a real
browser (headless Chromium, 1280×800). Click a thumbnail for the full
size. The images are hosted on a separate assets branch, so they are not
in this diff.

#### Street

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-light.png"
width="260" alt="Street: Carto (default) (light)"></a><br>Carto
(default) (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-carto-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-carto-dark.png"
width="260" alt="Street: Carto (default) (dark)"></a><br>Carto (default)
(dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/street-osmfr-hot-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/street-osmfr-hot-light.png"
width="260" alt="Street: OSM France Humanitarian (light)"></a><br>OSM
France Humanitarian (light) |

#### Gray

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-light.png"
width="260" alt="Gray: OSM France (light)"></a><br>OSM France (light) |
<a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-osmfr-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-osmfr-dark.png"
width="260" alt="Gray: OSM France (dark)"></a><br>OSM France (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-light.png"
width="260" alt="Gray: VersaTiles (light)"></a><br>VersaTiles (light) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-versatiles-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-versatiles-dark.png"
width="260" alt="Gray: VersaTiles (dark)"></a><br>VersaTiles (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-light.png"
width="260" alt="Gray: OpenFreeMap Positron (light)"></a><br>OpenFreeMap
Positron (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/gray-openfreemap-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/gray-openfreemap-dark.png"
width="260" alt="Gray: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap Dark
(dark) |

#### Topographic

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-cyclosm-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-cyclosm-light.png"
width="260" alt="Topo: CyclOSM (light)"></a><br>CyclOSM (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-eox-terrain-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-eox-terrain-light.png"
width="260" alt="Topo: EOX Terrain Light (light)"></a><br>EOX Terrain
Light (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/topo-opentopomap-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/topo-opentopomap-light.png"
width="260" alt="Topo: OpenTopoMap (opt-in) (light)"></a><br>OpenTopoMap
(opt-in) (light) |

#### Satellite

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-light.png"
width="260" alt="Satellite: Sentinel-2 2025 (light)"></a><br>Sentinel-2
2025 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-light.png"
width="260" alt="Satellite: Sentinel-2 2025 with labels
(light)"></a><br>Sentinel-2 2025 with labels (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-dark.png"
width="260" alt="Satellite: Sentinel-2 2025 (dark)"></a><br>Sentinel-2
2025 (dark) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2025-labels-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2025-labels-dark.png"
width="260" alt="Satellite: Sentinel-2 2025 with labels
(dark)"></a><br>Sentinel-2 2025 with labels (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-light.png"
width="260" alt="Satellite: Sentinel-2 2016 (light)"></a><br>Sentinel-2
2016 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/sat-s2-2016-labels-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/sat-s2-2016-labels-light.png"
width="260" alt="Satellite: Sentinel-2 2016 with labels
(light)"></a><br>Sentinel-2 2016 with labels (light) |

#### Vector (MapLibre)

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-liberty-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-liberty-light.png"
width="260" alt="Vector: OpenFreeMap Liberty
(light)"></a><br>OpenFreeMap Liberty (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-dark-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-dark-dark.png"
width="260" alt="Vector: OpenFreeMap Dark (dark)"></a><br>OpenFreeMap
Dark (dark) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-light.png"
width="260" alt="Vector: VersaTiles Colorful (light)"></a><br>VersaTiles
Colorful (light) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/vector-versatiles-colorful-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/vector-versatiles-colorful-dark.png"
width="260" alt="Vector: VersaTiles Colorful Dark
(dark)"></a><br>VersaTiles Colorful Dark (dark) | | |

#### Hillshade overlay

| | | |
|---|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-light.png"
width="260" alt="Overlay: hillshade over Sentinel-2
(light)"></a><br>hillshade over Sentinel-2 (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-light.png"
width="260" alt="Overlay: hillshade over Carto
(light)"></a><br>hillshade over Carto (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-s2-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-s2-dark.png"
width="260" alt="Overlay: hillshade over Sentinel-2
(dark)"></a><br>hillshade over Sentinel-2 (dark) |
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/overlay-hillshade-carto-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/overlay-hillshade-carto-dark.png"
width="260" alt="Overlay: hillshade over Carto (dark)"></a><br>hillshade
over Carto (dark) | | |

#### Repeater Planner

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-light.png"
width="260" alt="Planner: Sentinel-2 + hillshade
(light)"></a><br>Sentinel-2 + hillshade (light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/planner-s2-hillshade-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/planner-s2-hillshade-dark.png"
width="260" alt="Planner: Sentinel-2 + hillshade
(dark)"></a><br>Sentinel-2 + hillshade (dark) |

#### Layer picker

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-light.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-light.png"
width="260" alt="Layer picker open (light)"></a><br>Layer picker open
(light) | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/picker-dark.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/picker-dark.png"
width="260" alt="Layer picker open (dark)"></a><br>Layer picker open
(dark) |

#### Custom tile server

| | |
|---|---|
| <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-map.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-map.png"
width="260" alt="Custom raster provider on the map"></a><br>Custom
raster provider on the map | <a
href="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/custom-picker.png"><img
src="https://raw.githubusercontent.com/zfouts/CoreScope/35ce760b31b8365e511884fdc5e5ae873f4e28bb/map-providers-2428/thumbs/custom-picker.png"
width="260" alt="Custom raster provider in the picker"></a><br>Custom
raster provider in the picker |

## Testing

The full suite ran twice, each time on a fresh clone:

| | Pass 1 | Pass 2 |
|---|---|---|
| Commit (before rebase) | `8a56e841` | `9355169b` |
| `test-all.sh` (unit) | pass | pass |
| `npm run lint` | 0 errors (87 warnings, all pre-existing) | 0 errors
(87 warnings, all pre-existing) |
| Go `cmd/server` / `cmd/ingestor` | ok / ok | ok / ok |
| E2E, map-related files (CI fixture) | 44/44 | 44/44 + planner-on run |
| `test-e2e-playwright.js` | 149 passed, 3 skipped | 149 passed, 3
skipped |
| Race stress gate (seeds) | 101, 202: all pass | 303, 404: all pass |
| Stress with planner enabled | pass | pass (seed 505) |
| Gallery | 32/32 | 32/32 |

The rebased equivalents of the pass 1 and pass 2 commits are `fde2552c`
and `e4d5c447`.

**After the first CI run.** The two passes covered only the map-related
E2E files, and CI exposed two problems:

1. `test-issue-1206` failed 7 checks. This was the `unload` bug
described under "Fixes included".
2. The JS unit gate runs without `node_modules`, so the two new jsdom
suites crashed `test-all.sh`. Locally, jsdom was reachable from a parent
checkout, which hid this.

The second problem is fixed in `57720bf3`, following the
`domIntegration` convention in `scripts/non-unit-tests.json`.
`test-map-teardown-races.js` moved to `tests/e2e/`, and the three jsdom
zoom-limit tests split out into
`tests/e2e/test-map-vector-leaflet-zoom.js`, so 30 + 3 = 33 tests are
kept. Both run from E2E shard 2 and take a few seconds. The heavy suites
(race stress, perf, gallery) stay local-only and are not run in GitHub
Actions.

These checks ran on the fixed, rebased tree (`57720bf3`):

| Check | Result |
|---|---|
| `test-all.sh` from a copy with no `node_modules` (as CI runs it) |
pass |
| `test-test-inventory.js` | 399 suites classified |
| `npm run lint` | 0 errors |
| Go `cmd/server` / `cmd/ingestor` | ok / ok |
| `test-issue-1206-resize-observer-leak-e2e.js` | 13/13 (7 were failing)
|
| `test-e2e-playwright.js` | 149 passed, 3 skipped |
| `test-map-providers-e2e.js` / harness E2E | 17/17, 16/16 |
| `test-map-teardown-races.js` / `test-map-vector-leaflet-zoom.js` |
19/19, 3/3 |
| Race stress (3 iterations) | all scenarios pass |

The commits since pass 2 are `76c67184` (docs only), `7f8d792c` and
`57720bf3`.

**GitHub Actions on `0506fc1d` (rebased onto `master`, with the two-tab
fix): every check passed** (13 passed, 4 skipped as usual for a PR).
That covers Go build and test, all three E2E shards, the server race
detector and both image builds. The moved suites ran in shard 2
(`test-map-teardown-races.js` 19/19, `test-map-vector-leaflet-zoom.js`
3/3), and the unit gate ran `test-map-vector.js` (30/30).

**Race stress suite** (`tests/e2e/test-map-race-stress-e2e.js`). Each
gate run does 20 iterations of scenarios a to e and 50 cycles of f. The
scenarios cover:
- rapid provider switching;
- theme flips during a vector load;
- navigating away mid-load (including a 15 s MapLibre timeout);
- repeated map-page navigation with heap checks;
- overlay toggles;
- cross-page picker persistence.

Results for each run:
- Listener counts stay flat: at most 3 `mc-tile-provider-changed`
listeners and 5 `storage` listeners.
- At most one live WebGL context at a time.
- Heap growth in the second half of scenario d was 0.25 to 0.45 MB
across the four gate runs.
- Upstream `master` grows about 0.39 MB per map navigation loop in the
same scenario; this branch levels off.

## Maintainer review (addressed)

1. **Licence default.** All five new groups (`eox`, `osmfr`,
`openfreemap`, `versatiles`, `hillshade`) are opt-in: off unless
`enabled: true` (`"true"`, `1` or `{}` don't count). With an untouched
config the picker lists the same 6 base maps as before. Docs,
`config.example.json` (now `enabled: false` for each) and the changelog
say so, and each group's terms are listed before you turn it on.
2. **Saved pick.** The on-map picker now saves the pick (option 1 of
your two), as the docs promised. Behaviour, and what an other-theme pick
does:
- Every picker opens on the pinned map, on any page and after a reload,
until **Auto** is picked. A pin is shown whatever the theme, as an
explicit pick already was.
- Maps without a picker show it when it fits the current theme (`type`
is the theme or `any`), otherwise the theme's map. So a light pin isn't
drawn on a dark node map; satellite always is.
- A stored pin whose group is turned off reads as Auto, and comes back
if the group is turned on again; only picking Auto clears it.
- On a map with a picker, Auto always means the theme's map: the pin is
only the explicit layer, never loaded into the hidden Auto layer. A
choice in the customizer, and Reset All, clear the pin. A failed write
(quota full) never brings back an older stored pin.
- E2E `7a` to `7c` in `test-map-providers-e2e.js`: pick, reload, the
radio and tiles are the pick, the node detail map draws it, and a light
pin in dark is kept but not drawn. Auto clears the key; junk in it gives
Auto; an untouched config lists none of the new groups. 7a and 7b fail
with the save line removed. Unit tests are in `test-map-tile-pin.js`
(21).
3. **CI.** `test-map-providers-e2e.js` (24 checks, including `4b`, the
two-tab regression check) and `test-map-providers-harness-e2e.js` (16)
now run in E2E shard 3 against the fixture server: every tile host is
stubbed.

Non-blocking:
- `getTileUrl()` now returns the raster fallback for a vector pick
(`MC_rasterTileUrl`). There's a test in `test-tile-resolver.js` that
fails on the old code.
- The `map-vector.js` header and the binding's vendor README list the
Leaflet and binding internals to recheck on an upgrade.
- `docs/deployment.md` says SRI makes a JS-rewriting proxy drop vector
maps to raster, so serve `public/vendor/` unmodified.
- `scripts/map-perf/README.md` says where `SEED_DB` and `SHIFT_SH` come
from.
- The maps with `attributionControl: false` are left as they are. That
predates this PR, and with every new group opt-in, an upgrade adds no
uncredited imagery. Follow-up for all of them: #2441.
- The OpenFreeMap credit line is `OpenFreeMap © OpenMapTiles Data from
OpenStreetMap` (`2424f3c3`), the line OpenFreeMap gives for clients that
don't add it automatically.

Verified after these changes:
- `test-all.sh` and lint pass, with 0 errors.
- Both provider E2E suites pass: 24/24 and 16/16.
- Race stress, seed 979: scenarios a, b, d and e pass. c (15/15) and f
(37/37) pass after updating their model of the pick: they expected Auto
after a return, which was the old unsaved behaviour.
- The stress suite now enables the five groups itself, so it keeps
exercising them.

## Performance

A/B test: `master` against this branch on the large synthetic dataset,
with 9 runs per scenario and stubbed tiles (see Test data;
`scripts/map-perf/`). The full HTML report is not attached; ask if you
want it.

- **No budget failures** in either pass. The `raster-no-maplibre` budget
holds: no raster scenario loads MapLibre.
- **DOMContentLoaded is unchanged on every page**, even with the two new
scripts (`map-hillshade.js` and `map-vector.js`, about 635 lines in all;
MapLibre itself loads only on demand).
- **Switching providers 50 times (raster only), median ms per switch:**
  - dark: 146 → 108.5;
  - light: 208 → 208.5.
  - Heap growth during the run: 0.3 to 0.4% on both sides.
- **Flagged and checked:**
- Live and Analytics `loadEventMs` are flagged in some runs. This metric
has two modes on **both** branches: `load` fires either before Leaflet
adds its first tile `<img>`s (about 150 ms) or after they finish (about
400 ms). A direct probe of `master` alone gave 428, 404 and 124 ms. On
these scenarios, map creation, first tile, all tiles, first marker and
wall time are within a few ms on both sides.
- Carto and Positron `allTilesSinceMapMs` are 4 to 6 ms slower, just
above `master`'s p95. Tiles are stubbed locally, so this is not network
noise. The cause is not isolated. Both providers now go through the
shared layer builder, and on the same scenarios first tile, all tiles
and wall time are within 10 ms of `master` (Carto Voyager +2 to +3 ms;
Positron Dark +6 to +9 ms).
- The Analytics map shows one long task of about 50 ms against a
`master` median of 0, but `master`'s p95 is 52 to 84 ms. This is
disclosed in case it shows up for you.
- Other desktop apps were running during the perf runs, so expect some
noise.

## Commits

1. `c6bf12ed` feat(map): tile provider registry with custom servers,
gray, topo and satellite maps, and overlays
2. `9237230b` feat(map): client-side hillshade overlay
3. `781e41b4` feat(map): vector basemaps via vendored MapLibre GL 5.24.0
4. `dae36d2e` refactor(map): every map surface uses the shared basemap
builder
5. `3d4afcdc` fix(map): teardown races and listener leaks when leaving
map pages
6. `b798b2fe` test(server): pin map.tiles passthrough in
/api/config/client
7. `171b1bc4` test(map): E2E, race stress suite, perf harness and
screenshot gallery
8. `61f73875` docs(map): providers, custom tile servers, overlays and
self-hosting
9. `135a9b12` test(map): gallery shows opt-in OpenTopoMap and retries
slow tile servers
10. `3833572b` test(map): harness waits for vector basemaps; stress
suite fixes from test pass 1
11. `3e543cad` test(map-perf): redact tile-URL credentials; compare
switch50 like with like
12. `d5006111` docs(map): user guide for map styles, overlays and
map.tiles
13. `90ee1d5d` fix(map): MC_safeRemoveMap lets unload listeners run
before dropping them
14. `5fc57b5c` test(map): run the jsdom map suites in the E2E job, not
the unit gate
15. `bd36b634` fix(map): another tab's overlay pick that lands mid-click
is no longer removed
16. `38df65d2` fix(map): new tile groups are opt-in; a map picked in the
picker is saved
17. `eafbcac9` ci(e2e): run the map provider E2E suites in shard 3
18. `65b223ec` fix(map): getTileUrl() never returns a vector style url;
review notes
19. `f3402c5d` test(map): race stress enables the opt-in groups and
expects the saved pick
20. `e92b8631` fix(map): Auto on a picker map ignores the saved pick;
customizer clears it

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 22:26:46 +01:00
AdamandAdam 49dd4ee773 List my claimed repeaters on My account (#2430)
Follow-up to #2415, see #2414.

## Situation

A user can claim repeaters, but no page lists the ones they claimed.
Claims show on each node page ("Maintained by"), in the admin user
detail and in the "Download my data" export. A maintainer of several
repeaters has to remember them and open each node page to see whether
one went quiet.

## Change

**`GET /api/account/repeaters`** (browser session; device tokens answer
403) returns the caller's claims. For each one:

- `pubkey` and `name`.
- `known`: false once the node left the analyzer database. The claim
stays.
- `role` and `claimedAt`.
- `lastHeard`: the newer of the last advert and the last transmission
certainly from the node, the same sources as the node page's Last Heard.
- `lastRelayed`: when its hop prefix last relayed. Only used to mark a
silent repeater as uncertain, since the prefix can be shared.
- `status`: the saved repeater status, or null.
- `comaintainers`: the other active claimers' display names.

The list is sorted by name, with unnamed repeaters after, by pubkey.
It's built on the existing `RepeaterClaimsByUser`, `NotifyNodes` and
`LastHeardMap`.

**My account, My repeaters**, a new section after Companions:

- Silent repeaters come first. Health uses `getNodeStatus`, so the
threshold matches the node pages and map (`infraSilentMs`):
  - green "heard 2h ago";
  - yellow "last heard …, its prefix still relays";
  - red "silent, last heard 184d ago" or "never heard";
  - grey "no longer in the database".
- Each row links to the node page and shows the saved status ("no
details yet" without one), the claim date and "also maintained by …".
- *Unclaim* asks for confirmation, then uses the existing `DELETE
/api/repeaters/{pubkey}/claim`.
- The empty state says how to claim.
- `#/account?section=repeaters` deep-links to the section, like
`section=notifications`.

Editing stays on the node page, so there's only one editor. The admin
user detail and the export are unchanged.

Docs: `openapi.go` and the Repeater info section of
`docs/user-guide/accounts.md`.

## Notes

- A mobile client will want the same list. #2429 proposes device-token
scopes for repeater claims. If it lands, letting a `repeater:claim`
token read this route is a one-line addition to the bearer route table,
and I'd rather do that there than here.
- The two PRs touch nearby lines in `openapi.go`, `repeater_handlers.go`
(route registration) and `public/account.js`. Whichever merges second
needs a small rebase.

## Tests

- `cmd/server/my_repeaters_test.go`:
  - empty list;
  - ordering (named first, case-insensitive);
  - unheard, heard and departed nodes;
  - saved status;
  - co-maintainers, excluding the caller and disabled users;
  - another user's view;
  - unclaim removes the row;
  - anonymous gets 401 and a device token 403;
  - no analyzer DB.
- `tests/unit/test-user-management-ui.js`: health sort and chips,
escaping, the short-pubkey fallback, status and co-maintainer lines,
Unclaim on every row, and the empty state.
- `tests/e2e/test-user-management-e2e.js`:
  - the claimed repeater appears with its status and node link;
  - axe on the section;
- Unclaim: dismissing the confirm keeps the claim, accepting empties the
list.

Run locally:

- `make test`: all 16 modules OK.
- `sh test-all.sh` and `node tests/e2e/test-lint-network-calls.js`:
pass, run in `node:24-bookworm`.
- `npx eslint public/*.js`: 0 errors, the same 87 warnings as master.
- `test-user-management-e2e.js` against local e2etest servers with the
CI config: 46/46.
- Manually in Chrome, light and dark: heard, silent and co-maintained
rows, and the deep link.

Not run: the full Playwright shard matrix from CI.

---------

Co-authored-by: Adam <196335+anieto@users.noreply.github.com>
2026-11-20 16:25:51 +01:00
efiten 0cf22cdd79 feat: repeater info, claims and community-maintained details (#2415)
Closes #2414
2026-11-17 11:40:05 +01:00
efitenandClaude Opus 5.5 1577b57d67 docs(accounts): approved channels are backfilled since #2133 (#2385)
`docs/user-guide/accounts.md` said that messages received before a
channel proposal is approved stay encrypted. Since #2133 that is no
longer the case: when the key set grows, the ingestor re-decodes the
stored rows it could not decrypt (`cmd/ingestor/key_backfill.go`), and
the server tails `tx_rewrite_feed`, so no restart is needed.

Checked on an instance running #2133: approving four hashtag channels
triggered one pass (`[key-backfill] channel: 4 of 410 key(s) not yet
applied to history`, then `pass complete, 801 row(s) rewritten`, 13 s),
and the 801 rows are exactly the stored messages of those four channels,
the oldest from three months before the approval.

Docs only, one sentence. Not done: I did not check the other docs pages
for the same claim beyond a grep for "stay encrypted" and "received
before" in `docs/` and `README.md`, which found only this one.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 22:31:07 +01:00
efitenandClaude Opus 5.5 e149568c6c feat(observer-logins): personal MQTT logins for observers (#2306)
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.

Closes #2319.

The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.

## Changes in this PR

**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.

**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.

**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.

Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.

**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.

**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.

**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.

**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.

## Enabling it on an instance

Full steps in `docs/observer-mqtt-logins.md`. In short:

1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.

## Who can claim an observer

Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.

What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.

## Performance

`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.

## Verification

- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.

- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.

## Known behaviour

EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.

## Not in this PR

- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.

Note on cb9f4ff3: its message says the scheme problem was reported by an
operator. That report turned out to come from before the broker used the
hook, so it does not show this cause; the change rests on the firmware
source only.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 09:50:20 +01:00
efitenandClaude Opus 5.5 2bb000f2aa feat(notify): adaptive offline threshold and certain node analytics (#2260) (#2262)
Part 2 of #2260. **Builds on #2261** (certain liveness): review/merge
that first; until then this diff also shows its commit.

## Adaptive offline threshold

`userManagement.notifications.adaptiveOffline` — **off by default**:

```json
"notifications": {
  "enabled": true,
  "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 }
}
```

The fixed `infraSilentHours` (72 h) treats every repeater the same. With
this on, a watched node's `node.offline` window is learned from its own
**certain** traffic (the #2261 rule):

1. Certain packets per hour over the week **before its last certain
packet**. Ending there, not at now, keeps an outage from dragging the
rate down until the fixed window returns and the node reads "back
online".
2. Empty runs are classified: ≤ 6 h and recurring on another day at the
same hours = normal silence (kept; the window is never shorter than the
longest + 1 h); any other run = outage, left out of the learning so it
neither stretches the window nor blocks learning.
3. λ = 10th percentile of the remaining hourly counts, so quiet nights
don't alert.
4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson,
ln 10⁶ ≈ 13.8).
5. Clamped to [`floorMinutes`, the role's silent window].

Kept on the fixed window: λ < 1/hour, under 24 h of history in memory,
or a 10th-percentile hour heard by fewer than `minObservers` observers
(one observer dropping out must not read as the node going offline).
Silence is measured up to the newest packet in the store, so a stalling
feed stops every node's clock. A node stored offline comes back only on
evidence from the moment it went offline on. The notifier caches each
learned window for 10 minutes and drops unwatched nodes.

For the #2260 repeater (~840 packets/day) the mail would have gone out
about 30 minutes into the outage, instead of never (or 72 h with #2261
alone).

## Certain node analytics

- Each `activityTimeline` bucket carries `certain`: the part that can
only be this node (`certainFor`, the packet-level form of the
`certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other
endpoints are unchanged.
- Availability and longest silence use certain buckets only; the longest
silence now includes one **still running** (an outage in progress is the
silence worth seeing).
- The chart stacks certain (`--accent`) and uncertain (`--text-muted`) —
existing theme variables, so no new customizer entry. Descriptions
updated.

## Performance

`CertainActivity` walks a week of packets per watched node, at most once
per node per 10 minutes, with the read lock taken **per node**:

```
BenchmarkCertainActivity-16   12.1 ms/op   (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold)
BenchmarkNoteCertainRelays-16  121 ns/op   0 allocs
```

Node analytics adds one `certainFor` per packet of the requested node (a
prefix-map lookup, or the cached decoded JSON for its own packets).

## Tests

- `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet
nights set the rate; > 10 % empty hours, short history, too few
observers → fixed.
- `CertainActivity` ends at the last certain packet and counts only
certain packets — verified red by mutating the window to end at now.
- Notifier end to end: learned window passed → bad; inside → good; too
few observers / setting off → fixed 72 h; cache refresh and prune.
- `certainFor`, `longestSilence` (incl. running silence), config
defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket.
- Frontend: `NodeAnalyticsTimeline.series` (new
`tests/unit/test-node-analytics-certain-timeline.js`).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork.

## Validation

Server on the freshened e2e fixture: analytics buckets carry `certain`
(e.g. 89 of 90 packets certain for one repeater). **No browser
screenshot**: Playwright isn't installed on this machine.

## Later (rule 8)

`floorMinutes` / `minObservers` are server config, not theme settings;
an admin UI for notification settings would be the place, not the
customizer.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 21:20:42 +02:00
efitenandClaude Opus 5.5 a08f2f4e7c fix(liveness): count only evidence that can be the node alone (#2260) (#2261)
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold
follows in a separate PR).

## Problem

A repeater silent since 2026-10-18 showed **Active — Last relayed 4m
ago** and **Last Heard 36m**, and its offline mail never fired. Every
hit after that date was the 1-byte hop `67`, shared by nine nodes:

- the live resolver's affinity tiebreak (`resolvePathForObs`) put those
transmissions in `byNode`, which fed `stats.lastHeard` and
`LastHeardMap`;
- `last_relayed` folds in the 1-byte prefix bucket on purpose (#662),
and drove the status in `roles.js` and the notify offline check.

## Fix

The store keeps `certainHeard` per node: the newest transmission that
can only be that node.

| Evidence | Counts? |
|---|---|
| Own packet (`pubKey`, `srcPubKey`) | ✅ |
| Flood relay hop whose prefix, at the packet's path hash size, matches
exactly one path-capable node | ✅ |
| 1-byte hop shared with other repeaters | ❌ |
| Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌
|
| TRACE (path bytes are SNR) | ❌ |
| Destination (`destPubKey`) | ❌ |

A multibyte repeater legitimately appears as a 1-byte hop (the
originator picks the hash size, `Mesh::routeRecvPacket` appends at that
size), so only uniqueness decides. The rule is applied in every load
path through `indexResolvedPathHops` and the new
`indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live
ingest, late observations.

Liveness reads only that:

- `/api/nodes` and the node detail: `last_heard` is the newer of
`last_seen` and the certain last heard. Before, it was a plain copy of
`last_seen`.
- health / bulk-health `stats.lastHeard`.
- `LastHeardMap` → notify offline check. The separate relay input
(`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed.
- `roles.js getEffectiveHeardMs`: newest known timestamp, without
`last_relayed`. The side pane no longer says "Last relayed"; the Last
Relayed row stays (analytics) with an updated tooltip.
- New status **`uncertain`** (repeaters and rooms): nothing certain
within the silent window, but the shared prefix (`last_relayed`) still
in paths. Grey badge with explanation, an *Uncertain* filter on the
nodes list and the map, dimmed like stale. The offline mail for such a
node says only its shared prefix was still seen.
- A prefix also held by a path-capable node in `inactive_nodes` is not
unique; adverts with `signatureValid: false` are no evidence;
`/api/nodes/search` reports the same `last_heard`.

Attribution for analytics (`byNode`, packet counts, `last_relayed`,
relay counters, usefulness) is unchanged.

## Performance

Hot path: `noteCertainRelays` runs per observation on ingest under the
write lock: two hex bytes of header parsing plus one prefix-map lookup
per resolved hop.

```
BenchmarkNoteCertainRelays-16   10307640   116.2 ns/op   0 B/op   0 allocs/op   (8-hop 2-byte path, 2,000 repeaters)
BenchmarkLastHeardMap-16            1902   0.77 ms/op                            (5,000 nodes; was 3.7 ms scanning byNode)
```

Memory: one string per node ever certainly heard, dropped together with
the node's `byNode` list on eviction.

## Tests

- `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk,
LoadChunked — each with and without resolved_path — live ingest, late
observations) feeds `certainHeard`; each of the 8 call sites verified
red with a nil prefix map.
- `certain_heard_test.go`: hop width per route/payload type; retired
namesake prefix; failed signature; uncertain offline mail; shared vs
unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load
path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`,
detail and health agree.
- Regression for #2260: last certain packet on day X, 20 shared-prefix
hops after it → `LastHeardMap` stays at X and `node.offline` goes bad
after 72 h. Verified red with the uniqueness check mutated out.
- Notify: `TestEvaluateCertainHeard` replaces the relay-input test.
- Frontend: `last_relayed` alone no longer keeps a repeater or room
active; newest timestamp wins (an old `_liveSeen` can't hide a fresh
certain relay).
- `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a
trial merge into a downstream fork with 457 extra commits.

## Validation

Ran the server on the freshened e2e fixture: list, detail and health
report the same last heard; e.g. a repeater whose `last_relayed` is 3
min newer than anything certain keeps its certain time. **No browser
screenshot**: Playwright isn't installed on this machine; CI's E2E
covers the pages.

## Not changed

- The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a
shared hop only when exactly one candidate is adjacent in the neighbor
graph; it did not fire in this case. Left as is.
- The `lastHeard=` list filter still filters `last_seen` in SQL.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 19:11:30 +02:00
CoderNemesisandCoderNemesis d9342bbebd (feat): UserManagement, Add Support for Postal Email Delivery Provider (#2154)
This PR extends email support to add another email delivery provider
called Postal for user management. It is a, open source, self hosted
email delivery and reputation management service.

It adds 2 new additional configuration settings to the
userManagement.mail object which are "postalBaseUrl" and "postalApiKey".
It also adds "postal" as a supported value for the "provider" setting.
All existing functionality including email delivery notification is
supported by Postal.

Included test cases to support the new functionality and all pipelines
pass.

---------

Co-authored-by: CoderNemesis <rfedor@comchan.net>
2026-10-11 11:28:20 +02:00
efitenandClaude Opus 5.5 0737f10a5d feat(notify): optional external node alerts feed and the node.external event (#2177) (#2178)
Closes #2177.

## What

Node notifications can mail alerts computed outside CoreScope. With
`userManagement.notifications.externalAlerts` configured, users can
choose a new event, `node.external`, which is off by default and only
offered when configured:

```json
"notifications": {
  "enabled": true,
  "externalAlerts": { "url": "http://coverage-job/alerts.json", "maxAgeHours": 48, "label": "Coverage" }
}
```

The feed:

```json
{ "generatedAt": "2026-10-11T01:35:12Z",
  "alerts": [ { "pubkey": "<64 hex>", "key": "SE", "text": "never heard to the south-east", "url": "https://example.org/coverage" } ] }
```

Mail lines: `<name>: Coverage: never heard to the south-east, <time>`
when an alert appears, `<name>: Coverage resolved (SE), <time>` when it
is gone from a fresh feed.

## How

- `cmd/server/notify_external.go`: feed types, `parseExternalFeed` (RFC
3339 `generatedAt`, age check, per-alert validation),
`fetchExternalFeed` (200 only, 1 MB cap, 10 s timeout).
- `notify_eval.go`: `evalExternal`. Subjects are `<pubkey>/<key>`; a
`<pubkey>/*` row is the per-node baseline, so the first fresh feed after
watching stores the current alerts without a mail (like `foreign.new`),
and later alerts mail at once. `External == nil` (not configured, nobody
chose it, or stale) leaves every `node.external` state as it is.
- `notify_service.go`: the feed is fetched only when configured and at
least one user chose the event. Each new reason the feed is unusable is
logged once, and so is the recovery. Mail text and link: the alert's
`url`, otherwise the node page.
- `notify_handlers.go`: `availableEvents` and the new `externalLabel`
include the event only when configured; PUT answers 400 for
`node.external` otherwise.
- `internal/users/notify.go`: event constant, `OptionalNodeNotifyEvents`
(canonical order: node, optional, admin), `RemoveWatch` also deletes the
node's `<pubkey>/...` rows.
- Account page label, OpenAPI descriptions, `config.example.json`
comment, `docs/user-guide/accounts.md` section.

Decisions on the three open questions in #2177: the generic name
`node.external`; the resolved mail names the key instead of repeating
the text, so there is **no `users.db` schema change** (a bump would also
make older binaries refuse the file); one feed.

## Invariants

- Read/write separation (#1283): the server only reads the feed over
HTTP. State rows go to `users.db` through `internal/users`, like the
other events.
- A dead, garbled or old feed never mails "resolved":
`TestEvaluateExternalStaleFeedChangesNothing` and
`TestNotifierMailsExternalAlertsAndSkipsAStaleFeed` (HTTP 502 and a 72 h
old feed in a row, then a fresh empty feed resolves).

## Performance

The evaluator adds one pass over the stored states (already read every
tick) plus, per user with the event on and per watched node, the node's
current alerts. Bounded by `maxWatchesPerUser` (50) times the alerts per
node. The feed is fetched once per tick (default 5 minutes) only when
someone chose the event, capped at 1 MB. No packet-store lock is taken
for it. Nothing changes for instances without `externalAlerts`.

## Tests

- `cmd/server`: `go test` green locally (Windows;
`TestSaveGeoFilterPreservesFileMode` skipped as it needs Unix mode
bits). New: `notify_external_test.go` (feed parsing and limits, HTTP
fetch 200/404/oversized, config resolution, evaluator
baseline/new/resolved/returning/stale/not chosen/notifications off,
notifier end to end with a fake feed, account routes).
- `internal/users`: green, 2 new tests (optional event and canonical
order, `RemoveWatch` deletes external subjects only for that node).
- Frontend: `test-all.sh` green, 1 new test in
`test-notifications-ui.js` (label from `externalLabel`, escaped).
- `scripts/check-xss-sinks.sh --file public/notifications.js`: clean.

## Not done

- No E2E test: the account page change is one label, covered by the unit
test.
- Not run against a live instance yet. The first feed in use is the
coverage job of on8ar.eu (`alerts.json`, 29 alerts at the moment).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:54:35 +02:00
efitenandClaude Opus 5.5 c2a590b741 feat(companions): linked-only ingest, account Companions and My coverage (#2176)
Part 3 of 3 for #2173 (companion linking): the optional linked-only
ingest filter, the account page and the "My coverage" toggle, plus the
docs. Builds on #2174 and #2175 (both merged); `upstream/master` is
merged in, so the diff shows only this part.

## Changes in this PR

**Ingestor: linked-only ingest** (`cmd/ingestor/linked_companions.go`)
- With `clientRxCoverage.requireLinkedCompanion` on (default off) and
user management enabled, every `meshcore/client/<pubkey>/...` message
whose pubkey is not in `companion_links` is dropped before any client
handler runs, and counted (`client_unlinked_dropped` in the stats file).
- The ingestor opens `users.db` read-only (`mode=ro`, raw SQL, no
`internal/users` import), like the approved-channels reader, so the
#1283 write invariant holds. `TestLinkedCompanionSetIsReadOnly` pins it.
- The set refreshes every 60 s. A miss re-reads at most once per 5 s, so
a companion linked a moment ago is accepted within seconds while a flood
of unknown pubkeys costs one read per 5 s.
- It passes everything until its first successful read, and when
`users.db` has no `companion_links` table yet (a server older than
this). After the first read it keeps the last set on a read error. It is
a filter, not a security boundary: anyone with the shared broker
credential can still publish under a linked pubkey.

**Frontend**
- Account page: device tokens show under Devices as "CoreDrive RX" with
their label; a new Companions section lists linked companions (name,
short key, linked since, last seen) with an Unlink button.
- Coverage page (`#/rx-coverage`): a "My coverage" toggle for logged-in
users (`mine=1`). Only the newest coverage reply is drawn, and a refused
`mine=1` turns the toggle off. Under "My coverage" the page does not ask
for the "nothing received" cells from #2172, and their legend entry is
hidden.

**Docs**: user guide (accounts: devices, companions),
`docs/client-rx-coverage.md` (linking, linked-only ingest, the CORS
exception), `config.example.json`, and the spec status.

## Performance

The filter runs on every client message. Benchmarks with 1000 linked
companions (12-core dev box, `go test -bench`, 3 runs each):

| Path | Time | Allocations |
|---|---|---|
| linked pubkey (lock-free map lookup) | 54 ns/op | 0 |
| linked pubkey, parallel | 11-14 ns/op | 0 |
| unlinked pubkey inside the 5 s miss gap | 85-95 ns/op | 0 |
| one `users.db` re-read | 0.38 ms | 3030 |

The unlinked benchmark asserts that no extra `users.db` read happens. A
re-read happens at most once per 5 s on a miss and once a minute from
the refresh loop.

## Verification

- `cmd/ingestor`: 15 new tests (filter off changes nothing, linked
passes, unlinked dropped before every handler, miss refresh accepts a
new link, the 5 s cap holds under a flood, unlink honoured after the
periodic refresh, missing table and missing file, fail-closed only after
the first read, read-only open, startup, stats counter) and 4
benchmarks. Full ingestor suite green.
- Frontend unit tests for the account page (device tokens, companions
list escaping, empty state, Unlink confirm and cancel, load errors) and
`tests/unit/test-rx-coverage-mine.js`.
- E2E `tests/e2e/test-user-management-e2e.js` gains a companion step
(device token under Devices, a companion linked with a real Ed25519
signature under Companions, Unlink). Run locally against the e2etest
build: 28/28. The coverage suites (`test-rx-coverage-gaps-e2e.js`,
`-noise-`, `-viewport-`) still pass.
- Frontend unit suite (`test-all.sh`) green.

## Not in this PR

- Narrowing the "nothing received" cells to the caller's companions
under "My coverage" (the track query takes one companion today).
- Per-user broker credentials or ACLs, which would turn the filter into
an access control (non-goal in the spec).

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 18:38:27 +02:00
efitenandClaude Opus 5.5 2456762179 feat: account data export and users.db backup (#2141)
A small follow-up to parts A to E of #2128: users can download all data
the instance holds about them, and the server backs up `users.db`.

Stacked on #2140. Review the commits after that branch.

## The situation

- An instance with accounts holds personal data, but a user has no way
to get a copy (GDPR articles 15 and 20). Self-delete already exists.
- `users.db` is the only file the server writes, and nothing backs it
up: `GET /api/backup` snapshots the analyzer database only.

## What this PR adds

**Data export.** `GET /api/account/export` and a "Download my data"
button on the account page give one JSON file with:
- the profile, including activation and a pending email change;
- sessions (times and user agent);
- synced settings, own proposals, notification prefs and watches;
- audit rows where the user is actor or target, with other accounts as
an id only;
- the mail log.

Password hash, session and link tokens and the unsubscribe token are
left out: they are credentials. Each export is audited as `user.export`.

**users.db backup**
- Daily `VACUUM INTO` snapshots through `internal/users`, on by default
when user management is on. They go to `backups/` next to `users.db`,
the 7 newest are kept, the directory is 0700 and the files 0600.
Rotation only touches `users-<timestamp>.db` files and never the
snapshot it just wrote. Config: `userManagement.backup {enabled, dir,
keep}`.
- `GET /api/admin/users-backup`: an admin downloads a fresh snapshot to
keep off the server; audited as `user.backup`.
- `docs/user-guide/accounts.md` gains a restore procedure. It covers
what a restore brings back (accounts deleted after the snapshot, old
passwords, revoked sessions) and how to clean that up afterwards.

Spec:
[`docs/specs/2026-10-08-account-export-and-users-backup-design.md`](https://github.com/efiten/CoreScope/blob/feat/account-export-backup/docs/specs/2026-10-08-account-export-and-users-backup-design.md).

## Verification

- `internal/users` with `-race` and the `cmd/server` suite pass locally,
30 new Go tests. The file-mode tests skip on Windows.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E: 27 of 27 steps locally.
- On our production instance since 8 October 2026; the first snapshot
there was 143 kB.

## Not in this PR

- Importing an export into another instance.
- Off-site upload of the snapshots. The admin download covers that by
hand.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 22:16:49 +02:00
efitenandClaude Opus 5.5 de364a8bb6 feat: mail notifications for watched nodes (user management part E) (#2140)
Part E of #2128: a logged-in user watches nodes and gets one mail when a
watched node goes offline, comes back, or reports a low battery. Admins
can add instance events: a new foreign node, and an observer going
offline or back. This covers the per-user mail part of #775.

Stacked on #2139 (part D, which builds on #2138). Review the commits
after that branch.

## The situation

A sysop learns that a repeater went silent, or that its battery is
running down, only by opening CoreScope. #775 asks for notifications.
Accounts (part A) now give a verified address to mail and a mailer with
delivery status.

## What this PR adds

**Events.** They use the instance thresholds, so a mail does not
disagree with the node page:
- `node.offline`: silent for the role's `healthThresholds` window. Last
heard comes from the packet store, and repeaters and rooms count relayed
traffic (#1598).
- `node.battery`: advert telemetry below `batteryThresholds.lowMv`,
recovered at `lowMv + 100`.
- Admins only: `foreign.new` (once per node) and `observer.offline`.

**Store.** Schema v5 with `notification_prefs`, `notification_watches`
and `notification_state`.

**Server** (only with `userManagement.notifications.enabled`)
- A notifier next to the janitor, every `intervalMinutes` (5). The
evaluator is a pure function of watches, prefs, stored states and node
snapshots.
- No burst after a restart or a new watch: a subject's first evaluation
stores its state without mailing. The loop waits for the startup load.
While ingest is stale (newest packet older than 30 minutes) the offline
checks pause, and after recovery they wait one silent window.
- One mail per user per check. Limits: 20 per user and 100 per instance
per rolling 24 hours, 50 watches per user. A change over a limit is
recorded and never mailed later.
- Every mail has a one-click unsubscribe (`List-Unsubscribe` and
`List-Unsubscribe-Post`). The GET only redirects to a confirm page, so
link scanners cannot unsubscribe anyone. Node names are cleaned of
control and bidi characters before they go into a mail.
- Routes: `GET`/`PUT /api/account/notifications`, `PUT`/`DELETE
/api/account/notifications/watches/{pubkey}`, `POST
/api/account/notifications/watch-my-nodes` (copies the synced "my nodes"
list), `GET`/`POST /api/notifications/unsubscribe`. `mailer.Message`
gains `Headers`.

**Frontend.** A "Notify me" toggle on the node pages, a Notifications
section on the account page (`#/account?section=notifications`), an
unsubscribe page, and the notification mail count on the admin overview.

Spec:
[`docs/specs/2026-10-07-node-notifications-design.md`](https://github.com/efiten/CoreScope/blob/feat/notifications/docs/specs/2026-10-07-node-notifications-design.md).

## Performance

Each check reads watches, prefs and states from `users.db` (one query
each), the watched nodes from the analyzer DB in chunks of 500, and
last-heard times from the packet store.

| Measurement | Result |
|---|---|
| One check on our staging instance, 130,000 packets in memory, 1
watcher | 6.9 ms, of which 0.93 ms holding the store read lock |
| Evaluator benchmark, 100 users with 50 watches each, 2,000 nodes |
1.75 ms |

No work is added to ingest, broadcast or any request path.

## Verification

- `internal/users` and `internal/mailer` with `-race`; the `cmd/server`
suite plus `-race` on the notifier tests; 74 new Go tests. The same
Windows-only failure as noted in #2138 applies.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E: 26 of 26 steps locally. With notifications off
the new steps fail, so they do test the flag.
- On our staging and production instance since 7 October 2026.

## Not in this PR

- Other channels (Discord, Telegram, webhooks). Detection is separate
from delivery, so one can be added.
- The topology, RF and anomaly alerts of #775.
- If the whole server starts after a feed outage that already ended, the
grace window is unknown and a watcher can get one wrong offline mail.
The user guide says so.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 22:02:39 +02:00
efitenandClaude Opus 5.5 abbe6846ff feat: propose and approve hashtag channels (user management part D) (#2139)
Part D of #2128: logged-in users propose a hashtag channel, an admin
approves, rejects or later revokes it, and the ingestor decrypts
approved channels without a restart. It is the first consumer of a
generic proposal table, and answers the request in #2092.

Stacked on #2138 (part C). Review the commits after that branch; GitHub
shows C's commits here until it is merged.

## The situation

- An instance decrypts only the hashtag channels listed in
`hashChannels`. Users who want a city or club channel shown have to ask
the operator, who edits `config.json` and restarts the ingestor.
- #2092 asked for a suggest-and-approve flow; a downstream fork
(dborup/CoreScope#99) runs one with anonymous suggestions and a file
queue.

## What this PR adds

**Name rules (`internal/channel`).** `ValidateHashtagName`, mirrored in
the browser:
- at most 31 UTF-8 bytes including `#` (firmware
`ChannelDetails.name[32]`), case preserved, `Public` refused in any
case;
- control, bidi, separator and format characters refused except ZWJ,
plus invisible fillers (`Other_Default_Ignorable_Code_Point`, U+2800,
non-ASCII spaces).

Go and JS give the same verdict for every code point.

**Store.** Schema v4, one generic `proposals` table (kind, subject,
status pending/approved/rejected/revoked, proposer, reviewer, note). One
row per (kind, subject); limits are checked in the same transaction as
the change.

**Server** (only with `userManagement.channelProposals.enabled`)
- `POST /api/proposals`, `GET /api/account/proposals`, `GET
/api/admin/proposals`, `POST
/api/admin/proposals/{id}/approve|reject|revoke`.
- 400 for an invalid name. 409 for a duplicate, a rejected name, or a
name already in `hashChannels`/`channelKeys` ("this channel is already
decrypted on this instance"). 429 over a limit (`maxPending` 100,
`maxApproved` 128, `perUserPerDay` 5).
- `GET /api/channels` gains `approvedChannels`, so approved channels are
listed before they have traffic. With the feature off the response is
byte-identical.

**Ingestor.** Opens `users.db` with `mode=ro` every 60 seconds,
re-validates the names, and swaps its key map through an
`atomic.Pointer`. Configured channels always win, and a failed read
keeps the last good set. The ingestor never writes `users.db`; AGENTS.md
now says so.

**Frontend.** "Propose for everyone" in the add-channel dialog, an
"approved" marker in the channel list, "My proposals" on the account
page, and an admin "Proposals" tab with a warning on approve. The packet
detail pane now escapes the channel name: until now only the operator
chose those names.

Spec:
[`docs/specs/2026-10-07-channel-proposals-design.md`](https://github.com/efiten/CoreScope/blob/feat/channel-proposals/docs/specs/2026-10-07-channel-proposals-design.md).

## Performance

Every key is tried on each GRP_TXT that no key opens (`decodeGrpTxt`).
Benchmark of that worst case:

| Keys | Time per packet |
|---|---|
| 320 configured | 204 to 216 µs |
| 320 configured + 128 approved | 272 to 320 µs |

That is about +33%, linear in the key count and capped by `maxApproved`.
With the feature off the decoder gets the identical map and no ticker
runs.

## Verification

- `internal/channel`, `internal/users` (with `-race`), `cmd/server` and
`cmd/ingestor` pass locally, 53 new Go tests. The ingestor's
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced` needs symlink rights on
Windows and fails on `master` too.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E: 21 of 21 steps locally (propose, approve, the
channel appears, revoke removes it).
- On our staging and production instance since 7 October 2026.

## Not in this PR

- Private (PSK) channels: their keys stay in the browser (#725).
- Anonymous proposals.
- Names that need ZWNJ (some Persian and Urdu spellings) and subdivision
flags are refused, because ZWNJ and tag characters are format
characters. Allowing them later only widens the rule, so no approved
name gets stranded.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 21:51:42 +02:00
efitenandClaude Opus 5.5 18a634c115 feat: admin dashboard with overview and audit log (user management part C) (#2138)
Part C of #2128: an admin area with an overview, the user table and a
global audit log. Off by default like the rest of user management: with
`userManagement` off nothing changes, and non-admins get no new UI.

This is the first of 4 stacked PRs (C, D, E, then a small export/backup
follow-up). Each later one contains this branch; review them in order.

## The situation

- An admin could manage users one by one in `#/admin/users`, but nothing
showed whether the instance needs attention: accounts stuck in
activation, bouncing mail, someone guessing a password, an MQTT source
that dropped.
- The audit log existed (`internal/users/audit.go`) but could only be
read per user, and logins were not recorded.

## What this PR adds

**Store (`internal/users`)**
- Schema v3: an index on `audit_log(at)`.
- `AuditList` with filters (action or group prefix like `user.login.*`,
user as actor or target, period) and keyset pagination; `PruneAudit`;
`Stats` for the user figures.

**Server**
- `GET /api/admin/stats` (typed struct): accounts by status, admins,
registrations and active users over 7 and 30 days, logins and failed
logins in 24 hours, mail by final status, and the attention items
computed server-side.
- `GET /api/admin/audit`: filtered, newest first, `next` cursor.
- `GET /api/admin/users` gains `bouncing=1`.
- Logins are audited as `user.login` and `user.login.failed` (reason
`wrong_password`, `pending` or `disabled`). An unknown address writes no
row. The writes are asynchronous, so the login response does not wait on
`users.db`. Login rows are pruned after 90 days.

**Frontend**
- `#/admin?tab=overview|users|audit`: `admin.js` (tab shell),
`admin-overview.js` ("Needs attention", Users card, System card from the
existing health, MQTT and observer endpoints), `admin-audit.js` (filters
in the URL, "Load more"). `admin-users.js` becomes the Users tab; the
old `#/admin/users` link rewrites to it.
- `/api/healthz` is read on open and on Refresh only, not on the
60-second timer, because it walks every packet under a read lock.

Spec:
[`docs/specs/2026-10-07-admin-dashboard-design.md`](https://github.com/efiten/CoreScope/blob/feat/admin-dashboard/docs/specs/2026-10-07-admin-dashboard-design.md).

## Verification

- `internal/users` and `cmd/server`: `go vet` and `go test` pass
locally, 26 new Go tests. One upstream test,
`TestSaveGeoFilterPreservesFileMode`, also fails on Windows on plain
`master` (file modes) and is unrelated.
- `sh test-all.sh` exits 0; the XSS gate in diff mode passes.
- User-management E2E with the `e2etest` build: 13 of 13 steps locally.
- Running on our staging and production instance since 7 October 2026.

## Not in this PR

- Restricting existing pages (perf, MQTT status) to logged-in users or
admins. The admin area adds no access mechanism of its own, so that
stays a later router rule plus endpoint check.
- Server-enforced customizer tab restrictions (#1508).
- The 24-hour, 5-attempt and 10-minute attention thresholds are
constants for now (AGENTS.md rule 8: customizer later).

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 21:38:30 +02:00
efitenandClaude Opus 5.5 f3ae8ac25a feat: sync a logged-in user's settings across devices (part B) (#2130)
Part B of #2128: a logged-in user's settings follow them across devices.
Log in on a phone and your own nodes, favorites, customizer and filters
are there; a change on one device reaches the others within a minute or
when you return to the tab.

**This PR builds on #2129.** Until that one is merged, the diff here
includes it. The commits for this part start at `docs(specs): settings
sync for optional user management (sub-project B)`.

## The situation

Everything a visitor sets up lives in one browser's `localStorage`
(about 100 keys in `public/`). A second device or a cleared cache starts
from zero (#895).

## What this PR adds

**Storage.** `users.db` schema v2: one JSON document per user in
`user_settings`, with a revision number and a generation id. A write
succeeds only when the client's revision and generation match the stored
ones, so two devices cannot overwrite each other silently.

**Server.** `GET`, `PUT` and `DELETE /api/account/settings`, behind the
same session and CSRF checks as the account routes.
- The server owns the list of synced keys (61 keys,
[`settings_allowlist.go`](https://github.com/efiten/CoreScope/blob/feat/settings-sync/cmd/server/settings_allowlist.go))
and sends it to the client, so the two cannot drift.
- A hard denylist, checked first, refuses `meshcore-api-key`, every
`corescope_channel_*` key and `live-channel-colors` (#725). The colour
map is keyed by channel hash, and for a user-added channel that hash is
`user:<name>`, which would expose hashtag channel names.
- Documents are capped at 256 KiB, measured like `JSON.stringify`. PUT
is limited to 60 requests per hour per user. A stale revision gets 409
with the current document.

**Client**
([`settings-sync.js`](https://github.com/efiten/CoreScope/blob/feat/settings-sync/public/settings-sync.js)).
Inert unless the feature is on and someone is logged in.
- It wraps `localStorage.setItem` and `removeItem` for allowlisted keys
only and pushes 2 seconds after the last change.
- It pulls on login, page load, tab focus and every 60 seconds while the
tab is visible.
- **Merge:** three-way, against a per-device baseline that belongs to
one user and one document generation. Lists (own nodes, favorites, saved
filters) merge per item, so an item added anywhere is kept and an item
removed on one device does not come back from another. Single values:
the profile wins unless only this device changed it.
- Remote changes are written without a push, theme and colour-blind
preset are re-applied, and the current page re-renders (skipped on
account pages and while the geofilter editor is open).

**UI.**
- Logout asks: keep my settings on this device (default), remove them
from this device, or cancel. Channel keys are never removed: no copy
exists anywhere else.
- The account page gets a "Settings sync" section: last synced time,
"Sync now", what is and is not synced, and "Delete synced settings from
my account".

## Not synced

Layout and device state (panel and column widths, collapsed panels, map
positions, geofilter drafts), channel data (#725), the API key, and all
`sessionStorage`. The full list is in the
[spec](https://github.com/efiten/CoreScope/blob/feat/settings-sync/docs/specs/2026-10-06-user-settings-sync-design.md).

## Performance

- One GET per page load, tab focus and minute while visible; one
debounced PUT per burst of changes.
- The `setItem` wrapper costs one Set lookup per write for non-synced
keys. A synced write reads one small revision key, not the stored
document.
- The server reads or writes one row per request.

## Verification

- `internal/users` and `cmd/server`: `go vet` and `go test` pass locally
(22 new Go tests), including a test that every allowlisted key still
occurs in `public/`, and denylist tests.
- `tests/unit/test-settings-sync.js`: 79 passing (vm, real module). The
cases cover the merge table, two tabs sharing one storage, stale answers
after a push, delete while a push is in flight, and logout while the
final push fails.
- `sh test-all.sh` exits 0.
- `tests/e2e/test-user-management-e2e.js` (10 steps, 4 of them new)
passed locally with two browser contexts as two devices: a favorite and
the packet time window travel from device 1 to device 2, a removal does
not come back, and "remove from this device" clears the synced keys
while a channel key stays.
- Checked by hand on a staging instance with a desktop and a phone on
one account.

## Not in this PR

- On a shared browser where the previous user chose "keep", the next
user's first login merges those settings into their own account. The
user guide says to choose "remove" on shared computers.
- Saved filter expressions are synced as typed, including any channel
names written in them. The guide says so.
- Realtime push between devices; the minute pull is the sync interval.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 11:20:44 +02:00
efitenandClaude Opus 5.5 d232072f85 feat: optional user accounts (part A: foundation) (#2129)
Part A of #2128: optional, off-by-default user accounts. With the
feature off nothing changes; with it on, visitors can register and log
in, and admins manage users and use the operator actions without the API
key.

PR #2130 (settings sync) builds on this one. The two are meant to be
merged together.

## The situation

- Operator actions (geofilter save and prune, backup, perf reset) need
the shared `apiKey`. There is no per-person right.
- Nothing in CoreScope knows who a visitor is, so the requests in #2128
that need that (#1835, #2092, #1508, #730) have nothing to build on.

## What this PR adds

**Two new Go modules**
- `internal/users`: a separate `users.db` (SQLite through
`modernc.org/sqlite`) with users, sessions, single-use tokens, an audit
log and a mail log. Passwords use argon2id.
- `internal/mailer`: a `Mailer` interface with a Brevo client (send,
delivery events, webhook parsing) and an in-memory fake for tests.

**Server (`cmd/server`)**, active only with `userManagement.enabled`
- 24 routes, all documented in OpenAPI under the `users` tag
([`auth_routes.go`](https://github.com/efiten/CoreScope/blob/feat/user-management/cmd/server/auth_routes.go)):
  - auth: register, activate, login, logout, me, forgot, reset;
- account: profile, password, email change with confirmation, sessions,
self-delete;
- admin: list, detail, disable, enable, delete, role, resend activation,
manual activation, mail status refresh;
  - a Brevo webhook, registered only when `mail.webhookSecret` is set.
- `requireAdmin` replaces `requireAPIKey` at the 7 operator call sites:
the API key **or** an admin session. With the feature off it is the old
API-key gate (`TestRequireAdminWithoutUserManagementIsAPIKeyGate`).
- `/api/config/client` gets `userManagement: {enabled: true}` only when
the service started; with the feature off the response is
byte-identical.

**Frontend**
- `auth.js` (header account control, request helper that adds the CSRF
header), `account.js` (login, register, activate, forgot, reset, confirm
email, my account), `admin-users.js` (`#/admin/users`, deep-linked
filters), `account.css` (theme tokens only).
- On phones the top-bar control is hidden, so a conditional entry goes
into the bottom-nav "More" sheet and the nav drawer.
- The customizer geofilter tab and the Perf "Reset stats" button use the
admin session when there is one.

**Config.** A `userManagement` block (`config.example.json`,
[`docs/user-guide/accounts.md`](https://github.com/efiten/CoreScope/blob/feat/user-management/docs/user-guide/accounts.md)).
The Brevo key can come from `CORESCOPE_BREVO_API_KEY`. The server
refuses to start when the block is enabled but incomplete.

## Security choices

- Session cookie `cs_session`: HttpOnly, SameSite=Lax, Secure when
`publicBaseUrl` is https. Every cookie-authenticated state change needs
the `X-CS-CSRF` header and a matching Origin.
- Activation needs the token **and** the account password. Without the
password, an attacker who keeps re-registering a known address could get
the owner to activate an account that carries the attacker's password.
- Register, forgot and email change answer identically for known and
unknown addresses. A password reset ends all sessions, a password change
ends all other sessions, and both end outstanding email-change links.
- Rate limits: login 10 per 15 minutes, register and forgot 5 per hour,
per IP and per address. The bucket count is capped. `trustedProxies`
makes the per-IP limits see real client IPs behind a proxy.
- Server logs carry `#<user id>`, never addresses, tokens or passwords;
mail-provider error texts are redacted before logging.

## Performance

No change to an existing hot path with the feature off. With it on:
- One `users.db` lookup per authenticated request (session by token
hash).
- The admin user table rebuilds its `tbody` on each filter change.
`users.List` caps the result at 1000 rows (`internal/users/users.go`),
which bounds the rebuild.
- `map[string]interface{}` in `openapi.go`: 79 before, 78 after.

## Verification

- `internal/users`, `internal/mailer` and `cmd/server`: `go vet` and `go
test -race` pass locally. 121 new Go tests.
- `cmd/server` with `-tags e2etest`: vet and the e2e hook tests pass.
- `sh test-all.sh` exits 0. `tests/unit/test-user-management-ui.js`: 67
passing (vm, real modules).
- `tests/e2e/test-user-management-e2e.js` (6 steps) passed locally
against an `e2etest` build with the fake mailer and against a
feature-off build. CI builds the `e2etest` binary and runs the suite on
a second server (`deploy.yml`).
- On a staging instance with a real Brevo key: register, activation mail
delivered, activate, admin table, "Refresh status" showing sent,
deferred, delivered, opened and clicked.

## Not in this PR

- Settings sync (#2130), the admin dashboard, approval flows and
notifications (parts B to E of #2128).
- A `requireReadAuth` mode (#1835). Sessions from this PR are what such
a mode would accept.
- Binary size and build time with `modernc.org/sqlite` linked next to
`mattn/go-sqlite3` were not measured. Their driver names do not collide.
#1992 discusses the driver choice.
- No Brevo webhook was configured on staging; delivery status there came
from "Refresh status".

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 09:25:29 +02:00
efitenandClaude Opus 4.8 5c0de8fb41 feat(live): optional "Multibyte only" view filter (#1780) (#1781)
Closes #1780.

## What

Adds an opt-in **"Multibyte only"** toggle to the live map controls.
When ON, packets whose path hash size is `< 2` bytes (single-byte, or
unresolvable) are excluded from the entire live view — feed, map
polylines/rain, and the packet counter — in both LIVE and REPLAY modes.

- **Default OFF** — no behavior change for existing users.
- Persisted in `localStorage` under `live-multibyte-only`.
- Distinct from the existing global "hide 1-byte path hops" toggle: that
filters individual hops within a path at every render site; this filters
whole packets, on the live view only. They share no state.

## How

- **`public/hop-filter.js`** — new pure, dependency-free classifier
`MC_packetHashSize(rawHex, routeType)` returning `1|2|3`, or `0` when
unresolvable. Reads the path-length byte from `raw_hex` (`(pathByte >>
6) + 1`), offset `5` for transport routes (route_type 0/3) else `1` —
mirroring the existing `getPathLenOffset`/`computeBreakdownRanges` logic
in `app.js`. Lives next to the existing `hopByteLen`/`MC_*` family;
`app.js` is untouched (no duplication of the byte math).
- **`public/live.js`** — `groupIsMultibyte(packets)` consumes that
helper; applied at two render-time sites: the top of `renderPacketTree`
(above the counter increment, so the counter reflects multibyte-only)
and inside the `rebuildFeedList` group loop (so toggling re-filters the
buffered feed). Toggle markup + change handler mirror the existing
`liveFavoritesToggle` pattern.

## Why read from `raw_hex` and not the path hops

The hash size is a property of the whole packet and is present even for
zero-hop packets (where there are no hops to inspect), so reading the
path-length byte is correct in all cases. Unresolvable size is treated
as single-byte (excluded when ON) — we only show packets we can
positively confirm are multibyte.

## Performance (hot path)

The filter runs in the packet-render hot path, so: classification is
**O(1) per packet group** — it reads the first resolvable observation's
`raw_hex` (a short hex string, single `parseInt` of one byte) and
short-circuits. No per-packet API calls, no allocation in the loop, no
added O(n²). When the toggle is OFF (default) the check is a single
boolean guard and does nothing else. The buffered-feed re-filter reuses
the existing `rebuildFeedList` pass — no extra traversal.

## Tests

- **Unit** (`test-live-multibyte-filter.js`, 9 cases):
single/2-byte/3-byte classification, transport-route offset,
missing/short/garbage `raw_hex` → 0, whitespace tolerance.
- **E2E** (`test-live-multibyte-only-e2e.js`, Playwright): toggle
present and defaults OFF; ON hides a single-byte packet while a
multibyte one renders; OFF restores it; setting persists across reload.
Registered in the CI live-E2E block in `deploy.yml`.

## Docs

User-guide entry added in `docs/user-guide/live.md`.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 03:56:08 -07:00
317b59ab10 feat: area-based visual node filter — attribute packets by transmitter GPS (#804) (#839)
## Summary

- Adds configurable GPS polygon areas to `config.json`; nodes are
attributed to an area if their last-known position falls inside the
polygon
- New `Area: …` dropdown filter (matching the existing region filter
style) appears on all analytics, nodes, packets, map, and live screens
when areas are configured
- Backend resolves area membership with a 30s TTL cache; area filter
bypasses the 500-node cap on `/api/bulk-health` so all area nodes are
always returned
- Includes a polygon builder tool (`/area-map.html`) for drawing and
exporting area boundaries

## Changes

**Backend**
- `AreaEntry` type + `Areas` config field
- `GetNodePubkeysInArea` DB query + `resolveAreaNodes` (30s TTL,
`areaNodeMu` RWMutex)
- `PacketQuery.Area` + `filterPackets` polygon check
- `?area=` param propagated through all analytics, topology,
clock-health, and bulk-health routes
- `/api/config/areas` endpoint

**Frontend**
- `area-filter.js`: single-select dropdown, persists to localStorage,
cleans up stale keys on load
- Wired into analytics, nodes, packets, channels, map, and live pages
- Live map clears node markers on area change

**Docs & tools**
- `docs/user-guide/area-filter.md` — configuration and usage guide
- `docs/api-spec.md` — updated with new endpoint and `?area=` param
table
- `tools/area-map.html` — polygon builder for defining area boundaries
- Demo areas added to `config.example.json`

## Test plan

- [x] No areas configured → filter dropdown does not appear on any page
- [x] Areas configured → dropdown appears, "All" selected by default
- [x] Selecting an area filters nodes/packets/topology/map correctly
- [x] Selecting "All" restores unfiltered view
- [x] Selection persists across page reloads (localStorage)
- [x] Stale localStorage key (area removed from config) is cleared on
load
- [x] `/api/bulk-health?area=X` returns all nodes in area (no 500-node
cap)
- [x] `/api/config/areas` returns correct list

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
Co-authored-by: openclaw-bot <bot@openclaw.local>
2026-05-21 14:00:15 -07:00
efitenandClaude Sonnet 4.6 51f823bf7e feat: one-click prune nodes outside geofilter (#669 M4) (#738)
## Summary

- Adds `POST /api/admin/prune-geo-filter` endpoint — dry-run by default,
`?confirm=true` to permanently delete nodes outside the current
geofilter polygon + buffer. Requires `X-API-Key` header.
- Adds **Prune nodes** section inside the GeoFilter customizer tab
(write-access only, same `writeEnabled` gate as PUT). **Preview** lists
affected nodes; **Confirm delete** removes them.
- Adds `GetNodesForGeoPrune` and `DeleteNodesByPubkeys` DB helpers.
- Updates `docs/user-guide/geofilter.md` — documents the UI button as
primary workflow, CLI script as alternative.

> **Depends on M3** (`feat/geofilter-m3-customizer`, PR #736). Merge M3
first.

## Test plan

- [x] `cd cmd/server && go test ./...` — all pass
- [x] Customizer GeoFilter tab without `apiKey` — Prune section not
visible
- [x] With `apiKey` + polygon active — Prune section visible
- [x] **Preview** returns list of nodes outside polygon (no deletions)
- [x] **Confirm delete** removes nodes, list clears
- [x] `POST /api/admin/prune-geo-filter` without `X-API-Key` → 401
- [x] `POST /api/admin/prune-geo-filter` with no polygon configured →
400

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-21 03:19:31 +00:00
e460932668 fix(store): apply retentionHours cutoff in Load() to prevent OOM on cold start (#917)
## Problem

`Load()` loaded all transmissions from the DB regardless of
`retentionHours`, so `buildSubpathIndex()` processed the full DB history
on every startup. On a DB with ~280K paths this produces ~13.5M subpath
index entries, OOM-killing the process before it ever starts listening —
causing a supervisord crash loop with no useful error message.

## Fix

Apply the same `retentionHours` cutoff to `Load()`'s SQL that
`EvictStale()` already uses at runtime. Both conditions
(`retentionHours` window and `maxPackets` cap) are combined with AND so
neither safety limit is bypassed.

Startup now builds indexes only over the retention window, making
startup time and memory proportional to recent activity rather than
total DB history.

## Docs

- `config.example.json`: adds `retentionHours` to the `packetStore`
block with recommended value `168` (7 days) and a warning about `0` on
large DBs
- `docs/user-guide/configuration.md`: documents the field and adds an
explicit OOM warning

## Test plan

- [x] `cd cmd/server && go test ./... -run TestRetentionLoad` — covers
the retention-filtered load: verifies packets outside the window are
excluded, and that `retentionHours: 0` still loads everything
- [x] Deploy on an instance with a large DB (>100K paths) and
`retentionHours: 168` — server reaches "listening" in seconds instead of
OOM-crashing
- [x] Verify `config.example.json` has `retentionHours: 168` in the
`packetStore` block
- [x] Verify `docs/user-guide/configuration.md` documents the field and
warning

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
2026-05-01 06:47:55 +00:00
Kpa-clawbotandyou aeae7813bc fix: enable SQLite incremental auto-vacuum so DB shrinks after retention (#919) (#920)
Closes #919

## Summary

Enables SQLite incremental auto-vacuum so the database file actually
shrinks after retention reaper deletes old data. Previously, `DELETE`
operations freed pages internally but never returned disk space to the
OS.

## Changes

### 1. Auto-vacuum on new databases
- `PRAGMA auto_vacuum = INCREMENTAL` set via DSN pragma before
`journal_mode(WAL)` in the ingestor's `OpenStoreWithInterval`
- Must be set before any tables are created; DSN ordering ensures this

### 2. Post-reaper incremental vacuum
- `PRAGMA incremental_vacuum(N)` runs after every retention reaper cycle
(packets, metrics, observers, neighbor edges)
- N defaults to 1024 pages, configurable via `db.incrementalVacuumPages`
- Noop on `auto_vacuum=NONE` databases (safe before migration)
- Added to both server and ingestor

### 3. Opt-in full VACUUM for existing databases
- Startup check logs a clear warning if `auto_vacuum != INCREMENTAL`
- `db.vacuumOnStartup: true` config triggers one-time `PRAGMA
auto_vacuum = INCREMENTAL; VACUUM`
- Logs start/end time for operator visibility

### 4. Documentation
- `docs/user-guide/configuration.md`: retention section notes that
lowering retention doesn't immediately shrink the DB
- `docs/user-guide/database.md`: new guide covering WAL, auto-vacuum,
migration, manual VACUUM

### 5. Tests
- `TestNewDBHasIncrementalAutoVacuum` — fresh DB gets `auto_vacuum=2`
- `TestExistingDBHasAutoVacuumNone` — old DB stays at `auto_vacuum=0`
- `TestVacuumOnStartupMigratesDB` — full VACUUM sets `auto_vacuum=2`
- `TestIncrementalVacuumReducesFreelist` — DELETE + vacuum shrinks
freelist
- `TestCheckAutoVacuumLogs` — handles both modes without panic
- `TestConfigIncrementalVacuumPages` — config defaults and overrides

## Migration path for existing databases

1. On startup, CoreScope logs: `[db] auto_vacuum=NONE — DB needs
one-time VACUUM...`
2. Set `db.vacuumOnStartup: true` in config.json
3. Restart — VACUUM runs (blocks startup, minutes on large DBs)
4. Remove `vacuumOnStartup` after migration

## Test results

```
ok  github.com/corescope/server    19.448s
ok  github.com/corescope/ingestor  30.682s
```

---------

Co-authored-by: you <you@example.com>
2026-04-30 23:45:00 -07:00
efitenandClaude Sonnet 4.6 b7c2cb070c docs: geofilter manual + config.example.json entry (#734)
## Summary

- Add missing `geo_filter` block to `config.example.json` with polygon
example, `bufferKm`, and inline `_comment`
- Add `docs/user-guide/geofilter.md`: full operator guide covering
config schema, GeoFilter Builder workflow, and prune script as one-time
migration tool
- Add Geographic filtering section to `docs/user-guide/configuration.md`
with link to the full guide

Closes #669 (M1: documentation)

## Test plan

- [x] `config.example.json` parses cleanly (no JSON errors)
- [x] `docs/user-guide/geofilter.md` renders correctly in GitHub preview
- [x] Link from `configuration.md` to `geofilter.md` resolves

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:43:19 -07:00
efitenandClaude Sonnet 4.6 1de80a9eaf feat: serve geofilter builder from app, link from customizer (#735)
## Summary

Part of #669 — M2: Link the builder from the app.

- **`public/geofilter-builder.html`** — the existing
`tools/geofilter-builder.html` is now served by the static file server
at `/geofilter-builder.html`. Additions vs the original: a `← CoreScope`
back-link in the header, inline code comments explaining the output
format, and a help bar below the output panel with paste instructions
and a link to the documentation.
- **`public/customize-v2.js`** — adds a "Tools" section at the bottom of
the Export tab with a `🗺️ GeoFilter Builder →` link and a one-line
description.
- **`docs/user-guide/customization.md`** — documents the new GeoFilter
Builder entry in the Export tab.

> **Note:** `tools/geofilter-builder.html` is kept as-is for
local/offline use. The `public/` copy is what the server serves.

> **Depends on:** #734 (M1 docs) for `docs/user-guide/geofilter.md` —
the link in the help bar references that file. Can be merged
independently; the link still works once M1 lands.

## Test plan

- [x] Open the app, go to Customizer → Export tab — "Tools" section
appears with GeoFilter Builder link
- [x] Click the link — opens `/geofilter-builder.html` in a new tab
- [x] Builder loads the Leaflet map, draw 3+ points — JSON output
appears
- [x] Copy button works, output is valid `{ "geo_filter": { ... } }`
JSON
- [x] `← CoreScope` back-link navigates to `/`
- [x] Help bar shows paste instructions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:42:27 -07:00
you cde62166cb docs: v3.5.0 release notes + API documentation across README, deployment guide, FAQ
- Release notes for 95 commits since v3.4.1
- OpenAPI/Swagger docs: /api/spec and /api/docs called out everywhere
- Deployment guide: new API Documentation section
- README: API docs link added
- FAQ: 'Where is the API documentation?' entry
- Test plans for v3.4.2 validation
2026-04-08 06:55:25 +00:00
you 6aaa5cdc20 docs: add user guide — getting started, pages, config, FAQ 2026-04-05 07:09:54 +00:00