mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-22 22:17:57 +00:00
9a57b2fca610d815df80d212f4e1cf70851fc357
153
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
49dd4ee773 |
List my claimed repeaters on My account (#2430)
Follow-up to #2415, see #2414. ## Situation A user can claim repeaters, but no page lists the ones they claimed. Claims show on each node page ("Maintained by"), in the admin user detail and in the "Download my data" export. A maintainer of several repeaters has to remember them and open each node page to see whether one went quiet. ## Change **`GET /api/account/repeaters`** (browser session; device tokens answer 403) returns the caller's claims. For each one: - `pubkey` and `name`. - `known`: false once the node left the analyzer database. The claim stays. - `role` and `claimedAt`. - `lastHeard`: the newer of the last advert and the last transmission certainly from the node, the same sources as the node page's Last Heard. - `lastRelayed`: when its hop prefix last relayed. Only used to mark a silent repeater as uncertain, since the prefix can be shared. - `status`: the saved repeater status, or null. - `comaintainers`: the other active claimers' display names. The list is sorted by name, with unnamed repeaters after, by pubkey. It's built on the existing `RepeaterClaimsByUser`, `NotifyNodes` and `LastHeardMap`. **My account, My repeaters**, a new section after Companions: - Silent repeaters come first. Health uses `getNodeStatus`, so the threshold matches the node pages and map (`infraSilentMs`): - green "heard 2h ago"; - yellow "last heard …, its prefix still relays"; - red "silent, last heard 184d ago" or "never heard"; - grey "no longer in the database". - Each row links to the node page and shows the saved status ("no details yet" without one), the claim date and "also maintained by …". - *Unclaim* asks for confirmation, then uses the existing `DELETE /api/repeaters/{pubkey}/claim`. - The empty state says how to claim. - `#/account?section=repeaters` deep-links to the section, like `section=notifications`. Editing stays on the node page, so there's only one editor. The admin user detail and the export are unchanged. Docs: `openapi.go` and the Repeater info section of `docs/user-guide/accounts.md`. ## Notes - A mobile client will want the same list. #2429 proposes device-token scopes for repeater claims. If it lands, letting a `repeater:claim` token read this route is a one-line addition to the bearer route table, and I'd rather do that there than here. - The two PRs touch nearby lines in `openapi.go`, `repeater_handlers.go` (route registration) and `public/account.js`. Whichever merges second needs a small rebase. ## Tests - `cmd/server/my_repeaters_test.go`: - empty list; - ordering (named first, case-insensitive); - unheard, heard and departed nodes; - saved status; - co-maintainers, excluding the caller and disabled users; - another user's view; - unclaim removes the row; - anonymous gets 401 and a device token 403; - no analyzer DB. - `tests/unit/test-user-management-ui.js`: health sort and chips, escaping, the short-pubkey fallback, status and co-maintainer lines, Unclaim on every row, and the empty state. - `tests/e2e/test-user-management-e2e.js`: - the claimed repeater appears with its status and node link; - axe on the section; - Unclaim: dismissing the confirm keeps the claim, accepting empties the list. Run locally: - `make test`: all 16 modules OK. - `sh test-all.sh` and `node tests/e2e/test-lint-network-calls.js`: pass, run in `node:24-bookworm`. - `npx eslint public/*.js`: 0 errors, the same 87 warnings as master. - `test-user-management-e2e.js` against local e2etest servers with the CI config: 46/46. - Manually in Chrome, light and dark: heard, silent and co-maintained rows, and the deep link. Not run: the full Playwright shard matrix from CI. --------- Co-authored-by: Adam <196335+anieto@users.noreply.github.com> |
||
|
|
11eb72f578 |
feat(planner): Coverage tab, reach of a candidate repeater site (#2406)
The repeater planner (#2391) gets a fourth tab, **Coverage**: put a marker where you are thinking of placing a repeater, set its antenna height, and see on the map who would hear it. It answers the siting question "what does a repeater reach if we put it here", with two layers: handheld users and other repeaters. ## What it shows - **Users layer:** the chance that a handheld at 1.5 m hears the site, in 4 bands (10-25%, 25-50%, 50-75%, 75% and up; below 10% is not painted). The chance comes from a distance curve `P(d) = 1 / (1 + exp((ln d - ln d50) / slope))`, fitted on drive-test receptions of 41 Belgian repeaters (d50 2.7 km, slope 0.30). Terrain only lowers a cell by 2 bands where a 1.5 m receiver sits deeper than 1 first-Fresnel radius under the terrain (`cmd/server/planner_viewshed.go:19-21`). Antenna height does not change the curve: in the drive tests, handheld reach barely changed between 1, 7 and 15 m. The panel says so in one line. - **Repeaters layer:** first-Fresnel clearance from the site to every cell, for a receiver at "Repeaters at" height (default `planner.existingAntennaM`). 3 classes: line of sight (clearance 0% or more), slightly blocked (-20% to 0%), blocked (not painted). Every known repeater within range gets its link computed with the existing planner link code, a dot in its class colour, a line for line of sight and slightly blocked, and an entry in a list. - **Compute button:** changing site, height or range moves the marker and updates the URL hash right away, and dims an existing result with an "Outdated" note. Only "Compute coverage" starts a job. A shared link that carries a site computes once on open. The planner has 1 heavy-job slot shared with Area and allows 6 jobs per minute per IP, so recomputing on every change would burn both. State is in the hash: `#/tools/planner?mode=coverage&site=51.06,4.41&h=10&rx=5&range=30&layer=repeaters`. This builds on the repeater planner @zfouts wrote in #2391: the terrain tiles, the first-Fresnel analysis, the link code, the job protocol and the Area module pattern are all reused here. Thanks for that work, it made this tab a small addition instead of a new subsystem. ## Why the Repeaters classes are not the route LOS verdicts The route verdicts (clear 100%+, clear60 60%+, marginal 0%+, blocked) almost never show on Shuttle Radar Topography Mission (SRTM) terrain at mast heights. At 10 m / 5 m, 30 km around Mechelen, 109 of 70,980 cells reach clear60 or clear. SRTM includes trees and roofs, so the line between 2 masts rarely clears 60% of the Fresnel zone. I checked the clearance against one Belgian repeater (28 m, receivers at 5 m) and the repeaters live data shows it actually hears: | Distance | Heard, clearance >= 0 | Heard, clearance < 0 | |---|---|---| | 0-10 km | 9 of 11 | none in range | | 10-20 km | 3 of 4 | 1 of 8 | | 20-30 km | 0 of 6 | 3 of 35 | | 30-60 km | 1 of 2 | 0 of 84 | Every link heard within 16 km had a clearance between 2% and 54%, so "marginal" is where working links live. The map classes follow that (`repeaterClass`, `cmd/server/planner_viewshed.go:75-89`). Chain, Relay and Area keep the shared verdicts; `losVerdict` is unchanged. This rests on 17 heard links from 1 repeater, which is why the -20% band is narrow and the panel says line of sight stops being enough beyond about 20 km. ## Server - `GET /api/planner/viewshed?site=&h=&rx=&range=` (`cmd/server/planner_api.go:603`, route `cmd/server/routes.go:393`). Same job protocol, result cache, rate limits and opt-in as the other planner endpoints: 404 while `planner.enabled` is false. - One byte per cell: low nibble Users class 1-5, high nibble Repeaters class 1-3, 0 for no data. The grid is about 300 x 300 at every range: 0.2 km cells up to 30 km, `range/150` above (`planner_viewshed.go:47`). - Links skip rooms, stale repeaters and a repeater within 100 m of the site (that is the site itself). They carry no `elev_m` profile. - `truncated` is true only when the 300-node cap may have dropped a repeater within range (`viewshedTruncated`, `planner_api.go:694`). - Tile cap: `Viewshed` in the planner tile limits, sized from the largest offered range (60 km). - New config: `planner.userReach.{d50Km,slope}` with a `_comment` in `config.example.json`. Bounds 0.2-50 km and 0.05-2. - No writes, no new `map[string]interface{}`; response types `PlannerViewshedResponse/Params/Link` in `planner_types.go`. ## Frontend - `public/planner-coverage.js` + `.css`, mounted like `planner-area.js`. `planner.js` gets a `MODULES` table (`public/planner.js:31`) so Area and Coverage share one enter/leave path. - Canvas raster in an `L.imageOverlay`; the layer switch repaints locally without a new request. Requests go through `fetchJob` / `CSApi`. - Colours are CSS variables `--pl-cov-user-2..5` and `--pl-cov-rep-1..3` in `:root` and both dark blocks of `style.css`; the Repeaters ones point at `--status-*`, so a theme that changes those changes the map. ## Performance The job is opt-in and runs in the existing planner job slot, never in a hot path. Measured on real SRTM tiles, local build, 304 Belgian repeaters: | Site | Range | Mast | Job time | |---|---|---|---| | Maasvallei (hilly) | 30 km | 10 m | 2.07 s | | Maasvallei | 30 km | 30 m | 2.57 s | | Maasvallei | 60 km | 10 m | 3.98 s | | Mechelen (flat) | 30 km | 10 m | 2.61 s | On a staging instance (ARM64, cold tile cache) a 30 km job took 6.8 s including the tile downloads. Response size at 60 km with 115 links: 239,942 bytes with link elevation profiles, 149,984 without (what this PR ships). `BenchmarkViewshedMaxLeg` covers the 60 km worst case on a synthetic terrain view. ## Verification - Go: 26 viewshed tests and benchmarks in `planner_viewshed_test.go`, `planner_api_test.go`, `planner_config_test.go` and `planner_tiles_test.go`, among them: flat terrain equals the plain curve and `analyzeLOS`, a ridge shadows only its far side, missing terrain is "no data", the `truncated` rule (proven to fail on the old rule), links without `elev_m`, the repeater at the site left out. `go test ./...` in `cmd/server` passes. - Frontend: `tests/unit/test-planner-coverage.js` (23 tests, in `test-all.sh`): grid decode, class-to-colour mapping, unpainted classes, hash round trip, no fetch on parameter change, 1 fetch per click, deep-link autorun once, stale responses ignored, link classes at the boundaries, escaped repeater names. - E2E: 5 Coverage cases in `tests/e2e/test-planner-e2e.js` (stubbed viewshed): click and compute, layer switch without a request, outdated then recompute, deep link, leaving the tab removes the overlay. Planner E2E 25/25 locally. - eslint 0 errors. Browser: light and dark theme, Users and Repeaters on flat, hilly and coastal sites, deep links, no console errors; also on a staging instance with live data. ## Not in this PR - The `--pl-cov-*` colours are not in the customizer's `THEME_CSS_MAP` yet. They follow `--status-*` for the Repeaters layer; the Users colours need their own entries (AGENTS.md rule 8, later milestone). - In Flanders the Users layer is close to a circle: 0.3% of cells differ from the plain curve around Mechelen, 42% in the Maasvallei, 64% in the Ardennes. The terrain correction is untested in hilly areas, the drive tests had no repeater there. - The Repeaters calibration uses 1 repeater. Other operators with drive tests or neighbour data can tighten it. - A link's class comes from `clearance_pct`, rounded to 0.1%, while the grid uses the raw clearance: within 0.05% of a bound a link can land 1 class off the cell under it. List, line and dot always agree with each other. - An OS dark-mode switch without the theme toggle does not repaint the overlay until the next compute, the same as the planner's tile layer. --------- Co-authored-by: zfouts <zachary.fouts@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
12f9c5c686 |
docs(release): v3.18.0 release notes and changelog entry (#2420)
Release notes and changelog entry for v3.18.0, to be tagged on this PR's merge commit. - `docs/release-notes/v3.18.0.md`: the notes, in the same shape as v3.17.0. The CI/CD Pipeline uses this file as the GitHub release body. - `CHANGELOG.md`: a 3.18.0 entry under `[Unreleased]`. 11 commits since v3.17.0: 5 fix, 3 feat, 1 test, 1 refactor, 1 docs. "Read this before upgrading" covers the `users.db` schema v9 migration with user accounts on (#2415), the WebSocket path `/ws` (#2383) and config tile URLs winning on the Map and Live pages (#2386). Documentation only, no code change. |
||
|
|
0cf22cdd79 |
feat: repeater info, claims and community-maintained details (#2415)
Closes #2414 |
||
|
|
4ebb5ccc11 |
feat: radio line-of-sight on packet routes + repeater planner (#2391)
Both features are OPT-IN ("los" and "planner" config blocks, default
off); with them off nothing is fetched and payloads are unchanged.
Route view
- Multi-path picker discoverability: on-map "N hops shown · tap to see
all" badge, picker pulse (reduced-motion safe), one-time tip, row hover
arrow; "no GPS" hops counted apart from "unresolved".
- Line of sight for every leg of a packet route: clear / marginal /
blocked chips (first-Fresnel clearance, k=4/3), obstruction markers on
the map, docked terrain-profile panel with antenna/frequency inputs and
a &los= deep link. Backed by GET /api/elevation/profile, an Open-Meteo
proxy that only accepts known node positions, with weighted budgets,
back-off and a per-IP limit.
Repeater planner (#/tools/planner)
- Chain: LOS on every leg between any number of points or nodes.
- Relay: best single relay for A→B among known repeaters and open
terrain.
- Area: suggested new repeater sites that best join the existing
repeater backbone in a box up to 100×100 km, picked greedily to merge
clusters, then remove single points of failure, then reach unlinked
repeaters.
- Terrain: SRTM 1° tiles from AWS Open Data, in memory only (~90 m,
bounded LRU); the server stays read-only. Exact-length gunzip, fetch
budget with back-off, missing terrain reported rather than read as sea
level, fuzzed pixel maths.
- GET /api/planner/{profile,relay,coverage}: quantised cache keys, 202 +
progress then cached 200, separate compute slots, per-IP and global
limits, failed jobs remembered, unpolled jobs cancelled. Nodes pass the
same blacklist / hidden-name / geo_filter rules as /api/nodes. The Go
LOS maths matches public/los.js on shared test vectors.
Also: the per-IP client limiter is extracted from elevation.go into
client_limit.go (behaviour-preserving) so both features share it.
Perf (real SRTM): coverage cold 1.5-5.6 s (tile download), warm 10 ms;
worst real compute 545 ms; peak RSS 91 MiB; live /api/nodes and
/api/packets p95 at most 1.6x while a job computes (GOMAXPROCS=2).
Tests: Go suites for elevation, terrain (fuzz), LOS vectors, graph
(brute-force property test), search and planner API; frontend unit
suites in test-all.sh; Playwright e2e for route LOS and the planner.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
|
||
|
|
e7fd48a054 |
feat(frontend): serve the UI from a web container or a static host, three deployment modes (#2383)
## What this does The frontend half of serving the UI apart from the server. The server half was #2362. It gives CoreScope three deployment modes. The server container is the same image in all three, and it still serves everything, the classic UI included. | Mode | The UI is served by | API | Server CORS setting | Logins | |---|---|---|---|---| | All-in-one container (today) | the server container | same origin | none | yes | | Split containers (new) | a web container that forwards the API | same origin | none | yes | | Static host or CDN (new) | Cloudflare Pages, a CDN or any web server | another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI | ### Commit 1: the UI as static files, for any host - **One place for API URLs:** `public/deployment.js` says where the API is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket URL. A lint rule keeps raw `/api` fetches out of the other files. The embedded UI keeps `apiBase: ''`, so it behaves as before, except that its WebSocket now uses `/ws` instead of `/`. - **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`, which is `public/` byte for byte except for the cache buster in `index.html` and `deployment.js`. - It is `dist/web` rather than `dist/` because the Go binaries are built into `dist/` and `make clean` removes that folder. - `API_BASE=same-origin` is for a host that forwards `/api` and `/ws` itself, as the web container does. - **Anonymous build (`AUTH=false`):** for another origin. It has no account features: login, account, admin and other writes link to the same page on the classic UI. It sends no cookies, and its three JSON POSTs go as `text/plain`, so there is no preflight. - **Error notices:** a blocked API answer or an HTML answer each show a notice. - **Channel badge:** `beta`, `next` and `dev` builds show a badge that opens a pre-filled issue, plus an "Open in classic" link. - **Runbook:** `deploy/static/README.md` covers Cloudflare Pages, previews, server CORS and rollback. ### Commit 2: the web container and the three modes - **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It packages the UI with `build.sh` when the image is built, serves it, and forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`). `SITE_ADDRESS` set to a domain name turns on automatic HTTPS. - **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules as `deploy/static/_headers`, and compresses responses with zstd or gzip. - **`docker-compose.split.yml`:** the stock server image (its own Caddy off) plus the web container: `CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f docker-compose.split.yml up -d --build`. - `CORESCOPE_IMAGE` is required and must match the checkout's version, so a release server is never paired with a newer UI by default. - The server's built-in broker, which accepts anonymous clients, is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - **Docs:** `docs/deployment.md` has a new "Deployment modes" section, and `DEPLOY.md` points to it. ### Keeping new code on `CSApi` - **Lint rule:** `.eslintrc.json` now rejects every direct network call in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`, `sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in `href`, `src` and `action` (assigned, set with `setAttribute` or written into markup), `window.open`, `location.assign`, `location.replace` and `location =`. `api-base.js` is the one file exempt. - **HTML pages:** `eslint public/*.js` never sees the inline scripts and markup of `public/*.html`, so `test-lint-network-calls.js` applies the same rule to them, at their page lines. `area-map.html`'s two fetches carry an `eslint-disable` with the reason: its Server field can name another server. - **Why it changed:** the earlier rule only caught a literal `/api…` passed straight to `fetch`. - **Check:** CI already fails on lint errors, and `tests/e2e/test-lint-network-calls.js` pins exactly what the rule reports: 25 bad statements reported, 15 good ones not, `api-base.js` exempt. - **AGENTS.md:** a new "API calls" section covers: - which helper to use for each need; - the lint rule; - how anonymous builds gate login-only features; - loading `api-base.js` in vm and jsdom tests (the region quick-picks suite missed that, and failed shard 1 on the first CI run). ### Review follow-ups - **Account tokens:** the channel badge's pre-filled issue drops the query on `#/account/…` routes, so mail-link tokens never reach a GitHub URL. "Open in classic" keeps the full route, since the classic UI needs the token. - **HTML answers:** on a packaged UI (static host, web container), `api()` rejects an HTML answer before status and retry handling. An HTML 404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server returned an unexpected response" after one request. It used to give a generic error, or six "warming up" retries for a 503. JSON errors and the server's JSON 503 warm-up behave as before. The embedded UI keeps its old handling (`CSApi.embedded`). - **Logins:** `docs/deployment.md`, `Dockerfile.web` and `docker-compose.split.yml` say `userManagement.publicBaseUrl` must be the web container's public origin. ### Audit fixes - **No broken logins build:** `build.sh` refuses `AUTH=true` with a cross-origin `API_BASE`. The server allows no credentialed CORS, so every API request of such a build was blocked; in Chromium all 12 were, and the pages stayed empty. - **No accidental deletes:** `build.sh` refuses an `OUT` that is neither empty nor a previous build. It used to wipe whatever folder or file `OUT` named. - **Lint:** the rule now also reports `/api/` paths in `href`, `src` and `action` attributes in single-quoted strings and template literals. `tests/e2e/test-lint-network-calls.js` pins exactly what it reports. - **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published on localhost only. - **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since v3.16.0 and v3.15.0 predate the static build. It also names the server version the static build needs: before #2362 the warm-up headers stay hidden and wildcard previews are refused. ### Audit notes, fixed - **Cache busting on every page:** the tool pages (`area-map.html`, `geofilter-builder.html`) load their scripts with `?v=__BUST__`. The server and `build.sh` replace it in every top-level page, and `_headers` and the Caddyfile make every page revalidate. Behind Caddy, a returning visitor could otherwise get a new tool page with an old `api-base.js`. - **HTTP/3:** the split compose publishes 443 over UDP too, for the HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`. - **WebSocket on `/`:** a Go test pins that an upgrade on `/` still works, for UIs cached from before `/ws`. ### Second review - **Trusted proxies:** the docs said to trust the compose subnet, and also to publish port 3000 if wanted. Together they let a client spoof its address: the subnet holds the gateway, where a client reaching a published 3000 through Docker's proxy arrives from. The compose file now fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web container's address (`172.30.83.200`), and the docs say to trust that address alone. - **Classic UI unchanged:** the HTML-answer notice above applies to packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as before, and an unknown `/api` path fails to parse as before, with no notice. - **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md says what the rule cannot see: a URL assembled elsewhere and passed in a variable. - **AGENTS.md:** `build.sh` is described as packaging (a copy, `deployment.js` and `__BUST__`, never a code transform), and rule 3 says `__BUST__` is filled in every top-level page. - **CI builds the web image:** the image-check job (no publish) builds `Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`, the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It also validates the split compose file. - **Cross-origin logins:** a hand-edited `deployment.js` with `auth: true` and another origin now logs a warning naming the setting, instead of failing with CORS errors alone. ## Upgrade notes - **WebSocket path:** the UI's live feed connects to `/ws` instead of `/`. The server has answered `/ws` since #1793 and still upgrades `/`, so browsers holding the previous UI keep working. - A reverse proxy that forwards WebSocket upgrades only for an exact `/` must forward `/ws` too. - The nginx example in `docs/deployment.md` (`location /`) already covers it. - **Static UI from another origin:** - The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for the warm-up banner and wildcard preview origins. - `build.sh` refuses `AUTH=true` there, since the server allows no credentialed CORS. - **Split containers:** - `CORESCOPE_IMAGE` is required. - With user accounts on, `userManagement.publicBaseUrl` must be the web container's public origin. - The built-in MQTT broker is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - The compose network is `172.30.83.0/24`, with the web container at `.200`. If Docker reports that the pool overlaps another network, set `SPLIT_NET`. ## Testing - `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with new tests for busting every page and the WebSocket upgrade on `/`. - `sh test-all.sh` passes. New unit suites: - `test-static-build.js`: the build and its input checks, `same-origin` included, cross-origin logins refused, and an unrelated `OUT` left intact. - `test-web-container.js`: the Dockerfile, Caddyfile and compose file agree on packaging, forwarded paths, cache rules and the server port; the server image has no default and MQTT stays on localhost. - The `api-base`, anonymous mode, badge and `dev-static` suites. - `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes 47/47, the four pages included; it needs eslint, so CI runs it on E2E shard 1. - **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes 11/11. That is the static build on one origin against an API-only server on another, which CI runs on E2E shard 1. - **Split containers mode:** - The image builds, and `caddy validate` passes (Caddy 2.11.4). The new CI step passes locally, and fails when the Caddyfile stops forwarding `/api`. - Trusted proxies, with the stack up: the web container gets `172.30.83.200`. With only that address trusted, a spoofed `X-Forwarded-For` sent straight to a published 3000 is ignored, and one sent through Caddy is replaced with the client's address. With the subnet trusted, as the docs used to say, the spoofed address was believed: a deny entry for it rejected the upgrade. - With `docker-compose.split.yml` on the E2E fixture, seeded the way CI seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both through the web container and against the server directly. - Headers, checked with curl: - `no-cache` on `/`, `/index.html` and `/deployment.js`; - zstd on static assets; - `/api/*` proxied (`Via: 1.1 Caddy`); - a `101` WebSocket upgrade on `/ws`. - In Chromium: - every API request and the WebSocket stay on the web container's origin; - there are no page errors and no horizontal overflow at 390 px; - no badge shows on the `static` channel. - Logins through the web container: with user management on and `userManagement.publicBaseUrl` set to the web container's origin, a Chromium run passed every step, all on the web container's origin with no page errors: - registration, and activation from the mail link (which uses `publicBaseUrl`); - the session surviving a reload; - a CSRF-protected `PATCH /api/account`; - logout, then login with the form. - The same login with the backend's origin as `Origin` is refused with 403. The server checks `Origin` against `publicBaseUrl`, not `Host`, which is why the docs ask for it. ## Performance There are no hot-path changes. API calls go through `CSApi.url()`, a string concatenation. In the web container, Caddy serves files from disk and compresses them; API responses pass through with one proxy hop, the same as the all-in-one image's built-in Caddy. ## Notes - **Per-IP limits:** behind the web container, as behind the all-in-one image's Caddy, requests reach the server from the proxy's address. The per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`, the web container's fixed address. The docs say so. - **`depends_on`:** the web container starts with the server, not after its healthcheck, like the all-in-one image's Caddy. Gating on health would make every `up` wait for the first check (30 s) and fail outright when a large database loads for longer than the healthcheck allows; until the server listens, Caddy answers `/api` with 502. - **Not in this PR:** - publishing a `corescope-web` image (CI builds and smokes it, without publishing); - Cross-origin login—including a static frontend and API on sibling custom domains—is outside this PR. In the deployments supported by this PR, account support requires the frontend and API to share one browser origin, either served together or through a reverse proxy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0e23c24847 |
fix(scope-stats): report total and split unscoped by route (#2364) (#2368)
Fixes #2364. ## Problem Since #1838, `GetScopeStats` folds every non-transport transmission (FLOOD 1, DIRECT 2) into `summary.unscoped`, while `transportTotal` and `scoped` still cover transport routes (0, 3) only. The documented `transportTotal = scoped + unscoped` no longer holds, and a client computing an unscoped share against `transportTotal` gets >100% (the reporter shows `transportTotal 7620`, `unscoped 14320`). ## Fix — additive, no existing field changes value `summary` gains: | field | meaning | |---|---| | `total` | all observed transmissions, routes 0–3 (= `scoped + unscoped`) | | `unscopedTransport` | route 0/3 with Code1 = 0000 | | `unscopedFlood` | route 1 — unscoped by protocol | | `unscopedDirect` | route 2 — unscoped by protocol | Invariants (now asserted in `TestGetScopeStats`): ``` transportTotal = scoped + unscopedTransport unscoped = unscopedTransport + unscopedFlood + unscopedDirect total = scoped + unscoped ``` The non-transport count query keeps its single scan; it just splits route 1 vs 2 with `CASE`. On the design question in the issue: rather than changing what `unscoped` means (the Scopes tab and #1842 rely on it), the split lets a rollout view use `unscopedFlood` directly: repeaters apply region rejection to flood routes only (`isRouteFlood()` in `examples/simple_repeater/MyMesh.cpp`), so DIRECT traffic is not subject to flood-region filtering. `docs/api-spec.md` now says so, spells out the route mapping, and notes that `timeSeries` is transport-only (its `unscoped` matches `unscopedTransport`). ## Frontend Scopes tab uses `summary.total` when present and shows `N transport, N flood, N direct` on the Unscoped card; falls back unchanged for older servers. The cards are built by `scopeSummaryCardsHtml(summary)` (exposed as `window._analyticsScopeSummaryCardsHtml`). ## Tested - `go vet ./...` (cmd/server) clean, `gofmt -l` clean - `go test -run ScopeStats ./...` — pass (incl. extended `TestGetScopeStats` and `/api/scope-stats` route test) - `node --check public/analytics.js`; `tests/unit/test-issue-1375-scope-stats-fetch.js`, `test-issue-1979-scope-adverts-by-role.js` pass - new `tests/unit/test-issue-2364-scope-summary-cards.js` (vm sandbox, 11 assertions: shares of `total`, route split, fallback for servers without `total`/split, partial split, empty window); full `test-all.sh` green Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
c2e770ef65 |
docs: archive superseded repeater liveness plans (#2392)
Partial fix for #1385 (PR-E). Move the two April repeater-liveness documents from `docs/superpowers/` into existing `docs/archive/` with `git mv`. Preserve their dated filenames, content and headings; add historical notices clarifying that the proposed `relaying` status/signature is outdated and linking current guidance. Validation: exact commit blobs preserve 41,483 original bytes and 24 headings after removing only notices; four local links resolve; no tracked old-path references; whitespace and PII checks pass. TDD exemption: pure documentation, no behavior change. No runtime, configuration, customizer, test, hierarchy or ignore-rule changes. Application suites were not rerun. Both published Markdown pages and their guidance links render correctly. [Docs-only CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43199869942) and three independent reviews pass. The umbrella issue stays open pending maintainer reconciliation. Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
6dd62711bc |
docs(release): v3.17.0 release notes and changelog entry (#2390)
Release notes for v3.17.0: `docs/release-notes/v3.17.0.md` (used as the GitHub release body by `deploy.yml`) and the `CHANGELOG.md` entry. 24 commits since v3.16.0, counted with #2389, which is merged before this PR. Issues closed come from each PR's `closingIssuesReferences`. Docs only. After merge: tag `v3.17.0` on the merge commit once master CI is green and the `:edge` image carries that commit. |
||
|
|
873e8b0e16 |
feat(rx-coverage): QR code and link to the operator's CoreDrive RX deployment (#2389)
## What An operator who runs a CoreDrive RX deployment (the web and Android companion app that feeds `clientRxCoverage`) can now point visitors of the Coverage page at it. - New optional config key `clientRxCoverage.appUrl`. `/api/config/client` sends it as `clientRxAppUrl`, only while `clientRxCoverage.enabled` is true and the value is an absolute `https` URL with a host name. Any other value is left out and logged once at startup with a `[config]` line. - `#/rx-coverage` shows a panel under the subtitle when the field is present: a QR code of exactly that URL (vendored `qrcode.js`, same call as the node detail QR), the URL as a link, and one line on scanning it in the app or opening it on a phone. Without the field there is no panel. The page still needs no login. - The subtitle named the app "CoreScope-RX" and linked to `github.com/efiten/corescope-rx`, the app's old repository name. It now says "CoreDrive RX" and links to `github.com/efiten/coredrive-rx`, or carries no link when the panel already shows the deployment. ## Why The CoreDrive RX Android app is not tied to one server: on first start it asks for the address of a deployment. Scanning a QR code on the Coverage page is the shortest way for a visitor to get there, including people without an account. Disclosure: CoreDrive RX (`efiten/coredrive-rx`) is my own project. ## Changes - `cmd/server`: `config.go` (`validClientRxAppURL`, startup warning), `types.go`, `routes.go`, `main.go`, `openapi.go`. - `public/rx-coverage.js` (`appPanelHtml`, `appLinkHtml`), `public/roles.js` (`window.MC_CLIENT_RX_APP_URL`), `public/node-reach-coverage.css` (existing CSS variables only). - Docs: `config.example.json`, `docs/client-rx-coverage.md`, `docs/api-spec.md`. ## Tests - 3 new Go tests in `cmd/server/client_rx_app_url_test.go`: URL validation over 15 inputs, the endpoint field in 6 config states, the startup warning. Full `go test ./cmd/server` passes. - `tests/unit/test-rx-coverage-app-qr.js` (real `rx-coverage.js` and `qrcode.js` in a vm): panel present and absent, QR content, HTML escaping, subtitle link in both cases, old names gone. Added to `test-all.sh`. All 6 `tests/unit/test-rx-coverage-*.js` pass. ## Not done - No E2E test. - Browser-checked on a local server with and without `appUrl` in the light theme only. The QR keeps black modules on white in both themes, unlike the node QR, which is inverted in dark mode; dark mode itself was not checked in a browser. - `cmd/ingestor` is unchanged; it does not need the URL. - The same change runs on analyzer.on8ar.eu (a fork build) since 2026-11-10. |
||
|
|
fad303a4f4 |
docs: split the Docker minimum into running and building the image (#2361)
`docs/deployment.md` listed Docker 20.10+ as the minimum, without separating running the published image from building it. The Dockerfile needs BuildKit (`# syntax=docker/dockerfile:1` on line 1, `RUN --mount` from line 71), and BuildKit is the default builder only from Docker 23.0. On 20.10 to 22.x the legacy builder rejects `RUN --mount` unless `DOCKER_BUILDKIT=1` is set. Found while reviewing #2359. Changes: - `docs/deployment.md`: the requirements table gets two Docker rows, 20.10+ to run the published image, and 23.0+ (or 20.10+ with `DOCKER_BUILDKIT=1`) to build it, plus one sentence on why. - `docs/go-migration.md`: that guide builds the image, so its prerequisite becomes 23.0+ (or 20.10+ with `DOCKER_BUILDKIT=1`). Not verified: I did not run a build on Docker 20.10 or 22.x; the legacy-builder behaviour comes from Docker's documentation, not from a test here. Podman and buildah are not covered. Docs only. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1577b57d67 |
docs(accounts): approved channels are backfilled since #2133 (#2385)
`docs/user-guide/accounts.md` said that messages received before a channel proposal is approved stay encrypted. Since #2133 that is no longer the case: when the key set grows, the ingestor re-decodes the stored rows it could not decrypt (`cmd/ingestor/key_backfill.go`), and the server tails `tx_rewrite_feed`, so no restart is needed. Checked on an instance running #2133: approving four hashtag channels triggered one pass (`[key-backfill] channel: 4 of 410 key(s) not yet applied to history`, then `pass complete, 801 row(s) rewritten`, 13 s), and the 801 rows are exactly the stored messages of those four channels, the oldest from three months before the approval. Docs only, one sentence. Not done: I did not check the other docs pages for the same claim beyond a grep for "stay encrypted" and "received before" in `docs/` and `README.md`, which found only this one. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
199df07c21 |
docs(proto): correct route_type values to match the decoder (#2363) (#2367)
Fixes #2363. The published contract described `route_type` as `0=DIRECT, 1=FLOOD, 2=reserved, 3=TRANSPORT`. Both decoders (`cmd/ingestor/decoder.go`, `cmd/server/decoder.go` — `routeTypeNames`) and MeshCore's `docs/packet_format.md` use: ``` 0 = TRANSPORT_FLOOD, 1 = FLOOD, 2 = DIRECT, 3 = TRANSPORT_DIRECT ``` Updated: - `proto/decoded.proto` — `DecodedHeader.route_type` and the `route_type_name` value list - `proto/packet.proto` — `Transmission.route_type` (same stale comment) - `docs/api-spec.md` — Packet Object `route_type` line (same stale comment) Comment/doc-only; no generated code in-tree, no behaviour change. Co-Authored-By: Claude <noreply@anthropic.com> Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
b3a9f2a9e9 |
docs: make scheduled backups portable and bounded (#2353)
Fixes #2352. Follow-up to #2345.
The older deployment guide recommended copying a live telemetry
database. It now points to the consistent-snapshot procedure. The
scheduled-download example uses a trailing-X temporary filename, bounds
connection and total download time, and explains the weak-key denylist.
Validation: the parent reran the extracted command against eight
disposable HTTP cases: valid SQLite, unauthorized, server error, empty
content, HTML, corrupt SQLite, truncated transfer and timeout. Failed
downloads removed partial files and preserved existing completed
backups; the timeout exited 28. Shell syntax and diff checks passed. GNU
tools were exercised; BSD/macOS was not run natively.
TDD exemption: pure documentation, no application or repository test
files changed. No runtime/browser behaviour, dependencies or
configuration changed. Independent reviews are complete; the one
timeout-wording finding was addressed in
|
||
|
|
faf6f792ae |
docs(release): v3.16.0 release notes and changelog entry (#2355)
Release notes for v3.16.0 (`docs/release-notes/v3.16.0.md`) and the matching `CHANGELOG.md` entry. The tag pipeline uses this file as the GitHub release body. Range `v3.15.0..af2e96d9`, 47 commits: 23 fix, 9 feat, 6 docs, 2 test, 2 perf, 2 ci, 1 ui, 1 deps, 1 chore. "Read this before upgrading" covers three changes: - relay hops refresh Last Heard only with a unique prefix and only on flood packets (#2334, #2308, #2298, #2336); - `users.db` schema v6 to v8 with user accounts on (#2306, #2348); - Go 1.26 for source builds and Alpine 3.24 in the image (#2297). Not verified: the mosquitto, Caddy and supervisor versions are taken from #2297's local build, not from a pull of the published image. Performance numbers are copied from the PRs, not re-measured. Docs only, no code change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
af2e96d92a |
fix(reach): queue a cold request up to 3 s for a build slot before answering 429 (#2249)
Follow-up to #2127, from @dborup's review there. The limiter, most of its tests and the docs text are dborup's (dborup/CoreScope@195ec4c3), adapted to master; the commit carries `Co-authored-by: dborup`. ## What changes `GET /api/nodes/{pubkey}/reach` still runs at most 2 uncached reports at once. What changes is the next uncached request: - **Before:** 429 at once, `Retry-After: 5`. - **After:** it waits up to 3 s for a slot. It gets 429 with `Retry-After: 2` and a JSON body only if no slot frees in that time. Unchanged: cached answers never wait, and concurrent requests for the same node and `days` share one computation and one slot. One addition beyond dborup's version: the shared computation runs on the context of the request that started it. If that request disconnects, while queued or mid-scan, the other requests sharing it that are still connected now get 429 instead of 500. The 3 s wait made that case more likely. ## Why dborup's three points on #2127: 1. An instant 429 fails two tabs, or a quick switch between 7/14/30 days, when the database is not under real pressure. 2. The page cannot recover from it, because `api()` retries only 503. The panel retry is a separate PR so that this one stays server-only. 3. No test covered the promised property: a cached request still answers while every slot is held. Waiting parks a goroutine, not a database connection, so the pool is protected exactly as before. ## What I checked - **Tests.** `make test` (all 16 modules), `make vet`, `gofmt`, `sh test-all.sh` and eslint (0 errors) all pass. The reach tests also ran 10 times under `-race`. - **New tests.** 14 tests in `node_reach_cold_limit_test.go`: dborup's 10 plus 4 new. `TestNodeReachConcurrentBuildCap` now drives the new limiter through the production router. Point 3 is `TestNodeReach_WarmHitSkipsColdLimit`. The 4 new tests are: - a queued request is served when a slot frees; - a cache filled just before the flight skips the limiter; - coalesced waiters get 429 when the starter disconnects, queued or mid-scan (two tests). - **Mutations.** I wrote 21 from the code before running the tests, and the tests caught all 21: instant reject, cap +1, release that leaks, slot taken before the in-flight recheck, busy → 500, Retry-After header or value, status 503, dropped context mapping, and others. - **Real binary.** Master `2d09335c` and this branch, run one after the other on a copy of a 10.5M-observation, 11 GB database. Each burst fires its reach requests at once, while `/api/nodes?limit=50` and `/api/packets?limit=50` are polled every 200 ms. | Burst (distinct uncached keys) | master | this branch | |---|---|---| | 3 keys, `days=7`, warm page cache (2 runs) | 2 × 200, 1 × 429 after 0.01 s | 3 × 200; the third answered at 4.1–4.3 s | | 6 keys, three of them `days=14` (2 runs) | 2 × 200, 4 × 429 after ≤ 0.01 s | 2 × 200, 4 × 429 after 3.0 s | | 24 keys, cold page cache | 2 × 200, 22 × 429 | 2 × 200, 22 × 429 after 3.00 s | | `/api/nodes` during every burst | median 1 ms, max 53 ms | median 1 ms, max 66 ms | ## Not checked - **Slow scans.** On a database this size, uncached scans took from about 9 s to over 60 s, so the 3 s wait only helps when page cache is warm or the database is smaller. Otherwise the result is the same 429, 3 s later. A 30-day uncached scan hit the 60 s `WriteTimeout` on both builds; that is the scan #2214's opt-in hop index addresses. - **Pool contention under a live store.** On this copy the in-memory store loaded 0 packets, so `/api/nodes` latency does not show contention under a live load. - **Platforms.** One machine (linux/amd64). No browser check: the server change has no UI, and the panel retry comes with its own PR. **Base.** Rebased on master `7c00a429`, which includes #2214; every test and mutation above was re-run on the rebased commit. The real-binary runs used `2d09335c`, just before #2214, on a database without the hop index. That is the default after #2214 too (`hopIndexDays` unset means off). With the index on, scans are shorter, so more bursts fit in the 3 s wait; I did not measure that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Update after review (commits `c65f4d3b`, `1d2cd99d`, `04e49854`) efiten asked how many requests may wait at once. With only the 3 s timeout, the number of waiting distinct keys grew with the request rate. n30nex wrote a cap (nullrouten0/Baymc-CoreScope#1, tracked in #2264). Its two commits are now on this branch unchanged, as a fast-forward. - At most 8 distinct cold requests wait for a slot. Any further one gets 429 with `Retry-After: 2` at once, before a timer is set. - A waiter gives its place back on every exit: it gets a slot, disconnects or times out. - Cached answers and same-key sharing are unchanged. Requests for a key that is already queued or building wait on that one place and are not counted. `04e49854` corrects the code comment that said the ceiling was independent of request rate. **What I checked on `04e49854`:** - The two new tests fail on `28da016f` and pass now. - 12 mutations of the cap, all caught: cap 7 and cap 9; the place never returned, returned before waiting, returned only on promotion, leaked on cancel, leaked on timeout; a full queue that blocks, returns the wrong error or skips the counter; a timer armed before the check; a queue sized by the slot count. - `make test`, `make vet`, `make fmt-check`, `sh test-all.sh` and eslint 8.57.1 (0 errors) pass, and the reach tests pass 5 times under `-race`. On a local merge with current master `660cbbf5`, which includes #2277, the reach tests pass 3 times under `-race` and the `cmd/server` package passes. - Real binary: both heads, alternately, on a copy of a 2.5 GB soak database, with 24 distinct uncached `days=7` requests fired at once, two runs each: | | `28da016f` | `04e49854` | |---|---|---| | 200 | 2, 2 | 2, 3 | | 429 at once | 0, 0 | 14, 14 | | 429 after 3 s | 22, 22 | 8, 7 | **Not checked:** a sustained flood, a burst of same-key requests against the cap, `/api/nodes` latency during these bursts, and platforms other than linux/amd64. The real-binary table further up was measured before the cap. --------- Co-authored-by: nullrouten0 <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: dborup <kontakt@meshview.dk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: n30nex <blushellneon@outlook.com> Co-authored-by: efiten <erwin.fiten@gmail.com> |
||
|
|
1ed5e4ab3f |
fix(compare): exclude observer TX-only packets from overlap (#2346)
Red commit: `39295da8` ([intended CI assertion failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)). Green commit: `d1bda496`. Observer comparison counted an observer's TX-only packets as receptions, producing false overlap. It now requests `excludeTx=true`: the selected observer and non-TX direction must match the same observation in memory and SQLite v2/v3. Normal packet lists and stored TX records remain unchanged; missing directions retain legacy compatibility. Refs #667. This completes a comparison slice, not the issue's remaining observer counters or #671's fleet/SNR feature. Representative display fields and existing time-filter semantics are unchanged. API/OpenAPI docs describe the filter and multi-node bypass. Validation: - RED: Go tests compiled with 32 intended failures/16 control passes; unit and browser regressions reproduced false overlap. CI reached the new unit assertion. - GREEN: all 48 packet cases pass, plus grouped-cache isolation and OpenAPI tests. The added cache regression caught a malformed-radio cache-key collision before correction. - Full local server suite: 174s; ingestor: 185s; `npm run test:unit`: pass. Final focused packet/grouped/OpenAPI checks: pass. - E2E assertion added to `tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks pass, including correct 1/2 reception totals and 50% overlap. Performance: still two capped bulk requests, indexed candidate lookups retained, one direction check within the existing observation scan/SQL predicate, no new persistent structures. No configuration/customizer changes. Final-head CI passed; independent reviews found no must-fix items. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
8437d22356 |
docs: use consistent snapshots for backup and restore (#2345)
The deployment guide copied a running `meshcore.db`, which can omit committed WAL data or produce an inconsistent backup. It now uses the existing authenticated `/api/backup` snapshot, checks the download before publishing it, and explains how to restore with all database users stopped while preserving the old database and sidecars for rollback. Fixes #703, following the [maintainer's remaining documentation scope](https://github.com/Kpa-clawbot/CoreScope/issues/703#issuecomment-6163256620). - Protects API credentials in a header file and keeps incomplete downloads out of the completed backup set. - Links the separate account-backup guide; configuration and secrets remain separate backup responsibilities. - Replaces unsafe live-copy scheduling instructions with the same checked download procedure. Validation: - Exact shell example with real curl/SQLite: valid download, HTTP 401/500, empty, HTML, corrupt database, and truncated transfer. Failures preserved the previous backup and published no new `.db`. - Existing `TestBackup*` passed. Chromium downloaded an authenticated snapshot from a disposable instance; a row committed only in its nonempty WAL survived and `quick_check` returned `ok`. - Disposable stopped-file restore and rollback recovered the original WAL-only row and preserved the original files byte-for-byte. - `git diff --check` and privacy preflight passed. TDD exemption: pure docs; no application behavior changed and no test files modified. No new config/customizer settings. Live operator services were not used for validation. Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
b65412ab83 |
feat(observer-logins): turn on neighbour reports in the firmware block of the credentials mail (#2349)
The firmware block of the observer credentials mail now ends with `set
mqtt.neighbors on`, so firmware observers send the same `/neighbors`
report that the OpenHop block already turns on ("Publish Neighbours",
#2347). The ingestor reads that report for region scopes (#1865).
## Changes
- `cmd/server/observer_mail.go`: one extra line in the firmware block,
and one sentence in its note. The sentence says the command sends the
neighbour table and region scopes once a day, and that older firmware
does not know it and should skip it.
- `cmd/server/observer_mail_test.go`: the full firmware block assertion
includes the new line, and the slot test checks the line and the note.
- `docs/observer-mqtt-logins.md`: the mail section describes the new
line.
## Tests
- `go test ./...` in `cmd/server`: ok. Skipped
`TestSaveGeoFilterPreservesFileMode`, which needs Unix file modes; this
run was on Windows.
- `TestObserverCredentialsMail*` failed before the change, on the
missing line, and passes after it.
## Not checked
- The command name and the 24-hour default come from #1865. I did not
read the observer firmware source.
- The wording of the error older firmware gives for an unknown command
is not quoted in the mail. The note only says to skip the line.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
cadc666bd9 |
feat(observer-logins): OpenHop block in the credentials mail (#2347)
The observer credentials mail (#2306) had blocks for meshcoretomqtt, the Gessaman/Dutch-MeshCore firmware, Home Assistant and plain values. An operator running OpenHop (repeater software with its own MQTT uploader) could not tell which of these matched its broker form, and picked the format by trial. ## Changes - `cmd/server/observer_mail.go`: a fifth block, "4. OpenHop (repeater software)", before the plain values. It uses the field and option labels of the OpenHop broker form: - `Format: Letsmesh`, `Transport: TCP` or `Websockets` (from `observerLogins.transport`), host, port, username, password, `TLS: on/off`, `Use JWT Auth: off`, `Retain Status: on`, `Publish Neighbours: on`. - The note says Meshcoretomqtt works too, but the format labelled "openHop" (value `mqtt`) does not: it publishes to `meshcore/repeater/<name>`, which a personal login may not use. - Publish Neighbours is on because the ingestor reads the `neighbors` topic for region scopes (#1865). - `openapi.go` and `docs/observer-mqtt-logins.md`: "four formats" becomes five. ## Sources Read from OpenHop, not tested on a device by me: - format family and topics: `openhop_repeater/repeater/data_acquisition/mqtt_handler.py` (`MC2MQTT_FORMATS = ("meshcoretomqtt", "letsmesh", "waev")`, the `mqtt` format's `meshcore/repeater/<node_name>` base topic, transport values `tcp`/`websockets`); - labels: the built web UI (`Configuration-*.js`: "Add Broker", option labels `Letsmesh`, `Meshcoretomqtt`, `openHop`, `TCP`, `Websockets`, "Retain Status", "Publish Neighbours", "Use JWT Auth"). An operator on our instance set it up from a screenshot of that form with format Letsmesh, TCP, Retain Status and Publish Neighbours on; the observer publishes, and its `neighbors` report set `configured_scope` on its node. ## Tests - `TestObserverCredentialsMailOpenHopBlock`: the block text for TCP without TLS and for Websockets with TLS, the "openHop" warning, HTML escaping, and the block order. - `TestObserverCredentialsMailBlockTitles`: five titles in order. - `cmd/server` suite green locally (the two Windows-only failures skipped, as on master). ## Not in this PR - The websocket path: OpenHop uses paho's default, the doc already tells operators to match the listener path. - Anything with the other fields of the `neighbors` report (SNR, heard_secs_ago, default_scope): we are measuring how many observers send it first. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d29957fbdc |
feat(observer-logins): lock the shared login out of migrated observers (#2348)
Follow-up to #2306 / #2319. With personal observer logins, a migrated observer still is not safe from the shared login: the hook answers `ignore` for the shared login, so EMQX's built-in database lets it publish under any observer's topic, including one whose owner has moved to a personal login. This PR lets an operator close that per migrated observer, before the shared login is removed altogether. ## Changes **users.db v8.** `observer_bindings.confirmed_at`: the moment the owner's personal login first published under that pubkey. - An auto-bind (the owner's first publish) is confirmed at once. - An admin assignment stays unconfirmed until the owner's login publishes under that pubkey; that publish sets it. - Reassigning to another user or unbinding clears it; reassigning to the same owner keeps it. - Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin` rows stay NULL. **Config.** `userManagement.observerLogins.sharedUsernames` (default empty): the broker's shared login names. A name starting with `obs_` is refused at startup. **Authz hook.** For a login in `sharedUsernames`, a publish under `meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is confirmed, whatever the number of topic segments (the ingestor attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey). Logged as `publish denied: user=<name> observer=<8 hex>: confirmed owner`. Every other request from a shared login keeps `ignore`, so the built-in database decides as before. Personal logins are unchanged. An orphaned binding (owner account deleted) does not count as confirmed. **Admin.** The observers table of the Observer logins tab gets a Confirmed column; the bindings JSON gets `confirmedAt`. **Docs.** `docs/observer-mqtt-logins.md`: "Locking the shared login out of migrated observers" (what it denies, how to see `confirmedAt`, the risk below). `config.example.json` has the key. ## Risk for operators An observer that is confirmed but still runs a second uploader on the shared login loses that second uploader once its name is in `sharedUsernames`. The deny log line names the observer. ## Tests - `internal/users`: v7 to v8 migration on a real v7 database, auto confirm, admin assignment then owner confirm, non-owner and repeated confirm, reassign clears, orphan reassign. - `cmd/server`: the decision table (confirmed, unconfirmed, unbound, non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty and `client` iata, lowercase key; owner publish confirms, confirm failure still allows and retries; reassign, unbind and user removal clear the flag; the index load against a real store; config validation; the hook and admin endpoints end to end. - `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an auto-bound observer. - Local: `internal/users`, `cmd/server` (two Windows-only tests skipped, as on master) and `cmd/ingestor` green. The E2E assertion was not run locally; CI runs it. ## Not in this PR - Using `confirmed_at` for verified `/neighbors` reports: a separate PR, built on #2300. - Removing the shared login: still an operator step after migration. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
410db29fe6 |
chore: remove unused legacy Dockerfiles (#2314)
Fixes #2310. Partial follow-up to #1385 (PR-C). Remove unused `Dockerfile.go` and `Dockerfile.node`, which describe retired alternate builds. Current workflows, Makefile, Compose variants and operator scripts already select the canonical `Dockerfile`. Remove the obsolete formatter exclusions and label the old Node-to-Go migration guide historical, linking current deployment instructions. Historical release prose remains untouched. The canonical image, runtime files, dependencies and Compose behavior are unchanged. Validation: audited all 1,501 tracked files; old and new formatter inputs are exactly the same 633 real Go files. Both formatter commands, YAML/shell/workflow-reference checks, existing workflow/routing tests, COPY gate and test inventory pass. All 218 standalone frontend suites pass. The unchanged backend tree passed full server/ingestor suites, and the unchanged app passed 12 browser smoke cases. TDD exemption: behavior-preserving cleanup/refactor plus documentation; no test files modified. `make` and `actionlint` were unavailable locally; equivalent formatter and focused workflow checks ran directly. [CI passed on this head](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40230911566), including canonical container builds and all browser shards. Three independent reviews passed. The published historical guide and its current-deployment links render correctly in Chromium. No settings/customizer changes. |
||
|
|
23f938d1d4 |
docs: consolidate historical release notes (#2313)
Fixes #2311. Partial follow-up to #1385 (PR-B). Five historical release notes lived outside `docs/release-notes/`. Move v2.6.0, v3.0.0, v3.1.0, v3.4.0 and v3.6.0 into that directory using its existing version naming convention. Release content is preserved except for concise historical notices and four relative-link repairs. The v3.0.0 notice identifies its obsolete Node rollback instructions; current setup links point to README and DEPLOY. No new documentation hierarchy or index is introduced. Validation: all five moves are one-to-one; every document matches its original bytes after only the listed edits; all 14 local Markdown links resolve; no old-path references remain. Existing external links are unchanged. All 218 standalone frontend suites pass. The unchanged backend tree passed full server/ingestor suites, and the unchanged app passed 12 browser smoke cases. All five published documents were rendered in Chromium; historical notices and repaired link targets were verified. Three independent reviews passed. TDD exemption: pure documentation; no application, test, CI, configuration or deployment behavior changed. No settings/customizer implications. [CI passed](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40230718673); its docs-only classification correctly skipped application jobs. The full-suite results above are local validation. |
||
|
|
60615784d9 |
docs: archive historical planning documents (#2321)
Fixes #2318. Partial follow-up to #1385 (PR-A). Move eight historical planning documents from the root into `docs/archive/`, keeping their original filenames and headings. Each gets a brief archived-plan notice linking current README and deployment guidance. Historical proposals, implementation claims and code examples remain intact. Validation: eight one-to-one renames, byte-preserved original content, 16 valid added links, and no inbound or executable references requiring updates. Seven already-absent historical source paths remain plain text under the archive notice; no replacement links were invented. All other tracked files are unchanged. All 218 standalone suites passed locally. The unchanged backend tree passed full server/ingestor tests, and the unchanged app passed 12 browser smoke cases. TDD exemption: pure documentation; no test, application, configuration or deployment change. No customizer implications. All eight published documents and their guidance links were verified in Chromium. Three independent reviews passed. [CI passed](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40238488019); its docs-only classification correctly skipped application jobs. The full-suite results above are local validation. This only addresses PR-A; other cleanup remains tracked separately. |
||
|
|
e149568c6c |
feat(observer-logins): personal MQTT logins for observers (#2306)
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.
Closes #2319.
The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.
## Changes in this PR
**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.
**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.
**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.
Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.
**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.
**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.
**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.
**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.
## Enabling it on an instance
Full steps in `docs/observer-mqtt-logins.md`. In short:
1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.
## Who can claim an observer
Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.
What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.
## Performance
`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.
## Verification
- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.
- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.
## Known behaviour
EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.
## Not in this PR
- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.
Note on
|
||
|
|
7fa86fe1c1 |
docs(release-notes): v3.15.0 (#2263)
Release notes for v3.15.0: `docs/release-notes/v3.15.0.md` and the `CHANGELOG.md` entry. The release workflow uses the notes file as the GitHub release body. ## What is in it - 77 commits since v3.14.0 (76 merged pull requests plus this one): 27 fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs, 1 chore. Nineteen feats, so minor. - "Read this before upgrading" first. It covers: - the liveness change (#2261, #2262); - the `users.db` v6 migration and what it means for a rollback (#2174); - the backfill at every ingestor start (#2133) and the 24 h planner-statistics refresh (#2158); - the caches that trade freshness for latency (#2165, #2222, #2147, #2156); - changes in hop names and region placement (#2188, #2190, #2234, #2245, #2220); - the role thresholds on favorites (#2215); - the geo-filter save on a single-file mount (#2137); - the `/api/stats` polling in hidden tabs (#2162). - Features, fixes, performance, then tests/CI/docs. - 29 issues closed, taken from each pull request's `closingIssuesReferences`. - Contributors: pull request authors and issue reporters with counts. The commit co-author trailers only name the pull request's own author or tooling. ## How it was checked - I read every pull request body. Every number in the notes comes from a pull request body. - I checked the config keys and defaults against `config.example.json` on `2bb000f2`. - I checked the UNASKED and opt-in claims for #2125, #2133, #2235 and #2239 against the source. - For the other pull requests I relied on their bodies and did not open the diffs. ## Not covered - The uncertain count is a single count on one staging instance on the day of the change (1,682 repeaters and rooms). It is not a before/after comparison. - #2133 gives no measured cost for its backfill on a large database, and the notes say so. ## After merge Tag `v3.15.0` on the merge commit once its `:edge` image exists. Then check in the registry that `v3.15.0`, `v3.15`, `v3` and `latest` share one digest, and that the release has the notes as its body and both `corescope-decrypt` binaries. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2bb000f2aa |
feat(notify): adaptive offline threshold and certain node analytics (#2260) (#2262)
Part 2 of #2260. **Builds on #2261** (certain liveness): review/merge that first; until then this diff also shows its commit. ## Adaptive offline threshold `userManagement.notifications.adaptiveOffline` — **off by default**: ```json "notifications": { "enabled": true, "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 } } ``` The fixed `infraSilentHours` (72 h) treats every repeater the same. With this on, a watched node's `node.offline` window is learned from its own **certain** traffic (the #2261 rule): 1. Certain packets per hour over the week **before its last certain packet**. Ending there, not at now, keeps an outage from dragging the rate down until the fixed window returns and the node reads "back online". 2. Empty runs are classified: ≤ 6 h and recurring on another day at the same hours = normal silence (kept; the window is never shorter than the longest + 1 h); any other run = outage, left out of the learning so it neither stretches the window nor blocks learning. 3. λ = 10th percentile of the remaining hourly counts, so quiet nights don't alert. 4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson, ln 10⁶ ≈ 13.8). 5. Clamped to [`floorMinutes`, the role's silent window]. Kept on the fixed window: λ < 1/hour, under 24 h of history in memory, or a 10th-percentile hour heard by fewer than `minObservers` observers (one observer dropping out must not read as the node going offline). Silence is measured up to the newest packet in the store, so a stalling feed stops every node's clock. A node stored offline comes back only on evidence from the moment it went offline on. The notifier caches each learned window for 10 minutes and drops unwatched nodes. For the #2260 repeater (~840 packets/day) the mail would have gone out about 30 minutes into the outage, instead of never (or 72 h with #2261 alone). ## Certain node analytics - Each `activityTimeline` bucket carries `certain`: the part that can only be this node (`certainFor`, the packet-level form of the `certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other endpoints are unchanged. - Availability and longest silence use certain buckets only; the longest silence now includes one **still running** (an outage in progress is the silence worth seeing). - The chart stacks certain (`--accent`) and uncertain (`--text-muted`) — existing theme variables, so no new customizer entry. Descriptions updated. ## Performance `CertainActivity` walks a week of packets per watched node, at most once per node per 10 minutes, with the read lock taken **per node**: ``` BenchmarkCertainActivity-16 12.1 ms/op (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold) BenchmarkNoteCertainRelays-16 121 ns/op 0 allocs ``` Node analytics adds one `certainFor` per packet of the requested node (a prefix-map lookup, or the cached decoded JSON for its own packets). ## Tests - `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet nights set the rate; > 10 % empty hours, short history, too few observers → fixed. - `CertainActivity` ends at the last certain packet and counts only certain packets — verified red by mutating the window to end at now. - Notifier end to end: learned window passed → bad; inside → good; too few observers / setting off → fixed 72 h; cache refresh and prune. - `certainFor`, `longestSilence` (incl. running silence), config defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket. - Frontend: `NodeAnalyticsTimeline.series` (new `tests/unit/test-node-analytics-certain-timeline.js`). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork. ## Validation Server on the freshened e2e fixture: analytics buckets carry `certain` (e.g. 89 of 90 packets certain for one repeater). **No browser screenshot**: Playwright isn't installed on this machine. ## Later (rule 8) `floorMinutes` / `minObservers` are server config, not theme settings; an admin UI for notification settings would be the place, not the customizer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
a08f2f4e7c |
fix(liveness): count only evidence that can be the node alone (#2260) (#2261)
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold follows in a separate PR). ## Problem A repeater silent since 2026-10-18 showed **Active — Last relayed 4m ago** and **Last Heard 36m**, and its offline mail never fired. Every hit after that date was the 1-byte hop `67`, shared by nine nodes: - the live resolver's affinity tiebreak (`resolvePathForObs`) put those transmissions in `byNode`, which fed `stats.lastHeard` and `LastHeardMap`; - `last_relayed` folds in the 1-byte prefix bucket on purpose (#662), and drove the status in `roles.js` and the notify offline check. ## Fix The store keeps `certainHeard` per node: the newest transmission that can only be that node. | Evidence | Counts? | |---|---| | Own packet (`pubKey`, `srcPubKey`) | ✅ | | Flood relay hop whose prefix, at the packet's path hash size, matches exactly one path-capable node | ✅ | | 1-byte hop shared with other repeaters | ❌ | | Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌ | | TRACE (path bytes are SNR) | ❌ | | Destination (`destPubKey`) | ❌ | A multibyte repeater legitimately appears as a 1-byte hop (the originator picks the hash size, `Mesh::routeRecvPacket` appends at that size), so only uniqueness decides. The rule is applied in every load path through `indexResolvedPathHops` and the new `indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live ingest, late observations. Liveness reads only that: - `/api/nodes` and the node detail: `last_heard` is the newer of `last_seen` and the certain last heard. Before, it was a plain copy of `last_seen`. - health / bulk-health `stats.lastHeard`. - `LastHeardMap` → notify offline check. The separate relay input (`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed. - `roles.js getEffectiveHeardMs`: newest known timestamp, without `last_relayed`. The side pane no longer says "Last relayed"; the Last Relayed row stays (analytics) with an updated tooltip. - New status **`uncertain`** (repeaters and rooms): nothing certain within the silent window, but the shared prefix (`last_relayed`) still in paths. Grey badge with explanation, an *Uncertain* filter on the nodes list and the map, dimmed like stale. The offline mail for such a node says only its shared prefix was still seen. - A prefix also held by a path-capable node in `inactive_nodes` is not unique; adverts with `signatureValid: false` are no evidence; `/api/nodes/search` reports the same `last_heard`. Attribution for analytics (`byNode`, packet counts, `last_relayed`, relay counters, usefulness) is unchanged. ## Performance Hot path: `noteCertainRelays` runs per observation on ingest under the write lock: two hex bytes of header parsing plus one prefix-map lookup per resolved hop. ``` BenchmarkNoteCertainRelays-16 10307640 116.2 ns/op 0 B/op 0 allocs/op (8-hop 2-byte path, 2,000 repeaters) BenchmarkLastHeardMap-16 1902 0.77 ms/op (5,000 nodes; was 3.7 ms scanning byNode) ``` Memory: one string per node ever certainly heard, dropped together with the node's `byNode` list on eviction. ## Tests - `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk, LoadChunked — each with and without resolved_path — live ingest, late observations) feeds `certainHeard`; each of the 8 call sites verified red with a nil prefix map. - `certain_heard_test.go`: hop width per route/payload type; retired namesake prefix; failed signature; uncertain offline mail; shared vs unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`, detail and health agree. - Regression for #2260: last certain packet on day X, 20 shared-prefix hops after it → `LastHeardMap` stays at X and `node.offline` goes bad after 72 h. Verified red with the uniqueness check mutated out. - Notify: `TestEvaluateCertainHeard` replaces the relay-input test. - Frontend: `last_relayed` alone no longer keeps a repeater or room active; newest timestamp wins (an old `_liveSeen` can't hide a fresh certain relay). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork with 457 extra commits. ## Validation Ran the server on the freshened e2e fixture: list, detail and health report the same last heard; e.g. a repeater whose `last_relayed` is 3 min newer than anything certain keeps its certain time. **No browser screenshot**: Playwright isn't installed on this machine; CI's E2E covers the pages. ## Not changed - The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a shared hop only when exactly one candidate is adjacent in the neighbor graph; it did not fire in this case. Left as is. - The `lastHeard=` list filter still filters `last_seen` in SQL. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
dc98886d92 |
feat(nodes): name and show repeaters known only from the name they reported (#2238) (#2239)
Part of #2238. Path hops through such a repeater (`/api/nodes`, `/api/resolve-hops`, the frontend hop resolver) still show a bare prefix; that part of the issue stays open. ## Summary A repeater that never sends an advert can still tell a nearby companion its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`. This PR stores those answers, resolves names with them, and shows these repeaters on the map, in the nodes list and in search, marked as self-reported. ### Ingestor: store the names - `367ecde1` adds a new `names` subtopic. - It is gated on `clientNodeNames.enabled`, default off. - Answers go into a new table, `client_node_names`: one row per observation, `UNIQUE(target, rx_pubkey, observed_at)`. - **Validation.** - `target` must be 64 hex. - The name is trimmed and rejected when it is empty, over 31 runes (firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8, or contains a control character. - **Retention.** `retention.clientNodeNamesDays` (0 = keep). `config.example.json` sets 30, like the other client tables: a companion only asks a repeater for its name when CoreScope has none (`nameMissing` in CoreDrive's `src/names.js`), so pruning is also how a renamed repeater gets its new name. - **Compatibility.** An ingestor without this change drops `/names` silently, so CoreDrive v1.20.0 works against either version. ### Server: resolve by the reported name - `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and `client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys` (the coverage tooltips) both use it. - **Which name wins.** The advert name, otherwise the newest reported name by `observed_at`. A prefix that matches different pubkeys across the two tables stays ambiguous. - **Only repeaters heard directly.** A reported name counts only once its target has at least one `discover` or `advert` reception in `client_receptions`, keyed by the full pubkey, or by its 8-byte prefix when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names` starts with that prefix (`clientNameHeard`). A name for a pubkey nobody heard adds no candidate, so it cannot make a real node's prefix ambiguous. Without the uniqueness check, a made-up pubkey that shares a heard repeater's first 8 bytes borrowed that repeater's receptions and passed (fixed in `4c569f91`). - **Filters.** The blacklist and the hidden name prefixes apply to both sources. - **Missing table.** A database without the table resolves from `nodes` alone, and the server checks for the table again until the ingestor has created it. - **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?, name?, ambiguous}`. ### Server: estimated position - `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`. It lists every repeater in `client_node_names` that never sent an advert (no row in `nodes` or `inactive_nodes`) and was heard directly at least once (see above). A node with an advert but no position stays out: it is already in `/api/nodes` under its advert name. - **Gated** like the other endpoints on `client_receptions`: 404 when `clientRxCoverage.enabled` is off, because the counts and the estimate come from companion GPS fixes. - **Which receptions count.** Only direct ones: - `discover` and `advert` rows, keyed by the full pubkey, or by its 8-byte prefix when no other known pubkey shares it; - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone. `geo` rows are left out. - **How the estimate is made.** - Receptions within 100 m form one spot, with the best SNR of that spot. - An estimate needs at least 3 spots, at least 200 m apart. - The position is the SNR-weighted centroid (`10^(snr/20)`). The radius is the distance to the farthest spot, at least 500 m. - The response is cached for 5 minutes, and the compute time is logged. ### Map - `5fcde554` adds `public/map-estimated.js`, which draws each estimate on its own layer: - a hollow, dashed "≈" marker and a dashed uncertainty circle; - a popup with an "Estimated position" badge, the method, the caveat that one-sided driving skews the estimate, and a link to the coverage map; - a Display toggle, "Estimated positions": on by default, kept in localStorage, and also hidden by the repeater role filter and in route view. ### Nodes page and search - `12e6b345` adds a "Self-reported repeaters" section under the nodes table, on the All and Repeaters tabs, filtered by the search box. - **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a self-reported detail view replaces "Node not found": - who reported the name; - the receptions; - the estimate, or why there is none; - links to the map and the coverage map. An unknown key still shows "Node not found". - **Header search** adds matching self-reported repeaters, tagged "self-reported". - **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as they are. These repeaters have no packets, so the packet filters, the live view and the channel sender lookup must not match them. ### Docs - `03580437` corrects the `/regions` section of `docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the switch, but the code uses `clientRegions.enabled`. - The new subtopic, endpoint and config keys are documented in the same file. ## Trust Any companion the broker lets publish can report a name for any repeater, and the newest report wins. This is the same model as `/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to linked companions. A report for a repeater nobody heard directly is stored but not used: it is not listed and adds no prefix candidate. A companion can still report a wrong name for a repeater that was heard. The same companion can also publish a fake `discover` reception for a made-up pubkey, so the "heard directly" check filters junk, not a malicious companion; `requireLinkedCompanion` is the control for that (its gate runs before the `/names` handler too). ## Performance - **`lookupPrefixes`** adds one term per key on `client_node_names`. - It matches the prefix as a range, `target >= ? AND target < ?` (upper bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX idx_cnn_target (target>? AND target<?)`, and the reception check is `SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`. - It is still one query per chunk of 200 keys: 400 compound terms, under SQLite's default limit of 500. - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks. - **`/api/nodes/reported`** makes three kinds of query: - one grouped query on `client_node_names`; - one `lookupPrefixes` call for the short hashes; - one `IN` query per 400 keys on `client_receptions`, served by `idx_client_recept_heard_geo`. The spread check is O(spots²) per repeater, with at most one spot per 100 m visited. The result is cached for 5 minutes; concurrent misses wait for one computation, and the response is written after the lock is released. - **The map does not wait for it.** It draws its markers first and adds the estimates when the list arrives. The map and the nodes page share one cached fetch (`ReportedNodes.load()`). - **Not benchmarked.** On the instance where this runs, `client_node_names` holds one row today. I expect at most tens of such repeaters per instance, but have not measured that. ## Notes - **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2 occurrences: the handler's `msg` parameter and its `gps` field. That is the decoded MQTT message, passed in the same shape as `handleClientPacket` and `handleClientRegions` already take it. No API response or internal structure uses it. - **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min TTL. They should move to the customizer later (rule 8). - **Map filters.** The estimates follow the "Estimated positions" toggle and the repeater role filter, not the area, region or last-heard filters. The toggle's tooltip says so. - **Name characters.** Names are rejected for control characters but not for Unicode format characters (bidi overrides, zero-width joiners), because emoji sequences in node names use the zero-width joiner. All output goes through `escapeHtml`. - **Not done:** the home-page search box (`home.js`, `/api/nodes/search` with a Claim button) does not list these repeaters. Claiming a node without adverts would have nothing to show. ## Tests On this branch (based on `550829af`): - `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and `cmd/ingestor`. - **New Go tests:** - handler and validation cases; - idempotent redelivery; - newest `observed_at` wins; - prune; - the gate on and off, without falling through to the observer path; - client-only node, nameless `nodes` row, advert name wins, ambiguity across the two sources; - hidden and blacklisted nodes; - absent-then-created table; - spot merging, threshold, weighted centroid and radius floor; - `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an advert position and a missing table; - an advert without a position, a never-heard name, a name heard only in `rxlog` paths, and the `clientRxCoverage` gate (404). These fail on `12e6b345` and pass at head. - a made-up pubkey sharing a heard repeater's 8-byte prefix, and two reported targets sharing one (`4c569f91`). Both fail without that commit. - **New frontend unit suites:** `tests/unit/test-map-estimated.js` and `tests/unit/test-reported-nodes.js` pass. Both load the real `escapeHtml` from `app.js`. - **`test-all.sh`** passes, all suites (with Python available for the XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the four changed frontend files. - **E2E**, against a static server with a mocked API: - `test-map-estimated-e2e.js`: 6/6, including the repeater role filter and route view hiding the estimates. Mutating either hide makes its step fail. - `test-reported-nodes-e2e.js`: 6/6. - `test-issue-1150-404-state-e2e.js`: 5/5. - **Real data, API only.** On the fork's staging, `/api/nodes/reported` lists one repeater: 5 spots, 17 receptions, an estimate with a radius of 2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays listed under the stricter rules. Not re-checked after `4c569f91`. - **Not done:** a check of that repeater in a browser. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
93f9fc37e0 |
perf(ingestor): yield between committed retention batches (#2153)
Fixes #2151. Related to #2146. Yield to runnable goroutines after each committed full retention batch, once the writer lock is released. This gives the serial ingest consumer another opportunity to submit its next write. Batch size, cutoff and deletion semantics stay unchanged; dense fanout within one transaction remains unbounded. The branch is rebased onto upstream `b9593f71745b856651cfd6d6c80bb1f3f091e13e`. The duplicate test commit `99c2722d` is dropped because #2160 is already present upstream as `a5b60ca5`. The runtime and regression-test patch is unchanged from the previously reviewed `b70a9f6b`. The raw JSON profile is omitted from this branch. The [Markdown report](https://github.com/n30nex/CoreScope/blob/307432370f8a7d9d623223beac1ba15fdd2d6546/docs/performance/sqlite-retention-b5-2026-10-09.md) links to the [immutable original profile](https://github.com/n30nex/CoreScope/blob/323b18d4221e83c9827b4722145e2a9d5b9d9599/docs/performance/sqlite-retention-b5-2026-10-09.json); the linked profile was fetched and verified against its original Git blob. Performance scope: - The historical five-pair Linux comparison remains pinned to `9dbc287` versus `b70a9f6`, with 2,048,000 observations, 50 ingest events/s and 20 HTTP requests/s. - Additional-observation scheduled-to-complete p95/p99 improved by median paired ratios of 0.9503/0.9191, while service-time p99 rose 29.1% and cleanup took 0.5% longer. HTTP results were mixed, and multi-second ingestion queues remained. - Those figures do not measure this rebased candidate against current master. The current benchmark run is an execution smoke check, not a new performance comparison; the independent reviewer's results are not pooled into the historical study. - Added runtime work is one scheduler yield per full 250-transmission batch, with constant extra state and no additional SQL scans or per-row work. No config or customizer values change. Local validation on Windows with Go 1.27.2, CGO enabled, and package parallelism limited to one: - Focused retention regressions passed. - Full ingestor suite passed. - Focused retention regressions under the race detector passed. - Both retained benchmarks passed one iteration of every fixture (`typical16`, `dense256`, `singleHot`). - The full ingestor race suite passed in 264.31 seconds with `GOMAXPROCS=4`. An initial run during competing test jobs reached its 600-second limit; after those jobs finished, the retry passed with the same 600-second limit and all assertions intact. Post-push Chromium validation passed 148 checks with three explicit fixture-dependent skips. The server and frontend source are identical to upstream `b9593f71`; this browser pass checks the existing application journeys and does not measure retention performance. [Fresh CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/39433150610) passed on `30743237`, including the full Go job, ingestion race detector, all three browser shards and the two-architecture build. |
||
|
|
a164a44e7f |
fix(docker): run the Go services as an unprivileged user (#2125)
Runs `corescope-server` and `corescope-ingestor` unprivileged, opt-in with `RUN_AS_UID`. A code-execution bug in either one then no longer lands as root next to the mounted data and `config.json`. `supervisord` stays root, for mosquitto and caddy. | `RUN_AS_UID` | Go services run as | |---|---| | unset (default) | root, as before. Upgrades change nothing | | set | `RUN_AS_UID` / `RUN_AS_GID` | With `RUN_AS_UID` set: - Each start hands the data directory to that uid (`chown -h`), except `.env` and anything mounted into it on its own. Failures are logged. - `.env` is read only if it is root-owned and only root can write it. `RUN_AS_UID` and `RUN_AS_GID` are never taken from it. - `/app` stays root's. **Depends on #2137.** - A uid with no account in the image gets one at start; `caddy` and `mosquitto` are reused. A uid that can't be provided fails startup instead of falling back to root. The logic is in `docker/select-service-user.sh`, tested by `docker/test-select-service-user.sh`, which needs no Docker or root and runs in CI. See also "Running the Go services unprivileged" in `docs/deployment.md`. Verified in the built image (master + this + #2137) across 14 cases. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Mythos 5.1 <noreply@anthropic.com> Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> |
||
|
|
d9342bbebd |
(feat): UserManagement, Add Support for Postal Email Delivery Provider (#2154)
This PR extends email support to add another email delivery provider called Postal for user management. It is a, open source, self hosted email delivery and reputation management service. It adds 2 new additional configuration settings to the userManagement.mail object which are "postalBaseUrl" and "postalApiKey". It also adds "postal" as a supported value for the "provider" setting. All existing functionality including email delivery notification is supported by Postal. Included test cases to support the new functionality and all pipelines pass. --------- Co-authored-by: CoderNemesis <rfedor@comchan.net> |
||
|
|
a4ee3a7328 |
feat(scope-audit): observer region filter for both views (#2171)
## Summary Builds on the transport view from #2167 and #2169 (both merged); rebased onto `master` (`352a0ccd`), so the diff is this PR's six commits only. The Scope Audit was the one analysis page without the region selector. This adds the shared region filter to it, for both the declared and the transport view. `?region=SFO,SJC` means what it means on every other endpoint: count only forwarding **heard by observers in those regions**, not repeaters located there. - **API** (`GET /api/scope-audit`, both modes): - The region codes are resolved to their observers first (a handful of rows), then the audit's hop scan gets `AND o.observer_idx IN (…)` on v3, or `o.observer_id IN (…)` on v2. The 3.5M-row hop scan never joins `observers`. - `All` means no filter (shared `normalizeRegionCodes`). Codes with no observer match nothing, instead of silently widening to every observer. - Responses echo the normalised codes as `region`. The declared and transport caches and their singleflight keys now include the region, so each region set is cached separately with the existing TTLs. - Declared-region verification (`regionEvidence`) is deliberately left unfiltered: it corroborates what a repeater declares from its own traffic, which doesn't depend on where a hop was heard. - **Page:** the shared `RegionFilter` sits with the view and window controls; a change reloads the view, and the selection persists like every other page. The transport view's text box is relabelled **Scope** (it already sends `?scope=` since the rename in #2167), so the page no longer has two controls called Region. - **Docs:** `docs/api-spec.md` (parameter, both response shapes) and the OpenAPI parameters for the route (`mode`, `scope` and `region` were all missing). Partial fix for #2142 (its region filter for rollout tracking), together with #2167 and #2169. ## Performance The unfiltered path must not get slower: it's the same query with an empty clause. `BenchmarkComputeScopeTransport` (from #2167, 1,200 repeaters), the PR it stacks on vs this branch, `-count 8`: | | before | after | | |---|---|---|---| | sec/op | 220.3 ms | 218.0 ms | ~ (p=0.065) | | B/op | 43.77 MiB | 43.70 MiB | ~ | | allocs/op | 833.5k | 833.5k | ~ | **Filtered requests, measured on staging** (a copy of a production database): the first version drove `o.observer_idx IN (…)` from `idx_observations_observer_idx`, reading every observation the region's observers ever made. Fixed in `80414242` with a unary `+` (the same treatment as #2166), so the scan starts from the `first_seen` window and filters per row. Red test `67e063b1` asserts the query plan against a fixture with the real indexes. | scan, one region over 24h (13 observers) | plan | time | hop rows | |---|---|---|---| | `o.observer_idx IN (…)` | observer index | 13.44 s | 102,114 | | `+o.observer_idx IN (…)` | `first_seen` window | **0.25 s** | 102,114 | | unfiltered | `first_seen` window | 0.35 s | 225,983 | Cold end-to-end requests on staging after the fix: 24h declared, unfiltered 0.84 s · one region 0.48 s (was 16.7 s) · two regions 0.22 s · transport, one region 0.42 s · transport, six regions 0.75 s · 7d, one region 3.46 s. 24h transport, live data: 383 repeaters unfiltered, 204 heard by one country's observers. ## Tests - Red `629e4f36` (Go): extends the audit fixture with an `observers` table and `observations.observer_idx`. Covers the declared view ignoring a scope heard only by another region's observer, the transport view listing only repeaters heard in the region, `region=All`, per-region caching, a region with no observers, and the echoed codes. Five fail on assertions; `region=All` passes because the stub ignores the filter, which is the behaviour it pins. - Green `6f1b1e1c`: the implementation, plus a v2-schema case (`observer_id`), so both branches of the clause are covered. - Red `b4933d68` / green `08ecc172` (frontend): `apiPath` carries the selection in both views and adds nothing when none is selected. The region control on the page is a net-new UI surface. - `go test` (cmd/server) passes. Frontend suites: 193 of 194. The one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. ## Screenshot The same transport view with the shared region filter set to **SJC**: only forwarding heard by San Jose observers counts, so 7 repeaters are listed instead of 44, with their carried scopes counted from those observers alone. Taken against `test-fixtures/e2e-fixture.db` with fictional scopes.  ## Browser validation The branch's server was run locally with 14 configured regions (dropdown layout), checked in headless Chromium at 1280, 900, 700 and 390 px (touch): | check | 1280 | 900 | 700 | 390 touch | |---|---|---|---|---| | ticking SJC re-requests with `®ion=SJC` | yes | yes | yes | yes | | menu fully on screen | yes | yes | yes | yes | | page wider than viewport | no | no | no | no | | page errors | none | none | none | none | Found and fixed this way: the filter is the right-most control on desktop, and the menu (which opens rightwards) ran 66 px off the page. It now opens leftwards above 640 px. On a phone the controls wrap, the filter starts a line, and the default applies. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **S3776 on `ScopeAuditForwardingHeardIn`:** complexity 55. The function it generalises, `ScopeAuditForwarding`, is 54 on `master`; the region clause adds one point. - **S3776 on `computeScopeAudit`:** complexity 65, unchanged from `master`. - Both pre-date this PR. Splitting them is better as a refactor PR of its own than mixed into a feature. - The page code's findings were fixed in the transport-view PR it stacks on. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
352a0ccd19 |
feat(region-filter): configurable region quick picks (#2170)
## Summary **Independent of every other open PR.** Choosing a pick saves the selection, redraws the control and notifies the page's listeners itself, exactly as a click in the control does, so it re-queries straight away whether or not #2168 (`setSelected` notifies) is merged; with both in, nothing fires twice. Region quick picks are named groups of region (IATA) codes, offered as one-tap choices inside the region filter, e.g. "Northern California" for a dozen airport codes instead of ticking each box. One deployment has been injecting these into the page through an nginx `sub_filter` and a script that wraps `window.RegionFilter`; this makes them a configured feature. ```jsonc "regionQuickPicks": [ { "name": "Northern California", "description": "Bay Area and Sacramento observers.", "regions": ["SJC", "SFO", "OAK", "SMF"] } ] ``` - **Server:** `GET /api/config/region-quick-picks` returns the groups cleaned up: names and codes trimmed, codes upper-cased and de-duplicated in their configured order, groups without a name or codes dropped, at most 20 groups of 200 codes, always a list. `/api/config/regions` is unchanged. - **Region filter:** loads the groups with the regions and offers every group with at least one observer today, inside the region control: - dropdown layout: rows of the same kind as **All** (checkbox and bold name), directly under All with a divider before the single regions; the row is ticked while that pick is the selection, ticking it selects the pick and closes the menu, unticking goes back to all; - pill layout: the picks lead the same bar, set off from the single regions by a thin divider. - A tap selects the group's codes that have an observer and re-queries straight away. - The selector then reads the group's name ("South Bay ▾") instead of "2 Regions", and the button shows as pressed. Selecting the same codes by hand does the same. - Tapping the active group again goes back to all regions. - A failed fetch, or no configured groups, leaves the filter exactly as before. - **Mobile:** the rows take the menu's 44 px row height under `(pointer: coarse)`; the pill-bar picks reuse `.region-pill` and its touch target. Inside `.filter-bar` (Packets) the existing touch rule for inputs made every region checkbox a 44 px square; the checkbox is now 20 px and the 44 px row is the tap target. - **Semantics:** like every region filter, a group means "heard by observers in these regions", not "located there"; `config.example.json` says so. No existing issue covers this. ## Screenshot Nodes page with two picks configured (example config below), after tapping **South Bay**: the pick and its codes (SJC, MRY) are selected, and the list re-queried for them. Taken against `test-fixtures/e2e-fixture.db`.  ## Config example ```json "regionQuickPicks": [ { "name": "South Bay", "description": "San Jose and Monterey observers", "regions": ["SJC", "MRY"] }, { "name": "East and North Bay", "description": "Oakland and San Francisco observers", "regions": ["OAK", "SFO"] } ] ``` Each pick needs a `name` and at least one code; `description` is optional (shown as the tooltip). Codes are trimmed, upper-cased and de-duplicated; a pick with no code that has an observer today is not shown. At most 20 picks of 200 codes. ## Browser validation The branch's server was run locally against a migrated copy of `test-fixtures/e2e-fixture.db` (observers in MRY, OAK, SFO, SJC) with three groups configured: "South Bay" (SJC, MRY, SCZ), "East and North Bay" (OAK, SFO, sts) and "Nowhere yet" (XXX). Checked in headless Chromium: | check | desktop 1280 px | phone 390 px (touch) | |---|---|---| | groups shown | South Bay, East and North Bay ("Nowhere yet" hidden) | same | | tap South Bay selects | SJC, MRY (SCZ has no observer) | same | | next `/api/packets` request | `region=SJC,MRY` | `region=SJC,MRY` | | selector label | "South Bay ▾" | "South Bay ▾" | | second tap | back to all regions | same | | button height | 24 px | 44 px | | page wider than viewport | no | no | | page errors | none | none | Checked on Packets (dropdown layout) and Nodes (pill layout). Three layout problems found this way on the phone were fixed before opening: a sideways-scrolling row clipped at the screen edge, the selector squeezed into a narrow column (the filter group is a non-wrapping flex row), and the two wrapped rows spread apart on Nodes (`align-content`). **Revision 1:** the first version put the picks in a separate row beside the selector; they moved inside the dropdown. Re-checked with a 14-region list and three picks on Packets: desktop 1280 px dark and phone 390 px light both keep the menu on screen (menu 340 px wide, page not wider than the viewport), the pick closes the menu and the trigger names the pick. **Revision 2:** picks are checkbox rows like **All** instead of chips. Re-checked on Packets at 1280 px dark and 390 px light: rows render bold under All, the divider separates the single regions, ticking a pick ticks its codes and names the trigger, no sideways scroll. Unrelated and not changed: on Nodes at 390 px the region row (existing pills included) starts flush with the screen edge. ## Tests - Red commit `81ea67d6`: the config type, a stub normaliser and endpoint returning nothing, and tests. - Go: cleaning (trim, upper-case, de-duplicate, drop incomplete groups), caps, nil config, and the endpoint's JSON. - Frontend, `tests/e2e/test-region-quick-picks.js` (moved from `tests/unit/` in `fced69a7`; it needs jsdom, so it runs in the E2E job like `test-table-sort.js`): the real `region-filter.js` in **jsdom** against faked config endpoints. Nine cases: rendering and hiding groups with no observer, selecting and re-querying, the selector label, toggling back, a hand-made selection marked active, the pill layout, no groups, a failed fetch, HTML escaping. Seven fail on the stub, all on assertions. - Green commit `0e33210e`: the implementation, styles, `config.example.json` (`regionQuickPicks` + `_comment_regionQuickPicks`), `docs/api-spec.md`, and the OpenAPI route list (`TestOpenAPICompleteness` requires it). The jsdom test also had a cross-realm comparison bug fixed in this commit (`Array.from` on arrays coming out of the jsdom window). - `make test`: 16 of 16 modules pass. Frontend suites: 195 of 196; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. - Red commit `f991060d` (revision): picks live inside the dropdown menu before the checkboxes, choosing one closes the menu, and in the pill layout they share the region bar; two fail on assertions against the old layout. Green commit `1f720e35`. Frontend 195 of 196 (same environment-only failure). - Red `134f5256` / green `da0adb50` (second revision): picks are checkbox rows between All and the single regions with a divider, no chips in the menu; ticking and unticking a pick row. The green commit also repairs `style.css`: `1f720e35` had prepended its block to the top of the file (its edit matched an earlier `.region-pill {`) and left the previous rules in place. The file now differs from `master` by 13 lines. ## Review notes - Groups are offered only when at least one of their codes appears in `/api/config/regions`, and only those codes are selected. The stored selection is pruned to known codes on load anyway, so selecting absent codes would not survive a reload. - Customizer (AGENTS.md rule 8): groups are operator configuration; a customizer editor could come later. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `b84ace4e`: - `normalizeQuickPickCodes` split out of the config normaliser (S3776, 17 > 15); - a comment on why a failed quick-pick fetch is ignored (S2486); - `for…of` in `activePick` (S4138); - in the tests: `node:` specifiers, `includes()`, and an explicit string comparator for sorted selections (S7772, S7765, S2871). - **Kept on purpose:** - S2486 is still reported on that `catch`. Swallowing the error is the intended behaviour: the filter works as before, just without picks. - S1523 on the `vm` test loader, as above. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
696906315a |
feat(scope-audit): transport view of every repeater seen forwarding (#2142, API) (#2167)
## Summary Backend half of #2142 (the page is the stacked PR #2169). `GET /api/scope-audit` lists only repeaters that have answered a declared-regions request: 17 of almost 200 nodes on the instance the request came from. This adds a transport view, `GET /api/scope-audit?mode=transport&window=…[&scope=X]`, with one row per repeater **seen forwarding** in the window: - `transported`: the named region scopes it was seen carrying, each with packets and first/last seen (most packets first), plus unscoped, unmatched and ambiguous-hop counts; - `asked`, `declaredRegions`, `notObserved`, `configState`, `declaredAt` where a declared answer exists; `asked: false` with null declared fields where it does not ("not asked" is not "declares nothing"); - with `scope=X` (spelled `be`, `#be` or `BE`): `carriesScope` on every row and `carrying` / `notCarrying` counts, which is the rollout report the issue asks for. `mode=declared` (or no `mode`) is unchanged. An unknown `mode` is a 400. Partial fix for #2142: this PR provides the API; the page is #2169 and the observer region filter #2171. The issue should only be considered done once all three are in. ## How it follows the issue's notes - **The scan does not change.** It is the same single `ScopeAuditForwarding` pass, with every repeater and room in `nodes` (plus any declared target) as a target instead of only the declared ones. Same attribution rules: FLOOD-family route types only, `minForwarderHopHexLen` (no 1-byte hops), and a hop whose prefix matches more than one repeater is credited to none of them. - **Identity lookup with 1,000+ pubkeys:** `scopeAuditNodeIdentities` runs one `IN` query; exercised at 1,200 keys in the benchmark below. - **Row count and response size:** measured below. - **Cache:** the transport view is cached per window with the declared view's TTLs (30 s; 5 min for 7d), under the same singleflight group with a `transport|` key. The scope filter is applied to a copy of the cached result, so one scan serves every scope. (`scope=`, not `region=`: across the API `region=` is the observer IATA filter.) - **Wording:** "transported" is documented as *seen carrying*, not configured for; the declared side stays separate. ## Performance `BenchmarkComputeScopeTransport`: 1,200 repeaters (#1975 quotes an instance with 1,179), 30k flood transmissions in the last 24h on 4-hop paths of 3-byte prefixes, a third region-scoped: ``` BenchmarkComputeScopeTransport-12 5 219253727 ns/op 425.4 response-KB 1200 rows ``` 219 ms per compute, served from cache for 30 s (24h) or 5 min (7d); a 425 KB uncompressed response for 1,200 rows. This is new code with no "before"; the declared view's numbers are untouched (same scan, same cache). ### On real data (a staging copy of a production database and its live feed) | request | rows | result | time | |---|---|---|---| | declared view, 24h | 16 | (unchanged) | | | `mode=transport`, 24h | **385** repeaters seen forwarding, 16 of them with a declared answer | | 1.5 s cold | | `mode=transport&scope=<scope A>`, 7d | 664 | **51 carrying it, 613 not** | 14.4 s cold | | `mode=transport&scope=%23<SCOPE B>` (`#` and capitals), 7d, same window now cached | 664 | 237 carrying, 427 not | 0.35 s | The 7d cold compute is the same full-window hop scan the declared view already pays for 7d (its own TTL comment measures 16.7 s on a live-shaped database); like the declared view it is then cached for 5 minutes, and every scope filter is served from that one scan. - Commit `ceaeb56b` renamed the filter from `region=` to `scope=` after opening; the real-data figures above were taken before the rename and are otherwise the same code. ## Tests - Red commit `2a9b1286`: response types and a `mode=transport` stub that returns no rows, plus tests. Four fail on the stub: - repeaters that never answered are listed with `asked: false` and null declared fields (and the answered one with its declared list and `notObserved`); - `scope=fr` / `%23fr` / `FR` marks carriers and non-carriers and counts them; no scope means no `carriesScope` or counts; - an unknown mode is a 400; - the transport and declared views are cached separately (requesting one does not serve the other). - Also added, passing on the stub by construction: a repeater with no traffic in the window is not listed; the node blacklist applies. - Green commit `60c669f4`: the implementation, `docs/api-spec.md` (new `mode` and `scope` parameters, a Transport view section with the response shape), and the benchmark. - `make test`: 16 of 16 modules pass, including every existing scope-audit test. ## Review notes - Hidden-name and blacklist filtering use the same rules as the declared view. - `newTestStoreWithDB` now takes `testing.TB` so the benchmark can use it (callers unchanged). - No new `map[string]interface{}`; the response is a named struct. - No config added. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `060668b6`: S3776 (cognitive complexity 41 > 15) in the new `computeScopeTransport`. It is split into target, visibility, row and sort helpers; the response is the same and the transport tests are unchanged. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0737f10a5d |
feat(notify): optional external node alerts feed and the node.external event (#2177) (#2178)
Closes #2177. ## What Node notifications can mail alerts computed outside CoreScope. With `userManagement.notifications.externalAlerts` configured, users can choose a new event, `node.external`, which is off by default and only offered when configured: ```json "notifications": { "enabled": true, "externalAlerts": { "url": "http://coverage-job/alerts.json", "maxAgeHours": 48, "label": "Coverage" } } ``` The feed: ```json { "generatedAt": "2026-10-11T01:35:12Z", "alerts": [ { "pubkey": "<64 hex>", "key": "SE", "text": "never heard to the south-east", "url": "https://example.org/coverage" } ] } ``` Mail lines: `<name>: Coverage: never heard to the south-east, <time>` when an alert appears, `<name>: Coverage resolved (SE), <time>` when it is gone from a fresh feed. ## How - `cmd/server/notify_external.go`: feed types, `parseExternalFeed` (RFC 3339 `generatedAt`, age check, per-alert validation), `fetchExternalFeed` (200 only, 1 MB cap, 10 s timeout). - `notify_eval.go`: `evalExternal`. Subjects are `<pubkey>/<key>`; a `<pubkey>/*` row is the per-node baseline, so the first fresh feed after watching stores the current alerts without a mail (like `foreign.new`), and later alerts mail at once. `External == nil` (not configured, nobody chose it, or stale) leaves every `node.external` state as it is. - `notify_service.go`: the feed is fetched only when configured and at least one user chose the event. Each new reason the feed is unusable is logged once, and so is the recovery. Mail text and link: the alert's `url`, otherwise the node page. - `notify_handlers.go`: `availableEvents` and the new `externalLabel` include the event only when configured; PUT answers 400 for `node.external` otherwise. - `internal/users/notify.go`: event constant, `OptionalNodeNotifyEvents` (canonical order: node, optional, admin), `RemoveWatch` also deletes the node's `<pubkey>/...` rows. - Account page label, OpenAPI descriptions, `config.example.json` comment, `docs/user-guide/accounts.md` section. Decisions on the three open questions in #2177: the generic name `node.external`; the resolved mail names the key instead of repeating the text, so there is **no `users.db` schema change** (a bump would also make older binaries refuse the file); one feed. ## Invariants - Read/write separation (#1283): the server only reads the feed over HTTP. State rows go to `users.db` through `internal/users`, like the other events. - A dead, garbled or old feed never mails "resolved": `TestEvaluateExternalStaleFeedChangesNothing` and `TestNotifierMailsExternalAlertsAndSkipsAStaleFeed` (HTTP 502 and a 72 h old feed in a row, then a fresh empty feed resolves). ## Performance The evaluator adds one pass over the stored states (already read every tick) plus, per user with the event on and per watched node, the node's current alerts. Bounded by `maxWatchesPerUser` (50) times the alerts per node. The feed is fetched once per tick (default 5 minutes) only when someone chose the event, capped at 1 MB. No packet-store lock is taken for it. Nothing changes for instances without `externalAlerts`. ## Tests - `cmd/server`: `go test` green locally (Windows; `TestSaveGeoFilterPreservesFileMode` skipped as it needs Unix mode bits). New: `notify_external_test.go` (feed parsing and limits, HTTP fetch 200/404/oversized, config resolution, evaluator baseline/new/resolved/returning/stale/not chosen/notifications off, notifier end to end with a fake feed, account routes). - `internal/users`: green, 2 new tests (optional event and canonical order, `RemoveWatch` deletes external subjects only for that node). - Frontend: `test-all.sh` green, 1 new test in `test-notifications-ui.js` (label from `externalLabel`, escaped). - `scripts/check-xss-sinks.sh --file public/notifications.js`: clean. ## Not done - No E2E test: the account page change is one label, covered by the unit test. - Not run against a live instance yet. The first feed in use is the coverage job of on8ar.eu (`alerts.json`, 29 alerts at the moment). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c2a590b741 |
feat(companions): linked-only ingest, account Companions and My coverage (#2176)
Part 3 of 3 for #2173 (companion linking): the optional linked-only ingest filter, the account page and the "My coverage" toggle, plus the docs. Builds on #2174 and #2175 (both merged); `upstream/master` is merged in, so the diff shows only this part. ## Changes in this PR **Ingestor: linked-only ingest** (`cmd/ingestor/linked_companions.go`) - With `clientRxCoverage.requireLinkedCompanion` on (default off) and user management enabled, every `meshcore/client/<pubkey>/...` message whose pubkey is not in `companion_links` is dropped before any client handler runs, and counted (`client_unlinked_dropped` in the stats file). - The ingestor opens `users.db` read-only (`mode=ro`, raw SQL, no `internal/users` import), like the approved-channels reader, so the #1283 write invariant holds. `TestLinkedCompanionSetIsReadOnly` pins it. - The set refreshes every 60 s. A miss re-reads at most once per 5 s, so a companion linked a moment ago is accepted within seconds while a flood of unknown pubkeys costs one read per 5 s. - It passes everything until its first successful read, and when `users.db` has no `companion_links` table yet (a server older than this). After the first read it keeps the last set on a read error. It is a filter, not a security boundary: anyone with the shared broker credential can still publish under a linked pubkey. **Frontend** - Account page: device tokens show under Devices as "CoreDrive RX" with their label; a new Companions section lists linked companions (name, short key, linked since, last seen) with an Unlink button. - Coverage page (`#/rx-coverage`): a "My coverage" toggle for logged-in users (`mine=1`). Only the newest coverage reply is drawn, and a refused `mine=1` turns the toggle off. Under "My coverage" the page does not ask for the "nothing received" cells from #2172, and their legend entry is hidden. **Docs**: user guide (accounts: devices, companions), `docs/client-rx-coverage.md` (linking, linked-only ingest, the CORS exception), `config.example.json`, and the spec status. ## Performance The filter runs on every client message. Benchmarks with 1000 linked companions (12-core dev box, `go test -bench`, 3 runs each): | Path | Time | Allocations | |---|---|---| | linked pubkey (lock-free map lookup) | 54 ns/op | 0 | | linked pubkey, parallel | 11-14 ns/op | 0 | | unlinked pubkey inside the 5 s miss gap | 85-95 ns/op | 0 | | one `users.db` re-read | 0.38 ms | 3030 | The unlinked benchmark asserts that no extra `users.db` read happens. A re-read happens at most once per 5 s on a miss and once a minute from the refresh loop. ## Verification - `cmd/ingestor`: 15 new tests (filter off changes nothing, linked passes, unlinked dropped before every handler, miss refresh accepts a new link, the 5 s cap holds under a flood, unlink honoured after the periodic refresh, missing table and missing file, fail-closed only after the first read, read-only open, startup, stats counter) and 4 benchmarks. Full ingestor suite green. - Frontend unit tests for the account page (device tokens, companions list escaping, empty state, Unlink confirm and cancel, load errors) and `tests/unit/test-rx-coverage-mine.js`. - E2E `tests/e2e/test-user-management-e2e.js` gains a companion step (device token under Devices, a companion linked with a real Ed25519 signature under Companions, Unlink). Run locally against the e2etest build: 28/28. The coverage suites (`test-rx-coverage-gaps-e2e.js`, `-noise-`, `-viewport-`) still pass. - Frontend unit suite (`test-all.sh`) green. ## Not in this PR - Narrowing the "nothing received" cells to the caller's companions under "My coverage" (the track query takes one companion today). - Per-user broker credentials or ACLs, which would turn the filter into an access control (non-goal in the spec). --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
b2cda21a27 |
feat(server): companion linking routes, device tokens and bearer auth (#2175)
Part 2 of 3 for #2173 (companion linking): the server routes, bearer auth and the link handshake. Builds on #2174 (merged); `upstream/master` is merged in, so the diff shows only this part. Everything here is behind `userManagement.enabled` and answers 404 with it off. ## Changes in this PR **Device tokens and bearer auth** - `POST /api/auth/device-token`: email + password returns a bearer token, stored as a `kind = 'device'` session with a label. Same failure answers and the same rate limit as login. - Server-enforced scope for a bearer token: `/api/auth/me`, `/api/auth/logout`, `/api/account/companions*`, `/api/account/settings`. Everything else answers 403. - A bearer header only counts when there is no session cookie, so an auth proxy that adds its own `Authorization` header does not hide a logged-in browser. Cookies never accept a device token. - The sessions list on the account API carries `kind` and `label`; logout with a bearer token revokes that device. **Linking a companion** - `POST /api/account/companions/challenge` `{pubkey}` returns `{challenge, expiresAt, host}`: 32 random bytes, single use, 5 minutes, bound to the caller and the pubkey. - `POST /api/account/companions` `{pubkey, challenge, signature, name}`: the signature is the companion's Ed25519 signature over `corescope-link:<host>:<challenge>` (`internal/sigvalidate.VerifyMessage`). The challenge is consumed in every case. - A companion linked to another account moves to the caller (the newest proof wins). Both users get a `companion.transfer` audit row and the previous owner a mail. That mail is a security notice: it is sent whatever the node notification settings are. - `GET /api/account/companions` (with `lastSeenAt` from `client_receptions`) and `DELETE /api/account/companions/{pubkey}`. - Linking does not touch the synced `meshcore-my-nodes`: a companion is not a node to monitor. - Admin user detail lists linked companions; the account data export includes companions and device sessions. **Coverage and config** - `GET /api/rx-coverage?mine=1`: only coverage of the caller's linked companions (401 when logged out, 403 for a device token). With `gaps=1` (#2172) the reply carries no `gaps` member under `mine=1`, because the track query takes one companion and cannot narrow to a set yet (`TestRxCoverageMineOmitsGaps`). - `clientRxCoverage.requireLinkedCompanion` (default off) is read and reported in `/api/config/client`, which also announces `userManagement.companionLinking`, so CoreDrive RX shows its login only where linking exists. The ingestor side is part 3. - CORS: for origins in `corsAllowedOrigins`, the device-token routes alone also allow POST/PUT/DELETE and the `Authorization` and `Content-Type` headers. No credentialed CORS. Documented in `config.example.json`. - OpenAPI and `docs/api-spec.md` cover the new routes. ## Verification - 26 new tests in `cmd/server` and `internal/sigvalidate`, among them: `TestCompanionLinkFlow`, `TestCompanionLinkRejections` (signature by another key, malformed signature, challenge bound to another pubkey, unknown challenge; reuse and expiry are covered by the store tests in #2174), `TestCompanionTransfer`, `TestCompanionTransferMailIgnoresNotificationSettings`, `TestBearerScopedRoutes`, `TestBearerIgnoredWithCookie`, `TestDeviceTokenSharesLoginRateLimit`, `TestCORSBearerRoutes`, `TestCompanionRoutesAbsentWhenOff`, `TestRxCoverageMine`, `TestRxCoverageMineOmitsGaps` (fails without the `mine == nil` guard), `TestVerifyMessage`. - `cmd/server`, `internal/sigvalidate` and `internal/users`: `go vet` and `go test ./...` green, `gofmt` clean. - The linking flow runs on analyzer.on8ar.eu since 9 October 2026 with CoreDrive RX v1.19.0 (a real companion linked through the signed challenge). ## Not in this PR - The ingestor's linked-only filter, the account page (Devices, Companions), the "My coverage" toggle and the user docs: part 3. - Per-user broker credentials and per-user API keys (non-goals in the spec). --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
9897d52cf6 |
feat(users): companion linking store (schema v6, device sessions, link challenges) (#2174)
Part 1 of 3 for #2173 (companion linking). This PR adds only the `users.db` side: the schema and store methods the server (part 2) and ingestor (part 3) build on. No route, no UI and no behaviour change for a running server. ## Changes in this PR - **Design spec** `docs/specs/2026-10-08-companion-linking-design.md`, including the optional linked-only ingest. - **Schema v6** (`internal/users/schema.go`): - `sessions` gains `kind` (`'web'` or `'device'`), `label` and `scopes` (`''` = full, for web sessions). - `companion_links (pubkey PRIMARY KEY, user_id, name, linked_at)`, cascading on user delete. - `link_challenges`: single-use challenges bound to a user and a pubkey, with an expiry. - **Device sessions** (`sessions.go`): create a device session with a label and scopes, list sessions with kind and label. - **Challenges** (`challenges.go`): issue and consume a challenge exactly once; consumed, expired, or bound to another user or pubkey all fail. - **Companion links** (`companions.go`): link, list, unlink. Linking a pubkey that belongs to another user moves it and reports the previous owner (transfer detection), so the server can audit and mail. - **Janitor**: the existing janitor also prunes expired challenges (`cmd/server/auth_service.go`, 3 lines). ## Upgrade note The store migrates `users.db` to v6 on start. A binary that only knows v5 refuses to open a v6 file (`users: database schema version 6 is newer than this binary supports (5)`), so rolling back after this lands needs the pre-upgrade `users.db` backup. ## Verification - `internal/users`: new tests for the v6 migration from v5, device sessions, challenges (single use, expiry, binding) and companion links (transfer, cascade on user delete). `go vet` and `go test ./...` green. - `cmd/server` full suite green against the v6 store. ## Not in this PR - Routes, bearer auth and the link handshake: part 2. - Ingestor filter, account page and "My coverage": part 3. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
9dbc287579 |
docs: release notes for v3.14.0 (#2143)
Release notes and the CHANGELOG entry for v3.14.0. The release workflow reads `docs/release-notes/v3.14.0.md` from the tagged commit as the release body, so this lands before the tag. 23 commits since v3.13.1: 13 fix, 8 feat, 1 ci, and this docs commit. The headline is optional user accounts (#2128). "Read this before upgrading" covers the new limits on unauthenticated endpoints, the `traffic_share_score` drop (#2117), the region filter window (#2114) and the config file mode (#2126). After merge: tag this commit as `v3.14.0` once its `:edge` image is built, then dispatch the CI/CD pipeline on the tag. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2456762179 |
feat: account data export and users.db backup (#2141)
A small follow-up to parts A to E of #2128: users can download all data the instance holds about them, and the server backs up `users.db`. Stacked on #2140. Review the commits after that branch. ## The situation - An instance with accounts holds personal data, but a user has no way to get a copy (GDPR articles 15 and 20). Self-delete already exists. - `users.db` is the only file the server writes, and nothing backs it up: `GET /api/backup` snapshots the analyzer database only. ## What this PR adds **Data export.** `GET /api/account/export` and a "Download my data" button on the account page give one JSON file with: - the profile, including activation and a pending email change; - sessions (times and user agent); - synced settings, own proposals, notification prefs and watches; - audit rows where the user is actor or target, with other accounts as an id only; - the mail log. Password hash, session and link tokens and the unsubscribe token are left out: they are credentials. Each export is audited as `user.export`. **users.db backup** - Daily `VACUUM INTO` snapshots through `internal/users`, on by default when user management is on. They go to `backups/` next to `users.db`, the 7 newest are kept, the directory is 0700 and the files 0600. Rotation only touches `users-<timestamp>.db` files and never the snapshot it just wrote. Config: `userManagement.backup {enabled, dir, keep}`. - `GET /api/admin/users-backup`: an admin downloads a fresh snapshot to keep off the server; audited as `user.backup`. - `docs/user-guide/accounts.md` gains a restore procedure. It covers what a restore brings back (accounts deleted after the snapshot, old passwords, revoked sessions) and how to clean that up afterwards. Spec: [`docs/specs/2026-10-08-account-export-and-users-backup-design.md`](https://github.com/efiten/CoreScope/blob/feat/account-export-backup/docs/specs/2026-10-08-account-export-and-users-backup-design.md). ## Verification - `internal/users` with `-race` and the `cmd/server` suite pass locally, 30 new Go tests. The file-mode tests skip on Windows. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 27 of 27 steps locally. - On our production instance since 8 October 2026; the first snapshot there was 143 kB. ## Not in this PR - Importing an export into another instance. - Off-site upload of the snapshots. The admin download covers that by hand. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
de364a8bb6 |
feat: mail notifications for watched nodes (user management part E) (#2140)
Part E of #2128: a logged-in user watches nodes and gets one mail when a watched node goes offline, comes back, or reports a low battery. Admins can add instance events: a new foreign node, and an observer going offline or back. This covers the per-user mail part of #775. Stacked on #2139 (part D, which builds on #2138). Review the commits after that branch. ## The situation A sysop learns that a repeater went silent, or that its battery is running down, only by opening CoreScope. #775 asks for notifications. Accounts (part A) now give a verified address to mail and a mailer with delivery status. ## What this PR adds **Events.** They use the instance thresholds, so a mail does not disagree with the node page: - `node.offline`: silent for the role's `healthThresholds` window. Last heard comes from the packet store, and repeaters and rooms count relayed traffic (#1598). - `node.battery`: advert telemetry below `batteryThresholds.lowMv`, recovered at `lowMv + 100`. - Admins only: `foreign.new` (once per node) and `observer.offline`. **Store.** Schema v5 with `notification_prefs`, `notification_watches` and `notification_state`. **Server** (only with `userManagement.notifications.enabled`) - A notifier next to the janitor, every `intervalMinutes` (5). The evaluator is a pure function of watches, prefs, stored states and node snapshots. - No burst after a restart or a new watch: a subject's first evaluation stores its state without mailing. The loop waits for the startup load. While ingest is stale (newest packet older than 30 minutes) the offline checks pause, and after recovery they wait one silent window. - One mail per user per check. Limits: 20 per user and 100 per instance per rolling 24 hours, 50 watches per user. A change over a limit is recorded and never mailed later. - Every mail has a one-click unsubscribe (`List-Unsubscribe` and `List-Unsubscribe-Post`). The GET only redirects to a confirm page, so link scanners cannot unsubscribe anyone. Node names are cleaned of control and bidi characters before they go into a mail. - Routes: `GET`/`PUT /api/account/notifications`, `PUT`/`DELETE /api/account/notifications/watches/{pubkey}`, `POST /api/account/notifications/watch-my-nodes` (copies the synced "my nodes" list), `GET`/`POST /api/notifications/unsubscribe`. `mailer.Message` gains `Headers`. **Frontend.** A "Notify me" toggle on the node pages, a Notifications section on the account page (`#/account?section=notifications`), an unsubscribe page, and the notification mail count on the admin overview. Spec: [`docs/specs/2026-10-07-node-notifications-design.md`](https://github.com/efiten/CoreScope/blob/feat/notifications/docs/specs/2026-10-07-node-notifications-design.md). ## Performance Each check reads watches, prefs and states from `users.db` (one query each), the watched nodes from the analyzer DB in chunks of 500, and last-heard times from the packet store. | Measurement | Result | |---|---| | One check on our staging instance, 130,000 packets in memory, 1 watcher | 6.9 ms, of which 0.93 ms holding the store read lock | | Evaluator benchmark, 100 users with 50 watches each, 2,000 nodes | 1.75 ms | No work is added to ingest, broadcast or any request path. ## Verification - `internal/users` and `internal/mailer` with `-race`; the `cmd/server` suite plus `-race` on the notifier tests; 74 new Go tests. The same Windows-only failure as noted in #2138 applies. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 26 of 26 steps locally. With notifications off the new steps fail, so they do test the flag. - On our staging and production instance since 7 October 2026. ## Not in this PR - Other channels (Discord, Telegram, webhooks). Detection is separate from delivery, so one can be added. - The topology, RF and anomaly alerts of #775. - If the whole server starts after a feed outage that already ended, the grace window is unknown and a watcher can get one wrong offline mail. The user guide says so. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
abbe6846ff |
feat: propose and approve hashtag channels (user management part D) (#2139)
Part D of #2128: logged-in users propose a hashtag channel, an admin approves, rejects or later revokes it, and the ingestor decrypts approved channels without a restart. It is the first consumer of a generic proposal table, and answers the request in #2092. Stacked on #2138 (part C). Review the commits after that branch; GitHub shows C's commits here until it is merged. ## The situation - An instance decrypts only the hashtag channels listed in `hashChannels`. Users who want a city or club channel shown have to ask the operator, who edits `config.json` and restarts the ingestor. - #2092 asked for a suggest-and-approve flow; a downstream fork (dborup/CoreScope#99) runs one with anonymous suggestions and a file queue. ## What this PR adds **Name rules (`internal/channel`).** `ValidateHashtagName`, mirrored in the browser: - at most 31 UTF-8 bytes including `#` (firmware `ChannelDetails.name[32]`), case preserved, `Public` refused in any case; - control, bidi, separator and format characters refused except ZWJ, plus invisible fillers (`Other_Default_Ignorable_Code_Point`, U+2800, non-ASCII spaces). Go and JS give the same verdict for every code point. **Store.** Schema v4, one generic `proposals` table (kind, subject, status pending/approved/rejected/revoked, proposer, reviewer, note). One row per (kind, subject); limits are checked in the same transaction as the change. **Server** (only with `userManagement.channelProposals.enabled`) - `POST /api/proposals`, `GET /api/account/proposals`, `GET /api/admin/proposals`, `POST /api/admin/proposals/{id}/approve|reject|revoke`. - 400 for an invalid name. 409 for a duplicate, a rejected name, or a name already in `hashChannels`/`channelKeys` ("this channel is already decrypted on this instance"). 429 over a limit (`maxPending` 100, `maxApproved` 128, `perUserPerDay` 5). - `GET /api/channels` gains `approvedChannels`, so approved channels are listed before they have traffic. With the feature off the response is byte-identical. **Ingestor.** Opens `users.db` with `mode=ro` every 60 seconds, re-validates the names, and swaps its key map through an `atomic.Pointer`. Configured channels always win, and a failed read keeps the last good set. The ingestor never writes `users.db`; AGENTS.md now says so. **Frontend.** "Propose for everyone" in the add-channel dialog, an "approved" marker in the channel list, "My proposals" on the account page, and an admin "Proposals" tab with a warning on approve. The packet detail pane now escapes the channel name: until now only the operator chose those names. Spec: [`docs/specs/2026-10-07-channel-proposals-design.md`](https://github.com/efiten/CoreScope/blob/feat/channel-proposals/docs/specs/2026-10-07-channel-proposals-design.md). ## Performance Every key is tried on each GRP_TXT that no key opens (`decodeGrpTxt`). Benchmark of that worst case: | Keys | Time per packet | |---|---| | 320 configured | 204 to 216 µs | | 320 configured + 128 approved | 272 to 320 µs | That is about +33%, linear in the key count and capped by `maxApproved`. With the feature off the decoder gets the identical map and no ticker runs. ## Verification - `internal/channel`, `internal/users` (with `-race`), `cmd/server` and `cmd/ingestor` pass locally, 53 new Go tests. The ingestor's `TestWriteStatsAtomic_SymlinkAtDestIsReplaced` needs symlink rights on Windows and fails on `master` too. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 21 of 21 steps locally (propose, approve, the channel appears, revoke removes it). - On our staging and production instance since 7 October 2026. ## Not in this PR - Private (PSK) channels: their keys stay in the browser (#725). - Anonymous proposals. - Names that need ZWNJ (some Persian and Urdu spellings) and subdivision flags are refused, because ZWNJ and tag characters are format characters. Allowing them later only widens the rule, so no approved name gets stranded. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
18a634c115 |
feat: admin dashboard with overview and audit log (user management part C) (#2138)
Part C of #2128: an admin area with an overview, the user table and a global audit log. Off by default like the rest of user management: with `userManagement` off nothing changes, and non-admins get no new UI. This is the first of 4 stacked PRs (C, D, E, then a small export/backup follow-up). Each later one contains this branch; review them in order. ## The situation - An admin could manage users one by one in `#/admin/users`, but nothing showed whether the instance needs attention: accounts stuck in activation, bouncing mail, someone guessing a password, an MQTT source that dropped. - The audit log existed (`internal/users/audit.go`) but could only be read per user, and logins were not recorded. ## What this PR adds **Store (`internal/users`)** - Schema v3: an index on `audit_log(at)`. - `AuditList` with filters (action or group prefix like `user.login.*`, user as actor or target, period) and keyset pagination; `PruneAudit`; `Stats` for the user figures. **Server** - `GET /api/admin/stats` (typed struct): accounts by status, admins, registrations and active users over 7 and 30 days, logins and failed logins in 24 hours, mail by final status, and the attention items computed server-side. - `GET /api/admin/audit`: filtered, newest first, `next` cursor. - `GET /api/admin/users` gains `bouncing=1`. - Logins are audited as `user.login` and `user.login.failed` (reason `wrong_password`, `pending` or `disabled`). An unknown address writes no row. The writes are asynchronous, so the login response does not wait on `users.db`. Login rows are pruned after 90 days. **Frontend** - `#/admin?tab=overview|users|audit`: `admin.js` (tab shell), `admin-overview.js` ("Needs attention", Users card, System card from the existing health, MQTT and observer endpoints), `admin-audit.js` (filters in the URL, "Load more"). `admin-users.js` becomes the Users tab; the old `#/admin/users` link rewrites to it. - `/api/healthz` is read on open and on Refresh only, not on the 60-second timer, because it walks every packet under a read lock. Spec: [`docs/specs/2026-10-07-admin-dashboard-design.md`](https://github.com/efiten/CoreScope/blob/feat/admin-dashboard/docs/specs/2026-10-07-admin-dashboard-design.md). ## Verification - `internal/users` and `cmd/server`: `go vet` and `go test` pass locally, 26 new Go tests. One upstream test, `TestSaveGeoFilterPreservesFileMode`, also fails on Windows on plain `master` (file modes) and is unrelated. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E with the `e2etest` build: 13 of 13 steps locally. - Running on our staging and production instance since 7 October 2026. ## Not in this PR - Restricting existing pages (perf, MQTT status) to logged-in users or admins. The admin area adds no access mechanism of its own, so that stays a later router rule plus endpoint check. - Server-enforced customizer tab restrictions (#1508). - The 24-hour, 5-attempt and 10-minute attention thresholds are constants for now (AGENTS.md rule 8: customizer later). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f3ae8ac25a |
feat: sync a logged-in user's settings across devices (part B) (#2130)
Part B of #2128: a logged-in user's settings follow them across devices. Log in on a phone and your own nodes, favorites, customizer and filters are there; a change on one device reaches the others within a minute or when you return to the tab. **This PR builds on #2129.** Until that one is merged, the diff here includes it. The commits for this part start at `docs(specs): settings sync for optional user management (sub-project B)`. ## The situation Everything a visitor sets up lives in one browser's `localStorage` (about 100 keys in `public/`). A second device or a cleared cache starts from zero (#895). ## What this PR adds **Storage.** `users.db` schema v2: one JSON document per user in `user_settings`, with a revision number and a generation id. A write succeeds only when the client's revision and generation match the stored ones, so two devices cannot overwrite each other silently. **Server.** `GET`, `PUT` and `DELETE /api/account/settings`, behind the same session and CSRF checks as the account routes. - The server owns the list of synced keys (61 keys, [`settings_allowlist.go`](https://github.com/efiten/CoreScope/blob/feat/settings-sync/cmd/server/settings_allowlist.go)) and sends it to the client, so the two cannot drift. - A hard denylist, checked first, refuses `meshcore-api-key`, every `corescope_channel_*` key and `live-channel-colors` (#725). The colour map is keyed by channel hash, and for a user-added channel that hash is `user:<name>`, which would expose hashtag channel names. - Documents are capped at 256 KiB, measured like `JSON.stringify`. PUT is limited to 60 requests per hour per user. A stale revision gets 409 with the current document. **Client** ([`settings-sync.js`](https://github.com/efiten/CoreScope/blob/feat/settings-sync/public/settings-sync.js)). Inert unless the feature is on and someone is logged in. - It wraps `localStorage.setItem` and `removeItem` for allowlisted keys only and pushes 2 seconds after the last change. - It pulls on login, page load, tab focus and every 60 seconds while the tab is visible. - **Merge:** three-way, against a per-device baseline that belongs to one user and one document generation. Lists (own nodes, favorites, saved filters) merge per item, so an item added anywhere is kept and an item removed on one device does not come back from another. Single values: the profile wins unless only this device changed it. - Remote changes are written without a push, theme and colour-blind preset are re-applied, and the current page re-renders (skipped on account pages and while the geofilter editor is open). **UI.** - Logout asks: keep my settings on this device (default), remove them from this device, or cancel. Channel keys are never removed: no copy exists anywhere else. - The account page gets a "Settings sync" section: last synced time, "Sync now", what is and is not synced, and "Delete synced settings from my account". ## Not synced Layout and device state (panel and column widths, collapsed panels, map positions, geofilter drafts), channel data (#725), the API key, and all `sessionStorage`. The full list is in the [spec](https://github.com/efiten/CoreScope/blob/feat/settings-sync/docs/specs/2026-10-06-user-settings-sync-design.md). ## Performance - One GET per page load, tab focus and minute while visible; one debounced PUT per burst of changes. - The `setItem` wrapper costs one Set lookup per write for non-synced keys. A synced write reads one small revision key, not the stored document. - The server reads or writes one row per request. ## Verification - `internal/users` and `cmd/server`: `go vet` and `go test` pass locally (22 new Go tests), including a test that every allowlisted key still occurs in `public/`, and denylist tests. - `tests/unit/test-settings-sync.js`: 79 passing (vm, real module). The cases cover the merge table, two tabs sharing one storage, stale answers after a push, delete while a push is in flight, and logout while the final push fails. - `sh test-all.sh` exits 0. - `tests/e2e/test-user-management-e2e.js` (10 steps, 4 of them new) passed locally with two browser contexts as two devices: a favorite and the packet time window travel from device 1 to device 2, a removal does not come back, and "remove from this device" clears the synced keys while a channel key stays. - Checked by hand on a staging instance with a desktop and a phone on one account. ## Not in this PR - On a shared browser where the previous user chose "keep", the next user's first login merges those settings into their own account. The user guide says to choose "remove" on shared computers. - Saved filter expressions are synced as typed, including any channel names written in them. The guide says so. - Realtime push between devices; the minute pull is the sync interval. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
d232072f85 |
feat: optional user accounts (part A: foundation) (#2129)
Part A of #2128: optional, off-by-default user accounts. With the feature off nothing changes; with it on, visitors can register and log in, and admins manage users and use the operator actions without the API key. PR #2130 (settings sync) builds on this one. The two are meant to be merged together. ## The situation - Operator actions (geofilter save and prune, backup, perf reset) need the shared `apiKey`. There is no per-person right. - Nothing in CoreScope knows who a visitor is, so the requests in #2128 that need that (#1835, #2092, #1508, #730) have nothing to build on. ## What this PR adds **Two new Go modules** - `internal/users`: a separate `users.db` (SQLite through `modernc.org/sqlite`) with users, sessions, single-use tokens, an audit log and a mail log. Passwords use argon2id. - `internal/mailer`: a `Mailer` interface with a Brevo client (send, delivery events, webhook parsing) and an in-memory fake for tests. **Server (`cmd/server`)**, active only with `userManagement.enabled` - 24 routes, all documented in OpenAPI under the `users` tag ([`auth_routes.go`](https://github.com/efiten/CoreScope/blob/feat/user-management/cmd/server/auth_routes.go)): - auth: register, activate, login, logout, me, forgot, reset; - account: profile, password, email change with confirmation, sessions, self-delete; - admin: list, detail, disable, enable, delete, role, resend activation, manual activation, mail status refresh; - a Brevo webhook, registered only when `mail.webhookSecret` is set. - `requireAdmin` replaces `requireAPIKey` at the 7 operator call sites: the API key **or** an admin session. With the feature off it is the old API-key gate (`TestRequireAdminWithoutUserManagementIsAPIKeyGate`). - `/api/config/client` gets `userManagement: {enabled: true}` only when the service started; with the feature off the response is byte-identical. **Frontend** - `auth.js` (header account control, request helper that adds the CSRF header), `account.js` (login, register, activate, forgot, reset, confirm email, my account), `admin-users.js` (`#/admin/users`, deep-linked filters), `account.css` (theme tokens only). - On phones the top-bar control is hidden, so a conditional entry goes into the bottom-nav "More" sheet and the nav drawer. - The customizer geofilter tab and the Perf "Reset stats" button use the admin session when there is one. **Config.** A `userManagement` block (`config.example.json`, [`docs/user-guide/accounts.md`](https://github.com/efiten/CoreScope/blob/feat/user-management/docs/user-guide/accounts.md)). The Brevo key can come from `CORESCOPE_BREVO_API_KEY`. The server refuses to start when the block is enabled but incomplete. ## Security choices - Session cookie `cs_session`: HttpOnly, SameSite=Lax, Secure when `publicBaseUrl` is https. Every cookie-authenticated state change needs the `X-CS-CSRF` header and a matching Origin. - Activation needs the token **and** the account password. Without the password, an attacker who keeps re-registering a known address could get the owner to activate an account that carries the attacker's password. - Register, forgot and email change answer identically for known and unknown addresses. A password reset ends all sessions, a password change ends all other sessions, and both end outstanding email-change links. - Rate limits: login 10 per 15 minutes, register and forgot 5 per hour, per IP and per address. The bucket count is capped. `trustedProxies` makes the per-IP limits see real client IPs behind a proxy. - Server logs carry `#<user id>`, never addresses, tokens or passwords; mail-provider error texts are redacted before logging. ## Performance No change to an existing hot path with the feature off. With it on: - One `users.db` lookup per authenticated request (session by token hash). - The admin user table rebuilds its `tbody` on each filter change. `users.List` caps the result at 1000 rows (`internal/users/users.go`), which bounds the rebuild. - `map[string]interface{}` in `openapi.go`: 79 before, 78 after. ## Verification - `internal/users`, `internal/mailer` and `cmd/server`: `go vet` and `go test -race` pass locally. 121 new Go tests. - `cmd/server` with `-tags e2etest`: vet and the e2e hook tests pass. - `sh test-all.sh` exits 0. `tests/unit/test-user-management-ui.js`: 67 passing (vm, real modules). - `tests/e2e/test-user-management-e2e.js` (6 steps) passed locally against an `e2etest` build with the fake mailer and against a feature-off build. CI builds the `e2etest` binary and runs the suite on a second server (`deploy.yml`). - On a staging instance with a real Brevo key: register, activation mail delivered, activate, admin table, "Refresh status" showing sent, deferred, delivered, opened and clicked. ## Not in this PR - Settings sync (#2130), the admin dashboard, approval flows and notifications (parts B to E of #2128). - A `requireReadAuth` mode (#1835). Sessions from this PR are what such a mode would accept. - Binary size and build time with `modernc.org/sqlite` linked next to `mattn/go-sqlite3` were not measured. Their driver names do not collide. #1992 discusses the driver choice. - No Brevo webhook was configured on staging; delivery status there came from "Refresh status". --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
000d9ab030 |
feat(coverage): RF noise-floor layer on the Mobile RX coverage page (#2113)
The ingestor already stores the noise floor that CoreDrive RX companions
report with each GPS fix (`client_rf_samples`, opt-in through
`clientRfSamples`, `cmd/ingestor/client_rf_sample.go:62`), but nothing
reads it back. An operator who enables it collects the data and cannot
see it. This adds the read side: `GET /api/rf-noise` and a Signal/Noise
toggle on the Mobile RX coverage page.
## What it does
- **`GET /api/rf-noise?bbox=&z=&days=`** returns a GeoJSON hex grid with
the median, quietest and noisiest noise floor per cell, in the same
shape as `/api/rx-coverage`. It is registered always and 404s unless
`clientRfSamples.enabled` is true, the same pattern as the coverage
routes.
- **Stationary samples are excluded.** A parked companion logs hundreds
of readings at one point, which would otherwise define its cell.
- **Coverage page:** a Signal/Noise toggle, rendered only when
`/api/config/client` reports `clientRfSamples: true`. The noise layer
reuses the coverage colour tokens with the axis inverted, because a
lower dBm is quieter. Tiers are at -115 and -108 dBm.
- **Deep link:** `#/rx-coverage?layer=noise` opens on the noise layer.
- **Empty and failed answers** are labelled on the map ("No RF samples
in this view yet", or a retry hint), so a blank map never reads as
"feature off".
No new configuration key: it reads the existing `clientRfSamples`
section that the ingestor already uses. Default off, so nothing changes
for an instance that has not opted in.
## Where it comes from
Ported from the efiten/CoreScope fork, where it has run on
analyzer.on8ar.eu since September (fork commits `42d09f0e`, `80581c43`,
`cde95078`). The cherry-picks conflicted with upstream's newer
`routes.go`, `types.go` and `rx-coverage.js`, so the final state was
ported by hand. The fork-only `/scopes` route that sat next to it in the
same hunk is deliberately left out.
## Performance
The query is bounded by `sampled_at` (indexed, `idx_crf_prune`) and the
bbox, aggregation is one pass plus a per-cell sort, and the response is
capped at 5000 cells. Measured on analyzer.on8ar.eu, all of Belgium
(`bbox=49.4,2.4,51.6,6.5&z=9`), from a client in Belgium, so network
time included:
| window | samples | cells | response time |
|---|---|---|---|
| 7 days | 10,835 | 241 | 0.37 s |
| 30 days | 43,303 | 429 | 0.39 s |
That table holds 45,270 rows in total. It is only read when someone
opens the noise layer, never on ingest or WebSocket paths.
## Tests
- `cmd/server/rf_noise_test.go`: 8 tests for the aggregation (median and
extremes, stationary exclusion, the cell cap, empty input) and the gate
(404 when off, even with data present). All pass, and the full
`cmd/server` suite passes locally.
- `tests/unit/test-rx-coverage-noise.js` (new, in `test-all.sh`): the
colour axis runs the right way, including both tier bounds and a string
median from the API. Slices the real function out of `rx-coverage.js`
and fails on master's copy, which has no thresholds.
- `tests/e2e/test-rx-coverage-noise-e2e.js` (new, wired in `deploy.yml`
and `scripts/non-unit-tests.json`): no toggle when the flag is off;
Noise fetches `/api/rf-noise`, draws the cells, swaps legend and
subtitle, puts `layer=noise` in the hash; a deep link opens on the noise
layer and an empty answer shows the message. 3/3 locally; against
master's `rx-coverage.js` and `roles.js`, 1/3 (only the "flag off" case
passes, as it should).
- `test-rx-coverage-viewport-e2e.js` still passes. ESLint 8 reports 0
errors on the changed files.
## Not done
- The thresholds (-115 / -108 dBm) are fitted to the fork's own data
(1,241 moving samples at the time) and are constants in
`rx-coverage.js`. Per AGENTS.md rule 8 they belong in the customizer
eventually; not in this PR.
- The E2E suite mocks the API. The real endpoint is covered by the Go
tests and by the measurements above, not by a fixture with seeded
samples.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
9c3d76e14f |
docs(release): notes and changelog for v3.13.1 (#2112)
Release notes and CHANGELOG entry for v3.13.1, a patch release that ships #2111 (node-discover replies count as coverage). Once merged, the tag goes on the merge commit so `deploy.yml` picks up `docs/release-notes/v3.13.1.md` as the release body. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
00ee4f9e23 |
fix(coverage): attribute node-discover replies to the responder (#2111)
## Problem A CoreDrive RX companion that runs node-discover gets no coverage on an upstream instance. Reported today by an operator on v3.13.0: the app logged 32 discover replies from one repeater in 16 minutes (`heard 751a49f0c5dadc70 (8B, discover)`), all published, and `client_receptions` stayed at 0 rows for that companion while `client_rx_observations` held 93. Two gaps, both on the upstream side: 1. **Ingestor.** `deriveHeardKey` attributes a FLOOD `path[last]` and a 0-hop advert, and drops a 0-hop `CONTROL/DISCOVER_RESP` (firmware `CTL_TYPE_NODE_DISCOVER_RESP`). The reply carries the responder's own pubkey at offset 6 of the payload, which `decoder.go` already parses into `CtrlPubKey` (#1802), but the coverage path never used it. 2. **Server.** CoreDrive RX asks for `DISCOVER_PREFIX_ONLY`, so the firmware answers with an 8-byte prefix (`simple_repeater/MyMesh.cpp:799-805`). `coverageHeardKeyCandidates` only built the 64, 6 and 4 hex candidates, so a 16-hex `heard_key` would be stored and then matched by no per-node coverage query. ## Change - `cmd/ingestor/client_reception.go`: a third branch in `deriveHeardKey` for a discover response with no hops. Accepts exactly 8 or 32 bytes, nothing truncated, stored with `src='discover'`. - `cmd/server/rx_coverage.go`: adds the 16-hex prefix to `coverageHeardKeyCandidates`. - `docs/client-rx-coverage.md`: documents the `discover` source, the 8-byte keylen and the four-candidate lookup. The leaderboard and `/api/rx-coverage` read `client_receptions` without a key filter, so they pick the rows up without a change. Name resolution goes through `batchResolveHeardKeys`, which is a prefix lookup and handles 16 hex as is. ## Evidence from a deployment that has had this since 2026-08-19 On analyzer.on8ar.eu, `client_receptions` over the last 7 days by `src`: discover 4232, rxlog 4909, geo 548, advert 34. Discover replies are 44% of all coverage rows there (4232 of 9723); on an upstream instance those rows are not written. They cannot be backfilled afterwards either: `client_rx_observations` keeps no raw bytes, so the responder pubkey is gone. ## Tests - `TestDeriveHeardKey` and `TestBuildClientReception` gain discover cases: 8-byte and 32-byte keys accepted (32-byte uppercase input lowercased), a 3-byte and an empty key rejected, a non-discover CONTROL rejected, a discover response with hops not attributed. - `TestHandleClientPacketDiscoverRespWritesReception`: end to end, a raw 0-hop DISCOVER_RESP on the client topic writes one `client_receptions` row with `src='discover'`. - `TestCoverageHeardKeyCandidatesIncludesDiscoverPrefix`: the 16-hex prefix is among the per-node candidates. - Ran locally on Windows: `go test ./...` in `cmd/server` passes; in `cmd/ingestor` everything passes except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which needs the symlink privilege on Windows and fails on clean master too. Not done: no browser validation, as the change is ingestor and server only and the frontend reads the same endpoints. Not included: geographic resolution of 1-byte hops (`src='geo'`) and the RF noise layer, which are separate changes. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c2a9500060 |
docs(release): notes and changelog for v3.13.0 (#2110)
Release notes and CHANGELOG entry for v3.13.0, in the shape used for v3.12.0. Once merged, the tag goes on the merge commit so `deploy.yml` picks up `docs/release-notes/v3.13.0.md` as the release body (#2076). - 16 commits since v3.12.0: 10 fix, 4 test, 2 feat, so a minor bump. - "Read this before upgrading" covers the one-off advert route evidence backfill from #2088. Durations are measured on two instances (live 24m 1s on 16.8M observations, staging 36m 23s on 16.5M), and ingest during the backfill window is measured as unaffected on live. - Issues closed come from each PR's closing references: 14 from 16 PRs. Docs only. #2089 is deliberately not in this release. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |