Commit Graph
3164 Commits
Author SHA1 Message Date
efiten af732d18aa fix(repeater-info): log bulk read errors, cap long notes (#2416)
Follow-up to #2415, see #2414.
2026-11-17 12:56:27 +01:00
efiten 0cf22cdd79 feat: repeater info, claims and community-maintained details (#2415)
Closes #2414
2026-11-17 11:40:05 +01:00
4ebb5ccc11 feat: radio line-of-sight on packet routes + repeater planner (#2391)
Both features are OPT-IN ("los" and "planner" config blocks, default
off); with them off nothing is fetched and payloads are unchanged.

Route view
- Multi-path picker discoverability: on-map "N hops shown · tap to see
all" badge, picker pulse (reduced-motion safe), one-time tip, row hover
arrow; "no GPS" hops counted apart from "unresolved".
- Line of sight for every leg of a packet route: clear / marginal /
blocked chips (first-Fresnel clearance, k=4/3), obstruction markers on
the map, docked terrain-profile panel with antenna/frequency inputs and
a &los= deep link. Backed by GET /api/elevation/profile, an Open-Meteo
proxy that only accepts known node positions, with weighted budgets,
back-off and a per-IP limit.

Repeater planner (#/tools/planner)
- Chain: LOS on every leg between any number of points or nodes.
- Relay: best single relay for A→B among known repeaters and open
terrain.
- Area: suggested new repeater sites that best join the existing
repeater backbone in a box up to 100×100 km, picked greedily to merge
clusters, then remove single points of failure, then reach unlinked
repeaters.
- Terrain: SRTM 1° tiles from AWS Open Data, in memory only (~90 m,
bounded LRU); the server stays read-only. Exact-length gunzip, fetch
budget with back-off, missing terrain reported rather than read as sea
level, fuzzed pixel maths.
- GET /api/planner/{profile,relay,coverage}: quantised cache keys, 202 +
progress then cached 200, separate compute slots, per-IP and global
limits, failed jobs remembered, unpolled jobs cancelled. Nodes pass the
same blacklist / hidden-name / geo_filter rules as /api/nodes. The Go
LOS maths matches public/los.js on shared test vectors.

Also: the per-IP client limiter is extracted from elevation.go into
client_limit.go (behaviour-preserving) so both features share it.

Perf (real SRTM): coverage cold 1.5-5.6 s (tile download), warm 10 ms;
worst real compute 545 ms; peak RSS 91 MiB; live /api/nodes and
/api/packets p95 at most 1.6x while a job computes (GOMAXPROCS=2).

Tests: Go suites for elevation, terrain (fuzz), LOS vectors, graph
(brute-force property test), search and planner API; frontend unit
suites in test-all.sh; Playwright e2e for route LOS and the planner.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-11-16 16:15:36 +01:00
ZachandClaude Opus 5.5 138473d8c2 test(tile-resolver): load CSApi into the sandbox before roles.js (#2402)
## Problem

`tests/unit/test-tile-resolver.js` fails 0/26 on master with `CSApi is
not defined`, and master's CI on `e7fd48a0` is red because of it
([run](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43880608436)).

Two PRs that each passed CI on their own met on master:
- #2386 added this suite, which runs `public/roles.js` in a `vm`
sandbox.
- #2383 made `roles.js` call `CSApi.fetch('/api/config/client')` as it
loads.

The sandbox never had `CSApi`.

## Fix

The suite now loads `deployment.js` and `api-base.js` into the sandbox
first, through `tests/unit/api-base-sandbox.js`, as the other suites
that load `roles.js` already do. That's two lines in a test file, with
no app code changes.

## Verification

- `node tests/unit/test-tile-resolver.js`: 26 passed, 0 failed (it was
0/26).
- `sh test-all.sh` on this branch: exits 0.

## Related

#2391 needs this. It now includes #2383 from master, so its CI hits the
same failure. Once this lands, I'll merge master into #2391 again.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:59:55 +01:00
Paul PicazoandClaude Opus 5.5 e7fd48a054 feat(frontend): serve the UI from a web container or a static host, three deployment modes (#2383)
## What this does

The frontend half of serving the UI apart from the server. The server
half was #2362. It gives CoreScope three deployment modes. The server
container is the same image in all three, and it still serves
everything, the classic UI included.

| Mode | The UI is served by | API | Server CORS setting | Logins |
|---|---|---|---|---|
| All-in-one container (today) | the server container | same origin |
none | yes |
| Split containers (new) | a web container that forwards the API | same
origin | none | yes |
| Static host or CDN (new) | Cloudflare Pages, a CDN or any web server |
another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI |

### Commit 1: the UI as static files, for any host

- **One place for API URLs:** `public/deployment.js` says where the API
is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket
URL. A lint rule keeps raw `/api` fetches out of the other files. The
embedded UI keeps `apiBase: ''`, so it behaves as before, except that
its WebSocket now uses `/ws` instead of `/`.
- **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`,
which is `public/` byte for byte except for the cache buster in
`index.html` and `deployment.js`.
- It is `dist/web` rather than `dist/` because the Go binaries are built
into `dist/` and `make clean` removes that folder.
- `API_BASE=same-origin` is for a host that forwards `/api` and `/ws`
itself, as the web container does.
- **Anonymous build (`AUTH=false`):** for another origin. It has no
account features: login, account, admin and other writes link to the
same page on the classic UI. It sends no cookies, and its three JSON
POSTs go as `text/plain`, so there is no preflight.
- **Error notices:** a blocked API answer or an HTML answer each show a
notice.
- **Channel badge:** `beta`, `next` and `dev` builds show a badge that
opens a pre-filled issue, plus an "Open in classic" link.
- **Runbook:** `deploy/static/README.md` covers Cloudflare Pages,
previews, server CORS and rollback.

### Commit 2: the web container and the three modes

- **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It
packages the UI with `build.sh` when the image is built, serves it, and
forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`).
`SITE_ADDRESS` set to a domain name turns on automatic HTTPS.
- **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules
as `deploy/static/_headers`, and compresses responses with zstd or gzip.
- **`docker-compose.split.yml`:** the stock server image (its own Caddy
off) plus the web container:
`CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f
docker-compose.split.yml up -d --build`.
- `CORESCOPE_IMAGE` is required and must match the checkout's version,
so a release server is never paired with a newer UI by default.
- The server's built-in broker, which accepts anonymous clients, is
published on `127.0.0.1` unless `MQTT_BIND` says otherwise.
- **Docs:** `docs/deployment.md` has a new "Deployment modes" section,
and `DEPLOY.md` points to it.

### Keeping new code on `CSApi`

- **Lint rule:** `.eslintrc.json` now rejects every direct network call
in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`,
`sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in
`href`, `src` and `action` (assigned, set with `setAttribute` or written
into markup), `window.open`, `location.assign`, `location.replace` and
`location =`. `api-base.js` is the one file exempt.
- **HTML pages:** `eslint public/*.js` never sees the inline scripts and
markup of `public/*.html`, so `test-lint-network-calls.js` applies the
same rule to them, at their page lines. `area-map.html`'s two fetches
carry an `eslint-disable` with the reason: its Server field can name
another server.
- **Why it changed:** the earlier rule only caught a literal `/api…`
passed straight to `fetch`.
- **Check:** CI already fails on lint errors, and
`tests/e2e/test-lint-network-calls.js` pins exactly what the rule
reports: 25 bad statements reported, 15 good ones not, `api-base.js`
exempt.
- **AGENTS.md:** a new "API calls" section covers:
  - which helper to use for each need;
  - the lint rule;
  - how anonymous builds gate login-only features;
- loading `api-base.js` in vm and jsdom tests (the region quick-picks
suite missed that, and failed shard 1 on the first CI run).

### Review follow-ups

- **Account tokens:** the channel badge's pre-filled issue drops the
query on `#/account/…` routes, so mail-link tokens never reach a GitHub
URL. "Open in classic" keeps the full route, since the classic UI needs
the token.
- **HTML answers:** on a packaged UI (static host, web container),
`api()` rejects an HTML answer before status and retry handling. An HTML
404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server
returned an unexpected response" after one request. It used to give a
generic error, or six "warming up" retries for a 503. JSON errors and
the server's JSON 503 warm-up behave as before. The embedded UI keeps
its old handling (`CSApi.embedded`).
- **Logins:** `docs/deployment.md`, `Dockerfile.web` and
`docker-compose.split.yml` say `userManagement.publicBaseUrl` must be
the web container's public origin.

### Audit fixes

- **No broken logins build:** `build.sh` refuses `AUTH=true` with a
cross-origin `API_BASE`. The server allows no credentialed CORS, so
every API request of such a build was blocked; in Chromium all 12 were,
and the pages stayed empty.
- **No accidental deletes:** `build.sh` refuses an `OUT` that is neither
empty nor a previous build. It used to wipe whatever folder or file
`OUT` named.
- **Lint:** the rule now also reports `/api/` paths in `href`, `src` and
`action` attributes in single-quoted strings and template literals.
`tests/e2e/test-lint-network-calls.js` pins exactly what it reports.
- **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published
on localhost only.
- **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since
v3.16.0 and v3.15.0 predate the static build. It also names the server
version the static build needs: before #2362 the warm-up headers stay
hidden and wildcard previews are refused.

### Audit notes, fixed

- **Cache busting on every page:** the tool pages (`area-map.html`,
`geofilter-builder.html`) load their scripts with `?v=__BUST__`. The
server and `build.sh` replace it in every top-level page, and `_headers`
and the Caddyfile make every page revalidate. Behind Caddy, a returning
visitor could otherwise get a new tool page with an old `api-base.js`.
- **HTTP/3:** the split compose publishes 443 over UDP too, for the
HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with
Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`.
- **WebSocket on `/`:** a Go test pins that an upgrade on `/` still
works, for UIs cached from before `/ws`.

### Second review

- **Trusted proxies:** the docs said to trust the compose subnet, and
also to publish port 3000 if wanted. Together they let a client spoof
its address: the subnet holds the gateway, where a client reaching a
published 3000 through Docker's proxy arrives from. The compose file now
fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web
container's address (`172.30.83.200`), and the docs say to trust that
address alone.
- **Classic UI unchanged:** the HTML-answer notice above applies to
packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as
before, and an unknown `/api` path fails to parse as before, with no
notice.
- **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md
says what the rule cannot see: a URL assembled elsewhere and passed in a
variable.
- **AGENTS.md:** `build.sh` is described as packaging (a copy,
`deployment.js` and `__BUST__`, never a code transform), and rule 3 says
`__BUST__` is filled in every top-level page.
- **CI builds the web image:** the image-check job (no publish) builds
`Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`,
the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It
also validates the split compose file.
- **Cross-origin logins:** a hand-edited `deployment.js` with `auth:
true` and another origin now logs a warning naming the setting, instead
of failing with CORS errors alone.

## Upgrade notes

- **WebSocket path:** the UI's live feed connects to `/ws` instead of
`/`. The server has answered `/ws` since #1793 and still upgrades `/`,
so browsers holding the previous UI keep working.
- A reverse proxy that forwards WebSocket upgrades only for an exact `/`
must forward `/ws` too.
- The nginx example in `docs/deployment.md` (`location /`) already
covers it.
- **Static UI from another origin:**
- The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for
the warm-up banner and wildcard preview origins.
- `build.sh` refuses `AUTH=true` there, since the server allows no
credentialed CORS.
- **Split containers:**
  - `CORESCOPE_IMAGE` is required.
- With user accounts on, `userManagement.publicBaseUrl` must be the web
container's public origin.
- The built-in MQTT broker is published on `127.0.0.1` unless
`MQTT_BIND` says otherwise.
- The compose network is `172.30.83.0/24`, with the web container at
`.200`. If Docker reports that the pool overlaps another network, set
`SPLIT_NET`.

## Testing

- `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with
new tests for busting every page and the WebSocket upgrade on `/`.
- `sh test-all.sh` passes. New unit suites:
- `test-static-build.js`: the build and its input checks, `same-origin`
included, cross-origin logins refused, and an unrelated `OUT` left
intact.
- `test-web-container.js`: the Dockerfile, Caddyfile and compose file
agree on packaging, forwarded paths, cache rules and the server port;
the server image has no default and MQTT stays on localhost.
  - The `api-base`, anonymous mode, badge and `dev-static` suites.
- `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes
47/47, the four pages included; it needs eslint, so CI runs it on E2E
shard 1.
- **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes
11/11. That is the static build on one origin against an API-only server
on another, which CI runs on E2E shard 1.
- **Split containers mode:**
- The image builds, and `caddy validate` passes (Caddy 2.11.4). The new
CI step passes locally, and fails when the Caddyfile stops forwarding
`/api`.
- Trusted proxies, with the stack up: the web container gets
`172.30.83.200`. With only that address trusted, a spoofed
`X-Forwarded-For` sent straight to a published 3000 is ignored, and one
sent through Caddy is replaced with the client's address. With the
subnet trusted, as the docs used to say, the spoofed address was
believed: a deny entry for it rejected the upgrade.
- With `docker-compose.split.yml` on the E2E fixture, seeded the way CI
seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both
through the web container and against the server directly.
  - Headers, checked with curl:
    - `no-cache` on `/`, `/index.html` and `/deployment.js`;
    - zstd on static assets;
    - `/api/*` proxied (`Via: 1.1 Caddy`);
    - a `101` WebSocket upgrade on `/ws`.
  - In Chromium:
- every API request and the WebSocket stay on the web container's
origin;
    - there are no page errors and no horizontal overflow at 390 px;
    - no badge shows on the `static` channel.
- Logins through the web container: with user management on and
`userManagement.publicBaseUrl` set to the web container's origin, a
Chromium run passed every step, all on the web container's origin with
no page errors:
- registration, and activation from the mail link (which uses
`publicBaseUrl`);
    - the session surviving a reload;
    - a CSRF-protected `PATCH /api/account`;
    - logout, then login with the form.
- The same login with the backend's origin as `Origin` is refused with
403. The server checks `Origin` against `publicBaseUrl`, not `Host`,
which is why the docs ask for it.

## Performance

There are no hot-path changes. API calls go through `CSApi.url()`, a
string concatenation. In the web container, Caddy serves files from disk
and compresses them; API responses pass through with one proxy hop, the
same as the all-in-one image's built-in Caddy.

## Notes

- **Per-IP limits:** behind the web container, as behind the all-in-one
image's Caddy, requests reach the server from the proxy's address. The
per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`,
the web container's fixed address. The docs say so.
- **`depends_on`:** the web container starts with the server, not after
its healthcheck, like the all-in-one image's Caddy. Gating on health
would make every `up` wait for the first check (30 s) and fail outright
when a large database loads for longer than the healthcheck allows;
until the server listens, Caddy answers `/api` with 502.
- **Not in this PR:**
- publishing a `corescope-web` image (CI builds and smokes it, without
publishing);
- Cross-origin login—including a static frontend and API on sibling
custom domains—is outside this PR. In the deployments supported by this
PR, account support requires the frontend and API to share one browser
origin, either served together or through a reverse proxy.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:56:09 +01:00
n30nexandn30nex d80ce90976 fix(store): release evicted packet references from secondary indexes (#2327)
Addresses #2324 for upstream SQLite; PostgreSQL qualification remains
separate.

Eviction releases actual node memberships, pointer tails and cache roots
without SQL reconstruction. Borrowed distance snapshots and bounded
hash-migration ownership remain protected.

Review correction `0b1049b5` skips each distance-array copy when no
record is evicted from that array. When affected, it clones once and
filters the suffix beginning at the first match, preserving borrowed
snapshots and the surviving prefix. Work remains O(distance records); no
new locks, reader bookkeeping, settings or general speedup claim.

Tests-only `16509577` reaches [three real assertion failures in
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43200112374/job/131121073845).
GREEN leaves those tests unchanged. Cases cover
neither/hops-only/paths-only impact, repeated removals, live prefixes,
tail removal and snapshot contents. Parent independently reproduced RED
and reran GREEN.

Full native server/ingestor suites pass. Targeted checks: 167 passed;
race: 165 passed, with two existing timing skips that pass in the
ordinary run. Current master `a0b4e6cf` is normally merged; original
RED/GREEN history is preserved.

Historical November 10 measurements: synthetic retained heap 98.00→27.14
MiB with equal live data; path-update cost at 30k hops/10k paths
0.591→3.675ms and 2,824→5,368,584 B/op. The older ~30ms extra eviction
lock time at 240k transmissions is unremeasured. Large-window
qualification/optimization: #2393. Invalidated migration progress:
#2394; no SQL migration redesign here.

Browser verified: populated matching revision (500 transmissions/502
observations), 3/3 interactions. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43202356164)
and three independent reviews pass. No settings changes.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-15 12:24:43 +01:00
0e23c24847 fix(scope-stats): report total and split unscoped by route (#2364) (#2368)
Fixes #2364.

## Problem

Since #1838, `GetScopeStats` folds every non-transport transmission
(FLOOD 1, DIRECT 2) into `summary.unscoped`, while `transportTotal` and
`scoped` still cover transport routes (0, 3) only. The documented
`transportTotal = scoped + unscoped` no longer holds, and a client
computing an unscoped share against `transportTotal` gets >100% (the
reporter shows `transportTotal 7620`, `unscoped 14320`).

## Fix — additive, no existing field changes value

`summary` gains:

| field | meaning |
|---|---|
| `total` | all observed transmissions, routes 0–3 (= `scoped +
unscoped`) |
| `unscopedTransport` | route 0/3 with Code1 = 0000 |
| `unscopedFlood` | route 1 — unscoped by protocol |
| `unscopedDirect` | route 2 — unscoped by protocol |

Invariants (now asserted in `TestGetScopeStats`):

```
transportTotal = scoped + unscopedTransport
unscoped       = unscopedTransport + unscopedFlood + unscopedDirect
total          = scoped + unscoped
```

The non-transport count query keeps its single scan; it just splits
route 1 vs 2 with `CASE`.

On the design question in the issue: rather than changing what
`unscoped` means (the Scopes tab and #1842 rely on it), the split lets a
rollout view use `unscopedFlood` directly: repeaters apply region
rejection to flood routes only (`isRouteFlood()` in
`examples/simple_repeater/MyMesh.cpp`), so DIRECT traffic is not subject
to flood-region filtering. `docs/api-spec.md` now says so, spells out
the route mapping, and notes that `timeSeries` is transport-only (its
`unscoped` matches `unscopedTransport`).

## Frontend

Scopes tab uses `summary.total` when present and shows `N transport, N
flood, N direct` on the Unscoped card; falls back unchanged for older
servers. The cards are built by `scopeSummaryCardsHtml(summary)`
(exposed as `window._analyticsScopeSummaryCardsHtml`).

## Tested

- `go vet ./...` (cmd/server) clean, `gofmt -l` clean
- `go test -run ScopeStats ./...` — pass (incl. extended
`TestGetScopeStats` and `/api/scope-stats` route test)
- `node --check public/analytics.js`;
`tests/unit/test-issue-1375-scope-stats-fetch.js`,
`test-issue-1979-scope-adverts-by-role.js` pass
- new `tests/unit/test-issue-2364-scope-summary-cards.js` (vm sandbox,
11 assertions: shares of `total`, route split, fallback for servers
without `total`/split, partial split, empty window); full `test-all.sh`
green

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-11-15 12:24:27 +01:00
n30nexandn30nex ea87784257 refactor(live): reuse shared timestamp renderer (#2395)
Live timestamps duplicated the shared timestamp markup in `app.js`. Keep
`formatLiveTimestampHtml` as the input adapter and delegate its final
markup to `renderTimestampHtml`, retaining the existing missing-helper
fallback and Date/number/string normalization.

Partial follow-up to #1154 U10 and the [maintainer note on
#2283](https://github.com/Kpa-clawbot/CoreScope/pull/2283#pullrequestreview-5850899652).
This addresses only the Live wrapper; the umbrella remains open.

Validation:

- All 229 existing standalone frontend suites pass. Two shell-dependent
suites passed after selecting the installed shell/interpreter for the
local test process.
- 5,040 exact-base/candidate comparisons match byte-for-byte across
timestamp inputs, preferences, escaping, future warnings, and
missing-helper fallbacks.
- Actual Go fixture and Chromium: node-panel and replay-feed timestamps
checked on desktop and phone, with no page exceptions. The existing
phone portrait layout hides the feed; its replay DOM remains populated.
- ESLint: zero errors. No new settings, requests, dependencies, or
caches; constant work per timestamp, with no speedup claimed.

Pure-refactor TDD exemption: **tests/ files diff: no changes**. No test
files added or modified; existing assertions remain green. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43202375909)
and three independent reviews pass. Parent independently repeated the
Live unit, parity and post-push browser checks.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-15 12:24:20 +01:00
n30nexandn30nex c2e770ef65 docs: archive superseded repeater liveness plans (#2392)
Partial fix for #1385 (PR-E).

Move the two April repeater-liveness documents from `docs/superpowers/`
into existing `docs/archive/` with `git mv`. Preserve their dated
filenames, content and headings; add historical notices clarifying that
the proposed `relaying` status/signature is outdated and linking current
guidance.

Validation: exact commit blobs preserve 41,483 original bytes and 24
headings after removing only notices; four local links resolve; no
tracked old-path references; whitespace and PII checks pass.

TDD exemption: pure documentation, no behavior change. No runtime,
configuration, customizer, test, hierarchy or ignore-rule changes.
Application suites were not rerun. Both published Markdown pages and
their guidance links render correctly. [Docs-only
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43199869942)
and three independent reviews pass. The umbrella issue stays open
pending maintainer reconciliation.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-15 12:23:50 +01:00
TrixandOpenCode a0b4e6cf5f fix(nodes): light-mode map follows tile picks, key and theme changes (#1470/#1420) (#2386)
fix(nodes): light-mode map follows tile picks, key and theme changes
(#1470/#1420)

## Problem

The node-detail maps (`#/nodes/<pubkey>` full page + side pane — and
their
reusers `node-reach-map.js`, `rx-coverage.js`) pick a different basemap
in
**light mode** than the rest of the app:

- hardcode Carto Positron: the user's *Light Mode Provider* customizer
pick
  and the operator's `map.tiles.lightDefault` are ignored
- render Carto's `API KEY REQUIRED` watermark when `/api/config/client`
lands after the map is created — the tile requests stay keyless until a
  remount (they come back HTTP 200, so nothing surfaces as an error)
- never react to a theme flip or provider change while the page is open

Dark mode is unaffected, which is why it only shows up in light mode.

Reproduce on `master`: open `#/nodes/<id>` in light mode with
`mc-light-tile-provider = carto-voyager` — the inset requests
`light_all`
while the main map requests `rastertiles/voyager`.

## Root cause

`_applyTilesToNodeMap` (`public/nodes.js`) asked `window.getTileUrl()`
for a
single URL string, and roles.js's light branch returned the frozen
`TILE_LIGHT` scalar — the dark branch was the only one that consulted
the
runtime registry (`public/map-tile-providers.js`), which is what owns
localStorage picks, server defaults, the Carto key, and per-provider
attribution/maxZoom/refUrl/invertFilter. map.js and live.js keep the
only
other registry-aware resolvers of the same logic.

## What this PR does

- adds one resolver, `MC_resolveTile(isDark)`, in map-tile-providers.js,
  returning the full descriptor
  `{ type, id, url, attribution, maxZoom, refUrl, invertFilter }`:
  - `url` is always a resolved string — function-typed providers are
    invoked at call time (contract of #1614)
  - one precedence rule everywhere: explicit operator overrides
    (`cfg.tiles.dark|light`, `map.tiles.darkUrl|lightUrl`, tracked via
    `MC_LEGACY_TILE_OVERRIDE`) > user/server registry pick >
    `TILE_DARK`/`TILE_LIGHT` constants — **this is a deliberate behavior
    change on map.js/live.js, see below**
- fully lazy: registry rebuilds / pick changes are picked up by the next
    resolve, so config landing after page load fixes itself
- roles.js `getTileUrl()` becomes a thin wrapper over it; the public API
and
the #1614 sandbox contract are unchanged; `getActiveTileProvider()`
keeps
  its documented null-in-light contract
- `nodes.js _applyTilesToNodeMap` resolves through the descriptor and
stays
reactive: one shared `mc-tile-provider-changed` listener + one
`data-theme`
MutationObserver re-sync every registered map in place — old layers
removed
before the new ones are added, never stacked; maps whose container left
the
  DOM are pruned at sync time, so SPA navigation cannot leak listeners
- map.js `_resolveTileUrl` / live.js `_liveResolveTile` delegate to the
  shared resolver (their legacy bodies remain only as sandbox fallbacks)
- provider `maxZoom` replaces the hardcoded 18/19 caps (OpenTopoMap 17
and
  USGS 16 used to 404 above their max zoom)
- analytics.js subpath minimap gets provider `maxZoom` as well

## Behavior change (deliberate): operator URL overrides now win on every
surface

On `master`, the two tile-selection mechanisms conflicted and each
surface
resolved the conflict differently:

- `map.js` / `live.js`: user/server **registry pick won** — an
operator's
  `tiles.dark|light` / `map.tiles.darkUrl|lightUrl` was effectively dead
  config on those pages
- the legacy roles.js path (nodes page, reach map, rx-coverage,
analytics
  minimap): **operator override won**

This PR unifies on **operator-override-wins**, because that is the only
reading that gives live operator config a consistent meaning:

- these keys are still applied today. The deprecation warning says they
  would "be ignored in v3.5.0", and v3.5.0 shipped 2026-04-08 — the
instance line is now at v3.16.0 and the keys still take effect, so they
  are live configuration, not dead letters. An explicit operator pin
  should not be silently defeated by a per-user pick on some pages and
honored on others; whichever way the mechanism eventually goes, it must
  behave identically everywhere until it is actually removed.
- user choice is not reduced in the general case: picks remain
  authoritative for every provider the registry offers, and overrides
  only exist where an operator explicitly wrote a literal URL.

If maintainers would rather have pick-wins (operator URL acts only as a
fallback default), flipping the precedence in `MC_resolveTile` is a
small,
well-tested change — say so in review and it will be reworked.

Related cleanup, out of scope here: the stale "ignored in v3.5.0"
deprecation warning in `cmd/server/config.go` should either be honored
or
reworded now that the project is past v3.16.0.

## No change without operator overrides or picks

## Verification

Unit (all deterministic, `vm`-sandbox based):

- new `tests/unit/test-tile-resolver.js` — 22 cases: light/dark pick
honored,
operator `lightDefault`, disabled-provider fallback, override
precedence,
lazy function-URLs, key-after-async-config, in-place swap without
stacking,
  detached-map prune, multi-map single-listener sync, provider maxZoom;
  listed in `test-all.sh`
- existing suites green: `test-issue-1420-tile-providers` (47),
`test-issue-1470-node-tile-helper` (6),
`test-issue-1614-tile-url-function`
  (3), `test-frontend-helpers` (79), `test-settings-sync`,
  `test-packet-filter` (105), `test-aging`
- every one of the 227 suites listed in `test-all.sh`, run individually:
  all green except `test-manage-workflow` (2 failures that are local-env
only — CRLF bootstrap artifact — and pass in CI; `sh test-all.sh` aborts
  there locally under `set -e`, which is why the suites were run one by
  one this time; the scripted pipeline runs fully in CI)
- `eslint` on all six touched files: 0 errors

Browser (headless Chromium against this branch, `/api/config/client`
delayed
~800 ms to force the config race):

- light + `/#/nodes/<key>`: 12 keyless tile requests before config, 12
keyed
  re-requests after — no reload, no stacked layer
- light pick `carto-voyager`: inset requests `rastertiles/voyager` from
the
  start (parity with the main map)
- theme flip on an open node page: `dark_all` swaps in in place
- dark defaults unchanged (`dark_all`, keyed after config)

The Playwright E2E suite was not run locally — no Go toolchain on the
workstation to serve the backend — CI will cover it.

## Scope / follow-ups (intentionally not here)

- `home.js` mini-map (OSM only) and the geo-filter mini-maps in the
customizers are untouched on purpose: they are outside this bug's path,
  and migrating them to `MC_resolveTile` later is mechanical

## Performance

Not a hot path: one registry lookup + string build per map creation and
per
provider/theme-change event (each is rare); no per-packet or per-render
work;
the listener footprint is O(1) app-wide (single shared listener pair + a
pruned registry), so there is no growth across SPA navigations.

Co-authored-by: OpenCode <git-llm@trix.moe>
Co-authored-by: OpenCode:GLM-5.3-Flash <git-llm@trix.moe>
Assisted-by: OpenCode:GLM-5.3-Flash eslint

---------

Co-authored-by: OpenCode <git-llm@trix.moe>
2026-11-11 18:22:42 +01:00
efiten 6dd62711bc docs(release): v3.17.0 release notes and changelog entry (#2390)
Release notes for v3.17.0: `docs/release-notes/v3.17.0.md` (used as the
GitHub release body by `deploy.yml`) and the `CHANGELOG.md` entry.

24 commits since v3.16.0, counted with #2389, which is merged before
this PR. Issues closed come from each PR's `closingIssuesReferences`.

Docs only. After merge: tag `v3.17.0` on the merge commit once master CI
is green and the `:edge` image carries that commit.
v3.17.0
2026-11-11 00:26:33 +01:00
efiten 873e8b0e16 feat(rx-coverage): QR code and link to the operator's CoreDrive RX deployment (#2389)
## What

An operator who runs a CoreDrive RX deployment (the web and Android
companion app that feeds `clientRxCoverage`) can now point visitors of
the Coverage page at it.

- New optional config key `clientRxCoverage.appUrl`.
`/api/config/client` sends it as `clientRxAppUrl`, only while
`clientRxCoverage.enabled` is true and the value is an absolute `https`
URL with a host name. Any other value is left out and logged once at
startup with a `[config]` line.
- `#/rx-coverage` shows a panel under the subtitle when the field is
present: a QR code of exactly that URL (vendored `qrcode.js`, same call
as the node detail QR), the URL as a link, and one line on scanning it
in the app or opening it on a phone. Without the field there is no
panel. The page still needs no login.
- The subtitle named the app "CoreScope-RX" and linked to
`github.com/efiten/corescope-rx`, the app's old repository name. It now
says "CoreDrive RX" and links to `github.com/efiten/coredrive-rx`, or
carries no link when the panel already shows the deployment.

## Why

The CoreDrive RX Android app is not tied to one server: on first start
it asks for the address of a deployment. Scanning a QR code on the
Coverage page is the shortest way for a visitor to get there, including
people without an account.

Disclosure: CoreDrive RX (`efiten/coredrive-rx`) is my own project.

## Changes

- `cmd/server`: `config.go` (`validClientRxAppURL`, startup warning),
`types.go`, `routes.go`, `main.go`, `openapi.go`.
- `public/rx-coverage.js` (`appPanelHtml`, `appLinkHtml`),
`public/roles.js` (`window.MC_CLIENT_RX_APP_URL`),
`public/node-reach-coverage.css` (existing CSS variables only).
- Docs: `config.example.json`, `docs/client-rx-coverage.md`,
`docs/api-spec.md`.

## Tests

- 3 new Go tests in `cmd/server/client_rx_app_url_test.go`: URL
validation over 15 inputs, the endpoint field in 6 config states, the
startup warning. Full `go test ./cmd/server` passes.
- `tests/unit/test-rx-coverage-app-qr.js` (real `rx-coverage.js` and
`qrcode.js` in a vm): panel present and absent, QR content, HTML
escaping, subtitle link in both cases, old names gone. Added to
`test-all.sh`. All 6 `tests/unit/test-rx-coverage-*.js` pass.

## Not done

- No E2E test.
- Browser-checked on a local server with and without `appUrl` in the
light theme only. The QR keeps black modules on white in both themes,
unlike the node QR, which is inverted in dark mode; dark mode itself was
not checked in a browser.
- `cmd/ingestor` is unchanged; it does not need the URL.
- The same change runs on analyzer.on8ar.eu (a fork build) since
2026-11-10.
2026-11-11 00:25:32 +01:00
n30nexandn30nex 4140028165 fix(ui): keep slide-over close controls above navigation (#2304)
Fixes #2302.

Navigation covered tablet slide-over close controls and intercepted
pointer input. Two CSS substitutions put the panel/backdrop on the
existing modal layers. The regression uses actual close/backdrop clicks
and taps, retaining the existing keyboard, focus, scroll and resize
assertions.

Refreshed at `cd834802` by normally merging current master `d1bf4ca0`,
with zero conflicts. The exact three-file feature patch and original
RED/GREEN commits are preserved. Both feature tests are unchanged by
this refresh, including the supported 180-minute fixture window.

[Original RED
job](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212233915/job/121225789154):
40 passes, 17 pointer assertion failures, zero skips. The job completed
with failure before its aggregate run was cancelled. Merged #2289
resolves the old unrelated RF test failure.

Validation: four 57-case slide-over runs; three focus/scroll cases; 20
supplemental cases; 224 frontend suites; related navigation/gesture
checks; browser smoke with 149 passes/three existing fixture skips.
Parent rerun passes 57+3 after an initial-response timeout; tests were
unchanged. [Fresh
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42324795657)
passes, including the repeat gate.

Independent reviews found no introduced must-fixes; Kent Beck Gate:
PASS. Existing observer-focus and packet-navigation problems are
separately tracked in #2379 and #2380; runtime JavaScript is unchanged.

Browser verified: disposable Go fixture at `http://127.0.0.1:14964`.
E2E assertion added: `tests/e2e/test-slideover-1056-e2e.js`.

No new settings, dependencies, requests or hot-loop work.

## Preflight overrides

- check-red-commit: use the completed RED job’s 17 assertion failures;
its aggregate run was subsequently cancelled.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-11 00:07:56 +01:00
n30nexandn30nex 9e78e5c98d fix(live): show the newest certain node activity (#2283)
Fixes #2233.

Live node details use the newest certain activity from the node and
health responses for both status and Last Activity. Repeaters and rooms
with only recent shared-prefix relay evidence remain Uncertain.
Stale/missing values and a failed health request retain conservative
behavior.

Refreshed at `4d645dbc` by normally merging current master `d1bf4ca0`,
with zero conflicts. The three-file feature patch and original RED/GREEN
history are unchanged. [Original RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190088413)
remains; the preserved unit regression reaches 15 passes/eight assertion
failures locally.

Validation: all 224 frontend suites; 23 health-unit cases; eight health
browser groups including 12 Live cases; four RF airtime groups; and 48
Live cases across desktop/phone and light/dark. Parent reran the unit
and health browser suites. Backend trees are byte-identical to the
already-qualified master; no new native Go result is claimed.

The old unrelated RF test failure was reproduced during an actual theme
refresh. Upstream #2289’s locator waits correctly while retaining its
value/warning assertions; it is included in this refresh.

Browser verified: disposable Go fixture at `http://127.0.0.1:14963`.
Readable screenshots use the existing hidden-legend setting; the default
desktop legend overlap is unchanged.
E2E assertion added: `tests/e2e/test-health-status-e2e.js:259`.

Fixed-size work for one opened panel; no extra requests, dependencies or
settings. [Fresh
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42323634522)
passes. Independent correctness, UI and test-quality reviews found no
introduced must-fixes; Kent Beck Gate: PASS.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-11 00:07:42 +01:00
efitenandClaude Opus 5.5 fad303a4f4 docs: split the Docker minimum into running and building the image (#2361)
`docs/deployment.md` listed Docker 20.10+ as the minimum, without
separating running the published image from building it. The Dockerfile
needs BuildKit (`# syntax=docker/dockerfile:1` on line 1, `RUN --mount`
from line 71), and BuildKit is the default builder only from Docker
23.0. On 20.10 to 22.x the legacy builder rejects `RUN --mount` unless
`DOCKER_BUILDKIT=1` is set. Found while reviewing #2359.

Changes:
- `docs/deployment.md`: the requirements table gets two Docker rows,
20.10+ to run the published image, and 23.0+ (or 20.10+ with
`DOCKER_BUILDKIT=1`) to build it, plus one sentence on why.
- `docs/go-migration.md`: that guide builds the image, so its
prerequisite becomes 23.0+ (or 20.10+ with `DOCKER_BUILDKIT=1`).

Not verified: I did not run a build on Docker 20.10 or 22.x; the
legacy-builder behaviour comes from Docker's documentation, not from a
test here. Podman and buildah are not covered.

Docs only.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 00:00:52 +01:00
DSMoyleandClaude 9ddb29c946 fix/channels-mobile-header-overflow (#2382)
Fixes #2381. Drops the 56px header cap on mobile; title + Add on row 1,
region pills on a sideways-scrolling row 2. See the issue for details

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 00:00:39 +01:00
n30nexandn30nex afbae53b71 ui(nodes): add direct map and packet navigation (#2295)
Fixes #2290; addresses U6 of #1154.

Full node details, the side preview and tablet slide-over offer Open in
Map and View packets through existing deep links. A shared renderer
validates coordinates, including zero, and encodes identifiers without
extra fetches. Closing a preview preserves newly selected destinations
and titles; manual close returns to Nodes. Tablet targets stay separate,
and modified clicks retain native behavior.

Refreshed at `538cb169` by normally merging current master `d1bf4ca0`,
with zero conflicts. The original five commits and exact seven-file
feature patch are preserved. The [original RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203064817)
remains; the preserved RED also reproduces 39 assertions locally.
Upstream #2333 now supplies the correction for the previous unrelated
navigation-focus CI failure.

Current validation: all 225 frontend suites; 52 focused unit checks; 31
shortcut browser cases in each theme; 24 additional title/native-click
cases; navigation 60/60, branding 17/17 and slide-over 27/27. Parent
independently reran 52 unit and 31 browser cases. Native server and
ingestor suites pass. General browser smoke reports 149 passes and three
existing fixture skips. Unchanged serial retries resolved two documented
local test-server/navigation setup failures.

Browser verified: disposable local Go fixture at
`http://127.0.0.1:14954`.
E2E assertion added: `tests/e2e/test-node-quick-actions-e2e.js`.

Constant work for one detail view; no new requests, dependencies or
settings. [Final refreshed
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42318272198)
passes. Independent adversarial, UI and test-quality reviews found no
must-fixes; Kent Beck Gate: PASS.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-11 00:00:26 +01:00
n30nexandn30nex a59b42d8c2 fix(observers): keep charts and controls within narrow pages (#2344)
Fixes #2341.

Observer chart grids fit narrow screens, the doughnut stays square while
resizing, and empty canvases/long comparison choices no longer widen the
page. Radio formatting, requests and chart calculations are unchanged.

Refreshed at `f2ad231e` with a conflict-free normal merge of current
master `1577b57d`. Merged #2343 delayed-focus coverage and #2374 shared
payload labels are preserved. In response to review, the layout test now
checks one column at 760px and two at 900px, safely away from the exact
breakpoint. All original geometry, reload, disposal and request-count
assertions remain.

Original RED proof is retained: [four chart-width
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42011115036/job/127190202787)
and [four empty-data/control
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42016791855/job/127209299782).

Current validation: all 226 frontend suites and 68 ownership browser
cases pass. Parent independently reran four chart cases covering 32
layouts. Restoring current-base production through browser interception
makes all eight layout cases fail their overflow assertions; tests
remain sensitive to the bug.

Browser verified: disposable current-base Go fixture at
`http://127.0.0.1:14973`, desktop/mobile and both themes.
E2E assertion added: `tests/e2e/test-observer-detail-ownership-e2e.js`.

No new settings, dependencies, API calls or resize code. [Current
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42803853923)
and independent adversarial, visualization and Kent/TDD review pass at
`f2ad231e`.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-10 23:56:11 +01:00
n30nexandn30nex c03147cf0f fix(ui): share timestamp cells and retain future-clock warnings (#2387)
Partial fix for #1154 U10; follow-up to the maintainer's markup/warning
note on #2377.

Peer Interactions Last Contact now retains the same future-clock warning
as Nodes and Packets. One shared HTML renderer in app.js owns the
escaped timestamp, tooltip and existing warning markup; the three
surfaces delegate to it. Existing wrapper names/fallbacks, timestamp
preferences, placeholders and date algorithms are preserved. No new
settings, dependencies, requests or render triggers; work stays constant
per cell.

Tests-only RED `28e09e7c` reached [three unit assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42800749635/job/129777366878)
and [six browser missing-warning
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42800749635/job/129777366837).
GREEN `27f9fa49` leaves those tests unchanged and passes 115 unit
assertions, all 226 frontend suites and 10 browser cases. Parent
independently reran the unit and browser checks. Twenty-two
Nodes/Packets outputs match base exactly across 11 input/preference
cases.

Browser verified: disposable Go fixture at `http://127.0.0.1:14975`,
desktop/phone with UTC/local and relative/absolute preferences. Visible
warning/tooltip checks pass. The fixture's missing battery table and
existing final-row bottom-navigation overlap are unchanged limitations.
E2E assertion added:
`tests/e2e/test-issue-2218-node-analytics-stale-e2e.js`.

No production use of a test-only export, and no broader timestamp audit
claimed. [Current
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42804968476)
and independent adversarial, visualization and Kent/TDD review pass at
`27f9fa49`.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-10 23:55:59 +01:00
n30nexandn30nex 6c37fc59e1 chore: ignore root local worktree directories (#2388)
Partial fix for #1385 (PR-G).

Root `_wt-*` worktrees and `.worktrees/` folders currently appear as
untracked content. Add anchored, directory-only ignore rules for those
two local worktree conventions. Nested folders and similarly named files
remain visible.

Validation:
- Isolated Git repositories with empty global and local excludes: 21
before/after cases, 42 assertions passed.
- No currently tracked path matches either new rule; all other tracked
content and `.dockerignore` are unchanged.
- Existing planning-file and attachment visibility is preserved; `git
diff --check` passed.

TDD config-change exemption: no test files modified. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42809414204)
passes without test edits. No runtime, browser UI, or customizer
behavior changes. Parent reran the Git checks and post-push browser
smoke (3/3); served assets match the candidate. Three independent
reviews pass.

This covers only PR-G's transient worktree ignore audit. Other #1385
work, including PR-E documentation, remains outside this change; the
issue stays open.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-10 23:55:37 +01:00
efitenandClaude Opus 5.5 1577b57d67 docs(accounts): approved channels are backfilled since #2133 (#2385)
`docs/user-guide/accounts.md` said that messages received before a
channel proposal is approved stay encrypted. Since #2133 that is no
longer the case: when the key set grows, the ingestor re-decodes the
stored rows it could not decrypt (`cmd/ingestor/key_backfill.go`), and
the server tails `tx_rewrite_feed`, so no restart is needed.

Checked on an instance running #2133: approving four hashtag channels
triggered one pass (`[key-backfill] channel: 4 of 410 key(s) not yet
applied to history`, then `pass complete, 801 row(s) rewritten`, 13 s),
and the 801 rows are exactly the stored messages of those four channels,
the oldest from three months before the approval.

Docs only, one sentence. Not done: I did not check the other docs pages
for the same claim beyond a grep for "stay encrypted" and "received
before" in `docs/` and `README.md`, which found only this one.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 22:31:07 +01:00
Paul PicazoandClaude Opus 5.5 d66a219bcb feat(cors): let a UI on another origin connect, static host previews included (#2362)
## What this does

Server changes so the UI can be served from another origin, such as a
Cloudflare Pages project and its previews, against an existing backend.
It ships before the frontend, which is parked on
[`backup/static-frontend-1a`](https://github.com/ppicazo/CoreScope/tree/backup/static-frontend-1a)
and will come as its own PR.

- **Exposed headers:** allowed origins get
`Access-Control-Expose-Headers: Retry-After, X-CoreScope-Load-Status`.
The UI reads both, and cross-origin they are `null` without this.
- **Wildcard origins:** an entry may be `scheme://*.host[:port]`, such
as `https://*.myproject.pages.dev`. It matches exactly one leftmost
label (each deploy and branch preview) for both CORS and `/ws`, but not
`myproject.pages.dev` itself. The part after `*.` needs at least three
labels, so `*.pages.dev` is refused. Refused or malformed wildcards are
dropped at startup with a log line. That floor is a sanity check, not a
Public Suffix List lookup: `https://*.s3.amazonaws.com` passes and would
admit every S3 website endpoint, so a wildcard belongs only over a host
the operator controls.
- **Case:** exact entries now match case-insensitively for CORS (was
`==`), as they already did for `/ws`.
- **Test only:** pins that the three anonymous POSTs (`/api/decode`,
`/api/packets/observations`, `/api/paths/inspect`) accept a `text/plain`
JSON body, which keeps them free of a preflight.

Credentials stay off, so nothing a listed origin sends rides on cookies.
Beyond GET, a listed origin, and so every deployment under a wildcard
entry, can:

- call those three anonymous POSTs as simple requests. That predates
this PR; they compute an answer and write nothing.
`Access-Control-Allow-Methods: GET, HEAD, OPTIONS` only governs
preflighted requests.
- with `userManagement.enabled`, use the existing bearer tier for
companion linking (`/api/auth/device-token`, `/api/auth/me`,
`/api/auth/logout`, `/api/account/settings`, `/api/account/companions*`)
with POST, PUT and DELETE. Those routes need a device token in
`Authorization`, not a cookie, but a preview deployment gets them too.
On such a server, list a wildcard only if previews may use them.

## Configuring

```yaml
- CORS_ALLOWED_ORIGINS=https://existing-embed.example,https://myproject.pages.dev,https://*.myproject.pages.dev
```

The env var replaces `corsAllowedOrigins`, so keep the origins already
listed there.

## Testing

- `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`. The
new tests were written first and failed first.
- In Chromium, the static build from the backup branch, served on
`pr-1.proj.preview.localhost`, passed its split E2E (9/9, WebSocket
included) against this server.
- From the same setup, the project origin connected, a two-level
subdomain was blocked, and `X-CoreScope-Load-Status` read `ready` where
master gives `null`.

## Performance

These run per request in the CORS middleware and the `/ws` handshake,
and they allocate nothing. Same-origin requests skip them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 17:46:17 +01:00
n30nexandn30nex 899a3ad0b1 fix(map): normalize saved viewport longitudes (#2378)
Partial fix for #2375. Restoring a saved longitude such as -352.7
previously put markers one world-width off-screen while tiles looked
correct. Saved positions now use Leaflet’s native wrapping and the
existing viewport validator; movement saves canonical coordinates.
Invalid saved values fall back to configured defaults. Valid URL
viewports and node highlighting keep precedence.

This repairs saved state, not live dragging across world copies. No new
settings, dependencies, requests, marker loops or render triggers; added
work is constant per initialization/existing moveend.

The reported-node URL diagnosis is separate: no-slash returns JSON when
enabled and intentionally 404 when disabled; trailing-slash returns
HTML. See #2372. The endpoint/gate remain unchanged; #2375 stays open
for maintainer reconciliation.

RED `cb453c0d` produced [14 passes and 10 intended CI
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42317069830/job/128232241526).
Production fix `e6b70dfd` passed CI, 224 frontend suites, 24 browser
cases, six estimated-marker and three mobile checks. Marker placement
changes from x=262754/offscreen to x=610/visible in the same fixture.

Review follow-up `99a4746e` changes only test data: configured and saved
centers now differ, while invalid storage independently expects the
default. Parent rerun passes 24/24; a browser-only mutation discarding
saved coordinates now fails seven cases. All previous assertions and
production files are retained. The single review finding is resolved.

Browser verified: disposable Go fixture at `http://127.0.0.1:14956`,
real Leaflet rendering.
E2E assertion added: `tests/e2e/test-map-nodes-pagination-e2e.js:129`.

Final follow-up CI is blocked by the unchanged observer disposal
assertion (#2342; fix #2343). A maintainer must rerun the workflow; this
remains draft.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:40:40 +01:00
n30nexandn30nex 8420a7ac0f feat(map): identify historical foreign-advert flags (#2371)
Partial implementation of #730 M2; the issue remains open.

The map marks nodes whose stored `foreign` Boolean is true in dot and
hash-label modes. Titles, accessible names and popups explain that an
advert was flagged outside the configured area at ingestion. This
historical flag does not establish current location, bridging, an
observer or a recent count. Analytics/history remain separate.

The warning uses existing theme tokens and stays clear of nearby markers
and phone controls. Footprint preparation adds O(n) work and numeric
metadata to the existing marker list (around 2K nodes), with one
typography read per applicable render and no per-marker DOM measurement
or requests. Existing bounded-offset placement retains its worst-case
quadratic search; this is not a speedup claim. No new settings or render
triggers.

RED: `4ad11148` produced [44 missing-warning
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42012004213/job/127193151694).
Follow-up `c4a157de` produced [11 warning-occlusion
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42023035345/job/127230364384).

GREEN `ad465e32`: parent browser check 66/66, 224 frontend suites, 38
existing map/route checks and CSS audit. Incorrect-copy controls also
prove the tests reject a current-location inference.

Browser verified: local Go fixture, desktop/phone, both themes. E2E
assertion added: `tests/e2e/test-map-foreign-advert-e2e.js` covers
keyboard access, missing/false flags, no-GPS, refresh, clustering,
overlays, zoom and resize.

## Preflight overrides

- check-red-commit: initial RED includes the new browser regression plus
necessary CI/inventory registration, with no production changes; CI
reached the intended assertions.

[Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42026358900)
passes. Three independent reviews were consolidated; both findings are
fixed and parent-verified. Kent Beck Gate: PASS.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:13:33 +01:00
CoderNemesis 65990873ad feat(map): deep-link the route path picker by observer or All view (#2376)
## Summary

The map route deep link (`#/map?packet=<hash>[&obs=<id>]`) always opened
in the canonical-path view. Two optional query params now choose the
path picker's starting view, so a shared link shows exactly what the
sender meant.

| Param | Effect |
|---|---|
| `paths=all` | Opens the picker's **All** view (every unique path), the
same as clicking **All** |
| `observer=<pubkey>` | Full public key or a hex prefix of 8+
characters, case-insensitive. Picks that observer's **earliest**
observation (by timestamp, then id), makes it the canonical route and
isolates its path row, the same as clicking the row. "Back to packet"
then links to that observation. |

Examples:

```
#/map?packet=974ad4ecefbe3761&paths=all
#/map?packet=974ad4ecefbe3761&observer=a1b2c3d4
```

When params are combined, `obs=` wins over `observer=`, and `observer=`
wins over `paths=all`. An unknown or malformed observer falls back to
the current default view and logs a console warning.

## Changes

- `public/route-view-utils.js`: two new pure helpers.
- `resolveDeepLinkObservation(observations, params)` picks the
observation and the initial view.
- `pathGroupKey(rawHops, filterHops)` is the path picker's grouping key.
It was written out twice in `route-view.js` and now lives in one place,
which keeps the deep link and the picker rows in sync.
- `public/map.js`: `loadRouteFromDeepLink` uses the helper and passes
`initialPathView` through `drawPacketRouteMulti` to the renderer.
- `public/route-view.js`: the picker count and rows use `pathGroupKey`.
`sidebar._applyPathView` applies the view through the existing All and
row click handlers, in the frame after the route's initial fit.

## Performance

This doesn't touch a hot path. The work is one pass over a single
packet's observations (tens at most) plus a scan of the picker rows, and
it runs once per deep-link load. It adds no API calls.

## Config / customizer

No new configurable values.

## Tests

- New unit suite `tests/unit/test-map-deeplink-path-params.js` (29
checks). It covers prefix vs full key, case, short/non-hex keys,
earliest-observation choice with tie-break, the precedence rules,
`pathGroupKey`, and the source wiring.
- `tests/unit/test-issue-1633-hide-1byte-hops.js` replayed a copy of the
key logic. It now tests the real `pathGroupKey`.
- New Playwright suite `tests/e2e/test-map-deeplink-path-params-e2e.js`
(5 checks) on the seeded #1486 packet. It's registered in
`scripts/non-unit-tests.json` and CI shard 3. With the `route-view.js`
change removed, 3 of its 5 checks fail.
- Local runs against the CI-prepared fixture: `test-all.sh` passes.
Route-map a11y E2E 20/20, map mobile E2E 3/3, `test-e2e-playwright.js`
149/152 (3 skipped by the fixture, 0 failed).

## Browser validation

Checked in a browser against a local server, using a scratch copy of the
fixture that gives the seeded packet multi-hop paths through real GPS
nodes. One observer had two observations, and the API listed the later
one first.
- `observer=<prefix>` picked that observer's earliest observation and
drew only its path, with numbered markers and the map fitted to it.
- `paths=all` drew the same picture as clicking **All** by hand.
2026-11-08 17:13:22 +01:00
Marek WojtaszekandClaude Opus 5.5 3100d7a4c9 fix(ui): route remaining payload label surfaces through payload-labels.js (#2373) (#2374)
Fixes #2373

## Summary

Finishes the #1799 / #1804 migration: every frontend surface that names
a payload type now reads it from `public/payload-labels.js`. Node
Analytics, Observer detail and Observer comparison showed GRP_DATA,
MULTIPART and RAW_CUSTOM as `Type 6` / `Type 10` / `Type 15`. The map
route sidebar had RAW_CUSTOM on id 12; the firmware uses `0x0F`.

## Changes

- **`public/payload-labels.js`**: new
`PayloadLabels.api.displayName(id)` returns the canonical short label,
or `Type N` for ids the firmware does not define (e.g. a corrupted
header byte stays visible as `Type 13`). It lives on `.api` only; the
legacy root aliases are not extended.
- **`public/node-analytics.js`, `public/observer-detail.js`,
`public/compare.js`**: drop the local `PAYLOAD_LABELS` maps and use
`displayName()`.
- **`public/nodes.js`**: drop the local `PAYLOAD_TYPES` map; Recent
Adverts uses `shortById()` and keeps its `Packet` fallback.
- **`public/app.js`**: drop `PAYLOAD_TYPES`; `payloadTypeName()` uses
`shortById()` and keeps its `UNKNOWN` fallback.
- **`public/map.js`**: drop `PAYLOAD_TYPE_MAP`; `inferredType` uses
`enumNameById()`, which fixes RAW_CUSTOM (15 → `RAW_CUSTOM` instead of
`OTHER`).
- **`public/packets.js`**: `typeName()` is now `displayName` (same
output, one implementation).

Consumers resolve labels at render time rather than at file load, so the
many unit sandboxes that load these files without `payload-labels.js`
keep working. `index.html` already loads `payload-labels.js` before all
of them.

## Tests

- **New `tests/unit/test-payload-labels-surfaces.js`** (listed in
`test-all.sh`):
- drift guard: fails if any `public/*.js` other than `payload-labels.js`
pairs a numeric id with a short label or firmware enum name (it caught
the `map.js` copy, which was not on the original list);
- `displayName()` for every enum, string ids, and the `Type N` fallback;
- loads the real `node-analytics.js` in a vm sandbox and asserts the
doughnut labels for ids 6, 10, 15 and 13.
  - Before the fix: 11 failures. After: 82 passed.
- **`tests/e2e/test-issue-1799-label-vocab-e2e.js`**: two new steps
intercept `/api/nodes/:pk/analytics` and `/api/observers/:id/analytics`
(the fixture has no GRP_DATA in the store), inject ids 6, 10, 15 and 13,
and read the labels from `Chart.getChart()`. Against upstream `master`:
11 passed, 2 failed (`["Type 6","Type 10","Type 15","Type 13"]`). With
this branch: 13 passed.
- **Sandbox updates**: `test-compare-overlap.js` and
`test-observer-detail-ownership.js` now load `payload-labels.js` before
the page script, as `index.html` does.
`test-issue-1418-raw-hex-extraction.js` asserted the hand-rolled
`PAYLOAD_TYPE_MAP` literal; it now asserts that `inferredType` uses the
canonical map and that no local map exists.
- Unit suites, each `test-all.sh` entry run on its own: 223 pass on this
branch vs 222 on `master` (+1 new suite). `test-manage-workflow.js` and
`test-preflight-xss-gate.js` fail identically on `master` in my Windows
environment (CRLF in the shell fixture, no `python`); they are unrelated
to this change.
- E2E, run against a local server on the CI fixture (freshened, seeded
and migrated as in `deploy.yml`): `test-issue-1799-label-vocab-e2e.js`,
`test-charts-fluid-1058-e2e.js`, `test-issue-1147-section-order-e2e.js`,
`test-issue-1640-compare-discovery-e2e.js`,
`test-issue-1648-m2/m3-icons-e2e.js` and
`test-observer-detail-ownership-e2e.js` all pass.
`test-e2e-playwright.js` fails one step (`Version info lives on Perf
dashboard`, `#navStats` wait timeout) with the same result on `master`
locally.

## Performance

No hot-path change. Every lookup is still one object-key read per
packet: the packets-table sort accessor calls `displayName(t)` instead
of `TYPE_NAMES[t] ?? ...`. No new loops, requests or allocations.

## Browser validation

Checked against the local fixture server, with the analytics responses
patched in the page to contain GRP_DATA:

- Node Analytics → Packet Types legend: `Group Data, Advert, Channel
Msg, Direct Msg, Multipart, Raw Custom, Type 13`
- Observer detail → doughnut and the first Recent Packets row read
`Group Data`
- Observer comparison → packet table rows read `Group Data`
- Map route deep link (`#/map?packet=…`) renders the route panel; no new
console errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 17:13:10 +01:00
n30nexandn30nex 6618820836 fix(nodes): show timestamp context for Last Contact (#2377)
Partial fix for #1154 U10. Peer Interactions Last Contact now shows
timestamp context through the existing shared formatter: relative text
with an absolute tooltip, or the reverse when absolute display is
selected. Existing timezone, precision and custom-format preferences
apply; missing/invalid values remain a dash. Other U10 surfaces and
tracking items stay open.

The production change only formats the existing cell. Work stays O(P)
over the existing maximum 20 peers; no requests, settings, dependencies,
caches or render triggers are added. Tooltip and text are HTML-escaped.

RED `c47cce20` reached [11 unit assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42316165311/job/128229216777)
and [six browser missing-tooltip
assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42316165311/job/128229216764).
GREEN `2c29eccb` keeps those tests unchanged and passes 82 unit
assertions, 10 focused browser cases (parent rerun), and all 224
frontend suites. Existing browser smoke reports 149 passes and three
fixture skips. One separate navigation timeout passed on an unchanged
serial retry.

Browser verified: disposable Go fixture at `http://127.0.0.1:14953`,
desktop/phone; tests retain the real API response and supply
deterministic peer contacts. Its unrelated missing battery-metrics table
is a fixture limitation.
E2E assertion added:
`tests/e2e/test-issue-2218-node-analytics-stale-e2e.js:123`.

[Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42317025356)
passes. Independent adversarial, UI and test-quality reviews found no
must-fixes; Kent Beck Gate: PASS.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:13:00 +01:00
n30nexandn30nex 0e2ed89829 fix(packets): explain observer-report coverage (#2370)
Red commit: `e86294b6` ([four intended CI assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42009863423/job/127185896565)).

Partial fix for #667, Part B. The Packets page now explains that its
counts come from observer reports received by this instance—a partial
view, rather than network-wide totals or device firmware counters. The
note remains visible on phones and describes the table for assistive
technology.

This does not change TX/RX storage, counts, filtering or grouping, and
leaves the rest of #667 open. It invents no contributing-observer number
and makes no RX-only claim. No new settings, dependencies, requests or
processing work.

Validation on `08036329`: parent-run browser regression 10/10; required
frontend unit suites 932 assertions. The RED run completes six existing
cases and fails the four new missing-context assertions.

Browser verified: disposable local fixture at `http://127.0.0.1:14943`
(desktop, phone, populated/empty filters, grouping and refresh). E2E
assertion added: `tests/e2e/test-issue-1122-packets-filter-ux-e2e.js`.

Updated CI passes; independent adversarial, UI and test-quality reviews
found no must-fix issues.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:12:49 +01:00
199df07c21 docs(proto): correct route_type values to match the decoder (#2363) (#2367)
Fixes #2363.

The published contract described `route_type` as `0=DIRECT, 1=FLOOD,
2=reserved, 3=TRANSPORT`. Both decoders (`cmd/ingestor/decoder.go`,
`cmd/server/decoder.go` — `routeTypeNames`) and MeshCore's
`docs/packet_format.md` use:

```
0 = TRANSPORT_FLOOD, 1 = FLOOD, 2 = DIRECT, 3 = TRANSPORT_DIRECT
```

Updated:
- `proto/decoded.proto` — `DecodedHeader.route_type` and the
`route_type_name` value list
- `proto/packet.proto` — `Transmission.route_type` (same stale comment)
- `docs/api-spec.md` — Packet Object `route_type` line (same stale
comment)

Comment/doc-only; no generated code in-tree, no behaviour change.

Co-Authored-By: Claude <noreply@anthropic.com>

Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-11-08 17:12:37 +01:00
n30nexandn30nex c4dc46f349 test(e2e): wait for comparison focus before disposal assertions (#2343)
Red commit: `eedadc43` ([four intended CI assertion
failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40586787653/job/122480320591)).

Fixes #2342. The disposal test could capture comparison results before
the router's accessibility-focus frame added the heading tabindex, then
mistake that legitimate setup change for a late observer response.

Wait for the actual tabindex in attached state before capturing the
baseline, including when the mobile heading is hidden. A controlled
late-focus variant proves the regression. Full HTML equality, chart
cleanup, route and control assertions remain intact; no production
behavior or settings change.

Current head `cb537ebe` includes current master while preserving the
original RED/GREEN commits and newer progressive-render coverage. Local
validation: ownership browser suite 60/60 (independently rerun by
parent), navigation 60/60, branding 17/17 and all 224 standalone
frontend suites.

Browser verified: real local Go fixture server at
`http://127.0.0.1:14944`. E2E assertion added:
`tests/e2e/test-observer-detail-ownership-e2e.js`.

Updated CI passes; independent adversarial, UI and test-quality reviews
found no must-fix issues.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-11-08 17:12:03 +01:00
JadeandClaude d1bf4ca042 fix(docker): build natively on arm64 hosts (#2359)
Building the image natively on an arm64 machine (a Raspberry Pi 4 or 5
on 64-bit OS, Apple silicon) fails at the first `RUN` of the build
stage:

```
=> ERROR [builder  2/40] RUN apk add --no-cache curl xz && ...
0.841 exec /bin/sh: exec format error
```

**Cause:** the Dockerfile declares `ARG BUILDPLATFORM=linux/amd64`. A
default on that ARG overrides the value BuildKit sets for the machine
doing the build, so an arm64 host pulls the amd64 `golang` image and
can't run it. CI didn't catch it because its arm64 image is cross-built
on an amd64 runner, where `linux/amd64` was correct anyway.

**Fix:** declare `ARG BUILDPLATFORM` without a default. BuildKit, the
default builder since Docker 23 and what buildx uses, fills it in on
every host. amd64 builds and the CI cross-build are unchanged.

**Workaround on current master:** `docker build --build-arg
BUILDPLATFORM=linux/arm64 .`

**Test:** `tests/unit/test-dockerfile-buildplatform.js` fails on the old
Dockerfile and passes on this one. It checks three things:
- `BUILDPLATFORM` has no default.
- The Go builder stays pinned to `$BUILDPLATFORM`.
- No stage hard-codes a platform.

It's wired into `test-all.sh`.

Found while building the image on a Raspberry Pi running Raspberry Pi OS
Trixie (arm64) with Docker 29.


---

Co-authored-by: Claude <noreply@anthropic.com>
2026-10-31 12:13:29 +01:00
n30nexandn30nex 9341ec744b fix(observers): honour node visibility in neighbour reports (#2357)
Fixes #2356.

Observer neighbour reports now apply the existing node blacklist and
hidden-name-prefix rules before returning displayed identities. Names
are already batch-loaded, so the fix filters the request-owned slice
without another SQL query. It preserves reported totals, timestamps,
stored data, public/unknown neighbours and the existing reporter-only
observer-blacklist semantics. An all-hidden list remains `[]`.

The work is bounded by the existing 500-entry ingestion cap plus the
existing visibility checks; no new configuration, frontend change or
database write is introduced. The filter reuses the existing report
slice.

Validation:

- [RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40971452075/job/123789013428)
at `7341b334` compiled and reached the four expected visibility
assertions.
- GREEN `b38e95e6`: focused native checks passed in 6.607 s; parent
rerun passed in 1.865 s.
- Full native server passed in 189.743 s; parent ingestor suite passed
in 399.859 s. Existing neighbour frontend checks passed 7/7.
- Tests use the actual registered router, direct-node 404 controls,
casing/shared-name/unknown-node cases, metadata preservation and an
unchanged stored report.

Local standalone browser replay is unavailable because the
fixture-server launch was blocked. Fresh GREEN CI passed, including
server race checks and browser shards. Independent adversarial,
security-boundary and TDD reviews found no must-fix items. No
live-instance incident or deployment is claimed.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-31 12:01:19 +01:00
n30nexandn30nex d083f9ade9 test: cover partial observer chart construction failures (#2354)
Partial follow-up for #1778; addresses the chart-cleanup test gap
identified in the review of #2339.

The observer detail unit harness now throws during the second or fourth
chart constructor. Both cases require existing chart instances to be
destroyed, metadata/calibration to remain visible, the error to be
rendered as text, and a later time-window retry to recover without extra
requests.

Production cleanup already behaves correctly; this PR adds coverage
only. The parent-run ownership suite passes 30/30. Removing the actual
cleanup produces exactly the two intended assertion failures. The
remaining unit sources and runner match the current-master baseline
whose 222 frontend suites passed; only this updated unit file differs
within that test set.

TDD scope: no production change or new behavior. The negative control
removes existing cleanup to prove the new assertions detect its loss. No
configuration, dependencies or runtime performance changes. Exact-head
CI passed, including the browser shards. Independent adversarial,
correctness and TDD reviews found no must-fix items.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-31 12:00:06 +01:00
n30nexandn30nex b3a9f2a9e9 docs: make scheduled backups portable and bounded (#2353)
Fixes #2352. Follow-up to #2345.

The older deployment guide recommended copying a live telemetry
database. It now points to the consistent-snapshot procedure. The
scheduled-download example uses a trailing-X temporary filename, bounds
connection and total download time, and explains the weak-key denylist.

Validation: the parent reran the extracted command against eight
disposable HTTP cases: valid SQLite, unauthorized, server error, empty
content, HTML, corrupt SQLite, truncated transfer and timeout. Failed
downloads removed partial files and preserved existing completed
backups; the timeout exited 28. Shell syntax and diff checks passed. GNU
tools were exercised; BSD/macOS was not run natively.

TDD exemption: pure documentation, no application or repository test
files changed. No runtime/browser behaviour, dependencies or
configuration changed. Independent reviews are complete; the one
timeout-wording finding was addressed in 33d629b5 without changing the
validated command. Exact-head CI passed, including the browser shards.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-31 11:59:39 +01:00
n30nexandn30nex f8c3b32633 docs(api): describe readiness responses and ingestor liveness (#2358)
Partial follow-up for #1670.

Documents public `GET /api/healthz`: 503/loading, current in-memory
counts, the fixed legacy backfill compatibility object, and optional
ingestor liveness timestamps. Receipt and handler-entry clocks have
different reset semantics; handler entry does not prove a completed
database write. The legacy object does not track real migration
progress.

The 200 response has a typed schema, including its dynamic source-name
map. Only `/api/healthz` leaves the known-gap list (16 → 15). Handlers,
configuration and frontend behavior are unchanged; formal non-200
response-schema work remains separate.

Validation:

- The new documentation assertion failed before metadata was added;
actual handler-response checks passed.
- Full native server/ingestor suites passed in 257.438 s / 452.613 s.
- After review corrections, focused checks passed in 2.757 s and the
parent rerun in 2.097 s, including exact compatibility values and
retained-receipt/zero-handler timestamps.
- Independent adversarial, correctness and TDD reviews completed; the
parent verified both semantic corrections against actual call sites.
- [Exact-head CI
passed](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40979351261),
including browser shards.

TDD exemption: documentation/API metadata only. Local standalone browser
smoke was blocked; no local-browser or deployment claim is made.
Production untyped-map count is unchanged.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-31 11:59:20 +01:00
efitenandClaude Opus 5.5 faf6f792ae docs(release): v3.16.0 release notes and changelog entry (#2355)
Release notes for v3.16.0 (`docs/release-notes/v3.16.0.md`) and the
matching `CHANGELOG.md` entry. The tag pipeline uses this file as the
GitHub release body.

Range `v3.15.0..af2e96d9`, 47 commits: 23 fix, 9 feat, 6 docs, 2 test, 2
perf, 2 ci, 1 ui, 1 deps, 1 chore.

"Read this before upgrading" covers three changes:
- relay hops refresh Last Heard only with a unique prefix and only on
flood packets (#2334, #2308, #2298, #2336);
- `users.db` schema v6 to v8 with user accounts on (#2306, #2348);
- Go 1.26 for source builds and Alpine 3.24 in the image (#2297).

Not verified: the mosquitto, Caddy and supervisor versions are taken
from #2297's local build, not from a pull of the published image.
Performance numbers are copied from the PRs, not re-measured.

Docs only, no code change.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
v3.16.0
2026-10-29 23:11:30 +01:00
af2e96d92a fix(reach): queue a cold request up to 3 s for a build slot before answering 429 (#2249)
Follow-up to #2127, from @dborup's review there. The limiter, most of
its tests and the docs text are dborup's (dborup/CoreScope@195ec4c3),
adapted to master; the commit carries `Co-authored-by: dborup`.

## What changes

`GET /api/nodes/{pubkey}/reach` still runs at most 2 uncached reports at
once. What changes is the next uncached request:

- **Before:** 429 at once, `Retry-After: 5`.
- **After:** it waits up to 3 s for a slot. It gets 429 with
`Retry-After: 2` and a JSON body only if no slot frees in that time.

Unchanged: cached answers never wait, and concurrent requests for the
same node and `days` share one computation and one slot.

One addition beyond dborup's version: the shared computation runs on the
context of the request that started it. If that request disconnects,
while queued or mid-scan, the other requests sharing it that are still
connected now get 429 instead of 500. The 3 s wait made that case more
likely.

## Why

dborup's three points on #2127:

1. An instant 429 fails two tabs, or a quick switch between 7/14/30
days, when the database is not under real pressure.
2. The page cannot recover from it, because `api()` retries only 503.
The panel retry is a separate PR so that this one stays server-only.
3. No test covered the promised property: a cached request still answers
while every slot is held.

Waiting parks a goroutine, not a database connection, so the pool is
protected exactly as before.

## What I checked

- **Tests.** `make test` (all 16 modules), `make vet`, `gofmt`, `sh
test-all.sh` and eslint (0 errors) all pass. The reach tests also ran 10
times under `-race`.
- **New tests.** 14 tests in `node_reach_cold_limit_test.go`: dborup's
10 plus 4 new. `TestNodeReachConcurrentBuildCap` now drives the new
limiter through the production router. Point 3 is
`TestNodeReach_WarmHitSkipsColdLimit`. The 4 new tests are:
  - a queued request is served when a slot frees;
  - a cache filled just before the flight skips the limiter;
- coalesced waiters get 429 when the starter disconnects, queued or
mid-scan (two tests).
- **Mutations.** I wrote 21 from the code before running the tests, and
the tests caught all 21: instant reject, cap +1, release that leaks,
slot taken before the in-flight recheck, busy → 500, Retry-After header
or value, status 503, dropped context mapping, and others.
- **Real binary.** Master `2d09335c` and this branch, run one after the
other on a copy of a 10.5M-observation, 11 GB database. Each burst fires
its reach requests at once, while `/api/nodes?limit=50` and
`/api/packets?limit=50` are polled every 200 ms.

| Burst (distinct uncached keys) | master | this branch |
|---|---|---|
| 3 keys, `days=7`, warm page cache (2 runs) | 2 × 200, 1 × 429 after
0.01 s | 3 × 200; the third answered at 4.1–4.3 s |
| 6 keys, three of them `days=14` (2 runs) | 2 × 200, 4 × 429 after ≤
0.01 s | 2 × 200, 4 × 429 after 3.0 s |
| 24 keys, cold page cache | 2 × 200, 22 × 429 | 2 × 200, 22 × 429 after
3.00 s |
| `/api/nodes` during every burst | median 1 ms, max 53 ms | median 1
ms, max 66 ms |

## Not checked

- **Slow scans.** On a database this size, uncached scans took from
about 9 s to over 60 s, so the 3 s wait only helps when page cache is
warm or the database is smaller. Otherwise the result is the same 429, 3
s later. A 30-day uncached scan hit the 60 s `WriteTimeout` on both
builds; that is the scan #2214's opt-in hop index addresses.
- **Pool contention under a live store.** On this copy the in-memory
store loaded 0 packets, so `/api/nodes` latency does not show contention
under a live load.
- **Platforms.** One machine (linux/amd64). No browser check: the server
change has no UI, and the panel retry comes with its own PR.

**Base.** Rebased on master `7c00a429`, which includes #2214; every test
and mutation above was re-run on the rebased commit. The real-binary
runs used `2d09335c`, just before #2214, on a database without the hop
index. That is the default after #2214 too (`hopIndexDays` unset means
off). With the index on, scans are shorter, so more bursts fit in the 3
s wait; I did not measure that.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


---

## Update after review (commits `c65f4d3b`, `1d2cd99d`, `04e49854`)

efiten asked how many requests may wait at once. With only the 3 s
timeout, the number of waiting distinct keys grew with the request rate.
n30nex wrote a cap (nullrouten0/Baymc-CoreScope#1, tracked in #2264).
Its two commits are now on this branch unchanged, as a fast-forward.

- At most 8 distinct cold requests wait for a slot. Any further one gets
429 with `Retry-After: 2` at once, before a timer is set.
- A waiter gives its place back on every exit: it gets a slot,
disconnects or times out.
- Cached answers and same-key sharing are unchanged. Requests for a key
that is already queued or building wait on that one place and are not
counted. `04e49854` corrects the code comment that said the ceiling was
independent of request rate.

**What I checked on `04e49854`:**
- The two new tests fail on `28da016f` and pass now.
- 12 mutations of the cap, all caught: cap 7 and cap 9; the place never
returned, returned before waiting, returned only on promotion, leaked on
cancel, leaked on timeout; a full queue that blocks, returns the wrong
error or skips the counter; a timer armed before the check; a queue
sized by the slot count.
- `make test`, `make vet`, `make fmt-check`, `sh test-all.sh` and eslint
8.57.1 (0 errors) pass, and the reach tests pass 5 times under `-race`.
On a local merge with current master `660cbbf5`, which includes #2277,
the reach tests pass 3 times under `-race` and the `cmd/server` package
passes.
- Real binary: both heads, alternately, on a copy of a 2.5 GB soak
database, with 24 distinct uncached `days=7` requests fired at once, two
runs each:

| | `28da016f` | `04e49854` |
|---|---|---|
| 200 | 2, 2 | 2, 3 |
| 429 at once | 0, 0 | 14, 14 |
| 429 after 3 s | 22, 22 | 8, 7 |

**Not checked:** a sustained flood, a burst of same-key requests against
the cap, `/api/nodes` latency during these bursts, and platforms other
than linux/amd64. The real-binary table further up was measured before
the cap.

---------

Co-authored-by: nullrouten0 <201262769+nullrouten0@users.noreply.github.com>
Co-authored-by: dborup <kontakt@meshview.dk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: n30nex <blushellneon@outlook.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
2026-10-29 22:50:59 +01:00
efitenandClaude Opus 5.5 bb3c8cc001 fix(rx-coverage): My coverage and the observer filter on the noise layer (#2326)
"My coverage" and a selected observer only narrowed the signal layer on
the coverage page. On the noise layer the toggle stayed pressed, but
`/api/rf-noise` was called without `mine` or `rx` and returned every
companion's samples.

## Changes

- **Backend:** `/api/rf-noise` now accepts `rx` and `mine=1`, with the
same session rules as `/api/rx-coverage`: 401 without a session, 403 for
a device token, 404 with user management off. The `rx_pubkey` filter
moved into 1 helper, `appendRxFilter` (`cmd/server/rx_dashboard.go`),
which both queries use. Both parameters are in the OpenAPI entry.
- **Frontend:** `public/rx-coverage.js` fetches both layers through
`fetchLayer`. It builds the URL with the time window, observer and
toggle, and judges the reply with the existing `coverageReply`. The
request sequence (`signalSeq`, now `drawSeq`) covers both layers, so the
noise layer also drops a superseded reply. Before, a slow unfiltered
noise reply could overwrite a newer filtered one.

## Performance

The noise query gets 1 extra `WHERE` clause on `rx_pubkey`. There are no
new requests: the page still makes 1 request per redraw.

## Verification

- New Go tests in `cmd/server/rf_noise_mine_test.go`: `rx` narrows the
result (case-insensitive), `mine=1` returns 401 without a session,
returns empty with nothing linked, returns the linked companion's cells,
returns 403 for a device token, and returns 404 with user management
off. Both failed before the fix.
- `tests/unit/test-rx-coverage-mine.js` is updated for the new
`coverageUrl(path, ...)` signature and checks that both layers fetch
through `fetchLayer`.
- New step in `tests/e2e/test-rx-coverage-noise-e2e.js`: a
`?layer=noise&rx=...` link sends `rx`, and clicking "My coverage"
refetches with `mine=1`. It fails against the current master frontend
and passes with this change.
- Local runs (Windows): `go test` and `go vet` in `cmd/server`,
`test-all.sh`, and the noise, gaps and viewport E2E in Chrome. All
green. I did not run the full Playwright suite locally.

## Not in this PR

- The "nothing received" gap cells still skip `mine=1`, unchanged: the
track query cannot narrow to a set of companions yet.
- The time window (`days`) already worked on the noise layer, so it is
not touched.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:39:02 +01:00
TeTeHackoandClaude Opus 5.5 417c8dd00a feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) (#2300)
Closes #2270, closes #2271.

The ingestor took only the confirmed region scopes out of an observer's
`/neighbors` report (#1865) and dropped the rest. Nothing could
therefore say when an observer last reported, or whom it heard.

## Ingestor

`observer_neighbor_reports` (created in `dbschema.Apply`) keeps the
latest report of each observer, one row per observer, replaced by every
live report.

The report is unauthenticated, so `neighbors[]` is stored sanitised:

- only 64-hex keys
- `snr` and `heard_secs_ago` finite and in range
- `status` limited to `[a-z_]`
- `scopes` clamped
- at most 500 rows

`total_neighbors` is believed only when it is not below the number of
rows that arrived.

A **retained replay** only fills a missing row, and it is dated by the
report's own `timestamp`. After an ingestor restart, a long-dead
observer therefore does not look as if it just reported.

## API

The table is optional for the server (no `AssertReady` change); without
it the fields are simply absent.

- `GET /api/observers`: `neighbors_report_at` and `neighbors_count`,
from one batch query in the existing cached builder.
- `GET /api/observers/{id}`: also `neighbor_report`, which is
`{received_at, reported_at, neighbor_count, queried_neighbors,
self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr,
heard_secs_ago, status, scopes}]}`. Names and roles come from the node
table in one query.

## Frontend

`public/observer-neighbors.js` is shared by both pages and escapes
everything.

- **#2270, observers list:** a sortable **Neighbors** column, e.g. `3h
ago · 12`, or `—` when the observer never reported. The same value
appears in the slide-over.
- **#2270, observer page:** **Neighbors Report** and **Neighbors** stat
cards.
- **#2271, observer page:** a **Neighbors** table listing each neighbor
by name (a link to the node) or key prefix, with SNR, heard, scope query
status and scopes. The heading says when the list is shorter than the
count, e.g. `(2 of 3 listed)`.

## Tests

- **ingestor:** sanitising, the cap, live replace, the retained rules
(fill only, dated by the report, undated ignored) and the MQTT topic end
to end. The existing #1865 scope tests are unchanged and still pass.
- **server:** the list summary, the detail with names, and both
endpoints without the table.
- **frontend:**
  - unit tests of the helpers
- an XSS case for every rendered field; I checked that it fails when
escaping is removed
- `test-observers-headings` now expects the new column after `Last
Packet`
- **E2E:** `test-observer-neighbors-2270-e2e.js` intercepts the observer
endpoints, since the fixture has no reports. It is listed in
`scripts/non-unit-tests.json` and on shard 2.

Gates that pass:

- `go test -race` in `cmd/server`, `cmd/ingestor` and
`internal/dbschema`
- `sh test-all.sh`
- `eslint@8`, with 0 errors
- the XSS diff gate
- these E2E suites: the new one,
`test-issue-1639-observers-sort-e2e.js`,
`test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js`

Performance: the list adds one primary-key-sized query per cache fill,
and the detail adds two small queries per request.

Also fixes #2330 (folded in from #2337, second commit).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:36:53 +01:00
n30nexandn30nex 202bafea28 fix(observers): show metadata while analytics loads (#2339)
Partial follow-up to #1778, limited to observer details; other pages
remain separate.

Observer identity, status and radio information now render when the
basic request finishes. Analytics and clock calibration load into their
own regions, so either can finish first. An analytics failure preserves
usable metadata. Identity failures and obsolete responses cannot
overwrite the current view, and failed chart construction is cleaned up.

The same three requests start together. Generation ownership, request
deduplication, date-window behavior and disposal remain covered. No new
fetches, settings, dependencies or customizer options; secondary replies
preserve the actual metadata DOM node.

[RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426343054/job/121932999182)
proves 14 new assertion failures while all 11 existing unit cases pass.
Local RED browsers reproduce 20 new failures while all 32 old cases
pass. GREEN `52eabad7` passes 221 standalone suites, 28 ownership unit
checks and 56 browser cases across desktop/mobile and both themes; the
parent independently repeated the 56 browser cases.

All three independent review roles passed. Held-response assertions
prove the rendering behavior without claiming a server-latency speedup.
[Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427087948)
passes on this exact head, including all three browser shards, the
repeated slide-over stability gate and the architecture build.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:34:12 +01:00
n30nexandn30nex 1ed5e4ab3f fix(compare): exclude observer TX-only packets from overlap (#2346)
Red commit: `39295da8` ([intended CI assertion
failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)).
Green commit: `d1bda496`.

Observer comparison counted an observer's TX-only packets as receptions,
producing false overlap. It now requests `excludeTx=true`: the selected
observer and non-TX direction must match the same observation in memory
and SQLite v2/v3. Normal packet lists and stored TX records remain
unchanged; missing directions retain legacy compatibility.

Refs #667. This completes a comparison slice, not the issue's remaining
observer counters or #671's fleet/SNR feature. Representative display
fields and existing time-filter semantics are unchanged. API/OpenAPI
docs describe the filter and multi-node bypass.

Validation:
- RED: Go tests compiled with 32 intended failures/16 control passes;
unit and browser regressions reproduced false overlap. CI reached the
new unit assertion.
- GREEN: all 48 packet cases pass, plus grouped-cache isolation and
OpenAPI tests. The added cache regression caught a malformed-radio
cache-key collision before correction.
- Full local server suite: 174s; ingestor: 185s; `npm run test:unit`:
pass. Final focused packet/grouped/OpenAPI checks: pass.
- E2E assertion added to
`tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks
pass, including correct 1/2 reception totals and 50% overlap.

Performance: still two capped bulk requests, indexed candidate lookups
retained, one direction check within the existing observation scan/SQL
predicate, no new persistent structures. No configuration/customizer
changes.

Final-head CI passed; independent reviews found no must-fix items.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:34:06 +01:00
n30nexandn30nex c211ef7643 feat(nodes): show relay activity from certain forwarding evidence (#2336)
Fixes #2331 with the additive API option.

The Last Relayed row shows green **actively relaying** only when
`last_certain_relay` falls within the existing
`healthThresholds.relayActiveHours` window. Missing, invalid, future or
expired evidence adds no idle/alive claim. Existing `relay_active`,
attributed timestamps and counts retain their meanings.

The optional node list/detail field uses the existing certain-relay
path: an eligible observed frame, matching hop, unique prefix, no
retired namesake and existing signature checks. Originating an advert
does not count. Forwarding another node's advert can count, even with
zero non-advert attributions. Adverts with an unknown origin cannot
establish forwarding.

Evidence uses transmission `first_seen`, not late observation receipt
time, and follows node-membership eviction. Snapshots take one read lock
over requested keys, O(page size), with no packet scan or new database
query. Work per qualifying hop stays bounded. No new dependency or
setting; configuration documentation and OpenAPI explain the
distinction.

The test-first history includes assertion-failing RED commits and GREEN
fixes. The initial RED's nil-return compile stub is explicitly permitted
by `docs/agents/TDD.md`. Local native server/ingestor suites, focused
race checks, 809 frontend helper assertions and 70 browser cases pass.
Browser coverage includes both themes and full/side/tablet detail views.

[Review regression
RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40421464618/job/121916723339)
proves both malformed-origin failures. All three independent review
roles pass at `d1fe79cb`. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40422655728)
passes on that exact commit, including all three browser shards and the
two-architecture image check.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:33:59 +01:00
n30nex 1e1bfa0255 fix(ingestor): require unique hop prefixes for relay liveness (#2334)
Fixes #2329 (option 1: unique-prefix liveness).

The ingestor could select a likely relay through neighbour adjacency and
refresh its `last_seen` even when its prefix matched two nodes.
Contextual resolution remains available in stored `resolved_path`;
liveness now uses the existing unique-prefix resolver, sharing the
parsed path and prefix-index snapshot. The current flood-frame
eligibility gate remains unchanged. Retired-prefix catalogue parity with
the server is separate.

The native SQLite regression exercises real `InsertTransmission`: the
displayed path still selects the adjacent candidate, neither colliding
relay is touched, and the unique preceding hop is refreshed. Existing
direct/TRACE/flood checks remain intact. [RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40407478987/job/121868990704)
fails the timestamp/touch-count assertions; [GREEN race
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409860054/job/121877671472)
passes at `77be50e8`.

Full local server and ingestor suites, repeated focused race checks and
all 221 frontend suites pass. Existing test assertions are unchanged;
one misleading resolver comment is corrected. Frame/path assumptions
were checked against MeshCore firmware revision `81dbae59`.

Performance: one additional indexed lookup per hop, O(h), with no node
scan or extra database query. No schema, dependency, configuration or
customizer change. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409860054)
passes on `77be50e8`, including the race detector, all browser shards
and architecture builds. Three independent reviews found no blockers;
final repository checks pass. Ready for maintainer review.
2026-10-29 18:33:53 +01:00
n30nex f9b7c66a8b fix(ingestor): open buffered writes before startup packet retention (#2328)
Partial fix for #2325; leave the issue open for the remaining startup
work.

MQTT subscribed early but its buffer could not drain until the entire
packet-retention backlog finished. Required startup work and the
migration barrier still finish first; ingestion now opens before the
initial packet sweep, so buffered writes can use the gaps between the
existing 250-transmission transactions.

The sweep remains synchronous with its count/error logging. No new
worker, shutdown owner, setting, retention window or batch size. The
readiness log reflects the actual gate opening. Earlier non-packet
maintenance, initial vacuum ordering and diagnostics publication during
maintenance remain unchanged.

Native SQLite tests exercise blocked migrations, inter-batch writes,
FIFO/drop accounting, stopped buffers, callback counts, rollback/retry
and main-function wiring. Removing the migration wait fails the
regression. Existing tests are unchanged; GREEN strengthens the new RED
test with a blocked-gate assertion and early-failure cleanup.

[RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40308382930)
fails the intended assertions. [Final-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40311222867)
passes on `f8a4d175`, including Go/JS tests, race checks, all browser
shards and architecture builds. Full local ingestor tests, repeated
focused race checks and local browser/API smoke checks pass. Three
independent reviews found no blockers in this partial scope.

This removes the whole-sweep startup gate; it does not bound an
individual deletion transaction or claim a measured live latency
improvement.
2026-10-29 18:33:47 +01:00
n30nexandn30nex ef17d78909 docs(api): document node resolution and battery history (#2338)
Partial follow-up to #1670; the broader documentation issue stays open.

Documents the existing node-prefix resolution and node battery-history
GET APIs: parameters, normalization and lookup rules, defaults,
ambiguity/visibility, ordered samples, nullable latest values and error
media types. The generator now has concrete success schemas for both.
Error statuses remain documented in prose under its existing convention.

Removes exactly these two entries from the known-gap list: 18 → 16, with
115 of 131 routes described. No handler, query, configuration or
dependency changes.

[RED
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426035729/job/121931981058)
reaches the missing-documentation assertions. GREEN `521fb3ef` passes
both full native CGO suites and focused OpenAPI/real-handler contract
tests. Parent browser acceptance passed 28 checks through the rendered
Swagger UI and actual endpoints, including defaults, errors and
empty/null history.

All three independent review roles passed. [Final
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427081025)
passes on this exact head, including all browser shards and the
two-architecture image check.

---------

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:33:41 +01:00
n30nexandn30nex 8437d22356 docs: use consistent snapshots for backup and restore (#2345)
The deployment guide copied a running `meshcore.db`, which can omit
committed WAL data or produce an inconsistent backup. It now uses the
existing authenticated `/api/backup` snapshot, checks the download
before publishing it, and explains how to restore with all database
users stopped while preserving the old database and sidecars for
rollback.

Fixes #703, following the [maintainer's remaining documentation
scope](https://github.com/Kpa-clawbot/CoreScope/issues/703#issuecomment-6163256620).

- Protects API credentials in a header file and keeps incomplete
downloads out of the completed backup set.
- Links the separate account-backup guide; configuration and secrets
remain separate backup responsibilities.
- Replaces unsafe live-copy scheduling instructions with the same
checked download procedure.

Validation:
- Exact shell example with real curl/SQLite: valid download, HTTP
401/500, empty, HTML, corrupt database, and truncated transfer. Failures
preserved the previous backup and published no new `.db`.
- Existing `TestBackup*` passed. Chromium downloaded an authenticated
snapshot from a disposable instance; a row committed only in its
nonempty WAL survived and `quick_check` returned `ok`.
- Disposable stopped-file restore and rollback recovered the original
WAL-only row and preserved the original files byte-for-byte.
- `git diff --check` and privacy preflight passed.

TDD exemption: pure docs; no application behavior changed and no test
files modified. No new config/customizer settings. Live operator
services were not used for validation.

Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>
2026-10-29 18:33:14 +01:00
efitenandClaude Opus 5.5 15e2d2a4a7 fix(packets): read HB from the row's own raw_hex header (#2351)
## Problem

The HB column on the packets page can show `4`. Path mode 3 (hash size
4) is reserved: `Packet::isValidPathLen` in `src/Packet.cpp` returns
false for it, and `Dispatcher::checkRecv` drops such a packet. So HB 4
is never a real value.

The three HB cells in `public/packets.js` found `path_len` with
`getPathLenOffset(p.route_type)`, the route type of the
**transmission**. The raw_hex in a child row (and in a live-updated row)
is an **observation's** raw_hex. The packet hash does not cover the
route bits, so the same packet is heard as FLOOD by one observer and as
TRANSPORT_FLOOD by another, and the transmission keeps the route type of
its first observation.

Example from a production DB (transmission 3645334):

| source | raw_hex start | path_len at |
|---|---|---|
| transmission | `01 12 3A646A…` (FLOOD) | byte 1 |
| observation | `00 E2E0 0000 1D 3A646A…` (TRANSPORT_FLOOD) | byte 5 |

Read at the FLOOD offset, the observation gives `0xE2`, `0xE2 >> 6 = 3`,
`+ 1 = 4`. In the last 300,000 observations of that DB, about 4,300 have
a header route type that differs from their transmission's.

## Fix

`pathHashSize(rawHex)` in `public/app.js` already takes the route type
from the header byte of the same raw_hex, and the detail pane uses it.
The three HB cells now call it through a small `hashBytesText` helper
instead of carrying their own copy of the offset logic.

Behaviour change: a raw_hex with no hash size (reserved mode 3, a DIRECT
packet with no hops, missing raw_hex) shows the same placeholder as
TRACE rows instead of a number. Before, missing raw_hex showed `1`.

## Tests

- New `tests/unit/test-hb-column-header-route.js` (7 cases, registered
in `test-all.sh`): transport raw_hex on a FLOOD transmission gives 1,
flood raw_hex with 2-byte hashes on a TRANSPORT_FLOOD transmission gives
2, reserved mode 3 never gives 4, for the helper, the flat row and the
group header. It fails on `upstream/master` with `'4' !== '1'`.
- `sh test-all.sh`: all standalone frontend suites pass.
- Browser: local server on a migrated copy of the E2E fixture, with one
observation rewritten to a TRANSPORT_FLOOD raw_hex. The expanded group
shows HB 2 on all three rows. The old code reads `0xE2` on that child
and shows 4.

## Performance

Same per-cell work as before: one header parse and one byte parse per
rendered row, O(1), no new allocations beyond the existing regex test in
`pathHashSize`. Only visible rows are rendered.

## Not done

- The child-row call site is not hit by a unit test: child rows only
render when the group is in `expandedHashes`, which the test API does
not expose. It is covered by the helper test and the browser check
above.
- The HB sort accessor (`hash_byte_count` / `hash_size`, sort key `hb`
in `packets.js`) is unchanged; it does not read raw_hex.
- `transportBadge(c.route_type)` on child rows still follows the stored
route type, not the observation header. Left out of scope.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 06:42:51 +01:00
efitenandClaude Opus 5.5 b65412ab83 feat(observer-logins): turn on neighbour reports in the firmware block of the credentials mail (#2349)
The firmware block of the observer credentials mail now ends with `set
mqtt.neighbors on`, so firmware observers send the same `/neighbors`
report that the OpenHop block already turns on ("Publish Neighbours",
#2347). The ingestor reads that report for region scopes (#1865).

## Changes

- `cmd/server/observer_mail.go`: one extra line in the firmware block,
and one sentence in its note. The sentence says the command sends the
neighbour table and region scopes once a day, and that older firmware
does not know it and should skip it.
- `cmd/server/observer_mail_test.go`: the full firmware block assertion
includes the new line, and the slot test checks the line and the note.
- `docs/observer-mqtt-logins.md`: the mail section describes the new
line.

## Tests

- `go test ./...` in `cmd/server`: ok. Skipped
`TestSaveGeoFilterPreservesFileMode`, which needs Unix file modes; this
run was on Windows.
- `TestObserverCredentialsMail*` failed before the change, on the
missing line, and passes after it.

## Not checked

- The command name and the 24-hour default come from #1865. I did not
read the observer firmware source.
- The wording of the error older firmware gives for an unknown command
is not quoted in the mail. The note only says to skip the line.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 06:25:42 +01:00
efitenandClaude Opus 5.5 ddce76a685 ci(e2e): run the E2E shards on ubuntu-24.04 and fail on a broken browser install (#2350)
`ubuntu-latest` now hands out `ubuntu-26.04` runners. Every E2E shard
that lands on one fails before a single test runs:

```
browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium_headless_shell-1208/...
```

The step "Install Playwright browser" runs `npx playwright install
chromium 2>/dev/null || true`, so its own failure is invisible and the
job only breaks at the first browser launch.

## Evidence

- Run 40779894898 (PR #2349, a mail-text change): three attempts, each
failing on `Executable doesn't exist`. The job log shows `Image:
ubuntu-26.04`, OS `26.04.1 LTS`.
- The master run at 2026-10-28 21:31Z passed. Its shard log shows
`Image: ubuntu-24.04`.
- `playwright` in package-lock.json is 1.58.2.

## Change

- `e2e-shard` gets `runs-on: ubuntu-24.04`, with a comment saying when
to unpin. The other jobs do not install Playwright and stay on
`ubuntu-latest`.
- "Install Playwright browser" no longer has `2>/dev/null || true`, so a
failed install fails at that step.

## Not done

- I did not find out why the install fails on 26.04: its stderr was
discarded. Once this is merged, a run with the pin removed would show
it.
- No Playwright upgrade. That is a separate change, and it may make the
pin unnecessary.
- Before this change, `install-deps` errors were also swallowed. If it
fails on 24.04, this PR's own CI will show it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 00:07:30 +01:00
efitenandClaude Opus 5.5 cadc666bd9 feat(observer-logins): OpenHop block in the credentials mail (#2347)
The observer credentials mail (#2306) had blocks for meshcoretomqtt, the
Gessaman/Dutch-MeshCore firmware, Home Assistant and plain values. An
operator running OpenHop (repeater software with its own MQTT uploader)
could not tell which of these matched its broker form, and picked the
format by trial.

## Changes

- `cmd/server/observer_mail.go`: a fifth block, "4. OpenHop (repeater
software)", before the plain values. It uses the field and option labels
of the OpenHop broker form:
- `Format: Letsmesh`, `Transport: TCP` or `Websockets` (from
`observerLogins.transport`), host, port, username, password, `TLS:
on/off`, `Use JWT Auth: off`, `Retain Status: on`, `Publish Neighbours:
on`.
- The note says Meshcoretomqtt works too, but the format labelled
"openHop" (value `mqtt`) does not: it publishes to
`meshcore/repeater/<name>`, which a personal login may not use.
- Publish Neighbours is on because the ingestor reads the `neighbors`
topic for region scopes (#1865).
- `openapi.go` and `docs/observer-mqtt-logins.md`: "four formats"
becomes five.

## Sources

Read from OpenHop, not tested on a device by me:
- format family and topics:
`openhop_repeater/repeater/data_acquisition/mqtt_handler.py`
(`MC2MQTT_FORMATS = ("meshcoretomqtt", "letsmesh", "waev")`, the `mqtt`
format's `meshcore/repeater/<node_name>` base topic, transport values
`tcp`/`websockets`);
- labels: the built web UI (`Configuration-*.js`: "Add Broker", option
labels `Letsmesh`, `Meshcoretomqtt`, `openHop`, `TCP`, `Websockets`,
"Retain Status", "Publish Neighbours", "Use JWT Auth").

An operator on our instance set it up from a screenshot of that form
with format Letsmesh, TCP, Retain Status and Publish Neighbours on; the
observer publishes, and its `neighbors` report set `configured_scope` on
its node.

## Tests

- `TestObserverCredentialsMailOpenHopBlock`: the block text for TCP
without TLS and for Websockets with TLS, the "openHop" warning, HTML
escaping, and the block order.
- `TestObserverCredentialsMailBlockTitles`: five titles in order.
- `cmd/server` suite green locally (the two Windows-only failures
skipped, as on master).

## Not in this PR

- The websocket path: OpenHop uses paho's default, the doc already tells
operators to match the listener path.
- Anything with the other fields of the `neighbors` report (SNR,
heard_secs_ago, default_scope): we are measuring how many observers send
it first.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:31:43 +01:00