meshcore-analyzer

0 Fork 0

mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-11-11 19:37:33 +00:00

Author	SHA1	Message	Date
n30nexandn30nex	d083f9ade9	test: cover partial observer chart construction failures (#2354 ) Partial follow-up for #1778; addresses the chart-cleanup test gap identified in the review of #2339. The observer detail unit harness now throws during the second or fourth chart constructor. Both cases require existing chart instances to be destroyed, metadata/calibration to remain visible, the error to be rendered as text, and a later time-window retry to recover without extra requests. Production cleanup already behaves correctly; this PR adds coverage only. The parent-run ownership suite passes 30/30. Removing the actual cleanup produces exactly the two intended assertion failures. The remaining unit sources and runner match the current-master baseline whose 222 frontend suites passed; only this updated unit file differs within that test set. TDD scope: no production change or new behavior. The negative control removes existing cleanup to prove the new assertions detect its loss. No configuration, dependencies or runtime performance changes. Exact-head CI passed, including the browser shards. Independent adversarial, correctness and TDD reviews found no must-fix items. Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>	2026-10-31 12:00:06 +01:00
efitenandClaude Opus 5.5	bb3c8cc001	fix(rx-coverage): My coverage and the observer filter on the noise layer (#2326 ) "My coverage" and a selected observer only narrowed the signal layer on the coverage page. On the noise layer the toggle stayed pressed, but `/api/rf-noise` was called without `mine` or `rx` and returned every companion's samples. ## Changes - Backend: `/api/rf-noise` now accepts `rx` and `mine=1`, with the same session rules as `/api/rx-coverage`: 401 without a session, 403 for a device token, 404 with user management off. The `rx_pubkey` filter moved into 1 helper, `appendRxFilter` (`cmd/server/rx_dashboard.go`), which both queries use. Both parameters are in the OpenAPI entry. - Frontend: `public/rx-coverage.js` fetches both layers through `fetchLayer`. It builds the URL with the time window, observer and toggle, and judges the reply with the existing `coverageReply`. The request sequence (`signalSeq`, now `drawSeq`) covers both layers, so the noise layer also drops a superseded reply. Before, a slow unfiltered noise reply could overwrite a newer filtered one. ## Performance The noise query gets 1 extra `WHERE` clause on `rx_pubkey`. There are no new requests: the page still makes 1 request per redraw. ## Verification - New Go tests in `cmd/server/rf_noise_mine_test.go`: `rx` narrows the result (case-insensitive), `mine=1` returns 401 without a session, returns empty with nothing linked, returns the linked companion's cells, returns 403 for a device token, and returns 404 with user management off. Both failed before the fix. - `tests/unit/test-rx-coverage-mine.js` is updated for the new `coverageUrl(path, ...)` signature and checks that both layers fetch through `fetchLayer`. - New step in `tests/e2e/test-rx-coverage-noise-e2e.js`: a `?layer=noise&rx=...` link sends `rx`, and clicking "My coverage" refetches with `mine=1`. It fails against the current master frontend and passes with this change. - Local runs (Windows): `go test` and `go vet` in `cmd/server`, `test-all.sh`, and the noise, gaps and viewport E2E in Chrome. All green. I did not run the full Playwright suite locally. ## Not in this PR - The "nothing received" gap cells still skip `mine=1`, unchanged: the track query cannot narrow to a set of companions yet. - The time window (`days`) already worked on the noise layer, so it is not touched. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 22:39:02 +01:00
TeTeHackoandClaude Opus 5.5	417c8dd00a	feat(observers): latest /neighbors report on the observers list and page (#2270 , #2271 ) (#2300 ) Closes #2270, closes #2271. The ingestor took only the confirmed region scopes out of an observer's `/neighbors` report (#1865) and dropped the rest. Nothing could therefore say when an observer last reported, or whom it heard. ## Ingestor `observer_neighbor_reports` (created in `dbschema.Apply`) keeps the latest report of each observer, one row per observer, replaced by every live report. The report is unauthenticated, so `neighbors[]` is stored sanitised: - only 64-hex keys - `snr` and `heard_secs_ago` finite and in range - `status` limited to `[a-z_]` - `scopes` clamped - at most 500 rows `total_neighbors` is believed only when it is not below the number of rows that arrived. A retained replay only fills a missing row, and it is dated by the report's own `timestamp`. After an ingestor restart, a long-dead observer therefore does not look as if it just reported. ## API The table is optional for the server (no `AssertReady` change); without it the fields are simply absent. - `GET /api/observers`: `neighbors_report_at` and `neighbors_count`, from one batch query in the existing cached builder. - `GET /api/observers/{id}`: also `neighbor_report`, which is `{received_at, reported_at, neighbor_count, queried_neighbors, self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr, heard_secs_ago, status, scopes}]}`. Names and roles come from the node table in one query. ## Frontend `public/observer-neighbors.js` is shared by both pages and escapes everything. - #2270, observers list: a sortable Neighbors column, e.g. `3h ago · 12`, or `—` when the observer never reported. The same value appears in the slide-over. - #2270, observer page: Neighbors Report and Neighbors stat cards. - #2271, observer page: a Neighbors table listing each neighbor by name (a link to the node) or key prefix, with SNR, heard, scope query status and scopes. The heading says when the list is shorter than the count, e.g. `(2 of 3 listed)`. ## Tests - ingestor: sanitising, the cap, live replace, the retained rules (fill only, dated by the report, undated ignored) and the MQTT topic end to end. The existing #1865 scope tests are unchanged and still pass. - server: the list summary, the detail with names, and both endpoints without the table. - frontend: - unit tests of the helpers - an XSS case for every rendered field; I checked that it fails when escaping is removed - `test-observers-headings` now expects the new column after `Last Packet` - E2E: `test-observer-neighbors-2270-e2e.js` intercepts the observer endpoints, since the fixture has no reports. It is listed in `scripts/non-unit-tests.json` and on shard 2. Gates that pass: - `go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema` - `sh test-all.sh` - `eslint@8`, with 0 errors - the XSS diff gate - these E2E suites: the new one, `test-issue-1639-observers-sort-e2e.js`, `test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js` Performance: the list adds one primary-key-sized query per cache fill, and the detail adds two small queries per request. Also fixes #2330 (folded in from #2337, second commit). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 22:36:53 +01:00
n30nexandn30nex	202bafea28	fix(observers): show metadata while analytics loads (#2339 ) Partial follow-up to #1778, limited to observer details; other pages remain separate. Observer identity, status and radio information now render when the basic request finishes. Analytics and clock calibration load into their own regions, so either can finish first. An analytics failure preserves usable metadata. Identity failures and obsolete responses cannot overwrite the current view, and failed chart construction is cleaned up. The same three requests start together. Generation ownership, request deduplication, date-window behavior and disposal remain covered. No new fetches, settings, dependencies or customizer options; secondary replies preserve the actual metadata DOM node. [RED CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426343054/job/121932999182) proves 14 new assertion failures while all 11 existing unit cases pass. Local RED browsers reproduce 20 new failures while all 32 old cases pass. GREEN `52eabad7` passes 221 standalone suites, 28 ownership unit checks and 56 browser cases across desktop/mobile and both themes; the parent independently repeated the 56 browser cases. All three independent review roles passed. Held-response assertions prove the rendering behavior without claiming a server-latency speedup. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427087948) passes on this exact head, including all three browser shards, the repeated slide-over stability gate and the architecture build. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>	2026-10-29 18:34:12 +01:00
n30nexandn30nex	1ed5e4ab3f	fix(compare): exclude observer TX-only packets from overlap (#2346 ) Red commit: `39295da8` ([intended CI assertion failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)). Green commit: `d1bda496`. Observer comparison counted an observer's TX-only packets as receptions, producing false overlap. It now requests `excludeTx=true`: the selected observer and non-TX direction must match the same observation in memory and SQLite v2/v3. Normal packet lists and stored TX records remain unchanged; missing directions retain legacy compatibility. Refs #667. This completes a comparison slice, not the issue's remaining observer counters or #671's fleet/SNR feature. Representative display fields and existing time-filter semantics are unchanged. API/OpenAPI docs describe the filter and multi-node bypass. Validation: - RED: Go tests compiled with 32 intended failures/16 control passes; unit and browser regressions reproduced false overlap. CI reached the new unit assertion. - GREEN: all 48 packet cases pass, plus grouped-cache isolation and OpenAPI tests. The added cache regression caught a malformed-radio cache-key collision before correction. - Full local server suite: 174s; ingestor: 185s; `npm run test:unit`: pass. Final focused packet/grouped/OpenAPI checks: pass. - E2E assertion added to `tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks pass, including correct 1/2 reception totals and 50% overlap. Performance: still two capped bulk requests, indexed candidate lookups retained, one direction check within the existing observation scan/SQL predicate, no new persistent structures. No configuration/customizer changes. Final-head CI passed; independent reviews found no must-fix items. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>	2026-10-29 18:34:06 +01:00
n30nexandn30nex	c211ef7643	feat(nodes): show relay activity from certain forwarding evidence (#2336 ) Fixes #2331 with the additive API option. The Last Relayed row shows green actively relaying only when `last_certain_relay` falls within the existing `healthThresholds.relayActiveHours` window. Missing, invalid, future or expired evidence adds no idle/alive claim. Existing `relay_active`, attributed timestamps and counts retain their meanings. The optional node list/detail field uses the existing certain-relay path: an eligible observed frame, matching hop, unique prefix, no retired namesake and existing signature checks. Originating an advert does not count. Forwarding another node's advert can count, even with zero non-advert attributions. Adverts with an unknown origin cannot establish forwarding. Evidence uses transmission `first_seen`, not late observation receipt time, and follows node-membership eviction. Snapshots take one read lock over requested keys, O(page size), with no packet scan or new database query. Work per qualifying hop stays bounded. No new dependency or setting; configuration documentation and OpenAPI explain the distinction. The test-first history includes assertion-failing RED commits and GREEN fixes. The initial RED's nil-return compile stub is explicitly permitted by `docs/agents/TDD.md`. Local native server/ingestor suites, focused race checks, 809 frontend helper assertions and 70 browser cases pass. Browser coverage includes both themes and full/side/tablet detail views. [Review regression RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40421464618/job/121916723339) proves both malformed-origin failures. All three independent review roles pass at `d1fe79cb`. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40422655728) passes on that exact commit, including all three browser shards and the two-architecture image check. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com>	2026-10-29 18:33:59 +01:00
efitenandClaude Opus 5.5	15e2d2a4a7	fix(packets): read HB from the row's own raw_hex header (#2351 ) ## Problem The HB column on the packets page can show `4`. Path mode 3 (hash size 4) is reserved: `Packet::isValidPathLen` in `src/Packet.cpp` returns false for it, and `Dispatcher::checkRecv` drops such a packet. So HB 4 is never a real value. The three HB cells in `public/packets.js` found `path_len` with `getPathLenOffset(p.route_type)`, the route type of the transmission. The raw_hex in a child row (and in a live-updated row) is an observation's raw_hex. The packet hash does not cover the route bits, so the same packet is heard as FLOOD by one observer and as TRANSPORT_FLOOD by another, and the transmission keeps the route type of its first observation. Example from a production DB (transmission 3645334): \| source \| raw_hex start \| path_len at \| \|---\|---\|---\| \| transmission \| `01 12 3A646A…` (FLOOD) \| byte 1 \| \| observation \| `00 E2E0 0000 1D 3A646A…` (TRANSPORT_FLOOD) \| byte 5 \| Read at the FLOOD offset, the observation gives `0xE2`, `0xE2 >> 6 = 3`, `+ 1 = 4`. In the last 300,000 observations of that DB, about 4,300 have a header route type that differs from their transmission's. ## Fix `pathHashSize(rawHex)` in `public/app.js` already takes the route type from the header byte of the same raw_hex, and the detail pane uses it. The three HB cells now call it through a small `hashBytesText` helper instead of carrying their own copy of the offset logic. Behaviour change: a raw_hex with no hash size (reserved mode 3, a DIRECT packet with no hops, missing raw_hex) shows the same placeholder as TRACE rows instead of a number. Before, missing raw_hex showed `1`. ## Tests - New `tests/unit/test-hb-column-header-route.js` (7 cases, registered in `test-all.sh`): transport raw_hex on a FLOOD transmission gives 1, flood raw_hex with 2-byte hashes on a TRANSPORT_FLOOD transmission gives 2, reserved mode 3 never gives 4, for the helper, the flat row and the group header. It fails on `upstream/master` with `'4' !== '1'`. - `sh test-all.sh`: all standalone frontend suites pass. - Browser: local server on a migrated copy of the E2E fixture, with one observation rewritten to a TRANSPORT_FLOOD raw_hex. The expanded group shows HB 2 on all three rows. The old code reads `0xE2` on that child and shows 4. ## Performance Same per-cell work as before: one header parse and one byte parse per rendered row, O(1), no new allocations beyond the existing regex test in `pathHashSize`. Only visible rows are rendered. ## Not done - The child-row call site is not hit by a unit test: child rows only render when the group is in `expandedHashes`, which the test API does not expose. It is covered by the helper test and the browser check above. - The HB sort accessor (`hash_byte_count` / `hash_size`, sort key `hb` in `packets.js`) is unchanged; it does not read raw_hex. - `transportBadge(c.route_type)` on child rows still follows the stored route type, not the observation header. Left out of scope. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 06:42:51 +01:00
n30nex	3bf678709f	fix(ui): preserve navigation and menu focus during delayed updates (#2333 ) Fixes #2332. Fixes #2335. Two delayed UI updates could undo user actions. Rebuilding the More menu removed its focused link; a late packet-list response could replace a newly requested Nodes URL before `hashchange` ran. The menu now preserves the selected destination when it stays in More or moves inline, closes an empty menu, and leaves other controls' focus alone. Packet URL updates now require the packets route, including its existing detail/query forms. Observation selection and encoded filters remain supported. The changes add one lookup across the existing 12 navigation links and a URL ownership check. No packet loop, API request, dependency, setting or customizer change is added. [Packet RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409110743/job/121874676123) fails four wrong-route assertions. [Menu RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409898375/job/121877663579) fails the intended focused-destination assertion after the packet correction removes the competing navigation failure. GREEN is `57522143`. Local combined validation passes all 221 standalone frontend suites, 60 navigation browser checks, 17 branding/route checks and 11 filter/deep-link checks. The packet regression controls delivery of a real API response; substituting the original code still fails it. Existing exact title assertions remain; diagnostics now report the actual title. Backend source trees are unchanged from the tested baseline. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40410894023) passes on `57522143`, including all three browser shards and both architecture builds. Three independent source reviews found no blockers. Final repository checks and after-push browser verification pass. Ready for maintainer review.	2026-10-28 21:55:40 +01:00
efitenandClaude Opus 5.5	d29957fbdc	feat(observer-logins): lock the shared login out of migrated observers (#2348 ) Follow-up to #2306 / #2319. With personal observer logins, a migrated observer still is not safe from the shared login: the hook answers `ignore` for the shared login, so EMQX's built-in database lets it publish under any observer's topic, including one whose owner has moved to a personal login. This PR lets an operator close that per migrated observer, before the shared login is removed altogether. ## Changes users.db v8. `observer_bindings.confirmed_at`: the moment the owner's personal login first published under that pubkey. - An auto-bind (the owner's first publish) is confirmed at once. - An admin assignment stays unconfirmed until the owner's login publishes under that pubkey; that publish sets it. - Reassigning to another user or unbinding clears it; reassigning to the same owner keeps it. - Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin` rows stay NULL. Config. `userManagement.observerLogins.sharedUsernames` (default empty): the broker's shared login names. A name starting with `obs_` is refused at startup. Authz hook. For a login in `sharedUsernames`, a publish under `meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is confirmed, whatever the number of topic segments (the ingestor attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey). Logged as `publish denied: user=<name> observer=<8 hex>: confirmed owner`. Every other request from a shared login keeps `ignore`, so the built-in database decides as before. Personal logins are unchanged. An orphaned binding (owner account deleted) does not count as confirmed. Admin. The observers table of the Observer logins tab gets a Confirmed column; the bindings JSON gets `confirmedAt`. Docs. `docs/observer-mqtt-logins.md`: "Locking the shared login out of migrated observers" (what it denies, how to see `confirmedAt`, the risk below). `config.example.json` has the key. ## Risk for operators An observer that is confirmed but still runs a second uploader on the shared login loses that second uploader once its name is in `sharedUsernames`. The deny log line names the observer. ## Tests - `internal/users`: v7 to v8 migration on a real v7 database, auto confirm, admin assignment then owner confirm, non-owner and repeated confirm, reassign clears, orphan reassign. - `cmd/server`: the decision table (confirmed, unconfirmed, unbound, non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty and `client` iata, lowercase key; owner publish confirms, confirm failure still allows and retries; reassign, unbind and user removal clear the flag; the index load against a real store; config validation; the hook and admin endpoints end to end. - `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an auto-bound observer. - Local: `internal/users`, `cmd/server` (two Windows-only tests skipped, as on master) and `cmd/ingestor` green. The E2E assertion was not run locally; CI runs it. ## Not in this PR - Using `confirmed_at` for verified `/neighbors` reports: a separate PR, built on #2300. - Removing the shared login: still an operator step after migration. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-28 19:10:41 +01:00
TeTeHackoandClaude Opus 5.5	77a5f14925	feat(observers): radio setting per observation from the observer's radio history (#1869 ) (#2292 ) Part of #1869. ## Problem `observers.radio` keeps only the last `MHz,kHz,SF,CR` an observer reported. Once an observer is retuned (say from SF7 to the EU narrow SF8 setting to listen to a neighbouring mesh), every packet it ever heard reads as heard on the new setting, and there is no way to tell an SF7 packet from an SF8 one. ## What this does As #1869 proposes, the packet data model is untouched: the setting of an observation is resolved from its observer and time. Ingestor (all schema in `dbschema.Apply`) - `observer_radios`: one row per stretch of time an observer, and each radio of a multi-radio node, stayed on one setting. Every live status extends the current row and a change starts a new one; retained replays never extend a row. On the first run it is seeded with every observer's current radio, which is what the UI already showed for the past. - Settings are stored canonical through `dbschema.CanonicalRadio`, so `869.4320068,62.5,7,5` and `869.432,62.50,7,5` are one setting. - `observation_radios` (`observation_id → rx_radio_id`): multi-radio openHop nodes tag each packet with `rx_radio_id` and list `radios: [{id, radio}]` in their status, so their observations resolve per radio. - The table is sparse: single-radio nodes never send the field, so `observations` and every query that scans it stay untouched. - The row id comes from the observation upsert via `RETURNING id`. - Rows are pruned with their transmissions, and a purged observer takes its history along. Server (read-only) - No `AssertReady` change: an absent table reads as empty and is probed again later, the same rule as `advert_route_evidence`. - The history is held in memory. The live poller checks it on every tick (two primary-key lookups), so a retune or a new `rx_radio_id` is known before the observation is broadcast. A full reload runs every 10 min. - `radio` is added to observations, packets, grouped rows (taken from the header observation) and the live feed. - `GET /api/packets?radio=MHz,kHz,SF,CR` filters by setting. - Combined with `observer=`, it means that observer heard it on that setting. - A radio query is served from memory instead of falling back to SQL, which would return the window unfiltered. - `GET /api/radios` (named struct) and `radio_history` on `GET /api/observers/{id}`. Frontend - Packet filter fields `radio`, `freq`, `bw`, `sf`, `cr`, e.g. `sf == 8` or `freq > 869.5 && bw == 62.5`. - A Radio row in the packet detail and a Radio column in its observations table (only when the server reports one). - A radio history table on the observer page once there is more than one stretch. - `RfParams.radioText` formats the setting. It escapes all five characters and returns non-numeric input as it came, escaped. ## Performance `txMatchesRadio` runs once per observation inside `filterPackets`. - Each observation costs one map lookup by observer id. - Observers with a single stretch (nearly all) answer from that lookup directly. - Retuned observers add a binary search over their few stretches. - Multi-radio observers add one lookup by observation id. - The snapshot is loaded once per filter call. `BenchmarkTxMatchesRadio`, 30K packets / 150K observations (20 observers, 2 of them retuned): ``` BenchmarkTxMatchesRadio-24 602 2056924 ns/op 0 B/op 0 allocs/op ``` Memory is one entry per stretch plus one per multi-radio observation; nothing is added per observation otherwise. ## Tests - dbschema: seed runs once, canonical spellings, the tables exist without `observers.radio`. - ingestor: - stretches extend and split; canonical spellings do not split - two radios of one node hearing one transmission over two paths each keep their own radio - a retained status does not write history - prune and purge - `rx_radio_id` reaches `PacketData` - server: - resolution before, between and after stretches, and with an unknown radio id - `observer` + `radio`, and grouped rows carry `radio` - incremental refresh, `/api/radios`, `radio_history`, missing tables - the benchmark above - frontend: the new filter fields in `tests/unit/test-packet-filter.js`, and XSS cases through the real `RfParams.radioText` and the history renderer. - E2E: `tests/e2e/test-radio-setting-1869-e2e.js` checks the grouped rows' radio, `sf ==` on the packets page, the detail Radio row and the observer history. It is added to `scripts/non-unit-tests.json` and to shard 2. `go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema` passes, along with `sh test-all.sh`, `eslint@8` (0 errors) and the XSS diff gate. The new E2E test and `test-e2e-playwright.js` pass against the fixture. ## Browser validation This runs on a live instance with six observers, one of them retuned to 869.618/62.5/SF8/CR8: - The packet detail shows `Radio: 869.432 MHz · SF7 · BW 62.5 kHz · CR 4/5`. - `sf == 7` shows 106 of 106 grouped rows and `sf == 8` shows 0 of 107; the SF8 observer has not heard anything yet. - `/api/radios` lists both settings. - No page errors. ## Notes - The customizer (rule 8) gets nothing: the only constants are server-side refresh intervals. - Attribution timing: packets heard between a retune and the observer's next status message (minutes) are attributed to the previous setting, because a stretch starts when the status is handled. This is documented at `RecordObserverRadios`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-26 21:08:40 +01:00
n30nex	643a53e98d	fix(operators): refresh update metadata and wait for readiness (#2285 ) Red commit: `7277ca23` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190626951)). Fixes #2229. Management startup now defines logging helpers before normalizing CRLF environment files. Updates refresh version, revision and build time after checking out the target. Deployment verification and all five Compose healthchecks use `/api/healthz`, so an available statistics endpoint cannot report an unfinished startup as healthy. - The actual-script regression passes 15 checks: LF/CRLF/no environment file, repeat startup, all update targets, failed checkout, delayed/failed readiness, HTTPS readiness and unchanged informational statistics. - RED CI reproduced all 11 intended failures; GREEN `7ff9050b` passes the full standalone frontend suite and shell syntax checks. - Browser verified: an isolated Go fixture at http://127.0.0.1:18851 renders Home and returns ready=true from `/api/healthz`. - Git, Docker and network boundaries are mocked in management tests. No real container update was performed locally; CI validates the Compose/build path. - No new configuration, dependencies, schema changes or packet-processing work. Privacy, whitespace and scope checks passed. The external home-directory preflight bundle is unavailable; applicable checks were performed locally. Independent review found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191647297) passed on `7ff9050b`, including all three browser shards and both architecture builds.	2026-10-26 15:50:36 +01:00
n30nex	b5caea1d5f	fix(analytics): render safe, correctly sized battery status icons (#2282 ) Red commit: `4cc2efb3` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40189921192)). Fixes #2232. Battery badges now render their fixed status icon and append voltage as text, including zero. Chart sizing excludes inline icons, and status colors use the existing theme variables. Unknown status values stay muted. - The original browser regression failed on literal SVG; a second local run with only the JavaScript fix failed on oversized icons. - Validation on `eef0872e`: all standalone frontend suites; 47 focused unit checks; 41 Chromium checks, including both themes, hostile voltage, accessibility and 16×16 icon bounds. - Browser verified: http://127.0.0.1:18851 on an isolated Go fixture. - E2E assertion added: `tests/e2e/test-issue-1648-m2-icons-e2e.js:134`. - Constant work for one badge; no new requests, dependencies, settings or schema changes. Privacy, whitespace, CSS-variable and repository XSS checks passed. The external home-directory preflight bundle is unavailable; its applicable checks were performed locally. Independent review found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190620324) passed on `eef0872e`, including all three browser shards and both architecture builds.	2026-10-26 15:44:23 +01:00
n30nex	d07186af43	fix(nodes): preserve names and metadata in preview panels (#2315 ) Fixes #2309 and #2312. Partial follow-up to #1154 (U1 and tablet timestamp parity). Node previews omitted Last Relayed, Traffic share and Hash Prefix, while tablet previews read `advert_name` instead of the API's `name`. Shared rendering now presents those fields consistently across full detail, side preview and tablet, with the existing timestamp tooltips. Relay attribution is labelled neutrally: ambiguous relay evidence no longer claims a node is alive. Existing certainty/status logic is unchanged. Measured zero, legacy score fallback, role applicability, missing values and escaped names/hash tooltips are covered. A resize regression prevents duplicate score-row IDs across retained desktop and tablet previews. Only the selected node's metadata is formatted. Existing request counts remain five for full/side and two for tablet; no new fetch, cache, hot loop, setting, customizer value or dependency. Validation: 806 helper checks, all 218 standalone suites, 46 browser checks in both themes, eight health checks, and the main browser suite (149 passes, three existing fixture skips). The unchanged backend tree passed full server/ingestor suites. Independent reviews completed; the parent verified the two-line follow-up from its resize check. Test-only RED `e0f681ae` produced [eight expected assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40232409958/job/121286783705). Follow-up `2bb1ccd` repairs the help fixture's scroll range without relaxing its checks; all 13 help checks pass across three font/CPU configurations. [Final-head CI passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40234768485), including all browser shards and architecture builds.	2026-10-26 15:44:10 +01:00
n30nex	6e1abf0663	ui(nodes): identify path links as example packets (#2296 ) Red commit: `79abbba0` ([regression CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203075654)). Fixes #2291; addresses U7 of #1154. The two observed-path links now say View example packet →, accurately describing their existing sample-packet destination. Recent-advert labels and link behavior remain unchanged. - Production diff: two text literals in public/nodes.js. - Existing browser suite preserves contrast/highlighting assertions and adds exact labels, canonical destinations and real click-through from side/full/mobile views. - RED fails eight label assertions. GREEN passes 15/15 checks normally and with 4× CPU slowdown; all 218 standalone frontend suites pass. - Browser verified: http://127.0.0.1:18902. - E2E assertion added: tests/e2e/test-issue-1146-path-link-contrast-e2e.js. Initial loads and theme toggles wait for actual theme-refresh readiness before capture, fixing a reproduced screenshot-detachment race in the touched harness. - No runtime logic, requests, settings, dependencies or performance changes. Independent reviews and local preflight checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204585697) passed on `ae849cc6`, including all three browser shards and both architecture builds.	2026-10-26 15:43:57 +01:00
n30nex	fb101e46e0	fix(nav): fit primary links around late account controls (#2286 ) Red commit: `e5589b4c` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191348162)). Fixes #2216. Navigation measures the available flex strip instead of guessing component widths. Optional statistics and spacing yield when primary/active links would overlap More. Account insertion triggers a layout event immediately, including while authentication is pending or fails; normal user-change events retain their semantics. - Regression CI reproduced link overlap across narrow widths and late account controls. Review added a second RED test for the actual delayed authentication bootstrap before the three-line correction. - Validation on `f0ccbf4c`: full frontend suite; 89 user-management checks; 56 navigation checks; existing priority and clipping suites. Parent independently reran auth and browser checks. - Browser verified: http://127.0.0.1:18854; real bootstrap overlap of 14.66px becomes 21.38px clearance. Resizing, account changes, keyboard More and touch targets remain covered. - E2E assertion added: `tests/e2e/test-nav-fluid-1055-e2e.js:89`. - Work is bounded by the fixed navigation link set; events share one animation frame. No polling, observers, new requests, dependencies or settings. Privacy, whitespace, CSS-variable, XSS and ESLint checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. Independent review verified the bootstrap correction with no outstanding findings. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40192737545) passed on `f0ccbf4c`, including all three browser shards and both architecture builds.	2026-10-26 15:43:43 +01:00
n30nex	c853c64f1f	feat(ui): identify current views in browser tab titles (#2287 ) Red commit: `a62753de` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191439879)). Fixes #2259. Browser titles now identify the current page, Analytics tab, or selected node/packet while retaining the configured site name. A Reach view reads `Node abababababab \| Reach \| CoreScope`. Detail identifiers are limited to 12 characters; account/reset parameters stay out of titles. - Branding and page context have separate state, so delayed theme loads and edits cannot erase the view or accumulate suffixes. Selection/close paths that use replaceState also update the title. - Validation: all standalone frontend suites; 29 focused unit checks; 16 browser steps covering routes, history, reload, tabs, real row clicks, closing, branding and literal route text. An explicit All time selection keeps the browser test independent of fixture age. - Browser verified: http://127.0.0.1:18853 on an isolated Go fixture. - E2E assertion added: `tests/e2e/test-customize-branding-e2e.js:48`. - Fixed-size work at navigation/selection boundaries; no extra requests, DOM observers, render-loop work, dependencies or settings. Privacy, whitespace, XSS and ESLint checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. Independent review found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40193902834) passed on `4c037db5`, including all three browser shards and both architecture builds. The existing logo test now asserts the full page-and-brand title; its 6/6 checks and the 16/16 title/branding checks also pass locally.	2026-10-26 15:43:30 +01:00
TeTeHackoandClaude Opus 5.5	684f89f200	feat(frontend): serve Leaflet, leaflet.heat and Chart.js from vendor/, not unpkg.com (#2299 ) Refs #1554. `index.html`, `area-map.html` and `geofilter-builder.html` load Leaflet, leaflet.heat and Chart.js from unpkg.com on every page view. That causes three problems: - Privacy: every visitor sends a request to a third party. - Availability: where unpkg is blocked (script blockers, filtered networks) or down, the map and every chart are gone. The observer page then fails with `Error: Chart is not defined`. - #1554: the CSP target there, `script-src 'self'`, blocks these scripts just like the inline handlers the issue lists, but they are missing from its inventory. ## Change - Vendored libraries: `public/vendor/leaflet-1.9.4/` (with `images/`), `vendor/leaflet-heat-0.2.0.js` and `vendor/chart-4.5.1.umd.min.js`. - They are the unpkg dist files byte for byte, verified against the SRI hashes `index.html` already pins. - The two standalone pages loaded Leaflet without SRI; they now carry it. - This is the same pattern as `vendor/leaflet.markercluster.js`. - `.gitattributes`: `public/vendor/** -text`. Otherwise `* text=auto eol=lf` rewrites `leaflet.css`, which ships with CRLF, and the browser refuses it for the integrity mismatch. I hit exactly that while preparing this. - `scripts/instrument-frontend.sh`: now copies `vendor/` recursively. The flat `cp public/vendor/` left the E2E server without `vendor/leaflet-1.9.4/`, which showed up as `L is not defined`. - New test `tests/unit/test-vendored-libs.js`:* - no off-site `<script src>` or stylesheet in the served pages - every `integrity` attribute matches its vendored file - the images Leaflet's stylesheet references exist I checked that it fails against the old `index.html`. ## Checks - `sh test-all.sh` passes. - These E2E suites pass against the fixture: `test-e2e-playwright.js`, `test-analytics-fluid-charts.js`, `test-issue-1236-map-mobile-e2e.js` and `test-issue-1329-map-controls-accordion-e2e.js`. - On a live instance, a headless browser with `unpkg.com`, `jsdelivr` and `cdnjs` blocked: - the observer page renders all four charts - the map loads tiles and markers - `typeof L` and `typeof Chart` are defined - no CDN request is made and no page error occurs ## Not in this PR Swagger UI on `/api/docs` (`cmd/server/openapi.go`) still loads `swagger-ui-dist` from unpkg. I can vendor it in a follow-up if wanted. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-26 15:43:16 +01:00
n30nex	e61ff6e473	fix(observers): keep detail responses tied to the current view (#2323 ) Fixes #2316 and #2320. Related to #1778; progressive loading remains separate. Delayed observer responses could replace a newer observer or period, overwrite its error/status/calibration, and leave detached charts alive. Each load now captures its observer and days and checks ownership before success or failure can update the view. Leaving the page invalidates pending loads, including same-observer revisits. New visits also reset the request window to the selector's existing seven-day default. Previously, a new page could display seven days while requesting the preceding visit's 30-day window. Production change: 12 added lines and three replacements in one frontend file. No new requests, caches, settings or dependencies; the ownership check is O(1). Existing request deduplication, comparison controls, chart cleanup and current-request errors are preserved. Validation: 11 unit checks, all 219 standalone suites, 32 controlled browser cases across desktop/mobile and both themes, comparison 9/9, and observer-IATA 5/5. Core browser run exited successfully with 149 passes and three declared fixture skips; existing conditional subchecks remain. The unchanged backend tree passed both full backend suites. The parent independently reran the new unit/browser tests. Test-only RED `a73c227f` produced [nine unit and 28 browser assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40239824064). Three independent reviews passed. [Final-head CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40240673369) passes on `ba52d873` integrated with upstream `11ce1235`, including all browser shards and architecture builds.	2026-10-26 15:42:56 +01:00
n30nex	9f6f318857	test(rf-health): wait for airtime badges after grid refresh (#2289 ) Fixes #2288. The RF Health airtime test now waits for the requested observer's actual airtime badge. Waiting for any tile and then immediately reading the badge could race with a theme refresh that clears and rebuilds the grid. This is a test-only synchronization correction: two lines replaced, no application changes, and the existing value/warning assertions remain unchanged. No sleeps, timeout increases, skips or retries were added. - Existing RED evidence: [upstream-compatible CI failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190851550/job/121161457287). The test blob is identical on upstream `cb38ac35` and that PR's head. - A controlled local reproduction holds the next summary response during a real theme refresh. The old immediate read fails; the locator read waits for the correct badge. - Four local browser runs passed, including an independent parent run. Wrong-value and missing-warning mutations still fail the original assertion. Test inventory passed. - Browser verified: http://127.0.0.1:18851 on an isolated Go fixture. Privacy and whitespace checks passed. The external home-directory preflight bundle is unavailable; applicable checks were performed locally. Independent reviews found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40193032972) passed on `36d1bf4a`, including all three browser shards and both architecture builds.	2026-10-26 15:40:51 +01:00
efitenandClaude Opus 5.5	e149568c6c	feat(observer-logins): personal MQTT logins for observers (#2306 ) Observers publish to the broker with one shared MQTT login. Anyone who has that login can publish under any observer's topic, and the ingestor takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so the data in the analyzer DB is only as trustworthy as the shared password. This PR gives each user a personal MQTT login, generated and mailed by CoreScope, and lets the broker ask CoreScope whether a login may connect and publish. Closes #2319. The feature is optional and off by default (`userManagement.observerLogins.enabled`). It needs `userManagement.enabled`. With it off, nothing changes. ## Changes in this PR Broker hooks. `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the EMQX HTTP authenticator and authorizer. - They check a shared secret header, `X-CoreScope-Hook-Secret`, in constant time. - A username CoreScope does not know gets `ignore`, so EMQX falls through to its built-in database and existing users keep working. - A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`, with a 64-hex pubkey that is unbound or bound to the same user. - The first publish for an unbound pubkey binds that observer to the user (first come wins), until the user owns 20 observers (admin assignments count too). - An admin can rebind any observer. Credentials. - Username `obs_` + 12 random `[a-z0-9]`, password 32 random `[A-Za-z0-9]`, both from `crypto/rand`. - The password is stored twice: SHA-256 for the auth check, and sealed with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the user can ask for the same credentials again. - The UI and the API never show the password. It leaves the server only in the mail. Mail. One mail with four formats: - the meshcoretomqtt TOML; - the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same `mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3 (Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the bare host: the firmware picks the scheme from the port (`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad (2026-10-19) do not accept a scheme there (read from the source, not tested on a device). A scheme is written only when the port would pick the wrong one, for example TLS on 8884; - the Home Assistant meshcore-ha broker form fields; - the plain values. Each block starts with a numbered title and one line on where it goes. The account page can resend the mail, at most 3 times per 24 h. Account and admin UI. - The account page gets an "Observer MQTT login" section (`#/account?section=observers`) with the username, the bound observers and the resend button. - The admin page gets an "Observer logins" tab (`#/admin?tab=observer-logins`) that lists logins with revoke and reissue, and every observer with its owner, "not migrated" or "orphaned", with a reassign control. Notifications. Creating a login adds `observer.offline.own` ("My observers go offline") to the owner's notification events; mails go out once the user has notifications switched on, and the user can untick it. Storage. users.db schema v7 adds `mqtt_logins` and `observer_bindings`. The server still writes only users.db through `internal/users`, and the analyzer DB stays read-only. Docs. `docs/observer-mqtt-logins.md` covers the EMQX setup (authn and authz chains, the hook body templates, the closing `^obs_` deny rule and why it is needed), how to enable it on a running EMQX through the REST API step by step, with a check and a rollback per step, and how to migrate observers off a shared login. `config.example.json` has the block with placeholder values. ## Enabling it on an instance Full steps in `docs/observer-mqtt-logins.md`. In short: 1. CoreScope: add `userManagement.observerLogins` to `config.json` with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an `encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port, transport and TLS that observers should use. Restart. Keep the key safe: without it, mails cannot be resent and logins must be reissued. 2. EMQX, authorization: add a file source with `{deny, {username, {re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the front). 3. EMQX, authentication: add the HTTP authenticator pointing at `/api/mqtt-hook/emqx/authn` and move it to the front, before the built-in database. 4. Check: the shared login still connects, an unknown `obs_` login is refused, and `metrics.deny` on the authorizer stays 0 for existing clients. 5. Owners: they request a login on their account page and paste the block for their uploader. Each observer binds to them on its first publish. An uploader that was refused before step 3 may need a restart. 6. Later: remove the shared login from the broker once enough observers moved. ## Who can claim an observer Any active account can request a login, and its first publish under an unbound pubkey binds that observer, also when CoreScope already knew it from the shared login (first come wins, a deliberate choice: owners migrate without waiting for an admin). Observer pubkeys are public, and registration cannot be closed, so with this feature on anyone with a mail address can claim up to 20 unbound observers per account. Before, publishing needed the shared password. What limits it: each claim of a known observer mails the admins, an admin can unbind or reassign it and revoke the login, and a bound observer cannot be claimed by another login. The doc spells this out and suggests assigning known observers to their owners before announcing the feature. A stricter mode (admin approval of logins, or no auto-bind for known observers) is not in this PR. ## Performance `authorize()` runs for every publish that misses the EMQX authz cache. It takes two short read locks, does map lookups and one topic split. `BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96 B/op, 2 allocs/op. The admin observers table renders once per load and updates single rows after a change. ## Verification - All 16 Go modules: `bash scripts/allmod.sh test ./...` green. - Skipped `TestSaveGeoFilterPreservesFileMode` and `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on a clean master too. - `sh test-all.sh` green. - `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers: request, mail, resend, revoke, reissue, reassign, no password in the page or API body, and axe. The CI job for this suite gets the `observerLogins` block with fake values in `deploy.yml`. - Local EMQX 5.8.6 in Docker against a local CoreScope: - a personal login connects and publishes, which binds the observer; a second personal login is denied on that observer's topic, and subscribe is denied; - with CoreScope stopped, the built-in users still connect and publish, and new personal-login connects are refused; - the `^obs_` deny rule blocks an already-connected personal login while CoreScope is down; without the rule, the same publish was delivered. - Production: running on an EMQX 5.8.9 broker with about 50 clients since 2026-10-25, set up through the REST API as in the doc. After the hooks went in, the authorizer answered `ignore` for every existing login (0 denies, 0 dropped messages), and 4 observers were bound to their owners through personal logins on the same evening, with the shared login still in use by the rest. ## Known behaviour EMQX's HTTP client can mark the hook resource down for one health-check interval (15 s measured) when CoreScope closes an idle keep-alive connection just as a health check runs. During that window new personal-login connects are refused and the uploader reconnects. I saw it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout, and could not trigger it on demand. EMQX 5.8.6 has no setting for it, and 5.10.5 ships the same client code. The doc describes it and the alarm to watch for. ## Not in this PR - mosquitto (mosquitto-go-auth). EMQX only. - A websocket path field: the mail does not carry one, the doc tells operators to match the listener path. - Disabling the shared login. That is an operator step after migration, described in the doc. - Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only. Note on `cb9f4ff3`: its message says the scheme problem was reported by an operator. That report turned out to come from before the broker used the hook, so it does not show this cause; the change rests on the firmware source only. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-26 09:50:20 +01:00
n30nex	cb38ac356c	fix(packets): preserve All time across reloads and links (#2279 ) Desktop All time now survives reloads and shared links, including a selected older grouped packet and observation. An explicit zero is retained in storage and the URL; missing/invalid values still use the default. Mobile requests retain their existing time and row limits. Fixes #2231. Validation: RED reproduces the lost zero; all standalone frontend suites and the unchanged server suite pass. The local main browser suite (`test-e2e-playwright.js`) passes 149 checks with three explicit fixture-dependent skips. The gesture suite also preserves the existing time window when opening a packet trace. The new regression covers selection, reload, fresh links, older grouped observations and mobile bounds. Normalization is constant work; URL updates reuse the existing serializer and add no requests. No new settings or dependencies. Common ingestor validation is covered by the separate #2272 test-stability change. [Exact-head CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40160982457) passed on commit `5278ade6`, including all three browser shards.	2026-10-25 12:11:20 +01:00
n30nex	82aafb53c3	fix(analytics): count roles within selected filters (#2275 ) Analytics Nodes now counts roles from the same complete, filtered node list used by Network Status. Region/area selections no longer display the API's global role totals. The global API metadata remains available to other consumers. Fixes #2213. Validation: RED reproduces the mismatching totals; all standalone frontend suites and the unchanged server suite pass. Browser checks cover region and area changes and empty results. Regressions also cover 501-node pagination, unknown roles, unchanged metadata and request counts. Counting happens inside the existing O(n) status pass, using six bounded counters and no additional requests. No configuration changes. Common ingestor validation is covered by the separate #2272 test-stability change.	2026-10-25 12:10:36 +01:00
n30nex	7ebbed58dc	fix(analytics): anchor resize handles to table headers (#2274 ) Resize handles now use their static table header as the containing block. Already-positioned headers, including sticky Nodes headers, retain their existing positioning. Fixes #2217. Validation: assertion-failing RED commit followed by the fix; all standalone frontend suites and the unchanged server suite pass. The browser suite passes 48 checks covering desktop/mobile bounds, real dragging, saved widths and sticky headers. The browser regression waits for the initial theme render before measuring. The fix adds one positioning check per header during setup, with no new requests, dependencies or settings. Common ingestor validation is covered by the separate #2272 test-stability change.	2026-10-25 12:10:25 +01:00
efitenandClaude Opus 5.5	2bb000f2aa	feat(notify): adaptive offline threshold and certain node analytics (#2260 ) (#2262 ) Part 2 of #2260. Builds on #2261 (certain liveness): review/merge that first; until then this diff also shows its commit. ## Adaptive offline threshold `userManagement.notifications.adaptiveOffline` — off by default: ```json "notifications": { "enabled": true, "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 } } ``` The fixed `infraSilentHours` (72 h) treats every repeater the same. With this on, a watched node's `node.offline` window is learned from its own certain traffic (the #2261 rule): 1. Certain packets per hour over the week before its last certain packet. Ending there, not at now, keeps an outage from dragging the rate down until the fixed window returns and the node reads "back online". 2. Empty runs are classified: ≤ 6 h and recurring on another day at the same hours = normal silence (kept; the window is never shorter than the longest + 1 h); any other run = outage, left out of the learning so it neither stretches the window nor blocks learning. 3. λ = 10th percentile of the remaining hourly counts, so quiet nights don't alert. 4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson, ln 10⁶ ≈ 13.8). 5. Clamped to [`floorMinutes`, the role's silent window]. Kept on the fixed window: λ < 1/hour, under 24 h of history in memory, or a 10th-percentile hour heard by fewer than `minObservers` observers (one observer dropping out must not read as the node going offline). Silence is measured up to the newest packet in the store, so a stalling feed stops every node's clock. A node stored offline comes back only on evidence from the moment it went offline on. The notifier caches each learned window for 10 minutes and drops unwatched nodes. For the #2260 repeater (~840 packets/day) the mail would have gone out about 30 minutes into the outage, instead of never (or 72 h with #2261 alone). ## Certain node analytics - Each `activityTimeline` bucket carries `certain`: the part that can only be this node (`certainFor`, the packet-level form of the `certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other endpoints are unchanged. - Availability and longest silence use certain buckets only; the longest silence now includes one still running (an outage in progress is the silence worth seeing). - The chart stacks certain (`--accent`) and uncertain (`--text-muted`) — existing theme variables, so no new customizer entry. Descriptions updated. ## Performance `CertainActivity` walks a week of packets per watched node, at most once per node per 10 minutes, with the read lock taken per node: ``` BenchmarkCertainActivity-16 12.1 ms/op (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold) BenchmarkNoteCertainRelays-16 121 ns/op 0 allocs ``` Node analytics adds one `certainFor` per packet of the requested node (a prefix-map lookup, or the cached decoded JSON for its own packets). ## Tests - `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet nights set the rate; > 10 % empty hours, short history, too few observers → fixed. - `CertainActivity` ends at the last certain packet and counts only certain packets — verified red by mutating the window to end at now. - Notifier end to end: learned window passed → bad; inside → good; too few observers / setting off → fixed 72 h; cache refresh and prune. - `certainFor`, `longestSilence` (incl. running silence), config defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket. - Frontend: `NodeAnalyticsTimeline.series` (new `tests/unit/test-node-analytics-certain-timeline.js`). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork. ## Validation Server on the freshened e2e fixture: analytics buckets carry `certain` (e.g. 89 of 90 packets certain for one repeater). No browser screenshot: Playwright isn't installed on this machine. ## Later (rule 8) `floorMinutes` / `minObservers` are server config, not theme settings; an admin UI for notification settings would be the place, not the customizer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-22 21:20:42 +02:00
efitenandClaude Opus 5.5	a08f2f4e7c	fix(liveness): count only evidence that can be the node alone (#2260 ) (#2261 ) Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold follows in a separate PR). ## Problem A repeater silent since 2026-10-18 showed Active — Last relayed 4m ago and Last Heard 36m, and its offline mail never fired. Every hit after that date was the 1-byte hop `67`, shared by nine nodes: - the live resolver's affinity tiebreak (`resolvePathForObs`) put those transmissions in `byNode`, which fed `stats.lastHeard` and `LastHeardMap`; - `last_relayed` folds in the 1-byte prefix bucket on purpose (#662), and drove the status in `roles.js` and the notify offline check. ## Fix The store keeps `certainHeard` per node: the newest transmission that can only be that node. \| Evidence \| Counts? \| \|---\|---\| \| Own packet (`pubKey`, `srcPubKey`) \| ✅ \| \| Flood relay hop whose prefix, at the packet's path hash size, matches exactly one path-capable node \| ✅ \| \| 1-byte hop shared with other repeaters \| ❌ \| \| Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) \| ❌ \| \| TRACE (path bytes are SNR) \| ❌ \| \| Destination (`destPubKey`) \| ❌ \| A multibyte repeater legitimately appears as a 1-byte hop (the originator picks the hash size, `Mesh::routeRecvPacket` appends at that size), so only uniqueness decides. The rule is applied in every load path through `indexResolvedPathHops` and the new `indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live ingest, late observations. Liveness reads only that: - `/api/nodes` and the node detail: `last_heard` is the newer of `last_seen` and the certain last heard. Before, it was a plain copy of `last_seen`. - health / bulk-health `stats.lastHeard`. - `LastHeardMap` → notify offline check. The separate relay input (`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed. - `roles.js getEffectiveHeardMs`: newest known timestamp, without `last_relayed`. The side pane no longer says "Last relayed"; the Last Relayed row stays (analytics) with an updated tooltip. - New status `uncertain` (repeaters and rooms): nothing certain within the silent window, but the shared prefix (`last_relayed`) still in paths. Grey badge with explanation, an Uncertain filter on the nodes list and the map, dimmed like stale. The offline mail for such a node says only its shared prefix was still seen. - A prefix also held by a path-capable node in `inactive_nodes` is not unique; adverts with `signatureValid: false` are no evidence; `/api/nodes/search` reports the same `last_heard`. Attribution for analytics (`byNode`, packet counts, `last_relayed`, relay counters, usefulness) is unchanged. ## Performance Hot path: `noteCertainRelays` runs per observation on ingest under the write lock: two hex bytes of header parsing plus one prefix-map lookup per resolved hop. ``` BenchmarkNoteCertainRelays-16 10307640 116.2 ns/op 0 B/op 0 allocs/op (8-hop 2-byte path, 2,000 repeaters) BenchmarkLastHeardMap-16 1902 0.77 ms/op (5,000 nodes; was 3.7 ms scanning byNode) ``` Memory: one string per node ever certainly heard, dropped together with the node's `byNode` list on eviction. ## Tests - `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk, LoadChunked — each with and without resolved_path — live ingest, late observations) feeds `certainHeard`; each of the 8 call sites verified red with a nil prefix map. - `certain_heard_test.go`: hop width per route/payload type; retired namesake prefix; failed signature; uncertain offline mail; shared vs unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`, detail and health agree. - Regression for #2260: last certain packet on day X, 20 shared-prefix hops after it → `LastHeardMap` stays at X and `node.offline` goes bad after 72 h. Verified red with the uniqueness check mutated out. - Notify: `TestEvaluateCertainHeard` replaces the relay-input test. - Frontend: `last_relayed` alone no longer keeps a repeater or room active; newest timestamp wins (an old `_liveSeen` can't hide a fresh certain relay). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork with 457 extra commits. ## Validation Ran the server on the freshened e2e fixture: list, detail and health report the same last heard; e.g. a repeater whose `last_relayed` is 3 min newer than anything certain keeps its certain time. No browser screenshot: Playwright isn't installed on this machine; CI's E2E covers the pages. ## Not changed - The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a shared hop only when exactly one candidate is adjacent in the neighbor graph; it did not fire in this case. Left as is. - The `lastHeard=` list filter still filters `last_seen` in SQL. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-22 19:11:30 +02:00
AdamandClaude Opus 5.5	dbcdca9dc5	feat(region-filter): honor ?region= in the URL on every page (#2255 ) (#2256 ) Fixes #2255. Only the packets page read `?region=` from the hash, with its own one-off code. Live, nodes, observers, channels, analytics and scope audit ignored it, so a link like this opened with whatever region the visitor last picked: ``` https://<instance>/#/live?region=SJC ``` We link into our instance from a community directory (one link per region), and wanted those links to land on the live map or node list already filtered, not only on packets. ## What changes `public/region-filter.js` - `?region=` (comma-separated, trimmed) overrides the saved selection and is saved like a click in the control, the same as packets did through `setSelected()`. - Fresh load: it's applied when the script runs, before any page reads the selection. This matters for live, which fetches nodes before it calls `RegionFilter.init()`. - In-app navigation: `init()` applies it before its `await`, so a page that loads after `init()` uses it. If it changed the selection, listeners are told once regions load, so a page that already fetched (live) reloads instead of showing the old region's data under the new region's label. - Unknown codes are dropped by the existing cleanup in `fetchRegions()`. No param, or an empty one, leaves the saved selection alone. `public/packets.js` - Drops its own `_pendingUrlRegion` handling; `RegionFilter` covers it. - Removes its region and area listeners in `destroy()`. They were registered without a handle and outlived the page: after visiting packets, a region change on any other page reloaded packets and `replaceState`'d the URL to `#/packets?...` while you were on another page. That already happened with the dropdown on master; the in-app notify above would make it happen on every region link. ## Tests `tests/unit/test-region-filter-url-param.js`, 14 assertions: - URL region overrides the saved selection; comma lists, whitespace, unknown codes, empty param, no param, no hash - fresh load: selection is set before `init()` is called, and `init()` doesn't notify - in-app navigation: a changed region notifies listeners once, the same region doesn't - selection is saved to `localStorage` - packets keeps handles for both listeners and removes them in `destroy()` With `region-filter.js` from master, 5 of the first 8 assertions fail. How I checked: - `test-all.sh` suites all pass except `test-issue-1956-release-routing.js` and `test-preflight-xss-gate.js`. Both also fail on unmodified master on macOS: they shell out to bash-4 features (`mapfile`, and a syntax error in `check-xss-sinks.sh` under bash 3.2). - `eslint@8.57.1` on the two files: 0 errors, 3 warnings, the same as master. - Browser: served `public/` locally with `/api` proxied to a live instance and checked the requests each page made: - fresh load of `#/nodes?region=AUS`, `#/live?region=DFW`, `#/packets?region=HOU`: the first request carries the URL's region and the dropdown shows it - packets → `#/live?region=DFW`: live fetches with the old region, then reloads with DFW; the dropdown shows DFW - packets → `#/nodes?region=AUS`: no packets requests, and the URL stays on `#/nodes` I couldn't run the Playwright E2E suite locally (no Go toolchain on this machine). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-22 19:11:16 +02:00
efitenandClaude Opus 5.5	0e3ce48cff	feat(nodes): flag nodes whose LoRa settings differ from nearby nodes (#1268 ) (#2210 ) Refs #1268. #2209 is merged as `cab4f71c`, and master is merged into this branch (`47ce8264`), so the diff against master is this PR's own change: `ea4d1130` plus the fix in `5cbbe5d5`. ## What it does Two nodes hear each other only when SF and bandwidth match and their frequencies are close enough. `/api/rf-params` now also returns `deviations`: nodes whose setting differs from a majority of the nodes around them on the same band. - Peers: nodes with GPS and RF params within `rfParams.compareRadiusKm` (default 50) whose frequency is within 30 MHz. - Same channel: frequencies within a quarter of the bandwidth count as one channel (15.625 kHz at BW 62.5). The SX1262 tolerates a transmitter/receiver offset of up to 25% of BW without sensitivity loss, for SF5 to SF12 and all bandwidths. Source: the `FERR_L` row in the [S62F module datasheet](https://lora-alliance.org/wp-content/uploads/2020/07/S62F_SPEC_Ver.B.pdf), an SX1262 module; I did not find the figure in a Semtech document itself. That table also lists -50/-100/-200 ppm for SF12/SF11/SF10, which at 869 MHz is wider than BW/4 at 62.5 and 125 kHz, so BW/4 is the tighter bound there. - Flagged when: a setting the node cannot hear is shared by more than half of the group, the node itself counted, with at least 3 peers and no other incompatible setting at the same support. So 5 against 5 flags nobody, and 2 of 4 does not count as a majority. Each deviation carries the majority, how many peers are compatible with it, the peer count, and which of freq/sf/bw differ. - Coding rate is not compared. RadioLib starts SX126x radios with an explicit LoRa header (`SX126x::begin` sets `RADIOLIB_SX126X_LORA_HEADER_EXPLICIT` at `6d89348`, the commit the firmware pins in `platformio.ini`), and nothing in `firmware/src` switches to implicit. The header carries the CR, so a CR mismatch still decodes. I checked SX126x only, not SX127x or LR11x0. UI: - Nodes list: a badge such as `LoRa SF7` next to the name of a deviating node (the majority in its tooltip), and a `LoRa: differs from nearby (N)` filter kept in the URL as `?lora=differs`. - Node detail: one line, for example "Differs from 12 of 14 nearby nodes on this band, which use 869.618 MHz · SF8 · BW 62.5 kHz". - Badge and filter only exist while the server has map data. ## Why same-band only On our instance (1,110 nodes with params and GPS, measured with the first version of the rule), a plain neighbourhood majority flagged exactly 15 nodes, all of them a deliberate 433 MHz network in Limburg next to the 868 MHz one (names like `NL-ZAY-RP01 \| 433 X 868`). The same-band comparison flags none there. On our network this PR shows nothing today. The flags are for instances where a node really is set to a different SF, bandwidth or channel than the nodes around it. ## Cost - Points are bucketed on a radius-sized grid (rows of radius/110.57 degrees, longitude columns widened by 1/cos(lat)) and compared with a squared equirectangular distance. - `BenchmarkRfDeviations1k` (1,100 nodes over Belgium and the southern Netherlands): 4.5 ms. `BenchmarkRfDeviations10k` (10,000 nodes in the same box): 388 ms, 1.3 MB. Both on the first version. - After `5cbbe5d5`, on a slower machine, 3 runs each: 1k 4.4 to 9.5 ms, 10k 821 to 852 ms, 1.8 MB. On that machine the first version measured 7.6 to 9.3 ms and 819 to 861 ms, so the majority step adds no measurable time. It is one pass over the distinct settings around a node, and those are few (one per preset in use). - Inside the radius the work is close to O(n²): 10,000 nodes in one dense box means about 2,000 neighbours per node, and a request after the 5 minute TTL waits for that computation. - The result is cached per map snapshot for 5 minutes; concurrent requests wait for one computation. - On staging `/api/perf` reports `/api/rf-params` at 9.9 ms average, 26 ms max (8 requests). ## Tests - Go: deviation cases (SF, freq within band, BW, CR ignored, other band ignored, no GPS, no params, far away), split vote and too few peers, radius, a peer in another longitude column at 60°N, the cache (same snapshot within TTL, new snapshot, after TTL), the route with four inserted nodes, plus the two benchmarks. - Go, added in `5cbbe5d5`: a 5/5 split and 2 of 4 flag nobody; a 1 and a 15 kHz offset are not deviations while 32 kHz is, and offset peers add up into one majority; the BW/4 boundary; peers two grid columns away at 60°N; a peer at the north edge of the radius. Each fails on the first version. The two-column test also fails with a ±1 column search, which the existing longitude test did not catch. - Full `cmd/server` suite and `test-all.sh` pass locally on `5cbbe5d5`. - Frontend unit: 6 new assertions for `deviationBadge` and the detail line (13 in the file). - E2E: 4 new checks: badge only on the deviating node, filter count and result, `?lora=differs` written and restored, detail line. The disabled check now also asserts no filter and no badges. 8/8 locally. ## Validation On staging with live map data, with the first version: 1,115 nodes served, `deviations: {}`, and the filter shows `LoRa: differs from nearby (0)` with no badges. Because no real deviation exists there, the badge, filter and detail line were checked by serving one simulated deviation through Playwright route interception against the staging page. ## Not done - No map marker for deviating nodes. - `compareRadiusKm` is config-only; the customizer is a later step. The other values are hardcoded and belong in that step too: at least 3 peers, the 30 MHz band, the more-than-half rule, BW/4 and the 5 minute TTL. - Not re-run on staging after `5cbbe5d5`, and the E2E suite was not re-run locally. The fix changes only the server's choice of nodes, not the UI. - The detail line says the peers "use" the majority setting. With the BW/4 tolerance, some of them may be a few kHz off it. - Companions and room servers are compared too, not only repeaters. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-21 15:18:14 +02:00
efitenandClaude Opus 5.5	dc98886d92	feat(nodes): name and show repeaters known only from the name they reported (#2238 ) (#2239 ) Part of #2238. Path hops through such a repeater (`/api/nodes`, `/api/resolve-hops`, the frontend hop resolver) still show a bare prefix; that part of the issue stays open. ## Summary A repeater that never sends an advert can still tell a nearby companion its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`. This PR stores those answers, resolves names with them, and shows these repeaters on the map, in the nodes list and in search, marked as self-reported. ### Ingestor: store the names - `367ecde1` adds a new `names` subtopic. - It is gated on `clientNodeNames.enabled`, default off. - Answers go into a new table, `client_node_names`: one row per observation, `UNIQUE(target, rx_pubkey, observed_at)`. - Validation. - `target` must be 64 hex. - The name is trimmed and rejected when it is empty, over 31 runes (firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8, or contains a control character. - Retention. `retention.clientNodeNamesDays` (0 = keep). `config.example.json` sets 30, like the other client tables: a companion only asks a repeater for its name when CoreScope has none (`nameMissing` in CoreDrive's `src/names.js`), so pruning is also how a renamed repeater gets its new name. - Compatibility. An ingestor without this change drops `/names` silently, so CoreDrive v1.20.0 works against either version. ### Server: resolve by the reported name - `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and `client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys` (the coverage tooltips) both use it. - Which name wins. The advert name, otherwise the newest reported name by `observed_at`. A prefix that matches different pubkeys across the two tables stays ambiguous. - Only repeaters heard directly. A reported name counts only once its target has at least one `discover` or `advert` reception in `client_receptions`, keyed by the full pubkey, or by its 8-byte prefix when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names` starts with that prefix (`clientNameHeard`). A name for a pubkey nobody heard adds no candidate, so it cannot make a real node's prefix ambiguous. Without the uniqueness check, a made-up pubkey that shares a heard repeater's first 8 bytes borrowed that repeater's receptions and passed (fixed in `4c569f91`). - Filters. The blacklist and the hidden name prefixes apply to both sources. - Missing table. A database without the table resolves from `nodes` alone, and the server checks for the table again until the ingestor has created it. - Response shape. `/api/nodes/resolve` keeps `{prefix, pubkey?, name?, ambiguous}`. ### Server: estimated position - `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`. It lists every repeater in `client_node_names` that never sent an advert (no row in `nodes` or `inactive_nodes`) and was heard directly at least once (see above). A node with an advert but no position stays out: it is already in `/api/nodes` under its advert name. - Gated like the other endpoints on `client_receptions`: 404 when `clientRxCoverage.enabled` is off, because the counts and the estimate come from companion GPS fixes. - Which receptions count. Only direct ones: - `discover` and `advert` rows, keyed by the full pubkey, or by its 8-byte prefix when no other known pubkey shares it; - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone. `geo` rows are left out. - How the estimate is made. - Receptions within 100 m form one spot, with the best SNR of that spot. - An estimate needs at least 3 spots, at least 200 m apart. - The position is the SNR-weighted centroid (`10^(snr/20)`). The radius is the distance to the farthest spot, at least 500 m. - The response is cached for 5 minutes, and the compute time is logged. ### Map - `5fcde554` adds `public/map-estimated.js`, which draws each estimate on its own layer: - a hollow, dashed "≈" marker and a dashed uncertainty circle; - a popup with an "Estimated position" badge, the method, the caveat that one-sided driving skews the estimate, and a link to the coverage map; - a Display toggle, "Estimated positions": on by default, kept in localStorage, and also hidden by the repeater role filter and in route view. ### Nodes page and search - `12e6b345` adds a "Self-reported repeaters" section under the nodes table, on the All and Repeaters tabs, filtered by the search box. - `#/nodes/<pubkey>`. Where `/api/nodes/<pubkey>` answers 404, a self-reported detail view replaces "Node not found": - who reported the name; - the receptions; - the estimate, or why there is none; - links to the map and the coverage map. An unknown key still shows "Node not found". - Header search adds matching self-reported repeaters, tagged "self-reported". - Unchanged on purpose. `/api/nodes` and `/api/nodes/search` stay as they are. These repeaters have no packets, so the packet filters, the live view and the channel sender lookup must not match them. ### Docs - `03580437` corrects the `/regions` section of `docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the switch, but the code uses `clientRegions.enabled`. - The new subtopic, endpoint and config keys are documented in the same file. ## Trust Any companion the broker lets publish can report a name for any repeater, and the newest report wins. This is the same model as `/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to linked companions. A report for a repeater nobody heard directly is stored but not used: it is not listed and adds no prefix candidate. A companion can still report a wrong name for a repeater that was heard. The same companion can also publish a fake `discover` reception for a made-up pubkey, so the "heard directly" check filters junk, not a malicious companion; `requireLinkedCompanion` is the control for that (its gate runs before the `/names` handler too). ## Performance - `lookupPrefixes` adds one term per key on `client_node_names`. - It matches the prefix as a range, `target >= ? AND target < ?` (upper bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX idx_cnn_target (target>? AND target<?)`, and the reception check is `SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`. - It is still one query per chunk of 200 keys: 400 compound terms, under SQLite's default limit of 500. - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks. - `/api/nodes/reported` makes three kinds of query: - one grouped query on `client_node_names`; - one `lookupPrefixes` call for the short hashes; - one `IN` query per 400 keys on `client_receptions`, served by `idx_client_recept_heard_geo`. The spread check is O(spots²) per repeater, with at most one spot per 100 m visited. The result is cached for 5 minutes; concurrent misses wait for one computation, and the response is written after the lock is released. - The map does not wait for it. It draws its markers first and adds the estimates when the list arrives. The map and the nodes page share one cached fetch (`ReportedNodes.load()`). - Not benchmarked. On the instance where this runs, `client_node_names` holds one row today. I expect at most tens of such repeaters per instance, but have not measured that. ## Notes - `map[string]interface{}`. `cmd/ingestor/client_names.go` has 2 occurrences: the handler's `msg` parameter and its `gps` field. That is the decoded MQTT message, passed in the same shape as `handleClientPacket` and `handleClientRegions` already take it. No API response or internal structure uses it. - Thresholds are hardcoded: 100 m, 3 spots / 200 m, 500 m, 5 min TTL. They should move to the customizer later (rule 8). - Map filters. The estimates follow the "Estimated positions" toggle and the repeater role filter, not the area, region or last-heard filters. The toggle's tooltip says so. - Name characters. Names are rejected for control characters but not for Unicode format characters (bidi overrides, zero-width joiners), because emoji sequences in node names use the zero-width joiner. All output goes through `escapeHtml`. - Not done: the home-page search box (`home.js`, `/api/nodes/search` with a Claim button) does not list these repeaters. Claiming a node without adverts would have nothing to show. ## Tests On this branch (based on `550829af`): - `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and `cmd/ingestor`. - New Go tests: - handler and validation cases; - idempotent redelivery; - newest `observed_at` wins; - prune; - the gate on and off, without falling through to the observer path; - client-only node, nameless `nodes` row, advert name wins, ambiguity across the two sources; - hidden and blacklisted nodes; - absent-then-created table; - spot merging, threshold, weighted centroid and radius floor; - `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an advert position and a missing table; - an advert without a position, a never-heard name, a name heard only in `rxlog` paths, and the `clientRxCoverage` gate (404). These fail on `12e6b345` and pass at head. - a made-up pubkey sharing a heard repeater's 8-byte prefix, and two reported targets sharing one (`4c569f91`). Both fail without that commit. - New frontend unit suites: `tests/unit/test-map-estimated.js` and `tests/unit/test-reported-nodes.js` pass. Both load the real `escapeHtml` from `app.js`. - `test-all.sh` passes, all suites (with Python available for the XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the four changed frontend files. - E2E, against a static server with a mocked API: - `test-map-estimated-e2e.js`: 6/6, including the repeater role filter and route view hiding the estimates. Mutating either hide makes its step fail. - `test-reported-nodes-e2e.js`: 6/6. - `test-issue-1150-404-state-e2e.js`: 5/5. - Real data, API only. On the fork's staging, `/api/nodes/reported` lists one repeater: 5 spots, 17 receptions, an estimate with a radius of 2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays listed under the stricter rules. Not re-checked after `4c569f91`. - Not done: a check of that repeater in a browser. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-21 15:17:58 +02:00
nullrouten0 nullrouten0 Claude Opus 5.5	9534e1c25e	fix(reach): retry a busy (429) reach report in the reach page (#2250 ) Follow-up to #2127: @dborup's point 2 there, and @efiten's non-blocking note on the approval ("worth checking the node page's reach panel shows a retry hint on 429 rather than a generic error"). ## What changes - The reach page retries a 429 itself. It shows "Reach is busy computing other reports, retrying in N s", waits `Retry-After` seconds, and tries again. - When the header is missing or is not a number of seconds (an HTTP-date, 0, a negative number), it waits 2 s. It never waits more than 30 s. - After 5 retries it shows the old error ("Failed to load reach: API 429 …"). - Leaving the page stops the retries, both while it waits and while a request is still in flight. Picking another window while a request is in flight also drops that request's retries. - `api()` does not retry 429. Login, signup, activation, email change, companion linking, settings PUT and channel proposals also answer 429 from their own limits, and those must not auto-retry. `api()` now puts `status` and `retryAfter` on the errors it throws, so a caller can decide; the message text is unchanged. A unit test pins that `api()` makes exactly one request on a 429. - Fix in `api()`: the `_inflight` cleanup chain (`promise.finally(...)`) rejected unhandled on every failed call, so each API error also logged "Uncaught (in promise)". That would have added up to 5 more per busy reach load. The chain now catches; callers still get the rejection through the returned promise. This works with the server as it is on master (instant 429, `Retry-After: 5`), and with the bounded wait in #2249 (429 after 3 s, `Retry-After: 2`). Neither PR depends on the other. ## What I checked - Suites. `sh test-all.sh` passes, the XSS sink gate passes, and eslint has 0 errors and no new warnings. - Unit tests (`test-frontend-helpers.js`): - `reachRetryDelayMs`: Retry-After values; fallback on missing, zero, negative or HTTP-date values; the 30 s cap; giving up after 5 retries; no retry on 400, 404, 500, 503 or network errors. - `api()`: a 429 error carries `status` and `retryAfter`, makes exactly one request, and leaves no unhandled rejection. - The block sits above the file's `Promise.allSettled(pendingTests)` summary. Async tests registered below that line are never awaited, so they cannot fail the run; nothing below it is async today. - E2E (`test-node-reach-e2e.js`, already in CI suite 2): reach requests are answered through `page.route`, so the server does not have to be made busy. It checks: - two 429s, then the real report: the hint shows "retrying in 1 s", the retries wait (at least 1.8 s between the first and third request), and no error is shown; - always 429: 6 requests (1 + 5 retries), then the error; - leaving while it waits: no more requests; - leaving while the first request is still in flight: no hint written, no retry; - a 500 with Retry-After: one request, no retry. On master's frontend the new section fails at the first busy hint. Locally against the freshened `e2e-fixture.db`, the whole file takes about 17 s (2 runs). - Mutations. I wrote 15 from the code, and the tests caught all 15. They include: `api()` drops `status` or `retryAfter`, `api()` retries 429 itself, retry any error, retry limit off by one, fallback or cap changed, Retry-After 0 accepted, either "left the page" check removed, the wait not honoured, the hint showing ms, the retry count ignored, no error after giving up, and the cleanup chain unhandled again. Each mutation was checked against the file the server actually served before the tests ran. - Real browser (Chromium via Playwright, on a copy of a 10.5M-observation database): - Current master's server, 6 tabs opened at once on uncached 14-day reports: 4 got 429 with `Retry-After: 5` and showed the hint, and all 6 reports rendered within 11.5 s (one tab retried twice). Without this change those 4 tabs show "Failed to load reach: API 429". - With #2249's server: 4 of 6 got 429 with `Retry-After: 2`, and all rendered within 8.1 s. - The only console errors were Chrome's own "Failed to load resource" lines for the 429s; there was no "Uncaught (in promise)". ## Not checked - Mobile layout. The hint reuses the existing `.nq-load` style, so I did not take a screenshot at phone width. - Other browsers. Firefox and Safari were not tried; the code is plain `setTimeout` / `await`. - Window switches. While the hint is showing, the time-range buttons are not on screen; the same is true of the existing error and loading states. Picking a new window means waiting, or going back and opening reach again. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: nullrouten0 <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-21 11:58:24 +02:00
efitenandClaude Opus 5.5	d43f3c3777	feat(theme): ?theme=dark\|light URL parameter for one page load (#2236 ) (#2251 ) Fixes #2236. A display that can only be given a URL (digital signage, an iframe without cookies or login) had no way to choose dark or light. This adds `theme=dark` or `theme=light` to the hash query, next to `embed=1`: ``` https://<instance>/#/map?embed=1&theme=dark ``` ## What changes `public/index.html`, the pre-paint script - Reads `theme` from the hash query with `URLSearchParams`, as `shouldEmbedRoute` does for `embed`. Only `dark` and `light` count; anything else is ignored. - A valid value wins over the saved choice and the OS preference, and is set before first paint, so there is no light flash. - The value is handed to `app.js` as `window.__themeOverride`. - When `localStorage` throws (a sandboxed iframe), the URL value is still applied. `public/app.js`, the dark-mode setup - On load, a URL value is applied through `applyTheme(theme, false)`, which does not write `meshcore-theme`. The saved choice, and the account settings that `settings-sync.js` syncs from that key, stay as they were. - Theme changes from other tabs or from settings sync (the `storage` listener) are ignored while the URL value holds. Otherwise a logged-in display would flip within a minute. - A click on the dark-mode toggle ends the URL value and is saved as before. It works on every page, not only on the two `embed=1` pages. The value is read once per page load, so later hash changes (navigation, map viewport) keep it. ## Tests \| File \| Assertions \| Covers \| \|---\|---\|---\| \| `tests/unit/test-theme-url-param-2236.js` \| 8 \| Runs the real pre-paint script from `index.html`: `dark` and `light` win over a saved choice and the OS, nothing is written to `localStorage`, any page and position in the query, an unknown value, no param, `localStorage` throwing \| \| `tests/e2e/test-theme-url-param-2236-e2e.js` \| 6 steps \| `#/map?embed=1&theme=dark` over a saved "light" (dark, embed, saved still "light"), a reload without the param, `theme=light` over a saved "dark", a storage event while the override holds, the toggle ending it, `theme=blue` \| How I checked them: - With `index.html` and `app.js` from master, 6 of the 8 unit assertions fail, and the E2E fails at step 1 ("expected dark + embed + saved light"). - With only the `app.js` change reverted, the E2E fails at step 1. With only the `if (themeOverride) return;` line in the `storage` listener removed, it fails at step 4. - `sh test-all.sh` passes, except `test-issue-1956-release-routing.js`, which I skipped: it needs the workflow's `GITHUB_REF_NAME` and does not run in local git-bash. `npx eslint public/app.js` reports 0 errors (18 warnings, the same count as master). - E2E on a fixture seeded as in CI: \| Suite \| Result \| \|---\|---\| \| `test-e2e-playwright.js` \| 148 passed, 3 skipped \| \| `test-customize-theme-e2e.js` \| 8/8 \| \| the new #2236 suite \| 6/6 \| - `test-logo-theme-e2e.js` fails locally with "wordmark `<text>` overflows SVG viewBox", with the same numbers on master. It measures font metrics, which differ on this Windows machine. The new E2E is registered in `scripts/non-unit-tests.json` and `deploy.yml` (shard 2, next to the logo theme suites); the unit test is in `test-all.sh`. ## Not done - No screenshot: the E2E checks `data-theme` in headless Chromium, not how the page looks. - `live.js` Matrix mode forces dark and, when switched off, saves the theme it restores. With a URL value active, turning Matrix off saves that value as the stored choice. I left this as is, because a display that only shows a URL does not toggle Matrix mode. - Only `theme=`; the `?darkMode=1` form from the issue text is not supported, so there is one form to document. - I placed the E2E in shard 2 without measuring suite times. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 11:40:25 +02:00
efitenandClaude Opus 5.5	3960fadd96	fix(pages): drop late responses that belong to an earlier page visit (#2218 , #2230 , #2200 ) (#2248 ) Fixes #2218, fixes #2230, fixes #2200. Node analytics, the account page and analytics await their data, then write into the shared `#app` container. A response that lands after the user has navigated away replaces the new page. In #2218's reproduction, the Nodes list, with 200 rows, became an Analytics heading. ## The shared part `public/app.js` keeps one view generation, `viewGeneration()`: - `navigate()` bumps it before the old page's `destroy()`, so every page init counts as a new visit, including a switch to another node of the same page type. - A page captures the value at init, and after each `await` drops the result when the value has moved on. `.eslintrc.json` declares it as a global. ## Per page #2218, `node-analytics.js` - `loadAnalytics` and `loadBatteryChart` check the generation on both success and error. A page-local `loadSeq` makes a time-range click a new load on the same visit, so an older range can't land over a newer one. - `NodeHopAnalytics.load` already checked `container.isConnected`, so it is unchanged. #2230, `account.js` - The render after `CSAuth.ready()` runs only while the generation is unchanged. - `destroy()` was empty. It now clears `shown.app` and `shown.userId`. #2200, `analytics.js` - A page-local `_loadSeq` sits next to the generation. Results and errors of an older window/region selection or an earlier visit are dropped, so a late failure of an old load no longer shows an error over newer results. - The bulk load renders only the six tabs drawn from its data. A tab that loads its own data (Prefix Tool, Nodes, Roles, ...) is no longer rebuilt or replaced by the bulk result: a deep-linked one renders at init, and a window, region or area change re-renders it directly. - `renderTab` guards the six tabs drawn from the bulk data (overview, rf, topology, channels, hashsizes, collisions). Until the data arrives they show "Loading analytics…", or the current load's error. Overview no longer dereferences an undefined `rf`. - Three related fixes in the same lines: - The error message is now escaped with `esc()`; it was inserted raw. - `destroy()` removes the RegionFilter and AreaFilter listeners. They piled up on every visit and called `loadAnalytics` from other pages. - `destroy()` resets the stored error. ## Tests \| File \| Assertions \| Covers \| \|---\|---\|---\| \| `tests/unit/test-issue-2218-node-analytics-stale-render.js` \| 12 \| Loads the real router: the generation rises on every init, including another node. Late success and error after leaving, a stale visit of the same node, an older range landing last, late battery success and error. The current response, battery and error still render \| \| `tests/unit/test-issue-2230-account-stale-render.js` \| 6 \| Leaving before ready (login view and "not enabled"), re-entering before ready, the current render, an auth change after `destroy()` \| \| `tests/unit/test-issue-2200-analytics-stale-load.js` \| 19 \| An older selection landing last, a late rejection of an older selection, leave and re-enter (success and error), the current error rendered escaped and kept on a tab click, Overview and RF clicked before the data \| \| `tests/e2e/test-issue-2218-node-analytics-stale-e2e.js` \| 4 steps \| Holds `/api/nodes//analytics` with `page.route`, returns to `#/nodes`, then releases the response as a 200 and as a 500. The 200 rows stay and no Analytics heading appears; a normal visit still renders \| How I checked them: - With the production files reverted and the tests kept, 7 of 12, 4 of 6 and 7 of 12 unit assertions fail. With the fix, all pass. - The E2E fails on the unfixed frontend with "Nodes rows 200 before release, 0 after". - Single mutations, each reverted: - removing the success check in `loadAnalytics` (analytics) fails 2 assertions; - removing the check after `CSAuth.ready()` fails 3. Other test changes: - `test-analytics-theme-refresh-before-data.js` and `test-user-management-ui.js` get a `viewGeneration: () => 0` stub. - The new E2E is registered in `scripts/non-unit-tests.json` and `deploy.yml` (shard 2, next to the #748 suite). The unit tests are in `test-all.sh`. - The broad suite's "Node side panel Details link navigates" test selected an anchor that now opens Analytics, as #2218 notes. It now clicks `#nodesRight .node-detail-btn`. Results: - `sh test-all.sh` passes, and `npx eslint public/.js` reports 0 errors. - E2E on a fixture seeded as in CI: \| Suite \| Result \| \|---\|---\| \| `test-e2e-playwright.js` \| 148 passed, 3 skipped \| \| `test-user-management-e2e.js` \| 28/28 \| \| the new #2218 suite \| 4/4 \| \| `test-issue-748-node-analytics-coverage-e2e.js` \| 2/2 \| \| `test-analytics-fluid-charts.js` \| 8/8 \| \| `test-issue-1657-analytics-channels-group-sprites-e2e.js` \| 5/5 \| ## Not done - Two tabs that each fetch their own data can still overwrite each other after a fast tab switch between them. #2200 does not ask for that. - `nodes.js` `navigateToNode` calls its own `init` without the router, so it does not bump the generation. None of the three pages fixed here depends on that. - I placed the E2E in shard 2 without measuring suite times, which the `deploy.yml` comment asks for. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 10:39:15 +02:00
CoderNemesisandClaude Opus 5.5	e55c95db24	1-Byte Hop Path Inference for Packet and Map Analysis (#2245 ) ## Summary On meshes with 1-byte path hashes, most hop prefixes are shared by several repeaters. The packet detail pane currently resolves each hop on its own and stores the result by prefix. Two things go wrong as a result: - A prefix that appears twice in one path resolves to the same node both times. - Nothing checks that the chosen hops form a path a packet could actually take. This PR adds `HopResolver.inferPath()`, which infers the most likely node behind every hop of one observation's path, scoring the path as a whole and anchoring it on the observer that heard it. The packet detail pane and the map route view now use it, so both show the same nodes for the selected observation (`?obs=`). ## The problem, on a real packet Packet `57da37d1636f895a`, observation `1388126`: a 46-hop flood with 1-byte hashes. The server's `resolved_path` resolved 1 hop of 46, so the client resolver did the rest. Eight prefixes appear twice in the path. \| \| Current resolver \| `inferPath()` \| \|---\|---\|---\| \| Hops longer than 120 km \| 13 (21 longer than 60 km, two over 950 km) \| 0 \| \| Distance from the last hop to the observer \| 59 km \| 0.1 km \| \| Hops reported as unlisted (no listed node fits) \| 0, every hop guessed \| 13 \| \| Agreement on the shared path prefix across the packet's 23 observations \| — \| 97.8% \| The current resolver picks badly for three reasons: 1. Results are keyed by prefix. Hop 0 and hop 45 share a prefix, so both show the node near the start of the path, 59 km from the observer. 2. Picks are greedy. Each hop looks only at its nearest resolved neighbours, so nothing penalises a 1,100 km jump. 3. The detail pane read the wrong `resolved_path`. It used the packet row's, not the selected observation's, and when the lengths matched it never fell back for the hops that were `null`. ## Firmware basis The model relies on two facts from `firmware/src/Mesh.cpp`: - The last hop is the one the observer heard. `Mesh::routeRecvPacket` appends the relaying node's own hash to the end of a flood path. - A node appears at most once in a flood path. A node forwards a given packet once: `_tables->wasSeen()` / `markSeen()`. ## How the model works Each hop is either one of the located repeaters or room servers sharing its prefix, or unlisted: a node missing from the node list, or one without GPS. With 1-byte prefixes, unlisted is common, and naming it beats drawing a 300 km hop. \| Term \| Cost \| \|---\|---\| \| A hop between two located nodes \| distance / 50 km; past 100 km per hop, the cost rises 4× faster \| \| A hop with a neighbour-graph link \| the distance cost × 0.5 \| \| An unlisted hop \| 1.6, flat (about one 80 km hop) \| \| The last hop to the observer \| the same distance cost; falls back to the observer's IATA region centre (with 50 km slack) when the observer reports no GPS or 0,0 \| \| The first hop to the sender \| the same distance cost, when the sender's position is known (e.g. an ADVERT) \| - Search: dynamic programming over (hop, candidate) finds the cheapest sequence. When the best path uses a node twice, the node is banned from whichever position costs less to give up. - Confidence: each hop's confidence comes from the min-marginal costs of its alternatives, put through a softmax. A node the path already uses at another hop is not counted as an alternative. - Server hint: the observation's own `resolved_path` restricts a hop to the named node, but that hop can still be reported unlisted if the node doesn't fit. The tuning values live in `PATH_MODEL` in `hop-resolver.js`, with a TODO to expose them in the customizer (AGENTS rule 8). ## What changes in the UI Packet detail pane (`packets.js`): - The Path row, the cross-observer aggregate row and the byte breakdown render hops by position, from the inferred entries. - The detail pane no longer writes into the list's shared per-prefix cache (`hopNameCache`), so it can't change what the packets list shows. Hop display (`hop-display.js`): \| Hop state \| Display \| \|---\|---\| \| Confident (≥ 70%) \| The node's name; the tooltip says `NN% likely` and how many nodes share the prefix \| \| A guess (< 70%) \| Dashed underline, plus a candidate badge when there are alternatives \| \| Unlisted \| The hex prefix in muted text; the badge lists the rejected candidates \| \| The only known node, without GPS \| The name, with `(no GPS)` \| The candidate popover lists each candidate with its likelihood, or `hop N` when the path uses that node elsewhere. The "Path symbols" legend gains an entry for unlisted hops. Map route view (`map.js`, `route-view.js`): - The `#/map?packet=…&obs=…` deep link infers the chosen observation's hops with the same shared helper, so the map and the detail pane agree. - Unlisted hops render as `unlisted` placeholders rather than as the nearest node that happens to share the prefix. - When inferred hops are available, the `/api/resolve-hops` request is skipped. Shared helpers (`packet-helpers.js`): - `inferObservationPath(obs, observerPos, originPos)` - `ensureHopAffinity()`: loads the neighbour graph once per page, cached by `api()` for 5 minutes. - `routeHopFromInferred(hop, entry, role)` Cleanup: removed dead hop resolution from the "View route on map" button handler. It computed hops, sender and recipient, then navigated by URL and discarded them. Unchanged: the packets list, live view, node pages and `HopResolver.resolve()`. ## Performance AGENTS rule 0 asks for proof of performance claims; the figures follow. - Complexity: O(hops × 8 × candidates²). Hops more than 8 positions apart aren't compared directly; a longer run of unlisted hops costs a flat amount, tracked with a running minimum. Hop-to-hop costs are computed once and reused by every pass. - Where it runs: once per detail render (plus once for the aggregate row) and once per map deep link. Nothing runs in the list, live feed or WebSocket paths. - Measured (Node 22, warm JIT): \| Case \| Time \| \|---\|---\| \| 64 hops × 16 candidates, synthetic worst case \| 9–14 ms median \| \| Real 39–48 hop paths, 1,153 nodes (23 observations) \| 1–12 ms each \| - Enforced: `test-hop-path-inference.js` asserts a median under 40 ms (target 20 ms) for the 64 × 16 case. It times the module outside Node's `vm` sandbox, because the sandbox makes global lookups about 10× slower than a browser. - Network: one extra bulk request per page life (`/api/analytics/neighbor-graph`, cached server-side, TTL 5 min). No per-hop or per-packet requests. ## Tests New: - `tests/unit/test-hop-path-inference.js` (37 assertions). The model itself: - a repeated prefix resolves per position, and no node appears twice; - the observer anchors the last hop, with the IATA fallback when the observer has no position or reports 0,0; - a far-away lone candidate is reported unlisted; - a neighbour-graph link breaks a tie; - the server hint and the origin anchor are applied; - a node without GPS is handled; - output shape, and the performance gate. - `tests/unit/test-hop-path-detail.js` (32 assertions). Rendering and wiring: - inferred-hop rendering and the popover; - positional `renderPath`, including with the 1-byte hop filter on; - `renderDetail` infers the shown observation from its own `resolved_path` and leaves the list cache alone; - the neighbour graph is fetched once; - the shared map helpers, and source checks on the map wiring. - `tests/e2e/test-hop-path-inference-e2e.js` (6 steps, shard 1). Runs on fixture packet `57224a4a01c9c682`, observation 147, a 22-hop path that repeats one prefix three times: - every hop is inferred; - the repeats resolve per position; - the popover shows likelihoods; - the map's route list matches the detail pane hop for hop. Updated, because they pinned the old behaviour: - `test-issue-1418-deeplink-hops-channels.js`: it tested a copy of the old map priority logic. It now tests the real `inferObservationPath()`, including `resolved_path` as a hint. - `test-issue-2098-last-hop-region.js`: the detail pane no longer writes the prefix cache. The test now reads the region-anchored last hop from the rendered Path row. - `test-issue-2097-hop-ambiguity-badge.js`: the source check allows the new `entries` option while still asserting that the detail pane never uses the summarised form. - `.eslintrc.json`: declares the three new shared helpers as globals. Results on this branch: - `sh test-all.sh`: all standalone frontend suites pass. - `npx eslint public/.js`: 0 errors. All 85 warnings are pre-existing; none come from code this PR adds. - The "Run Playwright E2E tests (fail-fast)" step was run locally with its script taken verbatim from `deploy.yml` (`bash -eo pipefail`, fixture prepared as in CI): - shard 1: 27 of 27 passed; - shard 2: 47 of 47 passed; - shard 3: 50 of 50 passed. Note for reviewers: `test-issue-1692-packets-init-parallel-e2e.js` only passes while the freshened fixture's newest packets are less than 15 minutes old. The test sets the time window in `localStorage` after the page has already read it. CI meets that condition because shards start right after freshening. Unmodified `master` fails the same way when run late. Browser validation:* the local server ran on the fixture DB. Checked in the browser: - the Path row, the candidate popover and the byte breakdown; - the map route view, which matched the detail pane hop for hop with unlisted placeholders; - the console showed no errors. ## Known limitations and follow-ups - TRACE packets are not modelled yet. A TRACE's path is its planned route. The header path byte counts the hops actually forwarded (repeaters append SNR, not their hash, `Mesh.cpp`), and a trace may revisit a node on its way back (see the comment in `Packet::calculatePacketHash`). A follow-up will add a `heardAfter` / `allowRepeats` option, so a trace is anchored after the hops it actually completed and may repeat nodes. - Customizer (AGENTS rule 8): the `PATH_MODEL` tuning values are hardcoded for now. The unlisted cost, typical and maximum hop length, and the confidence threshold should become customizer settings. - Server-side `resolved_path`: the ingestor writes a hop as resolved whenever its prefix is unique in the node table, even when that node is hundreds of km from the neighbouring hops. `inferPath()` treats these as hints and can overrule them, but the stored data is still misleading. - The first hop without a sender position is often a close call. On encrypted channel messages, hop 0 is constrained only by what follows it, so 55/45-style splits between a long chain of known nodes and "unlisted" are expected. The badge and the percentage show this rather than hide it. ## Screenshots This is what a normal corescope instance would show for a 1 byte path. <img width="318" height="338" alt="image" src="https://github.com/user-attachments/assets/4056bc27-0b37-4107-9b25-31ac0f7864ec" /> <img width="430" height="601" alt="image" src="https://github.com/user-attachments/assets/01da5ee8-babf-4e41-a9ab-836b5d640549" /> Now the same packet but with the logic to infer each hope along the path different nodes are chosen as they're ranked and selected based on proximity to the next hop, <img width="436" height="646" alt="image" src="https://github.com/user-attachments/assets/bcca73b8-daf0-4031-8ea1-8c96d716c8b3" /> <img width="436" height="646" alt="image" src="https://github.com/user-attachments/assets/fc926d9e-158c-48ce-bee6-93086afb0775" /> The higher the confidence level then there's no need to show the exclamation mark to show that multiple nodes share the same prefix but it does continue to share that Examples of it currently running can be found at, https://corescope.baymc.org - modified instance https://analyzer.comchan.net - unmodified instance https://analyzer-stage.comchan.net - modified instance --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-19 20:00:07 +02:00
efitenandClaude Opus 5.5	2d09335cfb	fix(packets): keep the node filter pick on a slow click or late search (#2243 ) (#2244 ) Fixes #2243. ## Problem The packets node filter selected a suggestion on `click`, while the input's `blur` handler hides the dropdown 200 ms after focus leaves. A press held longer than that, or a search response that rebuilt the list between mousedown and mouseup, lost the pick: the typed text stayed and no `node=` filter was applied. ## Change - `public/packets.js`: suggestions select on `mousedown` with `preventDefault`, so the input keeps focus and the pick happens before any rebuild. `live.js` already does this for its node filter. - `public/packets.js`: a sequence counter drops search responses for an older query, bumped on every search and on a pick, so a late response can't replace or reopen the list. - `tests/e2e/test-packets-node-filter-e2e.js` (new), registered in `scripts/non-unit-tests.json` and on shard 1 in `deploy.yml`. ## Tests The new E2E has three cases: a normal click, a 400 ms press, and a search response released mid-click. \| Code \| Normal click \| 400 ms press \| Response mid-click \| \|---\|---\|---\|---\| \| master \| pass \| fail \| fail \| \| this PR without the sequence guard \| pass \| pass \| fail (list reopens) \| \| this PR \| pass \| pass \| pass \| Run locally against the freshened e2e fixture. `sh test-all.sh` passes (locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and `PYTHONUTF8=1`, unrelated to this change). ## Not done - Full `test-e2e-playwright.js` not run locally. - Could not reproduce on the reporter's instance with the browser tool, whose clicks are shorter than 200 ms. The 400 ms Playwright press reproduces it on the fixture. - `live.js` and `packets.js` still carry two separate node-autocomplete implementations. Merging them is out of scope here. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 21:30:57 +02:00
efitenandClaude Opus 5.5	a910a3e480	fix(packets): re-fetch when the observer filter changes (#2241 ) (#2242 ) Fixes #2241. ## Problem In grouped mode, ticking an observer on `#/packets` updated the URL and the trigger label but left the table unchanged until a reload. Since #1821 `applyObserverFilter` leaves grouped rows to the server's EXISTS filter, while the observer menu's change handler still only called `renderTableRows()`, which re-renders the rows already loaded without a new `/api/packets` request. ## Change - `public/packets.js`: the observer menu change handler calls `loadPackets()` instead of `renderTableRows()`, so the request carries the new `observer=` set. - `tests/unit/test-observer-menu-interactions.js`: two tests asserting the handler re-fetches, for ticking one observer and for "All Observers". Both failed before the change. ## Verification - `sh test-all.sh`: all suites pass (locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and `PYTHONUTF8=1`, unrelated to this change). - Browser, local server on the freshened e2e fixture: 55 rows; one observer ticked gives 2 rows, all from that observer; back to "All Observers" gives 53 rows. No reload in between. - Reproduced the original bug on an operator's v3.14.0 instance before the change. ## Not done - Playwright E2E suite not run locally. - Perf: one `/api/packets` request per checkbox toggle, the same request a reload or the node filter already makes. No per-item calls. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 21:07:27 +02:00
Alex BandClaude Opus 5.5	8b5ca5735c	fix(analytics): Packets / Hour readable on long histories (#2193 ) (#2194 ) Fixes #2193. ## Problem On an instance that keeps months of packets, the Overview tab's Packets / Hour chart becomes unreadable: 2,932 hourly bars in 720 units of plot, a date label every day, and the hours with no packets (which the server leaves out) squeezed off the axis. ## Before and after Both use the real `/api/analytics/rf` answer from the instance in the issue (2,933 hourly buckets over 124 days), served to the page in place of the fixture's own. Before (master, page on "All data"): ![before](https://raw.githubusercontent.com/A13xB0/CoreScope/e3cdf135/2026-10/pph-2193/before-desk.png) After, same data: ![after](https://raw.githubusercontent.com/A13xB0/CoreScope/7862abd8/2026-10/pph-2193/after-desk.png) Phone: <img src="https://raw.githubusercontent.com/A13xB0/CoreScope/7862abd8/2026-10/pph-2193/after-phone.png" width="300"> A short store (an instance holding 7 days in memory, plus a few old packets a day for weeks before that because they were heard again). The chart starts where the real data does (this shot predates the "since MM-DD" note now in the title when that crop applies): ![short store](https://raw.githubusercontent.com/A13xB0/CoreScope/7862abd8/2026-10/pph-2193/short-store.png) ## Change All in `public/analytics.js`; no API change. - `packetsPerHourCard()` replaces the inline block: - rolls the hours up to hourly, 6-hourly, daily or weekly bars, at most 180, and titles the card to match (Packets / Day); - counts missing hours as zero; - puts about 10 labels on round UTC times; - under "All data" only, starts on the first UTC day with at least 2% of a typical recent day (the median of the seven days before the last one), at its first hour with a tenth of a typical hour, and says "since MM-DD" in the title when that cuts anything. Scanning forward, an outage after the start is never taken for it. In a picked window (1h to 30d) the server has already cut the data, so nothing is cropped and a quieter early stretch stays visible. - Theme refresh before the data (separate commit; this also happens on master without the new chart): every load fires `theme-refresh` about 300 ms after the theme resolves, and the page re-rendered the current tab on it. When the analytics answers came later (a busy or just-restarted server), the Overview was drawn from an empty object and threw `Cannot read properties of undefined (reading 'totalTransmissions')`. It now skips that refresh until the data has landed; `loadAnalytics()` renders the tab with the current theme when it does. The span is still picked by the page's time window. ## Tests - `tests/unit/test-issue-2193-packets-per-hour.js` (in `test-all.sh`, red then green): 122 days gives 123 daily bars with 4 to 12 labels; 7 days gives 168 hourly bars; missing hours are drawn as zero; the chart starts at the data, not at the stragglers, and not after an outage; in a picked window nothing is cropped (90 days with 30 quiet ones gives 91 bars); a cropped chart says "since MM-DD" and an uncropped one does not; no data says so. - `tests/unit/test-analytics-theme-refresh-before-data.js` (in `test-all.sh`): drives `init()` with answers that never land, fires the `theme-refresh` handler, and checks that nothing throws and the tab is not re-rendered. Without the guard it fails with the production error. - `sh test-all.sh` passes; the #1925 neighbour-graph theme-refresh test and the analytics chart test still pass; the screenshots show no page errors and no sideways scroll at 1280 px or 390 px. ## Later - The 2% floor, the tenth-of-an-hour start and the 180-bar cap are hardcoded. Per AGENTS.md rule 8 they can move to the customizer in a later step. - Edge bars are partial periods (data starting at 17h gives a 7-hour first day; the last bar is still filling), and weekly bars align on epoch weeks (Thursday). Both are left as they are here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 11:46:28 +02:00
hermes-gadget	fdcec91578	fix(route-view): "N hops" counts only true hops, not payload endpoints (#2240 ) Fixes #2206 ## What Follow-up to #2207, as sketched in its review: the route meta (desktop header + mobile summary) printed `positions.length` as "N hops". For a DM, `positions` includes the payload-decoded sender and recipient rows — endpoints, not hops — so the count was inflated. Live on corescope.ukmesh.com right now, the reported 6-relay chain reads "8 hops · 913 km" with "from payload" chips on rows 1 and 8. ## Change - `public/route-view.js` — new `countHops(positions)` helper: counts rows without `_fromPayload` (both builders set the flag since #2207). The desktop header and the mobile summary both render it, so they can't disagree. Only the displayed figure changes; the row-count `total` still drives distances, stripe colours and row slicing. - `tests/unit/test-issue-2206-hop-count.js` — regression test (listed in `test-all.sh`): extracts `countHops()` from the shipping file and exercises it (6-relay DM → 6; payload-only → 0; plain chains and non-payload origin/dest unchanged), and pins both meta strings to the helper so a regression back to `total` fails. The count intentionally counts relay rows (per the review note), not edges: a 6-relay DM reads "6 hops". Frontend-only, static assets, no build step, no new dependencies. Cache buster untouched (AGENTS.md rule 3). ## Perf One `.filter()` over a ≤ ~10-entry array, once per sidebar build; no hot-path impact. ## Testing - `node tests/unit/test-issue-2206-hop-count.js` — 11 passed, 0 failed - `sh test-all.sh` — full unit suite green, exit 0 - Live cross-check against the deployed instances (pre-fix logic): `https://corescope.ukmesh.com/#/map?packet=44259e35b7c826a8&obs=97234` shows 8 rows, meta "8 hops · 913 km", payload chips on rows 1 and 8 → this change makes that same chain read "6 hops · 913 km". The new count ships with the next build that carries it; happy to re-check on a deployment once one does.	2026-10-17 10:39:08 +02:00
nullrouten0 nullrouten Claude Sonnet 5.5	127dd511a7	fix(health): judge favorites and home status cards by the node's role (#2215 ) ## The problem A starred repeater can show two different health statuses. The My Repeaters tab (#1761) and `getNodeStatus` judge it by the infra thresholds (degraded after 24 h, silent after 72 h). The favorites dropdown and the home page cards judge every node by the node thresholds (degraded after 1 h, silent after 24 h), because they never look at the node's role. \| repeater last heard \| favorites / home cards (before) \| My Repeaters tab \| after this PR \| \|---\|---\|---\|---\| \| 5 h ago \| degraded \| healthy \| healthy \| \| 30 h ago \| silent \| degraded \| degraded \| \| 80 h ago \| silent \| silent \| silent \| If you set `healthThresholds` in `config.json`, every view follows your values. ## The change Three call sites now use `getHealthThresholds(role)`, the helper the My Repeaters tab already uses: - `renderFavDropdown` in `app.js` - the "My Nodes" grid card in `home.js` - the hero search-result card in `home.js` The search card's fallback text, "Not heard in 24+ hours", was only true for non-infra nodes, so it now reads "Not heard recently". ## Scope - I kept the existing three-state healthy / degraded / silent logic and changed only which thresholds apply. - Not included: counting a repeater that is relaying traffic as alive (#1598, which `getNodeStatus` does). That is a bigger behavior change and would be its own PR. ## Risk Low. It is 9 lines in two files, and the helper and thresholds already exist and are used elsewhere. ## Tests New `tests/unit/test-health-card-role-thresholds.js`, added to `test-all.sh`. It checks that infra roles get looser thresholds than nodes, and that each of the three call sites goes through `getHealthThresholds(role)` with no node-only comparison left. On current `master` 5 of its 7 assertions fail; with this change all pass. `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning from an a11y self-test, which it also does on `master`. ## Update after review (commit `0f2376df`) The text above is the description efiten reviewed on 2026-10-15. This section describes the code as it is now. ### The change - New `getHealthStatus(role, heard, now)` in `roles.js`. It returns `healthy`, `degraded` or `silent` by the role's thresholds, or `null` when there is no usable timestamp. It only applies the cut-offs; each caller picks the timestamp. - All six places that compared an age against `degradedMs` / `silentMs` now call it: - the favorites dropdown (`app.js`); - the "My Nodes" card and the home health card (`home.js`); - the analytics network status and My Repeaters (`analytics.js`); - the live node panel (`live.js`). Each keeps its own labels and timestamp, so the last three show the same as before. `analytics.js` and `live.js` no longer guard with `typeof getHealthThresholds`. `roles.js` loads before both, and both already used it without a guard elsewhere: the old network status called `getHealthThresholds` directly, and `live.js` reads `ROLE_COLORS`. - The favorites dropdown and both home cards now fall back to the node's `last_seen` when the packet store has no packet for it. My Repeaters (`last_heard \|\| last_seen`) and the node page (`nodes.js:729`) already do this. Without it, with a packet store window under 72 h, a repeater could be silent on the cards and degraded on My Repeaters. The "ago" text uses the same timestamp. With the default thresholds, the fallback changes a status only when the packet store holds less than 72 h of packets. `retentionHours` unset (unlimited) or at the example's 168 h holds more than that. Otherwise a node with no packet in the window is silent either way, and the views now show how long ago it was last seen instead of "never" or nothing. - With no timestamp at all, the home health card said "Not heard in 24+ hours". It now says "No packets in the current window". ### Why a new helper and not `getNodeStatus` `getNodeStatus` has two states (active / stale) and counts a repeater that relays traffic as alive (#1598). Using it here would change more than the thresholds. `getHealthStatus` keeps these views' three states. ### Scope - Not included: counting a repeater that relays traffic as alive (#1598) in these views. That is a bigger behavior change and would be its own PR. - The live node panel still judges by `last_seen` alone, and the node page stays relay-aware. Both now use the shared thresholds; aligning their timestamps would also be its own change. ### Tests - `tests/unit/test-health-status.js` runs `roles.js` in `vm`. It covers: - repeater, room, companion, sensor, an unknown role and a missing role, at ages either side of 1 h, 24 h and 72 h; - ISO and epoch input; - null, empty and invalid timestamps; - a `healthThresholds` override through `/api/config/client`. One source guard remains: no age-vs-threshold comparison outside `roles.js`. - `tests/unit/test-my-repeaters-dashboard.js` now uses the real helper instead of a stub, which gave repeaters the node thresholds. It also checks that a repeater heard 30 h ago is Degraded. - New `tests/e2e/test-health-status-e2e.js` mocks `/api/nodes/<pk>/health` for six nodes. It checks five views: the favorites dropdown, the "My Nodes" card, the home health card, the analytics network status and the live node panel. It fails on `master` and on this PR's first commit. I put it on shard 2, which was the shortest shard in the last CI run. - Mutation check: I made 33 deliberate breaks. In the helper they cover each comparison and boundary. At each of the six call sites they cover the timestamp field, the role argument and at least one label mapping. Each one makes the suite fail. - I checked a `config.json` with `infraDegradedHours: 2` and `infraSilentHours: 6` in Chromium. Repeaters heard 3 h and 7 h ago show Degraded and Silent. - `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning, which comes from a deliberate fixture in the a11y self-test. ### Cost The analytics network status loops over every node. In headless Chromium, 30,000 synthetic nodes take 9.0 to 9.2 ms before this change and 9.1 ms after (median of 21 runs, on one machine). ### Risk Low. The change is frontend only, with no API, config or data change. Rolling back means reverting the commits. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-17 09:38:11 +02:00
hermes-gadgetandhermes-gadget	550829af6e	fix(route-view): badge payload-decoded sender/recipient rows in the sequence renderer (#2207 ) Refs #2206 ## What DM packets on the route view showed sender and recipient rows — decoded from the packet payload (`destHash`/`srcHash`, one byte each) rather than from the outer path bytes — as ordinary hops, without the "from payload" badge, whenever the chain renders through the multi-path/sequence builder. Only the single-path builder set `_fromPayload`, and the isolate/sequence list renderer never drew the chip. Effects: payload endpoints are indistinguishable from observed hops, and the displayed hop count is inflated (a 6-relay DM renders 8 rows, with the last "hop" being a single-byte recipient prefix resolved to whichever node shares it). ## Change - `public/map.js` — set `_fromPayload: true` on the sequence builder's origin (sender) and destination (recipient) entries (3 + 2 sites). - `public/route-view.js` — render the `mc-rt-status-chip mc-rt-status-payload` "from payload" chip in the isolate/sequence list, matching the canonical renderer. The chip markup now lives in one `payloadChip(p)` helper called by both row builders (it was duplicated between them, so they could drift apart). - `tests/unit/test-issue-2206-payload-badge.js` — regression test (listed in `test-all.sh`): extracts `payloadChip()` and `rowFor()` from the shipping `route-view.js` and exercises them in a vm sandbox, and pins every sender/recipient entry in `drawPacketRouteMulti` to `_fromPayload: true`. Frontend-only, static assets, no build step, no new dependencies. Per AGENTS.md rule 3 the cache buster in `public/index.html` is intentionally untouched — `__BUST__` is replaced automatically at server startup. ## Perf Two ternaries + one string concat per rendered row in a display-only list; no impact on hot paths. ## Testing - `node tests/unit/test-issue-2206-payload-badge.js` — 22 passed, 0 failed (new regression test) - `sh test-all.sh` — full unit suite green, exit 0 (201 suites) - Browser validation, re-run against the live deployments (which carry the equivalent patch): on https://corescope.nemesh.uk/#/map?packet=44259e35b7c826a8&obs=97234 the canonical chain marks rows 1 and 5 "from payload"; isolating the 6-hop path (`B4CE24→…→8C197F`) renders the 8-row chain with rows 1 and 8 marked. The refactor keeps the renderer output identical (pinned by the new test). The identical patched assets have been running on two independent deployments since 2026-10-13. ## Notes The underlying ambiguity (a 1-byte prefix matched against many nodes, resolved to the most-recently-seen candidate) is what the badge surfaces to the user; changing the resolution strategy itself is out of scope for this PR. The `N hops` header still counts every row — counting only non-payload rows would change every DM header, so it stays a follow-up; hence `Refs #2206`. --------- Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com>	2026-10-16 10:54:36 +02:00
efitenandClaude Opus 5.5	927a8eec8a	fix(nodes): node analytics rates cover the data in memory, and say so (#748 ) (#2211 ) Refs #748 (M3a: correct numbers within the in-memory window; the SQL history in M3b is a separate plan, see the end). ## Problem The node analytics range picker offers 24h, 7d, 30d and All (365d), but the store only holds its in-memory window (`retentionHours`, then eviction). `GetNodeAnalytics` still divided availability and packets/day by the requested range. Measured on our live instance (14-day window), repeater BE-BRE-ON8AR: \| Range \| Availability \| Packets/day \| \|---\|---\|---\| \| 7d \| 88.7% \| 91 \| \| 30d \| 42.6% \| 265 \| \| All \| 3.5% \| 22 \| Nothing on the page said that the data stopped at the edge of the window. ## What changes - `coverageStart` returns the edge of the in-memory window: the latest of `oldestLoaded` (background-load progress), `now - retentionHours`, and a new `evictedThrough` that eviction sets to the newest `FirstSeen` it removed, so memory-pressure eviction inside the retention window moves the edge too. - `s.packets[0]` is not that edge, and the first version of this PR used it. Chunks are selected by `last_seen`, so a transmission first heard weeks ago and heard again recently sits at the head with its old `FirstSeen`. On staging (7-day window) it returned 21/07. - When the edge lies more than a minute after the requested start, node analytics counts only transmissions from the edge, computes availability and packets/day over that span, and returns `timeRange.dataFrom` and `timeRange.partial`. - The minute of slack keeps a range equal to the retention window from reading as partial. Without it every 7d request on staging (168h) was partial, because the two edges came from separate clock reads. - The analytics page shows "Data from 2026-10-07: the server keeps about 7 days of packets in memory. Availability and packets/day cover that period, not the full 365 days." when partial. - UTC fix. The window bounds of node analytics and node hop analytics are formatted in UTC. They are compared as strings with UTC `FirstSeen` values, so on a non-UTC host the 7d window lost its oldest hours (2 at +02:00). Docker images run in UTC, which is why this did not show in production. - `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the edge is kept. ## Tests - Go, `node_analytics_coverage_test.go`: - full coverage within memory; - 30d over 3 days in memory; - empty store; - a re-heard 85-day-old transmission at the head of `s.packets` (edge stays at retention, packet not counted, no bucket before `dataFrom`); - memory-eviction edge; - eviction records `evictedThrough`; - retention 30 s shorter than the range is not partial, while 30d is; - window bounds under a +02:00 `time.Local`. - Two tests were checked to fail on the old code: the local-zone test (166 of 168 packets, a `+02:00` bound) and the retention test without the slack (`dataFrom 09:57:08` vs `from 09:56:38`). - Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). One run of five reported `FAIL` and four reruns were green; I did not capture which test failed. - Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js` (5 assertions on the real `node-analytics.js` in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`), against the real fixture response with no interception: 7d is not partial and shows no notice; All is partial and the notice names the real start date. 2/2 locally. ## Validation Staging (retentionHours 168), same repeater, after the background load completed: \| Range \| Before \| After \| \|---\|---\|---\| \| 7d \| 82.7% \| 83.3%, not partial \| \| 30d \| 19.3% \| 83.3%, partial from 2026-10-07 \| \| All \| 1.6% \| 83.3%, partial from 2026-10-07 \| In the browser the notice appears under All and not under 7d. ## Not done - M3b, older history from SQLite. In the database only the sender is indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay hops only in `observations.resolved_path` (filled for about 27% of observations on our live instance, not indexed). A fallback today would show only a node's own packets for older days, a different meaning than the in-memory part. Doing it properly needs a per-node index written by the ingestor (pubkey, transmission, role); that is a separate plan. - The other analytics endpoints with a range (observer analytics, RF) were not checked for the same dilution. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 09:27:21 +02:00
efitenandClaude Opus 5.5	cab4f71c64	feat(nodes): LoRa radio settings per node from the MeshCore map (#1268 ) (#2209 ) Fixes #1268. Frequency, spreading factor, bandwidth and coding rate are per-device settings that never travel in a packet. The public MeshCore map is the only source. This PR adds them to the node detail views, opt-in, and answers the open policy questions from the triage comments. ## Policy decisions \| Question (triage) \| Answer in this PR \| \|---\|---\| \| Opt-out for self-hosted instances \| Opt-in: `rfParams.nodesUrl` empty (default) = nothing fetched, `/api/rf-params` returns `enabled: false`. Same model as `knownChannelsUrl`. \| \| 33 MB every 6 h, thundering herd \| Random 0-60 s delay before the first fetch, then every `refreshHours` (default 6). The map sends gzip (17 MB on the wire). It sends no ETag or Last-Modified, so conditional requests are not possible. \| \| Size cap \| 128 MB on the decoded body (it is 51.5 MB today). Exceeding it fails the fetch instead of parsing a truncated body. \| \| Field shapes at the parse boundary \| 64-hex key; 100-1100 MHz; SF 5-12; bandwidth from the LoRa set (7.8 to 500 kHz); CR 5-8. Invalid rows are dropped. A body that is not a well-formed array fails the fetch. \| \| Keep stale cache on anomaly \| Any failure keeps the previous snapshot; a failed presets fetch keeps the previous presets. `fetched_at` is returned so an outage is visible rather than hidden. \| \| Freshness \| Only adverts within `maxAgeDays` (default 7) are kept at fetch time and re-checked at serve time. \| \| TLS pinning \| Not done: Go's certificate validation plus the field validation above. \| ## What changes - `cmd/server/rf_params_cache.go`: stream decoder, validation, cap reader, preset parser (presets with identical values are merged, titles joined), background refresh. In memory only, no DB writes, so the read-only server invariant holds. - `GET /api/rf-params`: `{enabled, fetched_at, source, presets, nodes}`; `nodes` holds only nodes this instance knows, keyed by lower-case pubkey. `/api/nodes` is unchanged, so its payload does not grow for every consumer and the 500-row paging is unaffected. - Node detail (side panel and full page): `LoRa` row with the four values, the preset name when all four match one, and the age of the map advert. Shown only when the feature is on and the server has fetched the map at least once. - No column in the Nodes table, on purpose: on our network such a column showed one preset title in 814 of 1,366 rows and nothing in 443. Showing only the nodes that differ from their region is the follow-up PR. - `config.example.json` documents the block with the two URLs to use. ## Cost - Measured on the real map payload (51.5 MB, 66,336 nodes): parse 0.52 s, 15 MB allocated in total, 26,815 nodes kept (advert within 7 days). Runs in a background goroutine every 6 h. - The endpoint walks the cached node list once with map lookups: O(nodes). - Frontend: one bulk fetch of `/api/rf-params` when a node detail opens (client TTL 5 min), not one per node. ## Tests - Go (`rf_params_cache_test.go`): parse with every rejection reason, malformed bodies (not an array, truncated, wrong type, empty), preset merge and invalid presets, cap reader (one byte over refused, exactly at the cap accepted), fail-soft against an `httptest` upstream (503, HTML body, presets-only failure), config defaults, route (disabled shape, only known nodes, entry aged past max age dropped at serve time). Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). - Frontend unit: `tests/unit/test-issue-1268-rf-params.js` (7 assertions on the real file in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-1268-rf-params-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`). It checks the real disabled state on the fixture server (no detail row), then serves `/api/rf-params` through `page.route`: no table column, full detail page with preset, side panel with raw values, and the hint for a node that is not on the map. 4/4 locally, plus the existing node table and node detail E2E files. ## Validation On a staging instance with `rfParams` set to the map and preset URLs: - Log: `[rf-params] 26821 nodes within max age, 20 presets`, about 6 s after start. - `/api/rf-params` returned 1,115 of our nodes: 988 on 869.618/SF8/BW62.5/CR8, 111 on the same with CR5, 15 on 433.65 MHz, 1 on SF7. - In the browser the detail row shows in the side panel and on the full page, and the Nodes table keeps its original columns. ## Not done - Highlighting nodes whose settings differ from their region's majority (a badge in the Nodes list plus a filter) is the follow-up (separate PR). In that follow-up a CR-only difference stays informational until it is confirmed whether MeshCore radios use the explicit LoRa header (then CR travels in the header and a mismatch still decodes). - `maxAgeDays` is config-only; the customizer is a later step. - No data is written to the DB, so the RF parameters are gone after a restart until the next fetch (at most 60 s plus the fetch time). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-16 09:26:45 +02:00
efitenandClaude Opus 5.5	1ae4c74913	feat(observers): rolling 1 h TX airtime and a duty-cycle warning (#661 ) (#2208 ) Fixes #661 (the narrowed scope from the 2026-10-14 comment). ## What changes - `GET /api/observers/metrics/summary` adds `tx_airtime_pct_1h` and `rx_airtime_pct_1h` per observer: the airtime share over the hour before now, whatever `window` is, from the cumulative `tx_air_secs` / `rx_air_secs` counters in `observer_metrics`. - A counter that goes down marks a reboot; only the stretch after the last reset counts. - Under 15 minutes of samples (one sample, or a reboot late in the hour) returns `null`. - Clamped to 0..100, rounded to 2 decimals. - The response also carries `duty_cycle_warn_pct`, from the new config key `dutyCycleWarnPct` (default 10, the EU 868 MHz g3 sub-band limit). Added to `config.example.json`. - RF Health tiles show `TX 1h: x%`; the observer list gets a sortable numeric `TX 1h` column. Both use the shared `public/observer-airtime.js` helper and flag values at or over the threshold with `.airtime-over` (`var(--status-red-text)`). - The handler now returns a named `MetricsSummaryResponse` struct instead of `map[string]interface{}` (one occurrence fewer). ## Why the warning matters Repeater and room-server firmware default to `airtime_factor = 1.0` (`examples/simple_repeater/MyMesh.cpp:890`), which `Dispatcher.cpp:26` turns into a 50% TX budget per 1 h window. The device therefore does not keep itself under the EU 10% limit; this view is where an operator can see it. ## Cost - Server: one extra query per summary request, `SELECT observer_id, timestamp, tx_air_secs, rx_air_secs FROM observer_metrics WHERE timestamp >= now-1h`, on the existing `idx_observer_metrics_timestamp`, then one O(rows) pass. On our instance that is 35 observers x about 12 samples (5 min interval). - Observer list: one extra bulk call `summary?window=1h` per load (client TTL 30 s, failure ignored), not a call per observer. ## Tests - Go: `TestRollingAirtimePct` (steady, reboot mid-hour, single sample, span under 15 min, reboot leaving under 15 min, nil counters, clamp, rounding), `TestGetMetricsSummaryAirtime1h` (samples older than 1 h ignored inside a 24 h window, single-sample observer gets null), `TestMetricsSummaryDutyCycleWarnPct` (default 10, config override, `observers` is `[]` not `null`). Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, which fails on Windows on clean master too). - Frontend unit: `tests/unit/test-issue-661-observer-airtime.js` (8 assertions, real file in `vm`), added to `test-all.sh`. `test-observers-headings.js` updated for the new column. Full `test-all.sh` passes locally. - E2E: `tests/e2e/test-issue-661-airtime-e2e.js`, registered in `deploy.yml` and `scripts/non-unit-tests.json`. The fixture DB has no `observer_metrics` table, so the summary is served through `page.route` with known values for the fixture's own observer ids. It checks the column header, over/under/unknown cells, sort order and the RF Health tile. 4/4 locally. It caught a real bug during development: `loadRFHealthData` declares a local `const window`, so the tile calls the helper as the global `ObserverAirtime`. ## Validation Deployed to a staging instance with live MQTT data (35 observers with metrics) and checked in the browser: - The observer list shows the column for 42 rows (33 with a value, 9 without enough samples). Sorting puts the highest share (5.3%) first. - RF Health shows `TX 1h` on all 35 tiles. With a fourth stat the line wrapped inside items on narrow tiles; the stats row now wraps between items only. - No observer was over 10% at the time, so the red state was checked by rendering one cell client-side at 12.5% (computed colour `rgb(185, 28, 28)` in light mode), and by the E2E test. ## Not done - The threshold is config-only; exposing it in the customizer is a follow-up. - No airtime comparison across observers (that is #671) and no per-packet airtime (Relay Airtime Share covers it). - Dark mode was not checked in the browser; `--status-red-text` is defined for both themes. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 21:07:20 +02:00
efitenandClaude Opus 5.5	fbeb9e4d71	ci: frontend coverage from every E2E suite, merged across the shards (#2204 ) Fixes #2203. ## Problem The frontend coverage badge (35.16% on run 38413268001) came from two sources on shard 2: one dump part-way through `test-e2e-playwright.js` and the `collect-frontend-coverage.js` tour. The other E2E suites run against the instrumented build but never save `window.__coverage__`, including the suites written for single pages (`path-inspector.js` 4.72%, `touch-gestures.js` 14% on that run). ## Change 1. `9d3bfad5` `test(e2e): red:` `tests/unit/test-e2e-coverage.js`, with fake pages, contexts and browsers (no Chromium needed). On the placeholder module the first assertion fails. 2. `c00bbb12` `feat(e2e):` the collector. - `scripts/e2e-coverage-hook.js` is a preload (`node --require`) that wraps `chromium.launch` from `playwright`. `playwright` re-exports `playwright-core`, so a suite's own `const { chromium } = require('playwright')` gets the same, wrapped object. - `scripts/e2e-coverage.js` reads `window.__coverage__` before every call that replaces or ends a page's document: `goto`, `reload`, `setContent`, `goBack`, `goForward`, `close`, and context and browser `close`. Pages from `newPage`, `newContext` and the context `page` event are all tracked. - Each read zeroes the page's counters, so a document read twice is not counted twice. Maps travel once per file; later reads carry counts only. - The page returns a JSON string. Returning the object cost about 10 s per suite, because Playwright serialises it node by node. - A read is given up after 5 s, so a busy page cannot hold up a suite's cleanup. - On exit the hook writes one file per suite to `.nyc_output/`, or nothing without coverage. A failed write is logged and never changes the exit code. 3. `64487c4c` `ci:` the workflow. - On master pushes the E2E run step sets `NODE_OPTIONS=--require ./scripts/e2e-coverage-hook.js`. On PRs it is empty, so PR runs are unchanged. - Each shard runs `nyc merge` (one file per suite is about 6 MB, mostly the same maps) and uploads `coverage-shard.json` with its results instead of `.badges/`. - The `e2e-test` job merges the three shards and makes the frontend badge (it used to come from shard 2 alone). - The merge, setup-node and badge steps are `continue-on-error`. When `nyc report` gives no statement percentage the step warns and writes no badge, so a registry hiccup cannot publish 0%. - `test-e2e-playwright.js` no longer writes its own `e2e-coverage.json`: the hook covers that suite completely, including the tests after the old dump. `collect-frontend-coverage.js` stays as an extra source. ## Performance Measured locally against an instrumented server (Windows, Chrome): \| Suite \| Without hook \| With hook \| \|---\|---\|---\| \| `test-path-inspector-coverage-e2e.js` \| 1.9 s \| 2.2 s \| \| `test-touch-gestures-coverage-e2e.js` \| 17 s \| 16 s \| \| `test-e2e-playwright.js` (stops at the same local-fixture failure in both runs) \| 52 s \| 52 s \| \| `browser.close()` on a page in a 30 s busy loop \| 153 ms \| 5.2 s \| The hook only loads on master pushes. ## Testing - `tests/unit/test-e2e-coverage.js` covers: - reads before each document-replacing call and before close; - pages opened by the page itself; - context and browser close; - summing without double counts; - a failed read, and a read that never answers; - the file writer, including a failing write. - Nine deliberate breaks of the library each turn it red: - dropping the counter reset; - dropping the `page` event, the `goto` read, the `goForward` read, the `close` read or the browser `close` read; - removing the timeout or the write guard, or writing an empty store. - The test also fails if it stops before its end. Without that check, a read that never returns would let node exit 0 half-way. - Real suites with the hook, measured on their own (the badge run had all suites): \| File \| Badge run 38413268001 \| Own suite with the hook \| \|---\|---\|---\| \| `path-inspector.js` \| 4.72% \| 52.75% \| \| `touch-gestures.js` \| 14% \| 72.96% \| - Exit codes and pass counts were unchanged with the hook for these suites (the first three were run before the 5 s read timeout was added): - `test-slideover-1056-e2e.js` (6 contexts); - `test-a11y-axe-1668.js` (`@axe-core/playwright`); - `test-issue-1150-404-state-e2e.js`; - the three above. - The badge step, run locally against two merged shard files: - It gives the same per-file numbers as `nyc report` on the inputs. - With no shard coverage, or with `nyc report` failing, it warns and writes no badge. - `sh test-all.sh` passes locally, apart from `test-issue-1956-release-routing.js`, which needs the workflow's `GITHUB_REF_NAME` and was skipped. The test inventory classifies 321 suites. `actionlint` 1.7.7 reports only the self-hosted runner label, as on master. Not tested: the shard and badge steps in CI. They only run on a master push, so the first master run after merge is the real test. I do not know what the merged number will be. I will compare the badge with the merged report in that run's `e2e-test` log. Not counted: a full page load the suite does not start through one of the wrapped calls, such as a link that leaves the SPA or `location.reload()` from page code. ## Not in this PR - The `tests/unit/` suites, which load frontend files through `vm.createContext` without instrumentation (noted in #2203). - Coverage on PR runs. The overhead above is small, so this could follow if wanted. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-13 22:04:03 +02:00
SaarMesh-Bot SaarMesh-Bot Claude	45680a22cf	fix(#2191 ): apply the channel filter to live WebSocket packets (#2192 ) Fixes #2191. ## Problem On the Packets page the channel filter only applies to the initial load (`/api/packets?channel=` → server-side `transmissions.channel_hash` match). The WebSocket handler re-checks time window, type, observer, region and node, but not `filters.channel`, and the channel filter is not re-applied at render time. With a channel selected, every new live packet is appended regardless of its channel. ## Fix - New pure helper `packetMatchesChannel(p, channel)` in `public/packets.js`, exported as `window._packetsMatchesChannelForTest` (same pattern as `_packetsBuildChannelOptionsForTest`). - It derives the value the server filters on from `decoded_json`, because WS packets don't carry `channel_hash`: - `channel` when the server decrypted the `GRP_TXT`, - otherwise `'enc_' + channelHashHex.toUpperCase()`. - Only `payload_type === 5` can match, mirroring the server (the column is only set for `GRP_TXT`). If a packet ever does carry `channel_hash`, that wins. - The WS filter calls it: `if (filters.channel && !packetMatchesChannel(p, filters.channel)) return false;` Checked the derivation against a production DB (live.saarmesh.de): it equals `channel_hash` for all 3,193 `GRP_TXT` rows of the last two days, 0 mismatches. ## Performance WS hot path, but the new check runs only while a channel filter is active, and for non-`GRP_TXT` packets it returns before any parsing. For `GRP_TXT` it is one `JSON.parse` of the already-received `decoded_json` string, O(1) per packet. WS messages are already debounced/coalesced by `debouncedOnWS`, so at typical rates (a few packets per second) this is negligible. No new state, no allocations beyond the parse, no API calls. ## Tests New `tests/unit/test-packets-live-channel-filter.js` (registered in `test-all.sh`), 10 cases: - no filter accepts everything - decrypted name match / mismatch - non-`GRP_TXT` rejected while a channel filter is active - `enc_<HH>` match incl. hex case normalisation and `enc_00` - `channel_hash` passthrough takes precedence - already-parsed `decoded_json` object - malformed / missing `decoded_json` doesn't throw and doesn't match - source check that the WS filter block actually applies the helper Without the `packets.js` change the new test fails; with it all pass. Existing `test-packets.js` (117), `test-packets-local-channels.js` (14), `test-packet-filter.js` (99) and `test-frontend-helpers.js` still pass. ## Browser validation The same change is deployed on live.saarmesh.de (hot-patched `packets.js`, server restarted so `__BUST__` refreshed; the served file contains the fix). I have not taken a screenshot of the filtered live table myself, so treat the in-browser check as pending until a reviewer or operator confirms. No config or customizer implications. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HJcbiRjQC5cSmydVFXbPbR Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-10-12 15:25:59 +02:00
efitenandClaude Opus 5.5	60af9bd469	fix(hops): narrow only the directly heard hop to the observer region (#2098 ) (#2188 ) Fixes #2098. ## Problem `HopResolver.resolve()` narrowed every hop of a path to the observer's IATA region (300 km around the code's coordinates). Only the last hop was heard by the observer directly; every earlier hop can be anywhere the mesh reaches, so the region says nothing about it. This stayed invisible on deployments whose observer codes were missing from `/api/iata-coords`: there `nodeInRegion()` returns `null` and nothing is filtered. A Belgian deployment's 42 observers use 10 codes and none was in the table. Filling the table alone would have switched on a filter that is wrong for every hop but the last. ## Why only the last hop #2098 lists three options and argues against simply repopulating the table, because the filter it would switch on is not sound. This PR does fill the table, but only after limiting the filter to the one hop where it is sound: - In a MeshCore path the last hop is the repeater the observer received the packet from, so it is within radio range of the observer. Measured below: p99 30 km for directly heard multi-byte hops. - An earlier hop says nothing about the observer's position. On the Belgian deployment nearly every observer hears nearly every flood, so a region filter on earlier hops would drop the right candidate whenever the packet started outside the observer's region. This was decided while preparing the PR, not on the issue thread, so the reasoning is recorded here. ## Change - `public/hop-resolver.js`: `resolve()` takes an optional `opts.directHops`, the hops the observer heard directly. Without it, the input is one path and its last hop is the direct one. The region check runs only on direct hops. An earlier hop gets `regional: false` and, with an observer, `regionSkipped: true`. `globalFallback` now means checked against the region with no candidate in it, so it is only set on a direct hop. - `public/hop-display.js`: the badge counts every candidate for `globalFallback` or `regionSkipped`. The dashed underline and the popover's "(global fallback)" stay for `globalFallback` only, so an earlier hop is not labelled as matched outside the region when no region check ran on it. - `public/packets.js`: `resolveHopsForPackets()` resolves the last hop of each path apart from the earlier hops. `hopCacheKey(h, observerId, direct)` keeps the two in separate cache entries, because the same prefix can be the last hop in one packet and a relayed hop in another. `renderPath()` reads the direct entry for the path's last hop, and the detail-pane resolve writes it. The region-narrowed entry never goes to the bare (observer-less) key, in the list or in the detail pane. The detail pane resolves and caches the shown observation's path with that observation's observer (`effectivePkt.observer_id`), the observer `renderPath` already used; before, it resolved with the packet row's observer, so when the first observation came from another observer the last hop missed its direct entry and showed unfiltered. The route button uses the same observer. - `cmd/server/routes.go`: `iataCoords` gains 104 entries from OurAirports `airports.csv` (public domain): every IATA code in Belgium, the Netherlands and Luxembourg, and the airports with scheduled service in Germany and France. AMS, CDG, FRA and MUC were already present and keep their values. ## Measured before building On 5000 packets from the Belgian deployment (3661 with a path): - 1-byte last hops: 1212, all ambiguous, 1204 of them without a server `resolved_path`, so they go through the client resolver. - Distance from the observer to a directly heard node, on 1892 unique multi-byte last hops: p95 25 km, p99 30 km, max 82 km. - With this change, on 906 1-byte last hops from observers whose code is now in the table: the badge counts 6.1 candidates on average instead of 6.6. The displayed name changes for 52, all from observers without their own lat/lon, where distance to the region now orders the candidates instead of list order. For observers with a position, no displayed name changes, because the #2097 observer anchor already picked the nearest candidate. ## Tests - `tests/unit/test-issue-2098-last-hop-region.js` (new, in `test-all.sh`, 25 assertions): - HopResolver: the last hop of a path is narrowed and an earlier hop is not; an earlier hop keeps every candidate and reports `regionSkipped`, not `globalFallback`; a direct hop with no candidate in region reports `globalFallback`; `directHops` overrides the position; a one-hop path is narrowed. - HopDisplay (real file): an earlier hop keeps a badge with every candidate but no fallback class or popover text; a direct hop outside the region keeps both. - packets.js (real file, real resolver and display): with one observer, a prefix that is the last hop of one packet and an earlier hop of another renders narrowed in the first and with a badge in the second; after the detail pane resolves a path, the observer-less render does not show the narrowed last hop; when the shown observation comes from another observer than the packet row, its last hop is narrowed to the region of the observer that heard it. - `TestIataCoordsIncludesBelgianObserverCodes`: ANR, BRU, GNE, KJK, LGG, OBL, OST and MST with their OurAirports coordinates. - Red commits first (`26000a22`, `bb91ff9d`, `189f55d7`, failing on assertions), then green. Mutation check on the packets.js tests: inverting `isDirect` fails 3 assertions, swapping the `directHops` argument fails 4. - `test-issue-2097-hop-ambiguity-badge.js`: two structural regexes now allow the new `direct` argument; every assertion still holds. - All 198 `node tests/unit/*` suites in `test-all.sh` pass. `cd cmd/server && go test ./...` passes (with `TestSaveGeoFilterPreservesFileMode` skipped, a Windows-only failure on master too); `go vet` clean. ## Browser validation Staging (this branch on top of the deployment's current build), headless Chrome on `#/packets/<hash>` for packets whose path ends in a 1-byte hop: the last hop's popover lists only candidates within the region with their distances (for example `7E`: 3 candidates at 46, 207 and 296 km), earlier hops list every candidate (`1D`: 13). The same prefix `7E` showed 3 in region as the last hop of one path and 4 as a relayed hop of another. No page errors. `/api/iata-coords` returns 158 codes. ## Browser validation of the follow-up Staging again (this branch at `4e129b49` on top of the deployment's build), headless Chrome: - Packets list, 41 paths of 2+ hops: the global-fallback class appears on 2 last hops and on 0 earlier hops. - Detail pane, 5 packets whose path ends in a 1-byte hop (1 to 13 hops shown): no earlier hop carries the class or "(global fallback)" in its popover, and every ambiguous earlier hop keeps its badge. Every last hop was checked against the region: 4 show "N candidates in region", 1 shows "(global fallback)" with the class. - Before the observer fix, 2 of 5 detail panes showed a last hop with neither label, because the shown observation came from another observer than the packet row (`b66a8f4c`: row from one observer with 13 hops, shown observation from another with 9). - No page errors. ## Not done - The 300 km radius is unchanged. The measured direct-hearing distances suggest it is wide, but a smaller radius would also have to hold for less dense networks, and I have no data for those. - `live.js` passes no observer, so the region filter does not run there, before or after this change. `map.js` passes the observer of the chosen path in path order, so once the resolver knows the observer regions it narrows that path's last hop only. - The "Cross-observer aggregate" line in the detail pane renders the longest path with the packet's own observer, so its last hop is narrowed to that observer's region even if another observer reported that path. The per-observer keying there predates this PR (#2097). - Two of the Belgian deployment's codes, HEP and NRW, are not airport codes and are not in OurAirports; those observers stay without a region filter. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-12 15:24:13 +02:00
Alex BandClaude Opus 5.5	955386b63f	test(map): settle the router's focus frame before the teardown snapshot (#2179 ) ## Summary Fixes an intermittent failure in `tests/e2e/test-map-nodes-pagination-e2e.js`, the "late response after map teardown" cases added for #2030 in #2048: ``` ✗ late /observers response after map teardown: old response must not change the destination page ``` It failed on #2164 ([run 38080812509](https://github.com/Kpa-clawbot/CoreScope/actions/runs/38080812509), shard 2/3), a Go-only change, and passed on the rerun. ## Cause `checkMapTeardown` compares `#app`'s HTML before and after releasing the old map's pending response. When it failed, the only difference was on the destination page's heading: ``` before: <div class="tools-landing"><h2>Tools</h2>… after: <div class="tools-landing"><h2 tabindex="-1">Tools</h2>… ``` That attribute comes from the router, not the old response. Since #630 (`app.js`, "SPA focus management") it focuses the new page's first heading in a `requestAnimationFrame` after the page renders. The before-snapshot was taken as soon as `#leaflet-map` detached, sometimes ahead of that frame. ## Fix The test now waits two frames before the before-snapshot, the same settle it already does before the after-snapshot. The router's frame is queued by then, so it always lands first. The change is test-only: no app code changes, and every assertion stays as it was. ## Tests Run locally with the e2e fixture (`tools/freshen-fixture.sh`, migrated) served by this branch's server, headless Chromium: \| \| runs \| failed \| \|---\|---\|---\| \| test on `master` (`352a0ccd`) \| 20 \| 2 \| \| this branch \| 50 \| 0 \| - Before: the two failures on `master` were "old response must not change the destination page" (teardown, `/config/regions` and `nodes`), plus one 30 s timeout in an "after map replacement" case in the same runs. - After: none of these failures or the timeout came back in 50 runs. ## Checks - Lint: ESLint passes with the repo config and with Node globals. SonarQube rules on the file report nothing on the added lines; its four existing minor findings are on lines this PR doesn't touch. - Repo rules: one logical change, explicit `git add`, no private data, no new dependencies. This is a test-only fix, so there's no red/green pair: the "red" is the intermittent failure above, measured as 2 in 20. Related: #2030, #2048 (where the test was added). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 13:28:25 +02:00
Alex BandClaude Opus 5.5	a4ee3a7328	feat(scope-audit): observer region filter for both views (#2171 ) ## Summary Builds on the transport view from #2167 and #2169 (both merged); rebased onto `master` (`352a0ccd`), so the diff is this PR's six commits only. The Scope Audit was the one analysis page without the region selector. This adds the shared region filter to it, for both the declared and the transport view. `?region=SFO,SJC` means what it means on every other endpoint: count only forwarding heard by observers in those regions, not repeaters located there. - API (`GET /api/scope-audit`, both modes): - The region codes are resolved to their observers first (a handful of rows), then the audit's hop scan gets `AND o.observer_idx IN (…)` on v3, or `o.observer_id IN (…)` on v2. The 3.5M-row hop scan never joins `observers`. - `All` means no filter (shared `normalizeRegionCodes`). Codes with no observer match nothing, instead of silently widening to every observer. - Responses echo the normalised codes as `region`. The declared and transport caches and their singleflight keys now include the region, so each region set is cached separately with the existing TTLs. - Declared-region verification (`regionEvidence`) is deliberately left unfiltered: it corroborates what a repeater declares from its own traffic, which doesn't depend on where a hop was heard. - Page: the shared `RegionFilter` sits with the view and window controls; a change reloads the view, and the selection persists like every other page. The transport view's text box is relabelled Scope (it already sends `?scope=` since the rename in #2167), so the page no longer has two controls called Region. - Docs: `docs/api-spec.md` (parameter, both response shapes) and the OpenAPI parameters for the route (`mode`, `scope` and `region` were all missing). Partial fix for #2142 (its region filter for rollout tracking), together with #2167 and #2169. ## Performance The unfiltered path must not get slower: it's the same query with an empty clause. `BenchmarkComputeScopeTransport` (from #2167, 1,200 repeaters), the PR it stacks on vs this branch, `-count 8`: \| \| before \| after \| \| \|---\|---\|---\|---\| \| sec/op \| 220.3 ms \| 218.0 ms \| ~ (p=0.065) \| \| B/op \| 43.77 MiB \| 43.70 MiB \| ~ \| \| allocs/op \| 833.5k \| 833.5k \| ~ \| Filtered requests, measured on staging (a copy of a production database): the first version drove `o.observer_idx IN (…)` from `idx_observations_observer_idx`, reading every observation the region's observers ever made. Fixed in `80414242` with a unary `+` (the same treatment as #2166), so the scan starts from the `first_seen` window and filters per row. Red test `67e063b1` asserts the query plan against a fixture with the real indexes. \| scan, one region over 24h (13 observers) \| plan \| time \| hop rows \| \|---\|---\|---\|---\| \| `o.observer_idx IN (…)` \| observer index \| 13.44 s \| 102,114 \| \| `+o.observer_idx IN (…)` \| `first_seen` window \| 0.25 s \| 102,114 \| \| unfiltered \| `first_seen` window \| 0.35 s \| 225,983 \| Cold end-to-end requests on staging after the fix: 24h declared, unfiltered 0.84 s · one region 0.48 s (was 16.7 s) · two regions 0.22 s · transport, one region 0.42 s · transport, six regions 0.75 s · 7d, one region 3.46 s. 24h transport, live data: 383 repeaters unfiltered, 204 heard by one country's observers. ## Tests - Red `629e4f36` (Go): extends the audit fixture with an `observers` table and `observations.observer_idx`. Covers the declared view ignoring a scope heard only by another region's observer, the transport view listing only repeaters heard in the region, `region=All`, per-region caching, a region with no observers, and the echoed codes. Five fail on assertions; `region=All` passes because the stub ignores the filter, which is the behaviour it pins. - Green `6f1b1e1c`: the implementation, plus a v2-schema case (`observer_id`), so both branches of the clause are covered. - Red `b4933d68` / green `08ecc172` (frontend): `apiPath` carries the selection in both views and adds nothing when none is selected. The region control on the page is a net-new UI surface. - `go test` (cmd/server) passes. Frontend suites: 193 of 194. The one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. ## Screenshot The same transport view with the shared region filter set to SJC: only forwarding heard by San Jose observers counts, so 7 repeaters are listed instead of 44, with their carried scopes counted from those observers alone. Taken against `test-fixtures/e2e-fixture.db` with fictional scopes. ![Scope Audit with the observer region filter on SJC](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/scope-audit-region-filter.png) ## Browser validation The branch's server was run locally with 14 configured regions (dropdown layout), checked in headless Chromium at 1280, 900, 700 and 390 px (touch): \| check \| 1280 \| 900 \| 700 \| 390 touch \| \|---\|---\|---\|---\|---\| \| ticking SJC re-requests with `&region=SJC` \| yes \| yes \| yes \| yes \| \| menu fully on screen \| yes \| yes \| yes \| yes \| \| page wider than viewport \| no \| no \| no \| no \| \| page errors \| none \| none \| none \| none \| Found and fixed this way: the filter is the right-most control on desktop, and the menu (which opens rightwards) ran 66 px off the page. It now opens leftwards above 640 px. On a phone the controls wrap, the filter starts a line, and the default applies. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - S3776 on `ScopeAuditForwardingHeardIn`: complexity 55. The function it generalises, `ScopeAuditForwarding`, is 54 on `master`; the region clause adds one point. - S3776 on `computeScopeAudit`: complexity 65, unchanged from `master`. - Both pre-date this PR. Splitting them is better as a refactor PR of its own than mixed into a feature. - The page code's findings were fixed in the transport-view PR it stacks on. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 11:17:12 +02:00
efitenandClaude Opus 5.5	55dd28164d	test(e2e): assert the configured-scope row after the theme-refresh re-render (#2180 ) ## What `test-issue-1648-m2-icons-e2e.js` now asserts the configured-scope row only after the page's `theme-refresh` re-render has fired and the detail has rendered again. ## Why On 2026-10-10 this assertion failed CI on four PRs (#2169, #2171, #2154, #2164), each time at the first live-map case: `confirmed scope row must be visible, including an empty scope`. A re-run of the failed jobs passed every time. The code path: - `customize-v2.js:814` dispatches `theme-changed` once `/api/config/theme` resolves, on every page load. - `app.js:1309-1315` turns that into `theme-refresh` 300 ms later. - `live.js:4713` re-runs `showNodeDetail(activeNodeDetailKey)` on `theme-refresh`, and `showNodeDetail` first sets `#nodeDetailContent` to "Loading…" (`live.js:2598`). This is the only caller besides the marker click (`live.js:3090`). - The test waited for `#nodeDetailContent table` and then called `row.isVisible()`, which does not wait. A refresh landing between the two fails the check. The same race could let the null-scope case pass on the "Loading…" panel, because `row.count() === 0` holds there too. The fix uses the same `addInitScript` flag pattern as `test-e2e-playwright.js:83`. ## Testing - Locally against a server on the E2E fixture: 5 of 5 runs pass, 27 of 27 checks each. - Not reproduced locally. The unchanged test also passed every local run: 5 plain, 3 with `/api/config/theme` delayed by 1 s, 5 with 8x CPU throttling. So the reasoning above comes from the code path and the four CI failures, not from a local red-then-green. - Test-only change, no product code touched. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-11 10:58:20 +02:00
Alex BandClaude Opus 5.5	352a0ccd19	feat(region-filter): configurable region quick picks (#2170 ) ## Summary Independent of every other open PR. Choosing a pick saves the selection, redraws the control and notifies the page's listeners itself, exactly as a click in the control does, so it re-queries straight away whether or not #2168 (`setSelected` notifies) is merged; with both in, nothing fires twice. Region quick picks are named groups of region (IATA) codes, offered as one-tap choices inside the region filter, e.g. "Northern California" for a dozen airport codes instead of ticking each box. One deployment has been injecting these into the page through an nginx `sub_filter` and a script that wraps `window.RegionFilter`; this makes them a configured feature. ```jsonc "regionQuickPicks": [ { "name": "Northern California", "description": "Bay Area and Sacramento observers.", "regions": ["SJC", "SFO", "OAK", "SMF"] } ] ``` - Server: `GET /api/config/region-quick-picks` returns the groups cleaned up: names and codes trimmed, codes upper-cased and de-duplicated in their configured order, groups without a name or codes dropped, at most 20 groups of 200 codes, always a list. `/api/config/regions` is unchanged. - Region filter: loads the groups with the regions and offers every group with at least one observer today, inside the region control: - dropdown layout: rows of the same kind as All (checkbox and bold name), directly under All with a divider before the single regions; the row is ticked while that pick is the selection, ticking it selects the pick and closes the menu, unticking goes back to all; - pill layout: the picks lead the same bar, set off from the single regions by a thin divider. - A tap selects the group's codes that have an observer and re-queries straight away. - The selector then reads the group's name ("South Bay ▾") instead of "2 Regions", and the button shows as pressed. Selecting the same codes by hand does the same. - Tapping the active group again goes back to all regions. - A failed fetch, or no configured groups, leaves the filter exactly as before. - Mobile: the rows take the menu's 44 px row height under `(pointer: coarse)`; the pill-bar picks reuse `.region-pill` and its touch target. Inside `.filter-bar` (Packets) the existing touch rule for inputs made every region checkbox a 44 px square; the checkbox is now 20 px and the 44 px row is the tap target. - Semantics: like every region filter, a group means "heard by observers in these regions", not "located there"; `config.example.json` says so. No existing issue covers this. ## Screenshot Nodes page with two picks configured (example config below), after tapping South Bay: the pick and its codes (SJC, MRY) are selected, and the list re-queried for them. Taken against `test-fixtures/e2e-fixture.db`. ![Region quick picks on the Nodes page](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/region-quick-picks.png) ## Config example ```json "regionQuickPicks": [ { "name": "South Bay", "description": "San Jose and Monterey observers", "regions": ["SJC", "MRY"] }, { "name": "East and North Bay", "description": "Oakland and San Francisco observers", "regions": ["OAK", "SFO"] } ] ``` Each pick needs a `name` and at least one code; `description` is optional (shown as the tooltip). Codes are trimmed, upper-cased and de-duplicated; a pick with no code that has an observer today is not shown. At most 20 picks of 200 codes. ## Browser validation The branch's server was run locally against a migrated copy of `test-fixtures/e2e-fixture.db` (observers in MRY, OAK, SFO, SJC) with three groups configured: "South Bay" (SJC, MRY, SCZ), "East and North Bay" (OAK, SFO, sts) and "Nowhere yet" (XXX). Checked in headless Chromium: \| check \| desktop 1280 px \| phone 390 px (touch) \| \|---\|---\|---\| \| groups shown \| South Bay, East and North Bay ("Nowhere yet" hidden) \| same \| \| tap South Bay selects \| SJC, MRY (SCZ has no observer) \| same \| \| next `/api/packets` request \| `region=SJC,MRY` \| `region=SJC,MRY` \| \| selector label \| "South Bay ▾" \| "South Bay ▾" \| \| second tap \| back to all regions \| same \| \| button height \| 24 px \| 44 px \| \| page wider than viewport \| no \| no \| \| page errors \| none \| none \| Checked on Packets (dropdown layout) and Nodes (pill layout). Three layout problems found this way on the phone were fixed before opening: a sideways-scrolling row clipped at the screen edge, the selector squeezed into a narrow column (the filter group is a non-wrapping flex row), and the two wrapped rows spread apart on Nodes (`align-content`). Revision 1: the first version put the picks in a separate row beside the selector; they moved inside the dropdown. Re-checked with a 14-region list and three picks on Packets: desktop 1280 px dark and phone 390 px light both keep the menu on screen (menu 340 px wide, page not wider than the viewport), the pick closes the menu and the trigger names the pick. Revision 2: picks are checkbox rows like All instead of chips. Re-checked on Packets at 1280 px dark and 390 px light: rows render bold under All, the divider separates the single regions, ticking a pick ticks its codes and names the trigger, no sideways scroll. Unrelated and not changed: on Nodes at 390 px the region row (existing pills included) starts flush with the screen edge. ## Tests - Red commit `81ea67d6`: the config type, a stub normaliser and endpoint returning nothing, and tests. - Go: cleaning (trim, upper-case, de-duplicate, drop incomplete groups), caps, nil config, and the endpoint's JSON. - Frontend, `tests/e2e/test-region-quick-picks.js` (moved from `tests/unit/` in `fced69a7`; it needs jsdom, so it runs in the E2E job like `test-table-sort.js`): the real `region-filter.js` in jsdom against faked config endpoints. Nine cases: rendering and hiding groups with no observer, selecting and re-querying, the selector label, toggling back, a hand-made selection marked active, the pill layout, no groups, a failed fetch, HTML escaping. Seven fail on the stub, all on assertions. - Green commit `0e33210e`: the implementation, styles, `config.example.json` (`regionQuickPicks` + `_comment_regionQuickPicks`), `docs/api-spec.md`, and the OpenAPI route list (`TestOpenAPICompleteness` requires it). The jsdom test also had a cross-realm comparison bug fixed in this commit (`Array.from` on arrays coming out of the jsdom window). - `make test`: 16 of 16 modules pass. Frontend suites: 195 of 196; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. - Red commit `f991060d` (revision): picks live inside the dropdown menu before the checkboxes, choosing one closes the menu, and in the pill layout they share the region bar; two fail on assertions against the old layout. Green commit `1f720e35`. Frontend 195 of 196 (same environment-only failure). - Red `134f5256` / green `da0adb50` (second revision): picks are checkbox rows between All and the single regions with a divider, no chips in the menu; ticking and unticking a pick row. The green commit also repairs `style.css`: `1f720e35` had prepended its block to the top of the file (its edit matched an earlier `.region-pill {`) and left the previous rules in place. The file now differs from `master` by 13 lines. ## Review notes - Groups are offered only when at least one of their codes appears in `/api/config/regions`, and only those codes are selected. The stored selection is pruned to known codes on load anyway, so selecting absent codes would not survive a reload. - Customizer (AGENTS.md rule 8): groups are operator configuration; a customizer editor could come later. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - Fixed in `b84ace4e`: - `normalizeQuickPickCodes` split out of the config normaliser (S3776, 17 > 15); - a comment on why a failed quick-pick fetch is ignored (S2486); - `for…of` in `activePick` (S4138); - in the tests: `node:` specifiers, `includes()`, and an explicit string comparator for sorted selections (S7772, S7765, S2871). - Kept on purpose: - S2486 is still reported on that `catch`. Swallowing the error is the intended behaviour: the filter works as before, just without picks. - S1523 on the `vm` test loader, as above. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 21:23:05 +02:00
Alex BandClaude Opus 5.5	99efaed33d	fix(region-filter): setSelected notifies change listeners on a real change (#2168 ) ## Summary `RegionFilter.setSelected()` saved the selection and redrew the control but never ran the `onChange` listeners; only a click (`toggleRegion`) did. A programmatic selection therefore updated the dropdown while the page kept showing data for the old selection until the next click. Affected today: the Packets page applies a `?region=` URL parameter through `setSelected`. It registers its listener afterwards, so its first load is fine, but any other page's listener that is still registered sees nothing. And anything that offers one-click region groups has to wrap `setSelected` to make it re-query (an nginx-injected overlay on one deployment does exactly that). This PR stands alone; #2170 (configurable quick picks) does not need it. `setSelected` now compares the new selection with the current one (order-insensitive; empty and null both mean "all") and, only when they differ, runs the listeners after saving and re-rendering, exactly as a click does. Re-applying the current selection stays silent, so a caller restoring state does not trigger a reload. No existing issue covers this. ## Tests - Red commit `75569b78`: `tests/unit/test-region-filter-setselected.js` loads the real `public/region-filter.js` in a `vm` sandbox. Two cases fail on `master` (a programmatic selection notifies once with the new codes; a clear notifies once with `null`). Two pass on both commits on purpose: re-applying the current selection in another order, and clearing an empty selection, must not notify. - Green commit `8a7ddfe1`: the fix. - Listed in `test-all.sh` in sorted order; `test-test-inventory.js` passes. `test-1108-region-hide-nodes.js` (the other region-filter suite) passes. Frontend suites: 194 of 195; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. ## Review notes - `packets.js` "clear all filters" calls `setSelected([])` and then reloads itself. When a region was selected, that path now also triggers the page's region listener, so the packets list is requested twice. It is a cosmetic double fetch on an explicit user action; left as is to keep this PR to the component. - No performance claim; this is a correctness fix. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - Fixed in `9e732706`: `node:` specifiers (S7772). - Kept on purpose: S1523 on `vm.runInContext`, which loads the repo's own `public/region-filter.js` as the other frontend unit tests do. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 21:22:54 +02:00
Alex BandClaude Opus 5.5	deb55dc82e	feat(scope-audit): transport view on the Scope Audit page (#2142 , page) (#2169 ) > Requires #2167 to be merged first. This page calls the transport view API that #2167 adds (`/api/scope-audit?mode=transport`), so it cannot be split from it; until #2167 merges this PR also shows its commits. ## Summary Page half of #2142. Stacked on #2167 (the API); review that first. #2171 adds the observer region filter on top. Partial fix for #2142 until all three are in. Adds a Declared / Transport toggle to `#/scope-audit`. The transport view calls `?mode=transport` and lists every repeater seen forwarding in the window: - Seen carrying: one chip per region scope with its packet count (first/last seen in the tooltip); - Declared: the repeater's declared regions coloured by whether they were seen carried (the declared view's own chips), or not asked when no declared-regions answer exists. Never asked is worded differently from declaring nothing; - Other traffic: unscoped, unmatched and ambiguous-hop counts. A Region box (suggestions from the scopes seen) adds a Carries X column, lists the repeaters not yet carrying it first, and summarises carriers against active non-carriers: the rollout report the issue asks for. Window, view and region are deep-linked (`#/scope-audit?window=7d&mode=transport&scope=be`) and survive a reload. The page says plainly that transported means seen carrying, not configured, and that over a short window a quiet region looks the same as a missing one. The intro text changes with the view. ## Screenshot The transport view with scope `be` typed in: every repeater seen forwarding in the last 24 h, whether it carries `be` (non-carriers first), the scopes it was seen carrying with counts, and its declared regions or not asked. Taken against `test-fixtures/e2e-fixture.db` with fictional `be` / `be-van` / `nl` scopes. ![Scope Audit transport view](https://raw.githubusercontent.com/A13xB0/CoreScope/pr-screenshots/2026-10/scope-audit-transport-view.png) ## Browser validation (staging, real data) Headless Chromium against a staging copy of a production database with its live feed: - `#/scope-audit?window=24h`: the declared view, 16 rows, as before. - Clicking Transport: 385 rows; the hash gains `mode=transport`. - Typing a scope as `#XX-YY` in capitals: normalised to `xx-yy`; the hash gains it (now `scope=xx-yy`); summary "385 repeaters seen forwarding in the last 24h · 16 with a declared-regions answer · 51 carried xx-yy · 334 active but not seen carrying it"; the first rows are the non-carriers. - Reloading that hash restores view and region. Switching back to Declared hides the Region box and restores the declared intro. No page errors. - Checked at 1280 px and 390 px wide. Two layout bugs found this way were fixed before opening: the region box inherited `width: 100%` from `.nodes-search`, and `display: flex` overrode the `hidden` attribute on the region bar. ## Tests `tests/unit/test-frontend-helpers.js` gains 10 cases on rendered markup and state, using the page's exported internals like the existing scope-audit tests: hash and API paths (and that a region never leaks into the declared view's link), region normalisation (`#`, capitals, empty), not asked vs declared chips (with the `` wildcard), carried-scope chips with counts and the filtered region marked, the carries column only with a region, HTML escaping of names, the summary counts, region suggestions, and the per-view intro. 768 passing in that file. Net-new UI surface, so per TDD.md the tests land in this PR rather than as a preceding red commit. Frontend suites on this branch: 193 of 194; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails identically on `master` locally (#2149 and #2160 address that). ## Review notes - Colours only through existing CSS variables (no hex literals); the filtered region is marked with the link colour, not a new status colour. - The window note's wording is in the same spirit as the declared view's `windowHonestyNote`. - No API calls per row: one request per view/window/region, cached client-side for 30 s like the declared view. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - Fixed* in `df53e770`: - one `modeQuery` helper for the page hash and the API path (S3358; the two copies were identical); - the carries-scope cell built before the row (S3358); - `localeCompare` for the region suggestions (S2871), `for…of` and `.dataset` (S4138, S7761), optional chaining (S6582); - object spread in the helper tests (S6661). - Optional chaining and spread are already used throughout `public/`. No behaviour change. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 21:22:37 +02:00

1 2 3