mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 12:18:47 +00:00
417c8dd00a243f29822f34e5c9aaadbd2e522213
66
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 TeTeHackoandClaude Opus 5.5
|
417c8dd00a |
feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) (#2300)
Closes #2270, closes #2271. The ingestor took only the confirmed region scopes out of an observer's `/neighbors` report (#1865) and dropped the rest. Nothing could therefore say when an observer last reported, or whom it heard. ## Ingestor `observer_neighbor_reports` (created in `dbschema.Apply`) keeps the latest report of each observer, one row per observer, replaced by every live report. The report is unauthenticated, so `neighbors[]` is stored sanitised: - only 64-hex keys - `snr` and `heard_secs_ago` finite and in range - `status` limited to `[a-z_]` - `scopes` clamped - at most 500 rows `total_neighbors` is believed only when it is not below the number of rows that arrived. A **retained replay** only fills a missing row, and it is dated by the report's own `timestamp`. After an ingestor restart, a long-dead observer therefore does not look as if it just reported. ## API The table is optional for the server (no `AssertReady` change); without it the fields are simply absent. - `GET /api/observers`: `neighbors_report_at` and `neighbors_count`, from one batch query in the existing cached builder. - `GET /api/observers/{id}`: also `neighbor_report`, which is `{received_at, reported_at, neighbor_count, queried_neighbors, self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr, heard_secs_ago, status, scopes}]}`. Names and roles come from the node table in one query. ## Frontend `public/observer-neighbors.js` is shared by both pages and escapes everything. - **#2270, observers list:** a sortable **Neighbors** column, e.g. `3h ago · 12`, or `—` when the observer never reported. The same value appears in the slide-over. - **#2270, observer page:** **Neighbors Report** and **Neighbors** stat cards. - **#2271, observer page:** a **Neighbors** table listing each neighbor by name (a link to the node) or key prefix, with SNR, heard, scope query status and scopes. The heading says when the list is shorter than the count, e.g. `(2 of 3 listed)`. ## Tests - **ingestor:** sanitising, the cap, live replace, the retained rules (fill only, dated by the report, undated ignored) and the MQTT topic end to end. The existing #1865 scope tests are unchanged and still pass. - **server:** the list summary, the detail with names, and both endpoints without the table. - **frontend:** - unit tests of the helpers - an XSS case for every rendered field; I checked that it fails when escaping is removed - `test-observers-headings` now expects the new column after `Last Packet` - **E2E:** `test-observer-neighbors-2270-e2e.js` intercepts the observer endpoints, since the fixture has no reports. It is listed in `scripts/non-unit-tests.json` and on shard 2. Gates that pass: - `go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema` - `sh test-all.sh` - `eslint@8`, with 0 errors - the XSS diff gate - these E2E suites: the new one, `test-issue-1639-observers-sort-e2e.js`, `test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js` Performance: the list adds one primary-key-sized query per cache fill, and the detail adds two small queries per request. Also fixes #2330 (folded in from #2337, second commit). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nexandn30nex
|
202bafea28 |
fix(observers): show metadata while analytics loads (#2339)
Partial follow-up to #1778, limited to observer details; other pages remain separate. Observer identity, status and radio information now render when the basic request finishes. Analytics and clock calibration load into their own regions, so either can finish first. An analytics failure preserves usable metadata. Identity failures and obsolete responses cannot overwrite the current view, and failed chart construction is cleaned up. The same three requests start together. Generation ownership, request deduplication, date-window behavior and disposal remain covered. No new fetches, settings, dependencies or customizer options; secondary replies preserve the actual metadata DOM node. [RED CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426343054/job/121932999182) proves 14 new assertion failures while all 11 existing unit cases pass. Local RED browsers reproduce 20 new failures while all 32 old cases pass. GREEN `52eabad7` passes 221 standalone suites, 28 ownership unit checks and 56 browser cases across desktop/mobile and both themes; the parent independently repeated the 56 browser cases. All three independent review roles passed. Held-response assertions prove the rendering behavior without claiming a server-latency speedup. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427087948) passes on this exact head, including all three browser shards, the repeated slide-over stability gate and the architecture build. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
n30nexandn30nex
|
1ed5e4ab3f |
fix(compare): exclude observer TX-only packets from overlap (#2346)
Red commit: `39295da8` ([intended CI assertion failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)). Green commit: `d1bda496`. Observer comparison counted an observer's TX-only packets as receptions, producing false overlap. It now requests `excludeTx=true`: the selected observer and non-TX direction must match the same observation in memory and SQLite v2/v3. Normal packet lists and stored TX records remain unchanged; missing directions retain legacy compatibility. Refs #667. This completes a comparison slice, not the issue's remaining observer counters or #671's fleet/SNR feature. Representative display fields and existing time-filter semantics are unchanged. API/OpenAPI docs describe the filter and multi-node bypass. Validation: - RED: Go tests compiled with 32 intended failures/16 control passes; unit and browser regressions reproduced false overlap. CI reached the new unit assertion. - GREEN: all 48 packet cases pass, plus grouped-cache isolation and OpenAPI tests. The added cache regression caught a malformed-radio cache-key collision before correction. - Full local server suite: 174s; ingestor: 185s; `npm run test:unit`: pass. Final focused packet/grouped/OpenAPI checks: pass. - E2E assertion added to `tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks pass, including correct 1/2 reception totals and 50% overlap. Performance: still two capped bulk requests, indexed candidate lookups retained, one direction check within the existing observation scan/SQL predicate, no new persistent structures. No configuration/customizer changes. Final-head CI passed; independent reviews found no must-fix items. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
n30nexandn30nex
|
c211ef7643 |
feat(nodes): show relay activity from certain forwarding evidence (#2336)
Fixes #2331 with the additive API option. The Last Relayed row shows green **actively relaying** only when `last_certain_relay` falls within the existing `healthThresholds.relayActiveHours` window. Missing, invalid, future or expired evidence adds no idle/alive claim. Existing `relay_active`, attributed timestamps and counts retain their meanings. The optional node list/detail field uses the existing certain-relay path: an eligible observed frame, matching hop, unique prefix, no retired namesake and existing signature checks. Originating an advert does not count. Forwarding another node's advert can count, even with zero non-advert attributions. Adverts with an unknown origin cannot establish forwarding. Evidence uses transmission `first_seen`, not late observation receipt time, and follows node-membership eviction. Snapshots take one read lock over requested keys, O(page size), with no packet scan or new database query. Work per qualifying hop stays bounded. No new dependency or setting; configuration documentation and OpenAPI explain the distinction. The test-first history includes assertion-failing RED commits and GREEN fixes. The initial RED's nil-return compile stub is explicitly permitted by `docs/agents/TDD.md`. Local native server/ingestor suites, focused race checks, 809 frontend helper assertions and 70 browser cases pass. Browser coverage includes both themes and full/side/tablet detail views. [Review regression RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40421464618/job/121916723339) proves both malformed-origin failures. All three independent review roles pass at `d1fe79cb`. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40422655728) passes on that exact commit, including all three browser shards and the two-architecture image check. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
n30nex
|
3bf678709f |
fix(ui): preserve navigation and menu focus during delayed updates (#2333)
Fixes #2332. Fixes #2335. Two delayed UI updates could undo user actions. Rebuilding the More menu removed its focused link; a late packet-list response could replace a newly requested Nodes URL before `hashchange` ran. The menu now preserves the selected destination when it stays in More or moves inline, closes an empty menu, and leaves other controls' focus alone. Packet URL updates now require the packets route, including its existing detail/query forms. Observation selection and encoded filters remain supported. The changes add one lookup across the existing 12 navigation links and a URL ownership check. No packet loop, API request, dependency, setting or customizer change is added. [Packet RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409110743/job/121874676123) fails four wrong-route assertions. [Menu RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409898375/job/121877663579) fails the intended focused-destination assertion after the packet correction removes the competing navigation failure. GREEN is `57522143`. Local combined validation passes all 221 standalone frontend suites, 60 navigation browser checks, 17 branding/route checks and 11 filter/deep-link checks. The packet regression controls delivery of a real API response; substituting the original code still fails it. Existing exact title assertions remain; diagnostics now report the actual title. Backend source trees are unchanged from the tested baseline. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40410894023) passes on `57522143`, including all three browser shards and both architecture builds. Three independent source reviews found no blockers. Final repository checks and after-push browser verification pass. Ready for maintainer review. |
||
|
efitenandClaude Opus 5.5
|
d29957fbdc |
feat(observer-logins): lock the shared login out of migrated observers (#2348)
Follow-up to #2306 / #2319. With personal observer logins, a migrated observer still is not safe from the shared login: the hook answers `ignore` for the shared login, so EMQX's built-in database lets it publish under any observer's topic, including one whose owner has moved to a personal login. This PR lets an operator close that per migrated observer, before the shared login is removed altogether. ## Changes **users.db v8.** `observer_bindings.confirmed_at`: the moment the owner's personal login first published under that pubkey. - An auto-bind (the owner's first publish) is confirmed at once. - An admin assignment stays unconfirmed until the owner's login publishes under that pubkey; that publish sets it. - Reassigning to another user or unbinding clears it; reassigning to the same owner keeps it. - Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin` rows stay NULL. **Config.** `userManagement.observerLogins.sharedUsernames` (default empty): the broker's shared login names. A name starting with `obs_` is refused at startup. **Authz hook.** For a login in `sharedUsernames`, a publish under `meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is confirmed, whatever the number of topic segments (the ingestor attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey). Logged as `publish denied: user=<name> observer=<8 hex>: confirmed owner`. Every other request from a shared login keeps `ignore`, so the built-in database decides as before. Personal logins are unchanged. An orphaned binding (owner account deleted) does not count as confirmed. **Admin.** The observers table of the Observer logins tab gets a Confirmed column; the bindings JSON gets `confirmedAt`. **Docs.** `docs/observer-mqtt-logins.md`: "Locking the shared login out of migrated observers" (what it denies, how to see `confirmedAt`, the risk below). `config.example.json` has the key. ## Risk for operators An observer that is confirmed but still runs a second uploader on the shared login loses that second uploader once its name is in `sharedUsernames`. The deny log line names the observer. ## Tests - `internal/users`: v7 to v8 migration on a real v7 database, auto confirm, admin assignment then owner confirm, non-owner and repeated confirm, reassign clears, orphan reassign. - `cmd/server`: the decision table (confirmed, unconfirmed, unbound, non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty and `client` iata, lowercase key; owner publish confirms, confirm failure still allows and retries; reassign, unbind and user removal clear the flag; the index load against a real store; config validation; the hook and admin endpoints end to end. - `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an auto-bound observer. - Local: `internal/users`, `cmd/server` (two Windows-only tests skipped, as on master) and `cmd/ingestor` green. The E2E assertion was not run locally; CI runs it. ## Not in this PR - Using `confirmed_at` for verified `/neighbors` reports: a separate PR, built on #2300. - Removing the shared login: still an operator step after migration. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
TeTeHackoandClaude Opus 5.5
|
77a5f14925 |
feat(observers): radio setting per observation from the observer's radio history (#1869) (#2292)
Part of #1869. ## Problem `observers.radio` keeps only the **last** `MHz,kHz,SF,CR` an observer reported. Once an observer is retuned (say from SF7 to the EU narrow SF8 setting to listen to a neighbouring mesh), every packet it ever heard reads as heard on the new setting, and there is no way to tell an SF7 packet from an SF8 one. ## What this does As #1869 proposes, the packet data model is untouched: the setting of an observation is resolved from its observer and time. **Ingestor** (all schema in `dbschema.Apply`) - `observer_radios`: one row per stretch of time an observer, and each radio of a multi-radio node, stayed on one setting. Every live status extends the current row and a change starts a new one; retained replays never extend a row. On the first run it is seeded with every observer's current radio, which is what the UI already showed for the past. - Settings are stored canonical through `dbschema.CanonicalRadio`, so `869.4320068,62.5,7,5` and `869.432,62.50,7,5` are one setting. - `observation_radios` (`observation_id → rx_radio_id`): multi-radio openHop nodes tag each packet with `rx_radio_id` and list `radios: [{id, radio}]` in their status, so their observations resolve per radio. - The table is sparse: single-radio nodes never send the field, so `observations` and every query that scans it stay untouched. - The row id comes from the observation upsert via `RETURNING id`. - Rows are pruned with their transmissions, and a purged observer takes its history along. **Server** (read-only) - No `AssertReady` change: an absent table reads as empty and is probed again later, the same rule as `advert_route_evidence`. - The history is held in memory. The live poller checks it on every tick (two primary-key lookups), so a retune or a new `rx_radio_id` is known before the observation is broadcast. A full reload runs every 10 min. - `radio` is added to observations, packets, grouped rows (taken from the header observation) and the live feed. - `GET /api/packets?radio=MHz,kHz,SF,CR` filters by setting. - Combined with `observer=`, it means *that* observer heard it on that setting. - A radio query is served from memory instead of falling back to SQL, which would return the window unfiltered. - `GET /api/radios` (named struct) and `radio_history` on `GET /api/observers/{id}`. **Frontend** - Packet filter fields `radio`, `freq`, `bw`, `sf`, `cr`, e.g. `sf == 8` or `freq > 869.5 && bw == 62.5`. - A Radio row in the packet detail and a Radio column in its observations table (only when the server reports one). - A radio history table on the observer page once there is more than one stretch. - `RfParams.radioText` formats the setting. It escapes all five characters and returns non-numeric input as it came, escaped. ## Performance `txMatchesRadio` runs once per observation inside `filterPackets`. - Each observation costs one map lookup by observer id. - Observers with a single stretch (nearly all) answer from that lookup directly. - Retuned observers add a binary search over their few stretches. - Multi-radio observers add one lookup by observation id. - The snapshot is loaded once per filter call. `BenchmarkTxMatchesRadio`, 30K packets / 150K observations (20 observers, 2 of them retuned): ``` BenchmarkTxMatchesRadio-24 602 2056924 ns/op 0 B/op 0 allocs/op ``` Memory is one entry per stretch plus one per multi-radio observation; nothing is added per observation otherwise. ## Tests - **dbschema:** seed runs once, canonical spellings, the tables exist without `observers.radio`. - **ingestor:** - stretches extend and split; canonical spellings do not split - two radios of one node hearing one transmission over two paths each keep their own radio - a retained status does not write history - prune and purge - `rx_radio_id` reaches `PacketData` - **server:** - resolution before, between and after stretches, and with an unknown radio id - `observer` + `radio`, and grouped rows carry `radio` - incremental refresh, `/api/radios`, `radio_history`, missing tables - the benchmark above - **frontend:** the new filter fields in `tests/unit/test-packet-filter.js`, and XSS cases through the real `RfParams.radioText` and the history renderer. - **E2E:** `tests/e2e/test-radio-setting-1869-e2e.js` checks the grouped rows' radio, `sf ==` on the packets page, the detail Radio row and the observer history. It is added to `scripts/non-unit-tests.json` and to shard 2. `go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema` passes, along with `sh test-all.sh`, `eslint@8` (0 errors) and the XSS diff gate. The new E2E test and `test-e2e-playwright.js` pass against the fixture. ## Browser validation This runs on a live instance with six observers, one of them retuned to 869.618/62.5/SF8/CR8: - The packet detail shows `Radio: 869.432 MHz · SF7 · BW 62.5 kHz · CR 4/5`. - `sf == 7` shows 106 of 106 grouped rows and `sf == 8` shows 0 of 107; the SF8 observer has not heard anything yet. - `/api/radios` lists both settings. - No page errors. ## Notes - **The customizer (rule 8)** gets nothing: the only constants are server-side refresh intervals. - **Attribution timing:** packets heard between a retune and the observer's next status message (minutes) are attributed to the previous setting, because a stretch starts when the status is handled. This is documented at `RecordObserverRadios`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
b5caea1d5f |
fix(analytics): render safe, correctly sized battery status icons (#2282)
Red commit: `4cc2efb3` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40189921192)). Fixes #2232. Battery badges now render their fixed status icon and append voltage as text, including zero. Chart sizing excludes inline icons, and status colors use the existing theme variables. Unknown status values stay muted. - The original browser regression failed on literal SVG; a second local run with only the JavaScript fix failed on oversized icons. - Validation on `eef0872e`: all standalone frontend suites; 47 focused unit checks; 41 Chromium checks, including both themes, hostile voltage, accessibility and 16×16 icon bounds. - Browser verified: http://127.0.0.1:18851 on an isolated Go fixture. - E2E assertion added: `tests/e2e/test-issue-1648-m2-icons-e2e.js:134`. - Constant work for one badge; no new requests, dependencies, settings or schema changes. Privacy, whitespace, CSS-variable and repository XSS checks passed. The external home-directory preflight bundle is unavailable; its applicable checks were performed locally. Independent review found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190620324) passed on `eef0872e`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
d07186af43 |
fix(nodes): preserve names and metadata in preview panels (#2315)
Fixes #2309 and #2312. Partial follow-up to #1154 (U1 and tablet timestamp parity). Node previews omitted Last Relayed, Traffic share and Hash Prefix, while tablet previews read `advert_name` instead of the API's `name`. Shared rendering now presents those fields consistently across full detail, side preview and tablet, with the existing timestamp tooltips. Relay attribution is labelled neutrally: ambiguous relay evidence no longer claims a node is alive. Existing certainty/status logic is unchanged. Measured zero, legacy score fallback, role applicability, missing values and escaped names/hash tooltips are covered. A resize regression prevents duplicate score-row IDs across retained desktop and tablet previews. Only the selected node's metadata is formatted. Existing request counts remain five for full/side and two for tablet; no new fetch, cache, hot loop, setting, customizer value or dependency. Validation: 806 helper checks, all 218 standalone suites, 46 browser checks in both themes, eight health checks, and the main browser suite (149 passes, three existing fixture skips). The unchanged backend tree passed full server/ingestor suites. Independent reviews completed; the parent verified the two-line follow-up from its resize check. Test-only RED `e0f681ae` produced [eight expected assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40232409958/job/121286783705). Follow-up `2bb1ccd` repairs the help fixture's scroll range without relaxing its checks; all 13 help checks pass across three font/CPU configurations. [Final-head CI passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40234768485), including all browser shards and architecture builds. |
||
|
n30nex
|
6e1abf0663 |
ui(nodes): identify path links as example packets (#2296)
Red commit: `79abbba0` ([regression CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203075654)). Fixes #2291; addresses U7 of #1154. The two observed-path links now say View example packet →, accurately describing their existing sample-packet destination. Recent-advert labels and link behavior remain unchanged. - Production diff: two text literals in public/nodes.js. - Existing browser suite preserves contrast/highlighting assertions and adds exact labels, canonical destinations and real click-through from side/full/mobile views. - RED fails eight label assertions. GREEN passes 15/15 checks normally and with 4× CPU slowdown; all 218 standalone frontend suites pass. - Browser verified: http://127.0.0.1:18902. - E2E assertion added: tests/e2e/test-issue-1146-path-link-contrast-e2e.js. Initial loads and theme toggles wait for actual theme-refresh readiness before capture, fixing a reproduced screenshot-detachment race in the touched harness. - No runtime logic, requests, settings, dependencies or performance changes. Independent reviews and local preflight checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204585697) passed on `ae849cc6`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
fb101e46e0 |
fix(nav): fit primary links around late account controls (#2286)
Red commit: `e5589b4c` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191348162)). Fixes #2216. Navigation measures the available flex strip instead of guessing component widths. Optional statistics and spacing yield when primary/active links would overlap More. Account insertion triggers a layout event immediately, including while authentication is pending or fails; normal user-change events retain their semantics. - Regression CI reproduced link overlap across narrow widths and late account controls. Review added a second RED test for the actual delayed authentication bootstrap before the three-line correction. - Validation on `f0ccbf4c`: full frontend suite; 89 user-management checks; 56 navigation checks; existing priority and clipping suites. Parent independently reran auth and browser checks. - Browser verified: http://127.0.0.1:18854; real bootstrap overlap of 14.66px becomes 21.38px clearance. Resizing, account changes, keyboard More and touch targets remain covered. - E2E assertion added: `tests/e2e/test-nav-fluid-1055-e2e.js:89`. - Work is bounded by the fixed navigation link set; events share one animation frame. No polling, observers, new requests, dependencies or settings. Privacy, whitespace, CSS-variable, XSS and ESLint checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. Independent review verified the bootstrap correction with no outstanding findings. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40192737545) passed on `f0ccbf4c`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
c853c64f1f |
feat(ui): identify current views in browser tab titles (#2287)
Red commit: `a62753de` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191439879)). Fixes #2259. Browser titles now identify the current page, Analytics tab, or selected node/packet while retaining the configured site name. A Reach view reads `Node abababababab | Reach | CoreScope`. Detail identifiers are limited to 12 characters; account/reset parameters stay out of titles. - Branding and page context have separate state, so delayed theme loads and edits cannot erase the view or accumulate suffixes. Selection/close paths that use replaceState also update the title. - Validation: all standalone frontend suites; 29 focused unit checks; 16 browser steps covering routes, history, reload, tabs, real row clicks, closing, branding and literal route text. An explicit All time selection keeps the browser test independent of fixture age. - Browser verified: http://127.0.0.1:18853 on an isolated Go fixture. - E2E assertion added: `tests/e2e/test-customize-branding-e2e.js:48`. - Fixed-size work at navigation/selection boundaries; no extra requests, DOM observers, render-loop work, dependencies or settings. Privacy, whitespace, XSS and ESLint checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. Independent review found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40193902834) passed on `4c037db5`, including all three browser shards and both architecture builds. The existing logo test now asserts the full page-and-brand title; its 6/6 checks and the 16/16 title/branding checks also pass locally. |
||
|
n30nex
|
e61ff6e473 |
fix(observers): keep detail responses tied to the current view (#2323)
Fixes #2316 and #2320. Related to #1778; progressive loading remains separate. Delayed observer responses could replace a newer observer or period, overwrite its error/status/calibration, and leave detached charts alive. Each load now captures its observer and days and checks ownership before success or failure can update the view. Leaving the page invalidates pending loads, including same-observer revisits. New visits also reset the request window to the selector's existing seven-day default. Previously, a new page could display seven days while requesting the preceding visit's 30-day window. Production change: 12 added lines and three replacements in one frontend file. No new requests, caches, settings or dependencies; the ownership check is O(1). Existing request deduplication, comparison controls, chart cleanup and current-request errors are preserved. Validation: 11 unit checks, all 219 standalone suites, 32 controlled browser cases across desktop/mobile and both themes, comparison 9/9, and observer-IATA 5/5. Core browser run exited successfully with 149 passes and three declared fixture skips; existing conditional subchecks remain. The unchanged backend tree passed both full backend suites. The parent independently reran the new unit/browser tests. Test-only RED `a73c227f` produced [nine unit and 28 browser assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40239824064). Three independent reviews passed. [Final-head CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40240673369) passes on `ba52d873` integrated with upstream `11ce1235`, including all browser shards and architecture builds. |
||
|
n30nex
|
9f6f318857 |
test(rf-health): wait for airtime badges after grid refresh (#2289)
Fixes #2288. The RF Health airtime test now waits for the requested observer's actual airtime badge. Waiting for any tile and then immediately reading the badge could race with a theme refresh that clears and rebuilds the grid. This is a test-only synchronization correction: two lines replaced, no application changes, and the existing value/warning assertions remain unchanged. No sleeps, timeout increases, skips or retries were added. - Existing RED evidence: [upstream-compatible CI failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190851550/job/121161457287). The test blob is identical on upstream `cb38ac35` and that PR's head. - A controlled local reproduction holds the next summary response during a real theme refresh. The old immediate read fails; the locator read waits for the correct badge. - Four local browser runs passed, including an independent parent run. Wrong-value and missing-warning mutations still fail the original assertion. Test inventory passed. - Browser verified: http://127.0.0.1:18851 on an isolated Go fixture. Privacy and whitespace checks passed. The external home-directory preflight bundle is unavailable; applicable checks were performed locally. Independent reviews found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40193032972) passed on `36d1bf4a`, including all three browser shards and both architecture builds. |
||
|
efitenandClaude Opus 5.5
|
e149568c6c |
feat(observer-logins): personal MQTT logins for observers (#2306)
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.
Closes #2319.
The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.
## Changes in this PR
**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.
**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.
**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.
Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.
**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.
**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.
**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.
**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.
## Enabling it on an instance
Full steps in `docs/observer-mqtt-logins.md`. In short:
1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.
## Who can claim an observer
Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.
What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.
## Performance
`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.
## Verification
- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.
- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.
## Known behaviour
EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.
## Not in this PR
- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.
Note on
|
||
|
n30nex
|
cb38ac356c |
fix(packets): preserve All time across reloads and links (#2279)
Desktop All time now survives reloads and shared links, including a
selected older grouped packet and observation. An explicit zero is
retained in storage and the URL; missing/invalid values still use the
default. Mobile requests retain their existing time and row limits.
Fixes #2231.
Validation: RED reproduces the lost zero; all standalone frontend suites
and the unchanged server suite pass. The local main browser suite
(`test-e2e-playwright.js`) passes 149 checks with three explicit
fixture-dependent skips. The gesture suite also preserves the existing
time window when opening a packet trace. The new regression covers
selection, reload, fresh links, older grouped observations and mobile
bounds.
Normalization is constant work; URL updates reuse the existing
serializer and add no requests. No new settings or dependencies. Common
ingestor validation is covered by the separate #2272 test-stability
change.
[Exact-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40160982457)
passed on commit
|
||
|
n30nex
|
82aafb53c3 |
fix(analytics): count roles within selected filters (#2275)
Analytics Nodes now counts roles from the same complete, filtered node list used by Network Status. Region/area selections no longer display the API's global role totals. The global API metadata remains available to other consumers. Fixes #2213. Validation: RED reproduces the mismatching totals; all standalone frontend suites and the unchanged server suite pass. Browser checks cover region and area changes and empty results. Regressions also cover 501-node pagination, unknown roles, unchanged metadata and request counts. Counting happens inside the existing O(n) status pass, using six bounded counters and no additional requests. No configuration changes. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
n30nex
|
7ebbed58dc |
fix(analytics): anchor resize handles to table headers (#2274)
Resize handles now use their static table header as the containing block. Already-positioned headers, including sticky Nodes headers, retain their existing positioning. Fixes #2217. Validation: assertion-failing RED commit followed by the fix; all standalone frontend suites and the unchanged server suite pass. The browser suite passes 48 checks covering desktop/mobile bounds, real dragging, saved widths and sticky headers. The browser regression waits for the initial theme render before measuring. The fix adds one positioning check per header during setup, with no new requests, dependencies or settings. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
efitenandClaude Opus 5.5
|
0e3ce48cff |
feat(nodes): flag nodes whose LoRa settings differ from nearby nodes (#1268) (#2210)
Refs #1268. #2209 is merged as `cab4f71c`, and master is merged into this branch (`47ce8264`), so the diff against master is this PR's own change: `ea4d1130` plus the fix in `5cbbe5d5`. ## What it does Two nodes hear each other only when SF and bandwidth match and their frequencies are close enough. `/api/rf-params` now also returns `deviations`: nodes whose setting differs from a majority of the nodes around them on the same band. - **Peers:** nodes with GPS and RF params within `rfParams.compareRadiusKm` (default 50) whose frequency is within 30 MHz. - **Same channel:** frequencies within a quarter of the bandwidth count as one channel (15.625 kHz at BW 62.5). The SX1262 tolerates a transmitter/receiver offset of up to 25% of BW without sensitivity loss, for SF5 to SF12 and all bandwidths. Source: the `FERR_L` row in the [S62F module datasheet](https://lora-alliance.org/wp-content/uploads/2020/07/S62F_SPEC_Ver.B.pdf), an SX1262 module; I did not find the figure in a Semtech document itself. That table also lists -50/-100/-200 ppm for SF12/SF11/SF10, which at 869 MHz is wider than BW/4 at 62.5 and 125 kHz, so BW/4 is the tighter bound there. - **Flagged when:** a setting the node cannot hear is shared by more than half of the group, the node itself counted, with at least 3 peers and no other incompatible setting at the same support. So 5 against 5 flags nobody, and 2 of 4 does not count as a majority. Each deviation carries the majority, how many peers are compatible with it, the peer count, and which of freq/sf/bw differ. - **Coding rate is not compared.** RadioLib starts SX126x radios with an explicit LoRa header (`SX126x::begin` sets `RADIOLIB_SX126X_LORA_HEADER_EXPLICIT` at `6d89348`, the commit the firmware pins in `platformio.ini`), and nothing in `firmware/src` switches to implicit. The header carries the CR, so a CR mismatch still decodes. I checked SX126x only, not SX127x or LR11x0. UI: - Nodes list: a badge such as `LoRa SF7` next to the name of a deviating node (the majority in its tooltip), and a `LoRa: differs from nearby (N)` filter kept in the URL as `?lora=differs`. - Node detail: one line, for example "Differs from 12 of 14 nearby nodes on this band, which use 869.618 MHz · SF8 · BW 62.5 kHz". - Badge and filter only exist while the server has map data. ## Why same-band only On our instance (1,110 nodes with params and GPS, measured with the first version of the rule), a plain neighbourhood majority flagged exactly 15 nodes, all of them a deliberate 433 MHz network in Limburg next to the 868 MHz one (names like `NL-ZAY-RP01 | 433 X 868`). The same-band comparison flags none there. On our network this PR shows nothing today. The flags are for instances where a node really is set to a different SF, bandwidth or channel than the nodes around it. ## Cost - Points are bucketed on a radius-sized grid (rows of radius/110.57 degrees, longitude columns widened by 1/cos(lat)) and compared with a squared equirectangular distance. - `BenchmarkRfDeviations1k` (1,100 nodes over Belgium and the southern Netherlands): 4.5 ms. `BenchmarkRfDeviations10k` (10,000 nodes in the same box): 388 ms, 1.3 MB. Both on the first version. - After `5cbbe5d5`, on a slower machine, 3 runs each: 1k 4.4 to 9.5 ms, 10k 821 to 852 ms, 1.8 MB. On that machine the first version measured 7.6 to 9.3 ms and 819 to 861 ms, so the majority step adds no measurable time. It is one pass over the distinct settings around a node, and those are few (one per preset in use). - Inside the radius the work is close to O(n²): 10,000 nodes in one dense box means about 2,000 neighbours per node, and a request after the 5 minute TTL waits for that computation. - The result is cached per map snapshot for 5 minutes; concurrent requests wait for one computation. - On staging `/api/perf` reports `/api/rf-params` at 9.9 ms average, 26 ms max (8 requests). ## Tests - Go: deviation cases (SF, freq within band, BW, CR ignored, other band ignored, no GPS, no params, far away), split vote and too few peers, radius, a peer in another longitude column at 60°N, the cache (same snapshot within TTL, new snapshot, after TTL), the route with four inserted nodes, plus the two benchmarks. - Go, added in `5cbbe5d5`: a 5/5 split and 2 of 4 flag nobody; a 1 and a 15 kHz offset are not deviations while 32 kHz is, and offset peers add up into one majority; the BW/4 boundary; peers two grid columns away at 60°N; a peer at the north edge of the radius. Each fails on the first version. The two-column test also fails with a ±1 column search, which the existing longitude test did not catch. - Full `cmd/server` suite and `test-all.sh` pass locally on `5cbbe5d5`. - Frontend unit: 6 new assertions for `deviationBadge` and the detail line (13 in the file). - E2E: 4 new checks: badge only on the deviating node, filter count and result, `?lora=differs` written and restored, detail line. The disabled check now also asserts no filter and no badges. 8/8 locally. ## Validation On staging with live map data, with the first version: 1,115 nodes served, `deviations: {}`, and the filter shows `LoRa: differs from nearby (0)` with no badges. Because no real deviation exists there, the badge, filter and detail line were checked by serving one simulated deviation through Playwright route interception against the staging page. ## Not done - No map marker for deviating nodes. - `compareRadiusKm` is config-only; the customizer is a later step. The other values are hardcoded and belong in that step too: at least 3 peers, the 30 MHz band, the more-than-half rule, BW/4 and the 5 minute TTL. - Not re-run on staging after `5cbbe5d5`, and the E2E suite was not re-run locally. The fix changes only the server's choice of nodes, not the UI. - The detail line says the peers "use" the majority setting. With the BW/4 tolerance, some of them may be a few kHz off it. - Companions and room servers are compared too, not only repeaters. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
dc98886d92 |
feat(nodes): name and show repeaters known only from the name they reported (#2238) (#2239)
Part of #2238. Path hops through such a repeater (`/api/nodes`, `/api/resolve-hops`, the frontend hop resolver) still show a bare prefix; that part of the issue stays open. ## Summary A repeater that never sends an advert can still tell a nearby companion its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`. This PR stores those answers, resolves names with them, and shows these repeaters on the map, in the nodes list and in search, marked as self-reported. ### Ingestor: store the names - `367ecde1` adds a new `names` subtopic. - It is gated on `clientNodeNames.enabled`, default off. - Answers go into a new table, `client_node_names`: one row per observation, `UNIQUE(target, rx_pubkey, observed_at)`. - **Validation.** - `target` must be 64 hex. - The name is trimmed and rejected when it is empty, over 31 runes (firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8, or contains a control character. - **Retention.** `retention.clientNodeNamesDays` (0 = keep). `config.example.json` sets 30, like the other client tables: a companion only asks a repeater for its name when CoreScope has none (`nameMissing` in CoreDrive's `src/names.js`), so pruning is also how a renamed repeater gets its new name. - **Compatibility.** An ingestor without this change drops `/names` silently, so CoreDrive v1.20.0 works against either version. ### Server: resolve by the reported name - `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and `client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys` (the coverage tooltips) both use it. - **Which name wins.** The advert name, otherwise the newest reported name by `observed_at`. A prefix that matches different pubkeys across the two tables stays ambiguous. - **Only repeaters heard directly.** A reported name counts only once its target has at least one `discover` or `advert` reception in `client_receptions`, keyed by the full pubkey, or by its 8-byte prefix when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names` starts with that prefix (`clientNameHeard`). A name for a pubkey nobody heard adds no candidate, so it cannot make a real node's prefix ambiguous. Without the uniqueness check, a made-up pubkey that shares a heard repeater's first 8 bytes borrowed that repeater's receptions and passed (fixed in `4c569f91`). - **Filters.** The blacklist and the hidden name prefixes apply to both sources. - **Missing table.** A database without the table resolves from `nodes` alone, and the server checks for the table again until the ingestor has created it. - **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?, name?, ambiguous}`. ### Server: estimated position - `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`. It lists every repeater in `client_node_names` that never sent an advert (no row in `nodes` or `inactive_nodes`) and was heard directly at least once (see above). A node with an advert but no position stays out: it is already in `/api/nodes` under its advert name. - **Gated** like the other endpoints on `client_receptions`: 404 when `clientRxCoverage.enabled` is off, because the counts and the estimate come from companion GPS fixes. - **Which receptions count.** Only direct ones: - `discover` and `advert` rows, keyed by the full pubkey, or by its 8-byte prefix when no other known pubkey shares it; - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone. `geo` rows are left out. - **How the estimate is made.** - Receptions within 100 m form one spot, with the best SNR of that spot. - An estimate needs at least 3 spots, at least 200 m apart. - The position is the SNR-weighted centroid (`10^(snr/20)`). The radius is the distance to the farthest spot, at least 500 m. - The response is cached for 5 minutes, and the compute time is logged. ### Map - `5fcde554` adds `public/map-estimated.js`, which draws each estimate on its own layer: - a hollow, dashed "≈" marker and a dashed uncertainty circle; - a popup with an "Estimated position" badge, the method, the caveat that one-sided driving skews the estimate, and a link to the coverage map; - a Display toggle, "Estimated positions": on by default, kept in localStorage, and also hidden by the repeater role filter and in route view. ### Nodes page and search - `12e6b345` adds a "Self-reported repeaters" section under the nodes table, on the All and Repeaters tabs, filtered by the search box. - **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a self-reported detail view replaces "Node not found": - who reported the name; - the receptions; - the estimate, or why there is none; - links to the map and the coverage map. An unknown key still shows "Node not found". - **Header search** adds matching self-reported repeaters, tagged "self-reported". - **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as they are. These repeaters have no packets, so the packet filters, the live view and the channel sender lookup must not match them. ### Docs - `03580437` corrects the `/regions` section of `docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the switch, but the code uses `clientRegions.enabled`. - The new subtopic, endpoint and config keys are documented in the same file. ## Trust Any companion the broker lets publish can report a name for any repeater, and the newest report wins. This is the same model as `/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to linked companions. A report for a repeater nobody heard directly is stored but not used: it is not listed and adds no prefix candidate. A companion can still report a wrong name for a repeater that was heard. The same companion can also publish a fake `discover` reception for a made-up pubkey, so the "heard directly" check filters junk, not a malicious companion; `requireLinkedCompanion` is the control for that (its gate runs before the `/names` handler too). ## Performance - **`lookupPrefixes`** adds one term per key on `client_node_names`. - It matches the prefix as a range, `target >= ? AND target < ?` (upper bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX idx_cnn_target (target>? AND target<?)`, and the reception check is `SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`. - It is still one query per chunk of 200 keys: 400 compound terms, under SQLite's default limit of 500. - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks. - **`/api/nodes/reported`** makes three kinds of query: - one grouped query on `client_node_names`; - one `lookupPrefixes` call for the short hashes; - one `IN` query per 400 keys on `client_receptions`, served by `idx_client_recept_heard_geo`. The spread check is O(spots²) per repeater, with at most one spot per 100 m visited. The result is cached for 5 minutes; concurrent misses wait for one computation, and the response is written after the lock is released. - **The map does not wait for it.** It draws its markers first and adds the estimates when the list arrives. The map and the nodes page share one cached fetch (`ReportedNodes.load()`). - **Not benchmarked.** On the instance where this runs, `client_node_names` holds one row today. I expect at most tens of such repeaters per instance, but have not measured that. ## Notes - **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2 occurrences: the handler's `msg` parameter and its `gps` field. That is the decoded MQTT message, passed in the same shape as `handleClientPacket` and `handleClientRegions` already take it. No API response or internal structure uses it. - **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min TTL. They should move to the customizer later (rule 8). - **Map filters.** The estimates follow the "Estimated positions" toggle and the repeater role filter, not the area, region or last-heard filters. The toggle's tooltip says so. - **Name characters.** Names are rejected for control characters but not for Unicode format characters (bidi overrides, zero-width joiners), because emoji sequences in node names use the zero-width joiner. All output goes through `escapeHtml`. - **Not done:** the home-page search box (`home.js`, `/api/nodes/search` with a Claim button) does not list these repeaters. Claiming a node without adverts would have nothing to show. ## Tests On this branch (based on `550829af`): - `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and `cmd/ingestor`. - **New Go tests:** - handler and validation cases; - idempotent redelivery; - newest `observed_at` wins; - prune; - the gate on and off, without falling through to the observer path; - client-only node, nameless `nodes` row, advert name wins, ambiguity across the two sources; - hidden and blacklisted nodes; - absent-then-created table; - spot merging, threshold, weighted centroid and radius floor; - `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an advert position and a missing table; - an advert without a position, a never-heard name, a name heard only in `rxlog` paths, and the `clientRxCoverage` gate (404). These fail on `12e6b345` and pass at head. - a made-up pubkey sharing a heard repeater's 8-byte prefix, and two reported targets sharing one (`4c569f91`). Both fail without that commit. - **New frontend unit suites:** `tests/unit/test-map-estimated.js` and `tests/unit/test-reported-nodes.js` pass. Both load the real `escapeHtml` from `app.js`. - **`test-all.sh`** passes, all suites (with Python available for the XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the four changed frontend files. - **E2E**, against a static server with a mocked API: - `test-map-estimated-e2e.js`: 6/6, including the repeater role filter and route view hiding the estimates. Mutating either hide makes its step fail. - `test-reported-nodes-e2e.js`: 6/6. - `test-issue-1150-404-state-e2e.js`: 5/5. - **Real data, API only.** On the fork's staging, `/api/nodes/reported` lists one repeater: 5 spots, 17 receptions, an estimate with a radius of 2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays listed under the stricter rules. Not re-checked after `4c569f91`. - **Not done:** a check of that repeater in a browser. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9534e1c25e |
fix(reach): retry a busy (429) reach report in the reach page (#2250)
Follow-up to #2127: @dborup's point 2 there, and @efiten's non-blocking note on the approval ("worth checking the node page's reach panel shows a retry hint on 429 rather than a generic error"). ## What changes - **The reach page retries a 429 itself.** It shows "Reach is busy computing other reports, retrying in N s", waits `Retry-After` seconds, and tries again. - When the header is missing or is not a number of seconds (an HTTP-date, 0, a negative number), it waits 2 s. It never waits more than 30 s. - After 5 retries it shows the old error ("Failed to load reach: API 429 …"). - Leaving the page stops the retries, both while it waits and while a request is still in flight. Picking another window while a request is in flight also drops that request's retries. - **`api()` does not retry 429.** Login, signup, activation, email change, companion linking, settings PUT and channel proposals also answer 429 from their own limits, and those must not auto-retry. `api()` now puts `status` and `retryAfter` on the errors it throws, so a caller can decide; the message text is unchanged. A unit test pins that `api()` makes exactly one request on a 429. - **Fix in `api()`:** the `_inflight` cleanup chain (`promise.finally(...)`) rejected unhandled on every failed call, so each API error also logged "Uncaught (in promise)". That would have added up to 5 more per busy reach load. The chain now catches; callers still get the rejection through the returned promise. This works with the server as it is on master (instant 429, `Retry-After: 5`), and with the bounded wait in #2249 (429 after 3 s, `Retry-After: 2`). Neither PR depends on the other. ## What I checked - **Suites.** `sh test-all.sh` passes, the XSS sink gate passes, and eslint has 0 errors and no new warnings. - **Unit tests** (`test-frontend-helpers.js`): - `reachRetryDelayMs`: Retry-After values; fallback on missing, zero, negative or HTTP-date values; the 30 s cap; giving up after 5 retries; no retry on 400, 404, 500, 503 or network errors. - `api()`: a 429 error carries `status` and `retryAfter`, makes exactly one request, and leaves no unhandled rejection. - The block sits above the file's `Promise.allSettled(pendingTests)` summary. Async tests registered below that line are never awaited, so they cannot fail the run; nothing below it is async today. - **E2E** (`test-node-reach-e2e.js`, already in CI suite 2): reach requests are answered through `page.route`, so the server does not have to be made busy. It checks: - two 429s, then the real report: the hint shows "retrying in 1 s", the retries wait (at least 1.8 s between the first and third request), and no error is shown; - always 429: 6 requests (1 + 5 retries), then the error; - leaving while it waits: no more requests; - leaving while the first request is still in flight: no hint written, no retry; - a 500 with Retry-After: one request, no retry. On master's frontend the new section fails at the first busy hint. Locally against the freshened `e2e-fixture.db`, the whole file takes about 17 s (2 runs). - **Mutations.** I wrote 15 from the code, and the tests caught all 15. They include: `api()` drops `status` or `retryAfter`, `api()` retries 429 itself, retry any error, retry limit off by one, fallback or cap changed, Retry-After 0 accepted, either "left the page" check removed, the wait not honoured, the hint showing ms, the retry count ignored, no error after giving up, and the cleanup chain unhandled again. Each mutation was checked against the file the server actually served before the tests ran. - **Real browser** (Chromium via Playwright, on a copy of a 10.5M-observation database): - Current master's server, 6 tabs opened at once on uncached 14-day reports: 4 got 429 with `Retry-After: 5` and showed the hint, and all 6 reports rendered within 11.5 s (one tab retried twice). Without this change those 4 tabs show "Failed to load reach: API 429". - With #2249's server: 4 of 6 got 429 with `Retry-After: 2`, and all rendered within 8.1 s. - The only console errors were Chrome's own "Failed to load resource" lines for the 429s; there was no "Uncaught (in promise)". ## Not checked - **Mobile layout.** The hint reuses the existing `.nq-load` style, so I did not take a screenshot at phone width. - **Other browsers.** Firefox and Safari were not tried; the code is plain `setTimeout` / `await`. - **Window switches.** While the hint is showing, the time-range buttons are not on screen; the same is true of the existing error and loading states. Picking a new window means waiting, or going back and opening reach again. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: nullrouten0 <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
d43f3c3777 |
feat(theme): ?theme=dark|light URL parameter for one page load (#2236) (#2251)
Fixes #2236. A display that can only be given a URL (digital signage, an iframe without cookies or login) had no way to choose dark or light. This adds `theme=dark` or `theme=light` to the hash query, next to `embed=1`: ``` https://<instance>/#/map?embed=1&theme=dark ``` ## What changes **`public/index.html`, the pre-paint script** - Reads `theme` from the hash query with `URLSearchParams`, as `shouldEmbedRoute` does for `embed`. Only `dark` and `light` count; anything else is ignored. - A valid value wins over the saved choice and the OS preference, and is set before first paint, so there is no light flash. - The value is handed to `app.js` as `window.__themeOverride`. - When `localStorage` throws (a sandboxed iframe), the URL value is still applied. **`public/app.js`, the dark-mode setup** - On load, a URL value is applied through `applyTheme(theme, false)`, which does not write `meshcore-theme`. The saved choice, and the account settings that `settings-sync.js` syncs from that key, stay as they were. - Theme changes from other tabs or from settings sync (the `storage` listener) are ignored while the URL value holds. Otherwise a logged-in display would flip within a minute. - A click on the dark-mode toggle ends the URL value and is saved as before. It works on every page, not only on the two `embed=1` pages. The value is read once per page load, so later hash changes (navigation, map viewport) keep it. ## Tests | File | Assertions | Covers | |---|---|---| | `tests/unit/test-theme-url-param-2236.js` | 8 | Runs the real pre-paint script from `index.html`: `dark` and `light` win over a saved choice and the OS, nothing is written to `localStorage`, any page and position in the query, an unknown value, no param, `localStorage` throwing | | `tests/e2e/test-theme-url-param-2236-e2e.js` | 6 steps | `#/map?embed=1&theme=dark` over a saved "light" (dark, embed, saved still "light"), a reload without the param, `theme=light` over a saved "dark", a storage event while the override holds, the toggle ending it, `theme=blue` | How I checked them: - With `index.html` and `app.js` from master, 6 of the 8 unit assertions fail, and the E2E fails at step 1 ("expected dark + embed + saved light"). - With only the `app.js` change reverted, the E2E fails at step 1. With only the `if (themeOverride) return;` line in the `storage` listener removed, it fails at step 4. - `sh test-all.sh` passes, except `test-issue-1956-release-routing.js`, which I skipped: it needs the workflow's `GITHUB_REF_NAME` and does not run in local git-bash. `npx eslint public/app.js` reports 0 errors (18 warnings, the same count as master). - E2E on a fixture seeded as in CI: | Suite | Result | |---|---| | `test-e2e-playwright.js` | 148 passed, 3 skipped | | `test-customize-theme-e2e.js` | 8/8 | | the new #2236 suite | 6/6 | - `test-logo-theme-e2e.js` fails locally with "wordmark `<text>` overflows SVG viewBox", with the same numbers on master. It measures font metrics, which differ on this Windows machine. The new E2E is registered in `scripts/non-unit-tests.json` and `deploy.yml` (shard 2, next to the logo theme suites); the unit test is in `test-all.sh`. ## Not done - No screenshot: the E2E checks `data-theme` in headless Chromium, not how the page looks. - `live.js` Matrix mode forces dark and, when switched off, saves the theme it restores. With a URL value active, turning Matrix off saves that value as the stored choice. I left this as is, because a display that only shows a URL does not toggle Matrix mode. - Only `theme=`; the `?darkMode=1` form from the issue text is not supported, so there is one form to document. - I placed the E2E in shard 2 without measuring suite times. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
3960fadd96 |
fix(pages): drop late responses that belong to an earlier page visit (#2218, #2230, #2200) (#2248)
Fixes #2218, fixes #2230, fixes #2200. Node analytics, the account page and analytics await their data, then write into the shared `#app` container. A response that lands after the user has navigated away replaces the new page. In #2218's reproduction, the Nodes list, with 200 rows, became an Analytics heading. ## The shared part `public/app.js` keeps one view generation, `viewGeneration()`: - `navigate()` bumps it before the old page's `destroy()`, so every page init counts as a new visit, including a switch to another node of the same page type. - A page captures the value at init, and after each `await` drops the result when the value has moved on. `.eslintrc.json` declares it as a global. ## Per page **#2218, `node-analytics.js`** - `loadAnalytics` and `loadBatteryChart` check the generation on both success and error. A page-local `loadSeq` makes a time-range click a new load on the same visit, so an older range can't land over a newer one. - `NodeHopAnalytics.load` already checked `container.isConnected`, so it is unchanged. **#2230, `account.js`** - The render after `CSAuth.ready()` runs only while the generation is unchanged. - `destroy()` was empty. It now clears `shown.app` and `shown.userId`. **#2200, `analytics.js`** - A page-local `_loadSeq` sits next to the generation. Results and errors of an older window/region selection or an earlier visit are dropped, so a late failure of an old load no longer shows an error over newer results. - The bulk load renders only the six tabs drawn from its data. A tab that loads its own data (Prefix Tool, Nodes, Roles, ...) is no longer rebuilt or replaced by the bulk result: a deep-linked one renders at init, and a window, region or area change re-renders it directly. - `renderTab` guards the six tabs drawn from the bulk data (overview, rf, topology, channels, hashsizes, collisions). Until the data arrives they show "Loading analytics…", or the current load's error. Overview no longer dereferences an undefined `rf`. - Three related fixes in the same lines: - The error message is now escaped with `esc()`; it was inserted raw. - `destroy()` removes the RegionFilter and AreaFilter listeners. They piled up on every visit and called `loadAnalytics` from other pages. - `destroy()` resets the stored error. ## Tests | File | Assertions | Covers | |---|---|---| | `tests/unit/test-issue-2218-node-analytics-stale-render.js` | 12 | Loads the real router: the generation rises on every init, including another node. Late success and error after leaving, a stale visit of the same node, an older range landing last, late battery success and error. The current response, battery and error still render | | `tests/unit/test-issue-2230-account-stale-render.js` | 6 | Leaving before ready (login view and "not enabled"), re-entering before ready, the current render, an auth change after `destroy()` | | `tests/unit/test-issue-2200-analytics-stale-load.js` | 19 | An older selection landing last, a late rejection of an older selection, leave and re-enter (success and error), the current error rendered escaped and kept on a tab click, Overview and RF clicked before the data | | `tests/e2e/test-issue-2218-node-analytics-stale-e2e.js` | 4 steps | Holds `/api/nodes/*/analytics` with `page.route`, returns to `#/nodes`, then releases the response as a 200 and as a 500. The 200 rows stay and no Analytics heading appears; a normal visit still renders | How I checked them: - With the production files reverted and the tests kept, 7 of 12, 4 of 6 and 7 of 12 unit assertions fail. With the fix, all pass. - The E2E fails on the unfixed frontend with "Nodes rows 200 before release, 0 after". - Single mutations, each reverted: - removing the success check in `loadAnalytics` (analytics) fails 2 assertions; - removing the check after `CSAuth.ready()` fails 3. Other test changes: - `test-analytics-theme-refresh-before-data.js` and `test-user-management-ui.js` get a `viewGeneration: () => 0` stub. - The new E2E is registered in `scripts/non-unit-tests.json` and `deploy.yml` (shard 2, next to the #748 suite). The unit tests are in `test-all.sh`. - The broad suite's "Node side panel Details link navigates" test selected an anchor that now opens Analytics, as #2218 notes. It now clicks `#nodesRight .node-detail-btn`. Results: - `sh test-all.sh` passes, and `npx eslint public/*.js` reports 0 errors. - E2E on a fixture seeded as in CI: | Suite | Result | |---|---| | `test-e2e-playwright.js` | 148 passed, 3 skipped | | `test-user-management-e2e.js` | 28/28 | | the new #2218 suite | 4/4 | | `test-issue-748-node-analytics-coverage-e2e.js` | 2/2 | | `test-analytics-fluid-charts.js` | 8/8 | | `test-issue-1657-analytics-channels-group-sprites-e2e.js` | 5/5 | ## Not done - Two tabs that each fetch their own data can still overwrite each other after a fast tab switch between them. #2200 does not ask for that. - `nodes.js` `navigateToNode` calls its own `init` without the router, so it does not bump the generation. None of the three pages fixed here depends on that. - I placed the E2E in shard 2 without measuring suite times, which the `deploy.yml` comment asks for. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
CoderNemesisandClaude Opus 5.5
|
e55c95db24 |
1-Byte Hop Path Inference for Packet and Map Analysis (#2245)
## Summary On meshes with 1-byte path hashes, most hop prefixes are shared by several repeaters. The packet detail pane currently resolves each hop on its own and stores the result by prefix. Two things go wrong as a result: - A prefix that appears twice in one path resolves to the same node both times. - Nothing checks that the chosen hops form a path a packet could actually take. This PR adds `HopResolver.inferPath()`, which infers the most likely node behind every hop of **one observation's path**, scoring the path as a whole and anchoring it on the observer that heard it. The packet detail pane and the map route view now use it, so both show the same nodes for the selected observation (`?obs=`). ## The problem, on a real packet Packet `57da37d1636f895a`, observation `1388126`: a 46-hop flood with 1-byte hashes. The server's `resolved_path` resolved 1 hop of 46, so the client resolver did the rest. Eight prefixes appear twice in the path. | | Current resolver | `inferPath()` | |---|---|---| | Hops longer than 120 km | 13 (21 longer than 60 km, two over 950 km) | 0 | | Distance from the last hop to the observer | 59 km | 0.1 km | | Hops reported as unlisted (no listed node fits) | 0, every hop guessed | 13 | | Agreement on the shared path prefix across the packet's 23 observations | — | 97.8% | The current resolver picks badly for three reasons: 1. **Results are keyed by prefix.** Hop 0 and hop 45 share a prefix, so both show the node near the start of the path, 59 km from the observer. 2. **Picks are greedy.** Each hop looks only at its nearest resolved neighbours, so nothing penalises a 1,100 km jump. 3. **The detail pane read the wrong `resolved_path`.** It used the packet row's, not the selected observation's, and when the lengths matched it never fell back for the hops that were `null`. ## Firmware basis The model relies on two facts from `firmware/src/Mesh.cpp`: - **The last hop is the one the observer heard.** `Mesh::routeRecvPacket` appends the relaying node's own hash to the end of a flood path. - **A node appears at most once in a flood path.** A node forwards a given packet once: `_tables->wasSeen()` / `markSeen()`. ## How the model works Each hop is either one of the **located** repeaters or room servers sharing its prefix, or **unlisted**: a node missing from the node list, or one without GPS. With 1-byte prefixes, unlisted is common, and naming it beats drawing a 300 km hop. | Term | Cost | |---|---| | A hop between two located nodes | distance / 50 km; past 100 km per hop, the cost rises 4× faster | | A hop with a neighbour-graph link | the distance cost × 0.5 | | An unlisted hop | 1.6, flat (about one 80 km hop) | | The last hop to the observer | the same distance cost; falls back to the observer's IATA region centre (with 50 km slack) when the observer reports no GPS or 0,0 | | The first hop to the sender | the same distance cost, when the sender's position is known (e.g. an ADVERT) | - **Search:** dynamic programming over (hop, candidate) finds the cheapest sequence. When the best path uses a node twice, the node is banned from whichever position costs less to give up. - **Confidence:** each hop's confidence comes from the min-marginal costs of its alternatives, put through a softmax. A node the path already uses at another hop is not counted as an alternative. - **Server hint:** the observation's own `resolved_path` restricts a hop to the named node, but that hop can still be reported unlisted if the node doesn't fit. The tuning values live in `PATH_MODEL` in `hop-resolver.js`, with a TODO to expose them in the customizer (AGENTS rule 8). ## What changes in the UI **Packet detail pane** (`packets.js`): - The Path row, the cross-observer aggregate row and the byte breakdown render hops **by position**, from the inferred entries. - The detail pane no longer writes into the list's shared per-prefix cache (`hopNameCache`), so it can't change what the packets list shows. **Hop display** (`hop-display.js`): | Hop state | Display | |---|---| | Confident (≥ 70%) | The node's name; the tooltip says `NN% likely` and how many nodes share the prefix | | A guess (< 70%) | Dashed underline, plus a candidate badge when there are alternatives | | Unlisted | The hex prefix in muted text; the badge lists the rejected candidates | | The only known node, without GPS | The name, with `(no GPS)` | The candidate popover lists each candidate with its likelihood, or `hop N` when the path uses that node elsewhere. The "Path symbols" legend gains an entry for unlisted hops. **Map route view** (`map.js`, `route-view.js`): - The `#/map?packet=…&obs=…` deep link infers the chosen observation's hops with the same shared helper, so the map and the detail pane agree. - Unlisted hops render as `unlisted` placeholders rather than as the nearest node that happens to share the prefix. - When inferred hops are available, the `/api/resolve-hops` request is skipped. **Shared helpers** (`packet-helpers.js`): - `inferObservationPath(obs, observerPos, originPos)` - `ensureHopAffinity()`: loads the neighbour graph once per page, cached by `api()` for 5 minutes. - `routeHopFromInferred(hop, entry, role)` **Cleanup:** removed dead hop resolution from the "View route on map" button handler. It computed hops, sender and recipient, then navigated by URL and discarded them. **Unchanged:** the packets list, live view, node pages and `HopResolver.resolve()`. ## Performance AGENTS rule 0 asks for proof of performance claims; the figures follow. - **Complexity:** O(hops × 8 × candidates²). Hops more than 8 positions apart aren't compared directly; a longer run of unlisted hops costs a flat amount, tracked with a running minimum. Hop-to-hop costs are computed once and reused by every pass. - **Where it runs:** once per detail render (plus once for the aggregate row) and once per map deep link. Nothing runs in the list, live feed or WebSocket paths. - **Measured** (Node 22, warm JIT): | Case | Time | |---|---| | 64 hops × 16 candidates, synthetic worst case | 9–14 ms median | | Real 39–48 hop paths, 1,153 nodes (23 observations) | 1–12 ms each | - **Enforced:** `test-hop-path-inference.js` asserts a median under 40 ms (target 20 ms) for the 64 × 16 case. It times the module outside Node's `vm` sandbox, because the sandbox makes global lookups about 10× slower than a browser. - **Network:** one extra bulk request per page life (`/api/analytics/neighbor-graph`, cached server-side, TTL 5 min). No per-hop or per-packet requests. ## Tests **New:** - `tests/unit/test-hop-path-inference.js` (37 assertions). The model itself: - a repeated prefix resolves per position, and no node appears twice; - the observer anchors the last hop, with the IATA fallback when the observer has no position or reports 0,0; - a far-away lone candidate is reported unlisted; - a neighbour-graph link breaks a tie; - the server hint and the origin anchor are applied; - a node without GPS is handled; - output shape, and the performance gate. - `tests/unit/test-hop-path-detail.js` (32 assertions). Rendering and wiring: - inferred-hop rendering and the popover; - positional `renderPath`, including with the 1-byte hop filter on; - `renderDetail` infers the shown observation from its own `resolved_path` and leaves the list cache alone; - the neighbour graph is fetched once; - the shared map helpers, and source checks on the map wiring. - `tests/e2e/test-hop-path-inference-e2e.js` (6 steps, shard 1). Runs on fixture packet `57224a4a01c9c682`, observation 147, a 22-hop path that repeats one prefix three times: - every hop is inferred; - the repeats resolve per position; - the popover shows likelihoods; - the map's route list matches the detail pane hop for hop. **Updated, because they pinned the old behaviour:** - `test-issue-1418-deeplink-hops-channels.js`: it tested a copy of the old map priority logic. It now tests the real `inferObservationPath()`, including `resolved_path` as a hint. - `test-issue-2098-last-hop-region.js`: the detail pane no longer writes the prefix cache. The test now reads the region-anchored last hop from the rendered Path row. - `test-issue-2097-hop-ambiguity-badge.js`: the source check allows the new `entries` option while still asserting that the detail pane never uses the summarised form. - `.eslintrc.json`: declares the three new shared helpers as globals. **Results on this branch:** - `sh test-all.sh`: all standalone frontend suites pass. - `npx eslint public/*.js`: 0 errors. All 85 warnings are pre-existing; none come from code this PR adds. - The "Run Playwright E2E tests (fail-fast)" step was run locally with its script taken verbatim from `deploy.yml` (`bash -eo pipefail`, fixture prepared as in CI): - shard 1: 27 of 27 passed; - shard 2: 47 of 47 passed; - shard 3: 50 of 50 passed. Note for reviewers: `test-issue-1692-packets-init-parallel-e2e.js` only passes while the freshened fixture's newest packets are less than 15 minutes old. The test sets the time window in `localStorage` after the page has already read it. CI meets that condition because shards start right after freshening. Unmodified `master` fails the same way when run late. **Browser validation:** the local server ran on the fixture DB. Checked in the browser: - the Path row, the candidate popover and the byte breakdown; - the map route view, which matched the detail pane hop for hop with unlisted placeholders; - the console showed no errors. ## Known limitations and follow-ups - **TRACE packets are not modelled yet.** A TRACE's path is its *planned* route. The header path byte counts the hops actually forwarded (repeaters append SNR, not their hash, `Mesh.cpp`), and a trace may revisit a node on its way back (see the comment in `Packet::calculatePacketHash`). A follow-up will add a `heardAfter` / `allowRepeats` option, so a trace is anchored after the hops it actually completed and may repeat nodes. - **Customizer (AGENTS rule 8):** the `PATH_MODEL` tuning values are hardcoded for now. The unlisted cost, typical and maximum hop length, and the confidence threshold should become customizer settings. - **Server-side `resolved_path`:** the ingestor writes a hop as resolved whenever its prefix is unique in the node table, even when that node is hundreds of km from the neighbouring hops. `inferPath()` treats these as hints and can overrule them, but the stored data is still misleading. - **The first hop without a sender position is often a close call.** On encrypted channel messages, hop 0 is constrained only by what follows it, so 55/45-style splits between a long chain of known nodes and "unlisted" are expected. The badge and the percentage show this rather than hide it. ## Screenshots This is what a normal corescope instance would show for a 1 byte path. <img width="318" height="338" alt="image" src="https://github.com/user-attachments/assets/4056bc27-0b37-4107-9b25-31ac0f7864ec" /> <img width="430" height="601" alt="image" src="https://github.com/user-attachments/assets/01da5ee8-babf-4e41-a9ab-836b5d640549" /> Now the same packet but with the logic to infer each hope along the path different nodes are chosen as they're ranked and selected based on proximity to the next hop, <img width="436" height="646" alt="image" src="https://github.com/user-attachments/assets/bcca73b8-daf0-4031-8ea1-8c96d716c8b3" /> <img width="436" height="646" alt="image" src="https://github.com/user-attachments/assets/fc926d9e-158c-48ce-bee6-93086afb0775" /> The higher the confidence level then there's no need to show the exclamation mark to show that multiple nodes share the same prefix but it does continue to share that Examples of it currently running can be found at, https://corescope.baymc.org - modified instance https://analyzer.comchan.net - unmodified instance https://analyzer-stage.comchan.net - modified instance --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
2d09335cfb |
fix(packets): keep the node filter pick on a slow click or late search (#2243) (#2244)
Fixes #2243. ## Problem The packets node filter selected a suggestion on `click`, while the input's `blur` handler hides the dropdown 200 ms after focus leaves. A press held longer than that, or a search response that rebuilt the list between mousedown and mouseup, lost the pick: the typed text stayed and no `node=` filter was applied. ## Change - `public/packets.js`: suggestions select on `mousedown` with `preventDefault`, so the input keeps focus and the pick happens before any rebuild. `live.js` already does this for its node filter. - `public/packets.js`: a sequence counter drops search responses for an older query, bumped on every search and on a pick, so a late response can't replace or reopen the list. - `tests/e2e/test-packets-node-filter-e2e.js` (new), registered in `scripts/non-unit-tests.json` and on shard 1 in `deploy.yml`. ## Tests The new E2E has three cases: a normal click, a 400 ms press, and a search response released mid-click. | Code | Normal click | 400 ms press | Response mid-click | |---|---|---|---| | master | pass | fail | fail | | this PR without the sequence guard | pass | pass | fail (list reopens) | | this PR | pass | pass | pass | Run locally against the freshened e2e fixture. `sh test-all.sh` passes (locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and `PYTHONUTF8=1`, unrelated to this change). ## Not done - Full `test-e2e-playwright.js` not run locally. - Could not reproduce on the reporter's instance with the browser tool, whose clicks are shorter than 200 ms. The 400 ms Playwright press reproduces it on the fixture. - `live.js` and `packets.js` still carry two separate node-autocomplete implementations. Merging them is out of scope here. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
127dd511a7 |
fix(health): judge favorites and home status cards by the node's role (#2215)
## The problem A starred repeater can show two different health statuses. The My Repeaters tab (#1761) and `getNodeStatus` judge it by the infra thresholds (degraded after 24 h, silent after 72 h). The favorites dropdown and the home page cards judge every node by the node thresholds (degraded after 1 h, silent after 24 h), because they never look at the node's role. | repeater last heard | favorites / home cards (before) | My Repeaters tab | after this PR | |---|---|---|---| | 5 h ago | degraded | healthy | healthy | | 30 h ago | silent | degraded | degraded | | 80 h ago | silent | silent | silent | If you set `healthThresholds` in `config.json`, every view follows your values. ## The change Three call sites now use `getHealthThresholds(role)`, the helper the My Repeaters tab already uses: - `renderFavDropdown` in `app.js` - the "My Nodes" grid card in `home.js` - the hero search-result card in `home.js` The search card's fallback text, "Not heard in 24+ hours", was only true for non-infra nodes, so it now reads "Not heard recently". ## Scope - I kept the existing three-state healthy / degraded / silent logic and changed only which thresholds apply. - Not included: counting a repeater that is relaying traffic as alive (#1598, which `getNodeStatus` does). That is a bigger behavior change and would be its own PR. ## Risk Low. It is 9 lines in two files, and the helper and thresholds already exist and are used elsewhere. ## Tests New `tests/unit/test-health-card-role-thresholds.js`, added to `test-all.sh`. It checks that infra roles get looser thresholds than nodes, and that each of the three call sites goes through `getHealthThresholds(role)` with no node-only comparison left. On current `master` 5 of its 7 assertions fail; with this change all pass. `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning from an a11y self-test, which it also does on `master`. ## Update after review (commit `0f2376df`) The text above is the description efiten reviewed on 2026-10-15. This section describes the code as it is now. ### The change - New `getHealthStatus(role, heard, now)` in `roles.js`. It returns `healthy`, `degraded` or `silent` by the role's thresholds, or `null` when there is no usable timestamp. It only applies the cut-offs; each caller picks the timestamp. - All six places that compared an age against `degradedMs` / `silentMs` now call it: - the favorites dropdown (`app.js`); - the "My Nodes" card and the home health card (`home.js`); - the analytics network status and My Repeaters (`analytics.js`); - the live node panel (`live.js`). Each keeps its own labels and timestamp, so the last three show the same as before. `analytics.js` and `live.js` no longer guard with `typeof getHealthThresholds`. `roles.js` loads before both, and both already used it without a guard elsewhere: the old network status called `getHealthThresholds` directly, and `live.js` reads `ROLE_COLORS`. - The favorites dropdown and both home cards now fall back to the node's `last_seen` when the packet store has no packet for it. My Repeaters (`last_heard || last_seen`) and the node page (`nodes.js:729`) already do this. Without it, with a packet store window under 72 h, a repeater could be silent on the cards and degraded on My Repeaters. The "ago" text uses the same timestamp. With the default thresholds, the fallback changes a status only when the packet store holds less than 72 h of packets. `retentionHours` unset (unlimited) or at the example's 168 h holds more than that. Otherwise a node with no packet in the window is silent either way, and the views now show how long ago it was last seen instead of "never" or nothing. - With no timestamp at all, the home health card said "Not heard in 24+ hours". It now says "No packets in the current window". ### Why a new helper and not `getNodeStatus` `getNodeStatus` has two states (active / stale) and counts a repeater that relays traffic as alive (#1598). Using it here would change more than the thresholds. `getHealthStatus` keeps these views' three states. ### Scope - Not included: counting a repeater that relays traffic as alive (#1598) in these views. That is a bigger behavior change and would be its own PR. - The live node panel still judges by `last_seen` alone, and the node page stays relay-aware. Both now use the shared thresholds; aligning their timestamps would also be its own change. ### Tests - `tests/unit/test-health-status.js` runs `roles.js` in `vm`. It covers: - repeater, room, companion, sensor, an unknown role and a missing role, at ages either side of 1 h, 24 h and 72 h; - ISO and epoch input; - null, empty and invalid timestamps; - a `healthThresholds` override through `/api/config/client`. One source guard remains: no age-vs-threshold comparison outside `roles.js`. - `tests/unit/test-my-repeaters-dashboard.js` now uses the real helper instead of a stub, which gave repeaters the node thresholds. It also checks that a repeater heard 30 h ago is Degraded. - New `tests/e2e/test-health-status-e2e.js` mocks `/api/nodes/<pk>/health` for six nodes. It checks five views: the favorites dropdown, the "My Nodes" card, the home health card, the analytics network status and the live node panel. It fails on `master` and on this PR's first commit. I put it on shard 2, which was the shortest shard in the last CI run. - Mutation check: I made 33 deliberate breaks. In the helper they cover each comparison and boundary. At each of the six call sites they cover the timestamp field, the role argument and at least one label mapping. Each one makes the suite fail. - I checked a `config.json` with `infraDegradedHours: 2` and `infraSilentHours: 6` in Chromium. Repeaters heard 3 h and 7 h ago show Degraded and Silent. - `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning, which comes from a deliberate fixture in the a11y self-test. ### Cost The analytics network status loops over every node. In headless Chromium, 30,000 synthetic nodes take 9.0 to 9.2 ms before this change and 9.1 ms after (median of 21 runs, on one machine). ### Risk Low. The change is frontend only, with no API, config or data change. Rolling back means reverting the commits. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
927a8eec8a |
fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211)
Refs #748 (M3a: correct numbers within the in-memory window; the SQL history in M3b is a separate plan, see the end). ## Problem The node analytics range picker offers 24h, 7d, 30d and All (365d), but the store only holds its in-memory window (`retentionHours`, then eviction). `GetNodeAnalytics` still divided availability and packets/day by the requested range. Measured on our live instance (14-day window), repeater BE-BRE-ON8AR: | Range | Availability | Packets/day | |---|---|---| | 7d | 88.7% | 91 | | 30d | 42.6% | 265 | | All | 3.5% | 22 | Nothing on the page said that the data stopped at the edge of the window. ## What changes - `coverageStart` returns the edge of the in-memory window: the latest of `oldestLoaded` (background-load progress), `now - retentionHours`, and a new `evictedThrough` that eviction sets to the newest `FirstSeen` it removed, so memory-pressure eviction inside the retention window moves the edge too. - `s.packets[0]` is not that edge, and the first version of this PR used it. Chunks are selected by `last_seen`, so a transmission first heard weeks ago and heard again recently sits at the head with its old `FirstSeen`. On staging (7-day window) it returned 21/07. - When the edge lies more than a minute after the requested start, node analytics counts only transmissions from the edge, computes availability and packets/day over that span, and returns `timeRange.dataFrom` and `timeRange.partial`. - The minute of slack keeps a range equal to the retention window from reading as partial. Without it every 7d request on staging (168h) was partial, because the two edges came from separate clock reads. - The analytics page shows "Data from 2026-10-07: the server keeps about 7 days of packets in memory. Availability and packets/day cover that period, not the full 365 days." when partial. - **UTC fix.** The window bounds of node analytics and node hop analytics are formatted in UTC. They are compared as strings with UTC `FirstSeen` values, so on a non-UTC host the 7d window lost its oldest hours (2 at +02:00). Docker images run in UTC, which is why this did not show in production. - `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the edge is kept. ## Tests - Go, `node_analytics_coverage_test.go`: - full coverage within memory; - 30d over 3 days in memory; - empty store; - a re-heard 85-day-old transmission at the head of `s.packets` (edge stays at retention, packet not counted, no bucket before `dataFrom`); - memory-eviction edge; - eviction records `evictedThrough`; - retention 30 s shorter than the range is not partial, while 30d is; - window bounds under a +02:00 `time.Local`. - Two tests were checked to fail on the old code: the local-zone test (166 of 168 packets, a `+02:00` bound) and the retention test without the slack (`dataFrom 09:57:08` vs `from 09:56:38`). - Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). One run of five reported `FAIL` and four reruns were green; I did not capture which test failed. - Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js` (5 assertions on the real `node-analytics.js` in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`), against the real fixture response with no interception: 7d is not partial and shows no notice; All is partial and the notice names the real start date. 2/2 locally. ## Validation Staging (retentionHours 168), same repeater, after the background load completed: | Range | Before | After | |---|---|---| | 7d | 82.7% | 83.3%, not partial | | 30d | 19.3% | 83.3%, partial from 2026-10-07 | | All | 1.6% | 83.3%, partial from 2026-10-07 | In the browser the notice appears under All and not under 7d. ## Not done - M3b, older history from SQLite. In the database only the sender is indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay hops only in `observations.resolved_path` (filled for about 27% of observations on our live instance, not indexed). A fallback today would show only a node's own packets for older days, a different meaning than the in-memory part. Doing it properly needs a per-node index written by the ingestor (pubkey, transmission, role); that is a separate plan. - The other analytics endpoints with a range (observer analytics, RF) were not checked for the same dilution. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
cab4f71c64 |
feat(nodes): LoRa radio settings per node from the MeshCore map (#1268) (#2209)
Fixes #1268. Frequency, spreading factor, bandwidth and coding rate are per-device settings that never travel in a packet. The public MeshCore map is the only source. This PR adds them to the node detail views, opt-in, and answers the open policy questions from the triage comments. ## Policy decisions | Question (triage) | Answer in this PR | |---|---| | Opt-out for self-hosted instances | Opt-**in**: `rfParams.nodesUrl` empty (default) = nothing fetched, `/api/rf-params` returns `enabled: false`. Same model as `knownChannelsUrl`. | | 33 MB every 6 h, thundering herd | Random 0-60 s delay before the first fetch, then every `refreshHours` (default 6). The map sends gzip (17 MB on the wire). It sends no ETag or Last-Modified, so conditional requests are not possible. | | Size cap | 128 MB on the decoded body (it is 51.5 MB today). Exceeding it fails the fetch instead of parsing a truncated body. | | Field shapes at the parse boundary | 64-hex key; 100-1100 MHz; SF 5-12; bandwidth from the LoRa set (7.8 to 500 kHz); CR 5-8. Invalid rows are dropped. A body that is not a well-formed array fails the fetch. | | Keep stale cache on anomaly | Any failure keeps the previous snapshot; a failed presets fetch keeps the previous presets. `fetched_at` is returned so an outage is visible rather than hidden. | | Freshness | Only adverts within `maxAgeDays` (default 7) are kept at fetch time and re-checked at serve time. | | TLS pinning | Not done: Go's certificate validation plus the field validation above. | ## What changes - `cmd/server/rf_params_cache.go`: stream decoder, validation, cap reader, preset parser (presets with identical values are merged, titles joined), background refresh. In memory only, no DB writes, so the read-only server invariant holds. - `GET /api/rf-params`: `{enabled, fetched_at, source, presets, nodes}`; `nodes` holds only nodes this instance knows, keyed by lower-case pubkey. `/api/nodes` is unchanged, so its payload does not grow for every consumer and the 500-row paging is unaffected. - Node detail (side panel and full page): `LoRa` row with the four values, the preset name when all four match one, and the age of the map advert. Shown only when the feature is on and the server has fetched the map at least once. - No column in the Nodes table, on purpose: on our network such a column showed one preset title in 814 of 1,366 rows and nothing in 443. Showing only the nodes that differ from their region is the follow-up PR. - `config.example.json` documents the block with the two URLs to use. ## Cost - Measured on the real map payload (51.5 MB, 66,336 nodes): parse 0.52 s, 15 MB allocated in total, 26,815 nodes kept (advert within 7 days). Runs in a background goroutine every 6 h. - The endpoint walks the cached node list once with map lookups: O(nodes). - Frontend: one bulk fetch of `/api/rf-params` when a node detail opens (client TTL 5 min), not one per node. ## Tests - Go (`rf_params_cache_test.go`): parse with every rejection reason, malformed bodies (not an array, truncated, wrong type, empty), preset merge and invalid presets, cap reader (one byte over refused, exactly at the cap accepted), fail-soft against an `httptest` upstream (503, HTML body, presets-only failure), config defaults, route (disabled shape, only known nodes, entry aged past max age dropped at serve time). Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). - Frontend unit: `tests/unit/test-issue-1268-rf-params.js` (7 assertions on the real file in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-1268-rf-params-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`). It checks the real disabled state on the fixture server (no detail row), then serves `/api/rf-params` through `page.route`: no table column, full detail page with preset, side panel with raw values, and the hint for a node that is not on the map. 4/4 locally, plus the existing node table and node detail E2E files. ## Validation On a staging instance with `rfParams` set to the map and preset URLs: - Log: `[rf-params] 26821 nodes within max age, 20 presets`, about 6 s after start. - `/api/rf-params` returned 1,115 of our nodes: 988 on 869.618/SF8/BW62.5/CR8, 111 on the same with CR5, 15 on 433.65 MHz, 1 on SF7. - In the browser the detail row shows in the side panel and on the full page, and the Nodes table keeps its original columns. ## Not done - Highlighting nodes whose settings differ from their region's majority (a badge in the Nodes list plus a filter) is the follow-up (separate PR). In that follow-up a CR-only difference stays informational until it is confirmed whether MeshCore radios use the explicit LoRa header (then CR travels in the header and a mismatch still decodes). - `maxAgeDays` is config-only; the customizer is a later step. - No data is written to the DB, so the RF parameters are gone after a restart until the next fetch (at most 60 s plus the fetch time). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
1ae4c74913 |
feat(observers): rolling 1 h TX airtime and a duty-cycle warning (#661) (#2208)
Fixes #661 (the narrowed scope from the 2026-10-14 comment). ## What changes - `GET /api/observers/metrics/summary` adds `tx_airtime_pct_1h` and `rx_airtime_pct_1h` per observer: the airtime share over the hour before now, whatever `window` is, from the cumulative `tx_air_secs` / `rx_air_secs` counters in `observer_metrics`. - A counter that goes down marks a reboot; only the stretch after the last reset counts. - Under 15 minutes of samples (one sample, or a reboot late in the hour) returns `null`. - Clamped to 0..100, rounded to 2 decimals. - The response also carries `duty_cycle_warn_pct`, from the new config key `dutyCycleWarnPct` (default 10, the EU 868 MHz g3 sub-band limit). Added to `config.example.json`. - RF Health tiles show `TX 1h: x%`; the observer list gets a sortable numeric `TX 1h` column. Both use the shared `public/observer-airtime.js` helper and flag values at or over the threshold with `.airtime-over` (`var(--status-red-text)`). - The handler now returns a named `MetricsSummaryResponse` struct instead of `map[string]interface{}` (one occurrence fewer). ## Why the warning matters Repeater and room-server firmware default to `airtime_factor = 1.0` (`examples/simple_repeater/MyMesh.cpp:890`), which `Dispatcher.cpp:26` turns into a 50% TX budget per 1 h window. The device therefore does not keep itself under the EU 10% limit; this view is where an operator can see it. ## Cost - Server: one extra query per summary request, `SELECT observer_id, timestamp, tx_air_secs, rx_air_secs FROM observer_metrics WHERE timestamp >= now-1h`, on the existing `idx_observer_metrics_timestamp`, then one O(rows) pass. On our instance that is 35 observers x about 12 samples (5 min interval). - Observer list: one extra bulk call `summary?window=1h` per load (client TTL 30 s, failure ignored), not a call per observer. ## Tests - Go: `TestRollingAirtimePct` (steady, reboot mid-hour, single sample, span under 15 min, reboot leaving under 15 min, nil counters, clamp, rounding), `TestGetMetricsSummaryAirtime1h` (samples older than 1 h ignored inside a 24 h window, single-sample observer gets null), `TestMetricsSummaryDutyCycleWarnPct` (default 10, config override, `observers` is `[]` not `null`). Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, which fails on Windows on clean master too). - Frontend unit: `tests/unit/test-issue-661-observer-airtime.js` (8 assertions, real file in `vm`), added to `test-all.sh`. `test-observers-headings.js` updated for the new column. Full `test-all.sh` passes locally. - E2E: `tests/e2e/test-issue-661-airtime-e2e.js`, registered in `deploy.yml` and `scripts/non-unit-tests.json`. The fixture DB has no `observer_metrics` table, so the summary is served through `page.route` with known values for the fixture's own observer ids. It checks the column header, over/under/unknown cells, sort order and the RF Health tile. 4/4 locally. It caught a real bug during development: `loadRFHealthData` declares a local `const window`, so the tile calls the helper as the global `ObserverAirtime`. ## Validation Deployed to a staging instance with live MQTT data (35 observers with metrics) and checked in the browser: - The observer list shows the column for 42 rows (33 with a value, 9 without enough samples). Sorting puts the highest share (5.3%) first. - RF Health shows `TX 1h` on all 35 tiles. With a fourth stat the line wrapped inside items on narrow tiles; the stats row now wraps between items only. - No observer was over 10% at the time, so the red state was checked by rendering one cell client-side at 12.5% (computed colour `rgb(185, 28, 28)` in light mode), and by the E2E test. ## Not done - The threshold is config-only; exposing it in the customizer is a follow-up. - No airtime comparison across observers (that is #671) and no per-packet airtime (Relay Airtime Share covers it). - Dark mode was not checked in the browser; `--status-red-text` is defined for both themes. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
fbeb9e4d71 |
ci: frontend coverage from every E2E suite, merged across the shards (#2204)
Fixes #2203. ## Problem The frontend coverage badge (35.16% on run 38413268001) came from two sources on shard 2: one dump part-way through `test-e2e-playwright.js` and the `collect-frontend-coverage.js` tour. The other E2E suites run against the instrumented build but never save `window.__coverage__`, including the suites written for single pages (`path-inspector.js` 4.72%, `touch-gestures.js` 14% on that run). ## Change 1. `9d3bfad5` `test(e2e): red:` `tests/unit/test-e2e-coverage.js`, with fake pages, contexts and browsers (no Chromium needed). On the placeholder module the first assertion fails. 2. `c00bbb12` `feat(e2e):` the collector. - `scripts/e2e-coverage-hook.js` is a preload (`node --require`) that wraps `chromium.launch` from `playwright`. `playwright` re-exports `playwright-core`, so a suite's own `const { chromium } = require('playwright')` gets the same, wrapped object. - `scripts/e2e-coverage.js` reads `window.__coverage__` before every call that replaces or ends a page's document: `goto`, `reload`, `setContent`, `goBack`, `goForward`, `close`, and context and browser `close`. Pages from `newPage`, `newContext` and the context `page` event are all tracked. - Each read zeroes the page's counters, so a document read twice is not counted twice. Maps travel once per file; later reads carry counts only. - The page returns a JSON string. Returning the object cost about 10 s per suite, because Playwright serialises it node by node. - A read is given up after 5 s, so a busy page cannot hold up a suite's cleanup. - On exit the hook writes one file per suite to `.nyc_output/`, or nothing without coverage. A failed write is logged and never changes the exit code. 3. `64487c4c` `ci:` the workflow. - On master pushes the E2E run step sets `NODE_OPTIONS=--require ./scripts/e2e-coverage-hook.js`. On PRs it is empty, so PR runs are unchanged. - Each shard runs `nyc merge` (one file per suite is about 6 MB, mostly the same maps) and uploads `coverage-shard.json` with its results instead of `.badges/`. - The `e2e-test` job merges the three shards and makes the frontend badge (it used to come from shard 2 alone). - The merge, setup-node and badge steps are `continue-on-error`. When `nyc report` gives no statement percentage the step warns and writes no badge, so a registry hiccup cannot publish 0%. - `test-e2e-playwright.js` no longer writes its own `e2e-coverage.json`: the hook covers that suite completely, including the tests after the old dump. `collect-frontend-coverage.js` stays as an extra source. ## Performance Measured locally against an instrumented server (Windows, Chrome): | Suite | Without hook | With hook | |---|---|---| | `test-path-inspector-coverage-e2e.js` | 1.9 s | 2.2 s | | `test-touch-gestures-coverage-e2e.js` | 17 s | 16 s | | `test-e2e-playwright.js` (stops at the same local-fixture failure in both runs) | 52 s | 52 s | | `browser.close()` on a page in a 30 s busy loop | 153 ms | 5.2 s | The hook only loads on master pushes. ## Testing - `tests/unit/test-e2e-coverage.js` covers: - reads before each document-replacing call and before close; - pages opened by the page itself; - context and browser close; - summing without double counts; - a failed read, and a read that never answers; - the file writer, including a failing write. - Nine deliberate breaks of the library each turn it red: - dropping the counter reset; - dropping the `page` event, the `goto` read, the `goForward` read, the `close` read or the browser `close` read; - removing the timeout or the write guard, or writing an empty store. - The test also fails if it stops before its end. Without that check, a read that never returns would let node exit 0 half-way. - Real suites with the hook, measured on their own (the badge run had all suites): | File | Badge run 38413268001 | Own suite with the hook | |---|---|---| | `path-inspector.js` | 4.72% | 52.75% | | `touch-gestures.js` | 14% | 72.96% | - Exit codes and pass counts were unchanged with the hook for these suites (the first three were run before the 5 s read timeout was added): - `test-slideover-1056-e2e.js` (6 contexts); - `test-a11y-axe-1668.js` (`@axe-core/playwright`); - `test-issue-1150-404-state-e2e.js`; - the three above. - The badge step, run locally against two merged shard files: - It gives the same per-file numbers as `nyc report` on the inputs. - With no shard coverage, or with `nyc report` failing, it warns and writes no badge. - `sh test-all.sh` passes locally, apart from `test-issue-1956-release-routing.js`, which needs the workflow's `GITHUB_REF_NAME` and was skipped. The test inventory classifies 321 suites. `actionlint` 1.7.7 reports only the self-hosted runner label, as on master. **Not tested:** the shard and badge steps in CI. They only run on a master push, so the first master run after merge is the real test. I do not know what the merged number will be. I will compare the badge with the merged report in that run's `e2e-test` log. **Not counted:** a full page load the suite does not start through one of the wrapped calls, such as a link that leaves the SPA or `location.reload()` from page code. ## Not in this PR - The `tests/unit/` suites, which load frontend files through `vm.createContext` without instrumentation (noted in #2203). - Coverage on PR runs. The overhead above is small, so this could follow if wanted. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
955386b63f |
test(map): settle the router's focus frame before the teardown snapshot (#2179)
## Summary Fixes an intermittent failure in `tests/e2e/test-map-nodes-pagination-e2e.js`, the "late response after map teardown" cases added for #2030 in #2048: ``` ✗ late /observers response after map teardown: old response must not change the destination page ``` It failed on #2164 ([run 38080812509](https://github.com/Kpa-clawbot/CoreScope/actions/runs/38080812509), shard 2/3), a Go-only change, and passed on the rerun. ## Cause `checkMapTeardown` compares `#app`'s HTML before and after releasing the old map's pending response. When it failed, the only difference was on the destination page's heading: ``` before: <div class="tools-landing"><h2>Tools</h2>… after: <div class="tools-landing"><h2 tabindex="-1">Tools</h2>… ``` That attribute comes from the router, not the old response. Since #630 (`app.js`, "SPA focus management") it focuses the new page's first heading in a `requestAnimationFrame` after the page renders. The before-snapshot was taken as soon as `#leaflet-map` detached, sometimes ahead of that frame. ## Fix The test now waits two frames before the before-snapshot, the same settle it already does before the after-snapshot. The router's frame is queued by then, so it always lands first. The change is test-only: no app code changes, and every assertion stays as it was. ## Tests Run locally with the e2e fixture (`tools/freshen-fixture.sh`, migrated) served by this branch's server, headless Chromium: | | runs | failed | |---|---|---| | test on `master` (`352a0ccd`) | 20 | 2 | | this branch | 50 | 0 | - **Before:** the two failures on `master` were "old response must not change the destination page" (teardown, `/config/regions` and `nodes`), plus one 30 s timeout in an "after map replacement" case in the same runs. - **After:** none of these failures or the timeout came back in 50 runs. ## Checks - **Lint:** ESLint passes with the repo config and with Node globals. SonarQube rules on the file report nothing on the added lines; its four existing minor findings are on lines this PR doesn't touch. - **Repo rules:** one logical change, explicit `git add`, no private data, no new dependencies. This is a test-only fix, so there's no red/green pair: the "red" is the intermittent failure above, measured as 2 in 20. Related: #2030, #2048 (where the test was added). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
55dd28164d |
test(e2e): assert the configured-scope row after the theme-refresh re-render (#2180)
## What `test-issue-1648-m2-icons-e2e.js` now asserts the configured-scope row only after the page's `theme-refresh` re-render has fired and the detail has rendered again. ## Why On 2026-10-10 this assertion failed CI on four PRs (#2169, #2171, #2154, #2164), each time at the first live-map case: `confirmed scope row must be visible, including an empty scope`. A re-run of the failed jobs passed every time. The code path: - `customize-v2.js:814` dispatches `theme-changed` once `/api/config/theme` resolves, on every page load. - `app.js:1309-1315` turns that into `theme-refresh` 300 ms later. - `live.js:4713` re-runs `showNodeDetail(activeNodeDetailKey)` on `theme-refresh`, and `showNodeDetail` first sets `#nodeDetailContent` to "Loading…" (`live.js:2598`). This is the only caller besides the marker click (`live.js:3090`). - The test waited for `#nodeDetailContent table` and then called `row.isVisible()`, which does not wait. A refresh landing between the two fails the check. The same race could let the null-scope case pass on the "Loading…" panel, because `row.count() === 0` holds there too. The fix uses the same `addInitScript` flag pattern as `test-e2e-playwright.js:83`. ## Testing - Locally against a server on the E2E fixture: 5 of 5 runs pass, 27 of 27 checks each. - **Not reproduced locally.** The unchanged test also passed every local run: 5 plain, 3 with `/api/config/theme` delayed by 1 s, 5 with 8x CPU throttling. So the reasoning above comes from the code path and the four CI failures, not from a local red-then-green. - Test-only change, no product code touched. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
352a0ccd19 |
feat(region-filter): configurable region quick picks (#2170)
## Summary **Independent of every other open PR.** Choosing a pick saves the selection, redraws the control and notifies the page's listeners itself, exactly as a click in the control does, so it re-queries straight away whether or not #2168 (`setSelected` notifies) is merged; with both in, nothing fires twice. Region quick picks are named groups of region (IATA) codes, offered as one-tap choices inside the region filter, e.g. "Northern California" for a dozen airport codes instead of ticking each box. One deployment has been injecting these into the page through an nginx `sub_filter` and a script that wraps `window.RegionFilter`; this makes them a configured feature. ```jsonc "regionQuickPicks": [ { "name": "Northern California", "description": "Bay Area and Sacramento observers.", "regions": ["SJC", "SFO", "OAK", "SMF"] } ] ``` - **Server:** `GET /api/config/region-quick-picks` returns the groups cleaned up: names and codes trimmed, codes upper-cased and de-duplicated in their configured order, groups without a name or codes dropped, at most 20 groups of 200 codes, always a list. `/api/config/regions` is unchanged. - **Region filter:** loads the groups with the regions and offers every group with at least one observer today, inside the region control: - dropdown layout: rows of the same kind as **All** (checkbox and bold name), directly under All with a divider before the single regions; the row is ticked while that pick is the selection, ticking it selects the pick and closes the menu, unticking goes back to all; - pill layout: the picks lead the same bar, set off from the single regions by a thin divider. - A tap selects the group's codes that have an observer and re-queries straight away. - The selector then reads the group's name ("South Bay ▾") instead of "2 Regions", and the button shows as pressed. Selecting the same codes by hand does the same. - Tapping the active group again goes back to all regions. - A failed fetch, or no configured groups, leaves the filter exactly as before. - **Mobile:** the rows take the menu's 44 px row height under `(pointer: coarse)`; the pill-bar picks reuse `.region-pill` and its touch target. Inside `.filter-bar` (Packets) the existing touch rule for inputs made every region checkbox a 44 px square; the checkbox is now 20 px and the 44 px row is the tap target. - **Semantics:** like every region filter, a group means "heard by observers in these regions", not "located there"; `config.example.json` says so. No existing issue covers this. ## Screenshot Nodes page with two picks configured (example config below), after tapping **South Bay**: the pick and its codes (SJC, MRY) are selected, and the list re-queried for them. Taken against `test-fixtures/e2e-fixture.db`.  ## Config example ```json "regionQuickPicks": [ { "name": "South Bay", "description": "San Jose and Monterey observers", "regions": ["SJC", "MRY"] }, { "name": "East and North Bay", "description": "Oakland and San Francisco observers", "regions": ["OAK", "SFO"] } ] ``` Each pick needs a `name` and at least one code; `description` is optional (shown as the tooltip). Codes are trimmed, upper-cased and de-duplicated; a pick with no code that has an observer today is not shown. At most 20 picks of 200 codes. ## Browser validation The branch's server was run locally against a migrated copy of `test-fixtures/e2e-fixture.db` (observers in MRY, OAK, SFO, SJC) with three groups configured: "South Bay" (SJC, MRY, SCZ), "East and North Bay" (OAK, SFO, sts) and "Nowhere yet" (XXX). Checked in headless Chromium: | check | desktop 1280 px | phone 390 px (touch) | |---|---|---| | groups shown | South Bay, East and North Bay ("Nowhere yet" hidden) | same | | tap South Bay selects | SJC, MRY (SCZ has no observer) | same | | next `/api/packets` request | `region=SJC,MRY` | `region=SJC,MRY` | | selector label | "South Bay ▾" | "South Bay ▾" | | second tap | back to all regions | same | | button height | 24 px | 44 px | | page wider than viewport | no | no | | page errors | none | none | Checked on Packets (dropdown layout) and Nodes (pill layout). Three layout problems found this way on the phone were fixed before opening: a sideways-scrolling row clipped at the screen edge, the selector squeezed into a narrow column (the filter group is a non-wrapping flex row), and the two wrapped rows spread apart on Nodes (`align-content`). **Revision 1:** the first version put the picks in a separate row beside the selector; they moved inside the dropdown. Re-checked with a 14-region list and three picks on Packets: desktop 1280 px dark and phone 390 px light both keep the menu on screen (menu 340 px wide, page not wider than the viewport), the pick closes the menu and the trigger names the pick. **Revision 2:** picks are checkbox rows like **All** instead of chips. Re-checked on Packets at 1280 px dark and 390 px light: rows render bold under All, the divider separates the single regions, ticking a pick ticks its codes and names the trigger, no sideways scroll. Unrelated and not changed: on Nodes at 390 px the region row (existing pills included) starts flush with the screen edge. ## Tests - Red commit `81ea67d6`: the config type, a stub normaliser and endpoint returning nothing, and tests. - Go: cleaning (trim, upper-case, de-duplicate, drop incomplete groups), caps, nil config, and the endpoint's JSON. - Frontend, `tests/e2e/test-region-quick-picks.js` (moved from `tests/unit/` in `fced69a7`; it needs jsdom, so it runs in the E2E job like `test-table-sort.js`): the real `region-filter.js` in **jsdom** against faked config endpoints. Nine cases: rendering and hiding groups with no observer, selecting and re-querying, the selector label, toggling back, a hand-made selection marked active, the pill layout, no groups, a failed fetch, HTML escaping. Seven fail on the stub, all on assertions. - Green commit `0e33210e`: the implementation, styles, `config.example.json` (`regionQuickPicks` + `_comment_regionQuickPicks`), `docs/api-spec.md`, and the OpenAPI route list (`TestOpenAPICompleteness` requires it). The jsdom test also had a cross-realm comparison bug fixed in this commit (`Array.from` on arrays coming out of the jsdom window). - `make test`: 16 of 16 modules pass. Frontend suites: 195 of 196; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. - Red commit `f991060d` (revision): picks live inside the dropdown menu before the checkboxes, choosing one closes the menu, and in the pill layout they share the region bar; two fail on assertions against the old layout. Green commit `1f720e35`. Frontend 195 of 196 (same environment-only failure). - Red `134f5256` / green `da0adb50` (second revision): picks are checkbox rows between All and the single regions with a divider, no chips in the menu; ticking and unticking a pick row. The green commit also repairs `style.css`: `1f720e35` had prepended its block to the top of the file (its edit matched an earlier `.region-pill {`) and left the previous rules in place. The file now differs from `master` by 13 lines. ## Review notes - Groups are offered only when at least one of their codes appears in `/api/config/regions`, and only those codes are selected. The stored selection is pruned to known codes on load anyway, so selecting absent codes would not survive a reload. - Customizer (AGENTS.md rule 8): groups are operator configuration; a customizer editor could come later. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `b84ace4e`: - `normalizeQuickPickCodes` split out of the config normaliser (S3776, 17 > 15); - a comment on why a failed quick-pick fetch is ignored (S2486); - `for…of` in `activePick` (S4138); - in the tests: `node:` specifiers, `includes()`, and an explicit string comparator for sorted selections (S7772, S7765, S2871). - **Kept on purpose:** - S2486 is still reported on that `catch`. Swallowing the error is the intended behaviour: the filter works as before, just without picks. - S1523 on the `vm` test loader, as above. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
c2a590b741 |
feat(companions): linked-only ingest, account Companions and My coverage (#2176)
Part 3 of 3 for #2173 (companion linking): the optional linked-only ingest filter, the account page and the "My coverage" toggle, plus the docs. Builds on #2174 and #2175 (both merged); `upstream/master` is merged in, so the diff shows only this part. ## Changes in this PR **Ingestor: linked-only ingest** (`cmd/ingestor/linked_companions.go`) - With `clientRxCoverage.requireLinkedCompanion` on (default off) and user management enabled, every `meshcore/client/<pubkey>/...` message whose pubkey is not in `companion_links` is dropped before any client handler runs, and counted (`client_unlinked_dropped` in the stats file). - The ingestor opens `users.db` read-only (`mode=ro`, raw SQL, no `internal/users` import), like the approved-channels reader, so the #1283 write invariant holds. `TestLinkedCompanionSetIsReadOnly` pins it. - The set refreshes every 60 s. A miss re-reads at most once per 5 s, so a companion linked a moment ago is accepted within seconds while a flood of unknown pubkeys costs one read per 5 s. - It passes everything until its first successful read, and when `users.db` has no `companion_links` table yet (a server older than this). After the first read it keeps the last set on a read error. It is a filter, not a security boundary: anyone with the shared broker credential can still publish under a linked pubkey. **Frontend** - Account page: device tokens show under Devices as "CoreDrive RX" with their label; a new Companions section lists linked companions (name, short key, linked since, last seen) with an Unlink button. - Coverage page (`#/rx-coverage`): a "My coverage" toggle for logged-in users (`mine=1`). Only the newest coverage reply is drawn, and a refused `mine=1` turns the toggle off. Under "My coverage" the page does not ask for the "nothing received" cells from #2172, and their legend entry is hidden. **Docs**: user guide (accounts: devices, companions), `docs/client-rx-coverage.md` (linking, linked-only ingest, the CORS exception), `config.example.json`, and the spec status. ## Performance The filter runs on every client message. Benchmarks with 1000 linked companions (12-core dev box, `go test -bench`, 3 runs each): | Path | Time | Allocations | |---|---|---| | linked pubkey (lock-free map lookup) | 54 ns/op | 0 | | linked pubkey, parallel | 11-14 ns/op | 0 | | unlinked pubkey inside the 5 s miss gap | 85-95 ns/op | 0 | | one `users.db` re-read | 0.38 ms | 3030 | The unlinked benchmark asserts that no extra `users.db` read happens. A re-read happens at most once per 5 s on a miss and once a minute from the refresh loop. ## Verification - `cmd/ingestor`: 15 new tests (filter off changes nothing, linked passes, unlinked dropped before every handler, miss refresh accepts a new link, the 5 s cap holds under a flood, unlink honoured after the periodic refresh, missing table and missing file, fail-closed only after the first read, read-only open, startup, stats counter) and 4 benchmarks. Full ingestor suite green. - Frontend unit tests for the account page (device tokens, companions list escaping, empty state, Unlink confirm and cancel, load errors) and `tests/unit/test-rx-coverage-mine.js`. - E2E `tests/e2e/test-user-management-e2e.js` gains a companion step (device token under Devices, a companion linked with a real Ed25519 signature under Companions, Unlink). Run locally against the e2etest build: 28/28. The coverage suites (`test-rx-coverage-gaps-e2e.js`, `-noise-`, `-viewport-`) still pass. - Frontend unit suite (`test-all.sh`) green. ## Not in this PR - Narrowing the "nothing received" cells to the caller's companions under "My coverage" (the track query takes one companion today). - Per-user broker credentials or ACLs, which would turn the filter into an access control (non-goal in the spec). --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
f3004d977e |
feat(rx-coverage): show where a companion drove and received nothing (#2172)
The Mobile RX coverage page (`#/rx-coverage`) shows only cells where a companion received something. Where a companion drove and received nothing is just as useful for operators, and the data is already there: the RF sample track (`client_rf_samples`, #1905/#1906) and the receptions (`client_receptions`) use the same hex grid (`hexCellAt`, `zoomToHexRes`). This PR adds those cells as a hatched layer under the reception cells. ## Changes in this PR **Server** - `GET /api/rx-coverage?...&gaps=1` adds a `gaps` member to the FeatureCollection: cells with moving (non-stationary) track samples and no reception in the same `days` window and `bbox`, each with `samples`. `gaps_truncated` is set when more than 5000 cells qualify (the most-sampled are kept, same pattern as `coverageFeatureCap` and `rfNoiseFeatureCap`). - Gaps are only returned when `clientRfSamples` is on and there is no `node` filter: with `?node=` the question would be "this node was not heard", which these cells do not answer. `?rx=` limits both the track and the receptions to that companion. - Without `gaps=1` the payload is unchanged: the `gaps` member is omitted. - New code lives in `cmd/server/rx_coverage_gaps.go` (pure `aggregateGaps` plus `queryTrackPoints`); the handler change in `rx_dashboard.go` is 14 lines. - `/api/rx-coverage` is now documented in `openapi.go` and removed from `openapi_known_gaps.json` (#1670). **Frontend** (`public/rx-coverage.js`, `public/node-reach-coverage.css`) - A "Nothing received" toggle on the signal layer, on by default, only rendered with `MC_CLIENT_RF_SAMPLES`. Hidden on the noise layer. - Gap cells use a 45-degree hatch (an SVG pattern in a dedicated renderer, so they sit under the reception cells). The hatch is denser for cells driven through more often (1 / 2-3 / 4+ samples). Tooltip: sample count. Legend entry: "driven, nothing received". - Off state is deep-linkable as `gaps=0`. - New token `--nq-cov-gap` for light and dark. Plain grey (`--nq-cov-grey`) already means "received, no SNR", so the hatch keeps the two apart. ## Threshold: one sample Measured on a live network (analyzer.on8ar.eu, Belgium, September to October 2026), with cells of about 250 m: a cell with track samples and no reception in one half of a month, driven again in the other half. How often did it receive something the second time? | Minimum samples | Cells "nothing received" | Driven again | Received something later | |---|---|---|---| | 1 | 3971 | 1263 | 29% | | 2 | 1565 | 592 | 32% | | 3 | 619 | 243 | 35% | | 5 | 142 | 63 | 40% | A higher threshold hides cells without making the rest more reliable, so the threshold is 1 and the wording is "nothing received in this period", never "no coverage". Using every decoded packet as reception evidence (including 1-byte path hashes) gave 25% to 32%, so the capture rule for 1-byte hashes is not what drives this. The median cell has 2 samples. ## Verification - Go: 7 new tests in `rx_coverage_gaps_test.go` (gap/no gap, stationary excluded, single sample, cap keeps the most-sampled, endpoint, `?rx=` filter, no `gaps` member when not asked / with `?node=` / with RF samples off). The endpoint tests fail without the handler change. Full `cmd/server` suite green locally, `go vet` and `gofmt` clean. - E2E: `tests/e2e/test-rx-coverage-gaps-e2e.js` (4 steps: no toggle with RF samples off, on by default with hatched cells under the reception cells, toggle off with `gaps=0` and deep link, noise layer hides it). 2 steps fail without the frontend change. Existing `test-rx-coverage-noise-e2e.js` and `test-rx-coverage-viewport-e2e.js` still pass. Listed in `scripts/non-unit-tests.json`. - Frontend unit suite (`test-all.sh`) green. - Real data on a staging instance (7 days, bbox 50.8,4.9,51.3,5.8, z=12): 478 reception cells and 379 gap cells, no cell in both, reception features identical with and without `gaps=1`, 0.48 s to 0.52 s per request both ways. - Browser: checked on staging with real data (light theme), and light, dark and 390 px wide with mocked data. ## Not in this PR - A per-node variant on the Reach page ("this repeater transmitted while a companion was nearby, and was not heard"). That needs the transmit moments from `observations` and is a heavier query. - `deploy.yml` does not get a line for the new E2E file, like the other entries that are only in `non-unit-tests.json` (#2037). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
29095658e9 |
fix(nodes): explain packet and observation counts (#2132)
Red commit: `d8a6d42` ([CI assertion](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37695889461)). Review-regression red: `f73643b` ([CI assertion](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37703995494)). Fixes #2131. Partial fix for #1154: U2 only; other audit items remain open. Both node views explain distinct transmissions versus recorded observations. The shared help is reachable by keyboard and touch, stays readable at either scroll edge, and remains open when the pointer enters its text. Escape dismisses help while preserving node context and focus; a second Escape keeps the existing navigation behavior. Count expressions and API requests are preserved. One controller per nodes page handles one active help element and one queued animation frame. Scroll/resize work is coalesced; listeners are removed on page teardown. Work is constant relative to packet/node counts, with no dataset scans or new settings/dependencies. E2E assertion added: `tests/e2e/test-e2e-playwright.js:114`. Thirteen cases cover exact counts, accessible description, keyboard/touch, hover transfer, Escape, and viewport/scroll bounds. Existing metric assertions are unchanged. Browser verified: http://localhost:63508 using Chromium, controlled fixtures and actual API data. Desktop/mobile screenshots retained locally; additional checks exercised resizing and repeated page cleanup. Local validation: 189 standalone suites; core browser 148 passed, 3 fixture-dependent skips (existing manual skips unchanged); location/metric suite 6 passed; syntax, theme variables and lint passed (0 errors, 92 existing warnings). ## Preflight overrides - OpenClaw preflight/profile unavailable; repository checks and local Chromium used. - Windows ingestor symlink fixture lacks permission. Backend files are unchanged; full Linux Go CI is required. |
||
|
efitenandClaude Opus 5.5
|
2456762179 |
feat: account data export and users.db backup (#2141)
A small follow-up to parts A to E of #2128: users can download all data the instance holds about them, and the server backs up `users.db`. Stacked on #2140. Review the commits after that branch. ## The situation - An instance with accounts holds personal data, but a user has no way to get a copy (GDPR articles 15 and 20). Self-delete already exists. - `users.db` is the only file the server writes, and nothing backs it up: `GET /api/backup` snapshots the analyzer database only. ## What this PR adds **Data export.** `GET /api/account/export` and a "Download my data" button on the account page give one JSON file with: - the profile, including activation and a pending email change; - sessions (times and user agent); - synced settings, own proposals, notification prefs and watches; - audit rows where the user is actor or target, with other accounts as an id only; - the mail log. Password hash, session and link tokens and the unsubscribe token are left out: they are credentials. Each export is audited as `user.export`. **users.db backup** - Daily `VACUUM INTO` snapshots through `internal/users`, on by default when user management is on. They go to `backups/` next to `users.db`, the 7 newest are kept, the directory is 0700 and the files 0600. Rotation only touches `users-<timestamp>.db` files and never the snapshot it just wrote. Config: `userManagement.backup {enabled, dir, keep}`. - `GET /api/admin/users-backup`: an admin downloads a fresh snapshot to keep off the server; audited as `user.backup`. - `docs/user-guide/accounts.md` gains a restore procedure. It covers what a restore brings back (accounts deleted after the snapshot, old passwords, revoked sessions) and how to clean that up afterwards. Spec: [`docs/specs/2026-10-08-account-export-and-users-backup-design.md`](https://github.com/efiten/CoreScope/blob/feat/account-export-backup/docs/specs/2026-10-08-account-export-and-users-backup-design.md). ## Verification - `internal/users` with `-race` and the `cmd/server` suite pass locally, 30 new Go tests. The file-mode tests skip on Windows. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 27 of 27 steps locally. - On our production instance since 8 October 2026; the first snapshot there was 143 kB. ## Not in this PR - Importing an export into another instance. - Off-site upload of the snapshots. The admin download covers that by hand. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
de364a8bb6 |
feat: mail notifications for watched nodes (user management part E) (#2140)
Part E of #2128: a logged-in user watches nodes and gets one mail when a watched node goes offline, comes back, or reports a low battery. Admins can add instance events: a new foreign node, and an observer going offline or back. This covers the per-user mail part of #775. Stacked on #2139 (part D, which builds on #2138). Review the commits after that branch. ## The situation A sysop learns that a repeater went silent, or that its battery is running down, only by opening CoreScope. #775 asks for notifications. Accounts (part A) now give a verified address to mail and a mailer with delivery status. ## What this PR adds **Events.** They use the instance thresholds, so a mail does not disagree with the node page: - `node.offline`: silent for the role's `healthThresholds` window. Last heard comes from the packet store, and repeaters and rooms count relayed traffic (#1598). - `node.battery`: advert telemetry below `batteryThresholds.lowMv`, recovered at `lowMv + 100`. - Admins only: `foreign.new` (once per node) and `observer.offline`. **Store.** Schema v5 with `notification_prefs`, `notification_watches` and `notification_state`. **Server** (only with `userManagement.notifications.enabled`) - A notifier next to the janitor, every `intervalMinutes` (5). The evaluator is a pure function of watches, prefs, stored states and node snapshots. - No burst after a restart or a new watch: a subject's first evaluation stores its state without mailing. The loop waits for the startup load. While ingest is stale (newest packet older than 30 minutes) the offline checks pause, and after recovery they wait one silent window. - One mail per user per check. Limits: 20 per user and 100 per instance per rolling 24 hours, 50 watches per user. A change over a limit is recorded and never mailed later. - Every mail has a one-click unsubscribe (`List-Unsubscribe` and `List-Unsubscribe-Post`). The GET only redirects to a confirm page, so link scanners cannot unsubscribe anyone. Node names are cleaned of control and bidi characters before they go into a mail. - Routes: `GET`/`PUT /api/account/notifications`, `PUT`/`DELETE /api/account/notifications/watches/{pubkey}`, `POST /api/account/notifications/watch-my-nodes` (copies the synced "my nodes" list), `GET`/`POST /api/notifications/unsubscribe`. `mailer.Message` gains `Headers`. **Frontend.** A "Notify me" toggle on the node pages, a Notifications section on the account page (`#/account?section=notifications`), an unsubscribe page, and the notification mail count on the admin overview. Spec: [`docs/specs/2026-10-07-node-notifications-design.md`](https://github.com/efiten/CoreScope/blob/feat/notifications/docs/specs/2026-10-07-node-notifications-design.md). ## Performance Each check reads watches, prefs and states from `users.db` (one query each), the watched nodes from the analyzer DB in chunks of 500, and last-heard times from the packet store. | Measurement | Result | |---|---| | One check on our staging instance, 130,000 packets in memory, 1 watcher | 6.9 ms, of which 0.93 ms holding the store read lock | | Evaluator benchmark, 100 users with 50 watches each, 2,000 nodes | 1.75 ms | No work is added to ingest, broadcast or any request path. ## Verification - `internal/users` and `internal/mailer` with `-race`; the `cmd/server` suite plus `-race` on the notifier tests; 74 new Go tests. The same Windows-only failure as noted in #2138 applies. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 26 of 26 steps locally. With notifications off the new steps fail, so they do test the flag. - On our staging and production instance since 7 October 2026. ## Not in this PR - Other channels (Discord, Telegram, webhooks). Detection is separate from delivery, so one can be added. - The topology, RF and anomaly alerts of #775. - If the whole server starts after a feed outage that already ended, the grace window is unknown and a watcher can get one wrong offline mail. The user guide says so. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
abbe6846ff |
feat: propose and approve hashtag channels (user management part D) (#2139)
Part D of #2128: logged-in users propose a hashtag channel, an admin approves, rejects or later revokes it, and the ingestor decrypts approved channels without a restart. It is the first consumer of a generic proposal table, and answers the request in #2092. Stacked on #2138 (part C). Review the commits after that branch; GitHub shows C's commits here until it is merged. ## The situation - An instance decrypts only the hashtag channels listed in `hashChannels`. Users who want a city or club channel shown have to ask the operator, who edits `config.json` and restarts the ingestor. - #2092 asked for a suggest-and-approve flow; a downstream fork (dborup/CoreScope#99) runs one with anonymous suggestions and a file queue. ## What this PR adds **Name rules (`internal/channel`).** `ValidateHashtagName`, mirrored in the browser: - at most 31 UTF-8 bytes including `#` (firmware `ChannelDetails.name[32]`), case preserved, `Public` refused in any case; - control, bidi, separator and format characters refused except ZWJ, plus invisible fillers (`Other_Default_Ignorable_Code_Point`, U+2800, non-ASCII spaces). Go and JS give the same verdict for every code point. **Store.** Schema v4, one generic `proposals` table (kind, subject, status pending/approved/rejected/revoked, proposer, reviewer, note). One row per (kind, subject); limits are checked in the same transaction as the change. **Server** (only with `userManagement.channelProposals.enabled`) - `POST /api/proposals`, `GET /api/account/proposals`, `GET /api/admin/proposals`, `POST /api/admin/proposals/{id}/approve|reject|revoke`. - 400 for an invalid name. 409 for a duplicate, a rejected name, or a name already in `hashChannels`/`channelKeys` ("this channel is already decrypted on this instance"). 429 over a limit (`maxPending` 100, `maxApproved` 128, `perUserPerDay` 5). - `GET /api/channels` gains `approvedChannels`, so approved channels are listed before they have traffic. With the feature off the response is byte-identical. **Ingestor.** Opens `users.db` with `mode=ro` every 60 seconds, re-validates the names, and swaps its key map through an `atomic.Pointer`. Configured channels always win, and a failed read keeps the last good set. The ingestor never writes `users.db`; AGENTS.md now says so. **Frontend.** "Propose for everyone" in the add-channel dialog, an "approved" marker in the channel list, "My proposals" on the account page, and an admin "Proposals" tab with a warning on approve. The packet detail pane now escapes the channel name: until now only the operator chose those names. Spec: [`docs/specs/2026-10-07-channel-proposals-design.md`](https://github.com/efiten/CoreScope/blob/feat/channel-proposals/docs/specs/2026-10-07-channel-proposals-design.md). ## Performance Every key is tried on each GRP_TXT that no key opens (`decodeGrpTxt`). Benchmark of that worst case: | Keys | Time per packet | |---|---| | 320 configured | 204 to 216 µs | | 320 configured + 128 approved | 272 to 320 µs | That is about +33%, linear in the key count and capped by `maxApproved`. With the feature off the decoder gets the identical map and no ticker runs. ## Verification - `internal/channel`, `internal/users` (with `-race`), `cmd/server` and `cmd/ingestor` pass locally, 53 new Go tests. The ingestor's `TestWriteStatsAtomic_SymlinkAtDestIsReplaced` needs symlink rights on Windows and fails on `master` too. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E: 21 of 21 steps locally (propose, approve, the channel appears, revoke removes it). - On our staging and production instance since 7 October 2026. ## Not in this PR - Private (PSK) channels: their keys stay in the browser (#725). - Anonymous proposals. - Names that need ZWNJ (some Persian and Urdu spellings) and subdivision flags are refused, because ZWNJ and tag characters are format characters. Allowing them later only widens the rule, so no approved name gets stranded. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
18a634c115 |
feat: admin dashboard with overview and audit log (user management part C) (#2138)
Part C of #2128: an admin area with an overview, the user table and a global audit log. Off by default like the rest of user management: with `userManagement` off nothing changes, and non-admins get no new UI. This is the first of 4 stacked PRs (C, D, E, then a small export/backup follow-up). Each later one contains this branch; review them in order. ## The situation - An admin could manage users one by one in `#/admin/users`, but nothing showed whether the instance needs attention: accounts stuck in activation, bouncing mail, someone guessing a password, an MQTT source that dropped. - The audit log existed (`internal/users/audit.go`) but could only be read per user, and logins were not recorded. ## What this PR adds **Store (`internal/users`)** - Schema v3: an index on `audit_log(at)`. - `AuditList` with filters (action or group prefix like `user.login.*`, user as actor or target, period) and keyset pagination; `PruneAudit`; `Stats` for the user figures. **Server** - `GET /api/admin/stats` (typed struct): accounts by status, admins, registrations and active users over 7 and 30 days, logins and failed logins in 24 hours, mail by final status, and the attention items computed server-side. - `GET /api/admin/audit`: filtered, newest first, `next` cursor. - `GET /api/admin/users` gains `bouncing=1`. - Logins are audited as `user.login` and `user.login.failed` (reason `wrong_password`, `pending` or `disabled`). An unknown address writes no row. The writes are asynchronous, so the login response does not wait on `users.db`. Login rows are pruned after 90 days. **Frontend** - `#/admin?tab=overview|users|audit`: `admin.js` (tab shell), `admin-overview.js` ("Needs attention", Users card, System card from the existing health, MQTT and observer endpoints), `admin-audit.js` (filters in the URL, "Load more"). `admin-users.js` becomes the Users tab; the old `#/admin/users` link rewrites to it. - `/api/healthz` is read on open and on Refresh only, not on the 60-second timer, because it walks every packet under a read lock. Spec: [`docs/specs/2026-10-07-admin-dashboard-design.md`](https://github.com/efiten/CoreScope/blob/feat/admin-dashboard/docs/specs/2026-10-07-admin-dashboard-design.md). ## Verification - `internal/users` and `cmd/server`: `go vet` and `go test` pass locally, 26 new Go tests. One upstream test, `TestSaveGeoFilterPreservesFileMode`, also fails on Windows on plain `master` (file modes) and is unrelated. - `sh test-all.sh` exits 0; the XSS gate in diff mode passes. - User-management E2E with the `e2etest` build: 13 of 13 steps locally. - Running on our staging and production instance since 7 October 2026. ## Not in this PR - Restricting existing pages (perf, MQTT status) to logged-in users or admins. The admin area adds no access mechanism of its own, so that stays a later router rule plus endpoint check. - Server-enforced customizer tab restrictions (#1508). - The 24-hour, 5-attempt and 10-minute attention thresholds are constants for now (AGENTS.md rule 8: customizer later). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
a4445b4985 |
ci: run E2E, Go tests and the image build side by side (#2135)
Closes #2134. ## What changes The test jobs and the image build now run side by side. Only the GHCR push waits for all of them. ``` changes ─┬─ go-test ─────────────────────────┐ ├─ race-test (ingestor changes only) │ ├─ e2e-shard ×3 ── e2e-test (gate) ───┼─ build-and-publish → deploy, publish └─ image-check (two-arch build) ──────┘ ``` - **`e2e-shard` (new, matrix of 3).** It needs only `changes`. The suite list stays in the workflow: each line now reads `suite <shard> <file> [VAR=value ...]`, with the same environment variables as before. - Shards are balanced on the measured suite times, about 6 minutes each. - A shard number outside 1..3 fails the step, so a typo cannot drop a suite silently. - `test-issue-1648-m4-icons-e2e.js` stays in the same shard as, and after, the a11y suite. Its distance-tab check only counts once the lazy distance index (#1011) has been built, and the a11y suite is what first requests it. - **`e2e-test`** is now a gate under the old check name. It runs only if all three shards passed, checks that three results arrived, and builds the same `e2e-badges` artifact as before. - **`image-check` (new).** It runs the two-arch build (no push) and the arm64 QEMU smoke on every event, beside the tests. It also computes the build metadata once, which the push reuses. - **`build-and-publish`** now needs `go-test`, `e2e-test` and `image-check`. On a PR every step is skipped. On push and tag refs it only pushes, with the same build args as `image-check`, so every layer should be a cache hit. - **The unused `docker compose build` of the staging image is gone.** `docker compose config --quiet` still validates the file. - `AXE_SCREENSHOT_DIR` now actually reaches the a11y suite. - `tests/unit/test-issue-1956-release-routing.js` follows the new jobs, and checks more than before: - a failure in `go-test`, `e2e-shard`, `e2e-test` or `image-check` skips `build-and-publish`; - the only non-publishing build lives in `image-check` and runs on every event; - the push takes its build args from `image-check`. ## Measurements The old numbers are from upstream; the new ones are from PR runs on my fork (a branch on top of this commit, plus one temporary commit that also triggers the workflow for PRs into a test base branch). | | Before (PR run 37766553643) | After (fork runs 37812660655 / 37814554422) | |---|---|---| | Go Build & Test | 4.9 min | 7.2 / 7.5 min, in parallel | | E2E | 18.4 min | 3 shards of 7.1–9.0 min, in parallel | | Image build | 6.9 min, after E2E | 5.9 / 5.4 min, in parallel | | **Total** | **30.5 min** | **9.6 / 10.6 min** | **Same tests:** I compared the second fork run with the E2E log of the upstream run, suite by suite. All 118 suites ran, and each one passed the same number of checks (1161 in total). **Cost:** more runner minutes, because each shard repeats about 2.5 minutes of setup. Standard runners are free for a public repository. ## Not verified here - **A master push.** On a fork the GHCR push cannot run. The expectation is that `Build and push to GHCR` gets cache hits from `image-check` and finishes in a minute or two instead of 4–4.5. The first master run after merge will show whether it does. - **Branch protection.** If it requires other job names than `🎭 Playwright E2E Tests` and `🏗️ Build & Publish Docker Image`, those may need adding. I cannot read the protection settings, so I could not check. ## Tests run locally - `sh test-all.sh` passes (on Windows with `PYTHONUTF8=1`). - `test-issue-1956-release-routing.js` passes, and fails as expected when `go-test` is removed from the `needs` of `build-and-publish`. - actionlint 1.7.7 reports only the existing self-hosted runner label. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
f3ae8ac25a |
feat: sync a logged-in user's settings across devices (part B) (#2130)
Part B of #2128: a logged-in user's settings follow them across devices. Log in on a phone and your own nodes, favorites, customizer and filters are there; a change on one device reaches the others within a minute or when you return to the tab. **This PR builds on #2129.** Until that one is merged, the diff here includes it. The commits for this part start at `docs(specs): settings sync for optional user management (sub-project B)`. ## The situation Everything a visitor sets up lives in one browser's `localStorage` (about 100 keys in `public/`). A second device or a cleared cache starts from zero (#895). ## What this PR adds **Storage.** `users.db` schema v2: one JSON document per user in `user_settings`, with a revision number and a generation id. A write succeeds only when the client's revision and generation match the stored ones, so two devices cannot overwrite each other silently. **Server.** `GET`, `PUT` and `DELETE /api/account/settings`, behind the same session and CSRF checks as the account routes. - The server owns the list of synced keys (61 keys, [`settings_allowlist.go`](https://github.com/efiten/CoreScope/blob/feat/settings-sync/cmd/server/settings_allowlist.go)) and sends it to the client, so the two cannot drift. - A hard denylist, checked first, refuses `meshcore-api-key`, every `corescope_channel_*` key and `live-channel-colors` (#725). The colour map is keyed by channel hash, and for a user-added channel that hash is `user:<name>`, which would expose hashtag channel names. - Documents are capped at 256 KiB, measured like `JSON.stringify`. PUT is limited to 60 requests per hour per user. A stale revision gets 409 with the current document. **Client** ([`settings-sync.js`](https://github.com/efiten/CoreScope/blob/feat/settings-sync/public/settings-sync.js)). Inert unless the feature is on and someone is logged in. - It wraps `localStorage.setItem` and `removeItem` for allowlisted keys only and pushes 2 seconds after the last change. - It pulls on login, page load, tab focus and every 60 seconds while the tab is visible. - **Merge:** three-way, against a per-device baseline that belongs to one user and one document generation. Lists (own nodes, favorites, saved filters) merge per item, so an item added anywhere is kept and an item removed on one device does not come back from another. Single values: the profile wins unless only this device changed it. - Remote changes are written without a push, theme and colour-blind preset are re-applied, and the current page re-renders (skipped on account pages and while the geofilter editor is open). **UI.** - Logout asks: keep my settings on this device (default), remove them from this device, or cancel. Channel keys are never removed: no copy exists anywhere else. - The account page gets a "Settings sync" section: last synced time, "Sync now", what is and is not synced, and "Delete synced settings from my account". ## Not synced Layout and device state (panel and column widths, collapsed panels, map positions, geofilter drafts), channel data (#725), the API key, and all `sessionStorage`. The full list is in the [spec](https://github.com/efiten/CoreScope/blob/feat/settings-sync/docs/specs/2026-10-06-user-settings-sync-design.md). ## Performance - One GET per page load, tab focus and minute while visible; one debounced PUT per burst of changes. - The `setItem` wrapper costs one Set lookup per write for non-synced keys. A synced write reads one small revision key, not the stored document. - The server reads or writes one row per request. ## Verification - `internal/users` and `cmd/server`: `go vet` and `go test` pass locally (22 new Go tests), including a test that every allowlisted key still occurs in `public/`, and denylist tests. - `tests/unit/test-settings-sync.js`: 79 passing (vm, real module). The cases cover the merge table, two tabs sharing one storage, stale answers after a push, delete while a push is in flight, and logout while the final push fails. - `sh test-all.sh` exits 0. - `tests/e2e/test-user-management-e2e.js` (10 steps, 4 of them new) passed locally with two browser contexts as two devices: a favorite and the packet time window travel from device 1 to device 2, a removal does not come back, and "remove from this device" clears the synced keys while a channel key stays. - Checked by hand on a staging instance with a desktop and a phone on one account. ## Not in this PR - On a shared browser where the previous user chose "keep", the next user's first login merges those settings into their own account. The user guide says to choose "remove" on shared computers. - Saved filter expressions are synced as typed, including any channel names written in them. The guide says so. - Realtime push between devices; the minute pull is the sync interval. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
d232072f85 |
feat: optional user accounts (part A: foundation) (#2129)
Part A of #2128: optional, off-by-default user accounts. With the feature off nothing changes; with it on, visitors can register and log in, and admins manage users and use the operator actions without the API key. PR #2130 (settings sync) builds on this one. The two are meant to be merged together. ## The situation - Operator actions (geofilter save and prune, backup, perf reset) need the shared `apiKey`. There is no per-person right. - Nothing in CoreScope knows who a visitor is, so the requests in #2128 that need that (#1835, #2092, #1508, #730) have nothing to build on. ## What this PR adds **Two new Go modules** - `internal/users`: a separate `users.db` (SQLite through `modernc.org/sqlite`) with users, sessions, single-use tokens, an audit log and a mail log. Passwords use argon2id. - `internal/mailer`: a `Mailer` interface with a Brevo client (send, delivery events, webhook parsing) and an in-memory fake for tests. **Server (`cmd/server`)**, active only with `userManagement.enabled` - 24 routes, all documented in OpenAPI under the `users` tag ([`auth_routes.go`](https://github.com/efiten/CoreScope/blob/feat/user-management/cmd/server/auth_routes.go)): - auth: register, activate, login, logout, me, forgot, reset; - account: profile, password, email change with confirmation, sessions, self-delete; - admin: list, detail, disable, enable, delete, role, resend activation, manual activation, mail status refresh; - a Brevo webhook, registered only when `mail.webhookSecret` is set. - `requireAdmin` replaces `requireAPIKey` at the 7 operator call sites: the API key **or** an admin session. With the feature off it is the old API-key gate (`TestRequireAdminWithoutUserManagementIsAPIKeyGate`). - `/api/config/client` gets `userManagement: {enabled: true}` only when the service started; with the feature off the response is byte-identical. **Frontend** - `auth.js` (header account control, request helper that adds the CSRF header), `account.js` (login, register, activate, forgot, reset, confirm email, my account), `admin-users.js` (`#/admin/users`, deep-linked filters), `account.css` (theme tokens only). - On phones the top-bar control is hidden, so a conditional entry goes into the bottom-nav "More" sheet and the nav drawer. - The customizer geofilter tab and the Perf "Reset stats" button use the admin session when there is one. **Config.** A `userManagement` block (`config.example.json`, [`docs/user-guide/accounts.md`](https://github.com/efiten/CoreScope/blob/feat/user-management/docs/user-guide/accounts.md)). The Brevo key can come from `CORESCOPE_BREVO_API_KEY`. The server refuses to start when the block is enabled but incomplete. ## Security choices - Session cookie `cs_session`: HttpOnly, SameSite=Lax, Secure when `publicBaseUrl` is https. Every cookie-authenticated state change needs the `X-CS-CSRF` header and a matching Origin. - Activation needs the token **and** the account password. Without the password, an attacker who keeps re-registering a known address could get the owner to activate an account that carries the attacker's password. - Register, forgot and email change answer identically for known and unknown addresses. A password reset ends all sessions, a password change ends all other sessions, and both end outstanding email-change links. - Rate limits: login 10 per 15 minutes, register and forgot 5 per hour, per IP and per address. The bucket count is capped. `trustedProxies` makes the per-IP limits see real client IPs behind a proxy. - Server logs carry `#<user id>`, never addresses, tokens or passwords; mail-provider error texts are redacted before logging. ## Performance No change to an existing hot path with the feature off. With it on: - One `users.db` lookup per authenticated request (session by token hash). - The admin user table rebuilds its `tbody` on each filter change. `users.List` caps the result at 1000 rows (`internal/users/users.go`), which bounds the rebuild. - `map[string]interface{}` in `openapi.go`: 79 before, 78 after. ## Verification - `internal/users`, `internal/mailer` and `cmd/server`: `go vet` and `go test -race` pass locally. 121 new Go tests. - `cmd/server` with `-tags e2etest`: vet and the e2e hook tests pass. - `sh test-all.sh` exits 0. `tests/unit/test-user-management-ui.js`: 67 passing (vm, real modules). - `tests/e2e/test-user-management-e2e.js` (6 steps) passed locally against an `e2etest` build with the fake mailer and against a feature-off build. CI builds the `e2etest` binary and runs the suite on a second server (`deploy.yml`). - On a staging instance with a real Brevo key: register, activation mail delivered, activate, admin table, "Refresh status" showing sent, deferred, delivered, opened and clicked. ## Not in this PR - Settings sync (#2130), the admin dashboard, approval flows and notifications (parts B to E of #2128). - A `requireReadAuth` mode (#1835). Sessions from this PR are what such a mode would accept. - Binary size and build time with `modernc.org/sqlite` linked next to `mattn/go-sqlite3` were not measured. Their driver names do not collide. #1992 discusses the driver choice. - No Brevo webhook was configured on staging; delivery status there came from "Refresh status". --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
000d9ab030 |
feat(coverage): RF noise-floor layer on the Mobile RX coverage page (#2113)
The ingestor already stores the noise floor that CoreDrive RX companions
report with each GPS fix (`client_rf_samples`, opt-in through
`clientRfSamples`, `cmd/ingestor/client_rf_sample.go:62`), but nothing
reads it back. An operator who enables it collects the data and cannot
see it. This adds the read side: `GET /api/rf-noise` and a Signal/Noise
toggle on the Mobile RX coverage page.
## What it does
- **`GET /api/rf-noise?bbox=&z=&days=`** returns a GeoJSON hex grid with
the median, quietest and noisiest noise floor per cell, in the same
shape as `/api/rx-coverage`. It is registered always and 404s unless
`clientRfSamples.enabled` is true, the same pattern as the coverage
routes.
- **Stationary samples are excluded.** A parked companion logs hundreds
of readings at one point, which would otherwise define its cell.
- **Coverage page:** a Signal/Noise toggle, rendered only when
`/api/config/client` reports `clientRfSamples: true`. The noise layer
reuses the coverage colour tokens with the axis inverted, because a
lower dBm is quieter. Tiers are at -115 and -108 dBm.
- **Deep link:** `#/rx-coverage?layer=noise` opens on the noise layer.
- **Empty and failed answers** are labelled on the map ("No RF samples
in this view yet", or a retry hint), so a blank map never reads as
"feature off".
No new configuration key: it reads the existing `clientRfSamples`
section that the ingestor already uses. Default off, so nothing changes
for an instance that has not opted in.
## Where it comes from
Ported from the efiten/CoreScope fork, where it has run on
analyzer.on8ar.eu since September (fork commits `42d09f0e`, `80581c43`,
`cde95078`). The cherry-picks conflicted with upstream's newer
`routes.go`, `types.go` and `rx-coverage.js`, so the final state was
ported by hand. The fork-only `/scopes` route that sat next to it in the
same hunk is deliberately left out.
## Performance
The query is bounded by `sampled_at` (indexed, `idx_crf_prune`) and the
bbox, aggregation is one pass plus a per-cell sort, and the response is
capped at 5000 cells. Measured on analyzer.on8ar.eu, all of Belgium
(`bbox=49.4,2.4,51.6,6.5&z=9`), from a client in Belgium, so network
time included:
| window | samples | cells | response time |
|---|---|---|---|
| 7 days | 10,835 | 241 | 0.37 s |
| 30 days | 43,303 | 429 | 0.39 s |
That table holds 45,270 rows in total. It is only read when someone
opens the noise layer, never on ingest or WebSocket paths.
## Tests
- `cmd/server/rf_noise_test.go`: 8 tests for the aggregation (median and
extremes, stationary exclusion, the cell cap, empty input) and the gate
(404 when off, even with data present). All pass, and the full
`cmd/server` suite passes locally.
- `tests/unit/test-rx-coverage-noise.js` (new, in `test-all.sh`): the
colour axis runs the right way, including both tier bounds and a string
median from the API. Slices the real function out of `rx-coverage.js`
and fails on master's copy, which has no thresholds.
- `tests/e2e/test-rx-coverage-noise-e2e.js` (new, wired in `deploy.yml`
and `scripts/non-unit-tests.json`): no toggle when the flag is off;
Noise fetches `/api/rf-noise`, draws the cells, swaps legend and
subtitle, puts `layer=noise` in the hash; a deep link opens on the noise
layer and an empty answer shows the message. 3/3 locally; against
master's `rx-coverage.js` and `roles.js`, 1/3 (only the "flag off" case
passes, as it should).
- `test-rx-coverage-viewport-e2e.js` still passes. ESLint 8 reports 0
errors on the changed files.
## Not done
- The thresholds (-115 / -108 dBm) are fitted to the fork's own data
(1,241 moving samples at the time) and are constants in
`rx-coverage.js`. Per AGENTS.md rule 8 they belong in the customizer
eventually; not in this PR.
- The E2E suite mocks the API. The real endpoint is covered by the Go
tests and by the measurements above, not by a fixture with seeded
samples.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
efitenandClaude Opus 5.5
|
ef12535364 |
test(map): wait for the new route sidebar before measuring it (#2109)
Step (9) of `tests/e2e/test-path-inspector-e2e.js` fails intermittently on master with `(3-9) fixture route coverage: 0 !== 700`: 3 of the 18 runs that executed it since 2026-09-30, including master runs [37120155356](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37120155356) and [37156362836](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37156362836). A red master run skips the image build, so `:edge` is not rebuilt. ## Cause The step clicks a candidate and then waits for any `.mc-rt-sidebar`. The previous route's sidebar survives the hash-only `goto` to `#/map`, and `drawPacketRoute` (`public/map.js:1080`) replaces it only after `/api/resolve-hops` answers. So the wait matched the old sidebar, and the checks that follow measured either the old sidebar or, if the replacement landed between the locator resolving and the evaluation, a detached element, whose `getBoundingClientRect()` width is 0. This is a test defect. The app keeps the previous route on screen while the next one loads, which is intended. ## Fix Remember the sidebar before the click and wait until a different one is in the DOM. Test-only, one file. ## Evidence Locally against the fixture, with `/api/resolve-hops` delayed through `page.route` in a throwaway copy of the test, measuring which sidebar the step-9 assertion sees: | Delay | Before | After | |---|---|---| | 0 ms | new sidebar, pass | new sidebar, pass | | 80 to 100 ms | old sidebar, `isConnected: false`, width 0 (3 of 5 runs) | new sidebar, pass (5 of 5) | | 1500 ms | old sidebar, pass: the step checked the wrong page | new sidebar, pass | The unmodified fixed test then passed 3 of 3 runs. ## Not done - I did not look for the same wait pattern in other suites. - The delay probe is not committed. It needs a timing window to be useful, and that window would be a fixed sleep in CI. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
9e49db9725 |
feat(nodes): group adverts by observed route evidence (#2085)
Red commit: `ac60248` ([two intended label failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37137429333)). Original grouping red: `767da9b` ([CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/36353796690)). Fixes #2073. #2088 is merged. Rebased onto master `415362c`; all eight patches are unchanged, including the test-first history. Recent Adverts groups the bounded sample by authoritative `advert_kind`: Flood, Mixed flood / direct (empty path), Direct (empty path), and Other / unknown. Missing evidence stays unknown. Each advert appears once, preserving reception metadata, order, packet links, counts and origin/history explanations. Wording describes observed remaining paths without inferring original send mode or RF distance. Validation at `6317baf`: - All 185 standalone suites passed; syntax, XSS and lint passed (zero errors, 92 existing warnings). - Browser verified: `http://127.0.0.1:51827` (temporary Go fixture server, now stopped), desktop pane/full view and mobile. Mixed evidence, exact links and caveats checked again after pushing. - The expanded fixture exposes the existing #2104 failure in the unmodified core harness. Supplemental integration with #2105's exact test correction passes 135 checks, with three existing skips. Both PRs remain separate. - [Fresh green CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37138366726) passed; three independent automated reviews found no must-fix issues. E2E assertion added: `tests/e2e/test-e2e-playwright.js:142`. Grouping is O(n) over the existing 20-advert sample. No new requests, settings, colors or dependencies. ## Preflight overrides External OpenClaw runner/profile unavailable; repository checks and actual local Chromium were used. |
||
|
n30nex
|
99a326a6b3 |
test(packets): verify observation hex after rerenders (#2105)
Red commit: `ca553e4` ([expected assertion failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37137426691)). Fixes #2104. The observation hex test now opens the existing grouped fixture directly, selects A → B → A with stable-ID locators, and asserts the selected ID and exact raw bytes after each render. Missing fixture data fails explicitly. Distinct observation frames are seeded after schema migration; this test's fixed sleeps and silent skips are removed. Rebased onto master `415362c` after #2090 merged. Both patches are unchanged. The previous #1122 layout blocker now passes against the actual current server and assets. Validation at `5db8017`: - All 185 standalone frontend suites passed. - Core browser: 133 passed, three existing skips; #1122 layout 6/6, filter 11/11, grouped-collapse 4/4. - Syntax, YAML, fixture SQL, XSS and lint passed (zero errors, 92 existing warnings). - Fresh red CI reached the intended detached-observation assertion. [Green CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37138187496) passed; three independent automated reviews found no must-fix issues. Only test code and fixture setup change; no configuration/customizer implications. ## Preflight overrides External OpenClaw runner/profile unavailable; repository checks and actual local Chromium were used. |
||
|
efitenandClaude Opus 5.5
|
415362c5af |
fix(packets): resolve path hops with the observer that heard them (#2097) (#2099)
Fixes #2097. Every 1-byte path prefix on this network is shared. Measured on the live deployment: 254 prefixes cover all 2043 nodes, nine of them on a single prefix, and not one node has a prefix to itself. The packets page printed one of those nine as a certainty. It could already have said otherwise. `hop-resolver.js` marks such a hop `ambiguous` and returns its full candidate list; `hop-display.js` renders a warning badge with a count and a candidate popover; `renderHop(h, observerId)` reads a per-observer cache key. None of it fired, because `HopResolver.resolve()` takes six parameters and `packets.js` passed one. Without `observerId`, `packetIata` is null, `nodeInRegion()` never runs, no candidate is flagged `regional`, `globalFallback` stays false, and `hop-display.js` computes a badge count of 0. The whole chain stayed silent while the data said it was a guess. ## What this changes **Resolution carries the observer.** `resolveHops()` takes it and writes the per-observer cache key that `renderHop()` has always read and nothing ever wrote. `resolveHopsForPackets()` groups a multi-packet call by observer so each group is filtered on its own. One site was the bug in miniature: it wrote its result under `hopNameCache[k + ':' + pkt.observer_id]` after resolving without that observer, so the key promised something the value was not. **The observer's position is the anchor.** `resolve()` has always accepted `observerLat`/`observerLon` as the anchor at the receiving end: when nothing later in the path is resolved, the observer is the next known position, which is what `pickByAffinity` needs. `observerPosition()` is the single lookup, used by both resolve paths. This matters more than the IATA route, which turns out to be dead on this network. `nodeInRegion()` looks the observer's code up in `/api/iata-coords`; **0 of 42 observers have their code in that table** (`ANR BRU GNE HEP KJK LGG MST NRW OBL OST` are all missing, while its 54 entries are `AMS`, `APC`, `ATL` and the like). So the 300 km filter has never excluded anything here. 27 of 42 observers report lat/lon directly, and that works. Filed separately as #2098, since it is a network-wide lookup that resolves nothing and hop resolution may not be its only consumer. **Each badge belongs to its pill.** The badge is a sibling after the pill, so a path read `A [8] → B [6] → C` with nothing to say which name the 8 qualified. Pill and badge now render inside one `white-space: nowrap` `.hop-group`, leaving the separator outside the pair. Only a hop that carries a badge is wrapped; wrapping every hop would change the layout of every path for nothing. That markup predates this work, but these commits are what made it visible — before them the badge essentially never appeared on this page. **The list summarises, the detail pane does not.** Every 1-byte hop getting its own badge turned a table row into a line of warning triangles. `renderPath()` takes `{ summary: true }` for the three list call sites: names with no per-hop badge, and one indicator reading "N of M hops have more than one candidate". The two detail-pane sites are unchanged and keep a badge per hop, because that is where the candidates actually get read. `HopDisplay` gains `opts.badge === false` for it, and the hop keeps its `hop-ambiguous` class either way so CSS can still mark it. ## Scope Deliberately not included: chaining the next resolved hop to narrow a candidate set to one. `pickByAffinity` already scores on graph edges and would do it; it needs neighbouring context this call does not yet supply, and that belongs in its own change to `hop-resolver.js`. What is here makes the uncertainty visible and picks a defensible candidate rather than the first in index order. ## Verification 30 unit tests in `tests/unit/test-issue-2097-hop-ambiguity-badge.js`, registered in `test-all.sh`. Full frontend suite exits 0. Four of them are structural guards over `packets.js`, and each was mutation-checked by reverting the change it protects: - no `HopResolver.resolve()` call passes the hops alone - no call passes an observer id and drops its position - one helper does the observer lookup - the detail pane calls `renderPath` without `summary` The behavioural test for the anchor runs with `iataCoords` deliberately empty, which is the live state, and lists the distant candidate first: without an anchor the resolver keeps candidate order and picks it, with one it does not, and the hop stays reported as ambiguous with all candidates listed. **Browser validated** on a staging deployment of this branch, against live data: | | list | detail pane | |---|---|---| | per-hop badges | 0 | 24 | | path indicators | 1 | 0 | | badges outside a `.hop-group` | — | 0 | On the packet that prompted the issue, the first hop now resolves to a repeater 10.6 km from the transmitter instead of one 126 km away, and the second hop resolves to the only candidate that has a recorded neighbour edge matching the next hop in the path. ## Note for reviewers The last commit exists because staging disagreed with the tests. After the anchor landed, the detail pane still picked the distant candidate: it resolves through its own `HopResolver.resolve` call, which had the observer id but a null position. The guard that now fails on exactly that shape is what would have caught it before the deploy. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
3b415fb007 |
fix(live): enable node filtering after its handlers are ready (#2103)
Red commit: `8974616` ([CI assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37064461392)). Fixes #2094. The Live node-filter input now starts natively disabled and becomes editable after its input, keyboard and blur handlers attach. Previously it accepted text while initial node loading was pending, then never processed that text without another keystroke. The change adds one HTML attribute and one initialization assignment. Failed initial loading still reaches the existing error handler and installs the controls. Saved filters, URL precedence, autocomplete and keyboard selection keep their existing behavior. No new settings, styling, dependencies, requests or packet-processing work. ## Validation - The existing CI-wired Live browser suite delays or aborts the real initial node request. It verifies rejected premature typing, later editability, actual suggestions, keyboard selection and URL state. Both new cases failed on the red commit; all six cases pass locally on `11fab46`. - All 183 standalone frontend suites passed, plus lint and XSS gates. Separate browser checks passed saved-filter restoration and URL override; desktop/mobile loading/ready screenshots were inspected. - A broader local core run exposed the unchanged stale observation-handle test tracked in #2104. It is outside this Live fix. Full [CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37067337557) passed at `11fab46`, including Go, browsers and both container architectures; three independent reviews are clear. E2E assertion added: `tests/e2e/test-1110-live-filter.js:69`. Preflight overrides: external script/OpenClaw profile unavailable; repository checks and local Chromium used. |
||
|
n30nex
|
b9fb3d244c |
feat(analytics): split adverts using recorded route evidence (#2088)
Red commits: `dc7dcbe`, `76e3217` ([assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37059977385)). Fixes #2041; supplies #2085's backend contract. Records at most two route bits per retained advert. Node flood counts now use the same evidence, including mixed and transport-flood adverts. Analytics failures log/count errors while core observation, path and liveness writes continue. The unrelated recent-advert limit clamp is removed. Direct adverts (empty path) describes the observed remaining path across valid hash widths. Firmware can forward an advert down to an empty path; this cannot establish original send mode or RF distance. The `zero_hop` API spelling remains compatible. Successful post-upgrade evidence writes survive observation replacement and restart. Older overwritten frames and failed writes can leave gaps. The UI explains automatic background backfill. ## Measured validation Synthetic Windows workloads; timings are not production guarantees: - 300K retained adverts, one-hour window: median 64.35→39.84 ms; 13.98 MB→145 KB allocated. Seven-day allocations unchanged; five paired samples. - 1M transmissions/11M observations: backfill 5m43s. Simple 10 Hz writer: all 3,423 observations persisted; p99 28 ms, maximum 323 ms. - Simulated pre-backfill cursor: retained evidence ready 4m21s; full catch-up 6m41s, 122 cache invalidations. Normal completed-migration startup bulk-loads current evidence. Portable opt-in scale tests and window benchmarks are included. All 183 frontend suites, full server tests, targeted races, lint and real-API desktop/mobile checks passed. Full [Linux CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37063210229) passed at `36def9c`; three independent reviews are clear. Local Windows testing hit the unchanged symlink-privilege test. No configuration changes. Preflight overrides: external script/OpenClaw profile unavailable; repository checks and Chromium used. |
