mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-12 13:18:03 +00:00
cb9f4ff392f410839e02de3aa0349d54600a8d7f
3101
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 efitenandClaude Opus 5.5
|
cb9f4ff392 |
fix(observer-logins): bare host in the firmware server line
Dutch-MeshCore firmware before merge 32dd8ad (2026-10-19) does not accept a scheme in mqtt1.server: it builds <proto>://<host>:<port> with proto from the port alone, so mqtt://host became mqtt://mqtt://host:1883. Reported by an observer operator on the live instance. The mail now writes the bare host and adds a scheme only when the port would pick the wrong one (for example TLS on 8884), which needs firmware from 2026-10-19 or later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
25ae52e52f |
fix(observer-logins): final review fixes
1. Audit actions renamed to observer.login.create/resend/revoke/reissue and
observer.binding.set so the admin audit filter accepts them; test pins
auditActionRE against all five, create/resend rows now asserted.
2. "three formats" is four formats in openapi.go and the doc.
3. hookSecret needs 16+ characters (env override applies first); the E2E
secret is lengthened to match.
4. broker.transport accepts only tcp or websockets.
5. openapi GET /api/account/export names observerLogin and observerBindings.
6. observer.binding.set audit detail carries previousUserId ("none" when
the key was unbound).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
efitenandClaude Opus 5.5
|
134c9f5555 |
feat(observer-logins): Home Assistant block in the credentials mail
Adds the meshcore-ha MQTT broker form fields as a fourth block, between the firmware commands and the plain values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
37104508cb |
docs(observer-logins): EMQX health-check race while CoreScope is up
EMQX 5.8.6 can mark the HTTP authenticator down for one health-check interval (15 s measured) when a health check lands between a closed keep-alive connection's gun_down and its client exit (ehttpc 0.6.0 do_after_gun_up). Personal logins are refused for that window. No EMQX setting changes it, and 5.10.5 (ehttpc 0.7.3) has the same code. Section 3 now says so with the measured counts, and the recovery line no longer states an unmeasured interval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
07a93f1852 |
docs(observer-logins): broker setup and migration guide
docs/observer-mqtt-logins.md covers the CoreScope config, the two hook endpoints, the EMQX authentication and authorization chains (tested on EMQX 5.8.6), the closing deny rule for obs_ usernames, the measured behaviour while CoreScope is down, and the migration away from the shared login. accounts.md links to it; config.example.json gets the observerLogins block with placeholder values. The credentials mail now gives the firmware server with a scheme (mqtt, mqtts, ws or wss). For a bare host the Dutch-MeshCore firmware picks the scheme from the port alone (MQTTBridge.cpp setupSlot: 8883 mqtts, 443 wss, else mqtt), so TLS or websockets on another port connected with the wrong transport. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
f691532f73 |
fix(observer-logins): admin tab updates rows, not whole tables
The inline Revoke/Reissue confirm re-rendered both tables, and every observer row carried a select listing every login. Now: - renderLogins() and renderObservers() are separate; confirm, cancel and revoke/reissue state touch only the logins table. - Each observer row shows its owner and a Reassign button; clicking it swaps one select into that row, with options built once per load. Cancel or Escape restores the button and focus. - A saved owner updates only that row and the not-migrated summary. - The select is disabled while the PUT is out; a refusal puts the saved value back. The E2E now reassigns the bound observer to "not bound" and back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
51b6934f8b |
feat(observer-logins): account and admin UI
Account page: an "Observer MQTT login" section below Companions (only with userManagement.observerLogins) with three states: request, login (username, created, bound observers, "Mail my credentials again") and revoked. #/account?section=observers scrolls to it. No password is rendered; no API returns one. Admin: an "Observer logins" tab (#/admin?tab=observer-logins) with the logins (Revoke / Reissue behind an inline confirm, no browser dialog) and every observer from one GET /api/observers joined client-side with the bindings: owner, "not migrated" or "orphaned", and a select to reassign or unbind. roles.js now copies observerLogins from /api/config/client into MC_USER_MGMT; without it neither the new UI nor the existing "My observers go offline" notification option could appear. The user-management E2E server (deploy.yml and the test header) runs with observerLogins enabled; the E2E covers request, mail with obs_, bind via the authz hook, resend, admin tab, revoke and reissue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
b65a75c88d |
feat(observer-logins): offline mails for an owner's own observers
New event observer.offline.own, open to every role and off by default. An owner gets the observer offline/online mails for observers bound to their own login; an admin with observer.offline too gets only the all-observers mails. Creating a login adds the event to the user's preferences. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Sonnet 5.5
|
6784d71baa |
feat(observer-logins): admin revoke, reissue and rebind
Admin routes to list logins and bindings, revoke or reissue a login and reassign or unbind an observer. The login index now follows account deletion (login removed, owned pubkeys orphaned) and disable/enable (separate flag, so enable does not undo a revocation). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
c370678f4b |
feat(observer-logins): account endpoints and credentials mail
GET/POST /api/account/observer-login and POST .../resend, gated on userManagement.observerLogins. The password is hashed for the broker and sealed for resend; it only leaves the server in the credentials mail (three pre blocks: meshcoretomqtt, Dutch-MeshCore firmware, plain). Resend is limited to 3 credential mails per 24 h (mail_log count). The account export gains the login metadata and bindings, never the credential material. /api/config/client announces observerLogins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
098067b13d |
test(observer-logins): wait for the touch goroutine in the throttle test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
9419101075 |
feat(observer-logins): EMQX authn/authz hook endpoints
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
7fc483b8a7 |
fix(observer-logins): cap automatic observer binds per user
A managed login could bind an unlimited number of invented pubkeys by publishing under them, growing users.db and the in-memory owners map without bound. Auto-bind now stops at 20 owned pubkeys per user (deny, no store call, logged); admin assignment is not capped. Ownership is counted per user in O(1). BenchmarkAuthorizeBound: 194.9 ns/op (was 191.9). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
c939bd8ef5 |
feat(observer-logins): credential generation, index and hook decisions
In-memory index of MQTT logins and observer bindings that answers the broker's authenticate and authorize hooks without touching users.db on the bound path. Binding table: owner allow, other/orphaned deny, absent binds through the store (first claim wins) outside the lock. BenchmarkAuthorizeBound (500 logins, 2000 bound observers): 191.9 ns/op. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
890ed0f822 |
feat(observer-logins): users.db v7 logins and observer bindings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
fc4f648905 |
feat(observer-logins): config block and secret box
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
c085ca8cba |
fix(rf-params): respect node visibility in responses (#2276)
The RF endpoint now applies the current blacklist and hidden-name policy to both exposed node metadata and deviations, including after cache warm-up. Unhiding a node restores its data without waiting for cache expiry. Fixes #2268. The shared RF snapshot and peer calculation remain unchanged; filtering copies only the response data. This adds a linear node pass and bounded deviation copies, with no additional database calls or cache invalidation. The feature remains opt-in. Validation: RED reproduces both response leaks. The full server and standalone frontend suites pass, as do focused race tests and the RF browser suite. Regressions cover warm/cold caches, policy updates, unhide, visible controls and default-off behavior. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
n30nex
|
82aafb53c3 |
fix(analytics): count roles within selected filters (#2275)
Analytics Nodes now counts roles from the same complete, filtered node list used by Network Status. Region/area selections no longer display the API's global role totals. The global API metadata remains available to other consumers. Fixes #2213. Validation: RED reproduces the mismatching totals; all standalone frontend suites and the unchanged server suite pass. Browser checks cover region and area changes and empty results. Regressions also cover 501-node pagination, unknown roles, unchanged metadata and request counts. Counting happens inside the existing O(n) status pass, using six bounded counters and no additional requests. No configuration changes. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
n30nex
|
7ebbed58dc |
fix(analytics): anchor resize handles to table headers (#2274)
Resize handles now use their static table header as the containing block. Already-positioned headers, including sticky Nodes headers, retain their existing positioning. Fixes #2217. Validation: assertion-failing RED commit followed by the fix; all standalone frontend suites and the unchanged server suite pass. The browser suite passes 48 checks covering desktop/mobile bounds, real dragging, saved widths and sticky headers. The browser regression waits for the initial theme render before measuring. The fix adds one positioning check per header during setup, with no new requests, dependencies or settings. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
n30nex
|
12e0a40a82 |
test: make hop coverage clock checks deterministic (#2273)
Three hop-coverage tests sampled the clock before database setup, then allowed only five seconds of drift. Slow setup made otherwise correct coverage checks fail. They now use the existing clock hook, restore it after fixture workers finish, and assert exact equality. Production code is unchanged. Fixes #2272. Validation: the original full run reproduced all three failures; the corrected full ingestor suite passes. Focused race checks pass, and row/deletion/ordering assertions remain intact. This is a test-only stabilization of existing failing assertions, not a new runtime feature. |
||
|
efitenandClaude Opus 5.5
|
7fa86fe1c1 |
docs(release-notes): v3.15.0 (#2263)
Release notes for v3.15.0: `docs/release-notes/v3.15.0.md` and the `CHANGELOG.md` entry. The release workflow uses the notes file as the GitHub release body. ## What is in it - 77 commits since v3.14.0 (76 merged pull requests plus this one): 27 fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs, 1 chore. Nineteen feats, so minor. - "Read this before upgrading" first. It covers: - the liveness change (#2261, #2262); - the `users.db` v6 migration and what it means for a rollback (#2174); - the backfill at every ingestor start (#2133) and the 24 h planner-statistics refresh (#2158); - the caches that trade freshness for latency (#2165, #2222, #2147, #2156); - changes in hop names and region placement (#2188, #2190, #2234, #2245, #2220); - the role thresholds on favorites (#2215); - the geo-filter save on a single-file mount (#2137); - the `/api/stats` polling in hidden tabs (#2162). - Features, fixes, performance, then tests/CI/docs. - 29 issues closed, taken from each pull request's `closingIssuesReferences`. - Contributors: pull request authors and issue reporters with counts. The commit co-author trailers only name the pull request's own author or tooling. ## How it was checked - I read every pull request body. Every number in the notes comes from a pull request body. - I checked the config keys and defaults against `config.example.json` on `2bb000f2`. - I checked the UNASKED and opt-in claims for #2125, #2133, #2235 and #2239 against the source. - For the other pull requests I relied on their bodies and did not open the diffs. ## Not covered - The uncertain count is a single count on one staging instance on the day of the change (1,682 repeaters and rooms). It is not a before/after comparison. - #2133 gives no measured cost for its backfill on a large database, and the notes say so. ## After merge Tag `v3.15.0` on the merge commit once its `:edge` image exists. Then check in the registry that `v3.15.0`, `v3.15`, `v3` and `latest` share one digest, and that the release has the notes as its body and both `corescope-decrypt` binaries. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v3.15.0 |
||
|
efitenandClaude Opus 5.5
|
2bb000f2aa |
feat(notify): adaptive offline threshold and certain node analytics (#2260) (#2262)
Part 2 of #2260. **Builds on #2261** (certain liveness): review/merge that first; until then this diff also shows its commit. ## Adaptive offline threshold `userManagement.notifications.adaptiveOffline` — **off by default**: ```json "notifications": { "enabled": true, "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 } } ``` The fixed `infraSilentHours` (72 h) treats every repeater the same. With this on, a watched node's `node.offline` window is learned from its own **certain** traffic (the #2261 rule): 1. Certain packets per hour over the week **before its last certain packet**. Ending there, not at now, keeps an outage from dragging the rate down until the fixed window returns and the node reads "back online". 2. Empty runs are classified: ≤ 6 h and recurring on another day at the same hours = normal silence (kept; the window is never shorter than the longest + 1 h); any other run = outage, left out of the learning so it neither stretches the window nor blocks learning. 3. λ = 10th percentile of the remaining hourly counts, so quiet nights don't alert. 4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson, ln 10⁶ ≈ 13.8). 5. Clamped to [`floorMinutes`, the role's silent window]. Kept on the fixed window: λ < 1/hour, under 24 h of history in memory, or a 10th-percentile hour heard by fewer than `minObservers` observers (one observer dropping out must not read as the node going offline). Silence is measured up to the newest packet in the store, so a stalling feed stops every node's clock. A node stored offline comes back only on evidence from the moment it went offline on. The notifier caches each learned window for 10 minutes and drops unwatched nodes. For the #2260 repeater (~840 packets/day) the mail would have gone out about 30 minutes into the outage, instead of never (or 72 h with #2261 alone). ## Certain node analytics - Each `activityTimeline` bucket carries `certain`: the part that can only be this node (`certainFor`, the packet-level form of the `certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other endpoints are unchanged. - Availability and longest silence use certain buckets only; the longest silence now includes one **still running** (an outage in progress is the silence worth seeing). - The chart stacks certain (`--accent`) and uncertain (`--text-muted`) — existing theme variables, so no new customizer entry. Descriptions updated. ## Performance `CertainActivity` walks a week of packets per watched node, at most once per node per 10 minutes, with the read lock taken **per node**: ``` BenchmarkCertainActivity-16 12.1 ms/op (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold) BenchmarkNoteCertainRelays-16 121 ns/op 0 allocs ``` Node analytics adds one `certainFor` per packet of the requested node (a prefix-map lookup, or the cached decoded JSON for its own packets). ## Tests - `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet nights set the rate; > 10 % empty hours, short history, too few observers → fixed. - `CertainActivity` ends at the last certain packet and counts only certain packets — verified red by mutating the window to end at now. - Notifier end to end: learned window passed → bad; inside → good; too few observers / setting off → fixed 72 h; cache refresh and prune. - `certainFor`, `longestSilence` (incl. running silence), config defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket. - Frontend: `NodeAnalyticsTimeline.series` (new `tests/unit/test-node-analytics-certain-timeline.js`). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork. ## Validation Server on the freshened e2e fixture: analytics buckets carry `certain` (e.g. 89 of 90 packets certain for one repeater). **No browser screenshot**: Playwright isn't installed on this machine. ## Later (rule 8) `floorMinutes` / `minObservers` are server config, not theme settings; an admin UI for notification settings would be the place, not the customizer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
a08f2f4e7c |
fix(liveness): count only evidence that can be the node alone (#2260) (#2261)
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold follows in a separate PR). ## Problem A repeater silent since 2026-10-18 showed **Active — Last relayed 4m ago** and **Last Heard 36m**, and its offline mail never fired. Every hit after that date was the 1-byte hop `67`, shared by nine nodes: - the live resolver's affinity tiebreak (`resolvePathForObs`) put those transmissions in `byNode`, which fed `stats.lastHeard` and `LastHeardMap`; - `last_relayed` folds in the 1-byte prefix bucket on purpose (#662), and drove the status in `roles.js` and the notify offline check. ## Fix The store keeps `certainHeard` per node: the newest transmission that can only be that node. | Evidence | Counts? | |---|---| | Own packet (`pubKey`, `srcPubKey`) | ✅ | | Flood relay hop whose prefix, at the packet's path hash size, matches exactly one path-capable node | ✅ | | 1-byte hop shared with other repeaters | ❌ | | Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌ | | TRACE (path bytes are SNR) | ❌ | | Destination (`destPubKey`) | ❌ | A multibyte repeater legitimately appears as a 1-byte hop (the originator picks the hash size, `Mesh::routeRecvPacket` appends at that size), so only uniqueness decides. The rule is applied in every load path through `indexResolvedPathHops` and the new `indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live ingest, late observations. Liveness reads only that: - `/api/nodes` and the node detail: `last_heard` is the newer of `last_seen` and the certain last heard. Before, it was a plain copy of `last_seen`. - health / bulk-health `stats.lastHeard`. - `LastHeardMap` → notify offline check. The separate relay input (`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed. - `roles.js getEffectiveHeardMs`: newest known timestamp, without `last_relayed`. The side pane no longer says "Last relayed"; the Last Relayed row stays (analytics) with an updated tooltip. - New status **`uncertain`** (repeaters and rooms): nothing certain within the silent window, but the shared prefix (`last_relayed`) still in paths. Grey badge with explanation, an *Uncertain* filter on the nodes list and the map, dimmed like stale. The offline mail for such a node says only its shared prefix was still seen. - A prefix also held by a path-capable node in `inactive_nodes` is not unique; adverts with `signatureValid: false` are no evidence; `/api/nodes/search` reports the same `last_heard`. Attribution for analytics (`byNode`, packet counts, `last_relayed`, relay counters, usefulness) is unchanged. ## Performance Hot path: `noteCertainRelays` runs per observation on ingest under the write lock: two hex bytes of header parsing plus one prefix-map lookup per resolved hop. ``` BenchmarkNoteCertainRelays-16 10307640 116.2 ns/op 0 B/op 0 allocs/op (8-hop 2-byte path, 2,000 repeaters) BenchmarkLastHeardMap-16 1902 0.77 ms/op (5,000 nodes; was 3.7 ms scanning byNode) ``` Memory: one string per node ever certainly heard, dropped together with the node's `byNode` list on eviction. ## Tests - `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk, LoadChunked — each with and without resolved_path — live ingest, late observations) feeds `certainHeard`; each of the 8 call sites verified red with a nil prefix map. - `certain_heard_test.go`: hop width per route/payload type; retired namesake prefix; failed signature; uncertain offline mail; shared vs unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`, detail and health agree. - Regression for #2260: last certain packet on day X, 20 shared-prefix hops after it → `LastHeardMap` stays at X and `node.offline` goes bad after 72 h. Verified red with the uniqueness check mutated out. - Notify: `TestEvaluateCertainHeard` replaces the relay-input test. - Frontend: `last_relayed` alone no longer keeps a repeater or room active; newest timestamp wins (an old `_liveSeen` can't hide a fresh certain relay). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork with 457 extra commits. ## Validation Ran the server on the freshened e2e fixture: list, detail and health report the same last heard; e.g. a repeater whose `last_relayed` is 3 min newer than anything certain keeps its certain time. **No browser screenshot**: Playwright isn't installed on this machine; CI's E2E covers the pages. ## Not changed - The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a shared hop only when exactly one candidate is adjacent in the neighbor graph; it did not fire in this case. Left as is. - The `lastHeard=` list filter still filters `last_seen` in SQL. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
AdamandClaude Opus 5.5
|
dbcdca9dc5 |
feat(region-filter): honor ?region= in the URL on every page (#2255) (#2256)
Fixes #2255. Only the packets page read `?region=` from the hash, with its own one-off code. Live, nodes, observers, channels, analytics and scope audit ignored it, so a link like this opened with whatever region the visitor last picked: ``` https://<instance>/#/live?region=SJC ``` We link into our instance from a community directory (one link per region), and wanted those links to land on the live map or node list already filtered, not only on packets. ## What changes **`public/region-filter.js`** - `?region=` (comma-separated, trimmed) overrides the saved selection and is saved like a click in the control, the same as packets did through `setSelected()`. - **Fresh load:** it's applied when the script runs, before any page reads the selection. This matters for live, which fetches nodes before it calls `RegionFilter.init()`. - **In-app navigation:** `init()` applies it before its `await`, so a page that loads after `init()` uses it. If it changed the selection, listeners are told once regions load, so a page that already fetched (live) reloads instead of showing the old region's data under the new region's label. - Unknown codes are dropped by the existing cleanup in `fetchRegions()`. No param, or an empty one, leaves the saved selection alone. **`public/packets.js`** - Drops its own `_pendingUrlRegion` handling; `RegionFilter` covers it. - Removes its region and area listeners in `destroy()`. They were registered without a handle and outlived the page: after visiting packets, a region change on any other page reloaded packets and `replaceState`'d the URL to `#/packets?...` while you were on another page. That already happened with the dropdown on master; the in-app notify above would make it happen on every region link. ## Tests `tests/unit/test-region-filter-url-param.js`, 14 assertions: - URL region overrides the saved selection; comma lists, whitespace, unknown codes, empty param, no param, no hash - fresh load: selection is set before `init()` is called, and `init()` doesn't notify - in-app navigation: a changed region notifies listeners once, the same region doesn't - selection is saved to `localStorage` - packets keeps handles for both listeners and removes them in `destroy()` With `region-filter.js` from master, 5 of the first 8 assertions fail. How I checked: - `test-all.sh` suites all pass except `test-issue-1956-release-routing.js` and `test-preflight-xss-gate.js`. Both also fail on unmodified master on macOS: they shell out to bash-4 features (`mapfile`, and a syntax error in `check-xss-sinks.sh` under bash 3.2). - `eslint@8.57.1` on the two files: 0 errors, 3 warnings, the same as master. - Browser: served `public/` locally with `/api` proxied to a live instance and checked the requests each page made: - fresh load of `#/nodes?region=AUS`, `#/live?region=DFW`, `#/packets?region=HOU`: the first request carries the URL's region and the dropdown shows it - packets → `#/live?region=DFW`: live fetches with the old region, then reloads with DFW; the dropdown shows DFW - packets → `#/nodes?region=AUS`: no packets requests, and the URL stays on `#/nodes` I couldn't run the Playwright E2E suite locally (no Go toolchain on this machine). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
0e3ce48cff |
feat(nodes): flag nodes whose LoRa settings differ from nearby nodes (#1268) (#2210)
Refs #1268. #2209 is merged as `cab4f71c`, and master is merged into this branch (`47ce8264`), so the diff against master is this PR's own change: `ea4d1130` plus the fix in `5cbbe5d5`. ## What it does Two nodes hear each other only when SF and bandwidth match and their frequencies are close enough. `/api/rf-params` now also returns `deviations`: nodes whose setting differs from a majority of the nodes around them on the same band. - **Peers:** nodes with GPS and RF params within `rfParams.compareRadiusKm` (default 50) whose frequency is within 30 MHz. - **Same channel:** frequencies within a quarter of the bandwidth count as one channel (15.625 kHz at BW 62.5). The SX1262 tolerates a transmitter/receiver offset of up to 25% of BW without sensitivity loss, for SF5 to SF12 and all bandwidths. Source: the `FERR_L` row in the [S62F module datasheet](https://lora-alliance.org/wp-content/uploads/2020/07/S62F_SPEC_Ver.B.pdf), an SX1262 module; I did not find the figure in a Semtech document itself. That table also lists -50/-100/-200 ppm for SF12/SF11/SF10, which at 869 MHz is wider than BW/4 at 62.5 and 125 kHz, so BW/4 is the tighter bound there. - **Flagged when:** a setting the node cannot hear is shared by more than half of the group, the node itself counted, with at least 3 peers and no other incompatible setting at the same support. So 5 against 5 flags nobody, and 2 of 4 does not count as a majority. Each deviation carries the majority, how many peers are compatible with it, the peer count, and which of freq/sf/bw differ. - **Coding rate is not compared.** RadioLib starts SX126x radios with an explicit LoRa header (`SX126x::begin` sets `RADIOLIB_SX126X_LORA_HEADER_EXPLICIT` at `6d89348`, the commit the firmware pins in `platformio.ini`), and nothing in `firmware/src` switches to implicit. The header carries the CR, so a CR mismatch still decodes. I checked SX126x only, not SX127x or LR11x0. UI: - Nodes list: a badge such as `LoRa SF7` next to the name of a deviating node (the majority in its tooltip), and a `LoRa: differs from nearby (N)` filter kept in the URL as `?lora=differs`. - Node detail: one line, for example "Differs from 12 of 14 nearby nodes on this band, which use 869.618 MHz · SF8 · BW 62.5 kHz". - Badge and filter only exist while the server has map data. ## Why same-band only On our instance (1,110 nodes with params and GPS, measured with the first version of the rule), a plain neighbourhood majority flagged exactly 15 nodes, all of them a deliberate 433 MHz network in Limburg next to the 868 MHz one (names like `NL-ZAY-RP01 | 433 X 868`). The same-band comparison flags none there. On our network this PR shows nothing today. The flags are for instances where a node really is set to a different SF, bandwidth or channel than the nodes around it. ## Cost - Points are bucketed on a radius-sized grid (rows of radius/110.57 degrees, longitude columns widened by 1/cos(lat)) and compared with a squared equirectangular distance. - `BenchmarkRfDeviations1k` (1,100 nodes over Belgium and the southern Netherlands): 4.5 ms. `BenchmarkRfDeviations10k` (10,000 nodes in the same box): 388 ms, 1.3 MB. Both on the first version. - After `5cbbe5d5`, on a slower machine, 3 runs each: 1k 4.4 to 9.5 ms, 10k 821 to 852 ms, 1.8 MB. On that machine the first version measured 7.6 to 9.3 ms and 819 to 861 ms, so the majority step adds no measurable time. It is one pass over the distinct settings around a node, and those are few (one per preset in use). - Inside the radius the work is close to O(n²): 10,000 nodes in one dense box means about 2,000 neighbours per node, and a request after the 5 minute TTL waits for that computation. - The result is cached per map snapshot for 5 minutes; concurrent requests wait for one computation. - On staging `/api/perf` reports `/api/rf-params` at 9.9 ms average, 26 ms max (8 requests). ## Tests - Go: deviation cases (SF, freq within band, BW, CR ignored, other band ignored, no GPS, no params, far away), split vote and too few peers, radius, a peer in another longitude column at 60°N, the cache (same snapshot within TTL, new snapshot, after TTL), the route with four inserted nodes, plus the two benchmarks. - Go, added in `5cbbe5d5`: a 5/5 split and 2 of 4 flag nobody; a 1 and a 15 kHz offset are not deviations while 32 kHz is, and offset peers add up into one majority; the BW/4 boundary; peers two grid columns away at 60°N; a peer at the north edge of the radius. Each fails on the first version. The two-column test also fails with a ±1 column search, which the existing longitude test did not catch. - Full `cmd/server` suite and `test-all.sh` pass locally on `5cbbe5d5`. - Frontend unit: 6 new assertions for `deviationBadge` and the detail line (13 in the file). - E2E: 4 new checks: badge only on the deviating node, filter count and result, `?lora=differs` written and restored, detail line. The disabled check now also asserts no filter and no badges. 8/8 locally. ## Validation On staging with live map data, with the first version: 1,115 nodes served, `deviations: {}`, and the filter shows `LoRa: differs from nearby (0)` with no badges. Because no real deviation exists there, the badge, filter and detail line were checked by serving one simulated deviation through Playwright route interception against the staging page. ## Not done - No map marker for deviating nodes. - `compareRadiusKm` is config-only; the customizer is a later step. The other values are hardcoded and belong in that step too: at least 3 peers, the 30 MHz band, the more-than-half rule, BW/4 and the 5 minute TTL. - Not re-run on staging after `5cbbe5d5`, and the E2E suite was not re-run locally. The fix changes only the server's choice of nodes, not the UI. - The detail line says the peers "use" the majority setting. With the BW/4 tolerance, some of them may be a few kHz off it. - Companions and room servers are compared too, not only repeaters. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
dc98886d92 |
feat(nodes): name and show repeaters known only from the name they reported (#2238) (#2239)
Part of #2238. Path hops through such a repeater (`/api/nodes`, `/api/resolve-hops`, the frontend hop resolver) still show a bare prefix; that part of the issue stays open. ## Summary A repeater that never sends an advert can still tell a nearby companion its name. CoreDrive RX v1.20.0 asks for it (`ANON_REQ_TYPE_OWNER`) and publishes the answer as `NODE_NAME` on `meshcore/client/{pk}/names`. This PR stores those answers, resolves names with them, and shows these repeaters on the map, in the nodes list and in search, marked as self-reported. ### Ingestor: store the names - `367ecde1` adds a new `names` subtopic. - It is gated on `clientNodeNames.enabled`, default off. - Answers go into a new table, `client_node_names`: one row per observation, `UNIQUE(target, rx_pubkey, observed_at)`. - **Validation.** - `target` must be 64 hex. - The name is trimmed and rejected when it is empty, over 31 runes (firmware `node_name[32]`, `src/helpers/CommonCLI.h:29`), invalid UTF-8, or contains a control character. - **Retention.** `retention.clientNodeNamesDays` (0 = keep). `config.example.json` sets 30, like the other client tables: a companion only asks a repeater for its name when CoreScope has none (`nameMissing` in CoreDrive's `src/names.js`), so pruning is also how a renamed repeater gets its new name. - **Compatibility.** An ingestor without this change drops `/names` silently, so CoreDrive v1.20.0 works against either version. ### Server: resolve by the reported name - `11d97b36` makes `lookupPrefixes` count candidates in both `nodes` and `client_node_names`. `/api/nodes/resolve` and `batchResolveHeardKeys` (the coverage tooltips) both use it. - **Which name wins.** The advert name, otherwise the newest reported name by `observed_at`. A prefix that matches different pubkeys across the two tables stays ambiguous. - **Only repeaters heard directly.** A reported name counts only once its target has at least one `discover` or `advert` reception in `client_receptions`, keyed by the full pubkey, or by its 8-byte prefix when no other pubkey in `nodes`, `inactive_nodes` or `client_node_names` starts with that prefix (`clientNameHeard`). A name for a pubkey nobody heard adds no candidate, so it cannot make a real node's prefix ambiguous. Without the uniqueness check, a made-up pubkey that shares a heard repeater's first 8 bytes borrowed that repeater's receptions and passed (fixed in `4c569f91`). - **Filters.** The blacklist and the hidden name prefixes apply to both sources. - **Missing table.** A database without the table resolves from `nodes` alone, and the server checks for the table again until the ingestor has created it. - **Response shape.** `/api/nodes/resolve` keeps `{prefix, pubkey?, name?, ambiguous}`. ### Server: estimated position - `98c76957` adds `GET /api/nodes/reported`, documented in `openapi.go`. It lists every repeater in `client_node_names` that never sent an advert (no row in `nodes` or `inactive_nodes`) and was heard directly at least once (see above). A node with an advert but no position stays out: it is already in `/api/nodes` under its advert name. - **Gated** like the other endpoints on `client_receptions`: 404 when `clientRxCoverage.enabled` is off, because the counts and the estimate come from companion GPS fixes. - **Which receptions count.** Only direct ones: - `discover` and `advert` rows, keyed by the full pubkey, or by its 8-byte prefix when no other known pubkey shares it; - `rxlog` rows whose 2-3 byte hash resolves to this repeater alone. `geo` rows are left out. - **How the estimate is made.** - Receptions within 100 m form one spot, with the best SNR of that spot. - An estimate needs at least 3 spots, at least 200 m apart. - The position is the SNR-weighted centroid (`10^(snr/20)`). The radius is the distance to the farthest spot, at least 500 m. - The response is cached for 5 minutes, and the compute time is logged. ### Map - `5fcde554` adds `public/map-estimated.js`, which draws each estimate on its own layer: - a hollow, dashed "≈" marker and a dashed uncertainty circle; - a popup with an "Estimated position" badge, the method, the caveat that one-sided driving skews the estimate, and a link to the coverage map; - a Display toggle, "Estimated positions": on by default, kept in localStorage, and also hidden by the repeater role filter and in route view. ### Nodes page and search - `12e6b345` adds a "Self-reported repeaters" section under the nodes table, on the All and Repeaters tabs, filtered by the search box. - **`#/nodes/<pubkey>`.** Where `/api/nodes/<pubkey>` answers 404, a self-reported detail view replaces "Node not found": - who reported the name; - the receptions; - the estimate, or why there is none; - links to the map and the coverage map. An unknown key still shows "Node not found". - **Header search** adds matching self-reported repeaters, tagged "self-reported". - **Unchanged on purpose.** `/api/nodes` and `/api/nodes/search` stay as they are. These repeaters have no packets, so the packet filters, the live view and the channel sender lookup must not match them. ### Docs - `03580437` corrects the `/regions` section of `docs/client-rx-coverage.md`: it named `clientRxCoverage.enabled` as the switch, but the code uses `clientRegions.enabled`. - The new subtopic, endpoint and config keys are documented in the same file. ## Trust Any companion the broker lets publish can report a name for any repeater, and the newest report wins. This is the same model as `/regions`. `clientRxCoverage.requireLinkedCompanion` limits it to linked companions. A report for a repeater nobody heard directly is stored but not used: it is not listed and adds no prefix candidate. A companion can still report a wrong name for a repeater that was heard. The same companion can also publish a fake `discover` reception for a made-up pubkey, so the "heard directly" check filters junk, not a malicious companion; `requireLinkedCompanion` is the control for that (its gate runs before the `/names` handler too). ## Performance - **`lookupPrefixes`** adds one term per key on `client_node_names`. - It matches the prefix as a range, `target >= ? AND target < ?` (upper bound prefix + `"g"`), not with `LIKE`: `LIKE` on this case-sensitive column cannot use the index. `EXPLAIN QUERY PLAN`: `SEARCH c USING INDEX idx_cnn_target (target>? AND target<?)`, and the reception check is `SEARCH r USING INDEX idx_client_recept_heard_geo (heard_key=?)`. - It is still one query per chunk of 200 keys: 400 compound terms, under SQLite's default limit of 500. - `TestLookupPrefixesManyKeys` covers 250 keys over two chunks. - **`/api/nodes/reported`** makes three kinds of query: - one grouped query on `client_node_names`; - one `lookupPrefixes` call for the short hashes; - one `IN` query per 400 keys on `client_receptions`, served by `idx_client_recept_heard_geo`. The spread check is O(spots²) per repeater, with at most one spot per 100 m visited. The result is cached for 5 minutes; concurrent misses wait for one computation, and the response is written after the lock is released. - **The map does not wait for it.** It draws its markers first and adds the estimates when the list arrives. The map and the nodes page share one cached fetch (`ReportedNodes.load()`). - **Not benchmarked.** On the instance where this runs, `client_node_names` holds one row today. I expect at most tens of such repeaters per instance, but have not measured that. ## Notes - **`map[string]interface{}`.** `cmd/ingestor/client_names.go` has 2 occurrences: the handler's `msg` parameter and its `gps` field. That is the decoded MQTT message, passed in the same shape as `handleClientPacket` and `handleClientRegions` already take it. No API response or internal structure uses it. - **Thresholds are hardcoded:** 100 m, 3 spots / 200 m, 500 m, 5 min TTL. They should move to the customizer later (rule 8). - **Map filters.** The estimates follow the "Estimated positions" toggle and the repeater role filter, not the area, region or last-heard filters. The toggle's tooltip says so. - **Name characters.** Names are rejected for control characters but not for Unicode format characters (bidi overrides, zero-width joiners), because emoji sequences in node names use the zero-width joiner. All output goes through `escapeHtml`. - **Not done:** the home-page search box (`home.js`, `/api/nodes/search` with a Claim button) does not list these repeaters. Claiming a node without adverts would have nothing to show. ## Tests On this branch (based on `550829af`): - `go vet ./...` and `go test -count=1 ./...` pass in `cmd/server` and `cmd/ingestor`. - **New Go tests:** - handler and validation cases; - idempotent redelivery; - newest `observed_at` wins; - prune; - the gate on and off, without falling through to the observer path; - client-only node, nameless `nodes` row, advert name wins, ambiguity across the two sources; - hidden and blacklisted nodes; - absent-then-created table; - spot merging, threshold, weighted centroid and radius floor; - `discover` / `rxlog` / `geo` handling, an ambiguous path hash, an advert position and a missing table; - an advert without a position, a never-heard name, a name heard only in `rxlog` paths, and the `clientRxCoverage` gate (404). These fail on `12e6b345` and pass at head. - a made-up pubkey sharing a heard repeater's 8-byte prefix, and two reported targets sharing one (`4c569f91`). Both fail without that commit. - **New frontend unit suites:** `tests/unit/test-map-estimated.js` and `tests/unit/test-reported-nodes.js` pass. Both load the real `escapeHtml` from `app.js`. - **`test-all.sh`** passes, all suites (with Python available for the XSS gate self-tests). `scripts/check-xss-sinks.sh --file` passes on the four changed frontend files. - **E2E**, against a static server with a mocked API: - `test-map-estimated-e2e.js`: 6/6, including the repeater role filter and route view hiding the estimates. Mutating either hide makes its step fail. - `test-reported-nodes-e2e.js`: 6/6. - `test-issue-1150-404-state-e2e.js`: 5/5. - **Real data, API only.** On the fork's staging, `/api/nodes/reported` lists one repeater: 5 spots, 17 receptions, an estimate with a radius of 2,305 m. It has no `nodes` row and 24 `discover` receptions, so it stays listed under the stricter rules. Not re-checked after `4c569f91`. - **Not done:** a check of that repeater in a browser. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
efd2aeabb4 |
perf(neighbors): rebuild the neighbor graph only in the background (#2221) (#2222)
Fixes #2221. ## Problem `getNeighborGraph` rebuilt `Server.neighborGraph` on the request goroutine once it passed its 5 min TTL. It held `neighborMu` for the whole build, which takes 11.1-11.4 s on a production store. The 5 min recomputer went through the same staleness check. A tick that landed just before the TTL skipped the rebuild and left it to the next request. As a result `/api/nodes/{pubkey}/neighbors` ran at p95 10.4 s and max 11.0 s (57 requests). ## Change - **`getNeighborGraph`** serves the current graph. It builds only when there is none yet, which happens at startup. - **`rebuildNeighborGraph`** builds from the packet store whatever the age of the current graph, outside `neighborMu`, and then swaps the pointer. - `recomputeNeighborGraphCache` calls it first on every tick. - Without a store it keeps the current graph, so tests that inject a graph are unaffected. - **`neighborBuildMu`** serializes builds, so a startup request and the first tick do not both build. Freshness is now bounded by the recompute interval (`neighborGraph.cacheRecomputeIntervalSeconds`, default 5 min) plus one build. Before this change it was the 5 min TTL plus one build, whatever the interval. So the bound is the same only at the default 300: - with a longer interval (for example 3600), `/api/nodes/{pk}/neighbors` can serve a graph up to that old, where before it was about 5 min; - with a shorter interval (for example 30), every tick does a full build, where before there was at most one build per 5 min. The config comment and `config.example.json` now say this (commit `de7ae981`). `IsStale`/`neighborGraphTTL` stay, because `path_inspect.go` still uses them for `store.graph`. ## Perf - **Request path:** after the first build it does no build at all. It takes one mutex and reads one pointer. - **Background:** the recompute tick now always builds, once every 5 min. Before, it built only on the ticks that found the graph stale, and a request built in the other cases. - **Total:** the number of builds stays about the same. Before: `/api/perf` on production shows p95 10404 ms. After: the follow-up comment gives client-side curl timings from staging (40 requests, p95 0.204 s), not `/api/perf`. Staging builds the graph in 1.03 s against 11 s on production, so the two are different instances. ## Tests `neighbor_graph_rebuild_test.go`: - a stale graph is served without a rebuild; - a graph is built when there is none yet; - a recompute tick replaces a graph that is not yet stale; - a recompute without a store keeps the injected graph. The first and third tests fail on master. `go vet` and `go test .` pass in `cmd/server`, and `-race` passes on the neighbor-graph tests. No frontend change. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
e8f6d4165a |
fix(observers): an observer's own transmissions are not receptions (#667) (#2220)
Refs #667 (M1, the first part). ## Problem Observations with `direction = 'tx'` are packets the observer's own radio sent, published by the MQTT bridge. Nothing in `cmd/server` or the ingestor's neighbor builder filtered on `direction`, so these rows counted as receptions. Two shapes, both seen in a production DB: - **Forwarded flood.** A repeater-observer appends its own hash before retransmitting (firmware `Mesh.cpp`), so the tx copy's path ends on the observer itself, for example tx `["2979","2029"]` next to rx `["2979"]` for the same transmission. - **Own advert.** The observer's own advert goes out with an empty path and no rx copy. Effects: - **The node detail "Heard By" card listed the observer as having heard itself.** On one repeater-observer that was the only row in the card (2,107 packets, no SNR). - **The neighbor graph counted a forwarded advert twice,** once from the rx copy and once from the tx copy (originator ↔ path[0]). This happened in both the in-memory build and the ingestor's `neighbor_edges` builder. ## Scale Last 2,000,000 observations on that DB, about 7.5 days: - 1,385 tx rows against 1,805,378 rx rows; - all tx rows have SNR and RSSI NULL; - every tx row with a non-empty path has an rx row from the same observer for the same transmission. ## Change - `StoreObs.isOwnTransmission()` (`store.go`). - These now skip tx rows: - `buildDirectHeardIndex` (`direct_heard.go`); - phase 1 of the neighbor graph build (`neighbor_graph.go`); - the ingestor's `buildAndPersistNeighborEdges` SQL, via `o.direction IS NOT 'tx'` (NULL rows from older bridges still count). ## Deliberately not changed - **Reach** (`node_reach.go`). A tx row with path `[A, X]` from observer X is the only row that shows X received A and forwarded it, because X is absent from the rx copy `[A]`. There it is valid evidence. - **Home regions** (`computeNodeHomeRegions`). A tx row of the observer's own zero-hop advert counts the observer's own IATA, which is where it is. - **Counters and region filters.** These include observation counts, `byObserver`, `packet_count` and the SQL region joins. There the effect is double counting of about 0.08% of rows, not a wrong fact. That is left for a follow-up if wanted. - **Existing `neighbor_edges` rows.** The builder is incremental from its watermark, so edges persisted before this change keep their counts. The Heard By card is rebuilt from the store, so it is correct after a restart. ## Tests - **Red then green:** - `TestBuildDirectHeardIndex_SkipsOwnTransmissions`: before the fix the observer was credited for itself; - `TestBuildNeighborGraph_SkipsOwnTransmissions`: X↔R1 count 2 instead of 1, plus a zero-hop edge from an empty-path tx row; - `TestNeighborEdgesBuilderSkipsOwnTransmissions`: 2 `neighbor_edges` rows from a single tx observation. - `go vet` and `go test .` pass in `cmd/server` and `cmd/ingestor`. No frontend change. I have not yet checked the "Heard By" card in a browser; that will be on a staging deploy. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
abbe19531d |
perf(paths): batch the resolved-path fallback for /nodes/{pk}/paths (#2223) (#2224)
Fixes #2223. ## Problem `loadCanonicalResolvedPaths` reads the resolved path of each candidate's longest observation in batches (#2146). A transmission whose longest observation has no stored `resolved_path` then went through `bestResolvedPath` on its own, with two queries: - `fetchResolvedPathForObs` re-read that same observation, which the batch had just read; - `fetchResolvedPathsForTx` read all of the transmission's observations. On a production database, 24% of the observations with a path have no `resolved_path`, so most `/api/nodes/{pubkey}/paths` requests paid for this once per such transmission. ## Change - **`loadFallbackResolvedPaths` batches the fallback.** It reads those transmissions' stored paths in one query per `resolvedPathBatchSize` transmissions: ```sql SELECT id, transmission_id, resolved_path FROM observations WHERE resolved_path IS NOT NULL AND transmission_id IN (...) ``` On a production database the planner runs it as `SEARCH observations USING INDEX idx_observations_tx_ts (transmission_id=?)`. It then picks the observation `bestResolvedPath` would pick: the longest `path_json` with a stored path, the first in snapshot order on a tie. The result goes into the resolved-path LRU, as before. - **`longestStoredResolvedPath`** holds that choice. `bestResolvedPath` now uses it as well, so the rule exists in one place. - **A failed batch still falls back.** If the first-stage batch fails, it goes through `bestResolvedPath` per transmission, as before. A failed fallback batch does the same for its transmissions. The answer is the same as before, and no cache is added. ## Perf Staging, 9 nodes, requests made from the host itself (`localhost:3001`), so no proxy or TLS. Each node 3 times, after one warm-up round: | | median | p90 | max | total for 27 requests | |---|---|---|---|---| | before | 0.174 s | 0.604 s | 0.866 s | 6.95 s | | after | 0.079 s | 0.433 s | 0.484 s | 3.22 s | CPU profile, 30 s each, under the same request loop: - **Requests completed:** 180 before, 261 after. - **CPU in `handleNodePaths` per request:** 154 ms before (27.77 s / 180), 107 ms after (27.90 s / 261). - **Fallback cost:** `bestResolvedPath` took 13.56 s of `loadCanonicalResolvedPaths` before. The batched `loadFallbackResolvedPaths` takes 6.57 s. Caveats: - The before run was on a process that had been up longer than the after run, so the two LRU states differ. - One node (`060679ad…`, about 11k transmissions) did not change: 0.455 s against 0.451 s. Its time is elsewhere in the handler, and this PR does not address it. ## Tests - **`TestLoadCanonicalResolvedPathsBatchesFallback`:** 1,001 transmissions whose longest observation has no stored path. - On master: 2,002 per-transmission queries. - Now: 0 per-transmission queries and 6 batched ones (3 for the longest observations, 3 for the fallback), with every path taken from the shorter observation. - **Existing tests:** `TestLoadCanonicalResolvedPathsMatchesBestResolvedPath` still passes. It covers the longest observation without a path (#810), ties, nothing stored, an empty stored path, and an observation absent from the snapshot. The LRU and query-count tests also still pass. - `go vet` and `go test .` in `cmd/server` pass. No frontend change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9534e1c25e |
fix(reach): retry a busy (429) reach report in the reach page (#2250)
Follow-up to #2127: @dborup's point 2 there, and @efiten's non-blocking note on the approval ("worth checking the node page's reach panel shows a retry hint on 429 rather than a generic error"). ## What changes - **The reach page retries a 429 itself.** It shows "Reach is busy computing other reports, retrying in N s", waits `Retry-After` seconds, and tries again. - When the header is missing or is not a number of seconds (an HTTP-date, 0, a negative number), it waits 2 s. It never waits more than 30 s. - After 5 retries it shows the old error ("Failed to load reach: API 429 …"). - Leaving the page stops the retries, both while it waits and while a request is still in flight. Picking another window while a request is in flight also drops that request's retries. - **`api()` does not retry 429.** Login, signup, activation, email change, companion linking, settings PUT and channel proposals also answer 429 from their own limits, and those must not auto-retry. `api()` now puts `status` and `retryAfter` on the errors it throws, so a caller can decide; the message text is unchanged. A unit test pins that `api()` makes exactly one request on a 429. - **Fix in `api()`:** the `_inflight` cleanup chain (`promise.finally(...)`) rejected unhandled on every failed call, so each API error also logged "Uncaught (in promise)". That would have added up to 5 more per busy reach load. The chain now catches; callers still get the rejection through the returned promise. This works with the server as it is on master (instant 429, `Retry-After: 5`), and with the bounded wait in #2249 (429 after 3 s, `Retry-After: 2`). Neither PR depends on the other. ## What I checked - **Suites.** `sh test-all.sh` passes, the XSS sink gate passes, and eslint has 0 errors and no new warnings. - **Unit tests** (`test-frontend-helpers.js`): - `reachRetryDelayMs`: Retry-After values; fallback on missing, zero, negative or HTTP-date values; the 30 s cap; giving up after 5 retries; no retry on 400, 404, 500, 503 or network errors. - `api()`: a 429 error carries `status` and `retryAfter`, makes exactly one request, and leaves no unhandled rejection. - The block sits above the file's `Promise.allSettled(pendingTests)` summary. Async tests registered below that line are never awaited, so they cannot fail the run; nothing below it is async today. - **E2E** (`test-node-reach-e2e.js`, already in CI suite 2): reach requests are answered through `page.route`, so the server does not have to be made busy. It checks: - two 429s, then the real report: the hint shows "retrying in 1 s", the retries wait (at least 1.8 s between the first and third request), and no error is shown; - always 429: 6 requests (1 + 5 retries), then the error; - leaving while it waits: no more requests; - leaving while the first request is still in flight: no hint written, no retry; - a 500 with Retry-After: one request, no retry. On master's frontend the new section fails at the first busy hint. Locally against the freshened `e2e-fixture.db`, the whole file takes about 17 s (2 runs). - **Mutations.** I wrote 15 from the code, and the tests caught all 15. They include: `api()` drops `status` or `retryAfter`, `api()` retries 429 itself, retry any error, retry limit off by one, fallback or cap changed, Retry-After 0 accepted, either "left the page" check removed, the wait not honoured, the hint showing ms, the retry count ignored, no error after giving up, and the cleanup chain unhandled again. Each mutation was checked against the file the server actually served before the tests ran. - **Real browser** (Chromium via Playwright, on a copy of a 10.5M-observation database): - Current master's server, 6 tabs opened at once on uncached 14-day reports: 4 got 429 with `Retry-After: 5` and showed the hint, and all 6 reports rendered within 11.5 s (one tab retried twice). Without this change those 4 tabs show "Failed to load reach: API 429". - With #2249's server: 4 of 6 got 429 with `Retry-After: 2`, and all rendered within 8.1 s. - The only console errors were Chrome's own "Failed to load resource" lines for the 429s; there was no "Uncaught (in promise)". ## Not checked - **Mobile layout.** The hint reuses the existing `.nq-load` style, so I did not take a screenshot at phone width. - **Other browsers.** Firefox and Safari were not tried; the code is plain `setTimeout` / `await`. - **Window switches.** While the hint is showing, the time-range buttons are not on screen; the same is true of the existing error and loading states. Picking a new window means waiting, or going back and opening reach again. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: nullrouten0 <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
6cafa3b596 |
chore: add SECURITY.md and OSS Scanner build files (#2257)
## What Two additions, no code changes: 1. `SECURITY.md`: a private reporting channel (`security@fiten.eu`), the supported-version policy (latest release only) and the scope. The repo had no security policy, and GitHub private vulnerability reporting is not enabled (that needs repo admin). 2. `.oss-scanner/Dockerfile` and `.oss-scanner/threat_model.md`: the files needed to enrol CoreScope in Anthropic's [OSS Scanner](https://red.anthropic.com/oss-scanner/), which periodically audits a project and mails findings to the security contact. Enrolment itself is a separate PR in [anthropics/oss-scanner](https://github.com/anthropics/oss-scanner) that points at these paths; it is opened only after this one merges. ## Dockerfile - `golang:1.27-trixie`, native cgo build with the system compiler. It is not the release image: no zig, no static musl, because the scanner only needs a working host build. - Fetches every Go module (`scripts/allmod.sh mod download`) and the npm packages, runs `make build` with symbols kept, then `make test` and `sh test-all.sh` as a non-fatal baseline, and finally sets `GOPROXY=off`. ## Threat model Lists where untrusted input enters (radio packets via MQTT, the MQTT topics and JSON, unauthenticated `/api` and `/ws`, the user-management flows), the properties that matter (no crash on malformed input, no XSS from radio data, auth boundaries, the read/write separation from `AGENTS.md`), how to run the tests, a severity scale, and what to leave alone (MeshCore protocol design, data that is public by design). ## Testing - Built on an amd64 Linux host with `docker build -f .oss-scanner/Dockerfile .`: all build steps passed, `make test` reported 16 of 16 modules OK, `test-all.sh` ended with "All standalone frontend suites passed". - In the finished image with `--network none`: `go vet ./...` and `go test` in `cmd/server` pass, `GOPROXY=off` is set. - `tools/validate.py` from oss-scanner passes with the corresponding `project.yaml`. Not done: `tools/check corescope` (needs a fresh build plus a Claude Code install in the image; no Docker host with enough disk was free). `.dockerignore` excludes `*.db`, so the two tests that read `test-fixtures/e2e-fixture.db` (`cmd/decrypt`, `cmd/migrate`) skip inside the image; the threat model says so. ## Question The 7-day acknowledgement in `SECURITY.md` is a commitment from the current maintainer. If another maintainer wants to be on the reporting address too, say so and it can be added to the scanner's `auto_ccs`. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
93f9fc37e0 |
perf(ingestor): yield between committed retention batches (#2153)
Fixes #2151. Related to #2146. Yield to runnable goroutines after each committed full retention batch, once the writer lock is released. This gives the serial ingest consumer another opportunity to submit its next write. Batch size, cutoff and deletion semantics stay unchanged; dense fanout within one transaction remains unbounded. The branch is rebased onto upstream `b9593f71745b856651cfd6d6c80bb1f3f091e13e`. The duplicate test commit `99c2722d` is dropped because #2160 is already present upstream as `a5b60ca5`. The runtime and regression-test patch is unchanged from the previously reviewed `b70a9f6b`. The raw JSON profile is omitted from this branch. The [Markdown report](https://github.com/n30nex/CoreScope/blob/307432370f8a7d9d623223beac1ba15fdd2d6546/docs/performance/sqlite-retention-b5-2026-10-09.md) links to the [immutable original profile](https://github.com/n30nex/CoreScope/blob/323b18d4221e83c9827b4722145e2a9d5b9d9599/docs/performance/sqlite-retention-b5-2026-10-09.json); the linked profile was fetched and verified against its original Git blob. Performance scope: - The historical five-pair Linux comparison remains pinned to `9dbc287` versus `b70a9f6`, with 2,048,000 observations, 50 ingest events/s and 20 HTTP requests/s. - Additional-observation scheduled-to-complete p95/p99 improved by median paired ratios of 0.9503/0.9191, while service-time p99 rose 29.1% and cleanup took 0.5% longer. HTTP results were mixed, and multi-second ingestion queues remained. - Those figures do not measure this rebased candidate against current master. The current benchmark run is an execution smoke check, not a new performance comparison; the independent reviewer's results are not pooled into the historical study. - Added runtime work is one scheduler yield per full 250-transmission batch, with constant extra state and no additional SQL scans or per-row work. No config or customizer values change. Local validation on Windows with Go 1.27.2, CGO enabled, and package parallelism limited to one: - Focused retention regressions passed. - Full ingestor suite passed. - Focused retention regressions under the race detector passed. - Both retained benchmarks passed one iteration of every fixture (`typical16`, `dense256`, `singleHot`). - The full ingestor race suite passed in 264.31 seconds with `GOMAXPROCS=4`. An initial run during competing test jobs reached its 600-second limit; after those jobs finished, the retry passed with the same 600-second limit and all assertions intact. Post-push Chromium validation passed 148 checks with three explicit fixture-dependent skips. The server and frontend source are identical to upstream `b9593f71`; this browser pass checks the existing application journeys and does not measure retention performance. [Fresh CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/39433150610) passed on `30743237`, including the full Go job, ingestion race detector, all three browser shards and the two-architecture build. |
||
|
n30nex
|
b9593f7174 |
fix(accounts): keep pending registration updates atomic (#2253)
Pending-account re-registration previously changed the password and display name in separate unconditional writes. A failed name update could leave only the password changed, and a stale pending lookup could authorize changes after activation. Update both fields in one statement constrained to `status = 'pending'`. If the account changed before the write, keep the generic registration response and issue no new activation message. Failed updates preserve the original credentials and activation link. Validation: reproduced the partial-write bug through the HTTP handler before the fix; 22 authentication tests and all 109 account-store tests pass on native SQLite. The four pending-registration store tests passed ten race-detector repetitions, covering competing registrations, a held activation writer, and active/disabled/deleted accounts. The complete server suite and full account-store race suite also pass. Local Chromium passed all 28 account journeys. No schema, configuration, UI, or dependency changes. Closes #2228. |
||
|
efitenandClaude Opus 5.5
|
d43f3c3777 |
feat(theme): ?theme=dark|light URL parameter for one page load (#2236) (#2251)
Fixes #2236. A display that can only be given a URL (digital signage, an iframe without cookies or login) had no way to choose dark or light. This adds `theme=dark` or `theme=light` to the hash query, next to `embed=1`: ``` https://<instance>/#/map?embed=1&theme=dark ``` ## What changes **`public/index.html`, the pre-paint script** - Reads `theme` from the hash query with `URLSearchParams`, as `shouldEmbedRoute` does for `embed`. Only `dark` and `light` count; anything else is ignored. - A valid value wins over the saved choice and the OS preference, and is set before first paint, so there is no light flash. - The value is handed to `app.js` as `window.__themeOverride`. - When `localStorage` throws (a sandboxed iframe), the URL value is still applied. **`public/app.js`, the dark-mode setup** - On load, a URL value is applied through `applyTheme(theme, false)`, which does not write `meshcore-theme`. The saved choice, and the account settings that `settings-sync.js` syncs from that key, stay as they were. - Theme changes from other tabs or from settings sync (the `storage` listener) are ignored while the URL value holds. Otherwise a logged-in display would flip within a minute. - A click on the dark-mode toggle ends the URL value and is saved as before. It works on every page, not only on the two `embed=1` pages. The value is read once per page load, so later hash changes (navigation, map viewport) keep it. ## Tests | File | Assertions | Covers | |---|---|---| | `tests/unit/test-theme-url-param-2236.js` | 8 | Runs the real pre-paint script from `index.html`: `dark` and `light` win over a saved choice and the OS, nothing is written to `localStorage`, any page and position in the query, an unknown value, no param, `localStorage` throwing | | `tests/e2e/test-theme-url-param-2236-e2e.js` | 6 steps | `#/map?embed=1&theme=dark` over a saved "light" (dark, embed, saved still "light"), a reload without the param, `theme=light` over a saved "dark", a storage event while the override holds, the toggle ending it, `theme=blue` | How I checked them: - With `index.html` and `app.js` from master, 6 of the 8 unit assertions fail, and the E2E fails at step 1 ("expected dark + embed + saved light"). - With only the `app.js` change reverted, the E2E fails at step 1. With only the `if (themeOverride) return;` line in the `storage` listener removed, it fails at step 4. - `sh test-all.sh` passes, except `test-issue-1956-release-routing.js`, which I skipped: it needs the workflow's `GITHUB_REF_NAME` and does not run in local git-bash. `npx eslint public/app.js` reports 0 errors (18 warnings, the same count as master). - E2E on a fixture seeded as in CI: | Suite | Result | |---|---| | `test-e2e-playwright.js` | 148 passed, 3 skipped | | `test-customize-theme-e2e.js` | 8/8 | | the new #2236 suite | 6/6 | - `test-logo-theme-e2e.js` fails locally with "wordmark `<text>` overflows SVG viewBox", with the same numbers on master. It measures font metrics, which differ on this Windows machine. The new E2E is registered in `scripts/non-unit-tests.json` and `deploy.yml` (shard 2, next to the logo theme suites); the unit test is in `test-all.sh`. ## Not done - No screenshot: the E2E checks `data-theme` in headless Chromium, not how the page looks. - `live.js` Matrix mode forces dark and, when switched off, saves the theme it restores. With a URL value active, turning Matrix off saves that value as the stored choice. I left this as is, because a display that only shows a URL does not toggle Matrix mode. - Only `theme=`; the `?darkMode=1` form from the issue text is not supported, so there is one form to document. - I placed the E2E in shard 2 without measuring suite times. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
30d58a5e15 |
fix(store): resolve region and area sets before taking s.mu (#2219) (#2247)
Fixes #2219. `resolveRegionObservers` and `resolveAreaNodes` query SQL when their cache is cold or a key is new. Nine callers looked them up while holding `s.mu.RLock()`, which `store_lock_sql_invariant_test.go` allowed through two explicit exemptions. One uncached region or area filter then blocked the poller's writer (`s.mu.Lock()`) and every reader queued behind it, `/api/healthz` included. ## What changes All in `cmd/server/store.go`. The lookups now run before the lock is taken: - **`QueryPackets`, `QueryGroupedPackets`:** follow the #2147 `nodePK` pattern. `resolvePacketFilterSets(q)` resolves node, region and area into `packetFilterSets` before the lock, and `queryPacketsLocked` and `filterPackets` take that struct instead of `nodePK`. - A region that resolves to no observers still returns nothing, and the area is then not looked up, as before. - **`GetChannels`, `GetEncryptedChannels`, `GetChannelMessages`, `computeAnalyticsChannels`, `computeAnalyticsRF`, `computeAnalyticsSubpathsWindowed`:** the region lookup moved above `s.mu.RLock()`. Their area lookups were already outside it. - **`computeAnalyticsSubpaths`:** resolves the region first and passes the set to `computeSubpathsSlow`, which no longer looks it up itself. Results, cache TTLs and cache code are unchanged. The other callers (`routes.go`, `clock_skew.go`, `neighbor_api.go`, `region_nodes.go`, `retransmission_pressure.go`, and the topology, distance, hash-size, collision and bulk-health paths in `store.go`) already looked up outside the lock, and are not changed. ## Tests **`store_lock_sql_invariant_test.go`:** - The two exemptions are removed. - With the exemptions removed and the old `store.go`, the test lists the 9 call sites. With this change it lists none. - `getCachedNodesAndPM` keeps its exemption; it is outside this issue. **`store_cold_lookup_lock_test.go`, new:** - `TestColdRegionAreaLookupDoesNotStallHealthz` has 11 subtests: raw and grouped packets, each with a region and with an area, plus the 7 region callers. Each one: - blocks the cold lookup on the existing `cacheLoadHook`; - queues an `s.mu.Lock()` writer; - requires `handleHealthz` to answer 200 within 1 s, and the writer to finish, while the lookup is still blocked. - `TestRegionSetResolvedBeforeLockSurvivesEviction` covers the eviction case from the issue: - the cache entry is replaced while the caller waits for the lock; - the result must still match the set the caller resolved; - a fix that only pre-warmed the cache and looked it up again under the lock gets 0 rows here. **How I checked them:** - With the old `store.go`, all 11 subtests fail ("healthz … blocked for more than 1s"), and so does the eviction test. - Moving only the `GetEncryptedChannels` lookup back under the lock fails `TestColdRegionAreaLookupDoesNotStallHealthz/GetEncryptedChannels`. - `go test ./...` in `cmd/server` passes; `go vet` and `gofmt -l` are clean. ## Not done - No `-race` run, no E2E, no browser check, and no latency measurement on a live instance, so this PR makes no latency claim. - `computeAnalyticsSubpaths` and `computeAnalyticsSubpathsWindowed` still call `getCachedNodesAndPM` under the lock. The invariant test still exempts that. - This PR and #2246 both change the first lines of `QueryGroupedPackets`. Whichever merges second needs a small rebase there. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
fe58dbd7d9 |
fix(server): read StoreTx state under s.mu in grouped packets and other readers (#2237) (#2246)
Fixes #2237. Ingest appends to `tx.Observations` and rewrites a transmission's header fields (`LatestSeen`, observer, path, counts) under `s.mu.Lock`. Several readers touched that state without the lock. This PR follows the approach of dborup/CoreScope#475, the reporter's fix: copy what each reader needs under the RLock it already takes, and sort or encode outside it. That fix does not apply to upstream as is (the fork has diverged), so this is a rewrite against current master. ## What changes, per site | Site | Before | After | |---|---|---| | `QueryGroupedPackets` (`store.go`) | Sort read `tx.LatestSeen` after `RUnlock`; the page (`groupedTxsToPage`, `storeTxDistinctIatas`) was built from the shared cache with no lock | The sort key is copied under the existing RLock (`groupedEntry`). The cache holds those entries. `groupedPage` copies only the requested page's rows under a short RLock; the maps are built after it | | `txGetParsedPath` (`store.go`) | Filled `tx.parsedPath` lazily under RLock, so concurrent readers wrote the same fields | `pickBestObservation` fills it through `setTxPath` where the path is written, under the write lock or before the tx is shared | | `BuildFromStoreWithOptions` (`neighbor_graph.go`) | Ranged `tx.Observations` after releasing the lock | Each tx's `Observations` slice header is copied with the packet list | | `/api/audio-lab/buckets` (`routes.go`) | Read path, observer and count after `RUnlock` | The sort stays outside the lock (`RawHex` never changes); the picked rows are read under a second short RLock | | `trackedBytes` (`/api/stats`, perf stats) | Read with no lock | Read under `s.mu` | | Observer analytics (`routes.go`, `observer_analytics.go`) | Read `tx.Hash` after `RUnlock`; the content-hash migration rewrites it | `txByID` holds copies made under the existing RLock (`txAnalyticsFields`) | `/api/nodes/{pk}/paths` and `GetNodeHopAnalytics`, items 2 in the issue, are not changed: on current master they already copy under the lock (`snapshotRPObs`) or run inside it. Appending to `tx.Observations` never changes the elements an already copied slice header covers, so copying the header is enough there. ## Tests `store_read_race_2237_test.go` adds one `-race` regression test per site. Each runs an ingest-style writer under `s.mu.Lock` against the reader: - `TestQueryGroupedPacketsIngestRace2237` (cold and cached paths) - `TestTxGetParsedPathReadersRace2237` - `TestBuildFromStoreIngestRace2237` - `TestAudioLabBucketsIngestRace2237` - `TestPerfStoreStatsTrackedBytesRace2237` - `TestStatsTrackedBytesRace2237` - `TestObserverAnalyticsHashMigrationRace2237` Plus `TestGroupedPageReflectsHeaderAfterIngest2237` (no race detector): header fields, `distinct_iatas`, count and order after ingest. How I checked them: - With the fix, `go test -race -count=3 -run 'Race2237'` passes. - Each site reverted on its own, one at a time, makes its test report `DATA RACE` (between 1 and 24 reports per site). - `go test ./...` in `cmd/server` passes; `go vet` and `gofmt -l` are clean. ## Performance `QueryGroupedPackets` is on the packets page's hot path. `BenchmarkQueryGroupedPackets2237`, 30K transmissions, median of 8 alternating runs per binary (Windows, noisy: single runs varied up to 2x): | Case | Before (ns/op) | After (ns/op) | B/op before → after | |---|---|---|---| | cold, limit 50 | 1,015,512 | 1,022,879 (+1%) | 1.62 MB → 2.10 MB | | cold, limit 1000 | 1,711,237 | 1,793,032 (+5%) | 2.93 MB → 3.64 MB | | cached, limit 50 | 29,439 | 32,608 (+11%, about 3 µs) | 69 KB → 82 KB | | cached, limit 1000 | 606,970 | 662,204 (+9%) | 1.38 MB → 1.61 MB | The extra cost is copying the page's rows. The lock is held for that copy only (at most `limit` rows), never for the sort or for building the maps. ## Not done - `BuildFromStoreWithOptions` still calls `tx.ParsedDecoded()` outside the lock, and `tx_rewrite_feed.go` resets `decodedOnce` / `parsedDecoded` under the write lock. That pair is still a race. Fixing it means parsing under the RLock on the first build, so I left it for a follow-up rather than widen this PR. - I ran the new tests under `-race`, not the whole package. - No load test against a live server; backend only, no browser check. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
3960fadd96 |
fix(pages): drop late responses that belong to an earlier page visit (#2218, #2230, #2200) (#2248)
Fixes #2218, fixes #2230, fixes #2200. Node analytics, the account page and analytics await their data, then write into the shared `#app` container. A response that lands after the user has navigated away replaces the new page. In #2218's reproduction, the Nodes list, with 200 rows, became an Analytics heading. ## The shared part `public/app.js` keeps one view generation, `viewGeneration()`: - `navigate()` bumps it before the old page's `destroy()`, so every page init counts as a new visit, including a switch to another node of the same page type. - A page captures the value at init, and after each `await` drops the result when the value has moved on. `.eslintrc.json` declares it as a global. ## Per page **#2218, `node-analytics.js`** - `loadAnalytics` and `loadBatteryChart` check the generation on both success and error. A page-local `loadSeq` makes a time-range click a new load on the same visit, so an older range can't land over a newer one. - `NodeHopAnalytics.load` already checked `container.isConnected`, so it is unchanged. **#2230, `account.js`** - The render after `CSAuth.ready()` runs only while the generation is unchanged. - `destroy()` was empty. It now clears `shown.app` and `shown.userId`. **#2200, `analytics.js`** - A page-local `_loadSeq` sits next to the generation. Results and errors of an older window/region selection or an earlier visit are dropped, so a late failure of an old load no longer shows an error over newer results. - The bulk load renders only the six tabs drawn from its data. A tab that loads its own data (Prefix Tool, Nodes, Roles, ...) is no longer rebuilt or replaced by the bulk result: a deep-linked one renders at init, and a window, region or area change re-renders it directly. - `renderTab` guards the six tabs drawn from the bulk data (overview, rf, topology, channels, hashsizes, collisions). Until the data arrives they show "Loading analytics…", or the current load's error. Overview no longer dereferences an undefined `rf`. - Three related fixes in the same lines: - The error message is now escaped with `esc()`; it was inserted raw. - `destroy()` removes the RegionFilter and AreaFilter listeners. They piled up on every visit and called `loadAnalytics` from other pages. - `destroy()` resets the stored error. ## Tests | File | Assertions | Covers | |---|---|---| | `tests/unit/test-issue-2218-node-analytics-stale-render.js` | 12 | Loads the real router: the generation rises on every init, including another node. Late success and error after leaving, a stale visit of the same node, an older range landing last, late battery success and error. The current response, battery and error still render | | `tests/unit/test-issue-2230-account-stale-render.js` | 6 | Leaving before ready (login view and "not enabled"), re-entering before ready, the current render, an auth change after `destroy()` | | `tests/unit/test-issue-2200-analytics-stale-load.js` | 19 | An older selection landing last, a late rejection of an older selection, leave and re-enter (success and error), the current error rendered escaped and kept on a tab click, Overview and RF clicked before the data | | `tests/e2e/test-issue-2218-node-analytics-stale-e2e.js` | 4 steps | Holds `/api/nodes/*/analytics` with `page.route`, returns to `#/nodes`, then releases the response as a 200 and as a 500. The 200 rows stay and no Analytics heading appears; a normal visit still renders | How I checked them: - With the production files reverted and the tests kept, 7 of 12, 4 of 6 and 7 of 12 unit assertions fail. With the fix, all pass. - The E2E fails on the unfixed frontend with "Nodes rows 200 before release, 0 after". - Single mutations, each reverted: - removing the success check in `loadAnalytics` (analytics) fails 2 assertions; - removing the check after `CSAuth.ready()` fails 3. Other test changes: - `test-analytics-theme-refresh-before-data.js` and `test-user-management-ui.js` get a `viewGeneration: () => 0` stub. - The new E2E is registered in `scripts/non-unit-tests.json` and `deploy.yml` (shard 2, next to the #748 suite). The unit tests are in `test-all.sh`. - The broad suite's "Node side panel Details link navigates" test selected an anchor that now opens Analytics, as #2218 notes. It now clicks `#nodesRight .node-detail-btn`. Results: - `sh test-all.sh` passes, and `npx eslint public/*.js` reports 0 errors. - E2E on a fixture seeded as in CI: | Suite | Result | |---|---| | `test-e2e-playwright.js` | 148 passed, 3 skipped | | `test-user-management-e2e.js` | 28/28 | | the new #2218 suite | 4/4 | | `test-issue-748-node-analytics-coverage-e2e.js` | 2/2 | | `test-analytics-fluid-charts.js` | 8/8 | | `test-issue-1657-analytics-channels-group-sprites-e2e.js` | 5/5 | ## Not done - Two tabs that each fetch their own data can still overwrite each other after a fast tab switch between them. #2200 does not ask for that. - `nodes.js` `navigateToNode` calls its own `init` without the router, so it does not bump the generation. None of the three pages fixed here depends on that. - I placed the E2E in shard 2 without measuring suite times, which the `deploy.yml` comment asks for. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
CoderNemesisandClaude Opus 5.5
|
e55c95db24 |
1-Byte Hop Path Inference for Packet and Map Analysis (#2245)
## Summary On meshes with 1-byte path hashes, most hop prefixes are shared by several repeaters. The packet detail pane currently resolves each hop on its own and stores the result by prefix. Two things go wrong as a result: - A prefix that appears twice in one path resolves to the same node both times. - Nothing checks that the chosen hops form a path a packet could actually take. This PR adds `HopResolver.inferPath()`, which infers the most likely node behind every hop of **one observation's path**, scoring the path as a whole and anchoring it on the observer that heard it. The packet detail pane and the map route view now use it, so both show the same nodes for the selected observation (`?obs=`). ## The problem, on a real packet Packet `57da37d1636f895a`, observation `1388126`: a 46-hop flood with 1-byte hashes. The server's `resolved_path` resolved 1 hop of 46, so the client resolver did the rest. Eight prefixes appear twice in the path. | | Current resolver | `inferPath()` | |---|---|---| | Hops longer than 120 km | 13 (21 longer than 60 km, two over 950 km) | 0 | | Distance from the last hop to the observer | 59 km | 0.1 km | | Hops reported as unlisted (no listed node fits) | 0, every hop guessed | 13 | | Agreement on the shared path prefix across the packet's 23 observations | — | 97.8% | The current resolver picks badly for three reasons: 1. **Results are keyed by prefix.** Hop 0 and hop 45 share a prefix, so both show the node near the start of the path, 59 km from the observer. 2. **Picks are greedy.** Each hop looks only at its nearest resolved neighbours, so nothing penalises a 1,100 km jump. 3. **The detail pane read the wrong `resolved_path`.** It used the packet row's, not the selected observation's, and when the lengths matched it never fell back for the hops that were `null`. ## Firmware basis The model relies on two facts from `firmware/src/Mesh.cpp`: - **The last hop is the one the observer heard.** `Mesh::routeRecvPacket` appends the relaying node's own hash to the end of a flood path. - **A node appears at most once in a flood path.** A node forwards a given packet once: `_tables->wasSeen()` / `markSeen()`. ## How the model works Each hop is either one of the **located** repeaters or room servers sharing its prefix, or **unlisted**: a node missing from the node list, or one without GPS. With 1-byte prefixes, unlisted is common, and naming it beats drawing a 300 km hop. | Term | Cost | |---|---| | A hop between two located nodes | distance / 50 km; past 100 km per hop, the cost rises 4× faster | | A hop with a neighbour-graph link | the distance cost × 0.5 | | An unlisted hop | 1.6, flat (about one 80 km hop) | | The last hop to the observer | the same distance cost; falls back to the observer's IATA region centre (with 50 km slack) when the observer reports no GPS or 0,0 | | The first hop to the sender | the same distance cost, when the sender's position is known (e.g. an ADVERT) | - **Search:** dynamic programming over (hop, candidate) finds the cheapest sequence. When the best path uses a node twice, the node is banned from whichever position costs less to give up. - **Confidence:** each hop's confidence comes from the min-marginal costs of its alternatives, put through a softmax. A node the path already uses at another hop is not counted as an alternative. - **Server hint:** the observation's own `resolved_path` restricts a hop to the named node, but that hop can still be reported unlisted if the node doesn't fit. The tuning values live in `PATH_MODEL` in `hop-resolver.js`, with a TODO to expose them in the customizer (AGENTS rule 8). ## What changes in the UI **Packet detail pane** (`packets.js`): - The Path row, the cross-observer aggregate row and the byte breakdown render hops **by position**, from the inferred entries. - The detail pane no longer writes into the list's shared per-prefix cache (`hopNameCache`), so it can't change what the packets list shows. **Hop display** (`hop-display.js`): | Hop state | Display | |---|---| | Confident (≥ 70%) | The node's name; the tooltip says `NN% likely` and how many nodes share the prefix | | A guess (< 70%) | Dashed underline, plus a candidate badge when there are alternatives | | Unlisted | The hex prefix in muted text; the badge lists the rejected candidates | | The only known node, without GPS | The name, with `(no GPS)` | The candidate popover lists each candidate with its likelihood, or `hop N` when the path uses that node elsewhere. The "Path symbols" legend gains an entry for unlisted hops. **Map route view** (`map.js`, `route-view.js`): - The `#/map?packet=…&obs=…` deep link infers the chosen observation's hops with the same shared helper, so the map and the detail pane agree. - Unlisted hops render as `unlisted` placeholders rather than as the nearest node that happens to share the prefix. - When inferred hops are available, the `/api/resolve-hops` request is skipped. **Shared helpers** (`packet-helpers.js`): - `inferObservationPath(obs, observerPos, originPos)` - `ensureHopAffinity()`: loads the neighbour graph once per page, cached by `api()` for 5 minutes. - `routeHopFromInferred(hop, entry, role)` **Cleanup:** removed dead hop resolution from the "View route on map" button handler. It computed hops, sender and recipient, then navigated by URL and discarded them. **Unchanged:** the packets list, live view, node pages and `HopResolver.resolve()`. ## Performance AGENTS rule 0 asks for proof of performance claims; the figures follow. - **Complexity:** O(hops × 8 × candidates²). Hops more than 8 positions apart aren't compared directly; a longer run of unlisted hops costs a flat amount, tracked with a running minimum. Hop-to-hop costs are computed once and reused by every pass. - **Where it runs:** once per detail render (plus once for the aggregate row) and once per map deep link. Nothing runs in the list, live feed or WebSocket paths. - **Measured** (Node 22, warm JIT): | Case | Time | |---|---| | 64 hops × 16 candidates, synthetic worst case | 9–14 ms median | | Real 39–48 hop paths, 1,153 nodes (23 observations) | 1–12 ms each | - **Enforced:** `test-hop-path-inference.js` asserts a median under 40 ms (target 20 ms) for the 64 × 16 case. It times the module outside Node's `vm` sandbox, because the sandbox makes global lookups about 10× slower than a browser. - **Network:** one extra bulk request per page life (`/api/analytics/neighbor-graph`, cached server-side, TTL 5 min). No per-hop or per-packet requests. ## Tests **New:** - `tests/unit/test-hop-path-inference.js` (37 assertions). The model itself: - a repeated prefix resolves per position, and no node appears twice; - the observer anchors the last hop, with the IATA fallback when the observer has no position or reports 0,0; - a far-away lone candidate is reported unlisted; - a neighbour-graph link breaks a tie; - the server hint and the origin anchor are applied; - a node without GPS is handled; - output shape, and the performance gate. - `tests/unit/test-hop-path-detail.js` (32 assertions). Rendering and wiring: - inferred-hop rendering and the popover; - positional `renderPath`, including with the 1-byte hop filter on; - `renderDetail` infers the shown observation from its own `resolved_path` and leaves the list cache alone; - the neighbour graph is fetched once; - the shared map helpers, and source checks on the map wiring. - `tests/e2e/test-hop-path-inference-e2e.js` (6 steps, shard 1). Runs on fixture packet `57224a4a01c9c682`, observation 147, a 22-hop path that repeats one prefix three times: - every hop is inferred; - the repeats resolve per position; - the popover shows likelihoods; - the map's route list matches the detail pane hop for hop. **Updated, because they pinned the old behaviour:** - `test-issue-1418-deeplink-hops-channels.js`: it tested a copy of the old map priority logic. It now tests the real `inferObservationPath()`, including `resolved_path` as a hint. - `test-issue-2098-last-hop-region.js`: the detail pane no longer writes the prefix cache. The test now reads the region-anchored last hop from the rendered Path row. - `test-issue-2097-hop-ambiguity-badge.js`: the source check allows the new `entries` option while still asserting that the detail pane never uses the summarised form. - `.eslintrc.json`: declares the three new shared helpers as globals. **Results on this branch:** - `sh test-all.sh`: all standalone frontend suites pass. - `npx eslint public/*.js`: 0 errors. All 85 warnings are pre-existing; none come from code this PR adds. - The "Run Playwright E2E tests (fail-fast)" step was run locally with its script taken verbatim from `deploy.yml` (`bash -eo pipefail`, fixture prepared as in CI): - shard 1: 27 of 27 passed; - shard 2: 47 of 47 passed; - shard 3: 50 of 50 passed. Note for reviewers: `test-issue-1692-packets-init-parallel-e2e.js` only passes while the freshened fixture's newest packets are less than 15 minutes old. The test sets the time window in `localStorage` after the page has already read it. CI meets that condition because shards start right after freshening. Unmodified `master` fails the same way when run late. **Browser validation:** the local server ran on the fixture DB. Checked in the browser: - the Path row, the candidate popover and the byte breakdown; - the map route view, which matched the detail pane hop for hop with unlisted placeholders; - the console showed no errors. ## Known limitations and follow-ups - **TRACE packets are not modelled yet.** A TRACE's path is its *planned* route. The header path byte counts the hops actually forwarded (repeaters append SNR, not their hash, `Mesh.cpp`), and a trace may revisit a node on its way back (see the comment in `Packet::calculatePacketHash`). A follow-up will add a `heardAfter` / `allowRepeats` option, so a trace is anchored after the hops it actually completed and may repeat nodes. - **Customizer (AGENTS rule 8):** the `PATH_MODEL` tuning values are hardcoded for now. The unlisted cost, typical and maximum hop length, and the confidence threshold should become customizer settings. - **Server-side `resolved_path`:** the ingestor writes a hop as resolved whenever its prefix is unique in the node table, even when that node is hundreds of km from the neighbouring hops. `inferPath()` treats these as hints and can overrule them, but the stored data is still misleading. - **The first hop without a sender position is often a close call.** On encrypted channel messages, hop 0 is constrained only by what follows it, so 55/45-style splits between a long chain of known nodes and "unlisted" are expected. The badge and the percentage show this rather than hide it. ## Screenshots This is what a normal corescope instance would show for a 1 byte path. <img width="318" height="338" alt="image" src="https://github.com/user-attachments/assets/4056bc27-0b37-4107-9b25-31ac0f7864ec" /> <img width="430" height="601" alt="image" src="https://github.com/user-attachments/assets/01da5ee8-babf-4e41-a9ab-836b5d640549" /> Now the same packet but with the logic to infer each hope along the path different nodes are chosen as they're ranked and selected based on proximity to the next hop, <img width="436" height="646" alt="image" src="https://github.com/user-attachments/assets/bcca73b8-daf0-4031-8ea1-8c96d716c8b3" /> <img width="436" height="646" alt="image" src="https://github.com/user-attachments/assets/fc926d9e-158c-48ce-bee6-93086afb0775" /> The higher the confidence level then there's no need to show the exclamation mark to show that multiple nodes share the same prefix but it does continue to share that Examples of it currently running can be found at, https://corescope.baymc.org - modified instance https://analyzer.comchan.net - unmodified instance https://analyzer-stage.comchan.net - modified instance --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7c00a42947 |
perf(reach): answer /nodes/{pk}/reach from a rolling hop index, not a path_json LIKE scan (#2214)
Refs #2146 (reach item: "an indexed hop table would remove it"). Follows #2155. ## What changes Reach finds the observations that relayed through a node with `path_json LIKE '%"XX"%'` over every observation in the window. The cost follows the window, not the node: a quiet repeater costs about as much as a busy one. When the database does not fit in the page cache this outlives the 60 s `WriteTimeout`, the request 500s, and nothing is cached. #2146 reports this endpoint at 14.9 s average and 74.7 s max on a 10.9 GB instance. Its 1.9 s figure after #2155 is a warm-cache timing; the scan reads the whole window, so cold it is far slower (below), and it holds one of the 4 read-pool connections the whole time. This PR adds an inverted index and uses it only when it is complete for the requested window: - `observation_hops(hop, timestamp, observation_id)`, `WITHOUT ROWID`. A token lookup is a seek and the window bound is a range on the same b-tree. - Triggers on `observations` maintain it (insert, delete, `path_json` update), so prune and path-repair need no changes. - `observation_hops_coverage.since` marks how far back the index is complete. If the window starts before `since`, or the table is missing (older DB, backfill unfinished), reach runs the existing `LIKE` scan unchanged. - The ingestor owns the schema, a resumable newest-first backfill, and a daily trim to `retention.hopIndexDays` (default 14). The server stays read-only and `AssertReady` does not require the table. The backfill starts after the ingest gate opens rather than inside `OpenStore`. My first version did the latter and held live ingest in the buffer for about 15 minutes on a 13.7 GB database (nothing dropped). Chunks are 15 minutes because one-hour chunks held the writer for 2 to 7 s per commit on EBS. ## Results ### Real data: vanilla vs this branch One snapshot of a real instance (11.5 GB, 10.7M observations over 14 days) served by both builds on the same machine (8 cores, 30 GB RAM, NVMe, warm cache). One cold request per cell, seconds, repeaters from busiest to quietest by 24 h relay count. | node (relays / 24 h) | 7 d vanilla | 7 d branch | 14 d vanilla | 14 d branch | |---|---|---|---|---| | 7,314 | 24.5 | 4.5 | 8.3 | 9.6 | | 7,307 | 1.7 | 2.4 | 1.3 | 3.1 | | 7,047 | 3.3 | 2.2 | 2.0 | 2.3 | | 2,151 | 20.0 | 0.44 | 29.6 | 2.0 | | 673 | 23.7 | 0.18 | 28.5 | 0.62 | | 240 | 21.8 | 0.01 | 27.8 | 0.04 | | 52 | 21.6 | 0.00 | 28.4 | 0.00 | - Nodes with few relays drop from 20 to 30 s to under a second. - The busiest relays are about the same, sometimes slower: both builds spend their time loading tens of thousands of matching rows by primary key, which the index does not speed up. - Single runs, and page-cache state moves them a lot: an earlier run, with the file freshly cached, had vanilla's quiet nodes at 4.5 to 5 s for 7 d, close to #2155's warm figure. Read orders of magnitude, not exact figures. Nothing failed on this fast machine. **Same answers.** Unit tests check indexed rows equal `LIKE` rows, fallback on partial coverage, and trigger upkeep. On real data, 21 node/window pairs (3 to 14 days) requested from both builds at the same moment were all identical once links were compared by public key. `LIKE` ignores case and the index does not; 0 of 10.1M observations in the window have a lower-case hop. Above `reachScanRowLimit` both paths truncate, and they can keep different subsets. ### Production instance t4g.large (2 vCPU, 8 GB, EBS), 13.7 GB database, about 4.5M observations per 7 days, so the cache held a fraction of it. - Before: `days=7` on a busy node returned 500 (`context canceled`), and `days=1` took 58.5 s. - With the index (38 probes of four nodes over 18 hours): `days=7` on that node returned 200 in 23 s. The busiest probe node had a median of 0.7 s and p95 18 s. The hottest (127k relay observations in 7 d) had a median of 1.3 s, p95 52 s, and one probe past 60 s at evening peak. The quietest had a median of 0.04 s. - On 2026-10-04 the host was resized to 4 vCPU and 16 GB so the cache holds the database. Same index afterwards (493 probes): hottest node median 1.4 s, p95 1.7 s, max 38 s; quietest median 0.05 s. (From 2026-10-03 the instance also ran a local change, not in this PR, that lets reach requests continue in the background, which only affects requests over 45 s.) So on a small cold host the index turned "always fails" into "usually works" but did not make the hottest relays reliably fast. That needs cheaper row loading (a compact side table measured 4 to 10x on cold reads; kept out of this PR) or background computation. ### Benchmarks (included) ``` go test ./cmd/server -run xxx -bench ScanReachRows -benchtime 5x # 600k observations, 11 hops avg, 7 d window in 14 d, warm cache hot_token/like 816 ms rare_token/like 486 ms hot_token/indexed 293 ms rare_token/indexed 1.7 ms go test ./internal/dbschema -run xxx -bench ObservationInsert -benchtime 20000x # one observation per transaction, WriterDSN (WAL, synchronous=FULL), 11 hops without_index 23 us/op with_index 127 us/op ``` The triggers add about 100 us per observation, roughly 0.1% of a core at the production instance's 8 observations per second. They cap one writer at about 8k observations per second instead of 43k. The benchmark ran in a temp directory, so it shows CPU and b-tree cost, not disk sync cost. Over 13 days on the production instance, 0 observations were dropped and newest-observation age was p99 4 s. Four of 623 samples were 20 s or more: 91 s and 20 s came within six minutes of a container restart, 31 s came 24 hours after one (when the ingestor's daily tickers fire), and 23 s had no restart nearby. Index coverage stayed between 14.0 and 15.2 days. **Disk.** On the snapshot (`dbstat`) the index is 1.98 GB for 10.1M observations in the window, about 196 bytes each, next to 6.81 GB for `observations`. It scales with hops per observation and `hopIndexDays`. Backfill of 77M rows took 15.5 min on the production instance. **Perf justification (AGENTS.md).** Reach goes from a scan over the whole window to a seek over the matching rows. Everything new is bounded: the index by `hopIndexDays` and the daily trim, the backfill by chunk size and a cancellable context, and the server adds no in-memory state. The ingest hot path pays the trigger cost above. ## Design notes and open questions - Why not the in-memory store: it defaults to 7 days (`retentionHours: 168`) and reach offers up to 30, so it could not answer the longer windows. The other option listed in #2146, driving the query from `observations.timestamp`, is what #2155 already did. - SQLite only: the triggers use `json_each`. The optional PostgreSQL backend in #2145 would need its own equivalent. - No off switch: the table and triggers are always installed and `hopIndexDays` only sets the window. If you would rather have this opt-in, or want a smaller default for small disks, I'll change it. ## Operator notes - Upgrade: the ingestor installs the table and triggers on start and backfills. Until a window is covered, reach behaves as today. - Rollback: an older binary keeps indexing new rows with no trim. To remove it with the ingestor stopped: `DROP TRIGGER trg_observation_hops_ai; DROP TRIGGER trg_observation_hops_ad; DROP TRIGGER trg_observation_hops_au; DROP TABLE observation_hops; DROP TABLE observation_hops_coverage;` - New key `retention.hopIndexDays`, documented in `config.example.json`. ## Tests `go test ./...` passes in `cmd/server`, `cmd/ingestor` and `internal/dbschema`. New: `node_reach_hops_test.go`, `observation_hops_test.go`, `observation_hops_backfill_test.go`, and the two benchmarks. ## Update after review (commit `2291bea2`) The text above is what was reviewed at `95e17976`. Where they differ, this section is current: - **Opt-in.** `retention.hopIndexDays` unset or 0 (the default, and the value in `config.example.json`) installs nothing. A value above 0 installs the table and triggers and backfills that many days. This replaces "default 14", the "No off switch" note and the Upgrade note above. - **Changing the setting** applies at the next ingestor start: raising it backfills the gap, lowering it trims, and 0 removes the triggers and both tables. On a 10.5M-observation copy the removal took 45 to 71 s, at startup before MQTT connects. - **Case.** Hops are stored and matched upper-cased, so the index ignores ASCII case the way LIKE does. This replaces the sentence on case under "Same answers". - **Rollback.** Set it to 0 and start the ingestor once, then roll back. The manual `DROP` list above still works with the ingestor stopped. - **Disk.** A full 14-day backfill on that copy built 2.17 GB for 9.68M observations, about 224 bytes each, in 13 min 37 s. - Backfill and trim are serialized, and the chunk SQL lives in `dbschema` next to the triggers. ## Update after review, round 2 (commit `95690e8c`) Where this differs from the two sections above, this one is current: - **Startup order.** The index is installed or removed in the startup block, after the MQTT subscription is up and before the ingest gate opens, so packets arriving during a removal are buffered (`ingestBufferSize`), not lost. This replaces "at startup before MQTT connects" in the first update. - **Mark moves cover their range.** Each backfill and trim transaction indexes or deletes everything between the old mark and the new one, so a row arriving between the chunk-picking read and the transaction is accounted for either way. - **Negative `hopIndexDays`** is a config error at startup; the ingestor refuses to run. This replaces "−5 is off" in the first update. - **Server.** If the index disappears between the coverage check and the scan, the request is answered from the `LIKE` scan instead of a 500. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
2d09335cfb |
fix(packets): keep the node filter pick on a slow click or late search (#2243) (#2244)
Fixes #2243. ## Problem The packets node filter selected a suggestion on `click`, while the input's `blur` handler hides the dropdown 200 ms after focus leaves. A press held longer than that, or a search response that rebuilt the list between mousedown and mouseup, lost the pick: the typed text stayed and no `node=` filter was applied. ## Change - `public/packets.js`: suggestions select on `mousedown` with `preventDefault`, so the input keeps focus and the pick happens before any rebuild. `live.js` already does this for its node filter. - `public/packets.js`: a sequence counter drops search responses for an older query, bumped on every search and on a pick, so a late response can't replace or reopen the list. - `tests/e2e/test-packets-node-filter-e2e.js` (new), registered in `scripts/non-unit-tests.json` and on shard 1 in `deploy.yml`. ## Tests The new E2E has three cases: a normal click, a 400 ms press, and a search response released mid-click. | Code | Normal click | 400 ms press | Response mid-click | |---|---|---|---| | master | pass | fail | fail | | this PR without the sequence guard | pass | pass | fail (list reopens) | | this PR | pass | pass | pass | Run locally against the freshened e2e fixture. `sh test-all.sh` passes (locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and `PYTHONUTF8=1`, unrelated to this change). ## Not done - Full `test-e2e-playwright.js` not run locally. - Could not reproduce on the reporter's instance with the browser tool, whose clicks are shorter than 200 ms. The 400 ms Playwright press reproduces it on the fixture. - `live.js` and `packets.js` still carry two separate node-autocomplete implementations. Merging them is out of scope here. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
a910a3e480 |
fix(packets): re-fetch when the observer filter changes (#2241) (#2242)
Fixes #2241. ## Problem In grouped mode, ticking an observer on `#/packets` updated the URL and the trigger label but left the table unchanged until a reload. Since #1821 `applyObserverFilter` leaves grouped rows to the server's EXISTS filter, while the observer menu's change handler still only called `renderTableRows()`, which re-renders the rows already loaded without a new `/api/packets` request. ## Change - `public/packets.js`: the observer menu change handler calls `loadPackets()` instead of `renderTableRows()`, so the request carries the new `observer=` set. - `tests/unit/test-observer-menu-interactions.js`: two tests asserting the handler re-fetches, for ticking one observer and for "All Observers". Both failed before the change. ## Verification - `sh test-all.sh`: all suites pass (locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and `PYTHONUTF8=1`, unrelated to this change). - Browser, local server on the freshened e2e fixture: 55 rows; one observer ticked gives 2 rows, all from that observer; back to "All Observers" gives 53 rows. No reload in between. - Reproduced the original bug on an operator's v3.14.0 instance before the change. ## Not done - Playwright E2E suite not run locally. - Perf: one `/api/packets` request per checkbox toggle, the same request a reload or the node filter already makes. No per-item calls. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
8b5ca5735c |
fix(analytics): Packets / Hour readable on long histories (#2193) (#2194)
Fixes #2193. ## Problem On an instance that keeps months of packets, the Overview tab's Packets / Hour chart becomes unreadable: 2,932 hourly bars in 720 units of plot, a date label every day, and the hours with no packets (which the server leaves out) squeezed off the axis. ## Before and after Both use the real `/api/analytics/rf` answer from the instance in the issue (2,933 hourly buckets over 124 days), served to the page in place of the fixture's own. **Before** (master, page on "All data"):  **After**, same data:  Phone: <img src="https://raw.githubusercontent.com/A13xB0/CoreScope/7862abd8/2026-10/pph-2193/after-phone.png" width="300"> **A short store** (an instance holding 7 days in memory, plus a few old packets a day for weeks before that because they were heard again). The chart starts where the real data does (this shot predates the "since MM-DD" note now in the title when that crop applies):  ## Change All in `public/analytics.js`; no API change. - **`packetsPerHourCard()`** replaces the inline block: - rolls the hours up to hourly, 6-hourly, daily or weekly bars, at most 180, and titles the card to match (Packets / Day); - counts missing hours as zero; - puts about 10 labels on round UTC times; - **under "All data" only**, starts on the first UTC day with at least 2% of a typical recent day (the median of the seven days before the last one), at its first hour with a tenth of a typical hour, and says "since MM-DD" in the title when that cuts anything. Scanning forward, an outage after the start is never taken for it. In a picked window (1h to 30d) the server has already cut the data, so nothing is cropped and a quieter early stretch stays visible. - **Theme refresh before the data** (separate commit; this also happens on master without the new chart): every load fires `theme-refresh` about 300 ms after the theme resolves, and the page re-rendered the current tab on it. When the analytics answers came later (a busy or just-restarted server), the Overview was drawn from an empty object and threw `Cannot read properties of undefined (reading 'totalTransmissions')`. It now skips that refresh until the data has landed; `loadAnalytics()` renders the tab with the current theme when it does. The span is still picked by the page's time window. ## Tests - `tests/unit/test-issue-2193-packets-per-hour.js` (in `test-all.sh`, red then green): 122 days gives 123 daily bars with 4 to 12 labels; 7 days gives 168 hourly bars; missing hours are drawn as zero; the chart starts at the data, not at the stragglers, and not after an outage; in a picked window nothing is cropped (90 days with 30 quiet ones gives 91 bars); a cropped chart says "since MM-DD" and an uncropped one does not; no data says so. - `tests/unit/test-analytics-theme-refresh-before-data.js` (in `test-all.sh`): drives `init()` with answers that never land, fires the `theme-refresh` handler, and checks that nothing throws and the tab is not re-rendered. Without the guard it fails with the production error. - `sh test-all.sh` passes; the #1925 neighbour-graph theme-refresh test and the analytics chart test still pass; the screenshots show no page errors and no sideways scroll at 1280 px or 390 px. ## Later - The 2% floor, the tenth-of-an-hour start and the 180-bar cap are hardcoded. Per AGENTS.md rule 8 they can move to the customizer in a later step. - Edge bars are partial periods (data starting at 17h gives a 7-hour first day; the last bar is still filling), and weekly bars align on epoch weeks (Thursday). Both are left as they are here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
9529bf9d41 |
perf(store): read resolved_path pubkeys in place, one copy each (#2197)
Refs #1870. ## Problem `Load`, `loadChunk` and `scanAndMergeChunk` decode each observation's `resolved_path` only to collect its non-empty pubkeys. On live data that's a JSON array of pubkeys and nulls, about 420 characters, present on most observations. Decoding into `[]*string` allocated a string and a pointer for every hop, plus two slices. ## Change - `resolvedPubkeysFromJSON` reads the usual array of escape-free strings and nulls in place. Anything else falls back to decoding, so it returns exactly `extractResolvedPubkeys(unmarshalResolvedPath(s))`. - Relay pubkeys repeat across nearly every observation, so each one goes through `internStr` (from #2183) and is kept once. A pubkey seen before is found by substring, without copying, and the indexes it feeds (`byNode` and friends) share that copy. Interned keys live for the process, evicted nodes' included; the table is bounded at 64k entries. - The three bulk-load call sites use it. - **Eviction too.** `resolvedPubkeysForEvictionBatch` runs on every eviction batch, about once a minute in steady state, and spent about 130 MB per five minutes on a busy week. It now reads each pubkey in place and de-duplicates each packet's short list by scanning it instead of building a map per packet. - Other callers of `unmarshalResolvedPath` are unchanged. ## Measured | | before | after | |---|---|---| | allocations for 3 known pubkeys | 13 | 1 | | eviction, allocations per evicted packet (3 observations × 4 hops) | 51 | 15 | | this path's allocations over startup (copy of a busy week) | 236 MB | 61 MB | | server CPU over the run | 175.6 s | 170.6 s | The startup rows need that dataset and are from a single run. The benchmarks are rerunnable (`go test -run X -bench ResolvedPubkeys -benchmem ./cmd/server`, medians of 3): | | master | this PR | |---|---|---| | `BenchmarkResolvedPubkeysFromJSON` (5 relays + 3 nulls, pool of 2,000) | n/a | 619 ns, 128 B, 1 alloc | | `BenchmarkResolvedPubkeysForEvictionBatch` (1,000 packets × 3 observations) | 14.3 ms, 3.96 MB, 65,495 allocs | 6.7 ms, 1.36 MB, 18,501 allocs | ## Tests - `TestResolvedPubkeysFromJSONAllocations`: red commit first, with the bound tightened in green. - `TestResolvedPubkeysForEvictionBatch`: keys, order, and an allocation bound. It has its own red commit. - `TestResolvedPubkeysFromJSONMatchesDecoding` compares against decode-then-extract on 26 edge cases and 20,000 random inputs. - `go test ./cmd/server` passes. No config or UI changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
cf342efc43 |
perf(store): key byNode's de-dup sets by transmission ID (#2199)
## Problem `nodeHashes` is the set behind `byNode`'s de-dup: for every node, the transmissions already listed under it. It was keyed by each transmission's 16-character hash string. On a week of a busy network that's about 650k entries and 21.5 MB, four times the size of `byNode` itself. ## Change - The sets are keyed by transmission ID. That's the same membership, since a store holds one transmission per hash. - Eviction deletes by ID. - The health endpoint's region filter matches regional observations by transmission ID. - `newNodeSets` and `nodeHasTx` wrap the map for callers and tests. ## Measured | | before | after | |---|---|---| | node sets' live heap (copy of a busy week) | 21.5 MB | 13.2 MB | | fixture, 500k (node, packet) pairs | 25.8 MB | 18.0 MB | ## Tests - `TestNodeSetRetainedHeap`: red commit first. - `TestAddToByNodeDedup`. - The existing eviction, routes and resolved-index tests updated to the ID-keyed sets. - `go test ./cmd/server` passes. No config or UI changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
hermes-gadget
|
fdcec91578 |
fix(route-view): "N hops" counts only true hops, not payload endpoints (#2240)
Fixes #2206 ## What Follow-up to #2207, as sketched in its review: the route meta (desktop header + mobile summary) printed `positions.length` as "N hops". For a DM, `positions` includes the payload-decoded sender and recipient rows — endpoints, not hops — so the count was inflated. Live on corescope.ukmesh.com right now, the reported 6-relay chain reads **"8 hops · 913 km"** with "from payload" chips on rows 1 and 8. ## Change - `public/route-view.js` — new `countHops(positions)` helper: counts rows **without** `_fromPayload` (both builders set the flag since #2207). The desktop header and the mobile summary both render it, so they can't disagree. Only the displayed figure changes; the row-count `total` still drives distances, stripe colours and row slicing. - `tests/unit/test-issue-2206-hop-count.js` — regression test (listed in `test-all.sh`): extracts `countHops()` from the shipping file and exercises it (6-relay DM → 6; payload-only → 0; plain chains and non-payload origin/dest unchanged), and pins both meta strings to the helper so a regression back to `total` fails. The count intentionally counts relay rows (per the review note), not edges: a 6-relay DM reads "6 hops". Frontend-only, static assets, no build step, no new dependencies. Cache buster untouched (AGENTS.md rule 3). ## Perf One `.filter()` over a ≤ ~10-entry array, once per sidebar build; no hot-path impact. ## Testing - `node tests/unit/test-issue-2206-hop-count.js` — 11 passed, 0 failed - `sh test-all.sh` — full unit suite green, exit 0 - Live cross-check against the deployed instances (pre-fix logic): `https://corescope.ukmesh.com/#/map?packet=44259e35b7c826a8&obs=97234` shows 8 rows, meta "8 hops · 913 km", payload chips on rows 1 and 8 → this change makes that same chain read "6 hops · 913 km". The new count ships with the next build that carries it; happy to re-check on a deployment once one does. |
||
|
|
127dd511a7 |
fix(health): judge favorites and home status cards by the node's role (#2215)
## The problem A starred repeater can show two different health statuses. The My Repeaters tab (#1761) and `getNodeStatus` judge it by the infra thresholds (degraded after 24 h, silent after 72 h). The favorites dropdown and the home page cards judge every node by the node thresholds (degraded after 1 h, silent after 24 h), because they never look at the node's role. | repeater last heard | favorites / home cards (before) | My Repeaters tab | after this PR | |---|---|---|---| | 5 h ago | degraded | healthy | healthy | | 30 h ago | silent | degraded | degraded | | 80 h ago | silent | silent | silent | If you set `healthThresholds` in `config.json`, every view follows your values. ## The change Three call sites now use `getHealthThresholds(role)`, the helper the My Repeaters tab already uses: - `renderFavDropdown` in `app.js` - the "My Nodes" grid card in `home.js` - the hero search-result card in `home.js` The search card's fallback text, "Not heard in 24+ hours", was only true for non-infra nodes, so it now reads "Not heard recently". ## Scope - I kept the existing three-state healthy / degraded / silent logic and changed only which thresholds apply. - Not included: counting a repeater that is relaying traffic as alive (#1598, which `getNodeStatus` does). That is a bigger behavior change and would be its own PR. ## Risk Low. It is 9 lines in two files, and the helper and thresholds already exist and are used elsewhere. ## Tests New `tests/unit/test-health-card-role-thresholds.js`, added to `test-all.sh`. It checks that infra roles get looser thresholds than nodes, and that each of the three call sites goes through `getHealthThresholds(role)` with no node-only comparison left. On current `master` 5 of its 7 assertions fail; with this change all pass. `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning from an a11y self-test, which it also does on `master`. ## Update after review (commit `0f2376df`) The text above is the description efiten reviewed on 2026-10-15. This section describes the code as it is now. ### The change - New `getHealthStatus(role, heard, now)` in `roles.js`. It returns `healthy`, `degraded` or `silent` by the role's thresholds, or `null` when there is no usable timestamp. It only applies the cut-offs; each caller picks the timestamp. - All six places that compared an age against `degradedMs` / `silentMs` now call it: - the favorites dropdown (`app.js`); - the "My Nodes" card and the home health card (`home.js`); - the analytics network status and My Repeaters (`analytics.js`); - the live node panel (`live.js`). Each keeps its own labels and timestamp, so the last three show the same as before. `analytics.js` and `live.js` no longer guard with `typeof getHealthThresholds`. `roles.js` loads before both, and both already used it without a guard elsewhere: the old network status called `getHealthThresholds` directly, and `live.js` reads `ROLE_COLORS`. - The favorites dropdown and both home cards now fall back to the node's `last_seen` when the packet store has no packet for it. My Repeaters (`last_heard || last_seen`) and the node page (`nodes.js:729`) already do this. Without it, with a packet store window under 72 h, a repeater could be silent on the cards and degraded on My Repeaters. The "ago" text uses the same timestamp. With the default thresholds, the fallback changes a status only when the packet store holds less than 72 h of packets. `retentionHours` unset (unlimited) or at the example's 168 h holds more than that. Otherwise a node with no packet in the window is silent either way, and the views now show how long ago it was last seen instead of "never" or nothing. - With no timestamp at all, the home health card said "Not heard in 24+ hours". It now says "No packets in the current window". ### Why a new helper and not `getNodeStatus` `getNodeStatus` has two states (active / stale) and counts a repeater that relays traffic as alive (#1598). Using it here would change more than the thresholds. `getHealthStatus` keeps these views' three states. ### Scope - Not included: counting a repeater that relays traffic as alive (#1598) in these views. That is a bigger behavior change and would be its own PR. - The live node panel still judges by `last_seen` alone, and the node page stays relay-aware. Both now use the shared thresholds; aligning their timestamps would also be its own change. ### Tests - `tests/unit/test-health-status.js` runs `roles.js` in `vm`. It covers: - repeater, room, companion, sensor, an unknown role and a missing role, at ages either side of 1 h, 24 h and 72 h; - ISO and epoch input; - null, empty and invalid timestamps; - a `healthThresholds` override through `/api/config/client`. One source guard remains: no age-vs-threshold comparison outside `roles.js`. - `tests/unit/test-my-repeaters-dashboard.js` now uses the real helper instead of a stub, which gave repeaters the node thresholds. It also checks that a repeater heard 30 h ago is Degraded. - New `tests/e2e/test-health-status-e2e.js` mocks `/api/nodes/<pk>/health` for six nodes. It checks five views: the favorites dropdown, the "My Nodes" card, the home health card, the analytics network status and the live node panel. It fails on `master` and on this PR's first commit. I put it on shard 2, which was the shortest shard in the last CI run. - Mutation check: I made 33 deliberate breaks. In the helper they cover each comparison and boundary. At each of the six call sites they cover the timestamp field, the role argument and at least one label mapping. Each one makes the suite fail. - I checked a `config.json` with `infraDegradedHours: 2` and `infraSilentHours: 6` in Chromium. Repeaters heard 3 h and 7 h ago show Degraded and Silent. - `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning, which comes from a deliberate fixture in the a11y self-test. ### Cost The analytics network status loops over every node. In headless Chromium, 30,000 synthetic nodes take 9.0 to 9.2 ms before this change and 9.1 ms after (median of 21 runs, on one machine). ### Risk Low. The change is frontend only, with no API, config or data change. Rolling back means reverting the commits. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
hermes-gadgetandhermes-gadget
|
550829af6e |
fix(route-view): badge payload-decoded sender/recipient rows in the sequence renderer (#2207)
Refs #2206 ## What DM packets on the route view showed **sender** and **recipient** rows — decoded from the packet payload (`destHash`/`srcHash`, one byte each) rather than from the outer path bytes — as ordinary hops, without the "from payload" badge, whenever the chain renders through the multi-path/sequence builder. Only the single-path builder set `_fromPayload`, and the isolate/sequence list renderer never drew the chip. Effects: payload endpoints are indistinguishable from observed hops, and the displayed hop count is inflated (a 6-relay DM renders 8 rows, with the last "hop" being a single-byte recipient prefix resolved to whichever node shares it). ## Change - `public/map.js` — set `_fromPayload: true` on the sequence builder's origin (sender) and destination (recipient) entries (3 + 2 sites). - `public/route-view.js` — render the `mc-rt-status-chip mc-rt-status-payload` "from payload" chip in the isolate/sequence list, matching the canonical renderer. The chip markup now lives in one `payloadChip(p)` helper called by both row builders (it was duplicated between them, so they could drift apart). - `tests/unit/test-issue-2206-payload-badge.js` — regression test (listed in `test-all.sh`): extracts `payloadChip()` and `rowFor()` from the shipping `route-view.js` and exercises them in a vm sandbox, and pins every sender/recipient entry in `drawPacketRouteMulti` to `_fromPayload: true`. Frontend-only, static assets, no build step, no new dependencies. Per AGENTS.md rule 3 the cache buster in `public/index.html` is intentionally untouched — `__BUST__` is replaced automatically at server startup. ## Perf Two ternaries + one string concat per rendered row in a display-only list; no impact on hot paths. ## Testing - `node tests/unit/test-issue-2206-payload-badge.js` — 22 passed, 0 failed (new regression test) - `sh test-all.sh` — full unit suite green, exit 0 (201 suites) - Browser validation, re-run against the live deployments (which carry the equivalent patch): on https://corescope.nemesh.uk/#/map?packet=44259e35b7c826a8&obs=97234 the canonical chain marks rows 1 and 5 "from payload"; isolating the 6-hop path (`B4CE24→…→8C197F`) renders the 8-row chain with rows 1 and 8 marked. The refactor keeps the renderer output identical (pinned by the new test). The identical patched assets have been running on two independent deployments since 2026-10-13. ## Notes The underlying ambiguity (a 1-byte prefix matched against many nodes, resolved to the most-recently-seen candidate) is what the badge surfaces to the user; changing the resolution strategy itself is out of scope for this PR. The `N hops` header still counts every row — counting only non-payload rows would change every DM header, so it stays a follow-up; hence `Refs #2206`. --------- Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com> |
||
|
|
cb0cd3c8dd |
fix(docker): read .env last, and restore RUN_AS_UID/GID from positional parameters (#2235)
Follow-up to #2125: the six non-blocking notes on its approval, with note 3 as the one that changes behaviour. **What changes** - `.env` is now read last. The hand-over of the data directory moves before it, and the only code that runs after the file is sourced reads `prepare_service_user`'s positional parameters: the account, the container's `RUN_AS_UID`/`RUN_AS_GID` (set or unset, and value) and the file's path are parked in `$1..$6` before the `.`. - The comment gives the real reason the account was already safe: the `.` happens inside `load_env_file`, so a `set -- root` in `.env` lands in that function's `$1`, not in `prepare_service_user`'s (note 1). It also says that a trusted `.env` is root's code and this does not defend against it (it can redefine functions or set `PATH`); what it cannot do by assigning variables is change the account, where the hand-over goes, or `RUN_AS_UID`/`RUN_AS_GID` as the services see them. - `svc_log` uses `printf` (note 5); the entrypoint check also pins that it does not assign `CORESCOPE_SERVICE_USER` (note 4); the test header is reflowed (note 6); `TEST_SHELL` picks the inner shell so the suite runs under dash, bash or busybox without editing. **Why** A sourced file shares the shell's variables: every global, the locals of `load_env_file`, and, by dynamic scoping, the locals of its caller too (checked under dash, bash 5.3 and busybox ash 1.37). So a trusted `.env` that assigned `_uid_before` or `_uid_was` skipped the restore and left `RUN_AS_UID=0` in the services' environment, as note 3 says. The same class reached the hand-over: with `SERVICE_DATA_DIR=/` in `.env`, the walk ran over `/`. Nothing but the entrypoint reads `RUN_AS_UID`/`RUN_AS_GID`, so the effect was a wrong value in the services' environment and a log line that said "ignoring" when it had not. **What I checked** - `docker/test-select-service-user.sh`: 116 assertions, 0 failed, under dash and bash. The new cases fail on master for the stated reasons and nothing else does. - New cases: `.env` running `set -- root` (note 1; this is the case that catches sourcing the file inline, the names case alone does not); `.env` assigning the restore's names, `_env_file`, `SERVICE_ENV_FILE` and `RUN_AS_*`, with uid 1000 and gid 1001 so a swapped parameter shows; `RUN_AS_GID` alone; `.env` pointing the three `SERVICE_*` paths elsewhere; a logged path containing `\012`. - Mutations, each applied to a copy and confirmed to make the suite fail: hand-over moved back after `.env`; restore from `load_env_file`'s locals (master); restore from `prepare_service_user`'s locals; file sourced inline in `prepare_service_user` with the refusal log kept; restore dropped; only the uid compared; gid restored from the uid's parameter; set/unset flipped; the log reading `SERVICE_ENV_FILE` after the `.`; the entrypoint reassigning the account. `svc_log` back to `echo` fails under dash only, which is where note 5 applies. - The image, built from this branch (linux/amd64), with a root-owned, root-only-writable `.env` containing all of the above plus `DISABLE_MOSQUITTO=true`, `DISABLE_CADDY=true` and `set -- root`: (A) `RUN_AS_UID` unset: root, nothing re-owned, `DISABLE_*` honoured, `RUN_AS_*` not in the services' environment. (B) `RUN_AS_UID=1000 RUN_AS_GID=1001` on A's data directory, root-owned `meshcore.db` inside: services run as 1000, their environment has `RUN_AS_UID=1000 RUN_AS_GID=1001`, `meshcore.db` and the directory are 1000:1001, `.env` stays root's, `/etc/passwd` stays root's. (C) `.env` owned by uid 1000: refused with the reason, services run as 1000. The same image with master's two scripts mounted over it, in B's shape: services' environment `RUN_AS_UID=0 RUN_AS_GID=0`, `/etc/passwd` owned by 1000, `.env` owned by 1000:1001. - `sh test-all.sh` passes; no JS or Go change. **Not checked** - Windows: the suite stops at the symlink fixture, as before. - busybox as `TEST_SHELL` on my machine: Ubuntu's busybox prefers its own applets over `PATH`, so the stubs are bypassed and 28 cases fail, on master and on this branch alike. Your WSL Debian run (103 passed) and the image itself, which runs Alpine's ash with real binaries, are what cover ash. - Rootless Docker and `userns-remap`: unchanged, docs gap already noted. Note 2: kept the names case as a second guard; the `set -- root` case is the one that pins the design. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Adam
|
4fdd494d9b |
fix(nodes): read the transport-direct path byte at offset 5 (#2225) (#2234)
Red commit:
|
||
|
efitenandClaude Opus 5.5
|
927a8eec8a |
fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211)
Refs #748 (M3a: correct numbers within the in-memory window; the SQL history in M3b is a separate plan, see the end). ## Problem The node analytics range picker offers 24h, 7d, 30d and All (365d), but the store only holds its in-memory window (`retentionHours`, then eviction). `GetNodeAnalytics` still divided availability and packets/day by the requested range. Measured on our live instance (14-day window), repeater BE-BRE-ON8AR: | Range | Availability | Packets/day | |---|---|---| | 7d | 88.7% | 91 | | 30d | 42.6% | 265 | | All | 3.5% | 22 | Nothing on the page said that the data stopped at the edge of the window. ## What changes - `coverageStart` returns the edge of the in-memory window: the latest of `oldestLoaded` (background-load progress), `now - retentionHours`, and a new `evictedThrough` that eviction sets to the newest `FirstSeen` it removed, so memory-pressure eviction inside the retention window moves the edge too. - `s.packets[0]` is not that edge, and the first version of this PR used it. Chunks are selected by `last_seen`, so a transmission first heard weeks ago and heard again recently sits at the head with its old `FirstSeen`. On staging (7-day window) it returned 21/07. - When the edge lies more than a minute after the requested start, node analytics counts only transmissions from the edge, computes availability and packets/day over that span, and returns `timeRange.dataFrom` and `timeRange.partial`. - The minute of slack keeps a range equal to the retention window from reading as partial. Without it every 7d request on staging (168h) was partial, because the two edges came from separate clock reads. - The analytics page shows "Data from 2026-10-07: the server keeps about 7 days of packets in memory. Availability and packets/day cover that period, not the full 365 days." when partial. - **UTC fix.** The window bounds of node analytics and node hop analytics are formatted in UTC. They are compared as strings with UTC `FirstSeen` values, so on a non-UTC host the 7d window lost its oldest hours (2 at +02:00). Docker images run in UTC, which is why this did not show in production. - `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the edge is kept. ## Tests - Go, `node_analytics_coverage_test.go`: - full coverage within memory; - 30d over 3 days in memory; - empty store; - a re-heard 85-day-old transmission at the head of `s.packets` (edge stays at retention, packet not counted, no bucket before `dataFrom`); - memory-eviction edge; - eviction records `evictedThrough`; - retention 30 s shorter than the range is not partial, while 30d is; - window bounds under a +02:00 `time.Local`. - Two tests were checked to fail on the old code: the local-zone test (166 of 168 packets, a `+02:00` bound) and the retention test without the slack (`dataFrom 09:57:08` vs `from 09:56:38`). - Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). One run of five reported `FAIL` and four reruns were green; I did not capture which test failed. - Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js` (5 assertions on the real `node-analytics.js` in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`), against the real fixture response with no interception: 7d is not partial and shows no notice; All is partial and the notice names the real start date. 2/2 locally. ## Validation Staging (retentionHours 168), same repeater, after the background load completed: | Range | Before | After | |---|---|---| | 7d | 82.7% | 83.3%, not partial | | 30d | 19.3% | 83.3%, partial from 2026-10-07 | | All | 1.6% | 83.3%, partial from 2026-10-07 | In the browser the notice appears under All and not under 7d. ## Not done - M3b, older history from SQLite. In the database only the sender is indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay hops only in `observations.resolved_path` (filled for about 27% of observations on our live instance, not indexed). A fallback today would show only a node's own packets for older days, a different meaning than the in-memory part. Doing it properly needs a per-node index written by the ingestor (pubkey, transmission, role); that is a separate plan. - The other analytics endpoints with a range (observer analytics, RF) were not checked for the same dilution. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
