mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 23:38:17 +00:00
7c00a42947ff7670dbce3849285fd65d37191a31
3063
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
7c00a42947 |
perf(reach): answer /nodes/{pk}/reach from a rolling hop index, not a path_json LIKE scan (#2214)
Refs #2146 (reach item: "an indexed hop table would remove it"). Follows #2155. ## What changes Reach finds the observations that relayed through a node with `path_json LIKE '%"XX"%'` over every observation in the window. The cost follows the window, not the node: a quiet repeater costs about as much as a busy one. When the database does not fit in the page cache this outlives the 60 s `WriteTimeout`, the request 500s, and nothing is cached. #2146 reports this endpoint at 14.9 s average and 74.7 s max on a 10.9 GB instance. Its 1.9 s figure after #2155 is a warm-cache timing; the scan reads the whole window, so cold it is far slower (below), and it holds one of the 4 read-pool connections the whole time. This PR adds an inverted index and uses it only when it is complete for the requested window: - `observation_hops(hop, timestamp, observation_id)`, `WITHOUT ROWID`. A token lookup is a seek and the window bound is a range on the same b-tree. - Triggers on `observations` maintain it (insert, delete, `path_json` update), so prune and path-repair need no changes. - `observation_hops_coverage.since` marks how far back the index is complete. If the window starts before `since`, or the table is missing (older DB, backfill unfinished), reach runs the existing `LIKE` scan unchanged. - The ingestor owns the schema, a resumable newest-first backfill, and a daily trim to `retention.hopIndexDays` (default 14). The server stays read-only and `AssertReady` does not require the table. The backfill starts after the ingest gate opens rather than inside `OpenStore`. My first version did the latter and held live ingest in the buffer for about 15 minutes on a 13.7 GB database (nothing dropped). Chunks are 15 minutes because one-hour chunks held the writer for 2 to 7 s per commit on EBS. ## Results ### Real data: vanilla vs this branch One snapshot of a real instance (11.5 GB, 10.7M observations over 14 days) served by both builds on the same machine (8 cores, 30 GB RAM, NVMe, warm cache). One cold request per cell, seconds, repeaters from busiest to quietest by 24 h relay count. | node (relays / 24 h) | 7 d vanilla | 7 d branch | 14 d vanilla | 14 d branch | |---|---|---|---|---| | 7,314 | 24.5 | 4.5 | 8.3 | 9.6 | | 7,307 | 1.7 | 2.4 | 1.3 | 3.1 | | 7,047 | 3.3 | 2.2 | 2.0 | 2.3 | | 2,151 | 20.0 | 0.44 | 29.6 | 2.0 | | 673 | 23.7 | 0.18 | 28.5 | 0.62 | | 240 | 21.8 | 0.01 | 27.8 | 0.04 | | 52 | 21.6 | 0.00 | 28.4 | 0.00 | - Nodes with few relays drop from 20 to 30 s to under a second. - The busiest relays are about the same, sometimes slower: both builds spend their time loading tens of thousands of matching rows by primary key, which the index does not speed up. - Single runs, and page-cache state moves them a lot: an earlier run, with the file freshly cached, had vanilla's quiet nodes at 4.5 to 5 s for 7 d, close to #2155's warm figure. Read orders of magnitude, not exact figures. Nothing failed on this fast machine. **Same answers.** Unit tests check indexed rows equal `LIKE` rows, fallback on partial coverage, and trigger upkeep. On real data, 21 node/window pairs (3 to 14 days) requested from both builds at the same moment were all identical once links were compared by public key. `LIKE` ignores case and the index does not; 0 of 10.1M observations in the window have a lower-case hop. Above `reachScanRowLimit` both paths truncate, and they can keep different subsets. ### Production instance t4g.large (2 vCPU, 8 GB, EBS), 13.7 GB database, about 4.5M observations per 7 days, so the cache held a fraction of it. - Before: `days=7` on a busy node returned 500 (`context canceled`), and `days=1` took 58.5 s. - With the index (38 probes of four nodes over 18 hours): `days=7` on that node returned 200 in 23 s. The busiest probe node had a median of 0.7 s and p95 18 s. The hottest (127k relay observations in 7 d) had a median of 1.3 s, p95 52 s, and one probe past 60 s at evening peak. The quietest had a median of 0.04 s. - On 2026-10-04 the host was resized to 4 vCPU and 16 GB so the cache holds the database. Same index afterwards (493 probes): hottest node median 1.4 s, p95 1.7 s, max 38 s; quietest median 0.05 s. (From 2026-10-03 the instance also ran a local change, not in this PR, that lets reach requests continue in the background, which only affects requests over 45 s.) So on a small cold host the index turned "always fails" into "usually works" but did not make the hottest relays reliably fast. That needs cheaper row loading (a compact side table measured 4 to 10x on cold reads; kept out of this PR) or background computation. ### Benchmarks (included) ``` go test ./cmd/server -run xxx -bench ScanReachRows -benchtime 5x # 600k observations, 11 hops avg, 7 d window in 14 d, warm cache hot_token/like 816 ms rare_token/like 486 ms hot_token/indexed 293 ms rare_token/indexed 1.7 ms go test ./internal/dbschema -run xxx -bench ObservationInsert -benchtime 20000x # one observation per transaction, WriterDSN (WAL, synchronous=FULL), 11 hops without_index 23 us/op with_index 127 us/op ``` The triggers add about 100 us per observation, roughly 0.1% of a core at the production instance's 8 observations per second. They cap one writer at about 8k observations per second instead of 43k. The benchmark ran in a temp directory, so it shows CPU and b-tree cost, not disk sync cost. Over 13 days on the production instance, 0 observations were dropped and newest-observation age was p99 4 s. Four of 623 samples were 20 s or more: 91 s and 20 s came within six minutes of a container restart, 31 s came 24 hours after one (when the ingestor's daily tickers fire), and 23 s had no restart nearby. Index coverage stayed between 14.0 and 15.2 days. **Disk.** On the snapshot (`dbstat`) the index is 1.98 GB for 10.1M observations in the window, about 196 bytes each, next to 6.81 GB for `observations`. It scales with hops per observation and `hopIndexDays`. Backfill of 77M rows took 15.5 min on the production instance. **Perf justification (AGENTS.md).** Reach goes from a scan over the whole window to a seek over the matching rows. Everything new is bounded: the index by `hopIndexDays` and the daily trim, the backfill by chunk size and a cancellable context, and the server adds no in-memory state. The ingest hot path pays the trigger cost above. ## Design notes and open questions - Why not the in-memory store: it defaults to 7 days (`retentionHours: 168`) and reach offers up to 30, so it could not answer the longer windows. The other option listed in #2146, driving the query from `observations.timestamp`, is what #2155 already did. - SQLite only: the triggers use `json_each`. The optional PostgreSQL backend in #2145 would need its own equivalent. - No off switch: the table and triggers are always installed and `hopIndexDays` only sets the window. If you would rather have this opt-in, or want a smaller default for small disks, I'll change it. ## Operator notes - Upgrade: the ingestor installs the table and triggers on start and backfills. Until a window is covered, reach behaves as today. - Rollback: an older binary keeps indexing new rows with no trim. To remove it with the ingestor stopped: `DROP TRIGGER trg_observation_hops_ai; DROP TRIGGER trg_observation_hops_ad; DROP TRIGGER trg_observation_hops_au; DROP TABLE observation_hops; DROP TABLE observation_hops_coverage;` - New key `retention.hopIndexDays`, documented in `config.example.json`. ## Tests `go test ./...` passes in `cmd/server`, `cmd/ingestor` and `internal/dbschema`. New: `node_reach_hops_test.go`, `observation_hops_test.go`, `observation_hops_backfill_test.go`, and the two benchmarks. ## Update after review (commit `2291bea2`) The text above is what was reviewed at `95e17976`. Where they differ, this section is current: - **Opt-in.** `retention.hopIndexDays` unset or 0 (the default, and the value in `config.example.json`) installs nothing. A value above 0 installs the table and triggers and backfills that many days. This replaces "default 14", the "No off switch" note and the Upgrade note above. - **Changing the setting** applies at the next ingestor start: raising it backfills the gap, lowering it trims, and 0 removes the triggers and both tables. On a 10.5M-observation copy the removal took 45 to 71 s, at startup before MQTT connects. - **Case.** Hops are stored and matched upper-cased, so the index ignores ASCII case the way LIKE does. This replaces the sentence on case under "Same answers". - **Rollback.** Set it to 0 and start the ingestor once, then roll back. The manual `DROP` list above still works with the ingestor stopped. - **Disk.** A full 14-day backfill on that copy built 2.17 GB for 9.68M observations, about 224 bytes each, in 13 min 37 s. - Backfill and trim are serialized, and the chunk SQL lives in `dbschema` next to the triggers. ## Update after review, round 2 (commit `95690e8c`) Where this differs from the two sections above, this one is current: - **Startup order.** The index is installed or removed in the startup block, after the MQTT subscription is up and before the ingest gate opens, so packets arriving during a removal are buffered (`ingestBufferSize`), not lost. This replaces "at startup before MQTT connects" in the first update. - **Mark moves cover their range.** Each backfill and trim transaction indexes or deletes everything between the old mark and the new one, so a row arriving between the chunk-picking read and the transaction is accounted for either way. - **Negative `hopIndexDays`** is a config error at startup; the ingestor refuses to run. This replaces "−5 is off" in the first update. - **Server.** If the index disappears between the coverage check and the scan, the request is answered from the `LIKE` scan instead of a 500. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
2d09335cfb |
fix(packets): keep the node filter pick on a slow click or late search (#2243) (#2244)
Fixes #2243. ## Problem The packets node filter selected a suggestion on `click`, while the input's `blur` handler hides the dropdown 200 ms after focus leaves. A press held longer than that, or a search response that rebuilt the list between mousedown and mouseup, lost the pick: the typed text stayed and no `node=` filter was applied. ## Change - `public/packets.js`: suggestions select on `mousedown` with `preventDefault`, so the input keeps focus and the pick happens before any rebuild. `live.js` already does this for its node filter. - `public/packets.js`: a sequence counter drops search responses for an older query, bumped on every search and on a pick, so a late response can't replace or reopen the list. - `tests/e2e/test-packets-node-filter-e2e.js` (new), registered in `scripts/non-unit-tests.json` and on shard 1 in `deploy.yml`. ## Tests The new E2E has three cases: a normal click, a 400 ms press, and a search response released mid-click. | Code | Normal click | 400 ms press | Response mid-click | |---|---|---|---| | master | pass | fail | fail | | this PR without the sequence guard | pass | pass | fail (list reopens) | | this PR | pass | pass | pass | Run locally against the freshened e2e fixture. `sh test-all.sh` passes (locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and `PYTHONUTF8=1`, unrelated to this change). ## Not done - Full `test-e2e-playwright.js` not run locally. - Could not reproduce on the reporter's instance with the browser tool, whose clicks are shorter than 200 ms. The 400 ms Playwright press reproduces it on the fixture. - `live.js` and `packets.js` still carry two separate node-autocomplete implementations. Merging them is out of scope here. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
a910a3e480 |
fix(packets): re-fetch when the observer filter changes (#2241) (#2242)
Fixes #2241. ## Problem In grouped mode, ticking an observer on `#/packets` updated the URL and the trigger label but left the table unchanged until a reload. Since #1821 `applyObserverFilter` leaves grouped rows to the server's EXISTS filter, while the observer menu's change handler still only called `renderTableRows()`, which re-renders the rows already loaded without a new `/api/packets` request. ## Change - `public/packets.js`: the observer menu change handler calls `loadPackets()` instead of `renderTableRows()`, so the request carries the new `observer=` set. - `tests/unit/test-observer-menu-interactions.js`: two tests asserting the handler re-fetches, for ticking one observer and for "All Observers". Both failed before the change. ## Verification - `sh test-all.sh`: all suites pass (locally `test-preflight-xss-gate.js` needs a real `python3` on PATH and `PYTHONUTF8=1`, unrelated to this change). - Browser, local server on the freshened e2e fixture: 55 rows; one observer ticked gives 2 rows, all from that observer; back to "All Observers" gives 53 rows. No reload in between. - Reproduced the original bug on an operator's v3.14.0 instance before the change. ## Not done - Playwright E2E suite not run locally. - Perf: one `/api/packets` request per checkbox toggle, the same request a reload or the node filter already makes. No per-item calls. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
8b5ca5735c |
fix(analytics): Packets / Hour readable on long histories (#2193) (#2194)
Fixes #2193. ## Problem On an instance that keeps months of packets, the Overview tab's Packets / Hour chart becomes unreadable: 2,932 hourly bars in 720 units of plot, a date label every day, and the hours with no packets (which the server leaves out) squeezed off the axis. ## Before and after Both use the real `/api/analytics/rf` answer from the instance in the issue (2,933 hourly buckets over 124 days), served to the page in place of the fixture's own. **Before** (master, page on "All data"):  **After**, same data:  Phone: <img src="https://raw.githubusercontent.com/A13xB0/CoreScope/7862abd8/2026-10/pph-2193/after-phone.png" width="300"> **A short store** (an instance holding 7 days in memory, plus a few old packets a day for weeks before that because they were heard again). The chart starts where the real data does (this shot predates the "since MM-DD" note now in the title when that crop applies):  ## Change All in `public/analytics.js`; no API change. - **`packetsPerHourCard()`** replaces the inline block: - rolls the hours up to hourly, 6-hourly, daily or weekly bars, at most 180, and titles the card to match (Packets / Day); - counts missing hours as zero; - puts about 10 labels on round UTC times; - **under "All data" only**, starts on the first UTC day with at least 2% of a typical recent day (the median of the seven days before the last one), at its first hour with a tenth of a typical hour, and says "since MM-DD" in the title when that cuts anything. Scanning forward, an outage after the start is never taken for it. In a picked window (1h to 30d) the server has already cut the data, so nothing is cropped and a quieter early stretch stays visible. - **Theme refresh before the data** (separate commit; this also happens on master without the new chart): every load fires `theme-refresh` about 300 ms after the theme resolves, and the page re-rendered the current tab on it. When the analytics answers came later (a busy or just-restarted server), the Overview was drawn from an empty object and threw `Cannot read properties of undefined (reading 'totalTransmissions')`. It now skips that refresh until the data has landed; `loadAnalytics()` renders the tab with the current theme when it does. The span is still picked by the page's time window. ## Tests - `tests/unit/test-issue-2193-packets-per-hour.js` (in `test-all.sh`, red then green): 122 days gives 123 daily bars with 4 to 12 labels; 7 days gives 168 hourly bars; missing hours are drawn as zero; the chart starts at the data, not at the stragglers, and not after an outage; in a picked window nothing is cropped (90 days with 30 quiet ones gives 91 bars); a cropped chart says "since MM-DD" and an uncropped one does not; no data says so. - `tests/unit/test-analytics-theme-refresh-before-data.js` (in `test-all.sh`): drives `init()` with answers that never land, fires the `theme-refresh` handler, and checks that nothing throws and the tab is not re-rendered. Without the guard it fails with the production error. - `sh test-all.sh` passes; the #1925 neighbour-graph theme-refresh test and the analytics chart test still pass; the screenshots show no page errors and no sideways scroll at 1280 px or 390 px. ## Later - The 2% floor, the tenth-of-an-hour start and the 180-bar cap are hardcoded. Per AGENTS.md rule 8 they can move to the customizer in a later step. - Edge bars are partial periods (data starting at 17h gives a 7-hour first day; the last bar is still filling), and weekly bars align on epoch weeks (Thursday). Both are left as they are here. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
9529bf9d41 |
perf(store): read resolved_path pubkeys in place, one copy each (#2197)
Refs #1870. ## Problem `Load`, `loadChunk` and `scanAndMergeChunk` decode each observation's `resolved_path` only to collect its non-empty pubkeys. On live data that's a JSON array of pubkeys and nulls, about 420 characters, present on most observations. Decoding into `[]*string` allocated a string and a pointer for every hop, plus two slices. ## Change - `resolvedPubkeysFromJSON` reads the usual array of escape-free strings and nulls in place. Anything else falls back to decoding, so it returns exactly `extractResolvedPubkeys(unmarshalResolvedPath(s))`. - Relay pubkeys repeat across nearly every observation, so each one goes through `internStr` (from #2183) and is kept once. A pubkey seen before is found by substring, without copying, and the indexes it feeds (`byNode` and friends) share that copy. Interned keys live for the process, evicted nodes' included; the table is bounded at 64k entries. - The three bulk-load call sites use it. - **Eviction too.** `resolvedPubkeysForEvictionBatch` runs on every eviction batch, about once a minute in steady state, and spent about 130 MB per five minutes on a busy week. It now reads each pubkey in place and de-duplicates each packet's short list by scanning it instead of building a map per packet. - Other callers of `unmarshalResolvedPath` are unchanged. ## Measured | | before | after | |---|---|---| | allocations for 3 known pubkeys | 13 | 1 | | eviction, allocations per evicted packet (3 observations × 4 hops) | 51 | 15 | | this path's allocations over startup (copy of a busy week) | 236 MB | 61 MB | | server CPU over the run | 175.6 s | 170.6 s | The startup rows need that dataset and are from a single run. The benchmarks are rerunnable (`go test -run X -bench ResolvedPubkeys -benchmem ./cmd/server`, medians of 3): | | master | this PR | |---|---|---| | `BenchmarkResolvedPubkeysFromJSON` (5 relays + 3 nulls, pool of 2,000) | n/a | 619 ns, 128 B, 1 alloc | | `BenchmarkResolvedPubkeysForEvictionBatch` (1,000 packets × 3 observations) | 14.3 ms, 3.96 MB, 65,495 allocs | 6.7 ms, 1.36 MB, 18,501 allocs | ## Tests - `TestResolvedPubkeysFromJSONAllocations`: red commit first, with the bound tightened in green. - `TestResolvedPubkeysForEvictionBatch`: keys, order, and an allocation bound. It has its own red commit. - `TestResolvedPubkeysFromJSONMatchesDecoding` compares against decode-then-extract on 26 edge cases and 20,000 random inputs. - `go test ./cmd/server` passes. No config or UI changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
cf342efc43 |
perf(store): key byNode's de-dup sets by transmission ID (#2199)
## Problem `nodeHashes` is the set behind `byNode`'s de-dup: for every node, the transmissions already listed under it. It was keyed by each transmission's 16-character hash string. On a week of a busy network that's about 650k entries and 21.5 MB, four times the size of `byNode` itself. ## Change - The sets are keyed by transmission ID. That's the same membership, since a store holds one transmission per hash. - Eviction deletes by ID. - The health endpoint's region filter matches regional observations by transmission ID. - `newNodeSets` and `nodeHasTx` wrap the map for callers and tests. ## Measured | | before | after | |---|---|---| | node sets' live heap (copy of a busy week) | 21.5 MB | 13.2 MB | | fixture, 500k (node, packet) pairs | 25.8 MB | 18.0 MB | ## Tests - `TestNodeSetRetainedHeap`: red commit first. - `TestAddToByNodeDedup`. - The existing eviction, routes and resolved-index tests updated to the ID-keyed sets. - `go test ./cmd/server` passes. No config or UI changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
hermes-gadget
|
fdcec91578 |
fix(route-view): "N hops" counts only true hops, not payload endpoints (#2240)
Fixes #2206 ## What Follow-up to #2207, as sketched in its review: the route meta (desktop header + mobile summary) printed `positions.length` as "N hops". For a DM, `positions` includes the payload-decoded sender and recipient rows — endpoints, not hops — so the count was inflated. Live on corescope.ukmesh.com right now, the reported 6-relay chain reads **"8 hops · 913 km"** with "from payload" chips on rows 1 and 8. ## Change - `public/route-view.js` — new `countHops(positions)` helper: counts rows **without** `_fromPayload` (both builders set the flag since #2207). The desktop header and the mobile summary both render it, so they can't disagree. Only the displayed figure changes; the row-count `total` still drives distances, stripe colours and row slicing. - `tests/unit/test-issue-2206-hop-count.js` — regression test (listed in `test-all.sh`): extracts `countHops()` from the shipping file and exercises it (6-relay DM → 6; payload-only → 0; plain chains and non-payload origin/dest unchanged), and pins both meta strings to the helper so a regression back to `total` fails. The count intentionally counts relay rows (per the review note), not edges: a 6-relay DM reads "6 hops". Frontend-only, static assets, no build step, no new dependencies. Cache buster untouched (AGENTS.md rule 3). ## Perf One `.filter()` over a ≤ ~10-entry array, once per sidebar build; no hot-path impact. ## Testing - `node tests/unit/test-issue-2206-hop-count.js` — 11 passed, 0 failed - `sh test-all.sh` — full unit suite green, exit 0 - Live cross-check against the deployed instances (pre-fix logic): `https://corescope.ukmesh.com/#/map?packet=44259e35b7c826a8&obs=97234` shows 8 rows, meta "8 hops · 913 km", payload chips on rows 1 and 8 → this change makes that same chain read "6 hops · 913 km". The new count ships with the next build that carries it; happy to re-check on a deployment once one does. |
||
|
|
127dd511a7 |
fix(health): judge favorites and home status cards by the node's role (#2215)
## The problem A starred repeater can show two different health statuses. The My Repeaters tab (#1761) and `getNodeStatus` judge it by the infra thresholds (degraded after 24 h, silent after 72 h). The favorites dropdown and the home page cards judge every node by the node thresholds (degraded after 1 h, silent after 24 h), because they never look at the node's role. | repeater last heard | favorites / home cards (before) | My Repeaters tab | after this PR | |---|---|---|---| | 5 h ago | degraded | healthy | healthy | | 30 h ago | silent | degraded | degraded | | 80 h ago | silent | silent | silent | If you set `healthThresholds` in `config.json`, every view follows your values. ## The change Three call sites now use `getHealthThresholds(role)`, the helper the My Repeaters tab already uses: - `renderFavDropdown` in `app.js` - the "My Nodes" grid card in `home.js` - the hero search-result card in `home.js` The search card's fallback text, "Not heard in 24+ hours", was only true for non-infra nodes, so it now reads "Not heard recently". ## Scope - I kept the existing three-state healthy / degraded / silent logic and changed only which thresholds apply. - Not included: counting a repeater that is relaying traffic as alive (#1598, which `getNodeStatus` does). That is a bigger behavior change and would be its own PR. ## Risk Low. It is 9 lines in two files, and the helper and thresholds already exist and are used elsewhere. ## Tests New `tests/unit/test-health-card-role-thresholds.js`, added to `test-all.sh`. It checks that infra roles get looser thresholds than nodes, and that each of the three call sites goes through `getHealthThresholds(role)` with no node-only comparison left. On current `master` 5 of its 7 assertions fail; with this change all pass. `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning from an a11y self-test, which it also does on `master`. ## Update after review (commit `0f2376df`) The text above is the description efiten reviewed on 2026-10-15. This section describes the code as it is now. ### The change - New `getHealthStatus(role, heard, now)` in `roles.js`. It returns `healthy`, `degraded` or `silent` by the role's thresholds, or `null` when there is no usable timestamp. It only applies the cut-offs; each caller picks the timestamp. - All six places that compared an age against `degradedMs` / `silentMs` now call it: - the favorites dropdown (`app.js`); - the "My Nodes" card and the home health card (`home.js`); - the analytics network status and My Repeaters (`analytics.js`); - the live node panel (`live.js`). Each keeps its own labels and timestamp, so the last three show the same as before. `analytics.js` and `live.js` no longer guard with `typeof getHealthThresholds`. `roles.js` loads before both, and both already used it without a guard elsewhere: the old network status called `getHealthThresholds` directly, and `live.js` reads `ROLE_COLORS`. - The favorites dropdown and both home cards now fall back to the node's `last_seen` when the packet store has no packet for it. My Repeaters (`last_heard || last_seen`) and the node page (`nodes.js:729`) already do this. Without it, with a packet store window under 72 h, a repeater could be silent on the cards and degraded on My Repeaters. The "ago" text uses the same timestamp. With the default thresholds, the fallback changes a status only when the packet store holds less than 72 h of packets. `retentionHours` unset (unlimited) or at the example's 168 h holds more than that. Otherwise a node with no packet in the window is silent either way, and the views now show how long ago it was last seen instead of "never" or nothing. - With no timestamp at all, the home health card said "Not heard in 24+ hours". It now says "No packets in the current window". ### Why a new helper and not `getNodeStatus` `getNodeStatus` has two states (active / stale) and counts a repeater that relays traffic as alive (#1598). Using it here would change more than the thresholds. `getHealthStatus` keeps these views' three states. ### Scope - Not included: counting a repeater that relays traffic as alive (#1598) in these views. That is a bigger behavior change and would be its own PR. - The live node panel still judges by `last_seen` alone, and the node page stays relay-aware. Both now use the shared thresholds; aligning their timestamps would also be its own change. ### Tests - `tests/unit/test-health-status.js` runs `roles.js` in `vm`. It covers: - repeater, room, companion, sensor, an unknown role and a missing role, at ages either side of 1 h, 24 h and 72 h; - ISO and epoch input; - null, empty and invalid timestamps; - a `healthThresholds` override through `/api/config/client`. One source guard remains: no age-vs-threshold comparison outside `roles.js`. - `tests/unit/test-my-repeaters-dashboard.js` now uses the real helper instead of a stub, which gave repeaters the node thresholds. It also checks that a repeater heard 30 h ago is Degraded. - New `tests/e2e/test-health-status-e2e.js` mocks `/api/nodes/<pk>/health` for six nodes. It checks five views: the favorites dropdown, the "My Nodes" card, the home health card, the analytics network status and the live node panel. It fails on `master` and on this PR's first commit. I put it on shard 2, which was the shortest shard in the last CI run. - Mutation check: I made 33 deliberate breaks. In the helper they cover each comparison and boundary. At each of the six call sites they cover the timestamp field, the role argument and at least one label mapping. Each one makes the suite fail. - I checked a `config.json` with `infraDegradedHours: 2` and `infraSilentHours: 6` in Chromium. Repeaters heard 3 h and 7 h ago show Degraded and Silent. - `sh test-all.sh` passes. It prints a `STALE ALLOWLIST` warning, which comes from a deliberate fixture in the a11y self-test. ### Cost The analytics network status loops over every node. In headless Chromium, 30,000 synthetic nodes take 9.0 to 9.2 ms before this change and 9.1 ms after (median of 21 runs, on one machine). ### Risk Low. The change is frontend only, with no API, config or data change. Rolling back means reverting the commits. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
hermes-gadgetandhermes-gadget
|
550829af6e |
fix(route-view): badge payload-decoded sender/recipient rows in the sequence renderer (#2207)
Refs #2206 ## What DM packets on the route view showed **sender** and **recipient** rows — decoded from the packet payload (`destHash`/`srcHash`, one byte each) rather than from the outer path bytes — as ordinary hops, without the "from payload" badge, whenever the chain renders through the multi-path/sequence builder. Only the single-path builder set `_fromPayload`, and the isolate/sequence list renderer never drew the chip. Effects: payload endpoints are indistinguishable from observed hops, and the displayed hop count is inflated (a 6-relay DM renders 8 rows, with the last "hop" being a single-byte recipient prefix resolved to whichever node shares it). ## Change - `public/map.js` — set `_fromPayload: true` on the sequence builder's origin (sender) and destination (recipient) entries (3 + 2 sites). - `public/route-view.js` — render the `mc-rt-status-chip mc-rt-status-payload` "from payload" chip in the isolate/sequence list, matching the canonical renderer. The chip markup now lives in one `payloadChip(p)` helper called by both row builders (it was duplicated between them, so they could drift apart). - `tests/unit/test-issue-2206-payload-badge.js` — regression test (listed in `test-all.sh`): extracts `payloadChip()` and `rowFor()` from the shipping `route-view.js` and exercises them in a vm sandbox, and pins every sender/recipient entry in `drawPacketRouteMulti` to `_fromPayload: true`. Frontend-only, static assets, no build step, no new dependencies. Per AGENTS.md rule 3 the cache buster in `public/index.html` is intentionally untouched — `__BUST__` is replaced automatically at server startup. ## Perf Two ternaries + one string concat per rendered row in a display-only list; no impact on hot paths. ## Testing - `node tests/unit/test-issue-2206-payload-badge.js` — 22 passed, 0 failed (new regression test) - `sh test-all.sh` — full unit suite green, exit 0 (201 suites) - Browser validation, re-run against the live deployments (which carry the equivalent patch): on https://corescope.nemesh.uk/#/map?packet=44259e35b7c826a8&obs=97234 the canonical chain marks rows 1 and 5 "from payload"; isolating the 6-hop path (`B4CE24→…→8C197F`) renders the 8-row chain with rows 1 and 8 marked. The refactor keeps the renderer output identical (pinned by the new test). The identical patched assets have been running on two independent deployments since 2026-10-13. ## Notes The underlying ambiguity (a 1-byte prefix matched against many nodes, resolved to the most-recently-seen candidate) is what the badge surfaces to the user; changing the resolution strategy itself is out of scope for this PR. The `N hops` header still counts every row — counting only non-payload rows would change every DM header, so it stays a follow-up; hence `Refs #2206`. --------- Co-authored-by: hermes-gadget <hermes-gadget@users.noreply.github.com> |
||
|
|
cb0cd3c8dd |
fix(docker): read .env last, and restore RUN_AS_UID/GID from positional parameters (#2235)
Follow-up to #2125: the six non-blocking notes on its approval, with note 3 as the one that changes behaviour. **What changes** - `.env` is now read last. The hand-over of the data directory moves before it, and the only code that runs after the file is sourced reads `prepare_service_user`'s positional parameters: the account, the container's `RUN_AS_UID`/`RUN_AS_GID` (set or unset, and value) and the file's path are parked in `$1..$6` before the `.`. - The comment gives the real reason the account was already safe: the `.` happens inside `load_env_file`, so a `set -- root` in `.env` lands in that function's `$1`, not in `prepare_service_user`'s (note 1). It also says that a trusted `.env` is root's code and this does not defend against it (it can redefine functions or set `PATH`); what it cannot do by assigning variables is change the account, where the hand-over goes, or `RUN_AS_UID`/`RUN_AS_GID` as the services see them. - `svc_log` uses `printf` (note 5); the entrypoint check also pins that it does not assign `CORESCOPE_SERVICE_USER` (note 4); the test header is reflowed (note 6); `TEST_SHELL` picks the inner shell so the suite runs under dash, bash or busybox without editing. **Why** A sourced file shares the shell's variables: every global, the locals of `load_env_file`, and, by dynamic scoping, the locals of its caller too (checked under dash, bash 5.3 and busybox ash 1.37). So a trusted `.env` that assigned `_uid_before` or `_uid_was` skipped the restore and left `RUN_AS_UID=0` in the services' environment, as note 3 says. The same class reached the hand-over: with `SERVICE_DATA_DIR=/` in `.env`, the walk ran over `/`. Nothing but the entrypoint reads `RUN_AS_UID`/`RUN_AS_GID`, so the effect was a wrong value in the services' environment and a log line that said "ignoring" when it had not. **What I checked** - `docker/test-select-service-user.sh`: 116 assertions, 0 failed, under dash and bash. The new cases fail on master for the stated reasons and nothing else does. - New cases: `.env` running `set -- root` (note 1; this is the case that catches sourcing the file inline, the names case alone does not); `.env` assigning the restore's names, `_env_file`, `SERVICE_ENV_FILE` and `RUN_AS_*`, with uid 1000 and gid 1001 so a swapped parameter shows; `RUN_AS_GID` alone; `.env` pointing the three `SERVICE_*` paths elsewhere; a logged path containing `\012`. - Mutations, each applied to a copy and confirmed to make the suite fail: hand-over moved back after `.env`; restore from `load_env_file`'s locals (master); restore from `prepare_service_user`'s locals; file sourced inline in `prepare_service_user` with the refusal log kept; restore dropped; only the uid compared; gid restored from the uid's parameter; set/unset flipped; the log reading `SERVICE_ENV_FILE` after the `.`; the entrypoint reassigning the account. `svc_log` back to `echo` fails under dash only, which is where note 5 applies. - The image, built from this branch (linux/amd64), with a root-owned, root-only-writable `.env` containing all of the above plus `DISABLE_MOSQUITTO=true`, `DISABLE_CADDY=true` and `set -- root`: (A) `RUN_AS_UID` unset: root, nothing re-owned, `DISABLE_*` honoured, `RUN_AS_*` not in the services' environment. (B) `RUN_AS_UID=1000 RUN_AS_GID=1001` on A's data directory, root-owned `meshcore.db` inside: services run as 1000, their environment has `RUN_AS_UID=1000 RUN_AS_GID=1001`, `meshcore.db` and the directory are 1000:1001, `.env` stays root's, `/etc/passwd` stays root's. (C) `.env` owned by uid 1000: refused with the reason, services run as 1000. The same image with master's two scripts mounted over it, in B's shape: services' environment `RUN_AS_UID=0 RUN_AS_GID=0`, `/etc/passwd` owned by 1000, `.env` owned by 1000:1001. - `sh test-all.sh` passes; no JS or Go change. **Not checked** - Windows: the suite stops at the symlink fixture, as before. - busybox as `TEST_SHELL` on my machine: Ubuntu's busybox prefers its own applets over `PATH`, so the stubs are bypassed and 28 cases fail, on master and on this branch alike. Your WSL Debian run (103 passed) and the image itself, which runs Alpine's ash with real binaries, are what cover ash. - Rootless Docker and `userns-remap`: unchanged, docs gap already noted. Note 2: kept the names case as a second guard; the `set -- root` case is the one that pins the design. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Adam
|
4fdd494d9b |
fix(nodes): read the transport-direct path byte at offset 5 (#2225) (#2234)
Red commit:
|
||
|
efitenandClaude Opus 5.5
|
927a8eec8a |
fix(nodes): node analytics rates cover the data in memory, and say so (#748) (#2211)
Refs #748 (M3a: correct numbers within the in-memory window; the SQL history in M3b is a separate plan, see the end). ## Problem The node analytics range picker offers 24h, 7d, 30d and All (365d), but the store only holds its in-memory window (`retentionHours`, then eviction). `GetNodeAnalytics` still divided availability and packets/day by the requested range. Measured on our live instance (14-day window), repeater BE-BRE-ON8AR: | Range | Availability | Packets/day | |---|---|---| | 7d | 88.7% | 91 | | 30d | 42.6% | 265 | | All | 3.5% | 22 | Nothing on the page said that the data stopped at the edge of the window. ## What changes - `coverageStart` returns the edge of the in-memory window: the latest of `oldestLoaded` (background-load progress), `now - retentionHours`, and a new `evictedThrough` that eviction sets to the newest `FirstSeen` it removed, so memory-pressure eviction inside the retention window moves the edge too. - `s.packets[0]` is not that edge, and the first version of this PR used it. Chunks are selected by `last_seen`, so a transmission first heard weeks ago and heard again recently sits at the head with its old `FirstSeen`. On staging (7-day window) it returned 21/07. - When the edge lies more than a minute after the requested start, node analytics counts only transmissions from the edge, computes availability and packets/day over that span, and returns `timeRange.dataFrom` and `timeRange.partial`. - The minute of slack keeps a range equal to the retention window from reading as partial. Without it every 7d request on staging (168h) was partial, because the two edges came from separate clock reads. - The analytics page shows "Data from 2026-10-07: the server keeps about 7 days of packets in memory. Availability and packets/day cover that period, not the full 365 days." when partial. - **UTC fix.** The window bounds of node analytics and node hop analytics are formatted in UTC. They are compared as strings with UTC `FirstSeen` values, so on a non-UTC host the 7d window lost its oldest hours (2 at +02:00). Docker images run in UTC, which is why this did not show in production. - `nodeTxsSince` (one caller) is inclusive, so a packet exactly on the edge is kept. ## Tests - Go, `node_analytics_coverage_test.go`: - full coverage within memory; - 30d over 3 days in memory; - empty store; - a re-heard 85-day-old transmission at the head of `s.packets` (edge stays at retention, packet not counted, no bucket before `dataFrom`); - memory-eviction edge; - eviction records `evictedThrough`; - retention 30 s shorter than the range is not partial, while 30d is; - window bounds under a +02:00 `time.Local`. - Two tests were checked to fail on the old code: the local-zone test (166 of 168 packets, a `+02:00` bound) and the retention test without the slack (`dataFrom 09:57:08` vs `from 09:56:38`). - Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). One run of five reported `FAIL` and four reruns were green; I did not capture which test failed. - Frontend unit: `tests/unit/test-issue-748-node-analytics-coverage.js` (5 assertions on the real `node-analytics.js` in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-748-node-analytics-coverage-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`), against the real fixture response with no interception: 7d is not partial and shows no notice; All is partial and the notice names the real start date. 2/2 locally. ## Validation Staging (retentionHours 168), same repeater, after the background load completed: | Range | Before | After | |---|---|---| | 7d | 82.7% | 83.3%, not partial | | 30d | 19.3% | 83.3%, partial from 2026-10-07 | | All | 1.6% | 83.3%, partial from 2026-10-07 | In the browser the notice appears under All and not under 7d. ## Not done - M3b, older history from SQLite. In the database only the sender is indexed (`from_pubkey`). Recipients sit in `decoded_json`, and relay hops only in `observations.resolved_path` (filled for about 27% of observations on our live instance, not indexed). A fallback today would show only a node's own packets for older days, a different meaning than the in-memory part. Doing it properly needs a per-node index written by the ingestor (pubkey, transmission, role); that is a separate plan. - The other analytics endpoints with a range (observer analytics, RF) were not checked for the same dilution. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
cab4f71c64 |
feat(nodes): LoRa radio settings per node from the MeshCore map (#1268) (#2209)
Fixes #1268. Frequency, spreading factor, bandwidth and coding rate are per-device settings that never travel in a packet. The public MeshCore map is the only source. This PR adds them to the node detail views, opt-in, and answers the open policy questions from the triage comments. ## Policy decisions | Question (triage) | Answer in this PR | |---|---| | Opt-out for self-hosted instances | Opt-**in**: `rfParams.nodesUrl` empty (default) = nothing fetched, `/api/rf-params` returns `enabled: false`. Same model as `knownChannelsUrl`. | | 33 MB every 6 h, thundering herd | Random 0-60 s delay before the first fetch, then every `refreshHours` (default 6). The map sends gzip (17 MB on the wire). It sends no ETag or Last-Modified, so conditional requests are not possible. | | Size cap | 128 MB on the decoded body (it is 51.5 MB today). Exceeding it fails the fetch instead of parsing a truncated body. | | Field shapes at the parse boundary | 64-hex key; 100-1100 MHz; SF 5-12; bandwidth from the LoRa set (7.8 to 500 kHz); CR 5-8. Invalid rows are dropped. A body that is not a well-formed array fails the fetch. | | Keep stale cache on anomaly | Any failure keeps the previous snapshot; a failed presets fetch keeps the previous presets. `fetched_at` is returned so an outage is visible rather than hidden. | | Freshness | Only adverts within `maxAgeDays` (default 7) are kept at fetch time and re-checked at serve time. | | TLS pinning | Not done: Go's certificate validation plus the field validation above. | ## What changes - `cmd/server/rf_params_cache.go`: stream decoder, validation, cap reader, preset parser (presets with identical values are merged, titles joined), background refresh. In memory only, no DB writes, so the read-only server invariant holds. - `GET /api/rf-params`: `{enabled, fetched_at, source, presets, nodes}`; `nodes` holds only nodes this instance knows, keyed by lower-case pubkey. `/api/nodes` is unchanged, so its payload does not grow for every consumer and the 500-row paging is unaffected. - Node detail (side panel and full page): `LoRa` row with the four values, the preset name when all four match one, and the age of the map advert. Shown only when the feature is on and the server has fetched the map at least once. - No column in the Nodes table, on purpose: on our network such a column showed one preset title in 814 of 1,366 rows and nothing in 443. Showing only the nodes that differ from their region is the follow-up PR. - `config.example.json` documents the block with the two URLs to use. ## Cost - Measured on the real map payload (51.5 MB, 66,336 nodes): parse 0.52 s, 15 MB allocated in total, 26,815 nodes kept (advert within 7 days). Runs in a background goroutine every 6 h. - The endpoint walks the cached node list once with map lookups: O(nodes). - Frontend: one bulk fetch of `/api/rf-params` when a node detail opens (client TTL 5 min), not one per node. ## Tests - Go (`rf_params_cache_test.go`): parse with every rejection reason, malformed bodies (not an array, truncated, wrong type, empty), preset merge and invalid presets, cap reader (one byte over refused, exactly at the cap accepted), fail-soft against an `httptest` upstream (503, HTML body, presets-only failure), config defaults, route (disabled shape, only known nodes, entry aged past max age dropped at serve time). Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, a Windows-only failure on clean master). - Frontend unit: `tests/unit/test-issue-1268-rf-params.js` (7 assertions on the real file in `vm`), in `test-all.sh`; the full suite passes. - E2E: `tests/e2e/test-issue-1268-rf-params-e2e.js` (registered in `deploy.yml` and `scripts/non-unit-tests.json`). It checks the real disabled state on the fixture server (no detail row), then serves `/api/rf-params` through `page.route`: no table column, full detail page with preset, side panel with raw values, and the hint for a node that is not on the map. 4/4 locally, plus the existing node table and node detail E2E files. ## Validation On a staging instance with `rfParams` set to the map and preset URLs: - Log: `[rf-params] 26821 nodes within max age, 20 presets`, about 6 s after start. - `/api/rf-params` returned 1,115 of our nodes: 988 on 869.618/SF8/BW62.5/CR8, 111 on the same with CR5, 15 on 433.65 MHz, 1 on SF7. - In the browser the detail row shows in the side panel and on the full page, and the Nodes table keeps its original columns. ## Not done - Highlighting nodes whose settings differ from their region's majority (a badge in the Nodes list plus a filter) is the follow-up (separate PR). In that follow-up a CR-only difference stays informational until it is confirmed whether MeshCore radios use the explicit LoRa header (then CR travels in the header and a mismatch still decodes). - `maxAgeDays` is config-only; the customizer is a later step. - No data is written to the DB, so the RF parameters are gone after a restart until the next fetch (at most 60 s plus the fetch time). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Adam
|
6d56f55168 |
fix(nodes): place nodes in regions by zero-hop home regions (#1879) (#2190)
Red commit: |
||
|
Alex BandClaude Opus 5.5
|
2cdb18e1dd |
perf(store): the decoded-payload cache drops fields nothing reads (#2198)
## Problem
`ParsedDecoded` caches each packet's decoded payload for as long as the
packet is stored, and the analytics passes fill it for every packet. It
kept fields that no server code reads:
- the ciphertext and MAC of encrypted payloads;
- advert signatures (and their validity flag);
- ISO copies of timestamps;
- raw path data;
- control-packet details.
Together those are about a third of the cached JSON.
## Change
- After decoding, the cache deletes those keys
(`decodedCacheDroppedKeys`).
- The live feed was the only place that sent the whole map.
`broadcastPayload` now passes the stored `DecodedJSON` through as a
`json.RawMessage` (after `json.Valid`), so clients still get every
field, byte for byte. Nothing in the server reads the broadcast payload
as a map; it is only serialised. This replaces the shallow copy of the
cache, so no JSON is decoded or map copied under the store's write lock
in either ingest path, and no nested map (such as `flags`) is shared
with the cache.
- `TestDecodedCacheDroppedKeysAreUnread` is a guard: it fails if any
server code starts naming a dropped key, so a future reader can't
silently get nothing.
## Measured
| | before | after |
|---|---|---|
| decoded cache's live heap (copy of a busy week) | 46 MB | 39.5 MB |
The saving is smaller than the JSON share suggests, because the
per-packet map overhead dominates. A typed struct would be the next
step, but that touches 27 call sites.
## Tests
- `TestParsedDecodedDropsUnreadKeys` and
`TestBroadcastPayloadKeepsEveryKey` (every key present in the broadcast,
and invalid JSON left out as before): red commit first.
- The guard test above.
- `go test ./cmd/server` passes. `map[string]interface{}` in
`cmd/server`: 933 on master, 932 here.
No config or UI changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Alex BandClaude Opus 5.5
|
92dd172f6e |
perf(store): bulk loaders reuse their row scan targets (#2201)
## Problem
`Load`, `loadChunk` and `scanAndMergeChunk` declared their ~20
`sql.Null*` scan targets, and built the `Scan` argument slice, inside
the row loop. Each target's address goes into the `[]interface{}`, so
each one escaped and was heap-allocated again on every row, along with
the slice and its appends. On a week of a busy network that was the
biggest flat cost inside `loadChunk`, about 330 MB.
## Change
- The targets and the argument slice are declared once, before the loop.
- `Scan` overwrites every target on each row.
- Nothing else takes their addresses; the `null*` helpers copy their
argument.
## Measured
| | before | after |
|---|---|---|
| allocations per row, fixture (`Load` / `LoadChunked` / `loadChunk`),
on current master | 85.9 / 87.1 / 86.0 | 63.9 / 65.1 / 64.0 |
| allocated over startup (copy of a busy week) | 5.41 GB | 5.05 GB |
| server CPU over the run | 175.6 s | 172.4 s |
The per-row figures are from the test on current master. The startup
figures were measured before #2183 and #2185 landed; this change saves
the same ~22 allocations per row on top of them.
**Scale:** this is the cold-load hot path, once per observation row
(about 400k rows on that week). The change removes per-row work and adds
none; complexity is unchanged.
## Tests
- `TestBulkLoadersReuseScanTargets` sets an allocations-per-row bound
for each loader, between the before and after values. Red commit first,
then green.
- `go vet ./...` is clean and `go test ./...` passes in `cmd/server`;
`-race -count=2` passes on the loader and observer-string tests.
- Local SonarQube: quality gate passed, no issues on added lines.
- `map[string]interface{}` in `cmd/server`: 934 on master and here.
No config or UI changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a164a44e7f |
fix(docker): run the Go services as an unprivileged user (#2125)
Runs `corescope-server` and `corescope-ingestor` unprivileged, opt-in with `RUN_AS_UID`. A code-execution bug in either one then no longer lands as root next to the mounted data and `config.json`. `supervisord` stays root, for mosquitto and caddy. | `RUN_AS_UID` | Go services run as | |---|---| | unset (default) | root, as before. Upgrades change nothing | | set | `RUN_AS_UID` / `RUN_AS_GID` | With `RUN_AS_UID` set: - Each start hands the data directory to that uid (`chown -h`), except `.env` and anything mounted into it on its own. Failures are logged. - `.env` is read only if it is root-owned and only root can write it. `RUN_AS_UID` and `RUN_AS_GID` are never taken from it. - `/app` stays root's. **Depends on #2137.** - A uid with no account in the image gets one at start; `caddy` and `mosquitto` are reused. A uid that can't be provided fails startup instead of falling back to root. The logic is in `docker/select-service-user.sh`, tested by `docker/test-select-service-user.sh`, which needs no Docker or root and runs in CI. See also "Running the Go services unprivileged" in `docs/deployment.md`. Verified in the built image (master + this + #2137) across 14 cases. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Mythos 5.1 <noreply@anthropic.com> Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> |
||
|
efitenandClaude Opus 5.5
|
1ae4c74913 |
feat(observers): rolling 1 h TX airtime and a duty-cycle warning (#661) (#2208)
Fixes #661 (the narrowed scope from the 2026-10-14 comment). ## What changes - `GET /api/observers/metrics/summary` adds `tx_airtime_pct_1h` and `rx_airtime_pct_1h` per observer: the airtime share over the hour before now, whatever `window` is, from the cumulative `tx_air_secs` / `rx_air_secs` counters in `observer_metrics`. - A counter that goes down marks a reboot; only the stretch after the last reset counts. - Under 15 minutes of samples (one sample, or a reboot late in the hour) returns `null`. - Clamped to 0..100, rounded to 2 decimals. - The response also carries `duty_cycle_warn_pct`, from the new config key `dutyCycleWarnPct` (default 10, the EU 868 MHz g3 sub-band limit). Added to `config.example.json`. - RF Health tiles show `TX 1h: x%`; the observer list gets a sortable numeric `TX 1h` column. Both use the shared `public/observer-airtime.js` helper and flag values at or over the threshold with `.airtime-over` (`var(--status-red-text)`). - The handler now returns a named `MetricsSummaryResponse` struct instead of `map[string]interface{}` (one occurrence fewer). ## Why the warning matters Repeater and room-server firmware default to `airtime_factor = 1.0` (`examples/simple_repeater/MyMesh.cpp:890`), which `Dispatcher.cpp:26` turns into a 50% TX budget per 1 h window. The device therefore does not keep itself under the EU 10% limit; this view is where an operator can see it. ## Cost - Server: one extra query per summary request, `SELECT observer_id, timestamp, tx_air_secs, rx_air_secs FROM observer_metrics WHERE timestamp >= now-1h`, on the existing `idx_observer_metrics_timestamp`, then one O(rows) pass. On our instance that is 35 observers x about 12 samples (5 min interval). - Observer list: one extra bulk call `summary?window=1h` per load (client TTL 30 s, failure ignored), not a call per observer. ## Tests - Go: `TestRollingAirtimePct` (steady, reboot mid-hour, single sample, span under 15 min, reboot leaving under 15 min, nil counters, clamp, rounding), `TestGetMetricsSummaryAirtime1h` (samples older than 1 h ignored inside a 24 h window, single-sample observer gets null), `TestMetricsSummaryDutyCycleWarnPct` (default 10, config override, `observers` is `[]` not `null`). Full `cmd/server` suite passes locally (skipping `TestSaveGeoFilterPreservesFileMode`, which fails on Windows on clean master too). - Frontend unit: `tests/unit/test-issue-661-observer-airtime.js` (8 assertions, real file in `vm`), added to `test-all.sh`. `test-observers-headings.js` updated for the new column. Full `test-all.sh` passes locally. - E2E: `tests/e2e/test-issue-661-airtime-e2e.js`, registered in `deploy.yml` and `scripts/non-unit-tests.json`. The fixture DB has no `observer_metrics` table, so the summary is served through `page.route` with known values for the fixture's own observer ids. It checks the column header, over/under/unknown cells, sort order and the RF Health tile. 4/4 locally. It caught a real bug during development: `loadRFHealthData` declares a local `const window`, so the tile calls the helper as the global `ObserverAirtime`. ## Validation Deployed to a staging instance with live MQTT data (35 observers with metrics) and checked in the browser: - The observer list shows the column for 42 rows (33 with a value, 9 without enough samples). Sorting puts the highest share (5.3%) first. - RF Health shows `TX 1h` on all 35 tiles. With a fourth stat the line wrapped inside items on narrow tiles; the stats row now wraps between items only. - No observer was over 10% at the time, so the red state was checked by rendering one cell client-side at 12.5% (computed colour `rgb(185, 28, 28)` in light mode), and by the E2E test. ## Not done - The threshold is config-only; exposing it in the customizer is a follow-up. - No airtime comparison across observers (that is #671) and no per-packet airtime (Relay Airtime Share covers it). - Dark mode was not checked in the browser; `--status-red-text` is defined for both themes. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
7c7c510fb8 |
docs(readme): correct stale claims and add the optional features (#2205)
## Summary The README is the repo's front page. I checked each claim against `master` and fixed what no longer holds. | README said | Now | Source | |---|---|---| | `POST /api/packets` for manual injection | route is GET only; points at `POST /api/decode` | `cmd/server/routes.go:317`, `:320` | | `PORT` env var overrides the port | not read anywhere; documents `port` and `-port` | `cmd/server/main.go:96` | | 5 store indexes | 7 (`byPathHop`, `byPayloadType` added) | `cmd/server/store.go:222-231` | | 11 analytics tabs | 17 `data-tab` values | `public/analytics.js` | | VCR up to 4× | up to 8× | `public/live.js:681` | | 14 Go modules | 16 `go.mod` files (also fixed in the `Makefile` header comment) | | | 327 / 53 / 380 Go tests, 150+ Node, 49 Playwright | 1827 / 710 / 2742 `func Test`, 199 unit files, 131 `test(` calls; counts removed so they stop going stale | | | live instance + Swagger on analyzer.00id.net | times out (from two networks, 2026-10-13); links analyzer.on8ar.eu, `/api/docs` and `/api/spec` return 200 there | curl | Added: Scope Audit in the feature list, and an "Optional Features" section for user accounts and mobile RX coverage, each linking its existing doc. ## Not checked - the performance figures (~300 MB for 56K packets, < 1 ms queries) - the "AES-128-ECB with rainbow table" line - whether the screenshots still match the current UI - the "-or-later" in the license line (`LICENSE` is GPL v3) Docs only, no tests run. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
fbeb9e4d71 |
ci: frontend coverage from every E2E suite, merged across the shards (#2204)
Fixes #2203. ## Problem The frontend coverage badge (35.16% on run 38413268001) came from two sources on shard 2: one dump part-way through `test-e2e-playwright.js` and the `collect-frontend-coverage.js` tour. The other E2E suites run against the instrumented build but never save `window.__coverage__`, including the suites written for single pages (`path-inspector.js` 4.72%, `touch-gestures.js` 14% on that run). ## Change 1. `9d3bfad5` `test(e2e): red:` `tests/unit/test-e2e-coverage.js`, with fake pages, contexts and browsers (no Chromium needed). On the placeholder module the first assertion fails. 2. `c00bbb12` `feat(e2e):` the collector. - `scripts/e2e-coverage-hook.js` is a preload (`node --require`) that wraps `chromium.launch` from `playwright`. `playwright` re-exports `playwright-core`, so a suite's own `const { chromium } = require('playwright')` gets the same, wrapped object. - `scripts/e2e-coverage.js` reads `window.__coverage__` before every call that replaces or ends a page's document: `goto`, `reload`, `setContent`, `goBack`, `goForward`, `close`, and context and browser `close`. Pages from `newPage`, `newContext` and the context `page` event are all tracked. - Each read zeroes the page's counters, so a document read twice is not counted twice. Maps travel once per file; later reads carry counts only. - The page returns a JSON string. Returning the object cost about 10 s per suite, because Playwright serialises it node by node. - A read is given up after 5 s, so a busy page cannot hold up a suite's cleanup. - On exit the hook writes one file per suite to `.nyc_output/`, or nothing without coverage. A failed write is logged and never changes the exit code. 3. `64487c4c` `ci:` the workflow. - On master pushes the E2E run step sets `NODE_OPTIONS=--require ./scripts/e2e-coverage-hook.js`. On PRs it is empty, so PR runs are unchanged. - Each shard runs `nyc merge` (one file per suite is about 6 MB, mostly the same maps) and uploads `coverage-shard.json` with its results instead of `.badges/`. - The `e2e-test` job merges the three shards and makes the frontend badge (it used to come from shard 2 alone). - The merge, setup-node and badge steps are `continue-on-error`. When `nyc report` gives no statement percentage the step warns and writes no badge, so a registry hiccup cannot publish 0%. - `test-e2e-playwright.js` no longer writes its own `e2e-coverage.json`: the hook covers that suite completely, including the tests after the old dump. `collect-frontend-coverage.js` stays as an extra source. ## Performance Measured locally against an instrumented server (Windows, Chrome): | Suite | Without hook | With hook | |---|---|---| | `test-path-inspector-coverage-e2e.js` | 1.9 s | 2.2 s | | `test-touch-gestures-coverage-e2e.js` | 17 s | 16 s | | `test-e2e-playwright.js` (stops at the same local-fixture failure in both runs) | 52 s | 52 s | | `browser.close()` on a page in a 30 s busy loop | 153 ms | 5.2 s | The hook only loads on master pushes. ## Testing - `tests/unit/test-e2e-coverage.js` covers: - reads before each document-replacing call and before close; - pages opened by the page itself; - context and browser close; - summing without double counts; - a failed read, and a read that never answers; - the file writer, including a failing write. - Nine deliberate breaks of the library each turn it red: - dropping the counter reset; - dropping the `page` event, the `goto` read, the `goForward` read, the `close` read or the browser `close` read; - removing the timeout or the write guard, or writing an empty store. - The test also fails if it stops before its end. Without that check, a read that never returns would let node exit 0 half-way. - Real suites with the hook, measured on their own (the badge run had all suites): | File | Badge run 38413268001 | Own suite with the hook | |---|---|---| | `path-inspector.js` | 4.72% | 52.75% | | `touch-gestures.js` | 14% | 72.96% | - Exit codes and pass counts were unchanged with the hook for these suites (the first three were run before the 5 s read timeout was added): - `test-slideover-1056-e2e.js` (6 contexts); - `test-a11y-axe-1668.js` (`@axe-core/playwright`); - `test-issue-1150-404-state-e2e.js`; - the three above. - The badge step, run locally against two merged shard files: - It gives the same per-file numbers as `nyc report` on the inputs. - With no shard coverage, or with `nyc report` failing, it warns and writes no badge. - `sh test-all.sh` passes locally, apart from `test-issue-1956-release-routing.js`, which needs the workflow's `GITHUB_REF_NAME` and was skipped. The test inventory classifies 321 suites. `actionlint` 1.7.7 reports only the self-hosted runner label, as on master. **Not tested:** the shard and badge steps in CI. They only run on a master push, so the first master run after merge is the real test. I do not know what the merged number will be. I will compare the badge with the merged report in that run's `e2e-test` log. **Not counted:** a full page load the suite does not start through one of the wrapped calls, such as a link that leaves the SPA or `location.reload()` from page code. ## Not in this PR - The `tests/unit/` suites, which load frontend files through `vm.createContext` without instrumentation (noted in #2203). - Coverage on PR runs. The overhead above is small, so this could follow if wanted. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
59708ee72e |
ci: publish the README badges to a badges branch with GITHUB_TOKEN (#2202)
Fixes #2136. ## Problem The README badges are stale: `e2e-tests` shows 821 passed (1161 when #2136 was filed), both Go coverage badges have not changed since 2026-03-30 and frontend coverage since June. There are two causes. 1. **The PAT.** The `publish` job wrote the badge JSON to `master` through the Contents API with `BADGE_PUSH_TOKEN`, a PAT that could push past the branch protection on `master`. That token returns 401: run 38360160932 on 2026-10-13 logs `gh: Bad credentials (HTTP 401)` for every badge. The step is `continue-on-error` and only echoed `Failed to update`, so every run stayed green. 2. **Old badges inside `e2e-badges`.** `.badges/` is tracked on `master`, so every e2e shard checkout and the aggregate checkout already hold the old files. The shard upload and the aggregate `cp` shipped them in `e2e-badges`, and `publish` downloaded `e2e-badges` after `go-badges`, so the old values overwrote the fresh ones. In run 38407143126, `go-badges` holds 85.8% and 77.1%, while `e2e-badges` holds 85.4%, 70.2% and frontend 36.64%, the same as `master`'s `.badges/`. Fixing only the token would have left three of the four README badges frozen. ## Change 1. `5f430966` `ci:` the `publish` job pushes the badges to an unprotected `badges` branch with its own `GITHUB_TOKEN`. - Job-level `permissions: contents: write`. No other job gains a permission. The checkout uses `persist-credentials: false`, since the push passes its own token. - The branch is a single orphan commit, force-pushed, so it does not grow. Master push runs are already serialised by the workflow `concurrency` group. - Each run starts from the branch as it is and overwrites only the badges it downloaded, so a missing artifact keeps those badges at their previous value. The first time, when `git ls-remote --exit-code` reports the branch absent (exit 2), it starts from `master`'s `.badges/`. - No downloaded artifacts, or any other `ls-remote`, fetch, extract, commit or push failure, ends in a `::warning::` in the run summary and leaves the branch alone. The step stays `continue-on-error`, so a badge problem never turns the pipeline red. - `go-badges` is downloaded after `e2e-badges`, so its files win. 2. `2134c1e5` `docs:` the four README badge URLs read `.../CoreScope/badges/<name>.json` instead of `.../master/.badges/<name>.json`. 3. `47e57482` `ci:` the frontend-coverage step in the e2e shards and the aggregate badge step start from an empty `.badges/`, so `e2e-badges` carries only what the run produced. This also stops shard 3's old `frontend-coverage.json` from replacing the one shard 2 generates. No PAT is involved any more, so nothing expires. ## Testing - The publish step script, taken from the YAML, run against a local bare repository standing in for `origin`: | Run | Downloaded | Result on `badges` | |---|---|---| | 1, branch absent | a, b | a, b new; c from `.badges/`; one commit | | 2 | c only | a, b kept from run 1; c new; still one commit | | 3 | none | `::warning::`, branch unchanged, step exit 0 | | 4, `origin` unreachable for the read | a | `::warning::`, branch unchanged, step exit 0 | | 5, push to a missing remote | a | `::warning::`, branch unchanged, step exit 0 | - The artifacts of run 38407143126 land flat in the download path, so `badges-new/*.json` matches them. - `shellcheck` at style level on the step script: no findings. - `actionlint` 1.7.7 (shellcheck off): one finding, the self-hosted runner label `meshcore-runner-2`, which `master` reports as well. **Not tested:** the e2e change and the push with a real `GITHUB_TOKEN`. Both only run on a push to `master`, so the first master run after merge is the real test. The `badges` branch already exists (`80fda6c0`, a copy of `master`'s `.badges/`), so the README badges keep working from the merge on, with the old values until that run. After it, I will check the `badges` branch contents against that run's go and frontend numbers, the step log and the rendered README. ## Not in this PR - Removing the now unused `.badges/` from `master`: the first run seeds the branch from it, so that goes in a follow-up once the branch exists. - Deleting the `BADGE_PUSH_TOKEN` secret: that needs a repository admin. - `frontend-tests.json`: no CI step produces it and no README badge uses it. It is carried over to the branch unchanged. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
b8e571c9c5 |
perf(store): build the relay-filtered candidate lists once (#2195)
## Problem `relayCandidates` runs once per hop of every observation the cold loader resolves, and again from the direct-heard and neighbor analytics. Once any listener-only observer was marked (#1290), every call built a filtered copy of the candidate `nodeInfo` structs, even when none of the candidates was a listener. On a week of a busy regional network that accounted for 1.1 GB of the 5.4 GB allocated over one startup, the biggest single source of startup churn. ## Change `markNonRelay` now builds a map once, holding the filtered list for each prefix whose candidates include a listener. A listener can only be in the lists of its own prefixes and its full key, so only those are filtered: at most 9 lists per listener, not every list in the prefix map. `relayCandidates` returns either that list or the prefix map's own list, without copying. Every caller only reads the slice (checked: direct-heard, neighbor persist, node hop analytics and `resolveWithContext`). ## Measured on a copy of that week | | before | after | |---|---|---| | allocated over startup | 5.44 GB | 4.32 GB | | server CPU over the run | 175.6 s | 169.1 s | | peak RSS | 1074 MB | 1077 MB (unchanged; the GC limit sets it) | ## Cost per rebuild against cost per lookup `markNonRelay` now does once per node cache rebuild (as often as every 30 s, `nodeCacheTTL`) the filtering that master did on every `relayCandidates` call. 5,000 repeaters and 3 listeners, median of 6 runs (`BenchmarkMarkNonRelay5000`, `BenchmarkRelayCandidates256`): | | master | first version of this PR | this PR now | |---|---|---|---| | `markNonRelay`, once per rebuild | 0.59 µs, 320 B, 3 allocs | 14.8 ms, 6.57 MB, 36,285 allocs | 20 µs, 20.6 KB, 33 allocs | | one `relayCandidates` lookup | ~7.8 µs, 6.7 KB | ~30 ns, 0 B | ~30 ns, 0 B | | 256 lookups (one per 2-character hop) | 2.0 ms, 1.71 MB, 1,453 allocs | 7.7 µs, 0 B | 7.7 µs, 0 B | The extra ~20 µs per rebuild is paid back after about 3 lookups. Every ingested observation looks up each hop of its path, so a running server makes far more than that between rebuilds, and a cold load makes millions (the 1.1 GB above). ## Tests - `TestRelayCandidatesDoesNotAllocate`: zero allocations per call, and listeners still filtered. - `TestRelayCandidatesAfterSecondMark`. - `TestMarkNonRelayVisitsOnlyListenerPrefixes`: at most 80 allocations for 3 listeners among 5,000 repeaters (36,285 before the fix). It has its own red commit. - `BenchmarkMarkNonRelay5000` and `BenchmarkRelayCandidates256` give the table above. - `TestMarkNonRelayMatchesFilteringEachList`: every prefix of the 5,000-repeater map matches filtering its list on the spot. - Red commit first, then green. - `go test ./cmd/server` passes. No config or UI changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
8d2ae405e8 |
perf(store): cache an observation's parsed time in 8 bytes (#2196)
Refs #2148. ## Problem `StoreObs` is the store's most numerous struct: 400k of them on a week of a busy regional network. Its lazy timestamp cache (#1481) is a `time.Time`, a `sync.Once` and a `bool`, together 40 bytes. On current master that makes the struct 200 bytes, which puts every observation in the 208-byte size class. ## Change The cache is now a single `atomic.Int64`: - `0` before the first call; - the parsed time as UTC nanoseconds after it; - or one of two sentinels: one for empty or unparseable strings, and one for times it can't hold exactly (a non-UTC offset, or outside `UnixNano`'s range). Those are re-parsed on each call so their location is preserved. Concurrent first calls may both parse; they store the same value. `StoreObs` is now 160 bytes, in the 160-byte size class: 48 bytes × 400k observations ≈ **19 MB** less heap. The first parse costs the same; cached reads now build the `time.Time` from an int. ## Tests - `TestStoreObsFitsIn160Bytes`: red commit first. - `TestStoreObsParsedTimeMatchesParse`: Z, nanosecond, offset, SQL, empty, garbage and epoch inputs give the same time and location as `time.Parse`, on the first and the cached call. - `TestStoreObsParsedTimeConcurrentFill`. - The existing `storeobs_parsedtime_test.go` still passes. - `go test ./cmd/server` passes, and `-race` passes on the ParsedTime, retransmission and observer analytics tests. ## This PR or #2149 #2149 fixes #2148 by reordering these same three fields. The two conflict, so here is how they compare. Sizes are `unsafe.Sizeof(StoreObs{})` on each branch, with Go's heap size class beside them: | | struct | heap per observation | saved per observation | saved at 400k observations | |---|---|---|---|---| | master | 200 B | 208 B | – | – | | #2149 (field reorder) | 192 B | 192 B | 16 B | ~6.4 MB | | this PR (one 8-byte field) | 160 B | 160 B | 48 B | ~19 MB | - **#2149:** a pure field reorder with no change in behaviour. It is the smallest, safest change and the easiest to review. It saves a third of what this PR does. - **This PR:** three times the saving, but more code. It adds two sentinel values, and re-parses on each call any timestamp with a non-UTC offset. The store writes UTC (`Z`), so that path is rare. It is tested against `time.Parse` on edge cases, concurrently, and under `-race`. Options: 1. **Take this PR.** #2149 then has nothing left to reorder. 2. **Take #2149 now, and this PR after it** as a follow-up for the further 32 bytes per observation. I'll rebase onto #2149 if it lands first. 3. **Take only #2149** if the smaller, simpler change is preferred. No config or UI changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
nullrouten0andClaude Opus 5
|
478f74a452 |
fix(config): write the geo filter through a symlinked config.json (#2137)
Found while reworking #2125, unrelated to that PR's change. In the Docker image the entrypoint symlinks `/app/config.json` to the bind-mounted `/app/data/config.json`. `SaveGeoFilter` takes the first path `os.Stat` resolves, so `configPath` stays the symlink, and the tmp+rename then **replaces the link with a regular file inside the container**: - the mounted `config.json` keeps the old `geo_filter`, so the filter the operator just saved appears nowhere on the host; - server and ingestor read the detached file from then on, so the UI keeps showing the new filter and nothing looks wrong; - the next container start reverts it: the entrypoint's `ln -sf` puts the link back, and re-creating the container drops the file anyway. **Fix:** resolve a symlink at `configPath` before the write, so the save lands in the file it points at and the link stays a link. Only a regular file is followed — a link to a device, fifo or directory is left alone and handled as before. This is deliberately the opposite of `writeStatsAtomic` (refs #1170), where replacing a destination symlink is the point because the default stats path can be world-writable `/tmp`. `config.json` is different: the link is one the image itself creates, and the directory is not world-writable. Happy to narrow it further (e.g. only follow a link that stays inside `configDir`) if you'd rather keep the two paths consistent. **Verification:** `TestSaveGeoFilterSymlinkedConfig` fails on master — ``` --- FAIL: TestSaveGeoFilterSymlinkedConfig config_symlink_test.go:47: .../config.json is no longer a symlink (mode -rw-------) — the rename replaced it config_symlink_test.go:60: geo_filter missing from .../data/config.json: {"port":3000,"_comment":"keep me"} ``` — and passes here. `TestSaveGeoFilterKeepsOwner` covers keeping the file's owner, and `TestSaveGeoFilterFifoSymlinkTarget` checks that only regular files are followed. `go test ./...` in `cmd/server` passes. **Single-file mounts:** when `config.json` is mounted as a single file, the save now fails with 500 (`EBUSY`) instead of writing a copy that the next start discards. Our own instance has no geo filter configured, which is why this went unnoticed there; its `/app/config.json` is still a symlink. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
e53c055d6e |
perf(store): share one copy of each observer string across observations (#2183)
Part of #1870: this interns the three fields the issue names (`ObserverID`, `ObserverName`, `ObserverIATA`), plus `Direction`. ## Problem Each `StoreObs` kept the row scan's own copy of its observer ID (64 hex characters for most observers), observer name, IATA code and direction. A busy regional network's 7-day window holds about 400k observations from a few hundred observers, so a few hundred distinct values were stored hundreds of thousands of times. ## Change - `internStr` returns the canonical copy of each value from a map guarded by an `RWMutex`, so every observation shares one copy. - The copies are held for the life of the process. That is what makes the sharing survive garbage collections. An earlier version of this PR used `unique.Make`, which holds its copies weakly, so a value interned again after a GC came back as a second copy (thanks to the review for catching that). - Observer strings number in the hundreds. The map is capped at 65,536 entries as a guard (AGENTS.md: no unbounded structures); past the cap a string is returned as it came, so nothing breaks. - A hit takes only the read lock and allocates nothing. A new value is cloned, so it doesn't pin a larger buffer it may be a slice of. - All five places that build a `StoreObs` use it: `Load`, `loadChunk`, `scanAndMergeChunk`, and both ingest paths. - The ingest paths intern while scanning rows, **before** taking `s.mu`, so interning adds nothing to the time API readers wait on the store. - `cmd/server/go.mod` stays at `go 1.22`. The earlier 1.27 bump was only there for `unique`, and is reverted here; the module upgrade belongs in its own PR. ## Scaling `BenchmarkInternStr` (a hit, the steady state): | | alone | 12 cores (per call) | allocs | |---|---|---|---| | plain `Mutex` map (the first version of this PR) | 20 ns | 57 ns | 0 | | `unique` (the second version, withdrawn) | 48 ns | 8 ns | 0 | | **`RWMutex` map (this PR)** | 17 ns | 26 ns | 0 | ## Measured on a copy of that window (`GOMEMLIMIT=750MiB`) The same database and harness, one run each, the `unique` version against this one: | | `unique` version | this PR | |---|---|---| | live heap at the last GC after load | 581 MB | **535 MB** | | string bytes allocated by row scans still live (`bytealg.MakeNoZero`) | 106 MB | **80 MB** | | peak RSS | 777 MB | 781 MB | | load to ready | 15 s | 15 s | Peak RSS is set by the load itself, not by these strings. Against master, the first measurement of this PR found the strings retained from row scans went from 84 MB to 58 MB. An earlier version of this PR (the `unique` one) also ran on a staging instance (live MQTT ingest, `GOMEMLIMIT=1GiB`) through five back-to-back restarts and a two-hour soak, with no OOM kills and every request succeeding. This version has not had a staging soak yet. ## Tests - `observer_strings_intern_test.go`: - one copy of each value across 50 observations, through `Load`, `LoadChunked` and `loadChunk`; - **a value interned again after `runtime.GC()` is the same copy** (new: red against the `unique` version, green now); - 16 goroutines interning concurrently get the same copy; - a hit allocates nothing; - `BenchmarkInternStr` (serial and parallel). - Red commits first, then green. - `cd cmd/server && go test ./...` passes at `go 1.22`, and `-race` passes on the intern tests. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b43fd4a470 |
feat(#2107): backfill unresolved channel/scope rows when the key set grows (#2133)
Part 1 of #2107, scoped as discussed there: the backfill on its own, no catalogue promotion. ## Problem Ingest decides a GRP_TXT's channel and a transport packet's region scope once, at insert time. A row stored before its key was known keeps `channel_hash = 'enc_XX'` / `scope_name = ''` forever, even after the key is added — via `hashChannels` plus a restart, or at runtime when the `autoRegionKeys` tier picks up a newly declared region. On live.saarmesh.de we have been closing that gap with an out-of-tree script that rewrites the DB and restarts the container; this moves it into the ingestor and removes the restart. ## What this does **Ingestor** — `cmd/ingestor/key_backfill.go` - Runs after the write path is ready, and again whenever the key set grows (kicked after every `autoRegionKeys` refresh; a no-op when nothing is new). - Touches **only unresolved rows**: `enc_*` GRP_TXT and transport rows with `scope_name = ''`. Every `UPDATE` carries the unresolved value in its `WHERE`, so a row resolved meanwhile by live ingest keeps its value. A decrypted channel or named scope is never rewritten. - **Same decision as ingest**: `DecodePacket` with the full channel key map → `PayloadJSON`, and `regionKeySnapshot.match()` with the full snapshot — including abstaining on an ambiguous collision (#1609). The scope-match counters are not touched; they describe live traffic. - **Bounded**: each batch covers a fixed window of 5000 transmission ids. Unresolved rows are sparse (hundreds in a million), and the ingestor has a single SQLite connection, so a "next N matching rows" query would scan most of the table in one statement while ingest waits. A fixed id window keeps every read a short primary-key range scan. The writer is yielded between windows. - **Resumable and transactional**: one transaction per window holds the rewrites, their change-feed rows and the advanced cursor (`key_backfill_state`). A cancelled or crashed run resumes at its last commit; nothing is applied twice. - **Key-set growth**: a run is tied to the fingerprint of the key set it started with. If the set changes mid-run, the run restarts from the beginning, so a key added halfway still sees all of history. Each key gets one complete pass (`key_backfill_done`); a restart with an unchanged key set does no work. A key that leaves the set forgets its pass and is replayed if it returns. - **Derived field**: a named ADVERT fills `nodes.default_scope` / `inactive_nodes.default_scope` only where it is still empty. A known `default_scope` is newer evidence than a backfilled advert and is never overwritten. **Change feed** — `internal/dbschema/key_backfill.go`, `cmd/server/tx_rewrite_feed.go` - Each rewrite appends a row to `tx_rewrite_feed` in the same transaction as the rewrite. - The server tails it exactly like `advert_route_evidence`: cursor captured before `Load`, absent table re-probed (not required by `AssertReady`, so an older ingestor simply means no feed), polled from `IngestNewObservations`. It refreshes `DecodedJSON` / `ScopeName` of loaded transmissions, resets the `ParsedDecoded` cache, recharges tracked bytes, and invalidates the channel caches. **No restart needed.** - The feed carries only the transmission id; current values are read from `transmissions`, so replaying a row is harmless. Rows older than 24h are pruned. ## Tests Ingestor (`key_backfill_test.go`) — fixtures are real wire frames: GRP_TXT encrypted AES-128-ECB with the firmware MAC, transport frames with the real region HMAC. - decrypts only unresolved rows; already-decrypted and unknown-key rows untouched - backfilled `decoded_json` / `channel_hash` are byte-identical to what `BuildPacketData` stores at ingest - unique match named; a constructed two-name collision on the same payload stays unmatched; named and non-transport rows untouched - ingest scope-match counters unchanged - ADVERT fills only an empty `default_scope` - idempotent: second pass writes nothing, done keys recorded, state cleared - cancel between commits → resume from the committed cursor, every row exactly once, in order - stale run fingerprint (key added mid-run) → restart reaches rows below the old cursor - a later key only adds its own rows; a removed key is replayed over the gap when it returns - a row resolved between decide and commit keeps its value and gets no feed entry - channel hash byte derivation matches the builtin Public key (0x11) - old feed rows pruned Server (`tx_rewrite_feed_test.go`) - rewrite reaches the loaded store through the normal `IngestNewObservations` poll, including a previously parsed `ParsedDecoded` (verified to fail without the `sync.Once` reset) - cursor starts at the load watermark - absent feed is a no-op and is picked up once created, without restart - failed read keeps the cursor and retries - feed row for an unloaded transmission advances the cursor and is ignored `gofmt` clean, `go vet` clean, no `go.mod` / `go.sum` changes. ## Validation against production data Ran the backfill on a snapshot of live.saarmesh.de taken **before** last night's run of our out-of-tree script (13,100 unresolved rows, 164 channel keys, 24 region keys incl. 2 derived), with the current config. It rewrote **the same 31 transmissions with the same channel names** as the script did on production (compared by id), named 0 scopes (script: 0), and a second pass wrote nothing. Browser validation (AGENTS rule 2): this changes no UI. The visible effect — a decrypted message or named scope appearing without a restart — is covered by the server feed tests; I have not yet run it against a deployed instance. ## Notes for review - Window size (5000 ids), inter-batch yield (10 ms) and feed retention (24h) are constants for now; per rule 8 they can move to config/customizer later if anyone needs to tune them. - `client_rx_observations.scope_name` is not backfilled here; happy to add it in a follow-up if wanted. - The orphaned "scope-repair" mention in `region_keys.go`'s `scopeMatch.Candidates` comment predates this PR; I left it alone. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01BKaogNXgy8dEQruqzR5foP --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
abc6ab9b59 |
fix(analytics): apply nodeBlacklist and hiddenNamePrefixes to Route Patterns (#2184) (#2189)
Fixes #2184. ## Problem `nodeBlacklist` and `hiddenNamePrefixes` did not apply to Route Patterns: - **`subpaths` and `subpaths-bulk`:** `filterBlacklistedFromSubpaths` asserted `data["subpaths"].([]interface{})`, while the store builds `[]map[string]interface{}`, so it returned its input unchanged. It then read `hops` as a list, which is an `int` (the run length). It also ran only when `nodeBlacklist` was set, so `hiddenNamePrefixes` alone never filtered anything. - **`subpath-detail`:** the check compared each raw hop (a prefix) with full keys, so it matched only a client that sent a full key. - **Parent paths** in the detail response were not filtered. ## Change All in `cmd/server/routes.go`; `store.go` is untouched, so this doesn't overlap #2182. - **Lists:** an entry is dropped when a name in its resolved `path` is the name of a blacklisted node (names looked up in the store's node cache) or has a hidden prefix. A subpath carries names and prefixes, not keys, and the store picks a node per prefix from context, so re-resolving a prefix in the handler would not be guaranteed to land on the same node. Matching the names the reader would see avoids that. Trade-off: another node with exactly the same name as a blacklisted one is hidden too. - **Detail:** 404 when a hop resolved to a blacklisted node (by the resolved key in `nodes[].pubkey`, so a node without a name is caught and one that only shares its name is not) or to a hidden-prefix name; parent paths that show one are dropped (by name, like the lists). The existing raw-hop check stays. - The filter runs when either `nodeBlacklist` or `hiddenNamePrefixes` is set. - `dropHiddenPaths` replaces `filterBlacklistedFromSubpaths` and returns a copy: the input can be the store's cached response, which the old function mutated in place. - `totalPaths` and `pct` are unchanged, so percentages stay relative to every stored path, the filtered ones included. **Perf:** per request, one pass over the node cache (about 2k nodes on the deployment below) to collect blacklisted names, then at most 8 lookups per entry on at most `limit` entries. It runs outside the store lock and only when a filter is configured. `map[string]interface{}` count in `routes.go`: 3 removed, 3 added. ## Tests - `subpaths_blacklist_2184_test.go`: on the seeded path `["aa","bb"]` (first hop resolves to TestRepeater), for `subpaths`, `subpaths-bulk` and `subpath-detail`: - unfiltered, the repeater is listed (so the test is not vacuous); - a blacklisted key, also in capitals, removes it and the detail returns 404; - a hidden name prefix removes it; - a different key with the same 1-byte prefix leaves it listed; - clearing the blacklist afterwards shows it again (the cached response was not changed). - subpath-detail: a blacklisted node without a name returns 404, a different node that only shares its name does not hide the detail. - Red commits first (`0ebee485`, `4acfb70a`, failing on assertions), then green. - `cd cmd/server && go test ./...` passes (`TestSaveGeoFilterPreservesFileMode` skipped, a Windows-only failure on master too); `go test -race -run 2184` passes; `go vet` clean. ## Validation on a live-fed staging instance `nodeBlacklist` temporarily set to the key of the node in the top Route Patterns entry (5848 occurrences), then restored: | | blacklist set | restored | |---|---|---| | `subpaths?limit=200` entries showing the node | 0 (185 entries left) | 15 (200 entries) | | `subpaths-bulk` groups 2-2 / 3-3 / 5-8 showing it | 0 / 0 / 0 | | | `subpath-detail` on a path through it | 404 | | | `subpath-detail` on a path without it | 200 | | ## Not done - Other analytics responses that carry node names (for example topology) were not audited for the same gap. - The UI still shows `pct` relative to all paths; whether a filtered list should renormalise is left open. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
45680a22cf |
fix(#2191): apply the channel filter to live WebSocket packets (#2192)
Fixes #2191. ## Problem On the Packets page the channel filter only applies to the initial load (`/api/packets?channel=` → server-side `transmissions.channel_hash` match). The WebSocket handler re-checks time window, type, observer, region and node, but not `filters.channel`, and the channel filter is not re-applied at render time. With a channel selected, every new live packet is appended regardless of its channel. ## Fix - New pure helper `packetMatchesChannel(p, channel)` in `public/packets.js`, exported as `window._packetsMatchesChannelForTest` (same pattern as `_packetsBuildChannelOptionsForTest`). - It derives the value the server filters on from `decoded_json`, because WS packets don't carry `channel_hash`: - `channel` when the server decrypted the `GRP_TXT`, - otherwise `'enc_' + channelHashHex.toUpperCase()`. - Only `payload_type === 5` can match, mirroring the server (the column is only set for `GRP_TXT`). If a packet ever does carry `channel_hash`, that wins. - The WS filter calls it: `if (filters.channel && !packetMatchesChannel(p, filters.channel)) return false;` Checked the derivation against a production DB (live.saarmesh.de): it equals `channel_hash` for all 3,193 `GRP_TXT` rows of the last two days, 0 mismatches. ## Performance WS hot path, but the new check runs only while a channel filter is active, and for non-`GRP_TXT` packets it returns before any parsing. For `GRP_TXT` it is one `JSON.parse` of the already-received `decoded_json` string, O(1) per packet. WS messages are already debounced/coalesced by `debouncedOnWS`, so at typical rates (a few packets per second) this is negligible. No new state, no allocations beyond the parse, no API calls. ## Tests New `tests/unit/test-packets-live-channel-filter.js` (registered in `test-all.sh`), 10 cases: - no filter accepts everything - decrypted name match / mismatch - non-`GRP_TXT` rejected while a channel filter is active - `enc_<HH>` match incl. hex case normalisation and `enc_00` - `channel_hash` passthrough takes precedence - already-parsed `decoded_json` object - malformed / missing `decoded_json` doesn't throw and doesn't match - source check that the WS filter block actually applies the helper Without the `packets.js` change the new test fails; with it all pass. Existing `test-packets.js` (117), `test-packets-local-channels.js` (14), `test-packet-filter.js` (99) and `test-frontend-helpers.js` still pass. ## Browser validation The same change is deployed on live.saarmesh.de (hot-patched `packets.js`, server restarted so `__BUST__` refreshed; the served file contains the fix). I have not taken a screenshot of the filtered live table myself, so treat the in-browser check as pending until a reviewer or operator confirms. No config or customizer implications. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HJcbiRjQC5cSmydVFXbPbR Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
60af9bd469 |
fix(hops): narrow only the directly heard hop to the observer region (#2098) (#2188)
Fixes #2098. ## Problem `HopResolver.resolve()` narrowed every hop of a path to the observer's IATA region (300 km around the code's coordinates). Only the last hop was heard by the observer directly; every earlier hop can be anywhere the mesh reaches, so the region says nothing about it. This stayed invisible on deployments whose observer codes were missing from `/api/iata-coords`: there `nodeInRegion()` returns `null` and nothing is filtered. A Belgian deployment's 42 observers use 10 codes and none was in the table. Filling the table alone would have switched on a filter that is wrong for every hop but the last. ## Why only the last hop #2098 lists three options and argues against simply repopulating the table, because the filter it would switch on is not sound. This PR does fill the table, but only after limiting the filter to the one hop where it is sound: - In a MeshCore path the last hop is the repeater the observer received the packet from, so it is within radio range of the observer. Measured below: p99 30 km for directly heard multi-byte hops. - An earlier hop says nothing about the observer's position. On the Belgian deployment nearly every observer hears nearly every flood, so a region filter on earlier hops would drop the right candidate whenever the packet started outside the observer's region. This was decided while preparing the PR, not on the issue thread, so the reasoning is recorded here. ## Change - **`public/hop-resolver.js`:** `resolve()` takes an optional `opts.directHops`, the hops the observer heard directly. Without it, the input is one path and its last hop is the direct one. The region check runs only on direct hops. An earlier hop gets `regional: false` and, with an observer, `regionSkipped: true`. `globalFallback` now means checked against the region with no candidate in it, so it is only set on a direct hop. - **`public/hop-display.js`:** the badge counts every candidate for `globalFallback` or `regionSkipped`. The dashed underline and the popover's "(global fallback)" stay for `globalFallback` only, so an earlier hop is not labelled as matched outside the region when no region check ran on it. - **`public/packets.js`:** `resolveHopsForPackets()` resolves the last hop of each path apart from the earlier hops. `hopCacheKey(h, observerId, direct)` keeps the two in separate cache entries, because the same prefix can be the last hop in one packet and a relayed hop in another. `renderPath()` reads the direct entry for the path's last hop, and the detail-pane resolve writes it. The region-narrowed entry never goes to the bare (observer-less) key, in the list or in the detail pane. The detail pane resolves and caches the shown observation's path with that observation's observer (`effectivePkt.observer_id`), the observer `renderPath` already used; before, it resolved with the packet row's observer, so when the first observation came from another observer the last hop missed its direct entry and showed unfiltered. The route button uses the same observer. - **`cmd/server/routes.go`:** `iataCoords` gains 104 entries from OurAirports `airports.csv` (public domain): every IATA code in Belgium, the Netherlands and Luxembourg, and the airports with scheduled service in Germany and France. AMS, CDG, FRA and MUC were already present and keep their values. ## Measured before building On 5000 packets from the Belgian deployment (3661 with a path): - 1-byte last hops: 1212, all ambiguous, 1204 of them without a server `resolved_path`, so they go through the client resolver. - Distance from the observer to a directly heard node, on 1892 unique multi-byte last hops: p95 25 km, p99 30 km, max 82 km. - With this change, on 906 1-byte last hops from observers whose code is now in the table: the badge counts 6.1 candidates on average instead of 6.6. The displayed name changes for 52, all from observers without their own lat/lon, where distance to the region now orders the candidates instead of list order. For observers with a position, no displayed name changes, because the #2097 observer anchor already picked the nearest candidate. ## Tests - `tests/unit/test-issue-2098-last-hop-region.js` (new, in `test-all.sh`, 25 assertions): - HopResolver: the last hop of a path is narrowed and an earlier hop is not; an earlier hop keeps every candidate and reports `regionSkipped`, not `globalFallback`; a direct hop with no candidate in region reports `globalFallback`; `directHops` overrides the position; a one-hop path is narrowed. - HopDisplay (real file): an earlier hop keeps a badge with every candidate but no fallback class or popover text; a direct hop outside the region keeps both. - packets.js (real file, real resolver and display): with one observer, a prefix that is the last hop of one packet and an earlier hop of another renders narrowed in the first and with a badge in the second; after the detail pane resolves a path, the observer-less render does not show the narrowed last hop; when the shown observation comes from another observer than the packet row, its last hop is narrowed to the region of the observer that heard it. - `TestIataCoordsIncludesBelgianObserverCodes`: ANR, BRU, GNE, KJK, LGG, OBL, OST and MST with their OurAirports coordinates. - Red commits first (`26000a22`, `bb91ff9d`, `189f55d7`, failing on assertions), then green. Mutation check on the packets.js tests: inverting `isDirect` fails 3 assertions, swapping the `directHops` argument fails 4. - `test-issue-2097-hop-ambiguity-badge.js`: two structural regexes now allow the new `direct` argument; every assertion still holds. - All 198 `node tests/unit/*` suites in `test-all.sh` pass. `cd cmd/server && go test ./...` passes (with `TestSaveGeoFilterPreservesFileMode` skipped, a Windows-only failure on master too); `go vet` clean. ## Browser validation Staging (this branch on top of the deployment's current build), headless Chrome on `#/packets/<hash>` for packets whose path ends in a 1-byte hop: the last hop's popover lists only candidates within the region with their distances (for example `7E`: 3 candidates at 46, 207 and 296 km), earlier hops list every candidate (`1D`: 13). The same prefix `7E` showed 3 in region as the last hop of one path and 4 as a relayed hop of another. No page errors. `/api/iata-coords` returns 158 codes. ## Browser validation of the follow-up Staging again (this branch at `4e129b49` on top of the deployment's build), headless Chrome: - Packets list, 41 paths of 2+ hops: the global-fallback class appears on 2 last hops and on 0 earlier hops. - Detail pane, 5 packets whose path ends in a 1-byte hop (1 to 13 hops shown): no earlier hop carries the class or "(global fallback)" in its popover, and every ambiguous earlier hop keeps its badge. Every last hop was checked against the region: 4 show "N candidates in region", 1 shows "(global fallback)" with the class. - Before the observer fix, 2 of 5 detail panes showed a last hop with neither label, because the shown observation came from another observer than the packet row (`b66a8f4c`: row from one observer with 13 hops, shown observation from another with 9). - No page errors. ## Not done - The 300 km radius is unchanged. The measured direct-hearing distances suggest it is wide, but a smaller radius would also have to hold for less dense networks, and I have no data for those. - `live.js` passes no observer, so the region filter does not run there, before or after this change. `map.js` passes the observer of the chosen path in path order, so once the resolver knows the observer regions it narrows that path's last hop only. - The "Cross-observer aggregate" line in the detail pane renders the longest path with the packet's own observer, so its last hop is narrowed to that observer's region even if another observer reported that path. The per-observer keying there predates this PR (#2097). - Two of the Belgian deployment's codes, HEP and NRW, are not airport codes and are not in OurAirports; those observers stay without a region filter. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
38ed9d6ce6 |
perf(store): count path hops without decoding the JSON (#2187)
## Problem
`pathLen` is how `pickBestObservation` chooses each transmission's best
(longest-path) observation, so it runs for every observation on every
load and ingest, and again from the resolved-path index. It decoded the
path JSON into a `[]interface{}` only to take its length.
## Change
- `countStringArray` counts the usual array of plain hex strings in
place, with no allocation (helpers `skipJSONSpace` and
`skipPlainJSONString`).
- Anything else falls back to the decoder exactly as before: escapes,
non-string elements, `null`, or malformed input. So the result is always
what decoding gives, including 0 for invalid JSON.
## Perf justification (AGENTS.md rule 0)
- **Complexity.** O(len(path_json)), a single pass with no allocation.
The old version was also linear but went through reflection and
allocated per element.
- **Scale.** It runs once per observation on every load and ingest (400k
observations in a busy regional network's 7-day window), plus each time
the best observation is re-picked.
Throwaway benchmark (not committed), current master vs this branch:
| `path_json` | master | this PR |
|---|---|---|
| `[]` | 215 ns, 32 B, 2 allocs | 4 ns, 0 allocs |
| 6 hops, 1-byte | 1825 ns, 488 B, 18 allocs | 27 ns, 0 allocs |
| 10 hops, 3-byte | 2815 ns, 936 B, 27 allocs | 72 ns, 0 allocs |
On a copy of that 7-day window:
| | master | this PR |
|---|---|---|
| allocated over startup (the whole server) | 5.41 GB | 5.10 GB |
| server CPU over the run | 175.6 s | 169.2 s |
`pathLen` accounted for about 275 MB of the startup allocations.
## Tests
- `TestPathLenDoesNotAllocate`: zero allocations for typical paths. Red
commit first (2–11 allocations), then green.
- `TestPathLenMatchesDecoding` compares the result with `json.Unmarshal`
into `[]interface{}` on 27 edge cases and 20,000 random inputs built
from JSON punctuation, escapes, `null`, objects and numbers.
- `cd cmd/server && go test ./...` passes.
- SonarQube: quality gate passed, no issues on added lines.
Backend only: no UI, API shape or config change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Alex BandClaude Opus 5.5
|
a48dc32b8a |
perf(store): build hop context without a map or lowered copies (#2186)
## Problem `buildHopContextPubkeys` (#1197) runs for every transmission in each topology and distance pass, inside `buildAggregateHopContextPubkeys` for every subpath query, and for each matched packet in subpath detail. On every call it: - made a de-dup map sized for 16 keys, to de-duplicate what's usually 2–5 keys; - lowered every hop into a new string just to look it up in the prefix map. Hops are upper-case hex on live data, so `strings.ToLower` always allocates; - lowered the observer ID twice, once to check it's hex and once to add it. ## Change - Duplicates are found by scanning the result slice. - Hops are lowered into a 64-byte stack buffer for the prefix-map lookup. `m[string(buf)]` doesn't allocate, and non-ASCII input falls back to `strings.ToLower`, so lookups are unchanged. - The observer ID is checked as hex in either case (`isHexAnyCase`) and lowered once. - The result is the same keys in the same order: sender, observer, then unambiguous hop anchors. ## Perf justification (AGENTS.md rule 0) - **Complexity.** The de-dup scan is O(k²) for the k keys of one transmission. k is at most 2 + path hops; MeshCore paths are under 64 hops and real values are 2–10. It replaces a map allocated on every call, and nothing grows with the number of transmissions or nodes. - **Scale.** It runs for every stored transmission in each analytics pass (topology and distance) and per request in subpath analytics, so the saving recurs on every recompute, not just at startup. Benchmarks, current master vs this branch, same machine: | | master | this PR | |---|---|---| | `BenchmarkBuildHopContextPubkeys` | 830 ns, 1192 B, 4 allocs | 507 ns, 128 B, 1 alloc | | `BenchmarkBuildAggregateHopContextPubkeys` | 5.6 ms, 6.65 MB, 20.4k allocs | 3.7 ms, 1.33 MB, 5.3k allocs | On a copy of a busy regional network's 7-day window (64k transmissions): | | master | this PR | |---|---|---| | this function's allocations over startup and the first analytics pass | 317 MB | 80 MB | | server CPU over the same run | 175.6 s | 165.6 s | ## Tests - `TestBuildHopContextPubkeysAllocatesOnlyItsResult` uses a fixture cased like live data (upper-case hops and observer ID): at most 2 allocations, the result and the lowered observer ID. Red commit first (9 allocations), then green. - `TestBuildHopContextPubkeysOrderAndDedup` pins the same keys in the same order. - The existing hop-context and disambiguation tests (#1198, #1202) pass. - `cd cmd/server && go test ./...` passes. - SonarQube: quality gate passed, no issues on added lines. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
8d39a9d465 |
perf(store): count subpaths per request instead of indexing every run (#2182)
## Problem `spTxIndex` kept every 2–8 hop run of every stored path as a map key, with one transmission pointer per occurrence. On a busy regional network's 7-day window (59k transmissions) that's about 0.9–1.07 M keys and 130–160 MB of live heap, roughly a quarter of the store. It was also rebuilt at the end of every startup, which is when memory peaks. Two smaller issues came with it: - Eviction never decremented `spTotalPaths`, so the percentages were divided by paths that were no longer stored. - Route Patterns took 1.2–3.6 s uncached, because the fast path still split, resolved and re-joined about a million raw keys per request. ## Change - **Count on request.** Subpaths are counted when the analytics are asked for, from the stored paths. Results stay cached for `rfCacheTTL`, as before. - **Pointer-free tallies.** `tallyNamedSubpaths` counts runs in one pointer-free `map[uint64]subpathTally`, keyed by a `maphash` of the resolved name built in a reused buffer. The GC doesn't scan the map, and counting allocates nothing but the map. - **Names only for the top entries.** Each tally records the first transmission and offset where its run appeared, so names and raw hops are built only for each group's top entries. `topSubpathTallies` picks those with a `container/heap` min-heap of `limit` entries instead of sorting every name, and the existing `rankSubpaths` formats the response, so no new response maps are added. - **Hash collisions.** If two names shared a hash their counts would merge; the odds are about 1 in 10⁷ for a few million names. - **Subpath detail.** `GetSubpathDetail` finds the run with `txsWithSubpath`, a scan of the stored paths. - **Counts unchanged:** runs of 2–8 hops, counted once per occurrence, case-folded. The tests pin this. - **Totals.** `spTotalPaths` stays, and eviction now decrements it. - **Memory estimate.** The O(path²) subpath term is dropped from the estimate. ## Perf justification (AGENTS.md rule 0) - **Complexity.** A cold Route Patterns request is O(Σ runs), about 1.5 M for this window: one pass over the stored paths, with one map lookup per run against a reused buffer. Ranking is O(tallies · log limit). Results are cached for `rfCacheTTL`, so this runs at most once per TTL per group set. - **Subpath detail** is O(stored hops), one pass with no allocation per packet. It's an on-click endpoint, not a hot path. - **Ingest and eviction** no longer touch a subpath index at all. Removal used to be a linear search of every key's transmission list. **Measured on a copy of that window**, upstream master vs this branch, same queries: | | master | this PR | |---|---|---| | peak RSS (`GOMEMLIMIT=750MiB`) | 948 MB | 767 MB | | in-use heap after load (750MiB) | 592 MB | 456 MB | | `/api/analytics/subpaths` cold (750MiB) | 3.56 s | 0.64 s | | `/api/analytics/subpaths-bulk` cold (750MiB) | 3.33 s | 0.70 s | | `/api/analytics/subpaths` cold (1GiB) | 1.18 s | 0.61 s | | `/api/analytics/subpath-detail` | 0.6 ms | 8 ms | | cached responses | 0.6 ms | 0.5 ms | At 750 MiB, master's request time is mostly GC pressure from the index. On the production-shaped fixture that `subpath_index_memory_test.go` builds (41k transmissions): | | master | this PR | |---|---|---| | retained by the store for subpaths | 197 MB | 0 MB | | allocated per bulk request | 336 MB | 111 MB | This has also run on a staging instance with live MQTT ingest for about nine hours, with Route Patterns answering in about 1 s. ## Tests - `subpaths_on_demand_test.go`: - per-occurrence, case-folded counts; - the 8-hop cap; - bulk matches single; - detail counts, including repeats, case and the length limits; - eviction drops evicted paths from the totals. - `TestSubpathIndexRetainedHeap`: the bound is now 5 MB (it was 225 MB). - `TestSubpathRequestAllocations`: 150 MB per request. - Each change has its red commit first, failing on assertions. - Tests that reached into `spTxIndex` now assert through `txsWithSubpath` and the public counts. - `cd cmd/server && go test ./...` passes. - SonarQube: quality gate passed, no issues on added lines. **Overlap with #2181:** both edit the same memory-estimate comment block in `store.go`. Whichever lands second needs a trivial rebase there; there's no code dependency. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
ca62790c81 |
fix(store): load the observer IATA in the background chunk loader (#2185)
## Problem With `hotStartupHours` set, startup loads the hot window first. `loadChunk` then fills the rest of the retention window in the background. Its query selected the observer ID and name but not the IATA code, and scanned no IATA, so every packet it loaded had an empty `ObserverIATA`. On a 7-day window with a 24 h hot start, that's six days of packets after every restart that: - have no `observer_iata` in the packets API; - never match the `iata` filter (#1188). `Load`, `scanAndMergeChunk` (the hot window) and both ingest paths already join `observers` to get the IATA code, so only background-loaded packets were affected. **Seen on a live instance** (`hotStartupHours: 24`, restarted about 18 h earlier), sampling `/api/packets?until=…`: | packets | with `observer_iata` | |---|---| | 20–30 h old (hot window) | 200 / 200 | | 46 h old (background-loaded) | 0 / 200 | | 60 h old (background-loaded) | 0 / 200 | With this fix on a staging instance, packets older than its hot window are 300 / 300. ## Change The chunk query selects `COALESCE(obs.iata, '')` for both schemas: the v3 query through the observers join it already had, and the legacy query through a new join on the observer ID, as `Load` does. `loadChunk` scans the column into `ObserverIATA`. ## Tests - `TestLoadChunkKeepsObserverIATA`: red commit first (fails on its assertion), then green. - `cd cmd/server && go test ./...` passes. - SonarQube: quality gate passed, no issues on added lines. **Overlap with #2183:** both edit the same lines of `loadChunk`'s `StoreObs` literal (#2183 interns the observer fields). Whichever lands second needs a trivial rebase; if this lands first, #2183 should intern `ObserverIATA` there too. There's no code dependency. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
dc7ebcd6f5 |
perf(store): check observation duplicates without per-transmission maps (#2181)
## Problem Every `StoreTx` carried two maps for observation de-duplication: - `obsKeys`, keyed by `observerID + "|" + pathJSON` (a concatenated string allocated for every observation); - `observerSet`, for `UniqueObserverCount`. On a busy regional network's 7-day window (64k transmissions, 400k observations), these maps and their concatenated keys held about 60 MB of heap. Most transmissions are heard a handful of times; the busiest in that week was heard 40 times. The maps cost a few hundred bytes per transmission to de-duplicate lists that are usually under ten long. ## Change - `hasObservation` and `hasObserver` scan `tx.Observations`, comparing the observer ID and path strings the observations already hold. These are the same two values the old map key was built from, so the answers are identical. - Once a transmission passes 64 observations (`obsIndexThreshold`), it builds the maps. They're keyed by a struct that shares the observations' strings, so no key bytes are allocated, and busy transmissions keep O(1) checks (the case #543 / #355 addressed). - The five places that appended an observation now call `addObservation`, which keeps `UniqueObserverCount` and `ObservationCount` in step. Those three lines were previously copied at each site. - The memory estimate no longer charges for the per-transmission maps or the concatenated keys. ## Perf justification (AGENTS.md rule 0) - **Complexity.** - Below the threshold, a duplicate check is O(k) for a transmission with k < 64 observations, so building it costs at most 63·64/2 = 2016 string-header comparisons. - At and above 64 observations, checks are O(1) map lookups, as before. - Nothing is O(n²) in the number of transmissions or observations overall: the scan is bounded per transmission. - **Scale.** Real data peaks at 40 observations per transmission. The scan compares string headers (length first, then bytes), and 64-character observer IDs differ early. `BenchmarkObsDedup` (a miss, the common case on ingest): | observations on tx | check used | time | |---|---|---| | 4 | scan | 15 ns | | 16 | scan | 70 ns | | 40 | scan | 151 ns | | 63 | scan | 233 ns | | 64+ | map (struct key) | 15 ns | For comparison, the old map lookup was about 15 ns too, plus building and allocating the `observerID|path` key string on every check. At 40 observations that's about 3 µs extra to build a whole transmission, against roughly 2 KB of maps no longer allocated for it. **Measured on a copy of that 7-day window** (`GOMEMLIMIT=750MiB`, same queries): | | before | after | |---|---|---| | in-use heap after load | 612 MB | 568 MB | | HeapAlloc after load | 660 MB | 564 MB | | peak RSS | 1077 MB | 1009 MB | | startup to ready | 31 s | 21 s | It has also run on a staging instance (live MQTT ingest) for about nine hours alongside other store changes, with no change in observation or observer counts. ## Tests - `obs_dedup_test.go`: - duplicates (same observer and path) are rejected, while a new path or a new observer is kept; - a transmission under the threshold carries no maps; - across the threshold the maps take over with the same answers; - empty observer IDs aren't counted as observers; - `BenchmarkObsDedup`. - Red commit first (stubs, failing on assertions), then green. - `cd cmd/server && go test ./...` passes. - SonarQube: quality gate passed, no issues on added lines. Backend only: no UI, API shape or config change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
955386b63f |
test(map): settle the router's focus frame before the teardown snapshot (#2179)
## Summary Fixes an intermittent failure in `tests/e2e/test-map-nodes-pagination-e2e.js`, the "late response after map teardown" cases added for #2030 in #2048: ``` ✗ late /observers response after map teardown: old response must not change the destination page ``` It failed on #2164 ([run 38080812509](https://github.com/Kpa-clawbot/CoreScope/actions/runs/38080812509), shard 2/3), a Go-only change, and passed on the rerun. ## Cause `checkMapTeardown` compares `#app`'s HTML before and after releasing the old map's pending response. When it failed, the only difference was on the destination page's heading: ``` before: <div class="tools-landing"><h2>Tools</h2>… after: <div class="tools-landing"><h2 tabindex="-1">Tools</h2>… ``` That attribute comes from the router, not the old response. Since #630 (`app.js`, "SPA focus management") it focuses the new page's first heading in a `requestAnimationFrame` after the page renders. The before-snapshot was taken as soon as `#leaflet-map` detached, sometimes ahead of that frame. ## Fix The test now waits two frames before the before-snapshot, the same settle it already does before the after-snapshot. The router's frame is queued by then, so it always lands first. The change is test-only: no app code changes, and every assertion stays as it was. ## Tests Run locally with the e2e fixture (`tools/freshen-fixture.sh`, migrated) served by this branch's server, headless Chromium: | | runs | failed | |---|---|---| | test on `master` (`352a0ccd`) | 20 | 2 | | this branch | 50 | 0 | - **Before:** the two failures on `master` were "old response must not change the destination page" (teardown, `/config/regions` and `nodes`), plus one 30 s timeout in an "after map replacement" case in the same runs. - **After:** none of these failures or the timeout came back in 50 runs. ## Checks - **Lint:** ESLint passes with the repo config and with Node globals. SonarQube rules on the file report nothing on the added lines; its four existing minor findings are on lines this PR doesn't touch. - **Repo rules:** one logical change, explicit `git add`, no private data, no new dependencies. This is a test-only fix, so there's no red/green pair: the "red" is the intermittent failure above, measured as 2 in 20. Related: #2030, #2048 (where the test was added). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
CoderNemesisandCoderNemesis
|
d9342bbebd |
(feat): UserManagement, Add Support for Postal Email Delivery Provider (#2154)
This PR extends email support to add another email delivery provider called Postal for user management. It is a, open source, self hosted email delivery and reputation management service. It adds 2 new additional configuration settings to the userManagement.mail object which are "postalBaseUrl" and "postalApiKey". It also adds "postal" as a supported value for the "provider" setting. All existing functionality including email delivery notification is supported by Postal. Included test cases to support the new functionality and all pipelines pass. --------- Co-authored-by: CoderNemesis <rfedor@comchan.net> |
||
|
Alex BandClaude Opus 5.5
|
a4ee3a7328 |
feat(scope-audit): observer region filter for both views (#2171)
## Summary Builds on the transport view from #2167 and #2169 (both merged); rebased onto `master` (`352a0ccd`), so the diff is this PR's six commits only. The Scope Audit was the one analysis page without the region selector. This adds the shared region filter to it, for both the declared and the transport view. `?region=SFO,SJC` means what it means on every other endpoint: count only forwarding **heard by observers in those regions**, not repeaters located there. - **API** (`GET /api/scope-audit`, both modes): - The region codes are resolved to their observers first (a handful of rows), then the audit's hop scan gets `AND o.observer_idx IN (…)` on v3, or `o.observer_id IN (…)` on v2. The 3.5M-row hop scan never joins `observers`. - `All` means no filter (shared `normalizeRegionCodes`). Codes with no observer match nothing, instead of silently widening to every observer. - Responses echo the normalised codes as `region`. The declared and transport caches and their singleflight keys now include the region, so each region set is cached separately with the existing TTLs. - Declared-region verification (`regionEvidence`) is deliberately left unfiltered: it corroborates what a repeater declares from its own traffic, which doesn't depend on where a hop was heard. - **Page:** the shared `RegionFilter` sits with the view and window controls; a change reloads the view, and the selection persists like every other page. The transport view's text box is relabelled **Scope** (it already sends `?scope=` since the rename in #2167), so the page no longer has two controls called Region. - **Docs:** `docs/api-spec.md` (parameter, both response shapes) and the OpenAPI parameters for the route (`mode`, `scope` and `region` were all missing). Partial fix for #2142 (its region filter for rollout tracking), together with #2167 and #2169. ## Performance The unfiltered path must not get slower: it's the same query with an empty clause. `BenchmarkComputeScopeTransport` (from #2167, 1,200 repeaters), the PR it stacks on vs this branch, `-count 8`: | | before | after | | |---|---|---|---| | sec/op | 220.3 ms | 218.0 ms | ~ (p=0.065) | | B/op | 43.77 MiB | 43.70 MiB | ~ | | allocs/op | 833.5k | 833.5k | ~ | **Filtered requests, measured on staging** (a copy of a production database): the first version drove `o.observer_idx IN (…)` from `idx_observations_observer_idx`, reading every observation the region's observers ever made. Fixed in `80414242` with a unary `+` (the same treatment as #2166), so the scan starts from the `first_seen` window and filters per row. Red test `67e063b1` asserts the query plan against a fixture with the real indexes. | scan, one region over 24h (13 observers) | plan | time | hop rows | |---|---|---|---| | `o.observer_idx IN (…)` | observer index | 13.44 s | 102,114 | | `+o.observer_idx IN (…)` | `first_seen` window | **0.25 s** | 102,114 | | unfiltered | `first_seen` window | 0.35 s | 225,983 | Cold end-to-end requests on staging after the fix: 24h declared, unfiltered 0.84 s · one region 0.48 s (was 16.7 s) · two regions 0.22 s · transport, one region 0.42 s · transport, six regions 0.75 s · 7d, one region 3.46 s. 24h transport, live data: 383 repeaters unfiltered, 204 heard by one country's observers. ## Tests - Red `629e4f36` (Go): extends the audit fixture with an `observers` table and `observations.observer_idx`. Covers the declared view ignoring a scope heard only by another region's observer, the transport view listing only repeaters heard in the region, `region=All`, per-region caching, a region with no observers, and the echoed codes. Five fail on assertions; `region=All` passes because the stub ignores the filter, which is the behaviour it pins. - Green `6f1b1e1c`: the implementation, plus a v2-schema case (`observer_id`), so both branches of the clause are covered. - Red `b4933d68` / green `08ecc172` (frontend): `apiPath` carries the selection in both views and adds nothing when none is selected. The region control on the page is a net-new UI surface. - `go test` (cmd/server) passes. Frontend suites: 193 of 194. The one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. ## Screenshot The same transport view with the shared region filter set to **SJC**: only forwarding heard by San Jose observers counts, so 7 repeaters are listed instead of 44, with their carried scopes counted from those observers alone. Taken against `test-fixtures/e2e-fixture.db` with fictional scopes.  ## Browser validation The branch's server was run locally with 14 configured regions (dropdown layout), checked in headless Chromium at 1280, 900, 700 and 390 px (touch): | check | 1280 | 900 | 700 | 390 touch | |---|---|---|---|---| | ticking SJC re-requests with `®ion=SJC` | yes | yes | yes | yes | | menu fully on screen | yes | yes | yes | yes | | page wider than viewport | no | no | no | no | | page errors | none | none | none | none | Found and fixed this way: the filter is the right-most control on desktop, and the menu (which opens rightwards) ran 66 px off the page. It now opens leftwards above 640 px. On a phone the controls wrap, the filter starts a line, and the default applies. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **S3776 on `ScopeAuditForwardingHeardIn`:** complexity 55. The function it generalises, `ScopeAuditForwarding`, is 54 on `master`; the region clause adds one point. - **S3776 on `computeScopeAudit`:** complexity 65, unchanged from `master`. - Both pre-date this PR. Splitting them is better as a refactor PR of its own than mixed into a feature. - The page code's findings were fixed in the transport-view PR it stacks on. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
55dd28164d |
test(e2e): assert the configured-scope row after the theme-refresh re-render (#2180)
## What `test-issue-1648-m2-icons-e2e.js` now asserts the configured-scope row only after the page's `theme-refresh` re-render has fired and the detail has rendered again. ## Why On 2026-10-10 this assertion failed CI on four PRs (#2169, #2171, #2154, #2164), each time at the first live-map case: `confirmed scope row must be visible, including an empty scope`. A re-run of the failed jobs passed every time. The code path: - `customize-v2.js:814` dispatches `theme-changed` once `/api/config/theme` resolves, on every page load. - `app.js:1309-1315` turns that into `theme-refresh` 300 ms later. - `live.js:4713` re-runs `showNodeDetail(activeNodeDetailKey)` on `theme-refresh`, and `showNodeDetail` first sets `#nodeDetailContent` to "Loading…" (`live.js:2598`). This is the only caller besides the marker click (`live.js:3090`). - The test waited for `#nodeDetailContent table` and then called `row.isVisible()`, which does not wait. A refresh landing between the two fails the check. The same race could let the null-scope case pass on the "Loading…" panel, because `row.count() === 0` holds there too. The fix uses the same `addInitScript` flag pattern as `test-e2e-playwright.js:83`. ## Testing - Locally against a server on the E2E fixture: 5 of 5 runs pass, 27 of 27 checks each. - **Not reproduced locally.** The unchanged test also passed every local run: 5 plain, 3 with `/api/config/theme` delayed by 1 s, 5 with 8x CPU throttling. So the reasoning above comes from the code path and the four CI failures, not from a local red-then-green. - Test-only change, no product code touched. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
4e91c5cf0b |
perf(analytics): opt-in pauseWhenIdle skips recomputes nobody reads (#2164)
## Summary The analytics recomputers (#1240) recompute eleven snapshots (topology, RF, distance, channels, hash collisions, hash sizes, roles, two clock-skew views, retransmissions, direct-heard) on a fixed interval, 300 s by default, whether or not anyone reads them. That is the right trade for a busy instance: reads always hit a warm cache. On a quiet one it means most of the server's allocation goes on results nobody looks at. Measured on one production instance (2 vCPU / 2 GB): - topology, RF and hash-sizes were each **read 23 times in 45 hours** while being recomputed every 5 minutes; - `analyticsRecomputer.runOnce` accounted for **30% of all bytes allocated** (121 GB of 391 GB in 45 h); - garbage collection was 87% of the process's CPU (30 s CPU profile: `gcDrain` 39.9 s of 45.9 s sampled), with an 805 MB live heap against a 1 GiB `GOMEMLIMIT`, so allocation rate translates directly into collector time. This PR adds an **opt-in** `analytics.pauseWhenIdle`: - a recomputer's tick is skipped when nothing has read its snapshot since the previous compute, and the recomputer is marked paused; - the next read still returns immediately with the snapshot it holds (`Load` stays non-blocking, as #1240 requires), and makes a non-blocking send that wakes the loop to recompute; reads after that get the fresh snapshot and the ticker resumes from there. **Default is off**, so behaviour is unchanged unless an operator turns it on. The trade-off for operators who do: the first read after a quiet spell can show a snapshot older than the interval (refreshed seconds later). It is documented in `config.example.json`. No existing issue covers this. ## Config example Off by default. To turn it on, add one key to the existing `analytics` block in `config.json`: ```json "analytics": { "defaultIntervalSeconds": 300, "pauseWhenIdle": true } ``` Nothing else changes: per-endpoint `recomputeIntervalSeconds` still apply while an endpoint is being read. There is no UI change, so no screenshot. ## Performance Staging A/B: the same 30 minutes of real production API traffic (5,640 requests from a day's nginx log, replayed at 2x against a copy of the production database with the live MQTT feed), on `master` and on this branch with `pauseWhenIdle: true`, each after a restart and warm-up. Allocation is the delta of `/debug/pprof/allocs` across the replay: | during the replay | master | this PR | |---|---|---| | allocated by `analyticsRecomputer.runOnce` | 459.7 MB | **53.4 MB (-88%)** | | total allocated by the process | 3,704 MB | 3,372 MB (-9%) | | GC cycles | 52 | 41 | | requests: p50 / p95 / errors | 9.9 ms / 57.0 ms / 0 | 10.4 ms / 60.1 ms / 0 | The replayed traffic included real analytics reads, so the paused recomputers were woken by them; the remaining 53 MB is that. Over 15 minutes each recomputer only ticks about three times, so on a longer quiet period the saving grows with the idle time. GC count and CPU on this shared host vary between runs on their own, so the allocation figures are the claim, not the GC count. There is no Go benchmark for this change: the behaviour is "how many computes happen in a period of no reads", which the tests below assert directly (16 computes on master vs at most 2 here over the same idle period). ## Tests - Red commit `24056c57`: config field, accessor stub and an unused recomputer flag, plus tests. Three fail on the stub: - `TestAnalyticsRecomputer_PauseWhenIdleSkipsUnreadTicks`: an unread recomputer with a 20 ms interval computes at most twice in 300 ms (16 times on the stub); - `TestConfig_AnalyticsPauseWhenIdle`: `pauseWhenIdle: true` enables it, absent or nil config leaves it off; - `TestStartAnalyticsRecomputers_AppliesPauseWhenIdle`: all eleven recomputers receive the flag. - Also added, passing on both commits on purpose: a read on a paused recomputer returns the existing snapshot and triggers a refresh; a recomputer that is being read keeps ticking; the default (off) keeps ticking unread. - Green commit `19de3636`: the behaviour, `config.example.json` entry with `_comment_pauseWhenIdle`, and the `Load` doc comment. - `go test -race` on the recomputer, warm-up (#1659) and config tests, three runs: pass. `make test`: 16 of 16 modules pass. The frontend tests that read `config.example.json` pass. ## Review notes - `roles` reads the nodes-clock-skew snapshot inside its own compute, which counts as a read, so that dependency keeps its source fresh while roles is in use. - The #1659 warm-up gate is untouched: a skipped tick never marks a first pass done. - Config documented per WORKFLOW.md section 7. Customizer (AGENTS.md rule 8): it is an operator setting, not a display value; no customizer entry proposed. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - No issues on the added lines. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
352a0ccd19 |
feat(region-filter): configurable region quick picks (#2170)
## Summary **Independent of every other open PR.** Choosing a pick saves the selection, redraws the control and notifies the page's listeners itself, exactly as a click in the control does, so it re-queries straight away whether or not #2168 (`setSelected` notifies) is merged; with both in, nothing fires twice. Region quick picks are named groups of region (IATA) codes, offered as one-tap choices inside the region filter, e.g. "Northern California" for a dozen airport codes instead of ticking each box. One deployment has been injecting these into the page through an nginx `sub_filter` and a script that wraps `window.RegionFilter`; this makes them a configured feature. ```jsonc "regionQuickPicks": [ { "name": "Northern California", "description": "Bay Area and Sacramento observers.", "regions": ["SJC", "SFO", "OAK", "SMF"] } ] ``` - **Server:** `GET /api/config/region-quick-picks` returns the groups cleaned up: names and codes trimmed, codes upper-cased and de-duplicated in their configured order, groups without a name or codes dropped, at most 20 groups of 200 codes, always a list. `/api/config/regions` is unchanged. - **Region filter:** loads the groups with the regions and offers every group with at least one observer today, inside the region control: - dropdown layout: rows of the same kind as **All** (checkbox and bold name), directly under All with a divider before the single regions; the row is ticked while that pick is the selection, ticking it selects the pick and closes the menu, unticking goes back to all; - pill layout: the picks lead the same bar, set off from the single regions by a thin divider. - A tap selects the group's codes that have an observer and re-queries straight away. - The selector then reads the group's name ("South Bay ▾") instead of "2 Regions", and the button shows as pressed. Selecting the same codes by hand does the same. - Tapping the active group again goes back to all regions. - A failed fetch, or no configured groups, leaves the filter exactly as before. - **Mobile:** the rows take the menu's 44 px row height under `(pointer: coarse)`; the pill-bar picks reuse `.region-pill` and its touch target. Inside `.filter-bar` (Packets) the existing touch rule for inputs made every region checkbox a 44 px square; the checkbox is now 20 px and the 44 px row is the tap target. - **Semantics:** like every region filter, a group means "heard by observers in these regions", not "located there"; `config.example.json` says so. No existing issue covers this. ## Screenshot Nodes page with two picks configured (example config below), after tapping **South Bay**: the pick and its codes (SJC, MRY) are selected, and the list re-queried for them. Taken against `test-fixtures/e2e-fixture.db`.  ## Config example ```json "regionQuickPicks": [ { "name": "South Bay", "description": "San Jose and Monterey observers", "regions": ["SJC", "MRY"] }, { "name": "East and North Bay", "description": "Oakland and San Francisco observers", "regions": ["OAK", "SFO"] } ] ``` Each pick needs a `name` and at least one code; `description` is optional (shown as the tooltip). Codes are trimmed, upper-cased and de-duplicated; a pick with no code that has an observer today is not shown. At most 20 picks of 200 codes. ## Browser validation The branch's server was run locally against a migrated copy of `test-fixtures/e2e-fixture.db` (observers in MRY, OAK, SFO, SJC) with three groups configured: "South Bay" (SJC, MRY, SCZ), "East and North Bay" (OAK, SFO, sts) and "Nowhere yet" (XXX). Checked in headless Chromium: | check | desktop 1280 px | phone 390 px (touch) | |---|---|---| | groups shown | South Bay, East and North Bay ("Nowhere yet" hidden) | same | | tap South Bay selects | SJC, MRY (SCZ has no observer) | same | | next `/api/packets` request | `region=SJC,MRY` | `region=SJC,MRY` | | selector label | "South Bay ▾" | "South Bay ▾" | | second tap | back to all regions | same | | button height | 24 px | 44 px | | page wider than viewport | no | no | | page errors | none | none | Checked on Packets (dropdown layout) and Nodes (pill layout). Three layout problems found this way on the phone were fixed before opening: a sideways-scrolling row clipped at the screen edge, the selector squeezed into a narrow column (the filter group is a non-wrapping flex row), and the two wrapped rows spread apart on Nodes (`align-content`). **Revision 1:** the first version put the picks in a separate row beside the selector; they moved inside the dropdown. Re-checked with a 14-region list and three picks on Packets: desktop 1280 px dark and phone 390 px light both keep the menu on screen (menu 340 px wide, page not wider than the viewport), the pick closes the menu and the trigger names the pick. **Revision 2:** picks are checkbox rows like **All** instead of chips. Re-checked on Packets at 1280 px dark and 390 px light: rows render bold under All, the divider separates the single regions, ticking a pick ticks its codes and names the trigger, no sideways scroll. Unrelated and not changed: on Nodes at 390 px the region row (existing pills included) starts flush with the screen edge. ## Tests - Red commit `81ea67d6`: the config type, a stub normaliser and endpoint returning nothing, and tests. - Go: cleaning (trim, upper-case, de-duplicate, drop incomplete groups), caps, nil config, and the endpoint's JSON. - Frontend, `tests/e2e/test-region-quick-picks.js` (moved from `tests/unit/` in `fced69a7`; it needs jsdom, so it runs in the E2E job like `test-table-sort.js`): the real `region-filter.js` in **jsdom** against faked config endpoints. Nine cases: rendering and hiding groups with no observer, selecting and re-querying, the selector label, toggling back, a hand-made selection marked active, the pill layout, no groups, a failed fetch, HTML escaping. Seven fail on the stub, all on assertions. - Green commit `0e33210e`: the implementation, styles, `config.example.json` (`regionQuickPicks` + `_comment_regionQuickPicks`), `docs/api-spec.md`, and the OpenAPI route list (`TestOpenAPICompleteness` requires it). The jsdom test also had a cross-realm comparison bug fixed in this commit (`Array.from` on arrays coming out of the jsdom window). - `make test`: 16 of 16 modules pass. Frontend suites: 195 of 196; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. - Red commit `f991060d` (revision): picks live inside the dropdown menu before the checkboxes, choosing one closes the menu, and in the pill layout they share the region bar; two fail on assertions against the old layout. Green commit `1f720e35`. Frontend 195 of 196 (same environment-only failure). - Red `134f5256` / green `da0adb50` (second revision): picks are checkbox rows between All and the single regions with a divider, no chips in the menu; ticking and unticking a pick row. The green commit also repairs `style.css`: `1f720e35` had prepended its block to the top of the file (its edit matched an earlier `.region-pill {`) and left the previous rules in place. The file now differs from `master` by 13 lines. ## Review notes - Groups are offered only when at least one of their codes appears in `/api/config/regions`, and only those codes are selected. The stored selection is pruned to known codes on load anyway, so selecting absent codes would not survive a reload. - Customizer (AGENTS.md rule 8): groups are operator configuration; a customizer editor could come later. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `b84ace4e`: - `normalizeQuickPickCodes` split out of the config normaliser (S3776, 17 > 15); - a comment on why a failed quick-pick fetch is ignored (S2486); - `for…of` in `activePick` (S4138); - in the tests: `node:` specifiers, `includes()`, and an explicit string comparator for sorted selections (S7772, S7765, S2871). - **Kept on purpose:** - S2486 is still reported on that `catch`. Swallowing the error is the intended behaviour: the filter works as before, just without picks. - S1523 on the `vm` test loader, as above. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
99efaed33d |
fix(region-filter): setSelected notifies change listeners on a real change (#2168)
## Summary `RegionFilter.setSelected()` saved the selection and redrew the control but never ran the `onChange` listeners; only a click (`toggleRegion`) did. A programmatic selection therefore updated the dropdown while the page kept showing data for the old selection until the next click. Affected today: the Packets page applies a `?region=` URL parameter through `setSelected`. It registers its listener afterwards, so its first load is fine, but any other page's listener that is still registered sees nothing. And anything that offers one-click region groups has to wrap `setSelected` to make it re-query (an nginx-injected overlay on one deployment does exactly that). This PR stands alone; #2170 (configurable quick picks) does not need it. `setSelected` now compares the new selection with the current one (order-insensitive; empty and null both mean "all") and, only when they differ, runs the listeners after saving and re-rendering, exactly as a click does. Re-applying the current selection stays silent, so a caller restoring state does not trigger a reload. No existing issue covers this. ## Tests - Red commit `75569b78`: `tests/unit/test-region-filter-setselected.js` loads the real `public/region-filter.js` in a `vm` sandbox. Two cases fail on `master` (a programmatic selection notifies once with the new codes; a clear notifies once with `null`). Two pass on both commits on purpose: re-applying the current selection in another order, and clearing an empty selection, must not notify. - Green commit `8a7ddfe1`: the fix. - Listed in `test-all.sh` in sorted order; `test-test-inventory.js` passes. `test-1108-region-hide-nodes.js` (the other region-filter suite) passes. Frontend suites: 194 of 195; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. ## Review notes - `packets.js` "clear all filters" calls `setSelected([])` and then reloads itself. When a region was selected, that path now also triggers the page's region listener, so the packets list is requested twice. It is a cosmetic double fetch on an explicit user action; left as is to keep this PR to the component. - No performance claim; this is a correctness fix. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `9e732706`: `node:` specifiers (S7772). - **Kept on purpose:** S1523 on `vm.runInContext`, which loads the repo's own `public/region-filter.js` as the other frontend unit tests do. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
deb55dc82e |
feat(scope-audit): transport view on the Scope Audit page (#2142, page) (#2169)
> **Requires #2167 to be merged first.** This page calls the transport view API that #2167 adds (`/api/scope-audit?mode=transport`), so it cannot be split from it; until #2167 merges this PR also shows its commits. ## Summary Page half of #2142. **Stacked on #2167** (the API); review that first. #2171 adds the observer region filter on top. Partial fix for #2142 until all three are in. Adds a **Declared / Transport** toggle to `#/scope-audit`. The transport view calls `?mode=transport` and lists every repeater seen forwarding in the window: - **Seen carrying:** one chip per region scope with its packet count (first/last seen in the tooltip); - **Declared:** the repeater's declared regions coloured by whether they were seen carried (the declared view's own chips), or *not asked* when no declared-regions answer exists. Never asked is worded differently from declaring nothing; - **Other traffic:** unscoped, unmatched and ambiguous-hop counts. A **Region** box (suggestions from the scopes seen) adds a *Carries X* column, lists the repeaters not yet carrying it first, and summarises carriers against active non-carriers: the rollout report the issue asks for. Window, view and region are deep-linked (`#/scope-audit?window=7d&mode=transport&scope=be`) and survive a reload. The page says plainly that *transported* means seen carrying, not configured, and that over a short window a quiet region looks the same as a missing one. The intro text changes with the view. ## Screenshot The transport view with scope `be` typed in: every repeater seen forwarding in the last 24 h, whether it carries `be` (non-carriers first), the scopes it was seen carrying with counts, and its declared regions or *not asked*. Taken against `test-fixtures/e2e-fixture.db` with fictional `be` / `be-van` / `nl` scopes.  ## Browser validation (staging, real data) Headless Chromium against a staging copy of a production database with its live feed: - `#/scope-audit?window=24h`: the declared view, 16 rows, as before. - Clicking **Transport**: 385 rows; the hash gains `mode=transport`. - Typing a scope as `#XX-YY` in capitals: normalised to `xx-yy`; the hash gains it (now `scope=xx-yy`); summary "385 repeaters seen forwarding in the last 24h · 16 with a declared-regions answer · 51 carried xx-yy · 334 active but not seen carrying it"; the first rows are the non-carriers. - Reloading that hash restores view and region. Switching back to Declared hides the Region box and restores the declared intro. No page errors. - Checked at 1280 px and 390 px wide. Two layout bugs found this way were fixed before opening: the region box inherited `width: 100%` from `.nodes-search`, and `display: flex` overrode the `hidden` attribute on the region bar. ## Tests `tests/unit/test-frontend-helpers.js` gains 10 cases on rendered markup and state, using the page's exported internals like the existing scope-audit tests: hash and API paths (and that a region never leaks into the declared view's link), region normalisation (`#`, capitals, empty), *not asked* vs declared chips (with the `*` wildcard), carried-scope chips with counts and the filtered region marked, the carries column only with a region, HTML escaping of names, the summary counts, region suggestions, and the per-view intro. 768 passing in that file. Net-new UI surface, so per TDD.md the tests land in this PR rather than as a preceding red commit. Frontend suites on this branch: 193 of 194; the one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails identically on `master` locally (#2149 and #2160 address that). ## Review notes - Colours only through existing CSS variables (no hex literals); the filtered region is marked with the link colour, not a new status colour. - The window note's wording is in the same spirit as the declared view's `windowHonestyNote`. - No API calls per row: one request per view/window/region, cached client-side for 30 s like the declared view. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `df53e770`: - one `modeQuery` helper for the page hash and the API path (S3358; the two copies were identical); - the carries-scope cell built before the row (S3358); - `localeCompare` for the region suggestions (S2871), `for…of` and `.dataset` (S4138, S7761), optional chaining (S6582); - object spread in the helper tests (S6661). - Optional chaining and spread are already used throughout `public/`. No behaviour change. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
696906315a |
feat(scope-audit): transport view of every repeater seen forwarding (#2142, API) (#2167)
## Summary Backend half of #2142 (the page is the stacked PR #2169). `GET /api/scope-audit` lists only repeaters that have answered a declared-regions request: 17 of almost 200 nodes on the instance the request came from. This adds a transport view, `GET /api/scope-audit?mode=transport&window=…[&scope=X]`, with one row per repeater **seen forwarding** in the window: - `transported`: the named region scopes it was seen carrying, each with packets and first/last seen (most packets first), plus unscoped, unmatched and ambiguous-hop counts; - `asked`, `declaredRegions`, `notObserved`, `configState`, `declaredAt` where a declared answer exists; `asked: false` with null declared fields where it does not ("not asked" is not "declares nothing"); - with `scope=X` (spelled `be`, `#be` or `BE`): `carriesScope` on every row and `carrying` / `notCarrying` counts, which is the rollout report the issue asks for. `mode=declared` (or no `mode`) is unchanged. An unknown `mode` is a 400. Partial fix for #2142: this PR provides the API; the page is #2169 and the observer region filter #2171. The issue should only be considered done once all three are in. ## How it follows the issue's notes - **The scan does not change.** It is the same single `ScopeAuditForwarding` pass, with every repeater and room in `nodes` (plus any declared target) as a target instead of only the declared ones. Same attribution rules: FLOOD-family route types only, `minForwarderHopHexLen` (no 1-byte hops), and a hop whose prefix matches more than one repeater is credited to none of them. - **Identity lookup with 1,000+ pubkeys:** `scopeAuditNodeIdentities` runs one `IN` query; exercised at 1,200 keys in the benchmark below. - **Row count and response size:** measured below. - **Cache:** the transport view is cached per window with the declared view's TTLs (30 s; 5 min for 7d), under the same singleflight group with a `transport|` key. The scope filter is applied to a copy of the cached result, so one scan serves every scope. (`scope=`, not `region=`: across the API `region=` is the observer IATA filter.) - **Wording:** "transported" is documented as *seen carrying*, not configured for; the declared side stays separate. ## Performance `BenchmarkComputeScopeTransport`: 1,200 repeaters (#1975 quotes an instance with 1,179), 30k flood transmissions in the last 24h on 4-hop paths of 3-byte prefixes, a third region-scoped: ``` BenchmarkComputeScopeTransport-12 5 219253727 ns/op 425.4 response-KB 1200 rows ``` 219 ms per compute, served from cache for 30 s (24h) or 5 min (7d); a 425 KB uncompressed response for 1,200 rows. This is new code with no "before"; the declared view's numbers are untouched (same scan, same cache). ### On real data (a staging copy of a production database and its live feed) | request | rows | result | time | |---|---|---|---| | declared view, 24h | 16 | (unchanged) | | | `mode=transport`, 24h | **385** repeaters seen forwarding, 16 of them with a declared answer | | 1.5 s cold | | `mode=transport&scope=<scope A>`, 7d | 664 | **51 carrying it, 613 not** | 14.4 s cold | | `mode=transport&scope=%23<SCOPE B>` (`#` and capitals), 7d, same window now cached | 664 | 237 carrying, 427 not | 0.35 s | The 7d cold compute is the same full-window hop scan the declared view already pays for 7d (its own TTL comment measures 16.7 s on a live-shaped database); like the declared view it is then cached for 5 minutes, and every scope filter is served from that one scan. - Commit `ceaeb56b` renamed the filter from `region=` to `scope=` after opening; the real-data figures above were taken before the rename and are otherwise the same code. ## Tests - Red commit `2a9b1286`: response types and a `mode=transport` stub that returns no rows, plus tests. Four fail on the stub: - repeaters that never answered are listed with `asked: false` and null declared fields (and the answered one with its declared list and `notObserved`); - `scope=fr` / `%23fr` / `FR` marks carriers and non-carriers and counts them; no scope means no `carriesScope` or counts; - an unknown mode is a 400; - the transport and declared views are cached separately (requesting one does not serve the other). - Also added, passing on the stub by construction: a repeater with no traffic in the window is not listed; the node blacklist applies. - Green commit `60c669f4`: the implementation, `docs/api-spec.md` (new `mode` and `scope` parameters, a Transport view section with the response shape), and the benchmark. - `make test`: 16 of 16 modules pass, including every existing scope-audit test. ## Review notes - Hidden-name and blacklist filtering use the same rules as the declared view. - `newTestStoreWithDB` now takes `testing.TB` so the benchmark can use it (callers unchanged). - No new `map[string]interface{}`; the response is a named struct. - No config added. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `060668b6`: S3776 (cognitive complexity 41 > 15) in the new `computeScopeTransport`. It is split into target, visibility, row and sort helpers; the response is the same and the transport tests are unchanged. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
8a3b923e81 |
perf(scope-stats): keep the per-region count on the first_seen index (#2166)
## Summary `/api/scope-stats` averaged **2.6 s on one production instance (max 17.7 s)**, whatever the window. Four of its five queries take 10-100 ms on that database. The fifth, the per-region count, took **2.5-2.9 s for every window, 1h included**: SQLite chose `idx_tx_scope_name` (`scope_name > ''`) and walked every scoped transmission ever stored (45 days of them), filtering by `first_seen` afterwards. ``` SEARCH transmissions USING INDEX idx_tx_scope_name (scope_name>?) | USE TEMP B-TREE FOR ORDER BY ``` A unary `+` on every `scope_name` reference in that query (`WHERE … +scope_name …`, `GROUP BY +scope_name`) stops the planner using that index. The query then searches the window on `idx_transmissions_first_seen` and groups in a temp b-tree, the same idiom the `advertsByRole` query in the same function already uses for `payload_type`. Putting `+` in the `WHERE` alone is not enough: the planner then walks `idx_tx_scope_name` to avoid sorting for the `GROUP BY`, which is just as slow; the `GROUP BY +scope_name` is what fixes it. ``` SEARCH transmissions USING INDEX idx_transmissions_first_seen (first_seen>?) | USE TEMP B-TREE FOR GROUP BY | USE TEMP B-TREE FOR ORDER BY ``` Related to #2146, which lists `/api/scope-stats` (2.3 s average on one instance) and records the 24 h summary as 7 ms warm there. On the instance measured here the per-region query planned onto `idx_tx_scope_name` and took 2.5-2.9 s for every window; this PR fixes that plan. Not marked as a fix: #2146 is about pool and lock waits. ## Results are identical On a copy of the production database, the old and new query run inside **one read transaction** (same snapshot) return exactly the same rows for 1h, 24h, 7d and 45d (5, 10, 12 and 18 regions; 73, 2,024, 13,822 and 76,083 transmissions). `TestScopeStatsByRegion_CountsMatchWindow` also checks the per-region counts against a direct count of the window. ## Performance ### The query alone, on a copy of the production database 311k transmissions, 76k of them region-scoped over 45 days; best of three: | window | before | after | |---|---|---| | 1h | 2,530 ms | 0.1 ms | | 24h | 2,570 ms | 11.6 ms | | 7d | 2,876 ms | 124 ms | ### The endpoint, on staging Cold `GET /api/scope-stats` (the response cache is 30 s, so each sample is a full compute) on a staging host with a copy of a production database and its live feed; two samples each, 32 s apart: | window | master | this PR | |---|---|---| | 1h | 3.08 s, 2.83 s | **16 ms, 12 ms** | | 24h | 2.69 s, 2.79 s | **131 ms, 101 ms** | | 7d | 3.35 s, 4.53 s | **626 ms, 663 ms** | ### Go benchmark `BenchmarkGetScopeStats24h`: all five scope-stats queries for 24h, on 300k transmissions spread over 45 days (a quarter region-scoped), with production's tables and both competing indexes and `ANALYZE` run as production does (#2072). 10 runs: ``` cpu: AMD Ryzen 5 3600XT 6-Core Processor │ master │ this PR │ GetScopeStats24h-12 33.186m ± 3% 8.058m ± 6% -75.72% (p=0.000 n=10) ``` The fixture is smaller and better cached than production, which is why the absolute times are lower than above; the plan change is the same. ## Tests - Red commit `a7dcf98f`: moves the query into `scopeStatsByRegionQuery` (unchanged), adds the fixture, the benchmark, `TestScopeStatsByRegion_UsesFirstSeenIndex` (`EXPLAIN QUERY PLAN` must use `idx_transmissions_first_seen` and not `idx_tx_scope_name`; fails on `master` with exactly the production plan shown above) and `TestScopeStatsByRegion_CountsMatchWindow` (passes on both commits; guards correctness). - Green commit `f9ba986b`: the `+` hints and a comment explaining them. - All existing scope-stats tests pass; `make test`: 16 of 16 modules pass. ## Review notes - Only the per-region query changes; the summary, non-transport, time-series and adverts-by-role queries already search by `first_seen`. - If `idx_transmissions_first_seen` were ever missing, the query would still run correctly (full scan); `+` only removes `idx_tx_scope_name` from consideration, unlike `INDEXED BY`, which would make the query fail. - `/api/scope-stats` still has no singleflight around its 30 s cache (unlike stats, channels and the scope audit). With the query now well under a second it matters much less; left for a separate change. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - No issues on the added lines. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
95672ac2be |
perf(server): share /api/nodes responses for identical queries for 15 s (#2165)
## Summary
`/api/nodes` scans every node row into a map, enriches it (hash size,
multi-byte capability, relay info, usefulness scores, declared scope)
and encodes the page on **every** request. On one production instance (2
vCPU / 2 GB) that was **95 GB of allocation in 45 hours, about 6.7 MB
per request and a quarter of all bytes allocated**, on a process where
garbage collection was 87% of CPU.
The queries repeat heavily. In that instance's nginx log for one day,
three pages of one query (`limit=500&offset=0|500|1000&lastHeard=30d`)
were 10,000 of the 14,053 `/api/nodes` requests.
This PR keeps the encoded response for 15 s per distinct query:
- `handleNodes` serves the cached bytes when it can; otherwise it builds
through `buildNodesResponse` (the old handler body, unchanged apart from
returning instead of writing) inside a singleflight, so concurrent
identical misses share one build.
- **Cache key:** blacklist generation | hidden-prefix generation |
sorted query string. Parameter order does not split the cache, and any
change to either filter list misses it at once. This is the pattern
`/api/nodes/{pubkey}/reach` already uses (#1629).
- `setGeoFilter` drops the cache, so a geo-filter change through the
admin endpoint shows immediately.
- **Bounded twice:** by `maxCacheEntries` (256) like the other keyed
caches, and by bytes, since a full-list response is several MB. Expired
entries are dropped on every insert, the total is capped at 32 MB, and a
single response over 8 MB is not cached.
- **15 s** matches the relay/usefulness caches the response is built
from (#1257); the client already keeps its own node list for 90 s.
Related to #2146: `/api/nodes` is in its endpoint table (144 s max on
one instance). This PR reduces how often the handler runs and allocates,
not the SQL wait the issue is about, so it is not a fix for it.
## Performance
### Go benchmarks
1,100 nodes, half repeaters, 10 runs each:
```
cpu: AMD Ryzen 5 3600XT 6-Core Processor
│ master │ this PR │
│ sec/op │ sec/op vs base │
HandleNodesRepeated-12 5991.14µ ± 3% 41.21µ ± 7% -99.31% (p=0.000 n=10)
HandleNodesUnique-12 6.010m ± 2% 5.967m ± 0% ~ (p=0.393 n=10)
│ B/op │ B/op vs base │
HandleNodesRepeated-12 1611.0Ki ± 1% 240.9Ki ± 0% -85.05% (p=0.000 n=10)
HandleNodesUnique-12 1.566Mi ± 1% 1.753Mi ± 1% +11.96% (p=0.000 n=10)
```
`Repeated` is one query asked over and over (the production pattern).
`Unique` makes every request a different query, so nothing is served
from cache: the time is unchanged, and it allocates 12% more because the
body is now built in memory so it can be kept.
### Expected hit rate
Replaying that day's 14,053 `/api/nodes` requests through a cache of
each lifetime:
| TTL | served from cache |
|---|---|
| 5 s | 39% |
| 10 s | 47% |
| **15 s** | **52%** |
| 30 s | 58% |
| 60 s | 69% |
### Staging A/B
The same 30 minutes of real production API traffic (5,640 requests, 279
of them `/api/nodes`), replayed at 2x on a staging host against a copy
of a production database, on `master` and on this branch:
| `/api/nodes` | master | this PR |
|---|---|---|
| p50 | 47.1 ms | **31.0 ms** |
| p95 | 165.4 ms | **114.4 ms** |
| max | 759.9 ms | 321.7 ms |
| allocated by `handleNodes` during the replay | 1,000 MB | **594 MB
(-41%)** |
| total allocated by the process | 3,704 MB | 3,349 MB (-10%) |
The -41% matches the expectation from the hit rate and the miss-path
cost: about 0.48 × 1.12 + 0.52 × 0.15 ≈ 0.61 of the old allocation.
## Tests
- Red commit `ab33bc29`: a test-only build hook, the tests and both
benchmarks. Three fail on `master`: identical queries are built once,
reordered parameters share an entry, ten concurrent misses share one
build. Three more guard against over-caching and pass on both commits:
different queries are built separately, an expired entry is rebuilt, a
geo-filter change drops entries.
- Green commit `ea0d1cf2`: the cache, plus
`TestHandleNodes_FilterListChangesMissCache` (hidden-prefix and
blacklist changes miss the cache) and `TestNodesCache_BoundedByBytes`
(stays under 32 MB; an oversized response is not cached).
- One existing test changed: `TestHandleNodesRegionUsesStore` already
clears the store's own 30 s region cache by hand before its second
request; it now clears this cache on the same line.
`TestHiddenNamePrefix_1181_NodesList`, which changes the hidden-prefix
list between requests, passes unchanged because the generation is in the
key.
- `go test -race` on the new tests passes. `make test`: 16 of 16 modules
pass.
## Review notes
- The response depends only on the query string and server state, not on
who is asking (no auth or header-dependent output in the handler), so a
shared cache is safe.
- `writeJSONBytes` writes the cached bytes with the same `Content-Type`
and status as `writeJSON`; the body has the trailing newline
`json.Encoder` adds, so output is byte-identical.
- No new `map[string]interface{}` added (the existing node maps are
unchanged; converting them to a struct is a separate, larger job).
- The TTL and byte budget are constants. They could be operator settings
later (AGENTS.md rule 8); not added here.
## SonarLint
SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.
- **S3776 on `buildNodesResponse`:** complexity 91. That is upstream's
existing handler body moved unchanged (it is also 91 on `master`); this
PR makes the handler itself simpler.
- **Fixed** in `25d40950`: the test hook moved to the cache-miss path,
so the builder stays identical to the old body rather than one point
above it. Splitting that function is a refactor of its own.
## Checked against the repo rules
- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Alex BandClaude Opus 5.5
|
323b8040d4 |
perf(frontend): throttle nav stats to one refresh per 15 s, none in hidden tabs (#2162)
## Summary The nav bar's packet / node / observer counts were refreshed three ways at once: a 15 s timer, every batch of live WebSocket packets (`debouncedOnWS`, 250 ms), and the `/stats` client cache being cleared every 5 s while packets arrive. None of them checked whether the tab was visible. So every open tab, including hidden ones, asked `/api/stats` every few seconds for as long as it stayed open. On one production instance's nginx log for a single day, `/api/stats` was **135,654 of 187,411** CoreScope API requests (72%), about 20,000 a day for each browser that kept a tab open. Each response is cheap thanks to the 10 s server-side cache (#1910), so this is about request volume, bandwidth and log noise, not server CPU. This PR adds `createVisibleThrottle()` to `app.js` and routes all three triggers through one throttled refresh: - at most one `/api/stats` request per 15 s from a visible tab (the old timer's own period); - none from a hidden tab; - one immediate refresh when a hidden tab becomes visible again. No existing issue covers this. ## Performance Measured with headless Chromium, counting `/api/stats` requests from a real tab: 60 s visible, then 60 s with the page hidden (`document.hidden` and a `visibilitychange` event), then back to visible. | `/api/stats` requests | production (current release) | staging, `master` | staging, this PR | |---|---|---|---| | visible tab, per minute | 9 | 4 | 4 (at +10, +25, +40, +55 s) | | hidden tab, per minute | 6 | 5 | **0** | | on return to visible | n/a | n/a | 1, immediately | The same check confirmed the nav bar still renders its counts (`… pkts · … nodes · … obs`). ## Tests - Red commit `2c81a3e4`: `tests/unit/test-nav-stats-throttle.js` loads the real `public/app.js` in a `vm` context. `createVisibleThrottle` lands as a pass-through stub, so the test fails on its first assertion (40 refreshes where 1 is expected). - Green commit `f0ba92ab`: the throttle, and the nav stats timer, WS hook and a new `visibilitychange` listener calling one throttled trigger. All 6 cases pass: one refresh per burst, again after the interval, at most 241 an hour, none while hidden, one on return, and a source check that `updateNavStats` is wrapped and not called by the timer directly. - Added to `test-all.sh` in sorted order; `test-test-inventory.js` passes. - Frontend suites: 194 of 195 pass. The one failure, `test-issue-1956-release-routing.js`, needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally. ## Review notes - `home.js` keeps its own one-shot `/stats` load on the home page; only the nav bar's polling changed. - A tab opened in the background loads its counts when it is first shown. - The 15 s interval is a constant at the call site. Could be exposed in the customizer later (AGENTS.md rule 8); not added here. - No Playwright E2E added: the change is timing behaviour, covered by the unit test against the real file and by the browser measurement above. Happy to add an E2E if wanted. ## SonarLint SonarQube Community rules (the SonarLint engine) were run on the files this PR touches, and only issues on lines it adds are listed. - **Fixed** in `9eb94ea1`: `node:` specifiers (S7772). - **Kept on purpose:** - S7773 wants `Number.isFinite`/`Number.parseInt` for the `isFinite` and `parseInt` handed to the `vm` sandbox, but `app.js` calls the global functions, so those are the ones it needs. - S1523 flags `vm.runInContext`, which is how the frontend unit tests load the repo's own `public/app.js`. ## Checked against the repo rules - Rebased onto `master` (`673442cc`). Each red commit was re-run and still fails on its assertions; each green commit passes (TDD.md). - `make test`: all 16 modules pass. Frontend suites: all pass except `test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME` and fails the same way on `master` locally (#2160 addresses it). - No private data: test fixtures and examples use the codes the existing tests use (SJC/SFO/LAX, `#be`-style scopes). - No new `map[string]interface{}`, no hard-coded colours. `git add` was explicit, and there is one logical change per commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Alex BandClaude Opus 5.5
|
be6aeffe45 |
perf(store): drop the duplicate subpath count map (#2163)
## Summary
The subpath index keeps every contiguous run of 2-8 hops from every
stored transmission's path, in two maps:
- `spIndex`: key → count
- `spTxIndex`: key → transmissions containing that subpath, one entry
per occurrence
Both were always updated together (the only production callers were
`addTxToSubpathIndexFull` / `removeTxFromSubpathIndexFull`, which touch
both), so every count equalled `len(spTxIndex[key])`. The existing
`TestSubpathTxIndexPopulated` asserted exactly that. The index therefore
held its 1M+ keys twice.
This PR removes `spIndex`. The two analytics consumers read `len(txs)`,
and the add/remove helpers take only the transaction map. No API
response changes.
No existing upstream issue covers this; it came out of profiling a
production instance where garbage collection was 87% of CPU (30 s CPU
profile: `gcDrain` 39.9 s of 45.9 s sampled) with an 805 MB live heap
against a 1 GiB `GOMEMLIMIT`. That heap profile put 186 MB in this
index; on that instance's 7-day window there are 1.2M distinct subpath
keys (79% seen once) and 3.07M entries.
## Performance
### Go benchmark
`BenchmarkBuildSubpathIndex` builds the index for a synthetic,
production-shaped window: 41k transmissions with path lengths drawn from
the production hop-count histogram, and hops from random walks over a
400-node graph with 3-byte prefixes. That gives 1.48M keys, 85% seen
once (production: 1.2M, 79%). No real data is used. 10 runs each:
```
cpu: AMD Ryzen 5 3600XT 6-Core Processor
│ master │ this PR │
│ sec/op │ sec/op vs base │
BuildSubpathIndex-12 2.157 ± 1% 1.610 ± 5% -25.33% (p=0.000 n=10)
│ retained-B │ retained-B vs base │
BuildSubpathIndex-12 238.5Mi ± 0% 187.7Mi ± 0% -21.31% (p=0.000 n=10)
│ B/op │ B/op vs base │
BuildSubpathIndex-12 572.0Mi ± 0% 465.6Mi ± 0% -18.61% (p=0.000 n=10)
```
(`retained-B` is the live heap the index keeps after a forced GC: 250.1
MB → 196.8 MB.)
### Staging heap
Live heap after a forced GC (`/debug/pprof/heap?gc=1`), on a staging
host running a copy of a production database and its live MQTT feed,
attributed to the subpath index builder:
| | master | this PR |
|---|---|---|
| subpath index (`buildSubpathIndex`, cum) | 172.4 MB | 116.1 MB |
| packets in memory at the time | 27,521 | 24,086 |
| per packet | 6.27 KB | 4.82 KB (**-23%**) |
The two snapshots were taken minutes apart with different numbers of
packets loaded, hence the per-packet figure; it agrees with the
benchmark's -21%.
The same A/B replay used for the other PRs (30 minutes of real
production API traffic) showed no latency change (p50 10.3 ms vs 10.7
ms, p95 59.5 ms vs 52.9 ms, 0 errors). GC count and CPU were lower in
this run (44 → 28 collections, 13.0% → 9.6% CPU), but runs on this
shared host vary by that much on their own, so no claim is made from
them.
## Tests
- Red commit `ded359ff`: the fixture, `TestSubpathFixtureShape` (keeps
the fixture production-like: 60-95% singleton keys),
`BenchmarkBuildSubpathIndex`, and `TestSubpathIndexRetainedHeap`, which
allows at most 225 MB for the 41k fixture. It fails on `master` with
250.1 MB.
- Green commit `e399c561`: the change. The heap test passes at about 197
MB.
- `make test`: 16 of 16 modules pass; `go test -race` over the subpath,
eviction, path-hop and resolved-index tests passes.
Test changes in the green commit are mechanical, as required for a
structural change:
- struct literals stop setting `spIndex`; where a literal did not set
`spTxIndex` it now does, because appending to a nil map panics where the
old code skipped the transaction map when passed `nil`;
- count assertions read `len(spTxIndex[key])`;
- the old "spIndex and spTxIndex agree" check becomes "no key is kept
with an empty list", which is what keeps `len()` a correct count.
## Review notes
- Removal still takes out one occurrence per subpath occurrence and
deletes a key whose list becomes empty, so `len()` stays equal to the
old count through eviction.
- The store's memory estimate (`perSubpathEntryBytes`) is unchanged
here; accounting accuracy is a separate item.
- No new `map[string]interface{}`.
## SonarLint
SonarQube Community rules (the SonarLint engine) were run on the files
this PR touches, and only issues on lines it adds are listed.
- **Fixed** in `8af9582f`: S3776 (cognitive complexity 16 > 15) in the
test fixture. Path generation is split into graph and hop-count helpers;
the random draws stay in the same order, so the fixture is unchanged.
## Checked against the repo rules
- Rebased onto `master` (`673442cc`). Each red commit was re-run and
still fails on its assertions; each green commit passes (TDD.md).
- `make test`: all 16 modules pass. Frontend suites: all pass except
`test-issue-1956-release-routing.js`, which needs CI's `GITHUB_REF_NAME`
and fails the same way on `master` locally (#2160 addresses it).
- No private data: test fixtures and examples use the codes the existing
tests use (SJC/SFO/LAX, `#be`-style scopes).
- No new `map[string]interface{}`, no hard-coded colours. `git add` was
explicit, and there is one logical change per commit.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
n30nex
|
7f2c999266 |
fix(ingestor): record cancelled background migrations separately (#2161)
Refs #1739. Depends on the small local-validation fix in #2160; merge that first. This branch contains its prerequisite commit plus one migration-status commit. An interrupted background migration currently records `failed`, making an operator shutdown look like a migration fault. Errors matching `context.Canceled`, including wrapped errors, now record `cancelled` and retain the error and completion time without logging `FAILED`. Ordinary errors, deadline expiry and panics remain failures even when the context is cancelled; a successful callback still records `done`. Cancelled work follows the existing retry path on the next run. Regression tests cover classification and clearing stale completion/error fields before retrying, then successful completion. The migration runbook documents the distinction. This covers the existing ingestor bookkeeping portion of #1739. The issue's server status mapper and banner depend on closed, unmerged #1735 and are absent from current master; this PR does not add that API/UI and leaves #1739 open for the remaining scope. No new configuration, customizer setting or schema migration is required. Validation: direct and wrapped cancellation failed the new regression before the change; focused tests and the race check pass afterward. Native full server/ingestor suites and all 194 standalone frontend suites pass with #2160. The browser smoke suite passed 148 checks, with three fixture-dependent skips and two pre-existing disabled cases. Windows-only filesystem capability skips are documented in #2160; Linux CI retains the security assertions. Performance impact: one error-chain check when a background migration finishes, using the existing single status update. No packet-ingestion loop, batching, polling or retention behavior changes, and no speedup is claimed. |
||
|
n30nex
|
a5b60ca594 |
test: make local validation independent of runner environment (#2160)
Fixes #2159. Related to #1995. Standalone release tests failed when `GITHUB_REF_NAME` was absent and could use an unrelated runner tag when it was present. Supply the simulated tag explicitly; exercise the real release-notes step with both missing and unrelated host values, including present and absent notes files. Two filesystem fixtures now report unsupported Windows capabilities explicitly: POSIX permission bits, and symlink creation denied with error 1314. Unix mode checks and all symlink safety assertions remain intact; other symlink setup errors still fail. Production code and workflows are unchanged. This is 29 added lines across three test files. Validation: reproduced all three failures before fixing them; focused regressions, the full standalone frontend suite, and native Go server/ingestor suites pass. The POSIX permission and unprivileged Windows symlink fixtures are explicitly skipped on this host and remain enabled in Linux CI. The browser smoke suite passed 148 checks, with three fixture-dependent skips and two pre-existing disabled cases. No configuration or customizer changes. |
||
|
efitenandClaude Opus 5.5
|
0737f10a5d |
feat(notify): optional external node alerts feed and the node.external event (#2177) (#2178)
Closes #2177. ## What Node notifications can mail alerts computed outside CoreScope. With `userManagement.notifications.externalAlerts` configured, users can choose a new event, `node.external`, which is off by default and only offered when configured: ```json "notifications": { "enabled": true, "externalAlerts": { "url": "http://coverage-job/alerts.json", "maxAgeHours": 48, "label": "Coverage" } } ``` The feed: ```json { "generatedAt": "2026-10-11T01:35:12Z", "alerts": [ { "pubkey": "<64 hex>", "key": "SE", "text": "never heard to the south-east", "url": "https://example.org/coverage" } ] } ``` Mail lines: `<name>: Coverage: never heard to the south-east, <time>` when an alert appears, `<name>: Coverage resolved (SE), <time>` when it is gone from a fresh feed. ## How - `cmd/server/notify_external.go`: feed types, `parseExternalFeed` (RFC 3339 `generatedAt`, age check, per-alert validation), `fetchExternalFeed` (200 only, 1 MB cap, 10 s timeout). - `notify_eval.go`: `evalExternal`. Subjects are `<pubkey>/<key>`; a `<pubkey>/*` row is the per-node baseline, so the first fresh feed after watching stores the current alerts without a mail (like `foreign.new`), and later alerts mail at once. `External == nil` (not configured, nobody chose it, or stale) leaves every `node.external` state as it is. - `notify_service.go`: the feed is fetched only when configured and at least one user chose the event. Each new reason the feed is unusable is logged once, and so is the recovery. Mail text and link: the alert's `url`, otherwise the node page. - `notify_handlers.go`: `availableEvents` and the new `externalLabel` include the event only when configured; PUT answers 400 for `node.external` otherwise. - `internal/users/notify.go`: event constant, `OptionalNodeNotifyEvents` (canonical order: node, optional, admin), `RemoveWatch` also deletes the node's `<pubkey>/...` rows. - Account page label, OpenAPI descriptions, `config.example.json` comment, `docs/user-guide/accounts.md` section. Decisions on the three open questions in #2177: the generic name `node.external`; the resolved mail names the key instead of repeating the text, so there is **no `users.db` schema change** (a bump would also make older binaries refuse the file); one feed. ## Invariants - Read/write separation (#1283): the server only reads the feed over HTTP. State rows go to `users.db` through `internal/users`, like the other events. - A dead, garbled or old feed never mails "resolved": `TestEvaluateExternalStaleFeedChangesNothing` and `TestNotifierMailsExternalAlertsAndSkipsAStaleFeed` (HTTP 502 and a 72 h old feed in a row, then a fresh empty feed resolves). ## Performance The evaluator adds one pass over the stored states (already read every tick) plus, per user with the event on and per watched node, the node's current alerts. Bounded by `maxWatchesPerUser` (50) times the alerts per node. The feed is fetched once per tick (default 5 minutes) only when someone chose the event, capped at 1 MB. No packet-store lock is taken for it. Nothing changes for instances without `externalAlerts`. ## Tests - `cmd/server`: `go test` green locally (Windows; `TestSaveGeoFilterPreservesFileMode` skipped as it needs Unix mode bits). New: `notify_external_test.go` (feed parsing and limits, HTTP fetch 200/404/oversized, config resolution, evaluator baseline/new/resolved/returning/stale/not chosen/notifications off, notifier end to end with a fake feed, account routes). - `internal/users`: green, 2 new tests (optional event and canonical order, `RemoveWatch` deletes external subjects only for that node). - Frontend: `test-all.sh` green, 1 new test in `test-notifications-ui.js` (label from `externalLabel`, escaped). - `scripts/check-xss-sinks.sh --file public/notifications.js`: clean. ## Not done - No E2E test: the account page change is one label, covered by the unit test. - Not run against a live instance yet. The first feed in use is the coverage job of on8ar.eu (`alerts.json`, 29 alerts at the moment). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
