mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-11 18:25:44 +00:00
5d2e14aba2b25bdcb77cfd07331b5f35ec0b6b3a
2870
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 efitenandClaude Opus 5
|
5d2e14aba2 |
fix(#1943): cancel the deferred swatch focus so arrow keys are not undone (#1945)
Closes #1943. The colour picker's keyboard navigation is broken, and the E2E flake that has been failing unrelated PRs (#1940, #1941, and master pushes `589fa987` and `859173f1`) was reporting it correctly. ## Cause `showPopover` deferred focusing the first swatch with an uncancellable `setTimeout(..., 0)` at `channel-color-picker.js:146`, and nothing cleared it on hide. The file contained **zero** `clearTimeout` calls. Reopen the popover while a swatch still holds focus and that timer lands after the user has already pressed an arrow key, pulling focus back to the first swatch. Proven, not argued. Instrumenting `HTMLElement.prototype.focus` with a stack trace, on one open: ``` focus(#f97316) @10205ms <- the keydown handler focus(#ef4444) @10208ms <- channel-color-picker.js:146:58 ``` Three milliseconds apart. ## The user-visible bug Worse than a flaky test. **Open the picker, arrow to a colour, press Enter, and the first colour is assigned instead of the one you chose.** Holding the timing still made the existing suite say so directly: ``` ✗ Enter should assign focused color (#f97316), got #ef4444 ``` ## Why the test looked flaky The revert happens on **every** open. Only whether the assertion reads before or after it varies, which is why an idle machine passes and a loaded runner does not. #1939 (mine) assumed the opposite: a race in which the handler had not yet moved focus, cured by waiting for it. #1943 has the measurement that disproves it. The failing step took **16 ms** while that wait has a **3 second** budget, so the wait was resolving successfully and then the value was reverted underneath it. It never helped. Its comment is corrected in this PR rather than left to mislead the next reader. ## Fix Keep a handle for the timer, cancel a pending one on both show and hide, and inside it do nothing when the popover has since been hidden or when focus already sits inside it. A fresh open still focuses the first swatch, which is what the accessibility behaviour is for. An open that inherits focus, or a user who has already navigated, is left alone. ## Verification - The **regression test added here fails on unmodified master** with `a late focus timer must not move focus after the user did` and passes with the fix. - It is **deterministic, not load-dependent**: it reproduces the exact sequence the stack trace identified (open, Escape, reopen, ArrowRight before the timer lands) rather than waiting for contention. It also asserts the Enter path, so the user-visible half is covered and not just focus position. - Full suite: 10 of 10, three consecutive runs. ## Note on the other flake This is one of two E2E failures blocking the queue. The other, #1925, is a different mechanism in a different file and is fixed separately in #1944. Together they should leave the E2E suite deterministic again. Same shape as @TeTeHacko's finding in #1940: something is operable before its setup has finished. That is now three instances in this codebase, so it may be worth a look as a pattern rather than three separate fixes. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
e2df9bbd3e |
fix(#1925): stop theme-refresh from discarding the neighbor-graph filter (#1944)
Closes #1925. This is the flake that failed #1942 and #1871, neither of which touches the feature. It is not a test problem. ## Cause Every page load fires exactly one delayed, full re-render of the active analytics tab: 1. `app.js` starts `/api/config/theme` without gating navigation on it, deliberately. 2. When it resolves, `_customizerV2.init()` runs `applyCSS()`, which dispatches `theme-changed`. 3. `app.js:1188` debounces that by 300 ms and dispatches `theme-refresh`. 4. `analytics.js:230` answered it with `renderTab(_currentTab)`. For the neighbor-graph tab step 4 is destructive. `renderTab` replaces `el.innerHTML`, so the role checkboxes are recreated with their defaults and companion is silently re-checked, and `_ngState` is rebuilt from the full 1400-node graph. The count is back over the 1000 limit, so `#ngSkipMsg` returns and the canvas is hidden. The re-entrancy epoch guard cannot prevent it. That guard stops a superseded `tick()` loop; this is a legitimate new top-level render pass that resets the very inputs the guard protects downstream of. **One mechanism produces both documented failure modes**, decided only by where that single re-render lands: | lands | result | |---|---| | before the first uncheck | harmless, test passes | | between an uncheck and the next `waitForFunction` poll | mode 1, the 15 s timeout | | after `waitForFunction` succeeded, before the follow-up `evaluate` | mode 2, "expected #ngSkipMsg gone again" (#1942) | Measured on an idle machine: the test's final evaluate at 542 ms, `theme-refresh` at 636 ms, `#ngSkipMsg` re-added at 667 ms. It passes locally by about 90 ms. On a loaded runner the test's Playwright round trips stretch while `theme-refresh` still lands at theme-fetch latency plus 300 ms, so it arrives mid-test. ## Fix On `theme-refresh`, restart the renderer instead of rebuilding the tab when the neighbor-graph tab is active and has state. Four lines. This stays theme-correct: node colors are read live per frame from `window.ROLE_COLORS`, role swatches use `.role-swatch--{role}` CSS tokens, stats and the skip message use CSS variables, and the one cached theme value, `_labelColor = cssVar('--text-primary')`, is re-read on restart at `analytics.js:3375`, inside `startGraphRenderer`. When `_ngState` is null it falls through to the old path. ## Verification Measured, not asserted: - **Deterministic reproduction** (hold `/api/config/theme` until just before the second filter-down, then stall 500 ms before the final evaluate; no synthetic events dispatched): **2 of 2 fail** on unmodified master with the exact #1942 message, **3 of 3 pass** with this fix. - The **unmodified** E2E test passes against the fixed build. - With the tab open and a filter applied, a `theme-refresh` leaves the filter intact, the canvas present, and produces no page errors. - The **regression test added here fails on unmodified master** with `theme-refresh reset the role filter (companion re-checked)` and passes with the fix. It dispatches the event directly, so it tests the cause instead of waiting for the race to appear. ## What this does not cover The same startup re-render silently discards user interaction in the first second or so on **any** analytics tab, not just this one. A user who clicks quickly after load loses that click. This change covers the neighbor-graph tab, because that is what #1925 is about and what is failing CI. The general fix, for example skipping the startup refresh when the effective config changed nothing, deserves its own issue rather than being smuggled in here. Side observation while tracing: `theme-changed` fires twice at startup, at about 311 ms and 323 ms. The debounce collapses them, so it is harmless, but it means the customizer pipeline runs twice. I did not identify the second dispatcher. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
9ae3387416 |
feat(ingestor): RF environment samples from mobile clients (#1906)
> **Stacked on #1905.** This branch contains #1905's commits; review and merge that one first. The diff unique to this PR is the `client_rf_samples` table, its handler, the delta query and its retention. ## What Everything CoreDrive RX records today is anchored to a *packet*. But a drive also passes through RF conditions that exist whether or not a packet arrives: the noise floor, how busy the channel is, how many receptions fail CRC. The radio measures all three and was never asked. This samples the companion's own counters along the GPS track and stores them, so the server can render a noise-floor map, a channel-utilisation map and a CRC-error-rate map. A fixed observer cannot produce those — it measures one point forever. **Zero airtime:** `CMD_GET_STATS` is a local Bluetooth query to the attached radio. Nothing is transmitted. ## Design points worth knowing - **Absolutes are stored; deltas are derived at query time.** A lost or reordered sample then costs one interval rather than corrupting a running total. `ClientRfDeltas` breaks the chain whenever `uptime_secs` fails to increase — that is the reboot and counter-wrap detector. - **Absent is not zero, end to end.** Firmware predating the `recv_errors` field cannot count CRC errors at all, and a stored `0` would read downstream as "a perfectly clean channel" — the opposite of "we don't know". Presence/absence is preserved through the app parser, the wire payload, a nullable column, and the delta view, which returns `nil` rather than `0` when either endpoint is unknown. Each of those five layers has its own test. - **`sampled_at` is millisecond precision, and it is load-bearing.** SQLite compares these strings lexicographically and `.` (0x2E) sorts before `Z` (0x5A), so a second-resolution retention cutoff would delete rows *inside* the window. The prune formats its cutoff with the same layout. ## Performance justification (touches the ingest hot path) - One INSERT per sample, gated behind an opt-in flag that defaults off. Sample rate is 15 s while moving and 5 min while parked, so roughly 240 rows per hour per active driver. - The delta query is a single `LAG(...) OVER` pass with no nested query inside the loop, so it cannot deadlock the single writer connection. Window functions are already used elsewhere in this codebase. - Retention has its own key and index (`sampled_at`); without it the table would grow unbounded, so `config.example.json` documents it inline. ## Safety for existing deployments Opt-in and default off on both sides (`clientRfSamples.enabled`, and `rfSampler` in the app). The coverage path is untouched — `Publisher.buildPayload` is byte-identical and a record with no `kind` field still routes to `/packets` unchanged. The MQTT dispatch was reshaped so that **anything on `meshcore/client/…` returns from that branch in every config state**, with the enable-gates inside rather than in the topic match. Previously a disabled gate let the message fall through to the observer path, where `parts[1]` — the literal string `client` — was read as a region and the phone's pubkey registered as an observer. The blacklist check now also runs ahead of the sub-topic switch, so it covers every present and future client sub-topic. ## Testing Full ingestor suite green. Notable coverage: a `/rf` message with the gate off writes nothing anywhere and does not fall through; a sample missing `uptime_secs` is rejected rather than stored as an unusable row; two samples 40 ms apart remain two rows; and the retention test seeds a row with a non-zero millisecond component inside the cutoff second, which is the only row that distinguishes a correct cutoff from an RFC3339 one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Joel ClawandJoel Claw
|
ac6fbaf9f3 |
perf: reuse ctx buffer in resolvePathForObs, cache ReadMemStats per store (#1873)
## Problem Three hot-path inefficiencies causing excess CPU and memory allocations: ### 1. `filterTxSlice` starts with nil slice `filterTxSlice` is called on the full `s.packets` slice (50k+ packets) for every query that doesn't hit a fast-path index. Starting with `var result []*StoreTx` means Go's append does ~15 growth+copy cycles (1→2→4→8→...→32768→65536) before reaching steady state. ### 2. `resolvePathForObs` allocates per hop Each hop in the path resolution loop allocates a new `ctx` slice (`make([]string, len(contextPKs), len(contextPKs)+2)`). For a 5-hop path, that's 5 allocations per observation. With 500+ observations per ingest batch, that's 2500+ small allocations. ### 3. `estimatedMemoryMB` calls `runtime.ReadMemStats` without caching `runtime.ReadMemStats()` triggers a STW (stop-the-world) pause. It's called from stats/debug endpoints (`GetStoreStats`, `GetPerfStoreStats`) that may be polled frequently. The routes.go layer already caches this with a 5s TTL, but the store layer doesn't. ## Fix 1. **Pre-allocate `filterTxSlice`**: `make([]*StoreTx, 0, n/2)` — the 2x over-allocation is cheaper than repeated growth+copy. 2. **Reuse ctx buffer**: Allocate one `ctx` buffer before the hop loop, reset to base length each iteration with `ctx = ctx[:ctxLen]`. 3. **Cache `ReadMemStats`**: 5-second TTL cache matching the routes.go pattern. Uses a package-level mutex (not on `PacketStore`) to avoid adding a field. ## Testing - `go build` passes - No behavior change — same results, fewer allocations --------- Co-authored-by: Joel Claw <358739783+Joel-Claw@users.noreply.github.com> |
||
|
efitenandClaude Opus 5
|
a8e8449170 |
test(#1616): wait for swatch focus to move instead of reading it immediately (#1939)
This is the test that has been keeping master red on both sides of today's queue run. ## The failure ``` ✗ ArrowRight cycles focus across swatches: ArrowRight should move focus to next swatch (was #ef4444, now #ef4444) ``` Observed on: | where | when | |---|---| | master push `589fa987` | 2026-08-31 — the last completed master run before today | | master push `859173f1` | 2026-09-02 — the first completed master run after #1938 | | PR #1884 | 2026-09-02, passed unchanged on a re-run | **Two out of two completed master runs.** Master has produced exactly two finished pipelines since 2026-08-31 and this test failed both, which is why the branch has had no green badge either side of a day of merges. ## The cause ```js await page.keyboard.press('ArrowRight'); const nextColor = await page.evaluate(() => document.activeElement.getAttribute('data-color')); ``` It reads `document.activeElement` on the tick after the key press. The keydown handler moves focus, but under CI load that can land after the evaluate has already run, so the assertion compares the swatch against itself and reports the same colour twice. **This file already knows about this.** The "outside click" step below carries a long comment about exactly this macrotask race for #1317, and the conclusion there was to wait on the real condition instead of a proxy. That step got the treatment and this one did not. ## The fix Wait for `activeElement` to be a `.cc-swatch` whose `data-color` differs from the one focused before the key press, with a 3s budget. The wait is wrapped so that a timeout falls through to the original assertion, which then reports the value actually observed rather than a bare Playwright timeout — a failing test should still say what it saw. **No product code changed.** One file, +18 lines, all of it the wait and the reasoning. ## What this does not claim It does not prove the focus handler is correct, only that the test stops racing it. If ArrowRight is ever genuinely broken, this still fails, and now with a useful message. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
9e13e0b05f |
feat(ingestor): full-packet RF observations from mobile clients (#1905)
## What A CoreDrive RX drive already carries far more RF information than reaches CoreScope, and it was being discarded twice: once in the mobile app (every packet it could not attribute to a directly-heard node was dropped before queueing) and once here (the ingestor decodes the *complete* packet, then keeps only `heard_key`/`snr`/`rssi`/`lat`/`lon`). This captures what was being thrown away, at **zero extra airtime** — nothing new is transmitted. - **`transmissions.code1` / `code2`** — the transport codes were decoded on every packet and used only to derive `scope_name`, then dropped. Storing them turns "which repeater forwards which scope" from a re-parse into a query. - **An async backfill** re-parses the `raw_hex` already on disk, so months of scope history become queryable with no new data collection. - **`client_rx_observations`** — a new diagnostic table holding every decodable packet a phone heard, with route type, transport codes, scope name, path-hash size, the full forwarder chain and the forwarder. ## Why it is safe for existing deployments Both halves are **opt-in and default off** (`clientRxObservations.enabled`, and `fullRfLog` on the app side), so an existing deployment sees no behaviour change and no volume change on upgrade. The coverage invariant is untouched: `client_receptions` keeps its rule — 0-hop advert pubkey or FLOOD `path[last]`, ≥2-byte hash — and an unattributable packet writes **zero** coverage rows. `deriveHeardKey`, `buildClientReception` and `InsertClientReception` are unmodified except for one guard described below. ## Performance justification (touches the ingest hot path) - **Backfill:** keyset-paginated by `id` in 5000-row batches, a single forward scan, `rows.Close()` before `Begin()` so it never deadlocks against `SetMaxOpenConns(1)`, and commits per batch so live ingest interleaves. Termination is driven by rows *scanned*, not rows decoded — an earlier count-based loop would have stopped at the first batch containing an undecodable row and then written its completion guard, permanently stranding the rest. - **Guard row is written if and only if the loop ran to genuine exhaustion.** Every error path leaves it unwritten so the next startup retries. - **Per-packet cost:** one extra INSERT on the client topic when enabled, gated behind an opt-in flag. No new work on the observer path. - **New indexes** cover the prune (`rx_at`), the flood-grouping (`pkt_hash, rx_at`), the per-repeater query (`forwarder, rx_at`) and the scope query (`scope_name, rx_at`). Retention has its own shorter window — this table is diagnostic, not archival. ## Two firmware-derived correctness points - **`pkt_hash` is `ComputeContentHash()`**, byte-identical to `transmissions.hash`, so dark-traffic queries are a plain equality join rather than a translation layer. - **TRACE packets are refused.** TRACE repurposes the header path bytes as per-hop SNR values, so deriving a `heard_key` from them invents a node that never existed. `packetpath.PathBytesAreHops` existed but was never wired into the client path; it became reachable only because the app half now publishes packets it previously dropped locally. ## Testing Full ingestor suite green. Notable coverage: a FLOOD-routed TRACE writes zero coverage rows and NULL `forwarder`; a `direction: "tx"` message writes no observation; a DIRECT route never sets `forwarder`; two forwarder copies of one flood remain two rows; the backfill's multi-batch path is exercised with an undecodable row in the first page; and a forced error asserts the migration guard stays unwritten. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
376c3e9f4a |
fix(packets): surface the transport region scope — detail pane row and a sortable Scope column (#1894)
## Summary `transmissions.scope_name` (#899) reached the database but never reached the UI. Two problems, one dead feature and one missing surface. ## 1. The detail pane's Scope row was dead `public/packets.js:3279` has rendered a **Scope** row since #899, gated on `pkt.scope_name != null`. It never fires in practice. `/api/packets` and `/api/packets/{id}` are served from the in-memory `PacketStore`. The store reads `scope_name` out of SQLite fine (`store.go:888`, `chunked_load.go:551` → `StoreTx.ScopeName`), but `txToMap()` did not put it in the JSON. Only packets old enough to have been evicted from the store — and thus served by the SQLite fallback in `db.go`, which does emit it — could ever show a scope. Verified against a live instance before the fix: ``` GET /api/packets/552e9687f1525537 → packet keys: ['_parsedPath','decoded_json','direction','first_seen','hash','id', 'observation_count','observations','observer_iata','observer_id', 'observer_name','path_json','payload_type','raw_hex','route_type','rssi','snr','timestamp'] ``` No `scope_name`. ### The NULL / "" distinction `StoreTx.ScopeName` was typed `string`, which collapses the two states the frontend distinguishes: | DB value | Meaning | UI | |---|---|---| | `NULL` | not transport-scoped | row hidden | | `""` | transport-scoped, region matched no configured key | muted "unknown scope" | | `"#be"` | matched region | the region name | `route_type` is **not** a usable proxy for that distinction: the ingestor writes NULL for a transport route whose `transport_code_1` is `0000` (`cmd/ingestor/db.go:1576` — `IsTransportScoped = route_type IN (0,3) AND Code1 ≠ "0000"`). So the field is now `*string`, with `nullStrPtr` preserving what `nullStrVal` collapsed. The two internal consumers (`TransportedScopes` #1751, `relayEntry.scope`) only care about non-empty named scopes and are unchanged in behaviour. ## 2. New: a Scope column on the packets table The scope was only reachable one packet at a time by opening the detail pane. It now has its own sortable column between Type and Observer, visible by default. The default view is **Group by Hash**, served by mappers that did not carry `scope_name` at all — so the column would have been empty in exactly the view most people look at. Both grouped paths now select and emit it: `groupedTxsToPage` in the store, and the dedicated grouped query in the DB fallback (v3 and legacy shapes). Rendering lives in `scopeCellHtml` (`public/app.js`, next to `transportBadge`) and is used on all three row-render sites — group header, expanded children, flat rows — so the column and the detail pane cannot drift apart. **Sorting** pins the empties last in both directions, as the nodes table already does for `default_scope`. Only ~8% of packets carry a scope, so an ascending sort would otherwise bury every scoped row under a wall of dashes. **Filtering**: `packet-filter.js` gains a `scope` field, so the cell is click-to-filter like Type and Observer, and `scope == "#be"` works in the filter bar. **Column prefs**: a `packets-known-cols` companion key. The `packets-visible-cols` array alone cannot distinguish "this column did not exist when you saved" from "you unchecked it", so any new column arrives silently hidden for every returning visitor. Keys absent from `known-cols` get the default treatment; keys the visitor actually hid stay hidden — there is a test for that second half specifically. ## Tests Each watched fail first. **Go** (`cmd/server/packet_scope_name_test.go`) - `txToMap` unit tests for all three states, including a JSON round-trip so a typed nil `*string` cannot pass as `null` - end-to-end through `/api/packets/{hash}` - `groupedTxsToPage` unit + end-to-end through `/api/packets?groupByHash=true`, across **both** the store-backed and DB-fallback paths - `transported_scopes_1751_test.go`: the "no scope" guard now covers both non-values (nil and a pointer to `""`) **Frontend** - `test-frontend-helpers.js`: `scopeCellHtml` three states + escaping - `test-packet-filter.js`: `scope` matching, case-insensitivity, and `FIELDS` registration - `test-packets-scope-column.js` (new Playwright e2e): header position, default visibility, one cell per row, em dash on non-transport rows, empties-last sorting, the Columns toggle, and the prefs backfill ## Verification Deployed and checked against a live instance: ``` /api/packets?groupByHash=true&limit=500 → scope_name present on 500/500, 59 with a matched region, 1 unknown-scope test-packets-scope-column.js → 7 passed, 0 failed cd cmd/server && go test ./... → ok ``` Two pre-existing failures, unrelated and equally red on an unmodified checkout: `test-e2e-playwright.js` "Customizer open does not overwrite server home config" and `test-observer-iata-1188-e2e.js` (timeout on `[data-loaded="true"]`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c598abd210 |
chore(#1859): pin eslint@8 + the lock file it needs (continues #1880) (#1937)
Continues #1880 by @SaarMesh-Bot. Their commit is unchanged and keeps their authorship; I added the lock file it was missing. ## What was wrong #1880 added `eslint@^8.57.1` to `devDependencies` but not to `package-lock.json`. CI runs `npm ci --production=false`, which requires the two to be in sync, so the pipeline died at **Install npm dependencies** before a single test ran: ``` npm error code EUSAGE npm error `npm ci` can only install packages when your package.json and npm error package-lock.json are in sync. npm error Missing: eslint@8.57.1 from lock file npm error Missing: @eslint-community/eslint-utils@4.10.1 from lock file ``` I approved that PR on 2026-08-30 on the grounds that it was two lines of devDependency with no runtime effect. I checked that `.eslintrc.json` exists so `npm run lint` would resolve, and did not check the lock file. That was the miss. ## What this adds One commit: the regenerated `package-lock.json`. Generated with `npm install --package-lock-only`, so `node_modules` was never touched and the change is confined to the lock file. The 13 removed lines are npm reorganising the existing `find-up` / `find-cache-dir` entries because eslint brings its own versions of them; nothing unrelated was bumped. ## Verification | command | result | |---|---| | `npm ci --production=false` | **added 394 packages**, exit 0 | | `npm run lint` | exit 0, **92 problems (0 errors, 92 warnings)** | The warnings are pre-existing unused-variable reports across `public/*.js`. Not addressed here: the point of this PR is that the command runs at all, and whether to act on 92 warnings is a decision for #1859 rather than something to smuggle in behind a lock file. ## Why this matters beyond itself #1881 is the other half of #1859 and adds the gofmt/go vet CI gate. It has been sitting at the end of the merge order all day because it forces a rebase on every open Go PR. It also should not land before this one: the `lint` script it complements is not installable until the lock file is right. @SaarMesh-Bot — your work, your credit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
859173f145 |
ci: unblock the master pipeline — gate the staging deploy, detach the badges job (#1938)
Fixes the problem @sylr reported on #1922. ## What is broken **master has produced no completed pipeline result since 2026-08-31.** Thirty-plus runs today alone were cancelled or left queued, while `go-test`, `e2e-test` and `build-and-publish` were passing inside them. The `deploy` job runs on `[self-hosted, meshcore-runner-2]`, and no such runner has picked up a job since at least 2026-08-31. It has no `timeout-minutes`, so it sits queued indefinitely. That job holds its run open, the run holds the concurrency group `ci-refs/heads/master`, and GitHub then cancels every subsequent master push while one waits. Two runs showing it, one from yesterday and one from right now: ``` |
||
|
|
25090230f2 |
fix(map): render the Esri labels overlay it was already named for (rebase of #1917) (#1935)
Continues #1917 by @nullrouten0. The commit is theirs, authorship unchanged; I only rebased it onto master. It went CONFLICTING because #1891 (the OpenTopoMap and USGS layers) landed in the same `BASE_STYLES` block, and both PRs also add cases to `test-issue-1420-tile-providers.js`. Resolution: kept #1891's two `usgs-*` entries and took this PR's `esri-darkgray-labels` line, which is the one that adds `refUrl`. Both test suites kept in full. Nothing else touched. Verified: `test-issue-1420-tile-providers.js` 47 passed, 0 failed, which includes this PR's four Esri cases and the Carto key cases that landed since. My review stands: approve. The id `esri-darkgray-labels` promised labels the layer control never stacked, and the test asserting that single-layer providers stay bare tile layers is the part that makes this safe to merge. Co-authored-by: nullrouten <nullrouten@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandJonathan Herlin
|
89544b1d08 |
perf: index, cache, and deflake /api/channels queries (rebase of #1887) (#1936)
Continues #1887 by @Jonher937. The commit is theirs, authorship unchanged; I only rebased it onto master. It went CONFLICTING because #1934 (prepared statements, originally @Joel-Claw's #1878) landed in the same `DB` struct. Both PRs add fields there and this one also replaces the single-slot channels cache. Resolution: kept this PR's keyed caches (`channelsCache`, `encChannelsCache`, `msgCache` plus their entry types and TTL constants) and kept master's thirteen prepared-statement fields alongside them. The old single-slot `channelsCacheKey`/`channelsCacheRes`/`channelsCacheExp` trio is gone, which is the point of this PR. Nothing else touched. Verified: `cmd/server` builds and the **full suite passes**, not just the channel tests. My review stands: approve, with two questions that do not block and are worth a look at some point. 1. `msgCache` is keyed by `hash|limit|offset|region`, and `offset` grows without bound as someone pages through a channel. Each entry also holds a full page of message maps, so a full 256-entry cache at `limit=50` holds around 12,800 maps. The other two caches are keyed by region only and genuinely low-cardinality as your comment says; this one is the odd one out. 2. `getMsgCache` returns the cached slice directly, so every hit hands the caller the same message maps. If any handler mutates one before serialising, it corrupts the cache for the next ten seconds. Same class as the finding on #1871, which was fixed there by copying at the two broadcast sites. Co-authored-by: Jonathan Herlin <jonte@jherlin.se> |
||
|
Jonathan Herlin
|
eb8f376c6c |
fix: use index from_pubkey in nodes region filter (#1882)
The region subquery in GetNodes was pulling the advert pubkey out of decoded_json with JSON_EXTRACT for every row the join touched, instead of reading the from_pubkey column that #1143 already added and indexed It looks like buildPacketWhere, GetRecentTransmissionsForNode, QueryMultiNodePackets etc. moved to from_pubkey already, but not this. |
||
|
Joel ClawandJoel Claw
|
4a776454ca |
perf: remove dead relayTimes field (#1931)
The `relayTimes` field (`map[string][]int64`) on `PacketStore` is never written to and never read. Its only references are the declaration at `store.go:180` and the `make()` in `NewPacketStore` at `store.go:644`. `relay_liveness_test.go` looks like a user at a glance but builds its own local `idx := make(map[string][]int64)` and passes that to `addTxToRelayTimeIndex`; the string "relayTimes" there is only inside a `t.Error` message. This is the surviving fragment of #1872, which no longer compiles after #1855 removed `lastSeenTouched` and `touchRelayLastSeen` from master. Verified against current master: both references are gone, build passes, all tests pass (32.2s). Co-authored-by: Joel Claw <358739783+Joel-Claw@users.noreply.github.com> |
||
|
Jonathan Herlin
|
a46a6d55eb |
fix: use replaceState for traces/roles redirects to preserve back-button history (#1883)
fix: use replaceState for traces/roles redirects to preserve back-button history The #/traces/<hash> and #/roles backward-compat redirects used location.hash = ..., which pushes a new history entry instead of replacing the current one. This trapped users navigating back from a trace view: the intermediate #/traces/<hash> entry would immediately re-redirect forward again on hashchange, so back button never reached the packets view they came from. |
||
|
efitenandClaude Opus 5
|
f081f91b88 |
fix(#1904): keep resolved full-pubkey hops across a path-hop index rebuild (#1907)
Fixes #1904. ## The bug `buildPathHopIndex` reassigned `s.byPathHop` to a fresh map and refilled it from every packet's raw `path_json` hops: ```go func (s *PacketStore) buildPathHopIndex() { s.byPathHop = make(map[string][]*StoreTx, 4096) for _, tx := range s.packets { addTxToPathHopIndex(s.byPathHop, tx) // raw hops only } ... } ``` `byPathHop` carries two kinds of key, though: those raw wire hops, and the resolved full pubkeys fed per observation by `indexResolvedPathHops`. The pubkey strings behind the second kind are retained nowhere — #800 replaced the per-`StoreTx` `ResolvedPath` field with a hash-only membership index (`resolvedPubkeyIndex` stores FNV hashes, not strings) — so the rebuild could not reproduce them and dropped them. All three call sites run post-load: `LoadChunked` (`chunked_load.go:459`), the background fill loader (`store.go:1573`), and the deferred startup build (`index_ready_1008.go:177`). The `resolved_path` branch of the chunk scan populates the index and is then silently undone a few hundred lines later, while the `resolved_path IS NULL` fallback right beside it is explicitly documented as "byNode ONLY — the resolved_path/path-hop indexes must NOT be populated here". The two branches disagreed about who owns the index. Consequence: after a cold start every lookup keyed by a node's full pubkey missed, so `relay_count_1h/24h`, `last_relayed`, `unscoped_relay_count_24h`, `transported_scopes` (#1751) and the usefulness Traffic axis all read zero until live ingestion slowly refilled the index. ## Evidence Fixture built from live data: 2512 nodes, 17,056 transmissions, 528,891 observations, 123,057 of them carrying a non-NULL `resolved_path`. ``` before [store] Built path-hop index: 2924 unique keys /api/nodes → 0 of 2000 nodes with transported_scopes 0 with relay_count_24h > 0 after [store] Built path-hop index: 3881 unique keys (172181 resolved-hop entries retained) /api/nodes → 726 with transported_scopes 741 with relay_count_24h > 0 ``` The 957 extra keys are the full pubkeys. ## The change `retainResolvedPathHops` re-merges the pre-rebuild map's entries that the raw-hop pass cannot reproduce. Entries are carried over **only for transmissions still in `s.packets`**. That filter is load-bearing rather than defensive. `removeTxFromPathHopIndex` strips raw hops only — it derives them from `txGetParsedPath` — and its companion `removeFromResolvedPubkeyIndex` cleans the hash index, not `byPathHop`. So evicted transmissions linger under their resolved keys, and the wipe this PR removes was the only thing that ever cleared them. Filtering on liveness keeps the index bounded by the eviction policy instead of converting that gap into a permanent leak. `TestBuildPathHopIndex_DropsResolvedHopsOfEvictedTx_1904` pins it. (The eviction gap itself is pre-existing and outside this change: between rebuilds, an evicted transmission still stays referenced under its resolved keys. Filed separately.) ## Perf `O(entries in prev)` with one scratch map reused across keys (`clear()` per key, the same idiom as `hopsSeen`), plus one `map[*StoreTx]struct{}` over `s.packets` for the liveness check. It runs only where `buildPathHopIndex` already ran — cold load and background-fill completion — never on an ingest or request path. Measured on the fixture above: index build stayed within the same `LoadChunked` step, 15.2s total for 17k transmissions / 527k observations. Memory: the retained entries point at transmissions already held by `s.packets`, so no `StoreTx` is kept alive beyond eviction; the cost is map/slice overhead for keys that the feature is supposed to have. ## Tests `cmd/server/pathhop_rebuild_1904_test.go`, red before / green after: 1. `TestBuildPathHopIndex_RetainsResolvedHops_1904` — a resolved full-pubkey key survives the rebuild alongside the raw hop. 2. `TestBuildPathHopIndex_DropsResolvedHopsOfEvictedTx_1904` — a resolved key whose transmission is no longer in `s.packets` is dropped, and the now-empty key is not left behind. 3. `TestBuildPathHopIndex_NoDuplicateOnRepeatedBuild_1904` — building twice does not double-append (`indexResolvedPathHops` dedups within a call, not across the several observations of one transmission, so `prev` can legitimately contain duplicates). ``` cd cmd/server && go test ./... ok github.com/corescope/server 85.5s go vet ./... clean ``` Frontend and ingestor suites are untouched by this change (Go server only, no `public/` files). ## Interaction with #1903 Both touch `byPathHop` semantics, so I verified them composed on the same fixture. With #1904 alone the resolved keys come back and #1902's prefix collision is plainly visible again (51% of 1-byte prefix groups reporting an identical scope set). With both: ``` f79616 BE repeater ['#be','#de','#eu','#nl'] relay24h=542 f752c2 DE/NRW repeater ['#de','#de-nw'] relay24h=343 f788ad BE repeater none relay24h=383 ``` Identical-set prefix groups fall to 8%, relay counts stay intact, and each node's scopes match what its own `resolved_path` rows say. The two changes are independent and compose cleanly. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandJoel Claw
|
2f711eb851 |
perf: use prepared statements for frequently-called server DB queries (rebase of #1878) (#1934)
Continues #1878 by @Joel-Claw, at their request. Both commits are theirs, authorship unchanged; I only rebased them onto master and resolved the conflict with #1909. ## The conflict, and how it is resolved Exactly the two places I named in the review on #1878: `OpenDB` and `Close()`. Both PRs rewrite them, and #1909 went first because it is the correctness fix. **`OpenDB`** — kept #1909's pinned-connection `detectSchema` and added this PR's `prepareStatements()` after it: ```go derr := d.detectSchema(ctx, sc) _ = sc.Close() if derr != nil { conn.Close(); return nil, fmt.Errorf("schema detection failed: %w", derr) } // Statements are prepared after schema detection so they can never be // compiled against a schema mode that turned out to be wrong (#1901). if err := d.prepareStatements(); err != nil { ... } ``` The ordering matters and is not arbitrary: preparing before detection would compile statements against a schema mode that #1909 exists to stop trusting. **`Close()`** — kept this PR's statement closing and **did not** restore the WAL checkpoint. #1909 removed it deliberately: the handle is `mode=ro`, so `PRAGMA wal_checkpoint(TRUNCATE)` can only ever fail with "disk I/O error (778)" and was emitting a misleading storage-fault line on every shutdown. That reasoning survives; the statement closing is added in front of it. ## Verification - Both commits cherry-picked onto `e5595ad9` - `cmd/server` builds - **Full `cmd/server` suite: ok, 0 failures** (not just the targeted DB tests — after master briefly went red today from a two-PR interaction, a full local run seemed worth the two minutes) ## Review points still open, none blocking From my review on #1878, unchanged by the rebase: 1. Every SQL string now exists twice, once prepared and once as the `stmtQueryRow` fallback literal, with nothing keeping them in sync. The fallback is genuinely needed — twelve test helpers build `&DB{conn: ...}` directly and never call `prepareStatements` — but a constructor for those helpers would remove the duplication. 2. `stmtCountObsLastHour` and `stmtCountObsLastDay` are byte-identical SQL. 3. `OpenDB` now refuses to start rather than degrading when a Prepare fails. Contained today, since none of the 13 prepared queries touch a schema-conditional column, but the failure mode changed. @Joel-Claw — your work, your credit. Ping me if you would rather take it back. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: Joel Claw <358739783+Joel-Claw@users.noreply.github.com> |
||
|
efitenandClaude Opus 5
|
aabbd50d27 |
feat(nodes): export the visible node list as MeshCore companion contacts JSON (#1889)
## What
Adds an **Export JSON** button to the Nodes topbar that downloads the
currently visible node list as a MeshCore companion-app config file:
```json
{
"contacts": [
{
"type": 2,
"name": "BE-MGU-RP03 | ON7YT",
"custom_name": null,
"public_key": "7a7a37d4819fb27440ed1439ca7d281fdecceb83b27e61a69745feb004d726a4",
"flags": 0,
"latitude": "51.07307",
"longitude": "5.5796",
"last_advert": 1786545431,
"last_modified": 1786545431,
"out_path_list": null
}
]
}
```
That is the exact shape the companion app itself writes, so the file
imports straight back into a companion app as contacts. The practical
use case is per-area: pick an area in the area filter, export, and hand
someone the repeaters for that region instead of having them wait for
adverts.
## Scope of the export
WYSIWYG — the button exports the rows the table is showing, in the
table's current order, so area, region, role tab, search, last-heard and
status filters all carry over. The button label shows how many contacts
the file will contain and is disabled when that count is zero.
## Field mapping
| JSON field | Source | Notes |
|---|---|---|
| `type` | `node.role` | repeater→2, companion→1, room→3, sensor→4;
unknown/empty→1 |
| `name` | `node.name` | unchanged, emoji included |
| `custom_name` | — | always `null` |
| `public_key` | `node.public_key` | full 64-hex |
| `flags` | — | always `0` |
| `latitude` / `longitude` | `node.lat` / `node.lon` | stringified, as
the format expects |
| `last_advert` | `node.last_seen` | RFC3339 → unix seconds |
| `last_modified` | mirrors `last_advert` | no separate source exists |
| `out_path_list` | — | always `null`; the companion app discovers
routes itself |
Nodes are skipped when they have no name, a pubkey shorter than 64 hex
chars, or no usable position (missing, non-numeric, or null island).
Filename: `corescope_nodes_<area|all>_YYYY-MM-DD-HHMMSS.json`.
## Shape of the change
The mapping lives in a self-contained `public/nodes-export.js`
(`window.NodesExport.buildContacts/filename/download`); `nodes.js` only
gains the button markup, a click handler and a small
`updateExportBtn()`. No backend change, no new API call — the export
reuses the already-fetched node list, so there is nothing per-node to
fetch.
## Tests
- `test-nodes-export.js` — field mapping and key order, role→type table,
skip rules, order preservation, filename format (added to
`test-all.sh`).
- `test-nodes-export-wiring.js` — `index.html` loads the module before
`nodes.js`; the button lives in the topbar and hands the filtered
`nodes` array plus `AreaFilter.getSelected()` to
`NodesExport.download()` (added to `test-all.sh`).
- `test-nodes-export-e2e.js` — Playwright: downloads the file, validates
the JSON shape per contact, asserts the button count matches the file,
and that narrowing the search shrinks the export set.
## Browser validation
Deployed to staging and checked in Chromium:
- Desktop 1400×900 — button renders at the right of the topbar next to
the count pills, `Export JSON (1962)`.
- Mobile 390×844 — topbar stacks, button visible, no horizontal overflow
(`scrollWidth == clientWidth`).
- E2E against staging: 1761 contacts exported,
`corescope_nodes_all_2026-08-12-164349.json`, shape validated, search
narrowing confirmed.
- With the `BE-LIM` area filter active: 44 contacts,
`corescope_nodes_BE-LIM_2026-08-12-164445.json`, type histogram `{1: 1,
2: 42, 3: 1}`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
efitenandClaude Opus 5
|
d821d9a390 |
feat(retention): add observerPurgeDays hard-delete for long-inactive observers (#1886)
## Problem `RemoveStaleObservers` only soft-deletes — it sets `inactive = 1` and the row stays forever. On a long-running deployment those rows just accumulate: on a two-year-old instance roughly 25% of the `observers` table was rows nobody can ever see again. There is currently no way to reclaim them. ## Fix A second retention stage. `PurgeStaleObservers` hard-deletes rows that are: - already `inactive = 1` (so the soft-delete stage owns the decision of *when* an observer goes stale), **and** - older than `retention.observerPurgeDays`, **and** - referenced by nothing. New config field `retention.observerPurgeDays`, default `0` = disabled. Existing deployments are unaffected until they opt in. Set it above both `observerDays` and `packetDays` — below those the reference guards keep every candidate row anyway. ## Why the reference guards are the point `observations.observer_idx` is a bare rowid with no foreign key. Deleting a still-referenced observer silently orphans history — `packets_v` stops resolving the observer and those packets get mis-attributed. Nothing errors; the data just quietly goes wrong. So the statement guards on all three referencing tables: ```sql AND NOT EXISTS (SELECT 1 FROM observations o WHERE o.observer_idx = observers.rowid) AND NOT EXISTS (SELECT 1 FROM observer_metrics m WHERE m.observer_id = observers.id) AND NOT EXISTS (SELECT 1 FROM dropped_packets d WHERE d.observer_id = observers.id) ``` This is correctness, not defensive padding — it was found the hard way, by orphaning 280 observation rows during a manual purge that skipped one of these checks. Each guard has its own test. ## Performance Each `NOT EXISTS` is an index seek per candidate row (`idx_observations_observer_idx`, `idx_dropped_observer`, the `observer_metrics` PK), and `observers` is O(100). It runs on the existing daily retention tick alongside `RemoveStaleObservers`, never on the ingest path. ## Tests Eight tests in `cmd/ingestor/observer_purge_test.go`, written before the implementation: - deletes an unreferenced stale row - keeps a row referenced by `observations` — and asserts zero orphans afterwards - keeps a row referenced by `observer_metrics` - keeps a row referenced by `dropped_packets` - keeps a row that is old enough but still `inactive = 0` - keeps a row inside the retention window - no-ops when disabled (`0` and `-1`) - config accessor table test ## Invariant Writes stay in `cmd/ingestor` per #1283. `cmd/server/readonly_invariant_test.go` now also forbids `PurgeStaleObservers` as a method on the server's `*DB`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
TeTeHackoandClaude Opus 5
|
97b6090344 |
fix(hash-size): key the zero-hop advert skip on the path byte, not the route type (#1913)
## Summary `computeNodeHashSizeInfo` skips zero-hop direct adverts by **route type**. It should skip them by the **content of the path byte**, because the two cases are no longer the same thing. A zero-hop direct advert carries no path, so its hop count is 0. Whether the two size bits next to it mean anything depends on the sender: - Firmware that predates [meshcore-dev/MeshCore#3293](https://github.com/meshcore-dev/MeshCore/pull/3293) does `packet->path_len = 0` in `Mesh::sendZeroHop()`, wiping the whole byte including the size bits. `0x00` genuinely says nothing about the node's `path.hash.mode` — skipping it is right, and #649 was right. - A sender that writes the size through `setPathHashSizeAndCount()` emits `0x40` (2 bytes) or `0x80` (3 bytes) with a zero hop count. On a zero-hop packet nothing else can set those bits, so they are a deliberate declaration. #653 landed the skip as `pathByte & 0x3F == 0`, which swallows the second case too. The diagnosis in #649 had actually proposed `pathByte == 0x00`; the review widened it on the reasoning that a zero hop count always implies zeroed size bits. That was true in April, when no firmware wrote them. It is not true now. On the Czech mesh (869.4 MHz), a 24h window of 10k packets holds **54 zero-hop direct adverts: 39 at `0x00` and 15 carrying a declared size** (14× `0x40`, 1× `0x80`). ## Why it matters for display, not just tidiness Measured on one node over a 7-day window. A companion was reconfigured from a 2-byte to a 3-byte path hash. Its first advert under the new setting was a zero-hop direct one on **24 Aug 15:36 UTC** declaring `0x80`. That packet was dropped, so the node kept reading as 2-byte until its next **flood** advert arrived on **25 Aug 10:18 UTC** — 18h42m serving a configuration the analyzer had already been told was stale, confirmed against both an unpatched and a patched instance. With local adverts typically every 2h and flood adverts every 25h, that gap is the normal case rather than a corner one. It bites hardest on an instance whose retention window is shorter than a flood advert interval: there the node has *no* countable advert at all and falls out of `hash_size` entirely (which is what #1912 is about on the rendering side). ## Change `(pathByte & 0x3F) == 0` → `pathByte == 0x00`, in `computeNodeHashSizeInfo` and in `computeAnalyticsHashSizes` so the two views agree. `isZeroHop` renamed to `isUndeclaredZeroHop` in the latter, since that is now what it means. No complexity change — same single byte comparison inside the existing scan. ## Measured A/B Two builds of the **same commit**, one with the change, both run read-only against the same copy of a real 181k-transmission / 973-node database: | | baseline | patched | |---|---|---| | nodes changed | — | **1** | | nodes regressed | — | **0** | | `hash_size_inconsistent` | 6 | **6** | | `multi_byte_status` split | 726 / 161 / 86 | unchanged | The flip-flop flag not moving is the point worth checking: a node that legitimately changes its mode mid-window is still handled by the recency decay from #1788, so reading these packets does not resurrect false "varies". ## Tests `cd cmd/server && go test ./...` → **ok**, 0 failures. Coverage 83.5%, unchanged from master. 5 new cases in `cmd/server/zerohop_hashsize_test.go`, two built from real off-air packets: - zero-hop DIRECT `0x40` → `HashSize 2` (was: dropped) - zero-hop DIRECT `0x80` → `HashSize 3` - zero-hop DIRECT `0x00` → still absent from the map, i.e. #649's behaviour preserved - TRANSPORT_DIRECT at path-byte offset 5, declared vs wiped - the declared size reaching `computeMultiByteCapability` as `confirmed`, which is what the map's multi-byte overlay reads **One existing test changed, flagging it explicitly:** `TestHashSizeTransportDirectZeroHopSkipped` used `0x40` as its "should be skipped" fixture. It now uses `0x00` — the case it was written to cover, since #747 was about the missing `RouteTransportDirect` skip rather than about the size bits. The `0x40` case is covered by the new tests with the opposite expectation. ## Deliberately not touched The decoders (`cmd/server/decoder.go:648`, `cmd/ingestor/decoder.go:1045`) still report `HashSize 0` for these packets, so per-packet views keep showing the size as unknown. Arguably they should follow the same rule, but that changes packet display rather than node attribution and felt like a separate call for you to make. ## Caveat worth stating This attributes a declared size to the pubkey inside the advert. That holds as long as the advert was transmitted by the node that owns it — the same assumption the existing zero-hop **flood** path already makes, so this change does not widen it. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
e5595ad92f |
fix: unbreak master — decouple the pathTrust builder test from the default (#1932)
**master is currently red.** This is the fix.
```
--- FAIL: TestNeighborEdgesBuilderPathTrustExcludesOneByte
neighbor_builder_test.go:301: 1-byte hop must not produce an edge under
the default threshold, got 1
```
## What happened
Two PRs that were each green on their own:
- **#1929** moved `DefaultMinHashBytesForMapping` from 2 to 1.
- **#1930** carries `TestNeighborEdgesBuilderPathTrustExcludesOneByte`,
written when the default was 2.
Neither pipeline saw the other, because a `pull_request` run tests the
merge commit as it stood when that run started. Both merged, and the
combination fails. My mistake for merging them in the same batch without
re-running one against the other.
## The fix
The test passed `nil` for the trust config and leaned on the package
default being 2:
```go
// nil == package default (MinHashBytesForMapping = 2).
store.buildAndPersistNeighborEdges(nil)
```
That coupling is the real defect. The test is about what happens **at
threshold 2**, not about what the default happens to be. It now says so:
```go
trust := &packetpath.TrustConfig{MinHashBytesForMapping: 2}
store.buildAndPersistNeighborEdges(trust)
```
It keeps testing exactly what it was written to test, and stops breaking
when the default moves. The sibling
`TestNeighborEdgesBuilderPathTrustAllowsTwoByte` already passes its own
fixture explicitly, so this brings the two into line.
**No production code changed.** `cmd/ingestor` PathTrust and Neighbor
tests pass.
## Worth recording
This is the failure mode I have been flagging on other PRs all day, and
I walked into it myself: green CI on a PR is a statement about the base
it was tested against, not about master. Two PRs can each be green and
still be red together. Nothing about the review process would have
caught it — only re-running one against the other, or a merge queue,
would.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
efitenandArcan Consulting - Michael J. Arcan
|
0fd22039cb |
feat(map): Important Links overlay (rebase of #1771 onto master) (#1928)
Continues #1771 by @ArcanConsulting. Both commits are theirs, authorship unchanged; I only rebased them onto current master. Opening it here rather than force-pushing to someone else's branch. ## Why the rebase was needed #1771 went CONFLICTING through no fault of its author: #1760 landed first and both PRs append a line to `test-all.sh` at the same spot. That was the entire conflict. ## What I changed One line, and it is the conflict resolution: `test-all.sh` now runs **both** test files rather than either. ``` node test-repeater-metric-scatter.js # from #1760 node test-top-routes-overlay.js # from this PR ``` Nothing else was touched. `public/map.js` and `test-issue-1329-map-controls-accordion-e2e.js` are byte-for-byte as the author wrote them. ## Verification on the rebased tree | | result | |---|---| | `test-top-routes-overlay.js` (this PR's own) | 20 passed, 0 failed | | `test-repeater-metric-scatter.js` (#1760's, must still pass) | 31 passed, 0 failed | | `test-frontend-helpers.js` | 627 passed, 0 failed | ## The one review point that still stands From my review on #1771, unchanged by the rebase and not something I fixed on the author's behalf: `test-top-routes-overlay.js` extracts the ranking core by `indexOf`-slicing `public/map.js` between the literals `const TOP_ROUTES_AXES` and `function clearTopRoutes`, then `new Function`s the result. There is a guard assertion for the rename case, which is thoughtful, but it still breaks on any reordering of map.js and it tests a string rather than the module. Two PRs in this same queue do it properly and are worth copying: #1821 exports `applyObserverFilter` through `_packetsTestAPI`, and #1912 puts `hashPrefixInfo` on `window`. Happy to take that as a follow-up rather than block the overlay on it. @ArcanConsulting — this is your work and the credit is yours. Say the word and I will close this and hand the rebase back, or push it to your branch instead if you would rather #1771 stayed the vehicle. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: Arcan Consulting - Michael J. Arcan <github@arcan-it.de> |
||
|
|
e8f32df4dc |
feat(#1784): gate ingestor neighbor-edge creation on the path-trust threshold (rebase of #1863) (#1930)
Continues #1863. Three of the four commits are @Saarlandpower's and @SaarMesh-Bot's, authorship unchanged. The fourth is mine and is explained below. ## Why a rebase was needed #1863 was stacked on #1824, and #1841 merged instead. Both carried the same pathTrust base from different commits, which is why the two conflicted while each reported MERGEABLE against master. Cherry-picking #1863's own three commits onto master applied cleanly with no conflicts, which confirms its actual work was always independent of that duplicated base. ## The fourth commit, and a correction to something I got wrong The three commits do not build on master: ``` cmd/ingestor/main.go:455:23: cfg.GetPathTrust undefined (type *Config has no field or method GetPathTrust) ``` **#1824 added the pathTrust config and helper to both `cmd/server/config.go` and `cmd/ingestor/config.go`. #1841 carried only the server half** — one of its own commits is titled "remove ingestor side". I then closed #1824 as superseded by #1841, which is true for the server side and wrong for the ingestor side. Master has no pathTrust code in `cmd/ingestor/config.go` at all. The fourth commit restores that half, unchanged from `beae2c1c`: the `packetpath` import, the `PathTrust` field, the `PathTrustConfig` alias, `GetPathTrust`, and `cmd/ingestor/config_test.go` verbatim (28 lines covering the default, an explicit value, and a nil `*Config` receiver). That code is @Bjorkan's and @SaarMesh-Bot's from #1824, not mine; I only put it back. ## Verification - All three original commits cherry-picked onto `b3a306b8` with **no conflicts** - `cmd/ingestor` builds, and its `PathTrust|Neighbor|Config` tests pass - `cmd/server` `Neighbor|PathTrust|AnonReq|Edge` tests pass ## Interaction with #1929 #1929 moves `DefaultMinHashBytesForMapping` from 2 to 1. With that in, this PR's ingestor gate is a no-op by default and only takes effect when an operator sets `minHashBytesForMapping` to 2 or 3, which is the opt-in shape #1784 asks for. The two are complementary; merge order between them does not matter. @Saarlandpower @SaarMesh-Bot — your work, your credit. Say the word and I will close this and hand the rebase back, or push it to the #1863 branch if you would rather that stayed the vehicle. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: Saarlandpower <Mail@mathiaskasper.de> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> |
||
|
h4badger
|
34b41fd5b6 |
Add topographic map layers (#1891)
This adds two optional map tile providers: - OpenTopoMap - USGS They are disabled by default, but can be quite useful for visualizing repeater sites with terrain features visible. |
||
|
|
1720060284 |
fix(#1827): avoid per-observation SQL fetch in handleObserverAnalytics hot loop (#1829)
## Summary Fixes the CPU/DoS issue in #1827: observer detail pages were saturating CPU on busy observers — 6-7 concurrently loaded tabs pegged 12 cores for seconds, and auto-refresh made it self-sustaining. ## Root cause `handleObserverAnalytics` iterated every observation in the requested window and called `enrichObs()` per observation just to read `payload_type` and `decoded_json` for the `packetTypes`/`nodesTimeline` aggregates. `enrichObs()` also runs an on-demand SQL `SELECT resolved_path FROM observations WHERE id=?` (`fetchResolvedPathForObs`) and builds a full response map — both of which are unused by this aggregation loop. `resolved_path` is only actually consumed by the `<=20` kept `recentPackets` entries. Per the triage in #1827 (@carmack): *"Replacing `enrichObs(obs)` with a direct `s.store.byTxID[obs.TransmissionID].PayloadType` read (as sketched in the body) drops a map alloc + interface boxes per obs on the loop that saturated the operator's 12 cores. Byte-identical output. That's ~90% of the value."* This PR implements exactly that fast-path. ## Change - Aggregate loop (`packetTypes`, `nodesTimeline`): read `payload_type`/`decoded_json` directly off the transmission via `s.store.byTxID[obs.TransmissionID]` — no SQL, no per-obs map allocation. - `recentPackets` (`<=20` entries): unchanged, still calls `enrichObs()` since it needs `resolved_path`/`raw_hex`/etc. for display. - Output is unchanged: `packetTypes`/`nodesTimeline` are computed from the exact same underlying fields (`tx.PayloadType`, `tx.DecodedJSON`), just without the O(N) SQL round-trips. ## Scope This is the concrete hot-path fix from #1827's triage — not the broader `/api/observers/{id}/analytics` endpoint-split proposal in #1828, which (per that issue's discussion) is a separate P3 follow-up. #1828's own triage converged on this same `byTxID` fast-path as "the ground-work minimum" before any endpoint splitting. ## Testing - Existing `TestObserverAnalytics` passes unchanged. - Extended `TestObserverAnalytics/default` to assert `packetTypes` counts come out correct (`{"4":2,"5":1}` for the seeded fixture) via the new `byTxID` path, and that `recentPackets` still carries `resolved_path` where present (confirming the `enrichObs()` path for those 20 entries is untouched). - `go build ./...` and `go vet ./...` clean in `cmd/server`. - Full `go test ./...` in `cmd/server`: passes except 4 pre-existing test-order-dependent failures in `TestHandleNodePaths_*` (unrelated to this change — reproduced identically on a fresh, unpatched clone of `upstream/master`). --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
1441734991 |
fix(#1901): detectSchema fails loud instead of caching wrong schema mode (#1909)
Fixes #1901. Thanks to @MarekWo for the exceptionally thorough report — root cause, repro, and a prioritised fix checklist in one. This implements it. ## Problem `detectSchema()` swallowed any probe-query error with a bare `return`, so a single transient failure of the first `PRAGMA table_info(observations)` at startup left `isV3` (and the feature flags) at their zero value **for the entire process lifetime**. The server then ran v2 SQL against a v3 DB: Packets page empty, `/api/channels/<name>/messages` → 500, logs full of `no such column: o.observer_id`, while the database was perfectly healthy. Nothing re-checked the flag, so only a manual restart recovered it. ## Fix Works through the report's checklist: - **Don't swallow the error.** `detectSchema` now returns `error` and `OpenDB` aborts on it. `main.go` already `log.Fatalf`s on an `OpenDB` failure, so the supervisord/Docker restart policy retries and a transient cause clears on the next attempt — strictly better than serving a broken read API. - **Log the mode unconditionally** — `[db] schema mode: v3 (observer_idx)` / `v2 (observer_id)`. A clean startup log is now positive evidence detection ran, not just an absence of errors. - **Run detection on a single pinned connection** (`conn.Conn(ctx)`) rather than an arbitrary pooled one, so the startup race in the report's hypothesis can't quietly hand detection a fresh, not-yet-openable handle — if the connection can't be acquired, we fail loud. - **`Close()` no longer checkpoints the read-only handle.** `PRAGMA wal_checkpoint(TRUNCATE)` on a `mode=ro` connection always failed with `disk I/O error (778)` and looked like a storage fault on every shutdown (the report's aside). The ingestor (the writer) owns WAL checkpointing. The three near-identical PRAGMA scan loops are consolidated into one `schemaColumns()` helper that returns errors instead of ignoring `Scan` failures. ### On the "single source of truth" item The report suggests deriving `isV3` from `dbschema.TableHasColumn(...)`. I kept the PRAGMA-scan structure here because `detectSchema` sets six flags from three tables in a single pass; swapping to `TableHasColumn` would mean six separate probe calls and wouldn't actually be cleaner. The goal it was aimed at — never cache a false negative — is met by making the existing scan fail loud. Happy to switch to the single-probe-per-column shape if you'd prefer it. ### Honest note on the connection `conn.Conn(ctx)` pins *a* single connection for all four probes and fails loud if it can't be acquired; it does not guarantee the literal connection `Ping()` validated (`database/sql` doesn't expose that). The fail-fast is what actually closes the bug — a mis-detected schema aborts startup instead of persisting for the process lifetime. ## Tests - `TestDetectSchemaFailsLoudOnProbeError` — injects a probe failure through a `rowQuerier` and asserts the error propagates and `isV3` stays unset (the invariant the old bare-`return` violated). - `TestDetectSchemaV3AndV2` — covers both schema shapes through `OpenDB`. `go vet ./cmd/server` and `go build` are clean; targeted `go test -run 'DetectSchema|OpenDB'` is green. Heads-up on the full `go test ./cmd/server` run: a handful of `TestHandleNodePaths_*` / `TestHandleAnalytics*` tests return `503 index loading`, plus one intentional panic test — these fail identically on pristine `master` (`a06ac8ac`) with this branch stashed, i.e. they're pre-existing/timing-related and untouched by this change. Out of scope (per the issue): frontend behaviour when the API 500s. 🤖 Authored with [Claude](https://claude.com) · Co-Authored-By trailer on the commit. Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
c5a71b34ec |
fix(#1890): drop the hardcoded og:url so shared links stay on the instance (#1893)
Fixes #1890. ## The problem `public/index.html:16` shipped this to every deployment: ```html <meta property="og:url" content="https://analyzer.00id.net"> ``` Open Graph consumers — Facebook and Messenger among them — treat `og:url` as the canonical destination. Clicking the preview of a link shared from *any* CoreScope instance navigated to that one host. The direct link text still resolved correctly, which is why this went unnoticed; the preview card and the surrounding message body did not. It is the only occurrence in the frontend. ## The change Remove the tag. `og:url` is optional — with no tag present, consumers fall back to the URL they crawled, which is correct for every deployment and needs no configuration. ## Why not the config-driven variant The issue also proposes deriving the URL from `config.json`. I did not take that shape, on purpose: `index.html` is pre-processed **once at startup** — `spaHandler` reads it and substitutes `__BUST__` (`cmd/server/main.go:565`), then serves the same byte slice for every request. A correct per-host `og:url` therefore needs either a new public-URL config key or per-request templating of the index. Both are decisions about config surface and request-path cost that belong to you, and neither is needed to stop the redirect. Happy to follow up with whichever shape you prefer — this PR is the part that is unambiguous. ## What is left alone `og:image` still points at `raw.githubusercontent.com/Kpa-clawbot/corescope/master/public/og-image.png`. That is the project's own asset, a shared project resource rather than a redirect target, so it is correct for every instance to reference it. ## Test `test-issue-1890-og-url.js`, a static scan, registered in `test-all.sh`: - no `og:url` meta tag - no `rel="canonical"` link - no `00id.net` reference anywhere in `index.html` - `og:title` / `og:description` / `og:image` still present That last assertion is deliberate: without it the guard could be satisfied by deleting the whole embed block. Watched fail first — 2 passed, 2 failed before the change, 4 passed after. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jonathan Herlin
|
647841c990 |
fix: stop watchdog force-reconnect from racing paho's own retry loop (#1897)
Relates to #1335, which was already closed by PR #1336 shipping the naive `client.Disconnect(250); client.Connect()` force-reconnect. That fix has its own bug: liveness.IsConnectedFn (paho's IsConnected()) reports true for the entire time paho is actively retrying, not just when genuinely connected, so the watchdog's stall check cannot tell a half-open TCP socket (the original #1335 case) from a broker that paho is already correctly reconnecting to. Unconditionally calling Disconnect(250) then Connect() on that second, transitional case races paho's status machine and permanently kills its retry loop, requiring another watchdog trigger to recover, sometimes compounding into 100+ minute outages. This is a different failure mode from #1749/PR #1853: that bug is a blocking log.Print() write freezing the entire watchdog loop before ForceReconnectFn is ever called. This bug only manifests once ForceReconnectFn does fire, so the two fixes are independent and touch disjoint files. buildForceReconnectFn now gates Disconnect() on IsConnectionOpen() (true only when status is strictly connected) so it only tears down a genuinely open connection, and logs Connect()'s error token instead of discarding it. |
||
|
|
0d6f59ab2d |
fix(#1864): decode ANON_REQ source pubkey instead of treating it like REQUEST (#1866)
Fixes #1864. ## Problem `PAYLOAD_TYPE_ANON_REQ` was effectively treated like `REQUEST`. The two differ on the wire: ``` REQUEST : <dest hash 1B> <source hash 1B> <hmac 2B> <encrypted> ANON_REQ: <dest hash 1B> <source pubkey 32B, full> <hmac 2B> <encrypted> ``` The decoders read the right bytes but surfaced the sender key as `ephemeralPubKey`, which meant: - `store.go`'s node indexer keys on `pubKey`/`destPubKey`/`srcPubKey`, so ANON_REQ packets were **not** indexed — they didn't show up on a node's packet view; and - the packets list "details" rendered a bare `anon → <destHash>`, throwing away the sender identity the packet actually carries. - the detail side-view byte breakdown fell through the REQ catch-all, mislabelling a nonexistent 1-byte "Src Hash" and placing MAC/Encrypted-Data at the wrong offsets (`+2`/`+4` instead of `+33`/`+35`). ## Fix **Backend** (`cmd/ingestor` + `cmd/server` decoders) - Surface the ANON_REQ sender key as `srcPubKey` (json) so it's indexed and resolvable. The frontend keeps a legacy `ephemeralPubKey` reader so packets decoded before this rename still resolve — no DB migration needed. - `TestDecodeAnonReqValid` now asserts the full 32-byte `srcPubKey`. **Frontend** - `hop-resolver.js`: new O(1) `nameForKey(pubkey)` using the existing `pubkeyIdx` (all nodes). - `getDetailPreview`: resolve the source pubkey to a node **name** when known, else show the first 8 hex chars — no more bare `anon`. - Detail side-view: explicit ANON_REQ breakdown — `Dest Hash (1B)` | `Src Public Key (32B)` (node-linked) | `MAC @+33` | `Encrypted Data @+35`. - Detail header `srcLabel` falls back to the resolved ANON_REQ sender. All rendered names are `escapeHtml`-wrapped. ## Testing - `go test ./...` green for both `cmd/ingestor` and `cmd/server` (incl. strengthened `TestDecodeAnonReqValid`). - `node --check` on `packets.js`; brace/paren balance + markers verified on `hop-resolver.js`. - No HTML sink lines added → XSS preflight gate unaffected; every interpolated name is escaped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
a3454e7508 |
fix(#1858): replace emoji map icon with Phosphor sprite in rx-coverage (#1860)
## Summary `public/rx-coverage.js` still carried a literal `🗺️` (U+1F5FA) in the Mobile RX coverage page header — missed by the #1648 emoji → Phosphor migration. Replaced with `ph-map-trifold` from the existing sprite, matching how every other page header renders (`analytics.js`, `home.js`, `node-analytics.js`, `customize-v2.js`). ```diff -'<h2 style="margin:4px 0 2px;font-size:18px">🗺️ Mobile RX coverage</h2>' + +'<h2 style="margin:4px 0 2px;font-size:18px"><svg class="ph-icon" aria-hidden="true"><use href="/icons/phosphor-sprite.svg#ph-map-trifold"/></svg> Mobile RX coverage</h2>' + ``` ## How it got through This is not a gap in the tooling — `test-issue-1648-m6-final-sweep.js` catches it correctly. On current `master` (`a06ac8ac`): ``` ✗ 1 emoji-as-icon violation(s): public/rx-coverage.js:29 [U+1F5FA] '<h2 style="margin:4px 0 2px;font-size:18px">🗺️ Mobile RX coverage</h2>' + ``` The gate is in `test-all.sh` but not in the CI test list in `deploy.yml`, so it never runs. That divergence is filed separately as #1858 — this PR is the concrete defect it let through. ## Test plan - [x] `node test-issue-1648-m6-final-sweep.js` — `✓ lint gate: 0 violations across public/** and cmd/**` (was 1 violation before) - [x] `node test-issue-1648-m6-lint-self.js` — green, including the anti-tautology probe (it requires a clean repo to run at all, so it was failing on master purely as a cascade from the above) - [x] `eslint public/rx-coverage.js` — 0 errors (1 pre-existing `no-unused-vars` warning on `selectedName`, untouched) - [x] `ph-map-trifold` confirmed present in `public/icons/phosphor-sprite.svg` Single-line change, no behaviour change beyond the icon glyph. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
52d08214bb |
fix(#1749): decouple watchdog emit from blocking I/O (root cause) (#1853)
Closes the gap left by #1810: that PR added defer/recover around the watchdog per-source work so a **panic** inside emit cannot kill the loop, but the actual production incident is caused by emit **blocking**, not panicking. ## Root cause In production `emit` is `log.Print`. `log.Print`'s underlying `write()` can block indefinitely if the sink is backpressured (Docker JSON-file log driver falling behind under load, a full stderr pipe, journald hiccups, etc.). A blocked syscall is not a panic -- `recover()` does nothing for it. Because emit was called **synchronously** inside the per-source work, a single stuck `write()` froze the entire tick loop forever -- no further source was ever checked and no further tick was ever processed again. This exactly reproduces the original #1749 incident even after #1810 landed: 3 independent MQTT sources going silent within ~60s of each other (one shared dependency -- the watchdog goroutine itself -- died, not 3 independent paho clients), zero WATCHDOG log lines for the rest of the 75-minute window, every other goroutine in the process continuing to run fine (a hang, not a crash), and only a full container restart recovering it. ## Fix `newAsyncEmit` decouples "decide to log" from "perform the write": the watchdog loop now only ever does a non-blocking channel send. A single background goroutine drains the channel and performs the (potentially blocking) write. If that goroutine itself gets stuck, the bounded queue (256) fills and further sends are dropped -- counted via the new `WatchdogLogDropCount`, surfaced through `/api/mqtt/status` and the ingestor stats snapshot alongside `WatchdogLastTickUnix` / `WatchdogPanicCount`. Worst case under a persistent backpressure event is now lost log lines (visible and counted), not a silently dead watchdog (invisible and undetectable -- the actual #1749 failure). ## Tests - `TestNewAsyncEmit_NeverBlocksWhenWriterStuck_1749` -- floods emit() past queue capacity while the writer is permanently blocked; every call must return immediately and drops must be counted. - `TestMQTTStallWatchdog_LoopSurvivesStuckWriter_1749` -- end-to-end, wires `runLivenessWatchdogLoop` exactly as production does (via `newAsyncEmit` around a permanently-blocking `realEmit`) with 3 registered sources, reproducing the incident shape and asserting the loop keeps ticking regardless. - `TestRunLivenessWatchdog_ProductionWiringUsesAsyncEmit_1749` -- smoke-tests the real entrypoint starts, ticks, and stops cleanly. - `WatchdogLogDropCount` round-trip tests in both the ingestor stats snapshot and the server's `/api/mqtt/status` handler, mirroring the existing `WatchdogPanicCount` coverage from #1810. All pre-existing watchdog/liveness tests (#1749, #1810 r1, force-reconnect) continue to pass unmodified; full ingestor suite green (verified 5x consecutive runs for flake-freedom). Note: the server package has pre-existing test-suite-wide flakiness in unrelated `TestHandleNodePaths_*` tests (confirmed reproducible on unmodified master too, non-deterministic which subset fails per run) -- unrelated to this change and out of scope here. --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
4fc42d30a7 |
fix(frontend): relay-aware staleness for infra nodes + dim-not-delete (#1598, PR A) (#1815)
Implements **PR A** from the #1598 triage fix path (r6 — all three operator decisions locked: confidence ≥0.75 [PR B], **dim-not-delete**, **in-place `getNodeStatus` signature extension**). ## Changes **`public/roles.js` — `getNodeStatus()`** - Now accepts a full node object (preferred); the legacy `(role, lastSeenMs)` signature keeps working unchanged. - For infra roles (repeater/room), freshness = `max(advert-based timestamp, last_relayed)`. Freshness precedence mirrors existing call sites: `_liveSeen` > `_lastHeard` > `last_heard` > `last_seen`. - `last_relayed` is only consulted for infra — companions keep pure advert/heard-based staleness (per @liquidraver's collision caveat; the ≥0.75-confidence `_liveSeen` refresh is PR B). **`public/live.js` — `pruneStaleNodes()`** - Repeater/room markers are **dimmed, never deleted**, regardless of `_fromAPI` origin. WS-only non-infra nodes are still removed to prevent unbounded memory growth. **Call sites** — all seven (`nodes.js` ×3, `map.js` ×3, `live.js` ×1) now pass the node object. The Nodes-page status explanation shows "Last relayed …" when relay participation is the fresher signal, so an Active badge next to an old "last heard" isn't confusing. **Tests** — 16 new `getNodeStatus` unit tests incl. the triage's backbone-repeater fixture (`last_seen`=25h, `last_relayed`=5min → `active`); `pruneStaleNodes` tests updated for dim-not-delete plus a new relay-aware prune test. ## Test results `node test-frontend-helpers.js`: **641 passed, 2 failed** — the 2 failures (favStar ★ assertions) are pre-existing on current master (verified on a clean upstream clone). ## Validation offer live.saarmesh.de currently has **160 infra nodes past `infraSilentMs`=72h while actively relaying** (incl. KatS disaster-relief repeaters) — a ready-made test population. Happy to run this branch there and report before/after node visibility. Refs #1598 (PR A of two; PR B = `_liveSeen` refresh on `resolved_path` ≥0.75 confidence). --------- Co-authored-by: Mathias Kasper <fallisaar@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: SaarMesh-Bot <bot@saarmesh.de> |
||
|
|
6528c7ba3e |
fix(#1854): move relay last_seen touch to the ingestor — server writes have been no-ops since mode=ro (#1855)
Fixes #1854. Refs #1598, #1611, #1845.
## The bug
`cmd/server/db.go:54` opens SQLite `mode=ro` (#1283/#1289).
`touchRelayLastSeen` → `TouchNodeLastSeen` issues `UPDATE nodes SET
last_seen` on that handle. It has failed on every call since, with the
error discarded at the call site:
```go
if err := s.db.TouchNodeLastSeen(pk, ts); err == nil {
s.lastSeenTouched[pk] = now
}
```
`nodes.last_seen` has therefore tracked ADVERT arrivals only. Verified
on live.saarmesh.de (1388 nodes): 1362 have `last_seen` within one
minute of their own most recent ADVERT. Reproduced directly with the
server's DSN in #1854.
Secondary effect: `lastSeenTouched` is populated only in the success
branch, so the debounce never engaged — the server retried the failing
UPDATE for every resolved pubkey in every decode window.
## The fix
The writer moves to `cmd/ingestor`, which owns `nodes` per #1283/#1287
and since #1547 already resolves hop prefixes to full pubkeys for
`observations.resolved_path`. The touch hooks into that existing
resolution point, so there is no new IPC surface and no second resolver.
Only unambiguously resolved hops qualify — a 1-byte prefix collision
cannot keep a silent node alive.
I considered the `internal/mbcapqueue` snapshot handoff used for
#903/#1324 and did not need it: that pattern exists because the
capability computation lives in the server's analytics cycle. Path
resolution already happens in the ingestor, so a file handoff would add
a hop for nothing.
`Store.TouchRelayNodes`:
- monotonic guard in SQL (`last_seen IS NULL OR last_seen < ?`) —
out-of-order ingest never rewinds
- 5-minute debounce keyed on `rxTime`, matching the interval the server
intended
- UPDATE only — unknown pubkeys never create rows
- unparsable `rxTime` is a no-op rather than writing garbage into the
node directory
- `Stats.RelayTouches` for `/api/perf` visibility
- debounce records the *attempt*, not the row match, so an unknown
pubkey is not retried per observation
## Server-side removal
`touchRelayLastSeen`, `DB.TouchNodeLastSeen`, the `lastSeenTouched` map
and the now-unused `allResolvedPKs` decode-window map are deleted.
`readonly_invariant_test.go` gains `UPDATE\s+nodes\s+SET\s+last_seen`.
`cmd/server/touch_last_seen_test.go` and two tests in
`resolved_index_test.go` go with it. Worth stating why they were green
for months: they build their `PacketStore` on `setupTestDB`, which opens
read-write. The production constraint is the one thing they did not
reproduce, which is why the added invariant regex — not a replacement
unit test — is the right guard here.
## Tests
Five tests in `cmd/ingestor/relay_touch_test.go`, committed red first
(
|
||
|
efitenandClaude Opus 5
|
176bb53335 |
fix(#1784): ship pathTrust default 1, not 2 (#1929)
Follows #1841. Moves the pathTrust default from 2 back to 1. ## Why #1784's first acceptance criterion is **"Default behaviour remains backward-compatible"**, and its example config shows `minHashBytesForMapping: 1`. What shipped is 2. The problem is not the value. It is that **there is no way to undo it from the UI.** #1841 adds no control for the threshold: it is `config.json` only, and changing it needs a restart. The customizer gains a hint that says exactly that. So an instance that upgrades without touching config switches to the stricter rule, and the only visible symptom is that the neighbour graph and the resolved paths quietly get smaller. The existing "Hide 1-byte path hops" toggle (#1633) is a *display* filter and does not change what counts as evidence, so it is not an escape hatch either. The two are easy to confuse. ## How much this actually moves Measured on a live instance via `/api/analytics/hash-sizes`, not estimated: | path-hop observations | count | share | |---|---|---| | 1-byte prefix | 116,923 | **56.0%** | | 2-byte prefix | 86,031 | 41.2% | | 3-byte prefix | 5,753 | 2.8% | | repeaters by observed hash size | count | |---|---| | 1-byte | **645 (41%)** | | 2-byte | 865 | | 3-byte | 63 | At threshold 2 the 1-byte column stops counting as mapping evidence. `MeetsPathTrust` also drops the legacy bucket-0 observations with it (pre-#1638 persisted neighbor edges that carry no per-mode breakdown), so already-stored edges lose their evidence status on upgrade too. ## What this does not change The knob works and is untouched. Operators who want the stricter behaviour set `minHashBytesForMapping` to 2 or 3, which is the opt-in #1784 describes. Only the default moves. Nothing about storage changes; packets and paths were never affected either way. ## Also fixes an inconsistency inside #1841 Five frontend consumers already fall back to **1** when `MC_getPathTrustThreshold()` is unavailable: `analytics.js`, `live.js`, `map.js`, `nodes.js`, `route-view.js`. Two fell back to **2**: `hop-filter.js` (the getter itself) and `customize-v2.js`. They now all agree. ## Tests - `internal/packetpath`: `TestMeetsPathTrust_ZeroValueOptIn` was asserting the old default *through behaviour*, so it would need rewriting on any future default change. It now asserts the property instead: an absent JSON field resolves to `DefaultMinHashBytesForMapping`, behaves identically to naming that value outright, and an explicit stricter setting still wins. Package tests pass. - `test-issue-1633-hide-1byte-hops.js`: the case pinning the getter's default is updated, with the reasoning in a comment so the next person sees why it is 1. **37 passed, 0 failed** (master baseline: 37 passed, 0 failed). - `cmd/server` config tests pass. ## One thing I want to flag rather than paper over The test I changed was named `default is 2 (operator-confirmed)`. I am overriding something that was confirmed with an operator, and I am not claiming that confirmation was wrong. My reading is that it was about the threshold being a *useful* value, which it is, rather than about it being the default in a build with no UI to change it. If the intent really was "2 out of the box for everyone", say so and I will close this. @Bjorkan as the issue author, @nullrouten0 and @Saarlandpower since you have touched adjacent code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
b3a306b81f |
fix(#1888): count only live observers in the store's /api/stats query (#1892)
Fixes #1888. ## The mismatch `/api/stats.totalObservers` and `/api/observers` counted different sets: | Source | Predicate | |---|---| | `cmd/server/store.go:2089` (store path) | `SELECT COUNT(*) FROM observers` — every row | | `cmd/server/db.go:336` (DB fallback) | `WHERE inactive IS NULL OR inactive = 0` | | `db.GetObservers()` → `/api/observers` | `WHERE inactive IS NULL OR inactive = 0` | `handleStats` uses the store path whenever a `PacketStore` exists (`routes.go:785`), which is every normal deployment. So the header count came from the unfiltered query while the Observers page listed the filtered set. The two stats implementations also disagreed with each other for the same database, which is a bug on its own. ## Reproduction The gap is exactly the observers the `observerDays` retention sweep has soft-deleted. On the instance I reproduced against: ``` GET /api/stats → totalObservers: 79 GET /api/observers → observers.length == 51 ``` ```sql SELECT 'all', COUNT(*) FROM observers -- 79 UNION ALL SELECT 'active', COUNT(*) FROM observers WHERE inactive IS NULL OR inactive = 0 -- 51 UNION ALL SELECT 'inactive', COUNT(*) FROM observers WHERE inactive = 1; -- 28 ``` 79 − 28 = 51. Same shape as the 82 vs 51 in the issue. ## The change One line: the store's stats query gets the same predicate the other two already use, so all three agree. ## Deliberately out of scope Two things the issue raises that this does **not** fix, called out so they are not mistaken for done: - **Config blacklist.** `buildObserversDefaultResponse` drops blacklisted observers in the handler loop (`routes.go:2752`), which no SQL count can see. A deployment with a non-empty `observerBlacklist` will still show a stats count higher than the list, by the number of blacklisted-but-live observers. Closing that needs config plumbing into the count and is a separate change — happy to follow up if wanted. - **Map controls.** The third surface named in the issue derives its count from node role aggregates (`roleCounts`), not from the observer set at all. That is a frontend concern and untouched here. ## Tests `cmd/server/observer_count_1888_test.go`, three cases, each watched fail first: 1. `TestStoreStatsTotalObserversExcludesSoftDeleted` — `TotalObservers = 5, want 4` 2. `TestStoreStatsTotalObserversMatchesObserverList` — `stats totalObservers = 5 but /api/observers lists 4` 3. `TestStoreAndDBStatsAgreeOnTotalObservers` — `store path reports 5 observers, DB fallback reports 4` The fixture includes a row with `inactive = NULL` alongside `inactive = 0` and `inactive = 1`, since `GetObservers` treats NULL as live and only the `1` may be excluded. `cd cmd/server && go test ./...` → ok (87s). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
SaarlandpowerandClaude
|
02c2338d64 |
fix(packets): observer filter hides multi-observer rows in grouped mode (#1748) (#1821)
Closes #1748. ## Root cause `renderTableRows()` in `public/packets.js` re-filtered the already server-filtered `/api/packets` (grouped) results client-side, comparing `filters.observer` against each row's `observer_id`. But that field is only the **representative** observer chosen for display — `QueryGroupedPackets` (`cmd/server/db.go:554-606`) picks the observation with the longest observed path via a `LEFT JOIN ... ORDER BY length(path_json) DESC LIMIT 1`, purely for display purposes. The server-side filter (`buildTransmissionWhere`, `cmd/server/db.go:725-790`) is already correct: it uses an `EXISTS` subquery over **all** observations of a transmission, so any row it returns was genuinely seen by at least one selected observer. The client then discarded rows *again*, checking only the representative's `observer_id`, with a fallback to `p._children` — which is `undefined` on initial page load (only fetched lazily on row-expand or when the observer-sort dropdown changes, see the `obsSortSel` change handler). Net effect: a multi-observer transmission stayed visible under an observer filter only when the filtered observer happened to also be the representative (longest-path) observer — which matches the reported "works only for whichever observer logged it first" behavior in dense meshes, where longest-path and earliest-seen correlate. ## Fix Skip the client-side observer re-filter entirely when `groupByHash` is active — the server's `EXISTS` filter is authoritative for grouped rows and needs no client-side correction. The flat/expanded-mode path (single-observation rows, each with its own exact `observer_id` from `buildPacketWhere`) keeps the existing children-aware filter unchanged, which already had test coverage under #537 for the case where `_children` is already populated. ## Tests Added 6 cases in `test-frontend-helpers.js` covering the specific gap #537's tests didn't reach — grouped mode with `_children` still `undefined` (the actual initial-load state that triggers this bug). New tests confirm: - A multi-observer row whose representative doesn't match the filter is kept in grouped mode (the core bug). - Grouped mode never re-filters client-side (trusts the server). - Flat mode behavior is unchanged (matches by own `observer_id`, falls back to already-loaded `_children`). Full run: `test-frontend-helpers.js` 631 passed / 2 failed (same 2 pre-existing `favStar` failures reproduce identically on unmodified `master` — unrelated). `test-packet-filter.js` 92/92. `test-aging.js` 18/18. Operator context: running CoreScope for SaarMesh (SaarLorLux, DE/FR/LU, 800+ nodes, 14 observers) — this was hiding a large share of traffic whenever filtering by a non-primary observer in our dense mesh. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
SaarlandpowerandClaude
|
0352c9a287 |
fix(clock-skew): restrict per-node skew to self-originated adverts (#1816, #1818) (#1820)
Closes #1816. Closes #1818 (confirmed duplicate of #1816 by the triage bot). ## Root cause `byNode` is an involvement index (`indexResolvedPathHops`, `store.go:1696-1705`, #1558/#1352): a transmission is indexed under every relay-hop pubkey found in an observation's `resolved_path`, not just its originator. `getNodeClockSkewLocked` (`clock_skew.go:489`) iterated every ADVERT transaction under a pubkey without checking who actually signed it, so a relay inherited the clock skew of every broken-clock node it forwarded as if it were its own. This produced: - Fleet-wide false `no_clock`/`bimodal_clock` classifications on healthy relays whose only "bad" samples were adverts they merely relayed. - Bit-identical `RecentMedianSkewSec` "clusters" across unrelated relays that all forwarded the same broken-clock originator. - Single relays showing a multi-day skew even though their own self-adverts are healthy, because 1-2 relayed adverts from a broken originator landed in the tail of their small recent-window sample (the #1818 "island" repro from @cwichura). ## Fix Add `txOriginatedBy(tx, pubkey)`: ADVERTs are self-signed, so `decoded["pubKey"]` is the originator per protocol (case-insensitive compare as a defensive measure). Apply it as a guard in both the main skew-aggregation loop and the per-hash evidence loop in `getNodeClockSkewLocked`. `byNode` itself is untouched — #1558/#1352 still rely on the broader involvement index for other consumers. ## Tests - Existing `clock_skew_test.go` / `clock_skew_issue1094_test.go` / `clock_skew_issue1285_test.go` fixtures built synthetic ADVERT transactions without a `pubKey` field and seeded `s.byNode` directly, bypassing the normal `indexByNode` path where every real ADVERT carries `pubKey`. Added `pubKey` to each fixture so it reflects a self-originated advert, which is what these tests already intended to represent. All pre-existing tests pass unchanged in behavior. - New `clock_skew_issue1816_test.go`: - `TestTxOriginatedBy` — unit coverage of the new guard (self, foreign, missing pubKey, case-insensitivity). - `TestIssue1816_RelayDoesNotInheritOriginatorSkew` — a relay with healthy self-adverts plus relayed adverts from a broken-clock originator (matching the report's +100.5k s band) must report `ok` severity based only on its own adverts. - `TestIssue1816_PureRelaysReportNoSkew_NoBitIdenticalCluster` — five relay pubkeys that only ever forward a broken originator's advert (never self-advert) must report `nil`, not a bit-identical copy of the originator's skew. - `TestIssue1818_TwoForeignAdvertsDoNotPoisonIslandNode` — reproduces the cwichura island scenario: 8 healthy self-adverts + 2 foreign adverts at ~10 days skew must not flip severity or pollute `RecentMedianSkewSec`. Full suite: `go test ./...` passes (one pre-existing, unrelated flaky test — `TestHandleNodePaths_PrefixCollision_1352`, an index-loading race — reproduces intermittently on unmodified `master` too). Operator context: running CoreScope for SaarMesh (SaarLorLux, DE/FR/LU, 800+ nodes); this bug was surfacing as fleet-wide clock-skew false positives on our infra nodes. Co-Authored-By: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
9ef4179ef1 |
fix(release): report correct version on fast-path retagged images (#1807) (#1814)
Fixes #1807. Implements the fix path from the triage (env → image-version file → baked version, zero rebuild cost): ## Changes **`cmd/server/main.go` — `resolveVersion()` fallback chain** 1. `CORESCOPE_VERSION` env (operator override) 2. `.image-version` file in the working dir (`/app` in the container) — mirrors the existing `.git-commit` pattern in `resolveCommit()` 3. ldflags-baked `Version` 4. `"unknown"` Edge builds are unaffected: no env, no file → baked `"edge"` as before. **`.github/workflows/release-fast-path.yml` — retag step** Instead of a plain `crane tag :edge → :vX.Y.Z`, the fast path now runs `crane mutate` on `:edge` with: - `--append` of a deterministic one-file layer containing `/app/.image-version` = `vX.Y.Z` - `--label org.opencontainers.image.version=vX.Y.Z` - `--tag :vX.Y.Z` `vX.Y`, `vX` and `latest` are then pointed at the mutated image. Still no rebuild — the mutation is a manifest + single ~100-byte layer operation. ## Notes - The release tags no longer share the exact digest with `:edge` (they carry one extra layer); the fallback SHA check is unaffected since it compares the `org.opencontainers.image.revision` label against `github.sha`. - The layer tar uses `--owner=0 --group=0 --mtime='UTC 2020-01-01'` for reproducibility. - Operators can also fix existing deployments immediately with `-e CORESCOPE_VERSION=v3.9.2`, no image change needed. ## Testing - `go build ./cmd/server` + `go vet` clean (golang:1.24) - Workflow YAML validated - Reporter context: running the affected v3.9.2 fast-path image in production (live.saarmesh.de), happy to verify the next tagged release end-to-end. --------- Co-authored-by: Mathias Kasper <fallisaar@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: SaarMesh-Bot <bot@saarmesh.de> |
||
|
Jesper B
|
8c48f2b49a |
feat(#1784): wire display consumers to path trust threshold (#1841)
Fixes #1784 ## Summary Step 4 of the #1784 multi-PR project — wires all frontend display consumers to respect the configured `pathTrust.minHashBytesForMapping` value. **Depends on:** #1840 (must be merged first — provides `MC_meetsPathTrust` / `MC_pathBelowTrust` / `MC_getPathTrustThreshold` helpers and `PATH_TRUST` global). ## Changes ### `map.js` — trust-gated route display - `drawPacketRoute` checks `MC_pathBelowTrust` before drawing polylines - When all path hops are below the configured threshold, shows a "Route not displayed" message with config guidance instead of speculative polylines - Cleans up the trust message control when a new route is drawn ### `analytics.js` — subpath trust filtering - `renderTable` in `renderSubpaths` filters route patterns whose hops are below trust threshold - Combined filter logic: both the 1-byte hide toggle AND trust threshold are applied together - Info line shows which filters are active ("1-byte hide + 2-byte trust", etc.) - No-data message explains which filters caused exclusion ### `route-view.js` — speculative path annotation - Path picker groups tagged with `belowTrust` flag when any hop doesn't meet the configured threshold - Speculative paths show `(speculative, <N-byte hops)` annotation with tooltip explaining the trust threshold - Tooltip includes guidance on changing `pathTrust.minHashBytesForMapping` in config.json ### `live.js` — Paths Through widget - `_pathHopsBelowTrust()` helper checks whether all hops in a path are below trust threshold - Widget fallback message distinguishes between "1-byte filtered" (display toggle) and "N-byte trust threshold" (server config) - Message includes the config key for operators to adjust ### `nodes.js` — confidence weight adjustment - `modeWeight` initialization considers the trust threshold - Hash modes below threshold get zero confidence weight - Bucket-0 (legacy/unknown) excluded at threshold >= 2 per #1784 bucket-0 policy ## Testing `test-issue-1633-hide-1byte-hops.js`: - 5 new source-grep guards verifying each consumer references the trust threshold helpers - All 26 tests passing (21 existing + 5 new) ## Files changed (6 files, +136/-6) ``` public/analytics.js | 28 ++++++++++++++++++--- public/live.js | 19 ++++++++++++++- public/map.js | 21 ++++++++++++++++ public/nodes.js | 8 ++++++ public/route-view.js | 16 +++++++++++- test-issue-1633-hide-1byte-hops.js | 50 ++++++++++++++++++++++++++++++ ``` --- **Depends on:** #1840 **Written by:** DeepSeek V4 Pro in Max Mode |
||
|
efitenandClaude Opus 5
|
4c45dec79f |
fix(#1902): don't attribute transported scopes from the 1-byte hop prefix (#1903)
Fixes #1902. ## The bug `byPathHop` is keyed on the raw hop string from `path_json`, and both relay-info paths look up the full pubkey **and** fold in `key[:2]` — the 1-byte wire prefix. `TransportedScopes` (#1751) accumulated over that folded set, so every node sharing a pubkey first byte reported the same scopes. On the live network all four active nodes with prefix `f7` returned an identical set: ``` f79616... BE repeater ['#be','#be-van','#de','#de-nw','#nl'] f7e718... BE repeater ['#be','#be-van','#de','#de-nw','#nl'] f788ad... BE repeater ['#be','#be-van','#de','#de-nw','#nl'] f752c2... DE/NRW repeat. ['#be','#be-van','#de','#de-nw','#nl'] ``` Their real sets, from unambiguous full-pubkey hops over the same 7 days, are disjoint: ``` f79616... (BE) #be 471, #eu 9, #nl 6, #de 3, #be-van 1 f752c2... (DE/NRW) #de 13, #de-nw 11 f7e718... (BE) (none) ``` A sysop reads a scope badge as a statement about how their repeater is configured, so a Belgian repeater badged `#de-nw` is a wrong answer, not an imprecise one. ## The change The prefix fold stays for the counters — that is the documented #662 trade-off, "a possible over-count for clearly false zeros", and `RelayCount1h/24h`, `LastRelayed` and `UnscopedRelayCount24h` are magnitudes where an over-count is tolerable. Scopes are not a magnitude. A 1-byte hop names one of N nodes and cannot substantiate a categorical claim. Entries reached only through the prefix bucket are now flagged (`relayEntry.viaPrefix` / a `viaPrefix` argument to the bulk `visit` closure) and excluded from scope accumulation only. Both computation paths are changed together so `/api/nodes` (bulk) and the node-detail endpoint (per-node) stay in parity: - `cmd/server/repeater_liveness.go` — `collectRelayEntriesLocked` / `computeRelayInfoFromEntries` - `cmd/server/repeater_enrich_bulk.go` — `computeRepeaterRelayInfoMap` The `public/nodes.js` tooltip is updated to describe what the field now actually means. Attribution does not collapse: `observations.resolved_path` carries full pubkeys for ~27% of observations on the live instance (408k of 1.54M over 7 days), and those rows produce the correct per-node sets above. A node with no resolved hop yet shows no badge rather than a borrowed one. ## Tests `TestTransportedScopes_CrossBucketFold` pinned the old behaviour ("a scope seen only in the prefix bucket must surface on the full key"), which is the bug. It is replaced by `TestTransportedScopes_PrefixBucketNotAttributed`, which asserts on **both** paths that: 1. a scope evidenced only by a 1-byte hop is not attributed; 2. a scope also present under the full key still is; 3. `RelayCount24h` still counts all three packets — narrowing scopes must not narrow the counters, i.e. the #662 fold is untouched. Red before the change, green after. ``` cd cmd/server && go test ./... ok github.com/corescope/server 98.2s node test-packet-filter.js 92 passed, 0 failed node test-aging.js 18 passed, 0 failed node test-frontend-helpers.js 625 passed, 2 failed ``` The two frontend failures (`favStar returns filled star for favorite`, `favStar returns empty star for non-favorite`) and `cmd/ingestor`'s `TestWriteStatsAtomic_SymlinkAtDestIsReplaced` are **pre-existing** — I ran them on a pristine `upstream/master` worktree and got byte-identical results (the ingestor one is a Windows symlink-privilege limitation, not a code failure). ## Perf No new work in any loop. The bulk path gains one bool argument to an existing closure and one `&& !viaPrefix` on a branch that already ran; the per-node path gains one bool field on `relayEntry`, which is stack/slice-local and not retained. Same complexity, same allocations. ## What I could not verify end-to-end, and why I built a fixture from live data (2512 nodes, 17k transmissions, 529k observations, including all eight `f7` nodes) and ran the before/after binaries against it. Neither reproduced the live field — both returned no `transported_scopes` and `relay_count_24h: 0` for every node. That turns out to be a **separate cold-start bug**: `LoadChunked` calls `indexResolvedPathHops` per observation while scanning chunks, which adds full-pubkey keys to `byPathHop`, and then the post-load block at `cmd/server/chunked_load.go:459` calls `buildPathHopIndex()`, which begins with `s.byPathHop = make(...)` and rebuilds from raw hops only. Every resolved full-pubkey key from the scan is discarded: ``` [store] Built path-hop index: 2924 unique keys <- raw hops only [store] LoadChunked: 17056 transmissions (527331 observations) ``` So on a freshly started server the full-pubkey buckets are empty and only refill from live ingestion. That is being filed separately; it is orthogonal to this change, but it does mean `transported_scopes` will be sparse for a while after any restart until it is fixed. This PR is therefore verified by unit tests on both computation paths plus the live-data derivation above, not by a local end-to-end run. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
9bd5f5a3a2 |
fix(ingestor): a retained status message is not observer liveness (#1885)
## Problem The broker replays every retained `status` message on subscribe, so each ingestor restart pushes all of them through the status path in `handleMessage`. That path stamps `last_seen` with `time.Now()` (deliberately, per #1465). Observed on a live deployment on 2026-08-11: **23 observers all carried `last_seen = 2026-08-06T08:20:09Z`** — 15 seconds after container start — and 18 of them had sent no actual packet in over a month. Their retained publish dates lined up almost 1:1 with `last_packet_at`, i.e. the replay was their only sign of "life": | observer | last real packet | retained status published | |---|---|---| | ON8AR - Observer | never | 2026-03-20 | | BE-BGS-RRY120-RES | never | 2026-04-02 | | A3BEF374 | 2026-04-30 | 2026-04-30 | | BE-BGS-RRY120-RUDY | 2026-05-18 | 2026-05-18 | | BE-JBE-ETG-O1 | 2026-06-10 | 2026-06-10 | That makes dead observers immortal, three ways per restart: 1. `last_seen` jumps forward, so `RemoveStaleObservers` can never age them out as long as a restart happens inside `observerDays`. 2. The unconditional `inactive = 0` reactivation at the end of `UpsertObserverAt` undoes any soft-delete that did land. 3. A metrics sample is filed at ingest time, dating a months-old reading as a present-tense measurement. `UpsertObserverAt`'s docstring already claimed retained replays were a no-op for `last_seen` thanks to the `MAX` guard. That held only while the caller passed the envelope timestamp; #1465 switched it to ingest time, which defeats the guard. ## Fix The retained path now updates metadata only, via a new `UpsertObserverRetained`: - no `last_seen` advance - no `inactive = 0` reactivation - no `packet_count` bump - no metrics sample - **no INSERT** — a retained-only observer the analyzer has never heard from live describes a past that may be months old and does not belong in the list. A live message from the same observer creates the row through the normal path moments later. Live status handling is unchanged. ## Tests Seven tests in `cmd/ingestor/retained_status_test.go`, written before the fix: - 4 that failed on the bug: `last_seen` advance, reactivation of a soft-deleted row, creation of a never-seen observer, metrics-sample insert - 2 regression guards pinning live (non-retained) behaviour: `last_seen` still advances, unknown observer still created - 1 asserting retained metadata is still applied — the snapshot is the observer's last known state, only the liveness signal is suppressed `mockMessage` gained a `retained` field so `Retained()` is controllable. Meta flattening is extracted to `observerMetaColumns` so both write paths bind identical args. Full `cmd/ingestor` suite passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Michael J. ArcanandWaydroid Builder
|
7402e8d9d9 |
feat(analytics): repeater metric scatter tab (#1760)
Repeater metric scatter tab. Closes #1763. --------- Co-authored-by: Waydroid Builder <build@waydroid.local> |
||
|
Joel ClawandJoel Claw
|
f49e3fcc26 |
perf: use cached ParsedDecoded() instead of repeated json.Unmarshal (#1871)
## Problem
`StoreTx.ParsedDecoded()` already caches the result of `json.Unmarshal`
on first call via `sync.Once`. However, 13 call sites in `store.go` and
1 in `routes.go` were independently unmarshaling `DecodedJSON` into
local `map[string]interface{}` variables on every access, completely
ignoring the cache.
## Impact
For a store with 50k+ transmissions, each analytics endpoint that
iterates all packets re-parses 50k JSON strings per request. With
multiple endpoints, this means hundreds of thousands of redundant
`json.Unmarshal` calls per page load, each allocating new maps and
slices.
## Fix
Replace each `var d map[string]interface{};
json.Unmarshal([]byte(tx.DecodedJSON), &d)` with `d :=
tx.ParsedDecoded()`, which returns the cached parse result (parsed once,
reused forever).
### Call sites changed (14 total):
- `untrackAdvertPubkey` — advert PK extraction during eviction
- Ingestion path — payload field for API responses
- `evictStaleInternal` — node cleanup during eviction
- `GetAnalyticsTopology` — node PK extraction
- `GetAnalyticsHashCollisions` — advert PK extraction
- `GetAnalyticsDistance` — region node PK building
- `resolveAreaNodes` — node PK extraction
- `GetRecentPackets` — payload field for API response
- `routes.go` byType grouping — type field extraction
### Left unchanged (3 sites):
Three call sites that unmarshal into typed structs (`grpDec`,
`decodedMsg`, `decodedGrp`) cannot use `ParsedDecoded()` since they need
specific struct types.
## Testing
- `go build` passes
- No behavior change — same data, same logic, just avoids redundant
parses
---------
Co-authored-by: Joel Claw <358739783+Joel-Claw@users.noreply.github.com>
|
||
|
efitenandClaude Opus 5
|
b38536911a |
fix(#1858): unstick npm test — three stale assertions and a Windows-only allowlist bug (#1895)
Partial fix for #1858. Clears four blockers; does **not** close the issue. ## Why partial matters here `test-all.sh` aborts at the first failing script. So the 9 red tests the issue counts are not 9 problems a contributor can work through — the run stops at `test-frontend-helpers.js` and everything after it is invisible. You have to fix them in order just to see the next one. That is what this does, as far as I could go without guessing. ## Three assertions that outlived their features Each was left behind by a merged PR that changed the thing being asserted. None could pass again without reverting that work. | Test | Asserted | Reality | |---|---|---| | `test-frontend-helpers.js` | `★` / `☆` glyphs in `favStar` | #1648 replaced them with `#ph-star-fill` / `#ph-star` sprite refs | | `test-channel-psk-ux.js` | literal `🔓` | same migration → `#ph-lock-open` | | `test-analytics-channels-integration.js` | a `#/analytics` chip in `channels.js` | #1367 / PR #1376 deliberately dropped it | The `favStar` rewrite also tightens two assertions that were not doing their job: `html.includes('on')` matched the word "favorites", and the negative case checked `!html.includes(' on')` rather than the class. Both now assert `aria-pressed` and the class explicitly. For the analytics chip I removed the assertion rather than weakening it, and left the reason in place so it does not get "restored" by someone reading a bare deletion: ```js // The "Channels page links to Analytics" assertion that used to live here was // removed: #1367 / PR #1376 ... deliberately deleted the `#/analytics` chip. ``` ## A Windows-only bug in the emoji sweep `test-issue-1648-m6-final-sweep.js` reports **64 violations on Windows and 1 on Linux**. `walkFiles` builds each path with `path.relative`, which yields `public\cb-presets.js` on Windows. Every entry in `tests/emoji-allowlist.txt` — and the ignore list in the same function — is written with `/`. So `matchesGlob` matched nothing, and every deliberately-annotated line (WCAG contrast `✓`/`✗` notes, `⌈⌉` ceiling brackets in audio-lab) was reported as a violation. One line, normalising to forward slashes. 64 → 1. The remaining 1 is the `🗺️` in `public/rx-coverage.js` that **PR #1860 already fixes**, and `test-issue-1648-m6-lint-self.js` cascades from it (`repo MUST be clean before anti-tautology probe`). Both go green when #1860 merges. Not duplicated here. ## Still red, deliberately untouched Four tests fail for reasons that need a behavioural decision rather than a test edit: - `test-issue-1438-customizer-mcrole.js` — `--mc-role-companion` empty - `test-issue-1446-cb-preset-cascade.js` — see below - `test-issue-1470-node-tile-helper.js` — `getActiveTileProvider` returns `undefined` for voyager; `getTileUrl` yields the `dark_all` URL instead - `test-issue-1485-live-anim-z.js` — animLayer hosts 2 animation shapes, ≥3 expected ### #1446 may not be a stale test Worth a second look before anyone edits it: - **Scenario 6** (roles.js + cb-presets.js, stored preset `deut`) **passes** — `--mc-role-repeater` = `#fe6100`. - **Scenario 5** is the same setup plus `customize-v2.js` and `_customizerV2.init({nodeColors:{repeater:'#aaaaaa'}})`, and the var comes back **empty** — not the preset colour, not the server colour. If that is the real behaviour, a colourblind user with a saved preset loses their role colours the moment the customizer initialises against a server config carrying `nodeColors`. I left it alone: the three-way cascade (preset vs server config vs user override) is exactly what #1446 defined, and picking a side changes colours for the users the feature exists for. That is your call, not mine. ## Not addressed: the structural half Three hand-maintained test lists (267 files, 159 in `deploy.yml`, 53 in `test-all.sh`, no CI invocation of the latter) is the root cause the issue identifies, and it is a workflow decision rather than a bug fix. Happy to do it in whatever shape you want — the obvious one being CI calling `test-all.sh` and the two lists collapsing into one — but that changes what gates merges, so it should be your choice. ## Verification ``` node test-frontend-helpers.js → 627 passed, 0 failed node test-channel-psk-ux.js → 20 passed, 0 failed node test-analytics-channels-integration.js → 23 passed node test-issue-1648-m6-final-sweep.js → 64 violations → 1 ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
8ce5291b7c |
fix(map): apply the CARTO key on every map surface (rebase of #1916 onto #1919) (#1926)
Continues #1916 by @nullrouten0. The commit is theirs, authorship
unchanged; I rebased it onto master and resolved the fallout from #1919.
Opening it here rather than force-pushing to someone else's branch.
## Why this is needed after #1919
#1919 shipped the Carto key, but only for the five `BASE_STYLES` entries
in `map-tile-providers.js`. Four map surfaces still request unkeyed
tiles and get them back stamped `API KEY REQUIRED` at HTTP 200:
- `public/roles.js` — `getTileUrl()` returns the bare `TILE_LIGHT`
constant in light mode and never consults the registry (the dark branch
has consulted it since #1461). Affects `analytics.js:2200` and
`nodes.js:94`, so the analytics map and the node-detail map stay
watermarked in light theme even with a key configured.
- `public/customize-v2.js:1838` and `:2047` — the two geo-filter maps in
the customizer.
- `public/geofilter-builder.html` — standalone page, outside the SPA, so
it fetches `/api/config/client` itself.
@nullrouten0 had already found and fixed all four, plus written the
docs, before #1919 was merged. Their diagnosis of the light-mode branch
is in the PR verbatim.
## What I changed while rebasing
1. **`carto.token` → `carto.key`.** #1919 shipped `key` and operators
already have it in their configs; renaming now would break them
silently. All of #1916's code, tests and docs follow suit.
2. **Dropped the duplicate key getter.** `_getCartoKeyParam` did the
same job as master's `_getCartoKey`; kept master's.
3. **Merged the two test suites.** Master's five cases plus four of
#1916's that master does not cover: the `api_key`-is-ignored assertion,
lazy resolution after async config, and both `MC_tileUrlById` cases. 42
passed, 0 failed.
4. **Corrected one factual claim in the docs.** #1916 stated CARTO
offers no referrer or origin restriction. CARTO does ask for a domain
when the key is issued. Whether that is enforced per request is not
something I verified, so the note now says exactly that rather than
asserting either way.
5. Merged the two config comments, keeping the operationally useful
part: unkeyed tiles return **200** with a watermark, so nothing errors
and no healthcheck fires. Verify by looking at a tile, not at a status
code.
## Verification
Rebased onto `7aa60c03`. No regressions:
| | base
|
||
|
Marek WojtaszekandClaude Opus 5
|
7aa60c0350 |
fix: support Carto basemap API key (tiles are watermarked without one) (#1919)
## Problem Carto began requiring an API key for its raster basemaps in 2026-08. Requests without a key still return `HTTP 200`, but the returned PNG is stamped `API KEY REQUIRED / carto.com/basemaps/apikey`. Because `carto-dark` / `carto-light` are the built-in defaults, **every CoreScope instance that has not changed tile providers now renders watermarked maps.** Reproduce without involving CoreScope at all — the watermark is baked into the bytes Carto serves: ```bash curl -o tile.png https://a.basemaps.cartocdn.com/light_all/11/1152/683.png # HTTP 200, and tile.png carries the watermark. (11/1152/683 is Lublin, PL.) ``` ## Why `carto.domain` cannot express this Carto takes the key as a query parameter on the **same** host: ``` https://basemaps.cartocdn.com/rastertiles/voyager/{z}/{x}/{y}.png?key=YOUR_KEY ``` The existing `tiles.providers.carto.domain` option builds `https://{s}.<domain>.cartocdn.com` — an enterprise subdomain, which is a different mechanism. There was no way to supply a key, so I added one rather than repurposing `domain`. ## What this changes Adds `tiles.providers.carto.key`, appended as `?key=` (URL-encoded) to all five Carto styles: `carto-dark`, `carto-light`, `carto-voyager`, `carto-voyager-dark`, `positron-dark`. ```json "carto": { "enabled": true, "key": "", "domain": "" } ``` When no key is set the emitted URLs are byte-identical to today's, so this is a no-op for anyone who has not requested one. `key` and `domain` compose, so enterprise users keep their subdomain and gain the key. Free keys are available at <https://carto.com/basemaps/apikey> (5M tiles/month, non-commercial tier). **Security note:** like the existing OSM and Stamen tokens, this key reaches the browser — unavoidable for raster tiles. The `_comment_carto` text in `config.example.json` tells operators to restrict the key by domain in the Carto dashboard. ## Tests 5 new cases in `test-issue-1420-tile-providers.js` — **38 passed, 0 failed**: 1. no `?key=` emitted when no key is configured (guards the no-op claim) 2. all five Carto styles append the key when it is set 3. the key is URL-encoded 4. the key does not leak into OSM / Esri provider URLs 5. `key` coexists with the enterprise `domain` option Also ran the required frontend suite: `test-packet-filter.js` (92 passed), `test-aging.js` (18 passed), `test-frontend-helpers.js` (625 passed, 2 failed). **The 2 failures are pre-existing on `master`** — I ran the same file from a clean `origin/master` worktree and got an identical 625/2. They are `favStar returns filled star for favorite` / `... for non-favorite`, unrelated to tiles. ## Performance Not a hot path. `url()` returns a Leaflet URL *template* that is built once per tile-layer construction (`map.js:289`, `live.js:1422`), not once per tile — `_getCartoKey()` is called at exactly the same frequency as the existing `_getCartoBase()`, and does one property read plus one `encodeURIComponent`. ## Validation Verified against a live instance (<https://analyzer.marwoj.net>) running this logic as a mounted patch, with a real Carto key: - **before** — `light_all` and `dark_all` tiles both watermarked - **after** — both clean, with the key restricted to the instance's domain in the Carto dashboard - confirmed the key survives `config.json` → `/api/config/client` → `MC_MAP_CFG` and reaches the tile URLs ## Heads-up for maintainers Carto is retiring raster basemaps and steering users to vector, so this fix has a shelf life. It restores the default experience now; a vector migration is a larger, separate piece of work. I am not sure whether you want an issue filed to track that — happy to open one if it helps. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5
|
589fa9878d |
fix(#1923): pin the packets time window in the slide-over E2E (#1924)
Closes #1923.
## What was wrong
`test-slideover-1056-e2e.js` runs twice in CI: inside the main E2E
suite, and again as the #1616 flake-gate, which is the last step before
coverage collection. That second run starts roughly 20 minutes after
`tools/freshen-fixture.sh` set the fixture's newest packet to "now".
The packets page defaults to a 15-minute time window
(`DEFAULT_TIME_WINDOW`, `public/packets.js:746`). By the time the
flake-gate runs, every fixture packet is outside it, the table renders
zero data rows, and `packets@800: page renders + first row exists` times
out after 30s. The two tests that need a row to click fail with it, so
one expired window reports as three failures.
Whether it fires depends on how long the preceding suite took, which is
why it read as flake and hit PRs that cannot have caused it. #1760 (a
frontend analytics tab, 2026-07-09) and #1872 (Go-only, deletes a dead
struct field, 2026-07-28) failed on the identical three tests three
weeks apart, and neither touches the packets page.
## The change
Pin `?timeWindow=1440` on both packets navigations in that file, so it
tests the slide-over rather than the clock. Two lines plus the comment
explaining why, no product code touched.
The value must be greater than 0. `public/packets.js:1092` reads the
parameter under `_urlTimeWindow > 0`, so `timeWindow=0` does **not**
disable the filter, it silently leaves the 15-minute default in place.
That is worth knowing before anyone "simplifies" it to zero.
## Verification
Reproduced first, then fixed. All against `upstream/master`
|
||
|
Kpa-clawbotandclawbot
|
a06ac8aceb |
fix(#1849): render — for TRACE Hop Bytes column (path bytes are SNR, not hop hashes) (#1850)
Fixes #1849. ## Problem The packets table "HB" (Hop Bytes / col-hashsize) column always shows `1` for TRACE packets. TRACE packets use header path bytes as per-hop SNR readings, not truncated hop hashes — see `internal/packetpath/route.go` `PathBytesAreHops(TRACE) = false`. The high-2-bit "hash_size" derivation applied to a SNR byte is meaningless (typically `1`). ## Fix At the 3 render sites in `public/packets.js` (`buildGroupRowHtml` header row, its child rows, `buildFlatRowHtml`), when `payload_type === 9` (TRACE) render `—` in the `col-hashsize` cell with a `title` tooltip explaining that TRACE path bytes are SNR readings and directing users to the sidebar decoder for the actual hop count. Non-TRACE rows unchanged. ## TDD - Red commit: `dd7a4e78` — `test-issue-1849-trace-hashbytes.js` asserts col-hashsize cell equals `—` with a title tooltip for TRACE, numeric for non-TRACE. CI must fail on this commit. - Green commit: `115368d0` — 3-site fix in `public/packets.js`. ## Verification ``` $ node test-issue-1849-trace-hashbytes.js ✅ All 4 tests passed ``` Preexisting failures in `test-packets.js` (13) are unchanged by this PR (verified via `git stash`) — unrelated to the surface touched here. ## Scope - `public/packets.js`: 3 render sites, +21/-8. - `test-issue-1849-trace-hashbytes.js`: new unit test. - `test-all.sh`: wire new test. No public API change. --------- Co-authored-by: clawbot <bot@example.invalid> |
||
|
Kpa-clawbotandcorescope-bot
|
8c3e397d39 |
fix(1846): drop 1200px cap on .observers-page (#1847)
Red commit:
|
||
|
Kpa-clawbotandcorescope-bot
|
c7f0c6931f |
fix(1843): restore QR quiet zone on node-details codes (#1844)
## Summary QR codes on node-details pages didn't scan. Two `public/nodes.js` render sites called `qr.createSvgTag(3, 0)`; the vendor lib treats the second arg as an **absolute pixel margin**, so `0` removed the QR quiet zone. QR spec requires ≥4 modules of light border, so every scanner rejected the code. With `cellSize=3`, four modules = 12 pixels. Changed both call sites to `createSvgTag(3, 12)`: - `nodes.js:813` — node-details list detail - `nodes.js:1705` — node-map overlay (also swaps background to transparent; with margin restored, dark modules stay 12px from the SVG edge so the transparent overlay still parses over the map tile) Out of scope, filed as follow-up in triage: contrast tuning of the transparent-overlay branch. ## TDD Red commit: `3be8552b` — [test-only, CI red](https://github.com/Kpa-clawbot/CoreScope/commit/3be8552baff64795504e8dcb888ed25cbd6a50be) Green commit: `6f7e83a6` — two-line fix + test passes ## Test `test-issue-1843-node-qr-quiet-zone.js` — Node/vm harness that loads `public/vendor/qrcode.js`, grep-asserts both `nodes.js` call sites use margin ≥ 12px, renders the SVG, and checks the `<path>` `M` coordinates keep all dark modules ≥ 12px from every viewBox edge (and viewBox = `modules*cellSize + 2*margin`). Verified red on parent commit (test fails with margin=0 leaving dark modules at 0,0), green after fix (dark modules at 12,12 with 12px free border). ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` — all gates clean. Browser verified: pending — will validate on staging after CI. (Frontend UX bug; DOM/grep test above exercises the same SVG code path that scanners consume.) E2E assertion added: `test-issue-1843-node-qr-quiet-zone.js:63` (viewBox + `<path>` coord grep on real-rendered SVG). Fixes #1843. --------- Co-authored-by: corescope-bot <bot@corescope.dev> |
||
|
Kpa-clawbotandcorescope-bot
|
59cf5130d1 |
fix(1838): fold non-transport routes into scope-stats Unscoped (#1842)
Fixes #1838 ## Problem `/api/scope-stats` reported 100% scoped whenever any region was configured. Reporter noticed on a scopeless instance that "unscoped" was always zero — the pie visual is misleading to operators deciding on `denyf *`. ## Root cause `cmd/server/db.go:22` restricted the entire scope-stats denominator to `route_type IN (0, 3)`. Per firmware `docs/packet_format.md § Route Types`: - `0` = `TRANSPORT_FLOOD` - `1` = `FLOOD` - `2` = `DIRECT` - `3` = `TRANSPORT_DIRECT` Only routes 0 and 3 carry `transport_code_1` (transport-level scope). Routes 1 and 2 are inherently unscoped by protocol. The existing SQL was correct for the "how many transport-scopable routes are actually scoped" question, but the denominator was silently promoted to "all traffic" in the UI. Bonus: the comment on `routeTypeTransportSQL` labelled routes 0+3 as "FLOOD (0) and DIRECT (3)" — wrong on both counts. ## Fix - `cmd/server/db.go` — corrected the `routeTypeTransportSQL` comment; added `routeTypeNonTransportSQL = "route_type IN (1, 2)"` alongside it. - `GetScopeStats` runs a second `COUNT(*)` over `route_type IN (1,2) AND first_seen >= ?` and folds that count into `Summary.Unscoped`. Same index path as the existing query — one extra scan per `/api/scope-stats` call (cached 30s per triage's carmack finding). - `public/analytics.js` — Scopes tab header explains the denominator (all observed transmissions) and which route types carry scope. Card notes now render `X% of all traffic` for Scoped/Unscoped and `X% of scoped` for Unknown Scope so the pie's denominator is explicit. ## TDD - Red: `5554ffe4` — extended `TestGetScopeStats` + `TestHandleScopeStats` with `route_type=1` and `route_type=2` rows and asserted `Unscoped = 3` (1 transport-NULL + 2 non-transport). Ran the tests and confirmed assertion failure (`Unscoped = 1, want 3`). - Green: `ebbb9253` — implementation + label copy. Full `go test ./cmd/server/...` passes (54s). ## Preflight overrides - check-branch-clean: justified — cross-stack fix by design (backend semantics change + matching frontend label copy). All 4 files are exactly the surface the triage comment identified. ## Verification - `go test ./cmd/server/...` — 54s, all pass. - Firmware confirmation: `firmware/docs/packet_format.md:20-24` (route type table). ## Files touched - `cmd/server/db.go` — comment fix + second COUNT query. - `cmd/server/db_test.go` — extended fixture. - `cmd/server/routes_test.go` — extended fixture + isolate from seed data. - `public/analytics.js` — labels and header copy. --------- Co-authored-by: corescope-bot <bot@corescope.dev> |
