2900 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Sylvain Rabot	3fbff01f64	build: upgrade Go toolchain to 1.27 (#1946) ... ## Summary - Bumps the Go toolchain used to build/test to 1.27: `golang:1.27-alpine` in `Dockerfile` and `Dockerfile.go`, and `go-version: '1.27'` in the three `actions/setup-go` steps in `.github/workflows/deploy.yml`. - Each module's `go.mod` `go` directive is intentionally left at `1.22` — no 1.27-only language features are being adopted, and a 1.27 toolchain builds a `go 1.22`-declared module without issue. ## Test plan - [x] `go build ./...` + `go vet ./...` for all 13 modules (`cmd/server`, `cmd/ingestor`, `cmd/migrate`, `cmd/decrypt`, 10 `internal/*` packages) under Go 1.27.0 - [x] `go test ./...` passes for `cmd/server`, `cmd/ingestor`, `cmd/migrate`, `cmd/decrypt` - [ ] `docker build` against the new `golang:1.27-alpine` base (Docker wasn't available in the sandbox this change was prepared in — needs a check in CI or locally)	2026-09-09 11:49:33 +02:00
efiten	2c8c1161b5	feat(#1975): network-wide Scope Audit page, fed by confirmed scopes (#1976) ... One row per repeater whose configured region list is known, answering a question no other view answers: you declare these regions, but were you seen forwarding them? default_scope says what a node's adverts were observed under and transported_scopes (#1751) says what it carried, but nothing lined the declared list up against observed forwarding. The declared side merges every confirmed-scope source the database carries, newest answer per node wins, rather than naming one. On a stock install only nodes.configured_scope (#1865/#1971) exists and it degrades to the one-source case; deployments that collect the same fact another way keep working. Reading a single hard-coded source would have rendered an empty page on the very instance the evidence came from. Declared and observed are compared through normScope, so a leading "#" and a bare region name are one region. Unobserved regions render neutral, not red: absence over a short window is weak evidence, which the page header already states in words. Ported from a long-running fork deployment with its 17 server tests, rewired to the upstream data source, plus 14 frontend cases asserting rendered markup.	2026-09-06 23:04:42 +02:00
efitenandClaude Opus 5	5b689f75fe	feat(#1845): filter nodes by how long they have been silent (#1973) ... Closes #1845 for the question in its title. The alerting ask that came later in the thread is deliberately **not** in here; see the bottom. ### The gap @Jonher937 asked to flag repeaters that stopped communicating for x days, to find remote gear that has died. Today the Nodes page has Active/Stale with thresholds fixed at 72h for infra and 24h for everything else, plus a Last Heard filter that selects nodes heard **within** a window. Neither answers "show me what has been quiet for over a week". ### What this adds A `Silent for` select beside Last Heard: 1d, 3d, 7d, 14d, 30d, each labelled with the number of nodes it would select. ``` [ All ] [ Active ] [ Stale ] Last Heard: Any v Silent for: over 7d (23) v ``` Counts are computed **before** the silence filter is applied, so the dropdown keeps showing what the other windows would select instead of collapsing to the one already chosen. The choice persists in `localStorage` like the neighbouring filters and is mirrored into the URL as `?silent=7d`, so the view can be pasted to whoever owns the silent gear. The URL sync is wrapped in try/catch, because it is a convenience and must never stop the filter working. ### The part worth reviewing: one definition of freshness, not two `getNodeStatus` has been relay-aware since #1598, while `nodes.js` separately computed `statusAge` from the ADVERT timestamp alone. Filtering on the latter would have listed a repeater as silent for ten days while its own badge on the same row said active, and it would have done so for **exactly** the nodes #1598 exists to protect. So the freshness rule is extracted into `window.getEffectiveHeardMs` in `roles.js`, and `getNodeStatus` now calls it. Behaviour is unchanged, there is now one source. Reviewers should look hardest at that refactor rather than at the select. A node never heard from at all scores `Infinity`, so it matches every window instead of silently dropping out of the filter. ### Why the thresholds are fixed values and not derived I measured the alternative before writing this, on a 1179-repeater mesh, and posted it on #1611: replacing a fixed threshold with `3 x per-node advert median` fixes 8 false "silent" flags and newly mis-flags **28 currently-active nodes**, because a 3h-median node gets a 9h threshold. Raising the global default to 144h rescues 7 and hides 27 genuinely dead repeaters. Both are net-negative. A user-chosen window sidesteps the whole question: the operator picks what "too long" means for their mesh, which is what @Jonher937 asked for in the first place. ### Verification - `test-frontend-helpers.js`: **666 → 680 passed, 0 failed**. Fourteen cases covering `NaN` rather than `0` when nothing is known (0 is a real timestamp and would sort as very old rather than unknown), the full `_liveSeen > _lastHeard > last_heard > last_seen` precedence, a recent relay beating a stale advert, a stale relay **not** dragging a fresh advert backwards, relay alone sufficing, `room` counting as infra while `companion` does not (a `last_relayed` on a companion is meaningless and must not rescue it), case-insensitive roles, the legacy `(role, ms)` call shape, and the 72h boundary asserted at 71h and 73h. - One of those failed on first run and **the test was wrong, not the code**: `9e7` ms is 25h, which is correctly active for infra. Fixed, and the boundary is now asserted explicitly so nobody repeats it. - `eslint` on the changed files: 0 errors. The warnings present are the same ones master already reports. - No server change, no new API, no new column. No cache-buster bump needed: `__BUST__` is substituted at startup in `cmd/server/main.go:570`. ### Not done **No alerting.** @fokcuk asked on the thread for notification when a repeater they look after goes silent, which is a different product: subscriber identity, an evaluation loop and delivery, none of which exist today. That deserves its own issue and a design call rather than being shimmed into `nodes.js`, which is also what the triage concluded. This PR gives the operator the view; it does not push to them. Sizes and boundaries (1d/3d/7d/14d/30d) are a judgement call. Say the word if a different set fits real operator habits better. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-06 21:43:20 +02:00
n30nex	2288e28d4e	fix: publish release artifacts after successful image retagging (#1964) ... Successful release fast paths publish image tags but never dispatch the job that creates the GitHub release and decrypt binaries. Dispatch `deploy.yml` from both image routes. A default-off `images_published` input skips E2E and image rebuilding only for an already-published tag; missing or mismatched images keep the complete fallback without requiring new inputs on older workflow definitions. Go validation still gates the release binaries, checkout and version flags retain the tagged source, and the existing release action uploads both architectures before publication. Missing binary files now fail publication. Fixes #1956. Validation: - `node test-issue-1956-release-routing.js` executes the actual workflow shell steps with registry and dispatch commands stubbed. Covers matching, missing and mismatched images; failed retag and Go validation; branch/PR boundaries; and both tagged binary commands. - The original test commit fails because a matching image dispatches zero artifact workflows; the fix passes the same assertion. - Existing release workflow Go checks, decrypt/channel tests, YAML parsing and actionlint pass. - Both static Linux amd64 and arm64 binaries cross-build with verified architecture and version metadata. Actual registry publication and GitHub release creation were not exercised. Existing immutable releases and old tags that contain older workflow definitions are outside this fix. Following #1922, this is a focused release-routing PR. A separate repair for #1858 rewrites the shared frontend test runner; merging this first lets that repair retain this regression in its authoritative list. Please assess current Go and E2E job results separately from workflow-approval or staging-runner state.	2026-09-06 21:20:16 +02:00
efitenandClaude Opus 5	1ffaad8eb1	feat(#1794): per-IP limits and a deny list on the /ws upgrade (#1974) ... Closes #1794. Follow-up to #1793, decided **before** the upgrade because the handshake is the resource being protected. - Deny list of addresses and CIDRs → 403 - Per-IP concurrent connection cap → 403 - Per-IP upgrade rate limit over a rolling minute → **429**, not 403: a temporary refusal should not read as "never come back" - Rejection counters split by cause in `/api/stats` under `websocket` ### The decision this feature lives or dies on Most CoreScope installs sit behind nginx, Caddy, Traefik or an ingress. `cdn_detection.go` says so in as many words: it deliberately excludes `X-Forwarded-For` from its CDN signals precisely because *every* reverse-proxied install sets it. For those deployments `r.RemoteAddr` is the proxy, `127.0.0.1` for every visitor on earth. A per-IP cap keyed on that address protects nobody and hands the sixth legitimate browser tab a 403. That is a self-inflicted outage wearing the costume of hardening. So: - **`X-Forwarded-For` is believed only from an address listed in `webSocket.trustedProxies`.** From anywhere else it is attacker-supplied, and trusting it would let anyone mint a fresh source IP per connection, which is strictly worse than having no limit at all. - **When the peer looks like a local reverse proxy and no `trustedProxies` is set, the per-IP limits are skipped**, and one warning names the setting that fixes it. Silently refusing real users is the worse failure. - **The deny list still applies there**, because it is the operator's explicit instruction rather than an inference. That is the answer to @mcode6726's question on the thread: it is neither "always the socket address" nor "always the header", and the operator decides which by naming their proxy. ### Two deliberate departures from the issue body **`maxConnsPerIP` ships as 0 (off), not 5.** Carrier-grade NAT puts thousands of unrelated mobile subscribers behind a single public IPv4. A cap of 5 refuses real visitors on phones while a scraper simply rents more addresses: all of the cost, none of the benefit. `upgradesPerMinPerIP` ships at **30 and on**, because that one *is* safe under CGNAT: a real client upgrades a handful of times per minute even while reconnecting, so 30 leaves ordinary traffic untouched while flattening a reconnect loop. A pointer type distinguishes "unset" from an explicit `0` that turns it off. **The default deny list is not shipped.** The thread proposed seeding 44 CIDRs for one VPS provider after a single scraper was seen at `23.111.177.6`. I have left it out: blanket-blocking a hosting provider by default breaks legitimate operators who host there, is undiscoverable by the person locked out (they see a bare 403), and ages badly as ranges get reassigned. The mechanism is here and `config.example.json` shows exactly how to configure it, so any operator who wants that list can have it in one line. If you want it shipped as a default anyway, that is your call as maintainer and it is a one-line change. ### Verification 19 tests, including all five the issue specifies as TDD requirements, each marked with the issue's own wording. Beyond those five: - a **bare address** in the deny list works, not just CIDR form. Operators write `1.2.3.4`, and silently ignoring that would be the worst possible failure for a deny list: it looks configured and blocks nothing - an unparseable deny entry is skipped and logged, not fatal. One typo must not take the server down - one client behind a trusted proxy does **not** exhaust another client's budget behind the same proxy, which is the entire point of honouring XFF - changing a forged XFF from an untrusted peer buys no fresh budget - `release` frees a slot and is **idempotent**, because `Unregister` can run twice for one client and double-crediting would leak slots - a **rejected** upgrade does not consume rate budget, or a retrying client could never recover once its window cleared - limits skipped for loopback and private peers; deny list applies anyway - a nil limiter allows everything, so a `Hub` built without `ConfigureLimits` behaves exactly as before - idle per-IP state is collected, while a record with a live connection never is Full `cmd/server` suite green, `gofmt` clean. ### Not done - No runtime config reload; restart required. Listed as optional in the issue. - No `WS_DENY_IPS` env override. Also listed as optional. - From the OWASP expansion in the first comment: `maxPayload` and the idle/read timeout are **already in master** (`SetReadLimit`, `SetReadDeadline`). The ping/pong heartbeat is not, and is not in this PR either; it is a separate change to the read/write pumps and belongs in its own review. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-06 21:11:29 +02:00
efitenandSaarMesh-Bot	9d6f08c144	feat(#1865): ingest observer /neighbors as confirmed scope evidence (#1971) ... Carries SaarMesh-Bot's implementation from the closed #1867 forward onto current master, 67 commits later, and surfaces the result on the per-node Reach report. The declared region list a repeater answers with is now stored on the node as configured_scope, normalised to the same leading-# syntax default_scope already uses so the two are directly comparable. That normalisation is the point @cwichura raised on #1865 and @dborup agreed with before the original PR closed. Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>	2026-09-06 21:06:12 +02:00
n30nex	43d83ee8ca	fix(nodes): dispose map timers with their owning view (#1970) ... Red commit: `2f1a50e` (local Chromium: 2 passed, 7 assertion failures). Ownership regression: `059ab49` (9 passed, 4 assertion failures). CI: [run](https://github.com/Kpa-clawbot/CoreScope/actions/runs/33988139479) awaits maintainer approval (`action_required`); 0 jobs started. Rapid navigation or closing node detail could leave a delayed resize targeting a removed or replacement map. Disposal now cancels its timer, each resize captures its own map, and delayed responses respect the view owning the current map. Stale side-pane responses are ignored before rendering. Fixes #1972. - E2E assertion added: `test-issue-1206-resize-observer-leak-e2e.js:216` and `:292`. This existing CI-selected suite covers navigation, replacement deadlines, close/Escape, no-location rendering, and late error/success responses. Existing observer-growth assertions remain intact; readiness waits replace fixed sleeps. - Browser verified: local fixture with real Chromium and Leaflet; 13 browser checks passed after push. Evidence: `data/node-map-validation/post-push-browser.log` and `data/node-map-validation/evidence.md`. - Validation: frontend unit suites 99/18/666 passed; JavaScript syntax, CSS variables, whitespace, PII and XSS checks passed. Backend unchanged; Go suites not rerun. - Independent adversarial, expert and TDD reviews found no required changes. Two original navigation checks initially timed out; the unchanged parent rerun passed 13/13 (`parent-browser-confirm.log`). - Performance/config: one timer handle, no packet/node loops or new requests. Tests enforce zero stale invalidations and one resize at the surviving map's deadline. Existing 100ms delay retained; no new settings or throughput claim. Fix commits: `2492a65`, `1dc090d`. ## Preflight overrides - External `run-all.sh` is unavailable. Scoped branch, red/green, PII, CSS, XSS and whitespace checks were run directly; no migrations, SQL attribution or image markup are added.	2026-09-06 21:04:09 +02:00
n30nex	a938176f83	fix(nodes): clearly mark the selected node in path chains (#1968) ... Red commit: `0988bc0` (local browser: 3 passed, 8 behavior assertion failures before the fix). The selected node now has a compact outline in long “Paths Through This Node” chains, in the side panel and full detail page. Matching uses complete public keys without case sensitivity; same-prefix siblings and unresolved hops stay unmarked. Existing links, escaped names, warnings and ambiguity underlines are preserved. Fixes #1153. Its prerequisite #1144 is already merged. - E2E assertion added: `test-issue-1146-path-link-contrast-e2e.js:220`. The existing CI-selected harness passes 11 checks across 18-hop paths, both themes, desktop/mobile, and the renderer fallback. Review follow-up `bd8118f` verifies the marked ambiguous hop's dashed underline. - Browser verified: `http://127.0.0.1:55635`; desktop/mobile path screenshots were inspected. The broader smoke runner exited successfully with fixture-dependent skips. - Required frontend checks pass: 99 filter, 18 aging, 666 helpers. CSS variables, seven CSS self-tests, 31 XSS sink checks, 17 XSS gate self-tests and XSS diff preflight pass. - Three independent reviews found no blocking issues. Traversal remains linear with no new requests, settings, dependencies or cache invalidation; styling uses the existing customizer token. ## Preflight overrides - The external preflight runner is absent; corresponding scoped gates passed. Red browser evidence is local, with upstream CI approval tracked separately under the process in #1922. - Existing rapid-navigation map resize timer errors remain visible in browser logs and are outside this change.	2026-09-06 21:04:03 +02:00
n30nex	108ea020f7	fix(nodes): remove misleading aggregate SNR headlines (#1969) ... Red commit: `2bf9be8` (local Chromium: 3 passed, 3 intended assertion failures). CI: [run](https://github.com/Kpa-clawbot/CoreScope/actions/runs/33988112224) awaits maintainer approval (`action_required`); 0 jobs started. Remove the unqualified aggregate Avg SNR row from node side-panel Overview and full-detail stats, following option 3 in #1149. Heard By retains each observer's SNR reading. Fixes #1149. - E2E assertion added: `test-issue-1281-location-row-e2e.js:224`. Three new browser cases cover desktop side/full and mobile full views with a numeric aggregate and distinct positive/negative observer readings. Existing packet-location assertions remain intact. - Browser verified: local Chromium; 6 cases passed after push. Screenshots: `coverage/issue-1149/issue-1149-desktop-side-panel.png`, `coverage/issue-1149/issue-1149-desktop-full-detail.png`, and `coverage/issue-1149/issue-1149-mobile-full-detail.png`. - Validation: packet filter 99/99, aging 18/18, frontend helpers 666/666; XSS, CSS-variable, syntax, whitespace and PII checks passed. - Independent reviews: adversarial, lifecycle expert and TDD reviewers found no required changes. One initial browser navigation timed out; the unchanged parent rerun passed 6/6. - Performance/config: two production row deletions; no new requests, loops, timers, settings or customizer implications. Backend unchanged; Go suites were not rerun. Fix commit: `d7c68f3`. ## Preflight overrides - External `run-all.sh` is unavailable on this host. Scoped branch, red/green, PII, CSS, XSS and whitespace checks were run directly. The diff adds no migrations, SQL attribution or image markup.	2026-09-06 21:03:58 +02:00
n30nex	eb1d733998	fix(analytics): preserve selected hash size in links (#1967) ... Red commit: `5a5ecb6` (local browser: 14 passed, 8 behavior assertion failures before the fix). Hash Issues links now restore `bytes=1|2|3` for the selected control and its matrix/collision data. Missing or malformed values default to one byte. Selector clicks, section/top links, tab-bar changes, filters and theme refreshes retain the chosen view through the existing URL helper. Fixes #1914. - E2E assertion added: `test-issue-1306-collisions-terminology-e2e.js:242`. The existing CI-selected harness passes 23 checks, including distinct nonempty collision rows for each byte size. Its original assertions remain. - Browser verified: `http://127.0.0.1:55634` with the local fixture API, plus reviewed matrix/risk screenshots. Region refresh passed; area coverage skips because the fixture has no areas. - Required frontend checks pass: 99 filter, 18 aging, 666 helpers; URL helpers pass 18. Three independent reviews found no blocking issues; their coverage suggestion is included in `fb482fe`. - Added work parses URL state and updates six links. Rendering and bulk requests are reused; no backend, configuration, dependency or CI-list changes. - A broader smoke run timed out at Live autocomplete (#1110); full-suite success is not established. ## Preflight overrides - The external `run-all.sh` is absent. Corresponding scope, PII, syntax, whitespace and CSS checks passed; no SQL, migration or image changes require those gates. - Red browser evidence is local. Upstream CI execution remains a separate approval gate, as discussed in #1922.	2026-09-06 21:02:14 +02:00
n30nex	cf67a5e5ec	fix: remove evicted resolved path hops and preserve relay snapshots (#1966) ... Eviction removes raw wire hops but leaves resolved full-key entries in `byPathHop`, retaining expired transmissions and stale relay counts/scopes. Filter every hop bucket once per eviction batch using the existing evicted-ID set, remove duplicate references and empty buckets, and clear discarded pointer slots. Bulk relay aggregation now owns its bucket snapshots before releasing the read lock, so eviction and raw-path updates cannot mutate an in-flight reader. Three existing handler test fixtures also wait for index readiness or explicitly simulate not-ready state, preserving their original 200/503 assertions. Fixes #1908. Validation: - Regression commits fail before their corresponding fixes: resolved keys/counts/scopes remain after eviction, and saved relay snapshots change during eviction. - Targeted eviction, relay, scope, cache and concurrent-reader checks pass under `-race`; coverage includes time/cap eviction, missing resolved-path prefetch, disabled membership indexing, duplicate references, retained backing arrays and surviving entries. - Local browser smoke: nodes, node details/path attribution and analytics render using the fixture-backed Go server. - The last full Windows server race run, before the final snapshot-copy correction, had one remaining DB-only timing failure (`TestGetChannelMessagesPerfLargeChannel`: 2.198s against a 1.5s budget). The final correction was checked with focused race tests. The unchanged ingestor suite also cannot create one symlink without Windows privileges. These thresholds/assertions were preserved; full Linux Go/E2E results still require upstream CI approval. Performance tradeoff: cleanup is O(total indexed pointers) per nonempty eviction batch, under the existing write lock. The minute-based ticker pays for one sweep instead of repeated scans of shared raw buckets. No per-transmission string index or dependency is added. Synthetic benchmark medians (three single-iteration runs, shared Windows host): | Transmissions | Evicted | Before | After | |---:|---:|---:|---:| | 30,000 | 1 | 1.07 ms | 14.19 ms | | 30,000 | 3,000 | 56.27 ms | 61.68 ms | | 30,000 | 7,500 | 83.58 ms | 65.54 ms | | 100,000 | 1 | 0.30 ms | 56.95 ms | | 100,000 | 10,000 | 949.93 ms | 190.71 ms | | 100,000 | 25,000 | 1,834.48 ms | 320.38 ms | Fixture: eight raw plus eight resolved hops per transmission, two observations, 2,048 relays; 480,000/1,600,000 hop entries. Timing includes acquiring the store lock and omits unrelated secondary indexes. Small batches now pay for the complete sweep; shared-host timing is noisy. Owning the bulk reader's arrays also has a measured cost on cold/bulk recomputation, rather than cached hits. Snapshot medians from three samples of ten iterations: | Transmissions / relay nodes | Before time / bytes per operation | After time / bytes per operation | |---|---:|---:| | 30,000 / 50 | 0.0068 ms / 5,416 B | 23.65 ms / 4,101,435 B | | 30,000 / 2,000 | 0.1374 ms / 196,768 B | 26.77 ms / 4,274,336 B | | 100,000 / 2,000 | 0.1376 ms / 196,768 B | 27.89 ms / 13,959,337 B | These are total snapshot costs, comparing the unsafe header-only snapshot with owned pointer arrays. Cleanup guarantees here apply to `byPathHop`; other indexes and existing periodic bulk-cache freshness are outside this change. Following #1922, this runtime fix is separate from the release-routing and frontend-runner PRs. Current Go and E2E job results should be assessed separately from workflow-approval or staging-runner state.	2026-09-06 21:00:44 +02:00
TeTeHacko	6ae7971da0	test(#1356): assert the rendered label, not where identifiers sit in map.js (#1933) ... Follow-up to the review on #1912, where this assertion cost a round trip. Independent of that PR — this branch is off current `master` and touches no code path it changes. ## The problem `#1356 V3.e`, `V3.f` and `V3.g` all describe what `makeRepeaterLabelIcon` **produces**, but all three assert it by grepping `public/map.js`. V3.e also bounds the distance between two identifiers: ```js assert(/MB_GLYPHS\[[^\]]+\][\s\S]{0,200}shortHash|shortHash[\s\S]{0,200}MB_GLYPHS\[/.test(mapSrc), 'makeRepeaterLabelIcon prepends MB_GLYPHS glyph to the hash text'); ``` Three separate failure modes, all observed: **1. It fails on edits that change nothing.** #1912 inserts one variable declaration in that function; the markup is byte-identical and the build went red. **2. It cannot tell code from prose about code.** My first attempt at fixing #1912 added a comment explaining the constraint — and the comment mentioned both identifiers, so it satisfied the grep by itself. With that comment present I moved the hash assignment away from the glyph, reintroducing the exact defect, and the test still reported green. A check that a comment can satisfy is worse than one that is merely brittle. **3. It does not assert the thing it is named after.** On `master` the match is not the declaration order at all. It is `MB_GLYPHS[...]` reaching the *later* `shortHash` inside `ariaStatus`, 212 characters downstream. Whether the glyph is actually prepended to the hash is incidental to whether this passes. That third point also corrects something I said on #1912, and it corrects it against myself: both the 312 you quoted and the 299 I "corrected" it to are the distance between the two **declarations**, which is not the distance the regex uses. Measuring the one it does use: | tree | `MB_GLYPHS[` → next `shortHash` | assertion | |---|---|---| | `master` | 212 | pass | | #1912 before the fix | 277 | fail | | moving `unknownWidth` below the glyph | **343** | fail | | moving `shortHash` below the glyph | 54 | pass | So moving `unknownWidth` down does not merely fall short — it makes the gap *worse*, because it lands between the glyph and `ariaStatus`. My earlier "233, still 33 over" was the wrong metric on the wrong pair. Apologies; the conclusion happened to hold but the reasoning did not. ## What this does Loads `map.js` in the same DOM-less `vm` sandbox `test-map-clustering.js` already uses, exposes `makeRepeaterLabelIcon` through the existing `window.__meshcoreMapInternals` hook, and asserts the emitted markup: - glyph, `U+2009` thin space, hash — in that order and adjacent; - no glyph and no thin space when there is no multi-byte status; - `aria-label` exactly `multi-byte <status>, hash <ID>`, and `repeater hash <ID>` without one; - the visible span carries `aria-hidden`. No browser, so it stays in the JS-unit-tests step rather than moving to Playwright. ## Mutation-tested, not eyeballed | mutation | old V3.e/f/g | new | |---|---|---| | glyph moved after the hash | **all silent** | caught | | plain space instead of `U+2009` | **all silent** | caught | | span loses `aria-hidden` | V3.g caught | caught | | `aria-label` loses its comma | **all silent** | caught | | 200 chars inserted between the identifiers (no behaviour change) | V3.e **fails** | passes | Full JS unit list from `.github/workflows/deploy.yml`: 65/65. ## Notes for review - V3.a–V3.d (MB_GLYPHS definitions, CSS variables, the border rule) are left as source/CSS greps. The glyph values are now covered implicitly by the rendered-output assertions, but converting the CSS ones needs a different approach and did not belong here. - The sandbox loader has no `try`/`catch` that warns and continues. If `map.js` stops loading, the suite must fail rather than silently skip every assertion below it. - If this lands, the ordering comment in #1912 becomes obsolete and I will drop it there. I deliberately did not touch it from this branch so the two do not conflict textually.	2026-09-05 15:05:43 +02:00
TeTeHackoandClaude Opus 5	75dfa1f4fb	fix(map): render an unobserved hash size as unknown, not as 1 byte (#1912) ... ## The bug `map.js` turns a missing `hash_size` into `1`: ```js var hs = node.hash_size || 1; ``` That field is **evidence**, not a default — `computeNodeHashSizeInfo` populates it only from adverts it could read a size out of, so a node with no countable advert in the retention window has no value at all. Rendering that absence as `1` states a 1-byte configuration nobody observed, and it does so in the one place where a reader is most likely to act on it. It is also inconsistent with the rest of the UI for the *same field on the same node*: | view | code | renders | |---|---|---| | node detail | `nodes.js:683` | `Hash Prefix: **Unknown**` | | analytics prefix table | `analytics.js:1553` | `(**?**B)` | | map popup + label + filter | `map.js:140`, `:1588`, `:1775` | `C8 **(1B)**` | On analyzer.meshcore.cz right now, **701 of 1007 nodes** have `hash_size: null`, so the map's 1-byte bucket is mostly nodes that were never measured. The Byte Size filter has the same problem from the other end: picking "1-byte" returns measured 1-byte nodes *and* every unheard node, which makes it hard to use for the thing it exists for. ## The fix - `roles.js`: shared `hashPrefixInfo(node)` → `{known, bytes, prefix}`, so the map stops re-deriving the prefix in three places and the "unknown" rule lives in one. - `map.js`: - **label** still draws a 1-byte prefix (it has to draw *something*) but carries `.hash-unconfirmed` and its `aria-label` says `…, hash size unknown`; - **popup** says `Unknown`, matching `nodes.js` wording; - **filter** gets its own `Unknown` bucket instead of folding unmeasured nodes into 1-byte. - `style.css`: dotted underline for the unconfirmed prefix — a shape cue rather than a colour one, so it survives forced-colors and colour-vision differences, consistent with the #1356 approach for these labels. `nodes.js` and `analytics.js` are left alone: they already behave correctly, and switching them to the helper would widen the diff without changing behaviour. Happy to do it in a follow-up if you'd rather have the call site count at zero. ## Tests `node test-frontend-helpers.js` → **635 passed, 2 failed**; the two failures are `favStar`, pre-existing on master (baseline run before this change: 625 passed, 2 failed — same two). 10 new cases: `hashPrefixInfo` across missing / null / 0 / 1 / 2 / 3-byte inputs plus missing pubkey and a null node, and a guard asserting `map.js` contains no bare `hash_size || 1` so this cannot quietly come back. ## Browser validation Headless Chromium against a live instance carrying real mesh data, same viewport (`#/map?lat=50.038502&lon=14.570556&zoom=17`), unpatched vs patched: | | before | after | |---|---|---| | `aria-label` | `repeater hash C8` | `repeater hash C8, hash size unknown` | | label class | `mc-mb-label` | `mc-mb-label hash-unconfirmed` | | filter buttons | `all,1,2,3` | `all,1,2,3,unknown` | The four nodes in that viewport that *do* have evidence (`157E`, `381E`, `FA74`, `C029`) render exactly as before. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-05 14:59:20 +02:00
efitenandClaude Opus 5	eb3d71f8f6	perf(#1910): collapse concurrent /stats work and serve the count cache stale (#1963) ... Addresses #1910. The Observers page hangs on "Loading..." for 10-20s; the reporter measured `/stats` at 10-17s under the mixed load that page produces, while the same endpoint stays under 70ms at 8x concurrency when it is the only one being hit. ## Cause Two cache layers guard the expensive work and **neither has single-flight**: | | | | |---|---|---| | `handleStats` | 10s cache | releases `statsMu` before rebuilding (`routes.go:774`) | | `GetStoreStats` | 30s cache | releases `statsCacheMu` before scanning (`store.go:2048`) | Both do check, release, then work. The moment either window expires, **every in-flight request does the whole thing itself**. The expensive part is a range scan over 24h of `observations` with two `SUM(CASE...)` over it. The column is indexed (`idx_observations_timestamp`), but the scan still visits every row in the window, and at 18k observers that is millions. The pool is `SetMaxOpenConns(4)` (`db.go:111`), and the page fires stats, observers, nodes, channels and clock-skew at once, so one cache miss turns a single scan into a queue of them. That is exactly the reported profile: fast alone, slow only when mixed. ## Changes 1. **Single-flight both layers.** Concurrent callers that miss the cache wait for the first one's result instead of each running the same queries. 2. **Serve the observation counts stale while refreshing in the background.** An expired cache answers from the previous value and kicks off one refresh, so a miss is never a wait. Single-flight alone would not have fixed the hang: the first caller still waits for the full scan. The second change is what removes it. ## Contract change, stated plainly `TestGetStoreStats_CacheExpiry` asserted that an expired cache returns **fresh DB values on the same call**. It no longer does. For `packetsLastHour` / `packetsLast24h` on a dashboard, answering with a value up to ~30s older instead of blocking for seconds looks like the right trade to me. But that is a judgement, not a bug fix, and **a reviewer should be able to reject it**. I did not quietly delete the test: it now asserts what still has to hold, that the refresh happens, and the new behaviour is pinned separately by `TestGetStoreStats_StaleCacheServedWithoutBlocking`. If you would rather keep the old contract, drop change 2 and keep change 1; the diff separates cleanly. ## Verification The new test **fails without the change**: ``` stale cache not served: got (0, 2), want (424242, 434343). An expired cache must answer from the previous value and refresh in the background, not block the request on the observations scan ``` `gofmt` clean, `go vet` clean, `cmd/server` suite ok in 267s. ## Two things I did not verify **The race detector.** It needs cgo and there is no gcc on this machine, so `go test -race` cannot run here. This change adds a background goroutine writing the cache under `statsCacheMu`, so that check matters. CI runs `go test -timeout 20m -race` for `cmd/server` (`deploy.yml:134`), which covers it before merge. **The 10-17s itself.** I have no database with 18k observers. The mechanism above explains the reported profile, including why the endpoint is fast in isolation, but I did not measure the figure. @dborup, if you can run a build from this branch, the number to watch is `/stats` under the same mixed-load command from your issue. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-05 09:57:44 +02:00
efitenandClaude Opus 5	83134d6e70	fix(#1896): say that the naive-clock notice clears itself (#1962) ... Closes #1896. The banner told operators their clock is naive but not that the notice goes away on its own, so people went looking for #1480 to find out. One sentence: > Clock is naive — per-packet timing clamped to ingest time. **Clears itself 24h after the last skew event.** ## Verified before writing it into the UI The issue asserts the 24h self-clear. Rather than repeat that, I checked it: - `cmd/server/observer_naive_clock.go:8` — `const observerNaiveClockWindow = 24 * time.Hour` - `applyObserverNaiveClock` applies the decay at read time and leaves `clock_naive` false once the last event is older than the window - its own comment: *"any event older than observerNaiveClockWindow is treated as absent so the chip and banner clear automatically without a background sweep"* So "24h after the last skew event" is accurate, including the fact that it needs no sweep and no restart. No test pinned the old string. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-04 17:05:01 +02:00
efitenandClaude Opus 5	71b892ba95	fix(#1899): scope the channel preview to the region filter (#1961) ... Closes #1899. The Channels sidebar preview (`lastMessage` / `lastSender`) ignored the active region filter, so an operator filtering on their own region saw a preview line from a message their observers never heard. ## Cause `GetChannels` scoped `msg_count` and `last_activity` correctly: the outer query joins `observations` and `observers` and filters on IATA. The `sample_json` subquery that feeds the preview joined neither, so it always returned the globally newest message on the channel. ## Fix Both region-filtered branches now scope the subquery the way the outer query does: v3 through `observations`/`observers` on `observer_idx`, v2 through the `EXISTS` on `observer_id`. The unfiltered branch is untouched, since there is no filter for it to respect. **One thing that is easy to get wrong here:** the subquery sits in the SELECT list, *ahead of* the WHERE, so its placeholders bind first. The region codes are appended twice, subquery set first, or every filtered call binds the wrong values. **Scope checked rather than assumed:** `GetEncryptedChannels` has the same shape and the same `regionPlaceholder` pattern, but selects no `sample_json`, so it does not have this bug and is left alone. ## Verification The regression test **fails on unmodified master**, with the reported symptom: ``` db_test.go:2465: preview sender = Bob, want Alice: SJC must not be shown Bob's message, which only SFO heard db_test.go:2469: preview message = heard in SFO, want "heard in SJC" ``` It asserts both directions, so it cannot pass by always picking the oldest row, and it asserts the unfiltered call still shows the globally newest message, which was never in question. `gofmt` clean, `go vet` clean, `cmd/server` suite ok in 98.5s. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-04 17:04:54 +02:00
efitenandClaude Opus 5	e6b31fc764	fix(#1927): stop plotting the usefulness composite on the traffic axis (#1960) ... Closes #1927. The scatter's **"Traffic share"** axis and the **"Traffic"** table column both fell back to `usefulness_score` when `traffic_share_score` was absent, with nothing marking the substitution. Two points on one axis could therefore measure different things. ## They really are different metrics Checked rather than assumed, in `cmd/server/usefulness_composite.go`: ```go node["traffic_share_score"] = trafficRaw // :147 the single traffic axis node["usefulness_score"] = composite // :152 0.30*bridge + 0.25*coverage + ... ``` `openapi.go:178,182` describes them the same way. So the fallback put a **composite** under a column and an axis that both promise share of non-advert traffic. Worth flagging, because it is easy to conclude the opposite: **#1456, which introduced the fallback, was a rename of the display label** from "Usefulness" to "Traffic share". That makes the two field names look interchangeable, and I nearly stopped there. They are not: #672 later gave `usefulness_score` its own composite meaning. ## Fix Your first preference in the issue: drop the fallback rather than mark it or relabel the axis. A node with no `traffic_share_score` now reads as unknown, so the table shows an em dash and the point is dropped from the plot by the existing `plottable` filter (`analytics.js:2728`), exactly the way a node with no bridge score already is. **No other change was needed** for that, which is what makes this the cheap option of the three. ## Tests The unit test that pinned the old behaviour is updated rather than deleted, so the expectation is now recorded the right way round: ```js assert(mapped[1].traffic === null && mapped[1].fav === false, 'a node with only usefulness_score has no traffic value; it must not be substituted'); ``` `test-repeater-metric-scatter.js`: 31 passed, 0 failed. Also corrected a comment above `_toScatterPoints` that still documented the removed fallback chain. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-04 17:04:48 +02:00
efitenandClaude Opus 5	e7b3a2e77f	chore(#1856): remove POST /api/packets, which has never worked (#1959) ... Part 1 of #1856. Part 2 (the hash migration reporting false success) is #1958. ## It has never worked `handlePostPacket` writes to the server's DB handle, and that handle is read-only. `cmd/server/db.go:106`: ```go dsn := fmt.Sprintf("file:%s?mode=ro&_journal_mode=WAL&_busy_timeout=5000", path) ``` Every call answered `500 attempt to write a readonly database`. **This is the second report.** #1196 raised it on 2026-06-13, a fix was merged that corrected v2 column names to v3, and the issue was closed. That fix could not have worked, because the column names were never why the write failed. Its comment is still sitting at `routes.go:1288`, next to code that has never executed successfully in production. ## Why remove rather than build a handoff **It cannot break a caller.** An endpoint that has only ever returned 500 has no working consumer. This is not a breaking API change, it is documentation catching up with reality. Nothing in `public/` calls it. **It was actively misleading.** `openapi.go` advertised it as "Ingest a packet" and it sits behind `requireAPIKey`, which reads as a live, protected write endpoint. **Its test hid the breakage.** `TestPostPacketPersistsV3Schema` asserted the observation row is written and passed for four months, because the test DB is opened read-write while production is not. That is how #1196 came to be closed as fixed. **Ingest is MQTT-only by design since #1283.** Re-adding an HTTP write path re-opens the invariant that change established. If manual injection is wanted later for testing or replay, it belongs on the ingestor side and deserves its own issue. The repository already has the handoff shape for that: the server writes `request-<id>.json` and the ingestor consumes it (`cmd/ingestor/prune_geofilter.go`). ## What went The route, `handlePostPacket` (103 lines), the now-unused `PacketIngestResponse` type, the `openapi.go` entry, the round-trip test, and the section plus table-of-contents line in `docs/api-spec.md`. The `packetpath` import in `routes.go` became unused and went with it. `+4/-225` across 6 files. ## The auth tests The four `requireAPIKey` tests used `"/api/packets"` only as a request path while building their own handler with `s.requireAPIKey(...)`, so they never touched the route. I checked that by **running them**, not by reading the code: ``` --- PASS: TestRequireAPIKey_RejectsWeakKey --- PASS: TestRequireAPIKey_AcceptsStrongKey --- PASS: TestRequireAPIKey_EmptyKeyDisablesEndpoints --- PASS: TestRequireAPIKey_WrongKeyUnauthorized ``` Their paths now point at `/api/admin/prune-geo-filter`, which still exists, so they no longer name a removed endpoint. Re-ran after that change: still 4 of 4. `/api/packets/observations` is a different endpoint and is untouched. Verified: `gofmt` clean, `go vet` clean, `cmd/server` suite ok in 62.9s. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-04 16:02:59 +02:00
efitenandClaude Opus 5	56d6d4c722	fix(#1856): stop the hash migration reporting success it never achieved (#1958) ... Part 2 of #1856. **Part 1 is deliberately not fixed here** and the issue stays open for it; reasoning at the end. ## The bug `migrateContentHashesAsync` set `store.hashMigrationComplete` in a deferred func that ran unconditionally. Every DB failure inside the loop takes a `continue` (begin tx, prepare, commit), so the loop always reaches that defer, **including when not a single batch was written**. That is not hypothetical. The server has held a `mode=ro` handle since #1283, so `Begin`, `Prepare` and `Commit` all fail, every batch is skipped, and `/api/stats` then answers `hashMigrationComplete: true` after migrating nothing. The migration is started unconditionally on every boot at `main.go:546`. ## The fix The three failure paths now count, and the defer only claims completion when the count is zero. When it is not, it logs once, naming the read-only handle as the expected cause and pointing at this issue, so an operator can tell "no work to do" apart from "could not do the work". **Nothing waits on the flag.** The only reader is `routes.go:828`, which reports it in `/api/stats`. Leaving it false on failure blocks nothing; it just stops the endpoint from lying. The in-memory index is untouched on failure. That was already true, because the index update runs only after a successful commit, and the test now asserts it so memory and disk cannot drift apart. ## Verification The regression test **fails on unmodified master**: ``` hash_migrate_test.go:115: hashMigrationComplete must stay false when no batch could be written; reporting true here is what #1856 called self-reported success ``` It closes the DB handle to make writes fail. That is deterministic and exercises the identical path as a read-only handle (`Begin` errors, batch skipped); the in-memory test DB cannot be reopened read-only. The existing happy-path test still passes, so the flag still turns true on a real migration. `gofmt` clean, `go vet` clean, `cmd/server` suite ok in 59.7s. ## Why part 1 is not in here `handlePostPacket` writes to the same read-only handle and therefore always answers 500. I checked the error path before assuming it was misleading: it already returns `"transmission insert: attempt to write a readonly database"`, so the message is accurate. The endpoint is not confusing, it is simply dead. The issue asks maintainers directly: *"is this endpoint still wanted? If ingestion is MQTT-only now, deleting it is simpler than routing it through a handoff."* That is a product decision, not a fix, and inventing a middle answer would only add code without settling it. Worth noting the repository already has a precedent for the handoff shape: the server writes `request-<id>.json` and the ingestor consumes it (`cmd/ingestor/prune_geofilter.go`). Two things a decision should account for: the endpoint is documented in `openapi.go:69` and guarded by `requireAPIKey`, and `routes_test.go:4850` asserts it writes an observation row using the v3 schema, which passes only because the test DB is read-write. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-04 15:15:16 +02:00
efitenandClaude Opus 5	9488e9c31c	fix(#1898, #1900): carry observer_id into replayed packets (#1957) ... Closes #1898. Closes #1900. Two issues, one omission. With a region filter active, VCR replay on the Live map rendered **nothing at all** (#1898), and the Replay button on packet detail **silently did nothing** (#1900). ## Cause `packetMatchesRegion` (`public/live.js:80-92`) matches a packet group by looking up `packets[i].observer_id` in the observer roster map. A packet whose `observer_id` is null is skipped, and when none match it returns `false` and the caller drops the whole group (`live.js:3374`). `dbPacketToLive()` returned `observer` (the resolved name) but never `observer_id`. So every replayed packet was skipped, and every group was dropped. The Replay button had the same gap: both branches passed `obsName(o.observer_id)` and threw the id itself away. **The value was there the whole time.** The VCR builds its entries with `Object.assign({}, p, obs, ...)`, so the observation's `observer_id` is on the input, and the server has returned `observer_id`, `observer_name` and `observer_iata` per packet since `cmd/server/db.go:345-347`. Only the object literal dropped it. ## Fix Carry `observer_id`, and `observer_iata` alongside it so `obsIataBadgeHtml` (`live.js:102-108`) can use the direct field for replayed packets instead of falling back to the roster map. Three lines of behaviour, in two files. ## Verification Four regression tests in `test-live-region-filter.js`. **Three fail without the fix**, checked by reverting `live.js` and re-running: ``` ❌ #1898: dbPacketToLive carries observer_id through ❌ #1898: a replayed packet survives an active region filter ✅ #1898: dropping observer_id is what broke it (guards the regression) ❌ #1898: observer_iata is carried so the badge needs no roster lookup ``` The one that passes either way does so on purpose: it asserts a packet carrying **no** `observer_id` is still dropped, pinning the mechanism so a future change cannot make the filter match everything. That test's sandbox needed `getParsedDecoded` and `getParsedPath`. `live.js:14` captures those from `packet-helpers.js` at load time and the sandbox does not load it, so they are stubbed in the sandbox definition rather than assigned afterwards. Assigning later is too late for that capture, which cost me two attempts. Other suites unaffected: `test-live.js` 95 passed, `test-packet-filter.js` 99, `test-frontend-helpers.js` 656. `test-1110-live-filter.js` fails identically on unmodified master with `ERR_CONNECTION_REFUSED`; it is an E2E test needing a server on port 13581. ## Note Both issues were filed separately and neither names the other. They are the same root cause in sibling code paths, which is why they are fixed together rather than in two PRs. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-04 15:03:08 +02:00
efitenandClaude Opus 5	cb6d230573	docs: renumber the release to v3.10.1 (#1953) ... v3.10.0 was tagged and then withdrawn. **Nothing was ever available under that number**: no container image and no release asset was ever published, so no user could have pulled it. This renames the notes and the CHANGELOG section. No product code changes. ## Why it had to be renumbered Three things, in the order they bit. **1. The image never built.** `release-fast-path.yml` re-tags `:edge` to `:vX.Y.Z` when the `:edge` revision label matches the tagged commit, and dispatches `deploy.yml` when it does not. The tagged commit was documentation-only, so the `paths-ignore` from #1949 meant no `:edge` existed for it and the fallback ran. That part behaved correctly. The fallback then published nothing, because every GHCR step was gated on `github.event_name == 'push'` and a dispatch is not a push. It built locally, reported `success`, and pushed nothing. Fixed in #1951, but that fix is not in the `v3.10.0` tag, and a `workflow_dispatch` runs the workflow file **from the ref it targets**. So the existing tag could not be made to publish. **2. The assets never uploaded.** I created the GitHub release by hand before the workflow reached it, and `action-gh-release` cannot update an immutable release. The correct procedure is to push the tag and let the workflow create the release. **3. The tag name cannot be reused.** GitHub's immutable releases keep a tag name reserved even after the release is deleted: ``` remote: - Cannot create ref due to creations being restricted. ``` I established that only after deleting the release, which is the wrong order. The lesson, written into the commit message so it survives: check whether a tag can be rewritten before removing anything that depends on it. ## What is in v3.10.1 The same 111 commits, plus the three CI fixes that landed after the v3.10.0 tag (#1949, #1950, #1951). Those are listed in their own section in the notes. **No product code differs** from what was tagged as v3.10.0. All 69 SHA references in the notes were re-verified after the rename. ## Procedure for this tag Push the tag and stop. The workflow creates the release and attaches the assets. Do not create it by hand. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> v3.10.1	2026-09-04 09:46:06 +02:00
efitenandClaude Opus 5	a3ee37011a	ci: scope docs-only skipping to jobs, so required checks still report (#1955) ... Same change as #1954, opened from a branch on this repository instead of from a fork. #1954 never received a workflow run: zero runs and zero check suites for its head commit, and closing and reopening it changed nothing. A manual `workflow_dispatch` on master ran immediately, so Actions itself is working; the `pull_request` event from the fork is what produces nothing. See the note at the end. Replaces the trigger-level `paths-ignore` from #1949 and #1950. That approach was wrong, and it is currently blocking #1953 from merging. ## What was wrong GitHub documents the distinction I had backwards: > a workflow skipped by path filtering keeps its checks **pending** and blocks the merge, while a **job** skipped by an `if:` conditional reports **Success** and does not. So the filtering has to live on the jobs, not on the trigger. I compounded it by claiming, in both the commit and the description of #1949, that master had no required checks: *"verified: the branch protection endpoint returns 404"*. **That verification was invalid.** A 404 there means the token cannot read protection details, not that none exist. The branch reports `protected=true`, and #1953 was refused with `the base branch policy prohibits the merge`. ## What this does A `🔎 Change scope` job computes whether anything outside `docs/`, `*.md` and `LICENSE` changed. `go-test`, `e2e-test`, `build-and-publish` and `release-artifacts` are gated on its output. A documentation-only pull request skips those jobs, they report Success, and the PR can merge. **Only pull requests are scoped.** A push or a dispatch always runs the full pipeline. That second part is deliberate and it fixes a separate failure. `release-fast-path.yml` re-tags `:edge` to `:vX.Y.Z` only when the `:edge` revision label matches the tagged commit. A master commit with no image breaks tagging, which is what happened to the v3.10.0 tag: the tagged commit was documentation-only, the fast path could not re-tag, it fell back to a dispatch, and the dispatch published nothing (#1951). Master pushes now always produce an image. The cost is running the pipeline on documentation commits to master; pull requests are where the queue pressure was. Two conservative defaults in the scope check: a non-`pull_request` event and an empty diff both count as code, so an unexpected shape runs everything rather than silently skipping. ## Note for whoever has repository settings access Fork pull requests stopped getting workflow runs between 22:36 and 05:40. #1949, #1950 and #1951 all came from the same fork and each got a run; #1954 got none, with no check suite created at all, which is different from a skipped run. `repos/.../actions/permissions` returns 403 for a non-admin token, so this could not be confirmed from the API. If the "Fork pull request workflows from outside collaborators" setting was tightened, that would explain it, and it would affect every outside contributor, not just this branch. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-04 09:44:18 +02:00
efitenandClaude Opus 5	bb8634312d	ci: publish images on a tag ref, not only on a push event (#1951) ... **v3.10.0 produced no container image.** The build job reported `success` and pushed nothing. ## What happened `release-fast-path.yml` re-tags `:edge` to `:vX.Y.Z` when the `:edge` revision label matches the tagged commit, and dispatches `deploy.yml` when it does not. For v3.10.0 the tagged commit was documentation-only. The `paths-ignore` added in #1949 means documentation-only commits skip `deploy.yml`, so no `:edge` image was ever built for that commit, the labels did not match, and the fallback ran. **That part worked exactly as designed** and correctly refused to re-tag an image built from a different commit. Then `deploy.yml` skipped all five GHCR steps, because each was gated on `github.event_name == 'push'` and a `workflow_dispatch` is not a push: ``` 4. Build Go Docker image (local staging): success 5. Set up Docker Buildx: skipped 7. Log in to GHCR: skipped 9. Build and push to GHCR: skipped ``` **So the fallback has never been able to publish an image.** It dispatches a pipeline that cannot push. That stayed invisible for as long as the fast path kept succeeding, which it did until a release note happened to be the last commit before the tag. ## Fix Gate those five steps on a push **or** a tag ref: ```yaml if: ${{ github.event_name == 'push' || startsWith(github.ref, 'refs/tags/v') }} ``` A dispatch aimed at a tag now publishes. A dispatch aimed at a branch still does not, so this does not turn every manual run into a release. ## The version stamp needed no change Verified rather than assumed. `Compute build metadata` keys on `GITHUB_REF`, not on the event: ```bash if [[ "$GITHUB_REF" == refs/tags/v* ]]; then APP_VERSION="${GITHUB_REF#refs/tags/}"; else APP_VERSION="edge"; fi ``` The failed v3.10.0 run already logged `Build: version=v3.10.0 commit=5bad23b`. Only the publishing was missing. ## Not covered here `Release Artifacts` failed on the same run for an unrelated reason: the GitHub release had been created by hand before the workflow reached it, and `action-gh-release` cannot update an immutable release. That one is process, not code. Push the tag and let the workflow create the release. Once this merges, re-dispatching `deploy.yml` against `v3.10.0` publishes the images for the existing tag. No re-tagging needed. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-03 23:05:16 +02:00
efitenandClaude Opus 5	5bad23b46a	docs: release notes for v3.10.0 (#1948) ... Release notes for the first tag since `v3.9.2` on 2026-06-13. **111 commits**, and no auto-generated coverage bumps fall in this range, so all 111 are substantive. Nothing here changes behaviour. It is `docs/release-notes/v3.10.0.md` plus a `CHANGELOG.md` section. ## Verification The header promises that every bullet ends with a SHA you can `git show`. That is checked mechanically rather than trusted: all **69** references were confirmed to point at a commit that exists and whose subject line contains the issue or PR number cited beside it. Zero mismatches. ## Two things operators need, and both are silent failures The urgency line leads with the first one on purpose. 1. **CARTO requires an API key** on its raster basemaps since 2026-08. Without one every tile is served watermarked with HTTP 200. Nothing errors, no healthcheck fires, and the only way to notice is to look at a tile. Anyone upgrading needs to set `map.tiles.providers.carto.key`. 2. **`pathTrust.minHashBytesForMapping` ships at 1**, which is the existing behaviour, so an upgrade changes nothing on its own. The note states what raising it to 2 would actually cost, with numbers from a live instance (56% of path-hop observations are 1-byte, 41% of repeaters use a 1-byte hash), because there is no UI to undo it. The relay `last_seen` fix is quantified the same way rather than described as "improved": for repeaters that relayed within the last hour, the gap between `last_relayed` and `last_seen` drops from a median of 12,062 s to 193 s, and the share more than five minutes behind falls from 96% to 39%. ## A theme worth naming Three of the highlights are the same defect in three places: something is operable before its own setup has finished. The Live view toggles are inert for about 100 ms after paint, the colour picker's deferred focus undid arrow-key navigation so Enter assigned the wrong colour, and an analytics theme-refresh discarded the filter you had just applied. All three were first written off as flaky tests, twice by me. Each is now fixed with a regression test that fails on the previous commit. ## Sequencing This should land, and the `v3.10.0` tag be cut, **before** the Go 1.27 upgrade in #1946. A toolchain bump changes the compiler, the runtime and `gofmt` for everything at once; landing it on top of 111 unpublished commits means a later regression cannot be separated from the toolchain. #1946 itself says no 1.27-only features are being adopted, so there is no cost to waiting one release. A tag first also gives a known-good bisect point. ## Not done The CHANGELOG has no `3.9.2` section and did not have one before this change. I left that gap alone rather than reconstructing it retroactively. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-03 21:27:32 +02:00
efitenandClaude Opus 5	6ccef5053b	ci: match root-level markdown in the docs paths-ignore (#1950) ... Follow-up to #1949, correcting a pattern I did not check before proposing it. #1949 used `'**/*.md'`. That reads as requiring a directory component, so it covers `docs/release-notes/v3.10.0.md` but not `CHANGELOG.md` or `README.md` at the repository root. Since `paths-ignore` skips only when **every** changed file matches, one uncovered root file is enough to run the whole pipeline anyway. GitHub's own example for "any file with this extension" is `'**.js'`, with no slash. `'**/*.md'` does not appear in their documentation at all. `'**.md'` covers root and subdirectories both. ## What this does not establish #1948 (`CHANGELOG.md` plus a release note) did start a full run after #1949 landed, and that is what prompted this. But there is a second candidate explanation I did not rule out: that PR's branch predates #1949, so its workflow file may simply not have carried the filter yet. I am not claiming to have proven which one it was. The fix is correct either way, and shipping the documented pattern is better than defending the first cause I noticed. The real test is the next docs-only PR opened from a branch that already contains the filter. The reasoning is in a comment above the `on:` block, not only in this description. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-03 21:27:00 +02:00
efitenandClaude Opus 5	55aabaa325	ci: skip the pipeline for documentation-only changes (#1949) ... Two documentation-only PRs were running the full pipeline simultaneously this afternoon: #1948 (`CHANGELOG.md` plus a release note) and #1947 (deleting a stale `docs/DEPLOYMENT.md`). Each spends about 12 minutes on Go Build & Test and about 16 minutes on Playwright to establish that a text file does not break a browser. **The cost is the queue, not the minutes.** On the same afternoon a `pull_request` run was created at 12:13 and its first job did not start until 16:19. Four hours in the queue. Every unnecessary run pushes the ones that matter further back, and this repository has been merging heavily today. ## Checked before adding the filter Rather than assumed: - **Nothing reads markdown at build or test time.** Grepping every Go and JS source for a runtime read (`ReadFile`, `readFileSync`, `os.Open`) of a `.md` path returns nothing. The `docs/` matches in `cmd/` and `test-*.js` are all comments pointing at documentation. - `/api/docs` serves Swagger UI generated from `cmd/server/openapi.go`, not from `docs/`. - `docs/` holds markdown plus screenshots (`png`, `gif`) and no build input. - **This workflow has no tag trigger**, so release tagging is unaffected; that runs from `release-fast-path.yml`. Worth stating explicitly given a `v3.10.0` tag is imminent. `paths-ignore` skips only when **every** changed file matches, so a PR touching both code and documentation still runs the full pipeline. ## The trap, stated in the file If required status checks are ever enabled on master, a skipped workflow never reports, and a docs-only PR would wait forever on a check that cannot arrive. At that point this needs to become a change-detection job with conditional heavy jobs rather than a trigger filter. Master has no required checks today. Verified: the branch protection endpoint returns 404. That caveat is in a comment above the `on:` block, not just in this description, because the person who enables required checks in six months will be reading the workflow and not this PR. ## Note This PR itself changes only `.github/workflows/deploy.yml`, so it is not documentation-only and will run the full pipeline, as it should. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-03 19:16:43 +02:00
Sylvain Rabot	b5dfac85dd	fix(docs): remove stale docs/DEPLOYMENT.md duplicate (#1947) ... ## Summary - `docs/DEPLOYMENT.md` and `docs/deployment.md` were both tracked in git, colliding into a single file on case-insensitive filesystems (default on macOS/Windows) and causing `git status` to report spurious modifications. - `docs/deployment.md` is the actively maintained guide (linked from `README.md` and `docs/deployment-behind-cdn.md`); `docs/DEPLOYMENT.md` was a stale duplicate untouched since the MeshCore → CoreScope rename. - Removed `docs/DEPLOYMENT.md` from the index, keeping `docs/deployment.md`. ## Test plan - [x] `git status` is clean on a case-insensitive checkout with no spurious modification - [x] Confirmed no remaining references to `docs/DEPLOYMENT.md` in the repo	2026-09-03 19:15:55 +02:00
3 people	40f664c587	chore(#1859): gofmt sweep + gofmt/go vet CI gate (rebase of #1881) (#1941) ... Rebase of #1881 by @SaarMesh-Bot onto current master. Their three commits are preserved, two of them cherry-picked with authorship intact; the sweep itself had to be regenerated. Opened as a new PR rather than force-pushing their branch. Closes #1881 once merged. Addresses parts 1 and 3 of #1859; part 2 landed as #1937. ## Why regenerated rather than merged The sweep in #1881 was cut on 2026-09-02 07:13 and roughly forty PRs landed after it, so it went `CONFLICTING/DIRTY`. Re-running `gofmt` on current master is cheaper and less error-prone than resolving 72 conflicts that are all whitespace. The drift it fixes also grew in the meantime: 66 files now, against 72 then, but spread differently. ## The three commits 1. **`style(#1859)`** — `gofmt -w` across the 14 modules. 66 files. 2. **`test(#1859)`** — @SaarMesh-Bot's fix for the one `go vet` copylocks finding, `cmd/ingestor/coverage_boost_test.go`: the range variable copied a `Config` embedding `sync.Once`. Cherry-picked unchanged. 3. **`ci(#1859)`** — @SaarMesh-Bot's CI step that fails on gofmt drift or vet findings, plus `.git-blame-ignore-revs`. Cherry-picked with one change, noted in the commit message: the ignore file pointed at `04bc80ee`, the sweep commit on their branch, which does not exist on this base and would make `git blame --ignore-revs-file` error. Repointed at `d3a02599`, the sweep here. ## Verification The claim "formatting only" is checked twice rather than asserted: - Every changed file is byte-identical to `gofmt(previous content)`. 0 of 66 deviate. - With line comments and all whitespace stripped, 0 of 66 files differ, so no code outside comments changed. 14 of the 66 also show doc-comment reflow. Since Go 1.19 `gofmt` re-indents indented comment blocks to tabs and inserts a blank comment line before them; the behavior matrix above `resolveHopWithContext` in `cmd/ingestor/path_resolver.go` is a clear example. That is gofmt's own output, not an edit, but it is worth naming because it makes the diff look larger than "whitespace" suggests. The gate was run locally exactly as the workflow runs it: `gofmt` clean, and `go vet` clean in all 14 modules, including `cmd/ingestor` which is what commit 2 fixes. Suites: `cmd/server` ok (80.7s), `internal/packetpath` ok (2.3s), `cmd/ingestor` passes except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails identically on bare master with "A required privilege is not held by the client" (Windows symlink privilege on my host, not code). ## Sequencing This should go last in the queue. The sweep touches 66 files, so merging it before the remaining open Go PRs gives each of them a conflict about nothing but formatting. After it lands the gate is active, and any PR with drift fails CI until it runs `gofmt -w`. Excluded from the sweep: the misnamed `Dockerfile.go`, which is a Dockerfile that gofmt cannot parse (the workflow excludes it too), and `docs/DEPLOYMENT.md`, which a case-insensitive filesystem surfaces as a spurious modification against `docs/deployment.md` and is unrelated. --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-03 18:52:03 +02:00
TeTeHacko	f167d6f338	test(#1923 follow-up): pin the packets window in the munger slide-over step (#1942) ... Follow-up to #1923/#1924 — one of the row-dependent packets navigations the pin sweep did not reach. ## The gap `test-slideover-1168-munger-e2e.js` navigates to bare `#/packets` and waits for `#pktTable tbody tr[data-action]` with an 8 s budget, on the default client-side window (`since = now − 15 min`). It is one of the row-dependent packets navigations #1924 did not reach. Most are already immune: #1924 pinned `?timeWindow=1440` in `test-slideover-1056-e2e.js`, `test-e2e-playwright.js` sets `meshcore-time-window=525600` in `gotoPackets()` and in the #1791 Group-Data step, and `test-issue-1122`/`1128` widen the window through the UI dropdown. Three other row-dependent packets navigations in the same job share the exposure and are **not** in this PR, to keep it to one file: - `test-gestures-1062-e2e.js` and `test-touch-gestures-coverage-e2e.js` also assert on the URL after in-app navigation, so a bare `?timeWindow=` query leaks into those assertions (`#/packets?hash=…` becomes `#/packets?timeWindow=1440&hash=…`); they want the localStorage window path. - the mobile branch of `test-observer-iata-1188-e2e.js` pins via localStorage, which `packets.js:736` clamps back to 15 min above 180 on a mobile viewport; switching it to the URL-param idiom this PR uses (which is applied after that clamp) would fix it, but it is a separate file. I can send those separately. (This step itself runs at an 800px viewport — `isMobile`, since the breakpoint is `innerWidth <= 1024` — and the pin still works precisely because the URL param is read after the mobile clamp, at `packets.js:1091-1094`, not from the clamped localStorage value.) ## Why it matters now, with numbers On two of the four master runs of 2026-09-02 this step executed at **freshen+8:06** (run 33678488159) and **freshen+10:13** (run 33684614144) — margins of 6:54 and 4:47 before the fixture's newest rows age out of the window. Every test added ahead of it shrinks that. The suite as a whole is closer still: in run 33684614144 the third repetition of the #1616 flake-gate ran **21:48:21→21:48:45 = freshen+14:45→+15:09**, i.e. already past the 15-minute mark — it survives only because of the #1924 pin. ## Verification Reproduced without waiting for the clock: shift the freshened fixture 20 minutes back (`first_seen` and `observations.timestamp`) and start the server on it — | | aged fixture | fresh fixture | |---|---|---| | step without the pin (master) | **fails** (selector timeout) | passes | | step with the pin (this PR) | passes | passes 3/3 | Same idiom and value as #1924, same caveat baked into the comment: the value must be > 0, because `packets.js` only reads the param under `_urlTimeWindow > 0`, so `timeWindow=0` silently keeps the default. One observation from the same investigation, offered separately from this PR: `tools/freshen-fixture.sh` computes its shift as `now − MAX(first_seen)`; if the max ever sits in the future, the offset goes negative and `printf('+%d seconds')` produces the invalid `'+-N seconds'`. On the NOT NULL `transmissions.first_seen` that aborts the script (set -e); on the nullable columns (`nodes.last_seen`, observers, neighbor_edges) `strftime` silently returns NULL. A one-line clamp to ≥0 would make it safe to run around an insert. Happy to send that separately if wanted.	2026-09-03 18:21:47 +02:00
TeTeHacko	ab62e86d2d	fix(live): wire the view toggles before init() awaits — they are inert for ~100 ms (#1940) ... Follow-up to the #1939 discussion, where @efiten asked for this PR. The multibyte E2E assertion that has been failing intermittently on master is a symptom of this; with this change the unmodified test passes reliably (3/3 idle, 8/8 under a 24-core load run that previously failed it 2 in 6). ## The defect `init()` writes the whole controls panel with `app.innerHTML` and only restores toggle state and attaches the `change` listeners ~330 lines later, behind two awaits (line numbers on master, as verified in the #1939 thread): | line | | |---|---| | 1104 | `app.innerHTML = …` — the checkboxes are in the DOM, clickable | | 1256 | `await (await fetch('/api/config/map')).json()` | | 1543 | `await loadNodes()` | | 1612–1614 | `.checked = <pref>` and `addEventListener('change', …)` | **A click inside that window is silently lost.** Measured on master, localhost, clicking `#liveMultibyteToggle` on the first animation frame in which it exists: | run | click at | immediately after | after 2.5 s | |---|---|---|---| | 1 | 481 ms | `checked=true`, `localStorage=null` | `checked=false`, `localStorage=null` | | 2 | 505 ms | `checked=true`, `localStorage=null` | `checked=false`, `localStorage=null` | | 3 | 438 ms | `checked=true`, `localStorage=null` | `checked=false`, `localStorage=null` | No handler runs, nothing reaches localStorage, and the later `.checked = <pref>` reverts the click with no feedback. Separately, the restored state itself appears only **93–112 ms (3–5 rendered frames)** after the control is painted — `ghost` and `colorHash` default ON, so they visibly flick on for every visitor. ## The fix - **`wireLiveControls()`** — synchronous, right after `app.innerHTML`: restores `.checked` and attaches listeners for the eight persisted toggles, as one table instead of eight near-identical blocks. The matrix↔heat interlock applies from the first paint too. - **`applyLiveControlEffects()`** — after the awaits: applies the effects that need state built there (matrix theme, rain canvas). - **`syncHeatToggleToMatrix()`** — the interlock, extracted; it previously existed as two identical copies. - Heat gets a module-level mirror (`heatEnabled`) like the other seven toggles, so the layer is only built when wanted. Previously it was built unconditionally and torn down ~270 lines later — invisible (no await in between, so no frame composited; the cost is only ~9 ms at 1000 nodes), but any throw between the two calls left the layer visible against the stored preference. `showHeatMap()` now guards on the map existing instead of relying on `nodeData` being empty at that moment. ## Verification - Click on the first painted frame now persists, 3/3 (`localStorage` written, survives). - Restored state present on the first painted frame: 0 unchecked frames in 5 runs (was 3–5). - All four heat×matrix load combinations render identically to master (layer present/absent, checked, disabled). - `test-live-multibyte-only-e2e.js` unmodified: 3/3, plus 8/8 under CPU load. - With stored matrix ON, the heat toggle is `checked=false, disabled=true` from the first frame. ## The probe (as requested) <details><summary>~30-line Playwright harness that demonstrates the inert control</summary> ```js const { chromium } = require('playwright'); (async () => { const b = await chromium.launch(); for (let run = 0; run < 3; run++) { const ctx = await b.newContext({ viewport: { width: 1400, height: 900 } }); const p = await ctx.newPage(); await p.addInitScript(() => { window.__r = { clickedAt: null, afterClick: null, lsAfterClick: null, final: null, lsFinal: null }; const tick = () => { const el = document.getElementById('liveMultibyteToggle'); if (el && window.__r.clickedAt === null) { window.__r.clickedAt = performance.now(); el.click(); window.__r.afterClick = el.checked; window.__r.lsAfterClick = localStorage.getItem('live-multibyte-only'); return; } requestAnimationFrame(tick); }; requestAnimationFrame(tick); }); await p.goto('http://localhost:13581/#/live', { waitUntil: 'domcontentloaded' }); await p.waitForTimeout(2500); const r = await p.evaluate(() => { const el = document.getElementById('liveMultibyteToggle'); window.__r.final = el ? el.checked : null; window.__r.lsFinal = localStorage.getItem('live-multibyte-only'); return window.__r; }); console.log(`run ${run+1}: click at ${Math.round(r.clickedAt)}ms -> checked=${r.afterClick}, ls=${r.lsAfterClick}` + ` || after 2.5s: checked=${r.final}, ls=${r.lsFinal}`); await ctx.close(); } await b.close(); })(); ``` </details> ## Deliberately out of scope (each verified, none regressed here) - `#liveAudioToggle` has the same window (MeshAudio persists `live-audio-enabled`), but its restore runs through `MeshAudio.restore()` and a slider panel — its own change. - `#liveGeoFilterToggle` stays hidden until its own config fetch, so its window is not user-reachable; the fullscreen control is created by Leaflet after the map exists. - Pre-existing: `clearNodeMarkers()` (VCR resume path) drops the heat layer and nothing rebuilds it. `heatEnabled` is the right gate for fixing that, but it is a separate behaviour change. Happy to also submit the deterministic version of the multibyte test (it forces the window open by delaying `/api/config/map`, so it fails on this bug 3/3 instead of intermittently) as a follow-up if wanted.	2026-09-03 18:19:54 +02:00
efitenandClaude Opus 5	5d2e14aba2	fix(#1943): cancel the deferred swatch focus so arrow keys are not undone (#1945) ... Closes #1943. The colour picker's keyboard navigation is broken, and the E2E flake that has been failing unrelated PRs (#1940, #1941, and master pushes `589fa987` and `859173f1`) was reporting it correctly. ## Cause `showPopover` deferred focusing the first swatch with an uncancellable `setTimeout(..., 0)` at `channel-color-picker.js:146`, and nothing cleared it on hide. The file contained **zero** `clearTimeout` calls. Reopen the popover while a swatch still holds focus and that timer lands after the user has already pressed an arrow key, pulling focus back to the first swatch. Proven, not argued. Instrumenting `HTMLElement.prototype.focus` with a stack trace, on one open: ``` focus(#f97316) @10205ms <- the keydown handler focus(#ef4444) @10208ms <- channel-color-picker.js:146:58 ``` Three milliseconds apart. ## The user-visible bug Worse than a flaky test. **Open the picker, arrow to a colour, press Enter, and the first colour is assigned instead of the one you chose.** Holding the timing still made the existing suite say so directly: ``` ✗ Enter should assign focused color (#f97316), got #ef4444 ``` ## Why the test looked flaky The revert happens on **every** open. Only whether the assertion reads before or after it varies, which is why an idle machine passes and a loaded runner does not. #1939 (mine) assumed the opposite: a race in which the handler had not yet moved focus, cured by waiting for it. #1943 has the measurement that disproves it. The failing step took **16 ms** while that wait has a **3 second** budget, so the wait was resolving successfully and then the value was reverted underneath it. It never helped. Its comment is corrected in this PR rather than left to mislead the next reader. ## Fix Keep a handle for the timer, cancel a pending one on both show and hide, and inside it do nothing when the popover has since been hidden or when focus already sits inside it. A fresh open still focuses the first swatch, which is what the accessibility behaviour is for. An open that inherits focus, or a user who has already navigated, is left alone. ## Verification - The **regression test added here fails on unmodified master** with `a late focus timer must not move focus after the user did` and passes with the fix. - It is **deterministic, not load-dependent**: it reproduces the exact sequence the stack trace identified (open, Escape, reopen, ArrowRight before the timer lands) rather than waiting for contention. It also asserts the Enter path, so the user-visible half is covered and not just focus position. - Full suite: 10 of 10, three consecutive runs. ## Note on the other flake This is one of two E2E failures blocking the queue. The other, #1925, is a different mechanism in a different file and is fixed separately in #1944. Together they should leave the E2E suite deterministic again. Same shape as @TeTeHacko's finding in #1940: something is operable before its setup has finished. That is now three instances in this codebase, so it may be worth a look as a pattern rather than three separate fixes. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-03 09:42:10 +02:00
efitenandClaude Opus 5	e2df9bbd3e	fix(#1925): stop theme-refresh from discarding the neighbor-graph filter (#1944) ... Closes #1925. This is the flake that failed #1942 and #1871, neither of which touches the feature. It is not a test problem. ## Cause Every page load fires exactly one delayed, full re-render of the active analytics tab: 1. `app.js` starts `/api/config/theme` without gating navigation on it, deliberately. 2. When it resolves, `_customizerV2.init()` runs `applyCSS()`, which dispatches `theme-changed`. 3. `app.js:1188` debounces that by 300 ms and dispatches `theme-refresh`. 4. `analytics.js:230` answered it with `renderTab(_currentTab)`. For the neighbor-graph tab step 4 is destructive. `renderTab` replaces `el.innerHTML`, so the role checkboxes are recreated with their defaults and companion is silently re-checked, and `_ngState` is rebuilt from the full 1400-node graph. The count is back over the 1000 limit, so `#ngSkipMsg` returns and the canvas is hidden. The re-entrancy epoch guard cannot prevent it. That guard stops a superseded `tick()` loop; this is a legitimate new top-level render pass that resets the very inputs the guard protects downstream of. **One mechanism produces both documented failure modes**, decided only by where that single re-render lands: | lands | result | |---|---| | before the first uncheck | harmless, test passes | | between an uncheck and the next `waitForFunction` poll | mode 1, the 15 s timeout | | after `waitForFunction` succeeded, before the follow-up `evaluate` | mode 2, "expected #ngSkipMsg gone again" (#1942) | Measured on an idle machine: the test's final evaluate at 542 ms, `theme-refresh` at 636 ms, `#ngSkipMsg` re-added at 667 ms. It passes locally by about 90 ms. On a loaded runner the test's Playwright round trips stretch while `theme-refresh` still lands at theme-fetch latency plus 300 ms, so it arrives mid-test. ## Fix On `theme-refresh`, restart the renderer instead of rebuilding the tab when the neighbor-graph tab is active and has state. Four lines. This stays theme-correct: node colors are read live per frame from `window.ROLE_COLORS`, role swatches use `.role-swatch--{role}` CSS tokens, stats and the skip message use CSS variables, and the one cached theme value, `_labelColor = cssVar('--text-primary')`, is re-read on restart at `analytics.js:3375`, inside `startGraphRenderer`. When `_ngState` is null it falls through to the old path. ## Verification Measured, not asserted: - **Deterministic reproduction** (hold `/api/config/theme` until just before the second filter-down, then stall 500 ms before the final evaluate; no synthetic events dispatched): **2 of 2 fail** on unmodified master with the exact #1942 message, **3 of 3 pass** with this fix. - The **unmodified** E2E test passes against the fixed build. - With the tab open and a filter applied, a `theme-refresh` leaves the filter intact, the canvas present, and produces no page errors. - The **regression test added here fails on unmodified master** with `theme-refresh reset the role filter (companion re-checked)` and passes with the fix. It dispatches the event directly, so it tests the cause instead of waiting for the race to appear. ## What this does not cover The same startup re-render silently discards user interaction in the first second or so on **any** analytics tab, not just this one. A user who clicks quickly after load loses that click. This change covers the neighbor-graph tab, because that is what #1925 is about and what is failing CI. The general fix, for example skipping the startup refresh when the effective config changed nothing, deserves its own issue rather than being smuggled in here. Side observation while tracing: `theme-changed` fires twice at startup, at about 311 ms and 323 ms. The debounce collapses them, so it is harmless, but it means the customizer pipeline runs twice. I did not identify the second dispatcher. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-03 09:41:36 +02:00
efitenandClaude Opus 5	9ae3387416	feat(ingestor): RF environment samples from mobile clients (#1906) ... > **Stacked on #1905.** This branch contains #1905's commits; review and merge that one first. The diff unique to this PR is the `client_rf_samples` table, its handler, the delta query and its retention. ## What Everything CoreDrive RX records today is anchored to a *packet*. But a drive also passes through RF conditions that exist whether or not a packet arrives: the noise floor, how busy the channel is, how many receptions fail CRC. The radio measures all three and was never asked. This samples the companion's own counters along the GPS track and stores them, so the server can render a noise-floor map, a channel-utilisation map and a CRC-error-rate map. A fixed observer cannot produce those — it measures one point forever. **Zero airtime:** `CMD_GET_STATS` is a local Bluetooth query to the attached radio. Nothing is transmitted. ## Design points worth knowing - **Absolutes are stored; deltas are derived at query time.** A lost or reordered sample then costs one interval rather than corrupting a running total. `ClientRfDeltas` breaks the chain whenever `uptime_secs` fails to increase — that is the reboot and counter-wrap detector. - **Absent is not zero, end to end.** Firmware predating the `recv_errors` field cannot count CRC errors at all, and a stored `0` would read downstream as "a perfectly clean channel" — the opposite of "we don't know". Presence/absence is preserved through the app parser, the wire payload, a nullable column, and the delta view, which returns `nil` rather than `0` when either endpoint is unknown. Each of those five layers has its own test. - **`sampled_at` is millisecond precision, and it is load-bearing.** SQLite compares these strings lexicographically and `.` (0x2E) sorts before `Z` (0x5A), so a second-resolution retention cutoff would delete rows *inside* the window. The prune formats its cutoff with the same layout. ## Performance justification (touches the ingest hot path) - One INSERT per sample, gated behind an opt-in flag that defaults off. Sample rate is 15 s while moving and 5 min while parked, so roughly 240 rows per hour per active driver. - The delta query is a single `LAG(...) OVER` pass with no nested query inside the loop, so it cannot deadlock the single writer connection. Window functions are already used elsewhere in this codebase. - Retention has its own key and index (`sampled_at`); without it the table would grow unbounded, so `config.example.json` documents it inline. ## Safety for existing deployments Opt-in and default off on both sides (`clientRfSamples.enabled`, and `rfSampler` in the app). The coverage path is untouched — `Publisher.buildPayload` is byte-identical and a record with no `kind` field still routes to `/packets` unchanged. The MQTT dispatch was reshaped so that **anything on `meshcore/client/…` returns from that branch in every config state**, with the enable-gates inside rather than in the topic match. Previously a disabled gate let the message fall through to the observer path, where `parts[1]` — the literal string `client` — was read as a region and the phone's pubkey registered as an observer. The blacklist check now also runs ahead of the sub-topic switch, so it covers every present and future client sub-topic. ## Testing Full ingestor suite green. Notable coverage: a `/rf` message with the gate off writes nothing anywhere and does not fall through; a sample missing `uptime_secs` is rejected rather than stored as an unusable row; two samples 40 ms apart remain two rows; and the retention test seeds a row with a non-zero millisecond component inside the cutoff second, which is the only row that distinguishes a correct cutoff from an RFC3339 one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 21:22:07 +00:00
Joel ClawandJoel Claw	ac6fbaf9f3	perf: reuse ctx buffer in resolvePathForObs, cache ReadMemStats per store (#1873) ... ## Problem Three hot-path inefficiencies causing excess CPU and memory allocations: ### 1. `filterTxSlice` starts with nil slice `filterTxSlice` is called on the full `s.packets` slice (50k+ packets) for every query that doesn't hit a fast-path index. Starting with `var result []*StoreTx` means Go's append does ~15 growth+copy cycles (1→2→4→8→...→32768→65536) before reaching steady state. ### 2. `resolvePathForObs` allocates per hop Each hop in the path resolution loop allocates a new `ctx` slice (`make([]string, len(contextPKs), len(contextPKs)+2)`). For a 5-hop path, that's 5 allocations per observation. With 500+ observations per ingest batch, that's 2500+ small allocations. ### 3. `estimatedMemoryMB` calls `runtime.ReadMemStats` without caching `runtime.ReadMemStats()` triggers a STW (stop-the-world) pause. It's called from stats/debug endpoints (`GetStoreStats`, `GetPerfStoreStats`) that may be polled frequently. The routes.go layer already caches this with a 5s TTL, but the store layer doesn't. ## Fix 1. **Pre-allocate `filterTxSlice`**: `make([]*StoreTx, 0, n/2)` — the 2x over-allocation is cheaper than repeated growth+copy. 2. **Reuse ctx buffer**: Allocate one `ctx` buffer before the hop loop, reset to base length each iteration with `ctx = ctx[:ctxLen]`. 3. **Cache `ReadMemStats`**: 5-second TTL cache matching the routes.go pattern. Uses a package-level mutex (not on `PacketStore`) to avoid adding a field. ## Testing - `go build` passes - No behavior change — same results, fewer allocations --------- Co-authored-by: Joel Claw <358739783+Joel-Claw@users.noreply.github.com>	2026-09-02 23:20:39 +02:00
efitenandClaude Opus 5	a8e8449170	test(#1616): wait for swatch focus to move instead of reading it immediately (#1939) ... This is the test that has been keeping master red on both sides of today's queue run. ## The failure ``` ✗ ArrowRight cycles focus across swatches: ArrowRight should move focus to next swatch (was #ef4444, now #ef4444) ``` Observed on: | where | when | |---|---| | master push `589fa987` | 2026-08-31 — the last completed master run before today | | master push `859173f1` | 2026-09-02 — the first completed master run after #1938 | | PR #1884 | 2026-09-02, passed unchanged on a re-run | **Two out of two completed master runs.** Master has produced exactly two finished pipelines since 2026-08-31 and this test failed both, which is why the branch has had no green badge either side of a day of merges. ## The cause ```js await page.keyboard.press('ArrowRight'); const nextColor = await page.evaluate(() => document.activeElement.getAttribute('data-color')); ``` It reads `document.activeElement` on the tick after the key press. The keydown handler moves focus, but under CI load that can land after the evaluate has already run, so the assertion compares the swatch against itself and reports the same colour twice. **This file already knows about this.** The "outside click" step below carries a long comment about exactly this macrotask race for #1317, and the conclusion there was to wait on the real condition instead of a proxy. That step got the treatment and this one did not. ## The fix Wait for `activeElement` to be a `.cc-swatch` whose `data-color` differs from the one focused before the key press, with a 3s budget. The wait is wrapped so that a timeout falls through to the original assertion, which then reports the value actually observed rather than a bare Playwright timeout — a failing test should still say what it saw. **No product code changed.** One file, +18 lines, all of it the wait and the reasoning. ## What this does not claim It does not prove the focus handler is correct, only that the test stops racing it. If ArrowRight is ever genuinely broken, this still fails, and now with a useful message. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 20:18:25 +00:00
efitenandClaude Opus 5	9e13e0b05f	feat(ingestor): full-packet RF observations from mobile clients (#1905) ... ## What A CoreDrive RX drive already carries far more RF information than reaches CoreScope, and it was being discarded twice: once in the mobile app (every packet it could not attribute to a directly-heard node was dropped before queueing) and once here (the ingestor decodes the *complete* packet, then keeps only `heard_key`/`snr`/`rssi`/`lat`/`lon`). This captures what was being thrown away, at **zero extra airtime** — nothing new is transmitted. - **`transmissions.code1` / `code2`** — the transport codes were decoded on every packet and used only to derive `scope_name`, then dropped. Storing them turns "which repeater forwards which scope" from a re-parse into a query. - **An async backfill** re-parses the `raw_hex` already on disk, so months of scope history become queryable with no new data collection. - **`client_rx_observations`** — a new diagnostic table holding every decodable packet a phone heard, with route type, transport codes, scope name, path-hash size, the full forwarder chain and the forwarder. ## Why it is safe for existing deployments Both halves are **opt-in and default off** (`clientRxObservations.enabled`, and `fullRfLog` on the app side), so an existing deployment sees no behaviour change and no volume change on upgrade. The coverage invariant is untouched: `client_receptions` keeps its rule — 0-hop advert pubkey or FLOOD `path[last]`, ≥2-byte hash — and an unattributable packet writes **zero** coverage rows. `deriveHeardKey`, `buildClientReception` and `InsertClientReception` are unmodified except for one guard described below. ## Performance justification (touches the ingest hot path) - **Backfill:** keyset-paginated by `id` in 5000-row batches, a single forward scan, `rows.Close()` before `Begin()` so it never deadlocks against `SetMaxOpenConns(1)`, and commits per batch so live ingest interleaves. Termination is driven by rows *scanned*, not rows decoded — an earlier count-based loop would have stopped at the first batch containing an undecodable row and then written its completion guard, permanently stranding the rest. - **Guard row is written if and only if the loop ran to genuine exhaustion.** Every error path leaves it unwritten so the next startup retries. - **Per-packet cost:** one extra INSERT on the client topic when enabled, gated behind an opt-in flag. No new work on the observer path. - **New indexes** cover the prune (`rx_at`), the flood-grouping (`pkt_hash, rx_at`), the per-repeater query (`forwarder, rx_at`) and the scope query (`scope_name, rx_at`). Retention has its own shorter window — this table is diagnostic, not archival. ## Two firmware-derived correctness points - **`pkt_hash` is `ComputeContentHash()`**, byte-identical to `transmissions.hash`, so dark-traffic queries are a plain equality join rather than a translation layer. - **TRACE packets are refused.** TRACE repurposes the header path bytes as per-hop SNR values, so deriving a `heard_key` from them invents a node that never existed. `packetpath.PathBytesAreHops` existed but was never wired into the client path; it became reachable only because the app half now publishes packets it previously dropped locally. ## Testing Full ingestor suite green. Notable coverage: a FLOOD-routed TRACE writes zero coverage rows and NULL `forwarder`; a `direction: "tx"` message writes no observation; a DIRECT route never sets `forwarder`; two forwarder copies of one flood remain two rows; the backfill's multi-batch path is exercised with an undecodable row in the first page; and a forced error asserts the migration guard stays unwritten. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 18:35:20 +02:00
efitenandClaude Opus 5	376c3e9f4a	fix(packets): surface the transport region scope — detail pane row and a sortable Scope column (#1894) ... ## Summary `transmissions.scope_name` (#899) reached the database but never reached the UI. Two problems, one dead feature and one missing surface. ## 1. The detail pane's Scope row was dead `public/packets.js:3279` has rendered a **Scope** row since #899, gated on `pkt.scope_name != null`. It never fires in practice. `/api/packets` and `/api/packets/{id}` are served from the in-memory `PacketStore`. The store reads `scope_name` out of SQLite fine (`store.go:888`, `chunked_load.go:551` → `StoreTx.ScopeName`), but `txToMap()` did not put it in the JSON. Only packets old enough to have been evicted from the store — and thus served by the SQLite fallback in `db.go`, which does emit it — could ever show a scope. Verified against a live instance before the fix: ``` GET /api/packets/552e9687f1525537 → packet keys: ['_parsedPath','decoded_json','direction','first_seen','hash','id', 'observation_count','observations','observer_iata','observer_id', 'observer_name','path_json','payload_type','raw_hex','route_type','rssi','snr','timestamp'] ``` No `scope_name`. ### The NULL / "" distinction `StoreTx.ScopeName` was typed `string`, which collapses the two states the frontend distinguishes: | DB value | Meaning | UI | |---|---|---| | `NULL` | not transport-scoped | row hidden | | `""` | transport-scoped, region matched no configured key | muted "unknown scope" | | `"#be"` | matched region | the region name | `route_type` is **not** a usable proxy for that distinction: the ingestor writes NULL for a transport route whose `transport_code_1` is `0000` (`cmd/ingestor/db.go:1576` — `IsTransportScoped = route_type IN (0,3) AND Code1 ≠ "0000"`). So the field is now `*string`, with `nullStrPtr` preserving what `nullStrVal` collapsed. The two internal consumers (`TransportedScopes` #1751, `relayEntry.scope`) only care about non-empty named scopes and are unchanged in behaviour. ## 2. New: a Scope column on the packets table The scope was only reachable one packet at a time by opening the detail pane. It now has its own sortable column between Type and Observer, visible by default. The default view is **Group by Hash**, served by mappers that did not carry `scope_name` at all — so the column would have been empty in exactly the view most people look at. Both grouped paths now select and emit it: `groupedTxsToPage` in the store, and the dedicated grouped query in the DB fallback (v3 and legacy shapes). Rendering lives in `scopeCellHtml` (`public/app.js`, next to `transportBadge`) and is used on all three row-render sites — group header, expanded children, flat rows — so the column and the detail pane cannot drift apart. **Sorting** pins the empties last in both directions, as the nodes table already does for `default_scope`. Only ~8% of packets carry a scope, so an ascending sort would otherwise bury every scoped row under a wall of dashes. **Filtering**: `packet-filter.js` gains a `scope` field, so the cell is click-to-filter like Type and Observer, and `scope == "#be"` works in the filter bar. **Column prefs**: a `packets-known-cols` companion key. The `packets-visible-cols` array alone cannot distinguish "this column did not exist when you saved" from "you unchecked it", so any new column arrives silently hidden for every returning visitor. Keys absent from `known-cols` get the default treatment; keys the visitor actually hid stay hidden — there is a test for that second half specifically. ## Tests Each watched fail first. **Go** (`cmd/server/packet_scope_name_test.go`) - `txToMap` unit tests for all three states, including a JSON round-trip so a typed nil `*string` cannot pass as `null` - end-to-end through `/api/packets/{hash}` - `groupedTxsToPage` unit + end-to-end through `/api/packets?groupByHash=true`, across **both** the store-backed and DB-fallback paths - `transported_scopes_1751_test.go`: the "no scope" guard now covers both non-values (nil and a pointer to `""`) **Frontend** - `test-frontend-helpers.js`: `scopeCellHtml` three states + escaping - `test-packet-filter.js`: `scope` matching, case-insensitivity, and `FIELDS` registration - `test-packets-scope-column.js` (new Playwright e2e): header position, default visibility, one cell per row, em dash on non-transport rows, empties-last sorting, the Columns toggle, and the prefs backfill ## Verification Deployed and checked against a live instance: ``` /api/packets?groupByHash=true&limit=500 → scope_name present on 500/500, 59 with a matched region, 1 unknown-scope test-packets-scope-column.js → 7 passed, 0 failed cd cmd/server && go test ./... → ok ``` Two pre-existing failures, unrelated and equally red on an unmodified checkout: `test-e2e-playwright.js` "Customizer open does not overwrite server home config" and `test-observer-iata-1188-e2e.js` (timeout on `[data-loaded="true"]`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 18:34:59 +02:00
3 people	c598abd210	chore(#1859): pin eslint@8 + the lock file it needs (continues #1880) (#1937) ... Continues #1880 by @SaarMesh-Bot. Their commit is unchanged and keeps their authorship; I added the lock file it was missing. ## What was wrong #1880 added `eslint@^8.57.1` to `devDependencies` but not to `package-lock.json`. CI runs `npm ci --production=false`, which requires the two to be in sync, so the pipeline died at **Install npm dependencies** before a single test ran: ``` npm error code EUSAGE npm error `npm ci` can only install packages when your package.json and npm error package-lock.json are in sync. npm error Missing: eslint@8.57.1 from lock file npm error Missing: @eslint-community/eslint-utils@4.10.1 from lock file ``` I approved that PR on 2026-08-30 on the grounds that it was two lines of devDependency with no runtime effect. I checked that `.eslintrc.json` exists so `npm run lint` would resolve, and did not check the lock file. That was the miss. ## What this adds One commit: the regenerated `package-lock.json`. Generated with `npm install --package-lock-only`, so `node_modules` was never touched and the change is confined to the lock file. The 13 removed lines are npm reorganising the existing `find-up` / `find-cache-dir` entries because eslint brings its own versions of them; nothing unrelated was bumped. ## Verification | command | result | |---|---| | `npm ci --production=false` | **added 394 packages**, exit 0 | | `npm run lint` | exit 0, **92 problems (0 errors, 92 warnings)** | The warnings are pre-existing unused-variable reports across `public/*.js`. Not addressed here: the point of this PR is that the command runs at all, and whether to act on 92 warnings is a decision for #1859 rather than something to smuggle in behind a lock file. ## Why this matters beyond itself #1881 is the other half of #1859 and adds the gofmt/go vet CI gate. It has been sitting at the end of the merge order all day because it forces a rebase on every open Go PR. It also should not land before this one: the `lint` script it complements is not installable until the lock file is right. @SaarMesh-Bot — your work, your credit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-09-02 18:34:55 +02:00
efitenandClaude Opus 5	859173f145	ci: unblock the master pipeline — gate the staging deploy, detach the badges job (#1938) ... Fixes the problem @sylr reported on #1922. ## What is broken **master has produced no completed pipeline result since 2026-08-31.** Thirty-plus runs today alone were cancelled or left queued, while `go-test`, `e2e-test` and `build-and-publish` were passing inside them. The `deploy` job runs on `[self-hosted, meshcore-runner-2]`, and no such runner has picked up a job since at least 2026-08-31. It has no `timeout-minutes`, so it sits queued indefinitely. That job holds its run open, the run holds the concurrency group `ci-refs/heads/master`, and GitHub then cancels every subsequent master push while one waits. Two runs showing it, one from yesterday and one from right now: ``` 8ce5291b (2026-09-01) 89544b1d (2026-09-02) ✅ Go Build & Test ok ✅ Go Build & Test ok 🎭 Playwright E2E ok 🎭 Playwright E2E ok 🏗️ Build & Publish ok 🏗️ Build & Publish ok 🚀 Deploy Staging cancel 🚀 Deploy Staging queued ← runner: none ``` The tests were fine the whole time. Only the deploy hangs, and it takes the branch's pipeline with it. ## What this changes **1. The deploy job is gated on a repository variable.** ```yaml if: | vars.ENABLE_STAGING_DEPLOY == 'true' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/master' timeout-minutes: 10 ``` Unset means it does not run. Setting `ENABLE_STAGING_DEPLOY` to `true` in Settings restores the old behaviour with **no code change**. The `timeout-minutes: 10` is there so that enabling it while the runner is still absent fails in ten minutes instead of blocking the branch again. A bare timeout would have unblocked the queue but left master permanently red on a job that cannot succeed while the runner is gone. Gating it means master goes green when the tests pass, which is what a branch pipeline is for. **2. The badges job stops depending on the deploy.** `needs: [deploy]` → `needs: [build-and-publish]`. That job downloads the `go-badges` and `e2e-badges` artifacts and commits them to `.badges/`. It never needed the deploy step, and that dependency is why the coverage badges also stopped updating. `build-and-publish` already requires both test jobs transitively, so ordering is unchanged. ## What this does not change Nothing about what the deploy job *does*. No test job is touched. One file, +26/-2. ## The part that is not mine to fix Why `meshcore-runner-2` is gone is the repository owner's to answer, and they have been unreachable since June, which is what #1922 is about. Whoever brings the runner back flips one variable and the deploy returns exactly as it was. Thanks @sylr — I had been working around this all day by verifying master locally after each batch, without understanding why master never completed. Your report is what made it legible. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 16:24:27 +00:00
3 people	25090230f2	fix(map): render the Esri labels overlay it was already named for (rebase of #1917) (#1935) ... Continues #1917 by @nullrouten0. The commit is theirs, authorship unchanged; I only rebased it onto master. It went CONFLICTING because #1891 (the OpenTopoMap and USGS layers) landed in the same `BASE_STYLES` block, and both PRs also add cases to `test-issue-1420-tile-providers.js`. Resolution: kept #1891's two `usgs-*` entries and took this PR's `esri-darkgray-labels` line, which is the one that adds `refUrl`. Both test suites kept in full. Nothing else touched. Verified: `test-issue-1420-tile-providers.js` 47 passed, 0 failed, which includes this PR's four Esri cases and the Carto key cases that landed since. My review stands: approve. The id `esri-darkgray-labels` promised labels the layer control never stacked, and the test asserting that single-layer providers stay bare tile layers is the part that makes this safe to merge. Co-authored-by: nullrouten <nullrouten@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 13:12:55 +00:00
efitenandJonathan Herlin	89544b1d08	perf: index, cache, and deflake /api/channels queries (rebase of #1887) (#1936) ... Continues #1887 by @Jonher937. The commit is theirs, authorship unchanged; I only rebased it onto master. It went CONFLICTING because #1934 (prepared statements, originally @Joel-Claw's #1878) landed in the same `DB` struct. Both PRs add fields there and this one also replaces the single-slot channels cache. Resolution: kept this PR's keyed caches (`channelsCache`, `encChannelsCache`, `msgCache` plus their entry types and TTL constants) and kept master's thirteen prepared-statement fields alongside them. The old single-slot `channelsCacheKey`/`channelsCacheRes`/`channelsCacheExp` trio is gone, which is the point of this PR. Nothing else touched. Verified: `cmd/server` builds and the **full suite passes**, not just the channel tests. My review stands: approve, with two questions that do not block and are worth a look at some point. 1. `msgCache` is keyed by `hash|limit|offset|region`, and `offset` grows without bound as someone pages through a channel. Each entry also holds a full page of message maps, so a full 256-entry cache at `limit=50` holds around 12,800 maps. The other two caches are keyed by region only and genuinely low-cardinality as your comment says; this one is the odd one out. 2. `getMsgCache` returns the cached slice directly, so every hit hands the caller the same message maps. If any handler mutates one before serialising, it corrupts the cache for the next ten seconds. Same class as the finding on #1871, which was fixed there by copying at the two broadcast sites. Co-authored-by: Jonathan Herlin <jonte@jherlin.se>	2026-09-02 13:10:43 +00:00
Jonathan Herlin	eb8f376c6c	fix: use index from_pubkey in nodes region filter (#1882) ... The region subquery in GetNodes was pulling the advert pubkey out of decoded_json with JSON_EXTRACT for every row the join touched, instead of reading the from_pubkey column that #1143 already added and indexed It looks like buildPacketWhere, GetRecentTransmissionsForNode, QueryMultiNodePackets etc. moved to from_pubkey already, but not this.	2026-09-02 15:03:41 +02:00
Joel ClawandJoel Claw	4a776454ca	perf: remove dead relayTimes field (#1931) ... The `relayTimes` field (`map[string][]int64`) on `PacketStore` is never written to and never read. Its only references are the declaration at `store.go:180` and the `make()` in `NewPacketStore` at `store.go:644`. `relay_liveness_test.go` looks like a user at a glance but builds its own local `idx := make(map[string][]int64)` and passes that to `addTxToRelayTimeIndex`; the string "relayTimes" there is only inside a `t.Error` message. This is the surviving fragment of #1872, which no longer compiles after #1855 removed `lastSeenTouched` and `touchRelayLastSeen` from master. Verified against current master: both references are gone, build passes, all tests pass (32.2s). Co-authored-by: Joel Claw <358739783+Joel-Claw@users.noreply.github.com>	2026-09-02 14:53:16 +02:00
Jonathan Herlin	a46a6d55eb	fix: use replaceState for traces/roles redirects to preserve back-button history (#1883) ... fix: use replaceState for traces/roles redirects to preserve back-button history The #/traces/<hash> and #/roles backward-compat redirects used location.hash = ..., which pushes a new history entry instead of replacing the current one. This trapped users navigating back from a trace view: the intermediate #/traces/<hash> entry would immediately re-redirect forward again on hashchange, so back button never reached the packets view they came from.	2026-09-02 14:51:30 +02:00
efitenandClaude Opus 5	f081f91b88	fix(#1904): keep resolved full-pubkey hops across a path-hop index rebuild (#1907) ... Fixes #1904. ## The bug `buildPathHopIndex` reassigned `s.byPathHop` to a fresh map and refilled it from every packet's raw `path_json` hops: ```go func (s *PacketStore) buildPathHopIndex() { s.byPathHop = make(map[string][]*StoreTx, 4096) for _, tx := range s.packets { addTxToPathHopIndex(s.byPathHop, tx) // raw hops only } ... } ``` `byPathHop` carries two kinds of key, though: those raw wire hops, and the resolved full pubkeys fed per observation by `indexResolvedPathHops`. The pubkey strings behind the second kind are retained nowhere — #800 replaced the per-`StoreTx` `ResolvedPath` field with a hash-only membership index (`resolvedPubkeyIndex` stores FNV hashes, not strings) — so the rebuild could not reproduce them and dropped them. All three call sites run post-load: `LoadChunked` (`chunked_load.go:459`), the background fill loader (`store.go:1573`), and the deferred startup build (`index_ready_1008.go:177`). The `resolved_path` branch of the chunk scan populates the index and is then silently undone a few hundred lines later, while the `resolved_path IS NULL` fallback right beside it is explicitly documented as "byNode ONLY — the resolved_path/path-hop indexes must NOT be populated here". The two branches disagreed about who owns the index. Consequence: after a cold start every lookup keyed by a node's full pubkey missed, so `relay_count_1h/24h`, `last_relayed`, `unscoped_relay_count_24h`, `transported_scopes` (#1751) and the usefulness Traffic axis all read zero until live ingestion slowly refilled the index. ## Evidence Fixture built from live data: 2512 nodes, 17,056 transmissions, 528,891 observations, 123,057 of them carrying a non-NULL `resolved_path`. ``` before [store] Built path-hop index: 2924 unique keys /api/nodes → 0 of 2000 nodes with transported_scopes 0 with relay_count_24h > 0 after [store] Built path-hop index: 3881 unique keys (172181 resolved-hop entries retained) /api/nodes → 726 with transported_scopes 741 with relay_count_24h > 0 ``` The 957 extra keys are the full pubkeys. ## The change `retainResolvedPathHops` re-merges the pre-rebuild map's entries that the raw-hop pass cannot reproduce. Entries are carried over **only for transmissions still in `s.packets`**. That filter is load-bearing rather than defensive. `removeTxFromPathHopIndex` strips raw hops only — it derives them from `txGetParsedPath` — and its companion `removeFromResolvedPubkeyIndex` cleans the hash index, not `byPathHop`. So evicted transmissions linger under their resolved keys, and the wipe this PR removes was the only thing that ever cleared them. Filtering on liveness keeps the index bounded by the eviction policy instead of converting that gap into a permanent leak. `TestBuildPathHopIndex_DropsResolvedHopsOfEvictedTx_1904` pins it. (The eviction gap itself is pre-existing and outside this change: between rebuilds, an evicted transmission still stays referenced under its resolved keys. Filed separately.) ## Perf `O(entries in prev)` with one scratch map reused across keys (`clear()` per key, the same idiom as `hopsSeen`), plus one `map[*StoreTx]struct{}` over `s.packets` for the liveness check. It runs only where `buildPathHopIndex` already ran — cold load and background-fill completion — never on an ingest or request path. Measured on the fixture above: index build stayed within the same `LoadChunked` step, 15.2s total for 17k transmissions / 527k observations. Memory: the retained entries point at transmissions already held by `s.packets`, so no `StoreTx` is kept alive beyond eviction; the cost is map/slice overhead for keys that the feature is supposed to have. ## Tests `cmd/server/pathhop_rebuild_1904_test.go`, red before / green after: 1. `TestBuildPathHopIndex_RetainsResolvedHops_1904` — a resolved full-pubkey key survives the rebuild alongside the raw hop. 2. `TestBuildPathHopIndex_DropsResolvedHopsOfEvictedTx_1904` — a resolved key whose transmission is no longer in `s.packets` is dropped, and the now-empty key is not left behind. 3. `TestBuildPathHopIndex_NoDuplicateOnRepeatedBuild_1904` — building twice does not double-append (`indexResolvedPathHops` dedups within a call, not across the several observations of one transmission, so `prev` can legitimately contain duplicates). ``` cd cmd/server && go test ./... ok github.com/corescope/server 85.5s go vet ./... clean ``` Frontend and ingestor suites are untouched by this change (Go server only, no `public/` files). ## Interaction with #1903 Both touch `byPathHop` semantics, so I verified them composed on the same fixture. With #1904 alone the resolved keys come back and #1902's prefix collision is plainly visible again (51% of 1-byte prefix groups reporting an identical scope set). With both: ``` f79616 BE repeater ['#be','#de','#eu','#nl'] relay24h=542 f752c2 DE/NRW repeater ['#de','#de-nw'] relay24h=343 f788ad BE repeater none relay24h=383 ``` Identical-set prefix groups fall to 8%, relay counts stay intact, and each node's scopes match what its own `resolved_path` rows say. The two changes are independent and compose cleanly. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:34:20 +02:00
efitenandJoel Claw	2f711eb851	perf: use prepared statements for frequently-called server DB queries (rebase of #1878) (#1934) ... Continues #1878 by @Joel-Claw, at their request. Both commits are theirs, authorship unchanged; I only rebased them onto master and resolved the conflict with #1909. ## The conflict, and how it is resolved Exactly the two places I named in the review on #1878: `OpenDB` and `Close()`. Both PRs rewrite them, and #1909 went first because it is the correctness fix. **`OpenDB`** — kept #1909's pinned-connection `detectSchema` and added this PR's `prepareStatements()` after it: ```go derr := d.detectSchema(ctx, sc) _ = sc.Close() if derr != nil { conn.Close(); return nil, fmt.Errorf("schema detection failed: %w", derr) } // Statements are prepared after schema detection so they can never be // compiled against a schema mode that turned out to be wrong (#1901). if err := d.prepareStatements(); err != nil { ... } ``` The ordering matters and is not arbitrary: preparing before detection would compile statements against a schema mode that #1909 exists to stop trusting. **`Close()`** — kept this PR's statement closing and **did not** restore the WAL checkpoint. #1909 removed it deliberately: the handle is `mode=ro`, so `PRAGMA wal_checkpoint(TRUNCATE)` can only ever fail with "disk I/O error (778)" and was emitting a misleading storage-fault line on every shutdown. That reasoning survives; the statement closing is added in front of it. ## Verification - Both commits cherry-picked onto `e5595ad9` - `cmd/server` builds - **Full `cmd/server` suite: ok, 0 failures** (not just the targeted DB tests — after master briefly went red today from a two-PR interaction, a full local run seemed worth the two minutes) ## Review points still open, none blocking From my review on #1878, unchanged by the rebase: 1. Every SQL string now exists twice, once prepared and once as the `stmtQueryRow` fallback literal, with nothing keeping them in sync. The fallback is genuinely needed — twelve test helpers build `&DB{conn: ...}` directly and never call `prepareStatements` — but a constructor for those helpers would remove the duplication. 2. `stmtCountObsLastHour` and `stmtCountObsLastDay` are byte-identical SQL. 3. `OpenDB` now refuses to start rather than degrading when a Prepare fails. Contained today, since none of the 13 prepared queries touch a schema-conditional column, but the failure mode changed. @Joel-Claw — your work, your credit. Ping me if you would rather take it back. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: Joel Claw <358739783+Joel-Claw@users.noreply.github.com>	2026-09-02 14:34:16 +02:00
efitenandClaude Opus 5	aabbd50d27	feat(nodes): export the visible node list as MeshCore companion contacts JSON (#1889) ... ## What Adds an **Export JSON** button to the Nodes topbar that downloads the currently visible node list as a MeshCore companion-app config file: ```json { "contacts": [ { "type": 2, "name": "BE-MGU-RP03 | ON7YT", "custom_name": null, "public_key": "7a7a37d4819fb27440ed1439ca7d281fdecceb83b27e61a69745feb004d726a4", "flags": 0, "latitude": "51.07307", "longitude": "5.5796", "last_advert": 1786545431, "last_modified": 1786545431, "out_path_list": null } ] } ``` That is the exact shape the companion app itself writes, so the file imports straight back into a companion app as contacts. The practical use case is per-area: pick an area in the area filter, export, and hand someone the repeaters for that region instead of having them wait for adverts. ## Scope of the export WYSIWYG — the button exports the rows the table is showing, in the table's current order, so area, region, role tab, search, last-heard and status filters all carry over. The button label shows how many contacts the file will contain and is disabled when that count is zero. ## Field mapping | JSON field | Source | Notes | |---|---|---| | `type` | `node.role` | repeater→2, companion→1, room→3, sensor→4; unknown/empty→1 | | `name` | `node.name` | unchanged, emoji included | | `custom_name` | — | always `null` | | `public_key` | `node.public_key` | full 64-hex | | `flags` | — | always `0` | | `latitude` / `longitude` | `node.lat` / `node.lon` | stringified, as the format expects | | `last_advert` | `node.last_seen` | RFC3339 → unix seconds | | `last_modified` | mirrors `last_advert` | no separate source exists | | `out_path_list` | — | always `null`; the companion app discovers routes itself | Nodes are skipped when they have no name, a pubkey shorter than 64 hex chars, or no usable position (missing, non-numeric, or null island). Filename: `corescope_nodes_<area|all>_YYYY-MM-DD-HHMMSS.json`. ## Shape of the change The mapping lives in a self-contained `public/nodes-export.js` (`window.NodesExport.buildContacts/filename/download`); `nodes.js` only gains the button markup, a click handler and a small `updateExportBtn()`. No backend change, no new API call — the export reuses the already-fetched node list, so there is nothing per-node to fetch. ## Tests - `test-nodes-export.js` — field mapping and key order, role→type table, skip rules, order preservation, filename format (added to `test-all.sh`). - `test-nodes-export-wiring.js` — `index.html` loads the module before `nodes.js`; the button lives in the topbar and hands the filtered `nodes` array plus `AreaFilter.getSelected()` to `NodesExport.download()` (added to `test-all.sh`). - `test-nodes-export-e2e.js` — Playwright: downloads the file, validates the JSON shape per contact, asserts the button count matches the file, and that narrowing the search shrinks the export set. ## Browser validation Deployed to staging and checked in Chromium: - Desktop 1400×900 — button renders at the right of the topbar next to the count pills, `Export JSON (1962)`. - Mobile 390×844 — topbar stacks, button visible, no horizontal overflow (`scrollWidth == clientWidth`). - E2E against staging: 1761 contacts exported, `corescope_nodes_all_2026-08-12-164349.json`, shape validated, search narrowing confirmed. - With the `BE-LIM` area filter active: 44 contacts, `corescope_nodes_BE-LIM_2026-08-12-164445.json`, type histogram `{1: 1, 2: 42, 3: 1}`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:22:51 +02:00
efitenandClaude Opus 5	d821d9a390	feat(retention): add observerPurgeDays hard-delete for long-inactive observers (#1886) ... ## Problem `RemoveStaleObservers` only soft-deletes — it sets `inactive = 1` and the row stays forever. On a long-running deployment those rows just accumulate: on a two-year-old instance roughly 25% of the `observers` table was rows nobody can ever see again. There is currently no way to reclaim them. ## Fix A second retention stage. `PurgeStaleObservers` hard-deletes rows that are: - already `inactive = 1` (so the soft-delete stage owns the decision of *when* an observer goes stale), **and** - older than `retention.observerPurgeDays`, **and** - referenced by nothing. New config field `retention.observerPurgeDays`, default `0` = disabled. Existing deployments are unaffected until they opt in. Set it above both `observerDays` and `packetDays` — below those the reference guards keep every candidate row anyway. ## Why the reference guards are the point `observations.observer_idx` is a bare rowid with no foreign key. Deleting a still-referenced observer silently orphans history — `packets_v` stops resolving the observer and those packets get mis-attributed. Nothing errors; the data just quietly goes wrong. So the statement guards on all three referencing tables: ```sql AND NOT EXISTS (SELECT 1 FROM observations o WHERE o.observer_idx = observers.rowid) AND NOT EXISTS (SELECT 1 FROM observer_metrics m WHERE m.observer_id = observers.id) AND NOT EXISTS (SELECT 1 FROM dropped_packets d WHERE d.observer_id = observers.id) ``` This is correctness, not defensive padding — it was found the hard way, by orphaning 280 observation rows during a manual purge that skipped one of these checks. Each guard has its own test. ## Performance Each `NOT EXISTS` is an index seek per candidate row (`idx_observations_observer_idx`, `idx_dropped_observer`, the `observer_metrics` PK), and `observers` is O(100). It runs on the existing daily retention tick alongside `RemoveStaleObservers`, never on the ingest path. ## Tests Eight tests in `cmd/ingestor/observer_purge_test.go`, written before the implementation: - deletes an unreferenced stale row - keeps a row referenced by `observations` — and asserts zero orphans afterwards - keeps a row referenced by `observer_metrics` - keeps a row referenced by `dropped_packets` - keeps a row that is old enough but still `inactive = 0` - keeps a row inside the retention window - no-ops when disabled (`0` and `-1`) - config accessor table test ## Invariant Writes stay in `cmd/ingestor` per #1283. `cmd/server/readonly_invariant_test.go` now also forbids `PurgeStaleObservers` as a method on the server's `*DB`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:22:30 +02:00
TeTeHackoandClaude Opus 5	97b6090344	fix(hash-size): key the zero-hop advert skip on the path byte, not the route type (#1913) ... ## Summary `computeNodeHashSizeInfo` skips zero-hop direct adverts by **route type**. It should skip them by the **content of the path byte**, because the two cases are no longer the same thing. A zero-hop direct advert carries no path, so its hop count is 0. Whether the two size bits next to it mean anything depends on the sender: - Firmware that predates [meshcore-dev/MeshCore#3293](https://github.com/meshcore-dev/MeshCore/pull/3293) does `packet->path_len = 0` in `Mesh::sendZeroHop()`, wiping the whole byte including the size bits. `0x00` genuinely says nothing about the node's `path.hash.mode` — skipping it is right, and #649 was right. - A sender that writes the size through `setPathHashSizeAndCount()` emits `0x40` (2 bytes) or `0x80` (3 bytes) with a zero hop count. On a zero-hop packet nothing else can set those bits, so they are a deliberate declaration. #653 landed the skip as `pathByte & 0x3F == 0`, which swallows the second case too. The diagnosis in #649 had actually proposed `pathByte == 0x00`; the review widened it on the reasoning that a zero hop count always implies zeroed size bits. That was true in April, when no firmware wrote them. It is not true now. On the Czech mesh (869.4 MHz), a 24h window of 10k packets holds **54 zero-hop direct adverts: 39 at `0x00` and 15 carrying a declared size** (14× `0x40`, 1× `0x80`). ## Why it matters for display, not just tidiness Measured on one node over a 7-day window. A companion was reconfigured from a 2-byte to a 3-byte path hash. Its first advert under the new setting was a zero-hop direct one on **24 Aug 15:36 UTC** declaring `0x80`. That packet was dropped, so the node kept reading as 2-byte until its next **flood** advert arrived on **25 Aug 10:18 UTC** — 18h42m serving a configuration the analyzer had already been told was stale, confirmed against both an unpatched and a patched instance. With local adverts typically every 2h and flood adverts every 25h, that gap is the normal case rather than a corner one. It bites hardest on an instance whose retention window is shorter than a flood advert interval: there the node has *no* countable advert at all and falls out of `hash_size` entirely (which is what #1912 is about on the rendering side). ## Change `(pathByte & 0x3F) == 0` → `pathByte == 0x00`, in `computeNodeHashSizeInfo` and in `computeAnalyticsHashSizes` so the two views agree. `isZeroHop` renamed to `isUndeclaredZeroHop` in the latter, since that is now what it means. No complexity change — same single byte comparison inside the existing scan. ## Measured A/B Two builds of the **same commit**, one with the change, both run read-only against the same copy of a real 181k-transmission / 973-node database: | | baseline | patched | |---|---|---| | nodes changed | — | **1** | | nodes regressed | — | **0** | | `hash_size_inconsistent` | 6 | **6** | | `multi_byte_status` split | 726 / 161 / 86 | unchanged | The flip-flop flag not moving is the point worth checking: a node that legitimately changes its mode mid-window is still handled by the recency decay from #1788, so reading these packets does not resurrect false "varies". ## Tests `cd cmd/server && go test ./...` → **ok**, 0 failures. Coverage 83.5%, unchanged from master. 5 new cases in `cmd/server/zerohop_hashsize_test.go`, two built from real off-air packets: - zero-hop DIRECT `0x40` → `HashSize 2` (was: dropped) - zero-hop DIRECT `0x80` → `HashSize 3` - zero-hop DIRECT `0x00` → still absent from the map, i.e. #649's behaviour preserved - TRANSPORT_DIRECT at path-byte offset 5, declared vs wiped - the declared size reaching `computeMultiByteCapability` as `confirmed`, which is what the map's multi-byte overlay reads **One existing test changed, flagging it explicitly:** `TestHashSizeTransportDirectZeroHopSkipped` used `0x40` as its "should be skipped" fixture. It now uses `0x00` — the case it was written to cover, since #747 was about the missing `RouteTransportDirect` skip rather than about the size bits. The `0x40` case is covered by the new tests with the opposite expectation. ## Deliberately not touched The decoders (`cmd/server/decoder.go:648`, `cmd/ingestor/decoder.go:1045`) still report `HashSize 0` for these packets, so per-packet views keep showing the size as unknown. Arguably they should follow the same rule, but that changes packet display rather than node attribution and felt like a separate call for you to make. ## Caveat worth stating This attributes a declared size to the pubkey inside the advert. That holds as long as the advert was transmitted by the node that owns it — the same assumption the existing zero-hop **flood** path already makes, so this change does not widen it. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:21:42 +02:00
efitenandClaude Opus 5	e5595ad92f	fix: unbreak master — decouple the pathTrust builder test from the default (#1932) ... **master is currently red.** This is the fix. ``` --- FAIL: TestNeighborEdgesBuilderPathTrustExcludesOneByte neighbor_builder_test.go:301: 1-byte hop must not produce an edge under the default threshold, got 1 ``` ## What happened Two PRs that were each green on their own: - **#1929** moved `DefaultMinHashBytesForMapping` from 2 to 1. - **#1930** carries `TestNeighborEdgesBuilderPathTrustExcludesOneByte`, written when the default was 2. Neither pipeline saw the other, because a `pull_request` run tests the merge commit as it stood when that run started. Both merged, and the combination fails. My mistake for merging them in the same batch without re-running one against the other. ## The fix The test passed `nil` for the trust config and leaned on the package default being 2: ```go // nil == package default (MinHashBytesForMapping = 2). store.buildAndPersistNeighborEdges(nil) ``` That coupling is the real defect. The test is about what happens **at threshold 2**, not about what the default happens to be. It now says so: ```go trust := &packetpath.TrustConfig{MinHashBytesForMapping: 2} store.buildAndPersistNeighborEdges(trust) ``` It keeps testing exactly what it was written to test, and stops breaking when the default moves. The sibling `TestNeighborEdgesBuilderPathTrustAllowsTwoByte` already passes its own fixture explicitly, so this brings the two into line. **No production code changed.** `cmd/ingestor` PathTrust and Neighbor tests pass. ## Worth recording This is the failure mode I have been flagging on other PRs all day, and I walked into it myself: green CI on a PR is a statement about the base it was tested against, not about master. Two PRs can each be green and still be red together. Nothing about the review process would have caught it — only re-running one against the other, or a merge queue, would. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 09:31:01 +00:00