mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 12:57:40 +00:00
417c8dd00a243f29822f34e5c9aaadbd2e522213
3126
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 TeTeHackoandClaude Opus 5.5
|
417c8dd00a |
feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) (#2300)
Closes #2270, closes #2271. The ingestor took only the confirmed region scopes out of an observer's `/neighbors` report (#1865) and dropped the rest. Nothing could therefore say when an observer last reported, or whom it heard. ## Ingestor `observer_neighbor_reports` (created in `dbschema.Apply`) keeps the latest report of each observer, one row per observer, replaced by every live report. The report is unauthenticated, so `neighbors[]` is stored sanitised: - only 64-hex keys - `snr` and `heard_secs_ago` finite and in range - `status` limited to `[a-z_]` - `scopes` clamped - at most 500 rows `total_neighbors` is believed only when it is not below the number of rows that arrived. A **retained replay** only fills a missing row, and it is dated by the report's own `timestamp`. After an ingestor restart, a long-dead observer therefore does not look as if it just reported. ## API The table is optional for the server (no `AssertReady` change); without it the fields are simply absent. - `GET /api/observers`: `neighbors_report_at` and `neighbors_count`, from one batch query in the existing cached builder. - `GET /api/observers/{id}`: also `neighbor_report`, which is `{received_at, reported_at, neighbor_count, queried_neighbors, self_scopes, self_default_scope, neighbors: [{pubkey, name, role, snr, heard_secs_ago, status, scopes}]}`. Names and roles come from the node table in one query. ## Frontend `public/observer-neighbors.js` is shared by both pages and escapes everything. - **#2270, observers list:** a sortable **Neighbors** column, e.g. `3h ago · 12`, or `—` when the observer never reported. The same value appears in the slide-over. - **#2270, observer page:** **Neighbors Report** and **Neighbors** stat cards. - **#2271, observer page:** a **Neighbors** table listing each neighbor by name (a link to the node) or key prefix, with SNR, heard, scope query status and scopes. The heading says when the list is shorter than the count, e.g. `(2 of 3 listed)`. ## Tests - **ingestor:** sanitising, the cap, live replace, the retained rules (fill only, dated by the report, undated ignored) and the MQTT topic end to end. The existing #1865 scope tests are unchanged and still pass. - **server:** the list summary, the detail with names, and both endpoints without the table. - **frontend:** - unit tests of the helpers - an XSS case for every rendered field; I checked that it fails when escaping is removed - `test-observers-headings` now expects the new column after `Last Packet` - **E2E:** `test-observer-neighbors-2270-e2e.js` intercepts the observer endpoints, since the fixture has no reports. It is listed in `scripts/non-unit-tests.json` and on shard 2. Gates that pass: - `go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema` - `sh test-all.sh` - `eslint@8`, with 0 errors - the XSS diff gate - these E2E suites: the new one, `test-issue-1639-observers-sort-e2e.js`, `test-observer-iata-1188-e2e.js` and `test-e2e-playwright.js` Performance: the list adds one primary-key-sized query per cache fill, and the detail adds two small queries per request. Also fixes #2330 (folded in from #2337, second commit). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nexandn30nex
|
202bafea28 |
fix(observers): show metadata while analytics loads (#2339)
Partial follow-up to #1778, limited to observer details; other pages remain separate. Observer identity, status and radio information now render when the basic request finishes. Analytics and clock calibration load into their own regions, so either can finish first. An analytics failure preserves usable metadata. Identity failures and obsolete responses cannot overwrite the current view, and failed chart construction is cleaned up. The same three requests start together. Generation ownership, request deduplication, date-window behavior and disposal remain covered. No new fetches, settings, dependencies or customizer options; secondary replies preserve the actual metadata DOM node. [RED CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426343054/job/121932999182) proves 14 new assertion failures while all 11 existing unit cases pass. Local RED browsers reproduce 20 new failures while all 32 old cases pass. GREEN `52eabad7` passes 221 standalone suites, 28 ownership unit checks and 56 browser cases across desktop/mobile and both themes; the parent independently repeated the 56 browser cases. All three independent review roles passed. Held-response assertions prove the rendering behavior without claiming a server-latency speedup. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427087948) passes on this exact head, including all three browser shards, the repeated slide-over stability gate and the architecture build. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
n30nexandn30nex
|
1ed5e4ab3f |
fix(compare): exclude observer TX-only packets from overlap (#2346)
Red commit: `39295da8` ([intended CI assertion failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40605770951/job/122544822813)). Green commit: `d1bda496`. Observer comparison counted an observer's TX-only packets as receptions, producing false overlap. It now requests `excludeTx=true`: the selected observer and non-TX direction must match the same observation in memory and SQLite v2/v3. Normal packet lists and stored TX records remain unchanged; missing directions retain legacy compatibility. Refs #667. This completes a comparison slice, not the issue's remaining observer counters or #671's fleet/SNR feature. Representative display fields and existing time-filter semantics are unchanged. API/OpenAPI docs describe the filter and multi-node bypass. Validation: - RED: Go tests compiled with 32 intended failures/16 control passes; unit and browser regressions reproduced false overlap. CI reached the new unit assertion. - GREEN: all 48 packet cases pass, plus grouped-cache isolation and OpenAPI tests. The added cache regression caught a malformed-radio cache-key collision before correction. - Full local server suite: 174s; ingestor: 185s; `npm run test:unit`: pass. Final focused packet/grouped/OpenAPI checks: pass. - E2E assertion added to `tests/e2e/test-issue-1640-compare-discovery-e2e.js`; all 10 checks pass, including correct 1/2 reception totals and 50% overlap. Performance: still two capped bulk requests, indexed candidate lookups retained, one direction check within the existing observation scan/SQL predicate, no new persistent structures. No configuration/customizer changes. Final-head CI passed; independent reviews found no must-fix items. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
n30nexandn30nex
|
c211ef7643 |
feat(nodes): show relay activity from certain forwarding evidence (#2336)
Fixes #2331 with the additive API option. The Last Relayed row shows green **actively relaying** only when `last_certain_relay` falls within the existing `healthThresholds.relayActiveHours` window. Missing, invalid, future or expired evidence adds no idle/alive claim. Existing `relay_active`, attributed timestamps and counts retain their meanings. The optional node list/detail field uses the existing certain-relay path: an eligible observed frame, matching hop, unique prefix, no retired namesake and existing signature checks. Originating an advert does not count. Forwarding another node's advert can count, even with zero non-advert attributions. Adverts with an unknown origin cannot establish forwarding. Evidence uses transmission `first_seen`, not late observation receipt time, and follows node-membership eviction. Snapshots take one read lock over requested keys, O(page size), with no packet scan or new database query. Work per qualifying hop stays bounded. No new dependency or setting; configuration documentation and OpenAPI explain the distinction. The test-first history includes assertion-failing RED commits and GREEN fixes. The initial RED's nil-return compile stub is explicitly permitted by `docs/agents/TDD.md`. Local native server/ingestor suites, focused race checks, 809 frontend helper assertions and 70 browser cases pass. Browser coverage includes both themes and full/side/tablet detail views. [Review regression RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40421464618/job/121916723339) proves both malformed-origin failures. All three independent review roles pass at `d1fe79cb`. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40422655728) passes on that exact commit, including all three browser shards and the two-architecture image check. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
n30nex
|
1e1bfa0255 |
fix(ingestor): require unique hop prefixes for relay liveness (#2334)
Fixes #2329 (option 1: unique-prefix liveness). The ingestor could select a likely relay through neighbour adjacency and refresh its `last_seen` even when its prefix matched two nodes. Contextual resolution remains available in stored `resolved_path`; liveness now uses the existing unique-prefix resolver, sharing the parsed path and prefix-index snapshot. The current flood-frame eligibility gate remains unchanged. Retired-prefix catalogue parity with the server is separate. The native SQLite regression exercises real `InsertTransmission`: the displayed path still selects the adjacent candidate, neither colliding relay is touched, and the unique preceding hop is refreshed. Existing direct/TRACE/flood checks remain intact. [RED CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40407478987/job/121868990704) fails the timestamp/touch-count assertions; [GREEN race CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409860054/job/121877671472) passes at `77be50e8`. Full local server and ingestor suites, repeated focused race checks and all 221 frontend suites pass. Existing test assertions are unchanged; one misleading resolver comment is corrected. Frame/path assumptions were checked against MeshCore firmware revision `81dbae59`. Performance: one additional indexed lookup per hop, O(h), with no node scan or extra database query. No schema, dependency, configuration or customizer change. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409860054) passes on `77be50e8`, including the race detector, all browser shards and architecture builds. Three independent reviews found no blockers; final repository checks pass. Ready for maintainer review. |
||
|
n30nex
|
f9b7c66a8b |
fix(ingestor): open buffered writes before startup packet retention (#2328)
Partial fix for #2325; leave the issue open for the remaining startup work. MQTT subscribed early but its buffer could not drain until the entire packet-retention backlog finished. Required startup work and the migration barrier still finish first; ingestion now opens before the initial packet sweep, so buffered writes can use the gaps between the existing 250-transmission transactions. The sweep remains synchronous with its count/error logging. No new worker, shutdown owner, setting, retention window or batch size. The readiness log reflects the actual gate opening. Earlier non-packet maintenance, initial vacuum ordering and diagnostics publication during maintenance remain unchanged. Native SQLite tests exercise blocked migrations, inter-batch writes, FIFO/drop accounting, stopped buffers, callback counts, rollback/retry and main-function wiring. Removing the migration wait fails the regression. Existing tests are unchanged; GREEN strengthens the new RED test with a blocked-gate assertion and early-failure cleanup. [RED CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40308382930) fails the intended assertions. [Final-head CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40311222867) passes on `f8a4d175`, including Go/JS tests, race checks, all browser shards and architecture builds. Full local ingestor tests, repeated focused race checks and local browser/API smoke checks pass. Three independent reviews found no blockers in this partial scope. This removes the whole-sweep startup gate; it does not bound an individual deletion transaction or claim a measured live latency improvement. |
||
|
n30nexandn30nex
|
ef17d78909 |
docs(api): document node resolution and battery history (#2338)
Partial follow-up to #1670; the broader documentation issue stays open. Documents the existing node-prefix resolution and node battery-history GET APIs: parameters, normalization and lookup rules, defaults, ambiguity/visibility, ordered samples, nullable latest values and error media types. The generator now has concrete success schemas for both. Error statuses remain documented in prose under its existing convention. Removes exactly these two entries from the known-gap list: 18 → 16, with 115 of 131 routes described. No handler, query, configuration or dependency changes. [RED CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426035729/job/121931981058) reaches the missing-documentation assertions. GREEN `521fb3ef` passes both full native CGO suites and focused OpenAPI/real-handler contract tests. Parent browser acceptance passed 28 checks through the rendered Swagger UI and actual endpoints, including defaults, errors and empty/null history. All three independent review roles passed. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427081025) passes on this exact head, including all browser shards and the two-architecture image check. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
n30nexandn30nex
|
8437d22356 |
docs: use consistent snapshots for backup and restore (#2345)
The deployment guide copied a running `meshcore.db`, which can omit committed WAL data or produce an inconsistent backup. It now uses the existing authenticated `/api/backup` snapshot, checks the download before publishing it, and explains how to restore with all database users stopped while preserving the old database and sidecars for rollback. Fixes #703, following the [maintainer's remaining documentation scope](https://github.com/Kpa-clawbot/CoreScope/issues/703#issuecomment-6163256620). - Protects API credentials in a header file and keeps incomplete downloads out of the completed backup set. - Links the separate account-backup guide; configuration and secrets remain separate backup responsibilities. - Replaces unsafe live-copy scheduling instructions with the same checked download procedure. Validation: - Exact shell example with real curl/SQLite: valid download, HTTP 401/500, empty, HTML, corrupt database, and truncated transfer. Failures preserved the previous backup and published no new `.db`. - Existing `TestBackup*` passed. Chromium downloaded an authenticated snapshot from a disposable instance; a row committed only in its nonempty WAL survived and `quick_check` returned `ok`. - Disposable stopped-file restore and rollback recovered the original WAL-only row and preserved the original files byte-for-byte. - `git diff --check` and privacy preflight passed. TDD exemption: pure docs; no application behavior changed and no test files modified. No new config/customizer settings. Live operator services were not used for validation. Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
efitenandClaude Opus 5.5
|
15e2d2a4a7 |
fix(packets): read HB from the row's own raw_hex header (#2351)
## Problem The HB column on the packets page can show `4`. Path mode 3 (hash size 4) is reserved: `Packet::isValidPathLen` in `src/Packet.cpp` returns false for it, and `Dispatcher::checkRecv` drops such a packet. So HB 4 is never a real value. The three HB cells in `public/packets.js` found `path_len` with `getPathLenOffset(p.route_type)`, the route type of the **transmission**. The raw_hex in a child row (and in a live-updated row) is an **observation's** raw_hex. The packet hash does not cover the route bits, so the same packet is heard as FLOOD by one observer and as TRANSPORT_FLOOD by another, and the transmission keeps the route type of its first observation. Example from a production DB (transmission 3645334): | source | raw_hex start | path_len at | |---|---|---| | transmission | `01 12 3A646A…` (FLOOD) | byte 1 | | observation | `00 E2E0 0000 1D 3A646A…` (TRANSPORT_FLOOD) | byte 5 | Read at the FLOOD offset, the observation gives `0xE2`, `0xE2 >> 6 = 3`, `+ 1 = 4`. In the last 300,000 observations of that DB, about 4,300 have a header route type that differs from their transmission's. ## Fix `pathHashSize(rawHex)` in `public/app.js` already takes the route type from the header byte of the same raw_hex, and the detail pane uses it. The three HB cells now call it through a small `hashBytesText` helper instead of carrying their own copy of the offset logic. Behaviour change: a raw_hex with no hash size (reserved mode 3, a DIRECT packet with no hops, missing raw_hex) shows the same placeholder as TRACE rows instead of a number. Before, missing raw_hex showed `1`. ## Tests - New `tests/unit/test-hb-column-header-route.js` (7 cases, registered in `test-all.sh`): transport raw_hex on a FLOOD transmission gives 1, flood raw_hex with 2-byte hashes on a TRANSPORT_FLOOD transmission gives 2, reserved mode 3 never gives 4, for the helper, the flat row and the group header. It fails on `upstream/master` with `'4' !== '1'`. - `sh test-all.sh`: all standalone frontend suites pass. - Browser: local server on a migrated copy of the E2E fixture, with one observation rewritten to a TRANSPORT_FLOOD raw_hex. The expanded group shows HB 2 on all three rows. The old code reads `0xE2` on that child and shows 4. ## Performance Same per-cell work as before: one header parse and one byte parse per rendered row, O(1), no new allocations beyond the existing regex test in `pathHashSize`. Only visible rows are rendered. ## Not done - The child-row call site is not hit by a unit test: child rows only render when the group is in `expandedHashes`, which the test API does not expose. It is covered by the helper test and the browser check above. - The HB sort accessor (`hash_byte_count` / `hash_size`, sort key `hb` in `packets.js`) is unchanged; it does not read raw_hex. - `transportBadge(c.route_type)` on child rows still follows the stored route type, not the observation header. Left out of scope. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
b65412ab83 |
feat(observer-logins): turn on neighbour reports in the firmware block of the credentials mail (#2349)
The firmware block of the observer credentials mail now ends with `set
mqtt.neighbors on`, so firmware observers send the same `/neighbors`
report that the OpenHop block already turns on ("Publish Neighbours",
#2347). The ingestor reads that report for region scopes (#1865).
## Changes
- `cmd/server/observer_mail.go`: one extra line in the firmware block,
and one sentence in its note. The sentence says the command sends the
neighbour table and region scopes once a day, and that older firmware
does not know it and should skip it.
- `cmd/server/observer_mail_test.go`: the full firmware block assertion
includes the new line, and the slot test checks the line and the note.
- `docs/observer-mqtt-logins.md`: the mail section describes the new
line.
## Tests
- `go test ./...` in `cmd/server`: ok. Skipped
`TestSaveGeoFilterPreservesFileMode`, which needs Unix file modes; this
run was on Windows.
- `TestObserverCredentialsMail*` failed before the change, on the
missing line, and passes after it.
## Not checked
- The command name and the 24-hour default come from #1865. I did not
read the observer firmware source.
- The wording of the error older firmware gives for an unknown command
is not quoted in the mail. The note only says to skip the line.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
efitenandClaude Opus 5.5
|
ddce76a685 |
ci(e2e): run the E2E shards on ubuntu-24.04 and fail on a broken browser install (#2350)
`ubuntu-latest` now hands out `ubuntu-26.04` runners. Every E2E shard that lands on one fails before a single test runs: ``` browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium_headless_shell-1208/... ``` The step "Install Playwright browser" runs `npx playwright install chromium 2>/dev/null || true`, so its own failure is invisible and the job only breaks at the first browser launch. ## Evidence - Run 40779894898 (PR #2349, a mail-text change): three attempts, each failing on `Executable doesn't exist`. The job log shows `Image: ubuntu-26.04`, OS `26.04.1 LTS`. - The master run at 2026-10-28 21:31Z passed. Its shard log shows `Image: ubuntu-24.04`. - `playwright` in package-lock.json is 1.58.2. ## Change - `e2e-shard` gets `runs-on: ubuntu-24.04`, with a comment saying when to unpin. The other jobs do not install Playwright and stay on `ubuntu-latest`. - "Install Playwright browser" no longer has `2>/dev/null || true`, so a failed install fails at that step. ## Not done - I did not find out why the install fails on 26.04: its stderr was discarded. Once this is merged, a run with the pin removed would show it. - No Playwright upgrade. That is a separate change, and it may make the pin unnecessary. - Before this change, `install-deps` errors were also swallowed. If it fails on 24.04, this PR's own CI will show it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
cadc666bd9 |
feat(observer-logins): OpenHop block in the credentials mail (#2347)
The observer credentials mail (#2306) had blocks for meshcoretomqtt, the Gessaman/Dutch-MeshCore firmware, Home Assistant and plain values. An operator running OpenHop (repeater software with its own MQTT uploader) could not tell which of these matched its broker form, and picked the format by trial. ## Changes - `cmd/server/observer_mail.go`: a fifth block, "4. OpenHop (repeater software)", before the plain values. It uses the field and option labels of the OpenHop broker form: - `Format: Letsmesh`, `Transport: TCP` or `Websockets` (from `observerLogins.transport`), host, port, username, password, `TLS: on/off`, `Use JWT Auth: off`, `Retain Status: on`, `Publish Neighbours: on`. - The note says Meshcoretomqtt works too, but the format labelled "openHop" (value `mqtt`) does not: it publishes to `meshcore/repeater/<name>`, which a personal login may not use. - Publish Neighbours is on because the ingestor reads the `neighbors` topic for region scopes (#1865). - `openapi.go` and `docs/observer-mqtt-logins.md`: "four formats" becomes five. ## Sources Read from OpenHop, not tested on a device by me: - format family and topics: `openhop_repeater/repeater/data_acquisition/mqtt_handler.py` (`MC2MQTT_FORMATS = ("meshcoretomqtt", "letsmesh", "waev")`, the `mqtt` format's `meshcore/repeater/<node_name>` base topic, transport values `tcp`/`websockets`); - labels: the built web UI (`Configuration-*.js`: "Add Broker", option labels `Letsmesh`, `Meshcoretomqtt`, `openHop`, `TCP`, `Websockets`, "Retain Status", "Publish Neighbours", "Use JWT Auth"). An operator on our instance set it up from a screenshot of that form with format Letsmesh, TCP, Retain Status and Publish Neighbours on; the observer publishes, and its `neighbors` report set `configured_scope` on its node. ## Tests - `TestObserverCredentialsMailOpenHopBlock`: the block text for TCP without TLS and for Websockets with TLS, the "openHop" warning, HTML escaping, and the block order. - `TestObserverCredentialsMailBlockTitles`: five titles in order. - `cmd/server` suite green locally (the two Windows-only failures skipped, as on master). ## Not in this PR - The websocket path: OpenHop uses paho's default, the doc already tells operators to match the listener path. - Anything with the other fields of the `neighbors` report (SNR, heard_secs_ago, default_scope): we are measuring how many observers send it first. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
3bf678709f |
fix(ui): preserve navigation and menu focus during delayed updates (#2333)
Fixes #2332. Fixes #2335. Two delayed UI updates could undo user actions. Rebuilding the More menu removed its focused link; a late packet-list response could replace a newly requested Nodes URL before `hashchange` ran. The menu now preserves the selected destination when it stays in More or moves inline, closes an empty menu, and leaves other controls' focus alone. Packet URL updates now require the packets route, including its existing detail/query forms. Observation selection and encoded filters remain supported. The changes add one lookup across the existing 12 navigation links and a URL ownership check. No packet loop, API request, dependency, setting or customizer change is added. [Packet RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409110743/job/121874676123) fails four wrong-route assertions. [Menu RED](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40409898375/job/121877663579) fails the intended focused-destination assertion after the packet correction removes the competing navigation failure. GREEN is `57522143`. Local combined validation passes all 221 standalone frontend suites, 60 navigation browser checks, 17 branding/route checks and 11 filter/deep-link checks. The packet regression controls delivery of a real API response; substituting the original code still fails it. Existing exact title assertions remain; diagnostics now report the actual title. Backend source trees are unchanged from the tested baseline. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40410894023) passes on `57522143`, including all three browser shards and both architecture builds. Three independent source reviews found no blockers. Final repository checks and after-push browser verification pass. Ready for maintainer review. |
||
|
efitenandClaude Opus 5.5
|
d29957fbdc |
feat(observer-logins): lock the shared login out of migrated observers (#2348)
Follow-up to #2306 / #2319. With personal observer logins, a migrated observer still is not safe from the shared login: the hook answers `ignore` for the shared login, so EMQX's built-in database lets it publish under any observer's topic, including one whose owner has moved to a personal login. This PR lets an operator close that per migrated observer, before the shared login is removed altogether. ## Changes **users.db v8.** `observer_bindings.confirmed_at`: the moment the owner's personal login first published under that pubkey. - An auto-bind (the owner's first publish) is confirmed at once. - An admin assignment stays unconfirmed until the owner's login publishes under that pubkey; that publish sets it. - Reassigning to another user or unbinding clears it; reassigning to the same owner keeps it. - Migration: existing `auto` rows get `confirmed_at = bound_at`, `admin` rows stay NULL. **Config.** `userManagement.observerLogins.sharedUsernames` (default empty): the broker's shared login names. A name starting with `obs_` is refused at startup. **Authz hook.** For a login in `sharedUsernames`, a publish under `meshcore/<x>/<pubkey>/...` is denied when that pubkey's binding is confirmed, whatever the number of topic segments (the ingestor attributes `meshcore/<x>/<pubkey>` and longer topics to that pubkey). Logged as `publish denied: user=<name> observer=<8 hex>: confirmed owner`. Every other request from a shared login keeps `ignore`, so the built-in database decides as before. Personal logins are unchanged. An orphaned binding (owner account deleted) does not count as confirmed. **Admin.** The observers table of the Observer logins tab gets a Confirmed column; the bindings JSON gets `confirmedAt`. **Docs.** `docs/observer-mqtt-logins.md`: "Locking the shared login out of migrated observers" (what it denies, how to see `confirmedAt`, the risk below). `config.example.json` has the key. ## Risk for operators An observer that is confirmed but still runs a second uploader on the shared login loses that second uploader once its name is in `sharedUsernames`. The deny log line names the observer. ## Tests - `internal/users`: v7 to v8 migration on a real v7 database, auto confirm, admin assignment then owner confirm, non-owner and repeated confirm, reassign clears, orphan reassign. - `cmd/server`: the decision table (confirmed, unconfirmed, unbound, non-publish), topics with 3, 4 and 5+ segments, trailing slash, empty and `client` iata, lowercase key; owner publish confirms, confirm failure still allows and retries; reassign, unbind and user removal clear the flag; the index load against a real store; config validation; the hook and admin endpoints end to end. - `tests/e2e/test-user-management-e2e.js`: the Confirmed column for an auto-bound observer. - Local: `internal/users`, `cmd/server` (two Windows-only tests skipped, as on master) and `cmd/ingestor` green. The E2E assertion was not run locally; CI runs it. ## Not in this PR - Using `confirmed_at` for verified `/neighbors` reports: a separate PR, built on #2300. - Removing the shared login: still an operator step after migration. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
TeTeHackoandClaude Opus 5.5
|
660cbbf5ff |
deps: golang.org/x/net v0.61.0, x/crypto v0.58.0, paho.mqtt.golang v1.5.1; runtime alpine:3.24 (#2297)
A Trivy scan of the published image found **19 HIGH and 20 MEDIUM** in Go dependencies, plus a runtime base past its support window. There is no Dependabot here, so this collects the bumps in one PR. | dependency | from | to | why | |---|---|---|---| | `golang.org/x/net` (ingestor) | v0.27.0 | v0.61.0 | CVE-2024-45338 and later advisories, fixed up to v0.60.0 | | `golang.org/x/crypto` (server, `internal/users`) | v0.31.0 | v0.58.0 | CVE-2025-22869, CVE-2025-47913 and later; `x/sys` follows to v0.49.0 | | `github.com/eclipse/paho.mqtt.golang` | v1.5.0 | v1.5.1 | CVE-2025-10543, an integer overflow in the UTF-8 string encoding | | runtime image | `alpine:3.20` | `alpine:3.24` | 3.20 is out of support; its musl 1.2.5-r3 carries CVE-2026-107891 | The Alpine change covers `Dockerfile`, `Dockerfile.go`, the arm64 smoke step in `deploy.yml` and the comment in `docker/test-select-service-user.sh`. **What changes besides the versions:** - The new `x/` modules declare `go 1.26.0`, so the `go` directive of `cmd/server`, `cmd/ingestor` and `internal/users` moves up from 1.22. The builder image and CI already run 1.27. - `x/sync` follows paho to v0.17.0. **What is left:** only `GO-2026-5932`. It flags `x/crypto/openpgp` as unmaintained; nothing here imports that package and no fix exists. ## Checks - Every module passes `go mod tidy`, `go build` and `go vet`. - `go test -race` passes in `internal/users`, `cmd/ingestor` and `cmd/server`. - A local image build on alpine 3.24.3: - ships mosquitto 2.1.2, caddy v2.11.4 and supervisor 4.3.0 - keeps the caddy and mosquitto accounts at the same uid:gid (100:101, 101:102), so `test-select-service-user.sh` still describes the image (116 assertions pass) - comes up healthy, ingests an MQTT status through mosquitto and serves the frontend 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
TeTeHackoandClaude Opus 5.5
|
77a5f14925 |
feat(observers): radio setting per observation from the observer's radio history (#1869) (#2292)
Part of #1869. ## Problem `observers.radio` keeps only the **last** `MHz,kHz,SF,CR` an observer reported. Once an observer is retuned (say from SF7 to the EU narrow SF8 setting to listen to a neighbouring mesh), every packet it ever heard reads as heard on the new setting, and there is no way to tell an SF7 packet from an SF8 one. ## What this does As #1869 proposes, the packet data model is untouched: the setting of an observation is resolved from its observer and time. **Ingestor** (all schema in `dbschema.Apply`) - `observer_radios`: one row per stretch of time an observer, and each radio of a multi-radio node, stayed on one setting. Every live status extends the current row and a change starts a new one; retained replays never extend a row. On the first run it is seeded with every observer's current radio, which is what the UI already showed for the past. - Settings are stored canonical through `dbschema.CanonicalRadio`, so `869.4320068,62.5,7,5` and `869.432,62.50,7,5` are one setting. - `observation_radios` (`observation_id → rx_radio_id`): multi-radio openHop nodes tag each packet with `rx_radio_id` and list `radios: [{id, radio}]` in their status, so their observations resolve per radio. - The table is sparse: single-radio nodes never send the field, so `observations` and every query that scans it stay untouched. - The row id comes from the observation upsert via `RETURNING id`. - Rows are pruned with their transmissions, and a purged observer takes its history along. **Server** (read-only) - No `AssertReady` change: an absent table reads as empty and is probed again later, the same rule as `advert_route_evidence`. - The history is held in memory. The live poller checks it on every tick (two primary-key lookups), so a retune or a new `rx_radio_id` is known before the observation is broadcast. A full reload runs every 10 min. - `radio` is added to observations, packets, grouped rows (taken from the header observation) and the live feed. - `GET /api/packets?radio=MHz,kHz,SF,CR` filters by setting. - Combined with `observer=`, it means *that* observer heard it on that setting. - A radio query is served from memory instead of falling back to SQL, which would return the window unfiltered. - `GET /api/radios` (named struct) and `radio_history` on `GET /api/observers/{id}`. **Frontend** - Packet filter fields `radio`, `freq`, `bw`, `sf`, `cr`, e.g. `sf == 8` or `freq > 869.5 && bw == 62.5`. - A Radio row in the packet detail and a Radio column in its observations table (only when the server reports one). - A radio history table on the observer page once there is more than one stretch. - `RfParams.radioText` formats the setting. It escapes all five characters and returns non-numeric input as it came, escaped. ## Performance `txMatchesRadio` runs once per observation inside `filterPackets`. - Each observation costs one map lookup by observer id. - Observers with a single stretch (nearly all) answer from that lookup directly. - Retuned observers add a binary search over their few stretches. - Multi-radio observers add one lookup by observation id. - The snapshot is loaded once per filter call. `BenchmarkTxMatchesRadio`, 30K packets / 150K observations (20 observers, 2 of them retuned): ``` BenchmarkTxMatchesRadio-24 602 2056924 ns/op 0 B/op 0 allocs/op ``` Memory is one entry per stretch plus one per multi-radio observation; nothing is added per observation otherwise. ## Tests - **dbschema:** seed runs once, canonical spellings, the tables exist without `observers.radio`. - **ingestor:** - stretches extend and split; canonical spellings do not split - two radios of one node hearing one transmission over two paths each keep their own radio - a retained status does not write history - prune and purge - `rx_radio_id` reaches `PacketData` - **server:** - resolution before, between and after stretches, and with an unknown radio id - `observer` + `radio`, and grouped rows carry `radio` - incremental refresh, `/api/radios`, `radio_history`, missing tables - the benchmark above - **frontend:** the new filter fields in `tests/unit/test-packet-filter.js`, and XSS cases through the real `RfParams.radioText` and the history renderer. - **E2E:** `tests/e2e/test-radio-setting-1869-e2e.js` checks the grouped rows' radio, `sf ==` on the packets page, the detail Radio row and the observer history. It is added to `scripts/non-unit-tests.json` and to shard 2. `go test -race` in `cmd/server`, `cmd/ingestor` and `internal/dbschema` passes, along with `sh test-all.sh`, `eslint@8` (0 errors) and the XSS diff gate. The new E2E test and `test-e2e-playwright.js` pass against the fixture. ## Browser validation This runs on a live instance with six observers, one of them retuned to 869.618/62.5/SF8/CR8: - The packet detail shows `Radio: 869.432 MHz · SF7 · BW 62.5 kHz · CR 4/5`. - `sf == 7` shows 106 of 106 grouped rows and `sf == 8` shows 0 of 107; the SF8 observer has not heard anything yet. - `/api/radios` lists both settings. - No page errors. ## Notes - **The customizer (rule 8)** gets nothing: the only constants are server-side refresh intervals. - **Attribution timing:** packets heard between a retune and the observer's next status message (minutes) are attributed to the previous setting, because a stretch starts when the status is handled. This is documented at `RecordObserverRadios`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
643a53e98d |
fix(operators): refresh update metadata and wait for readiness (#2285)
Red commit: `7277ca23` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190626951)). Fixes #2229. Management startup now defines logging helpers before normalizing CRLF environment files. Updates refresh version, revision and build time after checking out the target. Deployment verification and all five Compose healthchecks use `/api/healthz`, so an available statistics endpoint cannot report an unfinished startup as healthy. - The actual-script regression passes 15 checks: LF/CRLF/no environment file, repeat startup, all update targets, failed checkout, delayed/failed readiness, HTTPS readiness and unchanged informational statistics. - RED CI reproduced all 11 intended failures; GREEN `7ff9050b` passes the full standalone frontend suite and shell syntax checks. - Browser verified: an isolated Go fixture at http://127.0.0.1:18851 renders Home and returns ready=true from `/api/healthz`. - Git, Docker and network boundaries are mocked in management tests. No real container update was performed locally; CI validates the Compose/build path. - No new configuration, dependencies, schema changes or packet-processing work. Privacy, whitespace and scope checks passed. The external home-directory preflight bundle is unavailable; applicable checks were performed locally. Independent review found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191647297) passed on `7ff9050b`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
b5caea1d5f |
fix(analytics): render safe, correctly sized battery status icons (#2282)
Red commit: `4cc2efb3` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40189921192)). Fixes #2232. Battery badges now render their fixed status icon and append voltage as text, including zero. Chart sizing excludes inline icons, and status colors use the existing theme variables. Unknown status values stay muted. - The original browser regression failed on literal SVG; a second local run with only the JavaScript fix failed on oversized icons. - Validation on `eef0872e`: all standalone frontend suites; 47 focused unit checks; 41 Chromium checks, including both themes, hostile voltage, accessibility and 16×16 icon bounds. - Browser verified: http://127.0.0.1:18851 on an isolated Go fixture. - E2E assertion added: `tests/e2e/test-issue-1648-m2-icons-e2e.js:134`. - Constant work for one badge; no new requests, dependencies, settings or schema changes. Privacy, whitespace, CSS-variable and repository XSS checks passed. The external home-directory preflight bundle is unavailable; its applicable checks were performed locally. Independent review found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190620324) passed on `eef0872e`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
d07186af43 |
fix(nodes): preserve names and metadata in preview panels (#2315)
Fixes #2309 and #2312. Partial follow-up to #1154 (U1 and tablet timestamp parity). Node previews omitted Last Relayed, Traffic share and Hash Prefix, while tablet previews read `advert_name` instead of the API's `name`. Shared rendering now presents those fields consistently across full detail, side preview and tablet, with the existing timestamp tooltips. Relay attribution is labelled neutrally: ambiguous relay evidence no longer claims a node is alive. Existing certainty/status logic is unchanged. Measured zero, legacy score fallback, role applicability, missing values and escaped names/hash tooltips are covered. A resize regression prevents duplicate score-row IDs across retained desktop and tablet previews. Only the selected node's metadata is formatted. Existing request counts remain five for full/side and two for tablet; no new fetch, cache, hot loop, setting, customizer value or dependency. Validation: 806 helper checks, all 218 standalone suites, 46 browser checks in both themes, eight health checks, and the main browser suite (149 passes, three existing fixture skips). The unchanged backend tree passed full server/ingestor suites. Independent reviews completed; the parent verified the two-line follow-up from its resize check. Test-only RED `e0f681ae` produced [eight expected assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40232409958/job/121286783705). Follow-up `2bb1ccd` repairs the help fixture's scroll range without relaxing its checks; all 13 help checks pass across three font/CPU configurations. [Final-head CI passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40234768485), including all browser shards and architecture builds. |
||
|
n30nex
|
6e1abf0663 |
ui(nodes): identify path links as example packets (#2296)
Red commit: `79abbba0` ([regression CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203075654)). Fixes #2291; addresses U7 of #1154. The two observed-path links now say View example packet →, accurately describing their existing sample-packet destination. Recent-advert labels and link behavior remain unchanged. - Production diff: two text literals in public/nodes.js. - Existing browser suite preserves contrast/highlighting assertions and adds exact labels, canonical destinations and real click-through from side/full/mobile views. - RED fails eight label assertions. GREEN passes 15/15 checks normally and with 4× CPU slowdown; all 218 standalone frontend suites pass. - Browser verified: http://127.0.0.1:18902. - E2E assertion added: tests/e2e/test-issue-1146-path-link-contrast-e2e.js. Initial loads and theme toggles wait for actual theme-refresh readiness before capture, fixing a reproduced screenshot-detachment race in the touched harness. - No runtime logic, requests, settings, dependencies or performance changes. Independent reviews and local preflight checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204585697) passed on `ae849cc6`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
fb101e46e0 |
fix(nav): fit primary links around late account controls (#2286)
Red commit: `e5589b4c` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191348162)). Fixes #2216. Navigation measures the available flex strip instead of guessing component widths. Optional statistics and spacing yield when primary/active links would overlap More. Account insertion triggers a layout event immediately, including while authentication is pending or fails; normal user-change events retain their semantics. - Regression CI reproduced link overlap across narrow widths and late account controls. Review added a second RED test for the actual delayed authentication bootstrap before the three-line correction. - Validation on `f0ccbf4c`: full frontend suite; 89 user-management checks; 56 navigation checks; existing priority and clipping suites. Parent independently reran auth and browser checks. - Browser verified: http://127.0.0.1:18854; real bootstrap overlap of 14.66px becomes 21.38px clearance. Resizing, account changes, keyboard More and touch targets remain covered. - E2E assertion added: `tests/e2e/test-nav-fluid-1055-e2e.js:89`. - Work is bounded by the fixed navigation link set; events share one animation frame. No polling, observers, new requests, dependencies or settings. Privacy, whitespace, CSS-variable, XSS and ESLint checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. Independent review verified the bootstrap correction with no outstanding findings. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40192737545) passed on `f0ccbf4c`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
c853c64f1f |
feat(ui): identify current views in browser tab titles (#2287)
Red commit: `a62753de` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191439879)). Fixes #2259. Browser titles now identify the current page, Analytics tab, or selected node/packet while retaining the configured site name. A Reach view reads `Node abababababab | Reach | CoreScope`. Detail identifiers are limited to 12 characters; account/reset parameters stay out of titles. - Branding and page context have separate state, so delayed theme loads and edits cannot erase the view or accumulate suffixes. Selection/close paths that use replaceState also update the title. - Validation: all standalone frontend suites; 29 focused unit checks; 16 browser steps covering routes, history, reload, tabs, real row clicks, closing, branding and literal route text. An explicit All time selection keeps the browser test independent of fixture age. - Browser verified: http://127.0.0.1:18853 on an isolated Go fixture. - E2E assertion added: `tests/e2e/test-customize-branding-e2e.js:48`. - Fixed-size work at navigation/selection boundaries; no extra requests, DOM observers, render-loop work, dependencies or settings. Privacy, whitespace, XSS and ESLint checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. Independent review found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40193902834) passed on `4c037db5`, including all three browser shards and both architecture builds. The existing logo test now asserts the full page-and-brand title; its 6/6 checks and the 16/16 title/branding checks also pass locally. |
||
|
TeTeHackoandClaude Opus 5.5
|
684f89f200 |
feat(frontend): serve Leaflet, leaflet.heat and Chart.js from vendor/, not unpkg.com (#2299)
Refs #1554. `index.html`, `area-map.html` and `geofilter-builder.html` load Leaflet, leaflet.heat and Chart.js from **unpkg.com** on every page view. That causes three problems: - **Privacy:** every visitor sends a request to a third party. - **Availability:** where unpkg is blocked (script blockers, filtered networks) or down, the map and every chart are gone. The observer page then fails with `Error: Chart is not defined`. - **#1554:** the CSP target there, `script-src 'self'`, blocks these scripts just like the inline handlers the issue lists, but they are missing from its inventory. ## Change - **Vendored libraries:** `public/vendor/leaflet-1.9.4/` (with `images/`), `vendor/leaflet-heat-0.2.0.js` and `vendor/chart-4.5.1.umd.min.js`. - They are the unpkg dist files byte for byte, verified against the SRI hashes `index.html` already pins. - The two standalone pages loaded Leaflet without SRI; they now carry it. - This is the same pattern as `vendor/leaflet.markercluster.js`. - **`.gitattributes`:** `public/vendor/** -text`. Otherwise `* text=auto eol=lf` rewrites `leaflet.css`, which ships with CRLF, and the browser refuses it for the integrity mismatch. I hit exactly that while preparing this. - **`scripts/instrument-frontend.sh`:** now copies `vendor/` recursively. The flat `cp public/vendor/*` left the E2E server without `vendor/leaflet-1.9.4/`, which showed up as `L is not defined`. - **New test `tests/unit/test-vendored-libs.js`:** - no off-site `<script src>` or stylesheet in the served pages - every `integrity` attribute matches its vendored file - the images Leaflet's stylesheet references exist I checked that it fails against the old `index.html`. ## Checks - `sh test-all.sh` passes. - These E2E suites pass against the fixture: `test-e2e-playwright.js`, `test-analytics-fluid-charts.js`, `test-issue-1236-map-mobile-e2e.js` and `test-issue-1329-map-controls-accordion-e2e.js`. - On a live instance, a headless browser with `unpkg.com`, `jsdelivr` and `cdnjs` blocked: - the observer page renders all four charts - the map loads tiles and markers - `typeof L` and `typeof Chart` are defined - no CDN request is made and no page error occurs ## Not in this PR Swagger UI on `/api/docs` (`cmd/server/openapi.go`) still loads `swagger-ui-dist` from unpkg. I can vendor it in a follow-up if wanted. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
e61ff6e473 |
fix(observers): keep detail responses tied to the current view (#2323)
Fixes #2316 and #2320. Related to #1778; progressive loading remains separate. Delayed observer responses could replace a newer observer or period, overwrite its error/status/calibration, and leave detached charts alive. Each load now captures its observer and days and checks ownership before success or failure can update the view. Leaving the page invalidates pending loads, including same-observer revisits. New visits also reset the request window to the selector's existing seven-day default. Previously, a new page could display seven days while requesting the preceding visit's 30-day window. Production change: 12 added lines and three replacements in one frontend file. No new requests, caches, settings or dependencies; the ownership check is O(1). Existing request deduplication, comparison controls, chart cleanup and current-request errors are preserved. Validation: 11 unit checks, all 219 standalone suites, 32 controlled browser cases across desktop/mobile and both themes, comparison 9/9, and observer-IATA 5/5. Core browser run exited successfully with 149 passes and three declared fixture skips; existing conditional subchecks remain. The unchanged backend tree passed both full backend suites. The parent independently reran the new unit/browser tests. Test-only RED `a73c227f` produced [nine unit and 28 browser assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40239824064). Three independent reviews passed. [Final-head CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40240673369) passes on `ba52d873` integrated with upstream `11ce1235`, including all browser shards and architecture builds. |
||
|
n30nex
|
1dfcd0f5d3 |
fix(analytics): preserve distance invalidation during an active build (#2284)
Red commit: `38c0abe1` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190273036)). Fixes #2281. Background-fill completion could reset an active `sync.Once`, causing a fatal mutex unlock or letting an older distance build mark an incomplete index ready. Build admission now uses the existing mutex, and a generation check prevents stale completion from consuming invalidation. The loader also releases the store lock before acquiring the distance gate. - Deterministic regression drives the actual loader, recomputer and HTTP handler; the old runtime crash is contained in a child process. - Two cold observations stay below the normal 5% rebuild threshold. Ten concurrent callers must share one rebuild, and the final response must contain the cold path. - Validation on `b8a2333b`: full server suite, focused distance suite, ten repeated race runs plus a parent race rerun. Shared-base ingestor suite passed. - Browser verified: http://127.0.0.1:18855 returned 202 with Retry-After, then 200 with 20 paths and 20 hops. - Constant-time admission/invalidation and one fixed-size counter; existing computation costs remain unchanged. No schema, settings or API-shape changes. Privacy, whitespace and scope checks passed. The external home-directory preflight bundle is unavailable; applicable checks were performed locally. Independent review found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40191246966) passed on `b8a2333b`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
f66cd320da |
perf(graph): reuse indexed shortest-path workspaces (#2293)
Red commit: `8b3cffd0` ([regression CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203043161)). Fixes #2226. Bridge and coverage scoring now convert keys once and reuse indexed distance/predecessor workspaces and at most one heap item per vertex. Existing algorithms, normalization, snapshot publication and synchronous startup behavior remain intact. Five paired runs, deterministic 2,000-node/10,000-edge fixture, Go 1.27.2 on Windows/amd64, GOMAXPROCS=2: | Calculation | Median time | Allocated bytes/run | |---|---:|---:| | Bridge | 8.237s → 0.819s | 2,083,295,328 → 2,517,664 | | Coverage | 4.900s → 0.694s | 1,508,440,736 → 2,209,984 | The desktop was shared; baseline bridge runs ranged 7.81–17.56s. These are algorithm/allocation measurements, not startup/API latency or retained memory. Rebuilding committed `5f34b5bf` produced the exact measured binary hash. - Independent path oracle, edge/epsilon/tie/disconnected cases, allocation regression, race checks and full server/frontend suites pass. - Browser verified: isolated native servers at http://127.0.0.1:18903 and http://127.0.0.1:18904 return identical nonzero graph scores. - O(V+E) bounded workspace; O(V(V+E) log V) calculation. No persistent cache, schema, dependency or configuration change. Independent reviews and local preflight checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204557795) passed on `5f34b5bf`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
63d63ea90a |
perf(healthz): avoid scanning retained packets after startup (#2305)
Fixes #2301. Readiness probes currently scan every retained transmission while holding the packet-store read lock. Once `StartupLoadDone` closes, this reads the maintained observation counter beside the transmission count under that same lock. Count extraction becomes O(1), with no new cache, query or allocation. Startup retains the original scan: the background loader publishes indexes before its packet slice, so switching on readiness or `LoadComplete` would be premature. Response fields, readiness status, ingestion and configuration remain unchanged. Whole-handler benchmark medians, microseconds per probe: | Retained transmissions | Upstream | Candidate | | ---: | ---: | ---: | | 1,000 | 72.14 | 83.84 | | 30,000 | 320.85 | 65.74 | | 100,000 | 1,228.50 | 81.97 | Four alternating pairs, 10,000 iterations/size, real SQLite-loaded fixtures with three observations per transmission on average; Go 1.27.2, Windows/amd64, two CPUs. All samples retained; shared-host variability means no small-cache or production-latency improvement is claimed. Allocations remain 36/probe. The included benchmark supports reproduction against unchanged upstream production plus the identical harness. TDD exemption: pure performance refactor; **all pre-existing test files are byte-unchanged**. Added characterization tests pass before and after, covering loaders, deduplication, live updates, eviction, partial startup failures, concurrent snapshots and liveness. Independent review prompted test-only follow-up `2a0083b1`, which detects premature counter use; production remains identical to measured `1b8e6680`. Full server, race, ingestor and post-push browser/API checks pass locally. [Final-head CI passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40213482779), including all browser shards and architecture builds. No customizer changes. |
||
|
n30nex
|
abad8a26b3 |
fix(liveness): require evidence from each observed relay frame (#2298)
Red commit: `46e3b082` ([regression CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203680455)). Fixes #2266 and #2267. Certain relay activity now requires an eligible path from the observation that supplied it. A representative transmission header, destination attribution or catalogue uniqueness alone cannot supply that evidence. Later valid FLOOD evidence survives observer/path deduplication. The fix preserves valid own packets and excludes failed signatures, planned DIRECT/TRACE paths, ambiguous/retired prefixes and missing or mismatched headers. Firmware reference: meshcore-dev/MeshCore `81dbae59`. - One eligibility byte replaces an unused raw-frame field. StoreObs remains 152 bytes in the protected 160-byte allocation class; no wire frames are retained. Existing dedup maps gain one pointer per slot only at 64+ observations. - Processing is linear in retained paths, bounded by 64 wire path bytes. No new SQL, schema, configuration or maps. - Tests cover five load/ingest paths, late duplicates, analytics/adaptive thresholds, positive controls and concurrent readers. Review correction `4ad6c5ad` preserves lazy decoded caches while retaining escaped/legacy signature rejection. - Full server suite, focused/race checks and frontend checks pass. Browser verified on http://127.0.0.1:18905: both upstream false-positive cases disappear and a genuine later FLOOD is recognized. Independent review findings are addressed. Local preflight checks passed; external home-directory bundle unavailable. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40205015381) passed on `4ad6c5ad`, including all three browser shards and both architecture builds. |
||
|
AdamandClaude Opus 5.5
|
d777f4b7ce |
fix(ingestor): only flood relay hops refresh nodes.last_seen (#2307) (#2308)
Fixes #2307. The ingestor refreshed `nodes.last_seen` for every resolved hop in a path. For DIRECT routes those hops are still to come, and for TRACE `path_json` is the route the trace was asked to take, so tracing or messaging a switched-off repeater kept it looking recently heard. ## What changes **`cmd/ingestor/db.go`** - New `pathHopsTransmitted(data)`: true only when `PathBytesAreHops(payloadType)` and the route is FLOOD or TRANSPORT_FLOOD. It's the same rule as the server's `relayHopHexLen` in `certain_heard.go`. - `InsertTransmission` calls `touchRelayNodesLocked` only when that holds. `observations.resolved_path` is still written for every observation, so analytics are unchanged. - The route and payload type come from the observation's own decoded header (`PacketData`), not the transmission's first frame (see #2266). ## Perf There's one integer comparison per observation, before the existing touch. The touch now skips DIRECT and TRACE observations, so it does fewer `UPDATE nodes` calls, never more. ## Tests `cmd/ingestor/relay_touch_test.go`: `TestInsertTransmission_TouchesOnlyFloodRelays` goes through `InsertTransmission` with a real store and prefix index, using a node whose 2-byte prefix is unique: - FLOOD and TRANSPORT_FLOOD TXT_MSG: `last_seen` advances to the rx time - DIRECT TXT_MSG and DIRECT TRACE: `last_seen` is unchanged With `db.go` from master, the DIRECT and TRACE cases fail. How I checked: - `go vet ./...` and `go test ./...` in `cmd/ingestor` pass, and `gofmt -l` is clean (`golang:1.27` container, since there's no local Go toolchain). - No server, frontend or schema changes, so I didn't run the browser and E2E steps. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
410db29fe6 |
chore: remove unused legacy Dockerfiles (#2314)
Fixes #2310. Partial follow-up to #1385 (PR-C). Remove unused `Dockerfile.go` and `Dockerfile.node`, which describe retired alternate builds. Current workflows, Makefile, Compose variants and operator scripts already select the canonical `Dockerfile`. Remove the obsolete formatter exclusions and label the old Node-to-Go migration guide historical, linking current deployment instructions. Historical release prose remains untouched. The canonical image, runtime files, dependencies and Compose behavior are unchanged. Validation: audited all 1,501 tracked files; old and new formatter inputs are exactly the same 633 real Go files. Both formatter commands, YAML/shell/workflow-reference checks, existing workflow/routing tests, COPY gate and test inventory pass. All 218 standalone frontend suites pass. The unchanged backend tree passed full server/ingestor suites, and the unchanged app passed 12 browser smoke cases. TDD exemption: behavior-preserving cleanup/refactor plus documentation; no test files modified. `make` and `actionlint` were unavailable locally; equivalent formatter and focused workflow checks ran directly. [CI passed on this head](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40230911566), including canonical container builds and all browser shards. Three independent reviews passed. The published historical guide and its current-deployment links render correctly in Chromium. No settings/customizer changes. |
||
|
n30nex
|
3a85a98cc5 |
docs(api): describe the supported packet-list query parameters (#2294)
Red commit: `cee2a4bc` ([regression CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203052113)). Partial follow-up for #1670. GET /api/packets now documents all 16 supported query parameters, including hash, route, node(s), region, channel, area and expand. It removes unsupported sort/timeRange/search entries, corrects numeric types, and describes actual defaults, limits and memory/SQL/multi-node differences. This changes only packet-route metadata. Runtime API behavior and the schema generator stay unchanged. - Served-spec regressions fail on upstream while actual-route controls pass. - Focused OpenAPI/SwaggerUI tests, full server suite and full standalone frontend suite pass on `d8014193`. - Browser verified: http://127.0.0.1:18906/api/spec exposes the 16 supported parameters and integer types; the running packet route confirms default limit 50 and offset 0. - No new requests, dependencies, settings, schema or untyped-map occurrences. The umbrella remains open: current route coverage stays 98/122 with 24 allowlisted gaps. Other endpoint parameters/schemas and a generic parameter-presence gate are outside this PR. Independent reviews and local preflight checks passed. External home-directory preflight bundle unavailable; applicable checks were performed locally. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40204567683) passed on `d8014193`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
041daecd1e |
docs(api): describe performance and clock diagnostics (#2303)
Partial follow-up to #1670. Six existing read-only diagnostics endpoints lacked usable OpenAPI descriptions: process I/O, SQLite metrics, ingestor write counters, and fleet/node/observer clock skew. This documents their parameters, empty/error behavior and response root shapes. The known-gap list falls from 24 to 18. Descriptions preserve current behavior, including unknown/empty area fallback, the PacketStore cache ratio, stale write-counter snapshots, and the single-node 404. Schemas remain minimal; the existing generator still declares only the generic 200 response, with 404 explained in prose. No runtime handler, authentication, configuration or customizer changes. No added generic response maps. Validation: regression tests exercise the served spec and real router, including missing/malformed data, area selection and fallback, and calibration responses. Full server and ingestor tests pass locally; post-push Chromium verifies the served spec, six routes and Swagger rendering. Three independent reviews passed. Test-only RED `218a3bf3` produced the [six expected documentation failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212159661/job/121225534416). GREEN `1e2d8c08` adds the documentation and [passes CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212678062), including all browser shards and architecture builds. |
||
|
n30nex
|
9f6f318857 |
test(rf-health): wait for airtime badges after grid refresh (#2289)
Fixes #2288. The RF Health airtime test now waits for the requested observer's actual airtime badge. Waiting for any tile and then immediately reading the badge could race with a theme refresh that clears and rebuilds the grid. This is a test-only synchronization correction: two lines replaced, no application changes, and the existing value/warning assertions remain unchanged. No sleeps, timeout increases, skips or retries were added. - Existing RED evidence: [upstream-compatible CI failure](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190851550/job/121161457287). The test blob is identical on upstream `cb38ac35` and that PR's head. - A controlled local reproduction holds the next summary response during a real theme refresh. The old immediate read fails; the locator read waits for the correct badge. - Four local browser runs passed, including an independent parent run. Wrong-value and missing-warning mutations still fail the original assertion. Test inventory passed. - Browser verified: http://127.0.0.1:18851 on an isolated Go fixture. Privacy and whitespace checks passed. The external home-directory preflight bundle is unavailable; applicable checks were performed locally. Independent reviews found no outstanding fixes. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40193032972) passed on `36d1bf4a`, including all three browser shards and both architecture builds. |
||
|
n30nex
|
23f938d1d4 |
docs: consolidate historical release notes (#2313)
Fixes #2311. Partial follow-up to #1385 (PR-B). Five historical release notes lived outside `docs/release-notes/`. Move v2.6.0, v3.0.0, v3.1.0, v3.4.0 and v3.6.0 into that directory using its existing version naming convention. Release content is preserved except for concise historical notices and four relative-link repairs. The v3.0.0 notice identifies its obsolete Node rollback instructions; current setup links point to README and DEPLOY. No new documentation hierarchy or index is introduced. Validation: all five moves are one-to-one; every document matches its original bytes after only the listed edits; all 14 local Markdown links resolve; no old-path references remain. Existing external links are unchanged. All 218 standalone frontend suites pass. The unchanged backend tree passed full server/ingestor suites, and the unchanged app passed 12 browser smoke cases. All five published documents were rendered in Chromium; historical notices and repaired link targets were verified. Three independent reviews passed. TDD exemption: pure documentation; no application, test, CI, configuration or deployment behavior changed. No settings/customizer implications. [CI passed](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40230718673); its docs-only classification correctly skipped application jobs. The full-suite results above are local validation. |
||
|
n30nex
|
60615784d9 |
docs: archive historical planning documents (#2321)
Fixes #2318. Partial follow-up to #1385 (PR-A). Move eight historical planning documents from the root into `docs/archive/`, keeping their original filenames and headings. Each gets a brief archived-plan notice linking current README and deployment guidance. Historical proposals, implementation claims and code examples remain intact. Validation: eight one-to-one renames, byte-preserved original content, 16 valid added links, and no inbound or executable references requiring updates. Seven already-absent historical source paths remain plain text under the archive notice; no replacement links were invented. All other tracked files are unchanged. All 218 standalone suites passed locally. The unchanged backend tree passed full server/ingestor tests, and the unchanged app passed 12 browser smoke cases. TDD exemption: pure documentation; no test, application, configuration or deployment change. No customizer implications. All eight published documents and their guidance links were verified in Chromium. Three independent reviews passed. [CI passed](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40238488019); its docs-only classification correctly skipped application jobs. The full-suite results above are local validation. This only addresses PR-A; other cleanup remains tracked separately. |
||
|
n30nex
|
4ea80da234 |
ci: run JavaScript unit tests before backend validation (#2322)
Fixes #2317. Partial follow-up to #1995. JavaScript assertion failures currently reach their unit gate only after the Go validation work. Move the existing `Run JS unit tests` step directly after Go setup, before gofmt/vet and backend builds/tests. Failed unit checks then stop later work under the existing default failure behavior. This is an exact three-line block relocation within the same job. Its command remains `sh test-all.sh`. All ten jobs, twenty step bodies/options/conditions, other step ordering, dependencies, security checks, coverage outputs and publication gates are unchanged. No new job, package, timeout, cache or application setting. Validation: dependency audit found no need for prior Go outputs. All 218 standalone suites passed with no checkout build artifacts or external Node packages, after earlier shared Go tests had finished and cleaned up. The parent reran the suite, parsed-workflow equivalence checks and browser smoke. The unchanged backend tree passed both full backend suites. TDD exemption: configuration-only; no test files modified, and CI is green without test edits. Three independent reviews passed. [Final-head CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40238885639) confirms the unit step precedes Go validation; backend checks, all browser shards and architecture builds passed. No numerical speedup or successful-pipeline duration claim is made. |
||
|
efitenandClaude Opus 5.5
|
e149568c6c |
feat(observer-logins): personal MQTT logins for observers (#2306)
Observers publish to the broker with one shared MQTT login. Anyone who
has that login can publish under any observer's topic, and the ingestor
takes the observer id from the topic (`cmd/ingestor/main.go:1009`), so
the data in the analyzer DB is only as trustworthy as the shared
password. This PR gives each user a personal MQTT login, generated and
mailed by CoreScope, and lets the broker ask CoreScope whether a login
may connect and publish.
Closes #2319.
The feature is optional and off by default
(`userManagement.observerLogins.enabled`). It needs
`userManagement.enabled`. With it off, nothing changes.
## Changes in this PR
**Broker hooks.** `POST /api/mqtt-hook/emqx/authn` and `/authz`, for the
EMQX HTTP authenticator and authorizer.
- They check a shared secret header, `X-CoreScope-Hook-Secret`, in
constant time.
- A username CoreScope does not know gets `ignore`, so EMQX falls
through to its built-in database and existing users keep working.
- A personal login may only publish `meshcore/<iata>/<pubkey>/<type>`,
with a 64-hex pubkey that is unbound or bound to the same user.
- The first publish for an unbound pubkey binds that observer to the
user (first come wins), until the user owns 20 observers (admin
assignments count too).
- An admin can rebind any observer.
**Credentials.**
- Username `obs_` + 12 random `[a-z0-9]`, password 32 random
`[A-Za-z0-9]`, both from `crypto/rand`.
- The password is stored twice: SHA-256 for the auth check, and sealed
with AES-256-GCM (key from config or `CORESCOPE_OBSERVER_KEY`) so the
user can ask for the same credentials again.
- The UI and the API never show the password. It leaves the server only
in the mail.
**Mail.** One mail with four formats:
- the meshcoretomqtt TOML;
- the observer firmware CLI commands (Gessaman or Dutch-MeshCore, same
`mqttN` CLI), on slot 4: slots 1-2 (Gessaman, Let's Mesh US/EU) and 1-3
(Dutch-MeshCore, `MQTTDefaults.h:19-34`) hold default broker presets on
a fresh install, slot 4 is empty on both. `set mqtt4.server` gets the
bare host: the firmware picks the scheme from the port
(`MQTTBridge.cpp:2094-2145`), and builds before merge 32dd8ad
(2026-10-19) do not accept a scheme there (read from the source, not
tested on a device). A scheme is written only when the port would pick
the wrong one, for example TLS on 8884;
- the Home Assistant meshcore-ha broker form fields;
- the plain values.
Each block starts with a numbered title and one line on where it goes.
The account page can resend the mail, at most 3 times per 24 h.
**Account and admin UI.**
- The account page gets an "Observer MQTT login" section
(`#/account?section=observers`) with the username, the bound observers
and the resend button.
- The admin page gets an "Observer logins" tab
(`#/admin?tab=observer-logins`) that lists logins with revoke and
reissue, and every observer with its owner, "not migrated" or
"orphaned", with a reassign control.
**Notifications.** Creating a login adds `observer.offline.own` ("My
observers go offline") to the owner's notification events; mails go out
once the user has notifications switched on, and the user can untick it.
**Storage.** users.db schema v7 adds `mqtt_logins` and
`observer_bindings`. The server still writes only users.db through
`internal/users`, and the analyzer DB stays read-only.
**Docs.** `docs/observer-mqtt-logins.md` covers the EMQX setup (authn
and authz chains, the hook body templates, the closing `^obs_` deny rule
and why it is needed), how to enable it on a running EMQX through the
REST API step by step, with a check and a rollback per step, and how to
migrate observers off a shared login. `config.example.json` has the
block with placeholder values.
## Enabling it on an instance
Full steps in `docs/observer-mqtt-logins.md`. In short:
1. **CoreScope:** add `userManagement.observerLogins` to `config.json`
with `enabled: true`, a `hookSecret` (`openssl rand -hex 32`), an
`encryptionKey` (`openssl rand -base64 32`) and the `broker` host, port,
transport and TLS that observers should use. Restart. Keep the key safe:
without it, mails cannot be resent and logins must be reissued.
2. **EMQX, authorization:** add a file source with `{deny, {username,
{re, "^obs_"}}, all, ["#"]}.` and move it to the end, then add the HTTP
authorizer pointing at `/api/mqtt-hook/emqx/authz` (it lands at the
front).
3. **EMQX, authentication:** add the HTTP authenticator pointing at
`/api/mqtt-hook/emqx/authn` and move it to the front, before the
built-in database.
4. **Check:** the shared login still connects, an unknown `obs_` login
is refused, and `metrics.deny` on the authorizer stays 0 for existing
clients.
5. **Owners:** they request a login on their account page and paste the
block for their uploader. Each observer binds to them on its first
publish. An uploader that was refused before step 3 may need a restart.
6. **Later:** remove the shared login from the broker once enough
observers moved.
## Who can claim an observer
Any active account can request a login, and its first publish under an
unbound pubkey binds that observer, also when CoreScope already knew it
from the shared login (first come wins, a deliberate choice: owners
migrate without waiting for an admin). Observer pubkeys are public, and
registration cannot be closed, so with this feature on anyone with a
mail address can claim up to 20 unbound observers per account. Before,
publishing needed the shared password.
What limits it: each claim of a known observer mails the admins, an
admin can unbind or reassign it and revoke the login, and a bound
observer cannot be claimed by another login. The doc spells this out and
suggests assigning known observers to their owners before announcing the
feature. A stricter mode (admin approval of logins, or no auto-bind for
known observers) is not in this PR.
## Performance
`authorize()` runs for every publish that misses the EMQX authz cache.
It takes two short read locks, does map lookups and one topic split.
`BenchmarkAuthorizeBound` (2000 observers, 500 logins): 533.5 ns/op, 96
B/op, 2 allocs/op. The admin observers table renders once per load and
updates single rows after a change.
## Verification
- All 16 Go modules: `bash scripts/allmod.sh test ./...` green.
- Skipped `TestSaveGeoFilterPreservesFileMode` and
`TestWriteStatsAtomic_SymlinkAtDestIsReplaced`: both fail on Windows on
a clean master too.
- `sh test-all.sh` green.
- `tests/e2e/test-user-management-e2e.js` 36/36 locally. It covers:
request, mail, resend, revoke, reissue, reassign, no password in the
page or API body, and axe. The CI job for this suite gets the
`observerLogins` block with fake values in `deploy.yml`.
- Local EMQX 5.8.6 in Docker against a local CoreScope:
- a personal login connects and publishes, which binds the observer; a
second personal login is denied on that observer's topic, and subscribe
is denied;
- with CoreScope stopped, the built-in users still connect and publish,
and new personal-login connects are refused;
- the `^obs_` deny rule blocks an already-connected personal login while
CoreScope is down; without the rule, the same publish was delivered.
- Production: running on an EMQX 5.8.9 broker with about 50 clients
since 2026-10-25, set up through the REST API as in the doc. After the
hooks went in, the authorizer answered `ignore` for every existing login
(0 denies, 0 dropped messages), and 4 observers were bound to their
owners through personal logins on the same evening, with the shared
login still in use by the rest.
## Known behaviour
EMQX's HTTP client can mark the hook resource down for one health-check
interval (15 s measured) when CoreScope closes an idle keep-alive
connection just as a health check runs. During that window new
personal-login connects are refused and the uploader reconnects. I saw
it 3 times in about 25 minutes of runs with a forced 1-5 s idle timeout,
and could not trigger it on demand. EMQX 5.8.6 has no setting for it,
and 5.10.5 ships the same client code. The doc describes it and the
alarm to watch for.
## Not in this PR
- mosquitto (mosquitto-go-auth). EMQX only.
- A websocket path field: the mail does not carry one, the doc tells
operators to match the listener path.
- Disabling the shared login. That is an operator step after migration,
described in the doc.
- Tested on EMQX 5.8.6 (Docker) and 5.8.9 (production) only.
Note on
|
||
|
efitenandClaude Opus 5.5
|
11ce123505 |
fix(rx-coverage): English tooltips for the leaderboard score and cells columns (#2280)
## Problem The RX coverage leaderboard has two column tooltips in Dutch, since #1728 (`public/rx-coverage.js:340` and `:342`). The rest of the UI is English. ## Change - **score:** now says what `cmd/server/rx_dashboard.go:362-366` computes. Each ~150 m cell (`leaderboardHexRes = 13`, `rx_dashboard.go:264`) adds 1 divided by the number of observers that heard something there, so an unshared cell counts 1 and a cell shared with 3 others counts 0.25. - **cells:** a plain translation. No logic changes, so there is no new test. `sh test-all.sh` passes. Not done: no browser check of the tooltips. Noticed in the review of #2269, which had picked the Dutch text up as an English key. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
n30nex
|
cb38ac356c |
fix(packets): preserve All time across reloads and links (#2279)
Desktop All time now survives reloads and shared links, including a
selected older grouped packet and observation. An explicit zero is
retained in storage and the URL; missing/invalid values still use the
default. Mobile requests retain their existing time and row limits.
Fixes #2231.
Validation: RED reproduces the lost zero; all standalone frontend suites
and the unchanged server suite pass. The local main browser suite
(`test-e2e-playwright.js`) passes 149 checks with three explicit
fixture-dependent skips. The gesture suite also preserves the existing
time window when opening a packet trace. The new regression covers
selection, reload, fresh links, older grouped observations and mobile
bounds.
Normalization is constant work; URL updates reuse the existing
serializer and add no requests. No new settings or dependencies. Common
ingestor validation is covered by the separate #2272 test-stability
change.
[Exact-head
CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40160982457)
passed on commit
|
||
|
n30nex
|
b3367f5aea |
fix(analytics): keep distance indexing lazy during startup (#2278)
Chunked startup now leaves the distance index lazy, as the legacy loader already does. Requests wait for the initial chunks to finish, then trigger one build and publish the refreshed result. A terminal initial-load failure returns a clear error instead of retrying forever; a completed hot window remains usable if later background filling fails. Fixes #2227. Validation: RED reproduces eager and premature builds plus the terminal-error retry loop. Full server and unchanged frontend suites pass, with focused race checks. The browser/API check observed 202 then 200, nonempty distances, and exactly one build. Performance evidence is the test-enforced absence of eager/duplicate distance construction, with required path indexes retained. No timing speedup is claimed. No new settings. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
n30nex
|
77c0392947 |
fix(reach): avoid caching interrupted query results (#2277)
Reach now propagates failed scope lookups, rejects cancelled results before caching, and retains the prior complete degree snapshot when scanning or row iteration fails. Missing rows and optional scope columns still produce the legitimate empty result. Fixes #2258. Validation: RED reproduces partial degree snapshots and cached success after failed/cancelled follow-up queries. The full server and unchanged frontend suites pass; focused race and browser Reach checks pass. Handler regressions verify that failed builds do not replace caches and that a later request rebuilds successfully. The shared-build cancellation policy and query count are unchanged. This is independent of #2249's admission/queue work. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
n30nex
|
c085ca8cba |
fix(rf-params): respect node visibility in responses (#2276)
The RF endpoint now applies the current blacklist and hidden-name policy to both exposed node metadata and deviations, including after cache warm-up. Unhiding a node restores its data without waiting for cache expiry. Fixes #2268. The shared RF snapshot and peer calculation remain unchanged; filtering copies only the response data. This adds a linear node pass and bounded deviation copies, with no additional database calls or cache invalidation. The feature remains opt-in. Validation: RED reproduces both response leaks. The full server and standalone frontend suites pass, as do focused race tests and the RF browser suite. Regressions cover warm/cold caches, policy updates, unhide, visible controls and default-off behavior. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
n30nex
|
82aafb53c3 |
fix(analytics): count roles within selected filters (#2275)
Analytics Nodes now counts roles from the same complete, filtered node list used by Network Status. Region/area selections no longer display the API's global role totals. The global API metadata remains available to other consumers. Fixes #2213. Validation: RED reproduces the mismatching totals; all standalone frontend suites and the unchanged server suite pass. Browser checks cover region and area changes and empty results. Regressions also cover 501-node pagination, unknown roles, unchanged metadata and request counts. Counting happens inside the existing O(n) status pass, using six bounded counters and no additional requests. No configuration changes. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
n30nex
|
7ebbed58dc |
fix(analytics): anchor resize handles to table headers (#2274)
Resize handles now use their static table header as the containing block. Already-positioned headers, including sticky Nodes headers, retain their existing positioning. Fixes #2217. Validation: assertion-failing RED commit followed by the fix; all standalone frontend suites and the unchanged server suite pass. The browser suite passes 48 checks covering desktop/mobile bounds, real dragging, saved widths and sticky headers. The browser regression waits for the initial theme render before measuring. The fix adds one positioning check per header during setup, with no new requests, dependencies or settings. Common ingestor validation is covered by the separate #2272 test-stability change. |
||
|
n30nex
|
12e0a40a82 |
test: make hop coverage clock checks deterministic (#2273)
Three hop-coverage tests sampled the clock before database setup, then allowed only five seconds of drift. Slow setup made otherwise correct coverage checks fail. They now use the existing clock hook, restore it after fixture workers finish, and assert exact equality. Production code is unchanged. Fixes #2272. Validation: the original full run reproduced all three failures; the corrected full ingestor suite passes. Focused race checks pass, and row/deletion/ordering assertions remain intact. This is a test-only stabilization of existing failing assertions, not a new runtime feature. |
||
|
efitenandClaude Opus 5.5
|
7fa86fe1c1 |
docs(release-notes): v3.15.0 (#2263)
Release notes for v3.15.0: `docs/release-notes/v3.15.0.md` and the `CHANGELOG.md` entry. The release workflow uses the notes file as the GitHub release body. ## What is in it - 77 commits since v3.14.0 (76 merged pull requests plus this one): 27 fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs, 1 chore. Nineteen feats, so minor. - "Read this before upgrading" first. It covers: - the liveness change (#2261, #2262); - the `users.db` v6 migration and what it means for a rollback (#2174); - the backfill at every ingestor start (#2133) and the 24 h planner-statistics refresh (#2158); - the caches that trade freshness for latency (#2165, #2222, #2147, #2156); - changes in hop names and region placement (#2188, #2190, #2234, #2245, #2220); - the role thresholds on favorites (#2215); - the geo-filter save on a single-file mount (#2137); - the `/api/stats` polling in hidden tabs (#2162). - Features, fixes, performance, then tests/CI/docs. - 29 issues closed, taken from each pull request's `closingIssuesReferences`. - Contributors: pull request authors and issue reporters with counts. The commit co-author trailers only name the pull request's own author or tooling. ## How it was checked - I read every pull request body. Every number in the notes comes from a pull request body. - I checked the config keys and defaults against `config.example.json` on `2bb000f2`. - I checked the UNASKED and opt-in claims for #2125, #2133, #2235 and #2239 against the source. - For the other pull requests I relied on their bodies and did not open the diffs. ## Not covered - The uncertain count is a single count on one staging instance on the day of the change (1,682 repeaters and rooms). It is not a before/after comparison. - #2133 gives no measured cost for its backfill on a large database, and the notes say so. ## After merge Tag `v3.15.0` on the merge commit once its `:edge` image exists. Then check in the registry that `v3.15.0`, `v3.15`, `v3` and `latest` share one digest, and that the release has the notes as its body and both `corescope-decrypt` binaries. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>v3.15.0 |
||
|
efitenandClaude Opus 5.5
|
2bb000f2aa |
feat(notify): adaptive offline threshold and certain node analytics (#2260) (#2262)
Part 2 of #2260. **Builds on #2261** (certain liveness): review/merge that first; until then this diff also shows its commit. ## Adaptive offline threshold `userManagement.notifications.adaptiveOffline` — **off by default**: ```json "notifications": { "enabled": true, "adaptiveOffline": { "enabled": true, "floorMinutes": 30, "minObservers": 3 } } ``` The fixed `infraSilentHours` (72 h) treats every repeater the same. With this on, a watched node's `node.offline` window is learned from its own **certain** traffic (the #2261 rule): 1. Certain packets per hour over the week **before its last certain packet**. Ending there, not at now, keeps an outage from dragging the rate down until the fixed window returns and the node reads "back online". 2. Empty runs are classified: ≤ 6 h and recurring on another day at the same hours = normal silence (kept; the window is never shorter than the longest + 1 h); any other run = outage, left out of the learning so it neither stretches the window nor blocks learning. 3. λ = 10th percentile of the remaining hourly counts, so quiet nights don't alert. 4. Window = 14/λ hours: silence at rate λ has a < 10⁻⁶ chance (Poisson, ln 10⁶ ≈ 13.8). 5. Clamped to [`floorMinutes`, the role's silent window]. Kept on the fixed window: λ < 1/hour, under 24 h of history in memory, or a 10th-percentile hour heard by fewer than `minObservers` observers (one observer dropping out must not read as the node going offline). Silence is measured up to the newest packet in the store, so a stalling feed stops every node's clock. A node stored offline comes back only on evidence from the moment it went offline on. The notifier caches each learned window for 10 minutes and drops unwatched nodes. For the #2260 repeater (~840 packets/day) the mail would have gone out about 30 minutes into the outage, instead of never (or 72 h with #2261 alone). ## Certain node analytics - Each `activityTimeline` bucket carries `certain`: the part that can only be this node (`certainFor`, the packet-level form of the `certainHeard` rule). `TimeBucket.Certain` is `omitempty`, so other endpoints are unchanged. - Availability and longest silence use certain buckets only; the longest silence now includes one **still running** (an outage in progress is the silence worth seeing). - The chart stacks certain (`--accent`) and uncertain (`--text-muted`) — existing theme variables, so no new customizer entry. Descriptions updated. ## Performance `CertainActivity` walks a week of packets per watched node, at most once per node per 10 minutes, with the read lock taken **per node**: ``` BenchmarkCertainActivity-16 12.1 ms/op (50 watched repeaters × 6,000 packets; ~0.25 ms per lock hold) BenchmarkNoteCertainRelays-16 121 ns/op 0 allocs ``` Node analytics adds one `certainFor` per packet of the requested node (a prefix-map lookup, or the cached decoded JSON for its own packets). ## Tests - `adaptiveOfflineAfter`: busy repeater hits the floor; 5/hour; quiet nights set the rate; > 10 % empty hours, short history, too few observers → fixed. - `CertainActivity` ends at the last certain packet and counts only certain packets — verified red by mutating the window to end at now. - Notifier end to end: learned window passed → bad; inside → good; too few observers / setting off → fixed 72 h; cache refresh and prune. - `certainFor`, `longestSilence` (incl. running silence), config defaults, `/api/nodes/{pk}/analytics` splits a shared-prefix bucket. - Frontend: `NodeAnalyticsTimeline.series` (new `tests/unit/test-node-analytics-certain-timeline.js`). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork. ## Validation Server on the freshened e2e fixture: analytics buckets carry `certain` (e.g. 89 of 90 packets certain for one repeater). **No browser screenshot**: Playwright isn't installed on this machine. ## Later (rule 8) `floorMinutes` / `minObservers` are server config, not theme settings; an admin UI for notification settings would be the place, not the customizer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
a08f2f4e7c |
fix(liveness): count only evidence that can be the node alone (#2260) (#2261)
Fixes #2260 (part 1 of 2: liveness; the adaptive offline threshold follows in a separate PR). ## Problem A repeater silent since 2026-10-18 showed **Active — Last relayed 4m ago** and **Last Heard 36m**, and its offline mail never fired. Every hit after that date was the 1-byte hop `67`, shared by nine nodes: - the live resolver's affinity tiebreak (`resolvePathForObs`) put those transmissions in `byNode`, which fed `stats.lastHeard` and `LastHeardMap`; - `last_relayed` folds in the 1-byte prefix bucket on purpose (#662), and drove the status in `roles.js` and the notify offline check. ## Fix The store keeps `certainHeard` per node: the newest transmission that can only be that node. | Evidence | Counts? | |---|---| | Own packet (`pubKey`, `srcPubKey`) | ✅ | | Flood relay hop whose prefix, at the packet's path hash size, matches exactly one path-capable node | ✅ | | 1-byte hop shared with other repeaters | ❌ | | Direct-route path (hops still to come, `Mesh::removeSelfFromPath`) | ❌ | | TRACE (path bytes are SNR) | ❌ | | Destination (`destPubKey`) | ❌ | A multibyte repeater legitimately appears as a 1-byte hop (the originator picks the hash size, `Mesh::routeRecvPacket` appends at that size), so only uniqueness decides. The rule is applied in every load path through `indexResolvedPathHops` and the new `indexColdLoadRelayHops`: Load, loadChunk, scanAndMergeChunk, live ingest, late observations. Liveness reads only that: - `/api/nodes` and the node detail: `last_heard` is the newer of `last_seen` and the certain last heard. Before, it was a plain copy of `last_seen`. - health / bulk-health `stats.lastHeard`. - `LastHeardMap` → notify offline check. The separate relay input (`lastRelayed`, `relayTimes`, `notifyInput.Relayed`) is removed. - `roles.js getEffectiveHeardMs`: newest known timestamp, without `last_relayed`. The side pane no longer says "Last relayed"; the Last Relayed row stays (analytics) with an updated tooltip. - New status **`uncertain`** (repeaters and rooms): nothing certain within the silent window, but the shared prefix (`last_relayed`) still in paths. Grey badge with explanation, an *Uncertain* filter on the nodes list and the map, dimmed like stale. The offline mail for such a node says only its shared prefix was still seen. - A prefix also held by a path-capable node in `inactive_nodes` is not unique; adverts with `signatureValid: false` are no evidence; `/api/nodes/search` reports the same `last_heard`. Attribution for analytics (`byNode`, packet counts, `last_relayed`, relay counters, usefulness) is unchanged. ## Performance Hot path: `noteCertainRelays` runs per observation on ingest under the write lock: two hex bytes of header parsing plus one prefix-map lookup per resolved hop. ``` BenchmarkNoteCertainRelays-16 10307640 116.2 ns/op 0 B/op 0 allocs/op (8-hop 2-byte path, 2,000 repeaters) BenchmarkLastHeardMap-16 1902 0.77 ms/op (5,000 nodes; was 3.7 ms scanning byNode) ``` Memory: one string per node ever certainly heard, dropped together with the node's `byNode` list on eviction. ## Tests - `certain_heard_loadpaths_test.go`: every load path (Load, loadChunk, LoadChunked — each with and without resolved_path — live ingest, late observations) feeds `certainHeard`; each of the 8 call sites verified red with a nil prefix map. - `certain_heard_test.go`: hop width per route/payload type; retired namesake prefix; failed signature; uncertain offline mail; shared vs unique 1-byte vs 2-byte hops; direct/TRACE/nil prefix map; cold-load path; sender vs destination; eviction; `lastHeardValue`; `/api/nodes`, detail and health agree. - Regression for #2260: last certain packet on day X, 20 shared-prefix hops after it → `LastHeardMap` stays at X and `node.offline` goes bad after 72 h. Verified red with the uniqueness check mutated out. - Notify: `TestEvaluateCertainHeard` replaces the relay-input test. - Frontend: `last_relayed` alone no longer keeps a repeater or room active; newest timestamp wins (an old `_liveSeen` can't hide a fresh certain relay). - `cmd/server` `go test ./...` and `test-all.sh` pass locally, also on a trial merge into a downstream fork with 457 extra commits. ## Validation Ran the server on the freshened e2e fixture: list, detail and health report the same last heard; e.g. a repeater whose `last_relayed` is 3 min newer than anything certain keeps its certain time. **No browser screenshot**: Playwright isn't installed on this machine; CI's E2E covers the pages. ## Not changed - The ingestor's relay touch of `nodes.last_seen` (#1854) resolves a shared hop only when exactly one candidate is adjacent in the neighbor graph; it did not fire in this case. Left as is. - The `lastHeard=` list filter still filters `last_seen` in SQL. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
AdamandClaude Opus 5.5
|
dbcdca9dc5 |
feat(region-filter): honor ?region= in the URL on every page (#2255) (#2256)
Fixes #2255. Only the packets page read `?region=` from the hash, with its own one-off code. Live, nodes, observers, channels, analytics and scope audit ignored it, so a link like this opened with whatever region the visitor last picked: ``` https://<instance>/#/live?region=SJC ``` We link into our instance from a community directory (one link per region), and wanted those links to land on the live map or node list already filtered, not only on packets. ## What changes **`public/region-filter.js`** - `?region=` (comma-separated, trimmed) overrides the saved selection and is saved like a click in the control, the same as packets did through `setSelected()`. - **Fresh load:** it's applied when the script runs, before any page reads the selection. This matters for live, which fetches nodes before it calls `RegionFilter.init()`. - **In-app navigation:** `init()` applies it before its `await`, so a page that loads after `init()` uses it. If it changed the selection, listeners are told once regions load, so a page that already fetched (live) reloads instead of showing the old region's data under the new region's label. - Unknown codes are dropped by the existing cleanup in `fetchRegions()`. No param, or an empty one, leaves the saved selection alone. **`public/packets.js`** - Drops its own `_pendingUrlRegion` handling; `RegionFilter` covers it. - Removes its region and area listeners in `destroy()`. They were registered without a handle and outlived the page: after visiting packets, a region change on any other page reloaded packets and `replaceState`'d the URL to `#/packets?...` while you were on another page. That already happened with the dropdown on master; the in-app notify above would make it happen on every region link. ## Tests `tests/unit/test-region-filter-url-param.js`, 14 assertions: - URL region overrides the saved selection; comma lists, whitespace, unknown codes, empty param, no param, no hash - fresh load: selection is set before `init()` is called, and `init()` doesn't notify - in-app navigation: a changed region notifies listeners once, the same region doesn't - selection is saved to `localStorage` - packets keeps handles for both listeners and removes them in `destroy()` With `region-filter.js` from master, 5 of the first 8 assertions fail. How I checked: - `test-all.sh` suites all pass except `test-issue-1956-release-routing.js` and `test-preflight-xss-gate.js`. Both also fail on unmodified master on macOS: they shell out to bash-4 features (`mapfile`, and a syntax error in `check-xss-sinks.sh` under bash 3.2). - `eslint@8.57.1` on the two files: 0 errors, 3 warnings, the same as master. - Browser: served `public/` locally with `/api` proxied to a live instance and checked the requests each page made: - fresh load of `#/nodes?region=AUS`, `#/live?region=DFW`, `#/packets?region=HOU`: the first request carries the URL's region and the dropdown shows it - packets → `#/live?region=DFW`: live fetches with the old region, then reloads with DFW; the dropdown shows DFW - packets → `#/nodes?region=AUS`: no packets requests, and the URL stays on `#/nodes` I couldn't run the Playwright E2E suite locally (no Go toolchain on this machine). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
efitenandClaude Opus 5.5
|
0e3ce48cff |
feat(nodes): flag nodes whose LoRa settings differ from nearby nodes (#1268) (#2210)
Refs #1268. #2209 is merged as `cab4f71c`, and master is merged into this branch (`47ce8264`), so the diff against master is this PR's own change: `ea4d1130` plus the fix in `5cbbe5d5`. ## What it does Two nodes hear each other only when SF and bandwidth match and their frequencies are close enough. `/api/rf-params` now also returns `deviations`: nodes whose setting differs from a majority of the nodes around them on the same band. - **Peers:** nodes with GPS and RF params within `rfParams.compareRadiusKm` (default 50) whose frequency is within 30 MHz. - **Same channel:** frequencies within a quarter of the bandwidth count as one channel (15.625 kHz at BW 62.5). The SX1262 tolerates a transmitter/receiver offset of up to 25% of BW without sensitivity loss, for SF5 to SF12 and all bandwidths. Source: the `FERR_L` row in the [S62F module datasheet](https://lora-alliance.org/wp-content/uploads/2020/07/S62F_SPEC_Ver.B.pdf), an SX1262 module; I did not find the figure in a Semtech document itself. That table also lists -50/-100/-200 ppm for SF12/SF11/SF10, which at 869 MHz is wider than BW/4 at 62.5 and 125 kHz, so BW/4 is the tighter bound there. - **Flagged when:** a setting the node cannot hear is shared by more than half of the group, the node itself counted, with at least 3 peers and no other incompatible setting at the same support. So 5 against 5 flags nobody, and 2 of 4 does not count as a majority. Each deviation carries the majority, how many peers are compatible with it, the peer count, and which of freq/sf/bw differ. - **Coding rate is not compared.** RadioLib starts SX126x radios with an explicit LoRa header (`SX126x::begin` sets `RADIOLIB_SX126X_LORA_HEADER_EXPLICIT` at `6d89348`, the commit the firmware pins in `platformio.ini`), and nothing in `firmware/src` switches to implicit. The header carries the CR, so a CR mismatch still decodes. I checked SX126x only, not SX127x or LR11x0. UI: - Nodes list: a badge such as `LoRa SF7` next to the name of a deviating node (the majority in its tooltip), and a `LoRa: differs from nearby (N)` filter kept in the URL as `?lora=differs`. - Node detail: one line, for example "Differs from 12 of 14 nearby nodes on this band, which use 869.618 MHz · SF8 · BW 62.5 kHz". - Badge and filter only exist while the server has map data. ## Why same-band only On our instance (1,110 nodes with params and GPS, measured with the first version of the rule), a plain neighbourhood majority flagged exactly 15 nodes, all of them a deliberate 433 MHz network in Limburg next to the 868 MHz one (names like `NL-ZAY-RP01 | 433 X 868`). The same-band comparison flags none there. On our network this PR shows nothing today. The flags are for instances where a node really is set to a different SF, bandwidth or channel than the nodes around it. ## Cost - Points are bucketed on a radius-sized grid (rows of radius/110.57 degrees, longitude columns widened by 1/cos(lat)) and compared with a squared equirectangular distance. - `BenchmarkRfDeviations1k` (1,100 nodes over Belgium and the southern Netherlands): 4.5 ms. `BenchmarkRfDeviations10k` (10,000 nodes in the same box): 388 ms, 1.3 MB. Both on the first version. - After `5cbbe5d5`, on a slower machine, 3 runs each: 1k 4.4 to 9.5 ms, 10k 821 to 852 ms, 1.8 MB. On that machine the first version measured 7.6 to 9.3 ms and 819 to 861 ms, so the majority step adds no measurable time. It is one pass over the distinct settings around a node, and those are few (one per preset in use). - Inside the radius the work is close to O(n²): 10,000 nodes in one dense box means about 2,000 neighbours per node, and a request after the 5 minute TTL waits for that computation. - The result is cached per map snapshot for 5 minutes; concurrent requests wait for one computation. - On staging `/api/perf` reports `/api/rf-params` at 9.9 ms average, 26 ms max (8 requests). ## Tests - Go: deviation cases (SF, freq within band, BW, CR ignored, other band ignored, no GPS, no params, far away), split vote and too few peers, radius, a peer in another longitude column at 60°N, the cache (same snapshot within TTL, new snapshot, after TTL), the route with four inserted nodes, plus the two benchmarks. - Go, added in `5cbbe5d5`: a 5/5 split and 2 of 4 flag nobody; a 1 and a 15 kHz offset are not deviations while 32 kHz is, and offset peers add up into one majority; the BW/4 boundary; peers two grid columns away at 60°N; a peer at the north edge of the radius. Each fails on the first version. The two-column test also fails with a ±1 column search, which the existing longitude test did not catch. - Full `cmd/server` suite and `test-all.sh` pass locally on `5cbbe5d5`. - Frontend unit: 6 new assertions for `deviationBadge` and the detail line (13 in the file). - E2E: 4 new checks: badge only on the deviating node, filter count and result, `?lora=differs` written and restored, detail line. The disabled check now also asserts no filter and no badges. 8/8 locally. ## Validation On staging with live map data, with the first version: 1,115 nodes served, `deviations: {}`, and the filter shows `LoRa: differs from nearby (0)` with no badges. Because no real deviation exists there, the badge, filter and detail line were checked by serving one simulated deviation through Playwright route interception against the staging page. ## Not done - No map marker for deviating nodes. - `compareRadiusKm` is config-only; the customizer is a later step. The other values are hardcoded and belong in that step too: at least 3 peers, the 30 MHz band, the more-than-half rule, BW/4 and the 5 minute TTL. - Not re-run on staging after `5cbbe5d5`, and the E2E suite was not re-run locally. The fix changes only the server's choice of nodes, not the UI. - The detail line says the peers "use" the majority setting. With the BW/4 tolerance, some of them may be a few kHz off it. - Companions and room servers are compared too, not only repeaters. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |