mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-22 22:57:59 +00:00
01d2bbbbfdf7d22d1d48596b0bb21e3b8db3a335
3187
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
01d2bbbbfd |
fix(account): keep Unclaim and Unlink disabled until the list reloads (#2440)
Follow-up to #2430, covering the two points from @efiten's re-review. - **Unclaim and Unlink stay disabled until the list reloads.** Both click handlers now return the list reload from the DELETE's `.then` (`public/account.js`), so the button is re-enabled only after the redrawn list has replaced it. Before, a click between the DELETE's answer and the list GET's answer sent a second DELETE. - **The dropped-from-node-list note shows in every state.** `repeaterHealth` adds "no longer in the node list" whenever `r.inactive` is set, so an inactive repeater that was heard recently or whose prefix still relays is marked too, not only a silent one. Tests (`tests/unit/test-user-management-ui.js`, 103/103): - For both Unclaim and Unlink, the button stays disabled while the list GET after the DELETE is pending, and a click in that window sends no second DELETE. - An inactive repeater carries the note when active, uncertain, stale and never heard, and none of those rows links to a node page. `tests/e2e/test-lint-network-calls.js` 47/47. Co-authored-by: Adam <196335+anieto@users.noreply.github.com> |
||
|
|
9a57b2fca6 |
test(server): set the local zone in a child process, not via time.Local (#2438)
## Summary `TestNodeAnalyticsWindowIgnoresLocalZone` set the zone by assigning the global `time.Local`. Every `time.Now()` reads that variable, so if a timer left over from an earlier test fires during this test, the race detector reports a data race. It counts as a failure of this test, whichever test left the timer. It showed up as an intermittent failure of the server race detector (shard 3/4) on a PR run (#2436). Three local runs of the same shard passed. The code under test is fine; only the test was racy. ## Change The test now re-runs itself in a child process with `TZ=Etc/GMT-2` (UTC+2) and writes nothing global: - **Embedded zone data:** `time/tzdata` is embedded in the test binary, so the zone resolves without the host's zoneinfo. - **Zone check:** the child first checks that its local offset really is +02:00. - **Test name:** the child's `-test.run` pattern is built from `t.Name()`. - **Vacuous pass guarded:** the parent requires `--- PASS: <name>` in the child's output, so a rename can't make it pass without running. - **Windows:** the test is skipped there before the re-exec. Go on Windows takes the local zone from the OS and ignores `TZ`, so the child cannot be put in UTC+2. - **Bounded child:** the child runs under a 60 s context, so a hung child fails with its output. This follows the existing re-exec pattern in `distance_background_fill_test.go`. `TestMain` is just `os.Exit(m.Run())`, so the child runs only this test. It works under the sharded `-test.run` in the race job, and in coverage builds. ## Testing - `go test -race -run TestNodeAnalyticsWindowIgnoresLocalZone`: pass. gofmt and vet are clean. - **Still guards the original bug:** with `time.Now().UTC()` changed back to `time.Now()` in `GetNodeAnalytics`, it fails as before. The 7d window holds 166 of 168 packets, and `From` is `...+02:00`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
49dd4ee773 |
List my claimed repeaters on My account (#2430)
Follow-up to #2415, see #2414. ## Situation A user can claim repeaters, but no page lists the ones they claimed. Claims show on each node page ("Maintained by"), in the admin user detail and in the "Download my data" export. A maintainer of several repeaters has to remember them and open each node page to see whether one went quiet. ## Change **`GET /api/account/repeaters`** (browser session; device tokens answer 403) returns the caller's claims. For each one: - `pubkey` and `name`. - `known`: false once the node left the analyzer database. The claim stays. - `role` and `claimedAt`. - `lastHeard`: the newer of the last advert and the last transmission certainly from the node, the same sources as the node page's Last Heard. - `lastRelayed`: when its hop prefix last relayed. Only used to mark a silent repeater as uncertain, since the prefix can be shared. - `status`: the saved repeater status, or null. - `comaintainers`: the other active claimers' display names. The list is sorted by name, with unnamed repeaters after, by pubkey. It's built on the existing `RepeaterClaimsByUser`, `NotifyNodes` and `LastHeardMap`. **My account, My repeaters**, a new section after Companions: - Silent repeaters come first. Health uses `getNodeStatus`, so the threshold matches the node pages and map (`infraSilentMs`): - green "heard 2h ago"; - yellow "last heard …, its prefix still relays"; - red "silent, last heard 184d ago" or "never heard"; - grey "no longer in the database". - Each row links to the node page and shows the saved status ("no details yet" without one), the claim date and "also maintained by …". - *Unclaim* asks for confirmation, then uses the existing `DELETE /api/repeaters/{pubkey}/claim`. - The empty state says how to claim. - `#/account?section=repeaters` deep-links to the section, like `section=notifications`. Editing stays on the node page, so there's only one editor. The admin user detail and the export are unchanged. Docs: `openapi.go` and the Repeater info section of `docs/user-guide/accounts.md`. ## Notes - A mobile client will want the same list. #2429 proposes device-token scopes for repeater claims. If it lands, letting a `repeater:claim` token read this route is a one-line addition to the bearer route table, and I'd rather do that there than here. - The two PRs touch nearby lines in `openapi.go`, `repeater_handlers.go` (route registration) and `public/account.js`. Whichever merges second needs a small rebase. ## Tests - `cmd/server/my_repeaters_test.go`: - empty list; - ordering (named first, case-insensitive); - unheard, heard and departed nodes; - saved status; - co-maintainers, excluding the caller and disabled users; - another user's view; - unclaim removes the row; - anonymous gets 401 and a device token 403; - no analyzer DB. - `tests/unit/test-user-management-ui.js`: health sort and chips, escaping, the short-pubkey fallback, status and co-maintainer lines, Unclaim on every row, and the empty state. - `tests/e2e/test-user-management-e2e.js`: - the claimed repeater appears with its status and node link; - axe on the section; - Unclaim: dismissing the confirm keeps the claim, accepting empties the list. Run locally: - `make test`: all 16 modules OK. - `sh test-all.sh` and `node tests/e2e/test-lint-network-calls.js`: pass, run in `node:24-bookworm`. - `npx eslint public/*.js`: 0 errors, the same 87 warnings as master. - `test-user-management-e2e.js` against local e2etest servers with the CI config: 46/46. - Manually in Chrome, light and dark: heard, silent and co-maintained rows, and the deep link. Not run: the full Playwright shard matrix from CI. --------- Co-authored-by: Adam <196335+anieto@users.noreply.github.com> |
||
|
|
6c78a1d2c1 |
fix(ingestor): detect rejected SUBACK and retry subscription (#2413)
## Problem `paho.mqtt.golang` does not surface a SUBACK failure return code (`0x80`) via `token.Error()`. The subscribe token completes "successfully", so the client stays connected **with zero subscriptions**. The ingestor logs `subscribed to meshcore/#` and then silently ingests nothing until it is restarted. ### Observed in production (live.saarmesh.de) 1. EMQX was restarted. CoreScope reconnected about 15 s later. 2. EMQX accepts CONNECT before its authorization sources are loaded. With `authorization.no_match = deny`, the SUBSCRIBE sent from `OnConnect` was rejected. 3. The ingestor logged `subscribed to meshcore/#`, but `emqx ctl clients list` showed `subscriptions=0` for the CoreScope client. 4. The liveness watchdog reported `NEVER received a message …`. By design (#1335) it does not force a reconnect for `NeverReceived`, so ingestion stayed dead (about 40 min until a manual container restart). ## Fix - `subscribeOnce`: bounds the wait with a timeout and treats a SUBACK return code `>= 0x80` for the topic as an error. It matches on the `Result() map[string]byte` method that `*mqtt.SubscribeToken` implements, so it can be tested. - `subscribeWithRetry`: on failure, retries in the background with exponential backoff (2 s → 60 s) while the client stays connected. The loop ends on disconnect, and the next `OnConnect` starts a fresh attempt. `OnConnect` itself is never blocked by the retries. - `OnConnect` now calls `subscribeWithRetry` per topic instead of the previous unbounded `token.Wait()` + `token.Error()` check. ## Tests New `subscribe_retry_test.go` with a fake client/token covers: - granted SUBACK → success - rejected SUBACK (`0x80`) → error - token error → error - rejection ×2 then grant → recovers, and retries stop after success - success → exactly one SUBSCRIBE - disconnect during retry → loop stops `go vet` is clean, `gofmt` is clean, and the full `cmd/ingestor` test suite passes. The same change, backported onto v3.17.0, is running on live.saarmesh.de and subscribes cleanly after a restart. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01XfZdhsbrRHCYG2ksxRUo6B --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ca74bc12f9 |
fix(repeater-info): keep an open edit form on screen through a re-render (#2437)
## What On the full node page, an open repeater-info edit form now stays on screen when the page re-renders (`theme-refresh`). Before, the fresh section stayed empty until `GET /api/repeaters/<pk>/info` answered. ## Why `loadFullNode` (`public/nodes.js`) replaces the page body and calls `CSRepeaterInfo.mount` on a fresh element. `mount` kept the typed values in its `drafts` map, but only showed the form again after the GET, so for one round trip the form was gone. That made `tests/e2e/test-user-management-e2e.js` flaky. Every page load fires one `theme-refresh` about 300 ms after `/api/config/theme` resolves. On CI that refresh sometimes landed between filling the form and the `page.evaluate` that marks the notes field, which then threw `Cannot read properties of null (reading 'dataset')`. The three later repeater-info steps failed as a consequence. Since 2026-11-10 this happened in 2 of about 150 CI runs: run 44506140120 (#2435) and run 44187421862 (a docs-only branch). Reproduced locally with `/info` delayed 600 ms and a `theme-refresh` after filling the form: the notes field was missing during the refresh on the old code, and present on this branch (3 of 3 runs). ## Changes `public/repeater-info.js`: - `mount` shows an open draft at once, in the same task as the re-render. The GET only replaces it when the user lost edit rights. - Cancel before the GET answers no longer throws on a null `info`; the section stays hidden until the view can render. - A failed GET leaves an open form in place. - A save on the early form is not overwritten by the older GET answer. - A section the page already replaced ignores its late GET, so it cannot delete the draft its successor owns. `tests/e2e/test-user-management-e2e.js:745`: the side-pane step collapsed whitespace with `/s+/`; now `/\s+/`. ## Tests - `tests/unit/test-repeater-info.js`: 6 new cases, 27 pass. All 6 fail on the old `repeater-info.js`. - `tests/e2e/test-user-management-e2e.js`: 44/44 locally against an `e2etest` build, run twice. - `test-all.sh`: green locally, except the three suites that also fail on master on Windows (`test-manage-workflow.js`, `test-preflight-xss-gate.js`, `test-dev-static.js`). ## Not done - The side pane (`#node-repeater-info-pane`) still has a short gap. On `theme-refresh`, `selectNode` shows "Loading…" before `fetchNodeDetail` and only then mounts. The draft comes back, but the form is off screen during that fetch. Fixing it means changing the pane render path in `nodes.js`, which is outside this PR. - Focus and caret are still lost when the form is rebuilt. That was already the case before this PR. |
||
|
|
87b70562fd |
fix(node-analytics): show silent hours in the activity timeline (#2435)
## Problem
The node analytics activity timeline only shows the hours that hold
packets. `GetNodeAnalytics` sends sparse hourly buckets
(`cmd/server/store.go:10328`) and `buildActivityChart` used them as
category labels, so a silent stretch takes no room on the axis. An 11 h
outage of one repeater collapsed into two neighbouring bars, and two
repeaters that went down 12 h apart looked as if they dropped at the
same moment.
## Change
- `public/node-analytics.js`: `fillTimeline(tl, timeRange)` places every
hour from the start of the covered window (`timeRange.dataFrom` when
partial, else `timeRange.from`) to `timeRange.to` on the axis. Silent
hours get `{count: 0, certain: 0}`, so the certain/uncertain split from
#2260 stacks them as empty. Buckets the server sent are kept as they
are, also one outside the window. A missing or malformed `timeRange`
leaves the timeline unchanged.
- Above 8 days of covered window the hours are summed per UTC day (count
and certain), and the bar is labelled with its date plus "UTC". 337
hourly bars (14 days in memory) are about 1.5 px wide on a 700 px
canvas, so a quiet node would look empty. 7d stays hourly (169 bars).
- Buckets from older ingestors (`"2006-01-02 15:04:05"`, space instead
of T) are normalised so they land on their own hour.
- Exported as `NodeAnalyticsTimeline.fill` next to `series`.
Server stats are not touched: availability counts the hours with certain
packets (`store.go:10540`) and the longest silence is the gap between
them (`store.go:10549`), both correct on the sparse list. Sending zero
buckets from the server would break both.
## Scale
One bar per hour up to 8 days covered (25 for 24h, 169 for 7d), one per
day above it (31 for 30d, 366 for a full year). The fill is one pass
over the hours of the window: 8761 iterations for 365 days.
## Tests
- New `tests/unit/test-node-analytics-timeline-gaps.js` (13 cases, real
`node-analytics.js` in a vm), listed in `test-all.sh`: gaps filled with
zeros, window edges rounded to the hour, partial window starts at
`dataFrom`, empty node gives a full row of zeros, filled buckets go
through `series` as 0/0, bad `timeRange` returns the input,
out-of-window bucket kept, end on the hour included, 7d stays hourly, >8
days sums per day, 365 days gives 366 buckets, legacy space-format
bucket lands on its hour.
- Every suite in `test-all.sh` run one by one: all pass except
`test-manage-workflow.js` and `test-preflight-xss-gate.js`, which fail
the same way on this checkout without the change (Windows line endings /
console encoding).
## Browser check
Local server on the e2e fixture with one transmission moved 6 h back:
the 24h chart shows 10 hourly bars, five of them empty between the lone
early packet and the recent ones. Before the change the same data drew 4
adjacent bars. With the same transmission moved 10 days back, 30d shows
11 day bars labelled `2026-11-09 UTC` to `2026-11-19 UTC`, the lone
early packet visible at the left.
Not done: no E2E test added, no check on a mobile viewport.
|
||
|
|
b2c8693cbc |
ci(e2e): run the axe gate's viewport/theme pairs concurrently (#2433)
Refs #2424. After #2432 the E2E shards are the longest path of a PR run: 9m34s, 11m00s and 10m51s on its run 44394972332. ## Where the E2E time goes From the timestamped logs of that run: 1,655 s of suites over three shards, plus about 1 min of setup per runner. - **`test-a11y-axe-1668.js`: 359 s, 22% of the total, alone on shard 1.** It checks 104 cells (26 routes × 2 themes × 2 viewports) one after another, and each cell waits a fixed 1.5 s for the SPA to render. - **Next largest:** `test-e2e-playwright.js` 153 s, `test-issue-1147-section-order` 89 s, `test-observer-detail-ownership` 88 s, and the slide-over flake gate about 100 s. - **117 suites under 10 s each:** 409 s together, mostly the start of a node process and a browser. ## What changes 1. **The axe gate runs its four (viewport, theme) pairs at the same time.** - Each pair already had its own browser context; the route loop moved unchanged into `runCombo()`, and `main()` awaits the four with `Promise.all`. - Each pair buffers its log lines, which print afterwards in the old order, viewport headers included. - The 1.5 s render wait is unchanged, so axe sees the same page state; the waits only overlap. 2. **Four suites move to shard 1, which gains about 240 s:** - from shard 2: `test-issue-1147-section-order` (89 s) and `test-health-status` (29 s); - from shard 3: `test-observer-detail-ownership` (88 s) and `test-issue-1281-location-row` (25 s). None of them uses the shard-specific servers (`:13582` on shard 3, `:8788`/`:13591` on shard 1). Expected, with the axe gate scaled by the local ratio: about 479 / 468 / 461 s of tests per shard, down from 490 / 586 / 574. ## Checked locally Windows, Chrome, the e2e fixture migrated, server on `:13581`: - **Same output, three times faster:** | version | time | result | |---|---|---| | upstream/master | 282 s | 104 cells, net=0 | | this branch | 92 s | 104 cells, net=0 | The per-cell lines (`✅/❌`, viewport headers, `SUMMARY`) are identical in both runs (`diff` of the grep). - **The failure path still works.** Against a port with no server (`AXE_ROUTES_ONLY=/packets`), all four cells report `❌ net=1`, four screenshots are written, and the exit code is 1. - **Unit tests pass:** `test-test-inventory.js`, the a11y unit tests that load this file (selftest, routes coverage), and `test-issue-1956-release-routing.js`. ## Not checked - **The CI time.** It comes from this PR's run. Four pairs at once on a 4-core runner may speed up less than on my laptop; I'll post the shard times. - **The moved suites, run locally.** Their code is unchanged and they use the shared server on `:13581`. On shard 1 they run after the split-frontend suite instead of their old neighbours. - **The other suites themselves.** `test-e2e-playwright.js` holds 54 `waitForTimeout` calls. Replacing fixed waits with readiness checks would change what each suite waits for, so that is not in this PR. |
||
|
|
58c937c462 |
test(planner): trim the 30 km viewshed equivalence cases (#2431)
Refs #2424. Second step after #2426: `TestViewshedMatchesReference` is the slowest test in the server package. ## What changes Only `cmd/server/planner_viewshed_test.go`. 1. **30 km equivalence cases.** Of the three 30 km terrains (fake, flat, ridge) only the ridge case stays. Up to 30 km the cell size is the 0.2 km floor (`viewshedCellKm` = `max(0.2, r/150)`, `planner_viewshed.go:50`), so the 30 km cases add a bigger grid and longer rays (more samples per profile, `planProfileSamples`), not a different cell size. The ridge case keeps one full-size grid in the regular suite, with level ground west of the ridge and shadow east of it. 2. **Missing-tile case.** The old case could not reach its missing tile. The tile `{31,-98}` starts at lat 31; the 30 km bbox around lat 30.5, expanded by 1 km, ends near lat 30.78. So it was a second copy of the 30 km fake case. It now uses a site at lat 30.95 (0.05 degrees, about 5.6 km, south of the tile edge) with a 12 km range. 3. **New guard `TestViewshedMissingTileCaseReachesTheTile`.** It checks that the case touches the missing tile (`MissingTiles()` = 1 of at least 2), that every in-range cell north of the edge is NoData, and that cells south of it have data. With the old site (lat 30.5) it fails: `MissingTiles() = 0 of 1, want 1 of at least 2`. ## Measured Native, Windows, Go 1.26.1, `go test -count=1 -run '^TestViewshedMatchesReference$' -v .` with a temporary timer per case: | case | before | after | |---|---|---| | fake/30km | 3.49 s | removed | | flat/30km | 3.07 s | removed | | ridge/30km | 2.78 s | kept | | missing-tile | 4.05 s (30 km) | 12 km | | whole test | 17.0 s | 6.2 s | Before this change the test was 16.1 of 101 s for the whole server package in a local `-json` run (16%). `go test -count=1 -run 'TestViewshed|TestPlanner|TestClearance|TestCoverage' .` passes; `gofmt` and `go vet` are clean. ## Not done - Not measured under `-race` and coverage, the way CI runs it. The CI time comes from the hosted run of this PR. - No equivalence case above 30 km, where the cell grows past 0.2 km. The old suite had none either; `TestViewshedCellKm` covers the formula and `BenchmarkViewshedWorkCount` the 60 km counts. - The CI job split (running `-race` in shards next to the coverage run) is a separate PR. |
||
|
|
f4d15f8456 |
ci: run the server race detector in four shards beside go-test (#2432)
Refs #2424. The Go Build & Test job is the longest path of the pipeline: 23.5 min on #2410's last run, against 10 to 13 min for the E2E shards. Of that, the server step (`go test -race -coverprofile`) took 21 min. ## Why it is slow - The server package has 2,364 tests and none call `t.Parallel()`, so they run one after another in one process. - Natively, without `-race`, the whole package takes 101 s locally (Windows, `go test -json`). Under `-race` with coverage on the hosted runner it takes about 18 min. - No single test explains it: 2,151 of the tests take under 0.1 s each. The slowest one, `TestViewshedMatchesReference`, is trimmed separately in #2431. ## What changes - **`go-test`**: the server suite runs with coverage only, no `-race`. Coverage badges and the e2etest step are unchanged. - **New `server-race-shard` matrix, 4 runners.** - Each shard builds the race test binary once (`go test -race -c`) and lists the tests with `-test.list`, keeping `Test`, `Example` and `Fuzz` names, which is what `go test` runs by default. - It runs every fourth name with `-test.run "^(A|B|...)$"`. - Every test still runs under `-race`, in exactly one shard. A shard with no tests fails. - **New `server-race` gate**, built like `e2e-test`: it requires every shard. `build-and-publish` requires the gate. - **Tag runs**: the shards skip on a tag run with `images_published`, the same condition as the E2E shards. The `:edge` commit was already checked. - **`tests/unit/test-issue-1956-release-routing.js`** routes both new jobs. It asserts that a failed shard fails the gate rather than skipping it, and that a failed shard or gate fails `build-and-publish`. ## Checked locally Without `-race`, Windows, test binary from this branch: - The four shard lists hold 591 tests each. Their union is the full list of 2,364 names, with no duplicates. - Each shard ran on its own and passed: 16 s, 25 s, 39 s and 33 s. This shows no test depends on state left behind by a test in another shard, at least without `-race`. - Shard 2 run with `-v` reported exactly 591 top-level results. - Each `-test.run` pattern is about 21 KB, well under Linux's 128 KB per-argument limit. - The routing test passes. ## Not checked - The hosted wall time. The shard times will come from this PR's own run. Each shard compiles the race build of cgo SQLite itself, because the go-build cache that `go-test` saves holds non-race objects. I'll post the measured times in a comment. - Whether branch protection requires specific check names. The API answers 404 for me. If "✅ Go Build & Test" is required, "🏁 Race detector (server)" should be added next to it. Otherwise a race failure turns the run red but does not block a merge. |
||
|
|
0af70cf2c1 |
fix(build): pin musl thread stacks at 8 MiB and check the artifacts (#2410)
## Summary Pins musl thread stacks at 8 MiB on every link, and fixes a crash in **native Alpine builds** (`make build`, `make test`, or `go test` on an Alpine host). **The released images are not affected.** An earlier version of this description said public endpoints could crash the shipped image. That was wrong, and review caught it: see "Who is affected". ## Root cause - musl sizes every non-main thread's stack from the ELF `PT_GNU_STACK` size, with a floor of 128 KiB and a cap of 8 MiB (glibc defaults to 8 MiB). - cgo calls run on the calling thread's stack, and SQLite's `multiSelect` recurses about 850 bytes per `UNION ALL` term. - `lookupPrefixes` (up to 400 terms) and `sharedEightBytePrefixes` (up to 450 terms) overflow a 128 KiB stack and SIGSEGV the whole process. ## Who is affected | Link | `PT_GNU_STACK` without the flag | Thread stack | Wide-UNION tests | |---|---|---|---| | Alpine gcc + GNU ld (`make build` / `make test` / `go test` on Alpine) | `0` | 128 KiB | **SIGSEGV** | | zig 0.16.0, x86_64-linux-musl (Dockerfile, `crossbuild`, release job) | `0x1000000` (16 MiB) | 8 MiB | pass 3/3 | | zig 0.16.0, aarch64-linux-musl | `0x1000000` (16 MiB) | 8 MiB | pass 3/3, run on an arm64 node | So for the released images the flag pins a value that zig's default already provides. It stops being a property of a linker default, and a build check now fails if it is lost. For native Alpine links it is the fix: with the flag those tests pass. Control: a zig arm64 link with `-Wl,-z,stack-size=131072` gets 128 KiB threads and the same tests SIGSEGV in cgo on the arm64 node. ## Fix - Every musl link passes `-Wl,-z,stack-size=8388608`: the `Dockerfile`, the `Makefile` (`build` and `test` on Linux, and `crossbuild`) and the release job in `deploy.yml`. - This reserves address space only. Memory is committed as pages are touched. - **The artifacts are checked.** The Dockerfile's builder stage and the release job's "Verify release binaries" step read `PT_GNU_STACK` with `readelf` and fail unless it is exactly 8 MiB. Verified: the builder stage passes for `linux/amd64` and `linux/arm64`, and fails with `--build-arg STACK_LDFLAG=` (`/corescope-server: PT_GNU_STACK is not 8 MiB`, memsz `0x1000000`). ## Testing - `TestMuslThreadStackWideUnionQueries` (`cmd/server/musl_thread_stack_linux_test.go`, Linux only) runs both queries off the main thread and asserts exact results, so a failed chunk cannot pass as "nothing matched". Under `golang:1.27-alpine` with gcc: SIGSEGV without the flag, pass with it. On CI's glibc runners it passes either way, which is why the artifact checks above exist. - `tests/unit/test-issue-1956-release-routing.js`: the release verify step must check both binaries, and a `0x000000` or `0x1000000` header must fail the release. - `tests/unit/test-dockerfile-buildplatform.js`: all three `go build` lines carry `STACK_LDFLAG`, and the `readelf` check covers all three binaries. - `sh test-all.sh` passes. `go vet` passes for `cmd/server` on Linux with gofmt clean. - Not verified: a Windows `go vet`. The fix there is the `_linux_test.go` file name; no Windows machine was available to run it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Merge order These four PRs were split out of #2396 and are **independent — they can merge in any order.** Verified: every pair is conflict-free, and all four merged together pass `go vet`, `go test ./...` (all 16 modules) and `sh test-all.sh`. (That combined run predates the review fixes on #2409 and #2410.) Suggested order, none of it urgent: 1. **#2408** — never lose rows during startup / a failed poll. 2. **#2407** — explicit `observers.idx` primary key. 3. **#2409** — content-hash migration moved into the ingestor. 4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not affected; it fixes native Alpine builds. After these land, #2396 (Postgres) will be rebased onto master. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: efiten <erwin.fiten@gmail.com> |
||
|
|
7ae6645337 |
fix(#1856): run the content-hash migration in the ingestor (#2409)
## Summary Fixes #1856. The content-hash migration from #787 has done nothing since #1283. It ran in the server on the read-only analyzer handle, so every batch failed. Before #1958 it also reported `hashMigrationComplete: true` anyway. It also meant the server issued `UPDATE` and `DELETE` against a database it must never write. The migration now runs in the ingestor as the async migration `content_hash_payload_bits_v1`. Split out of #2396 at review's request. ## Who is affected Only installs holding transmissions from **before #787 (April 18)** that did not run a server between #787 and #1283 (April 18 to May 18), the only month the old server-side pass could commit. `retention.packetDays` defaults to keep-forever, so long-lived installs can qualify. For everyone else this is a one-time scan that changes nothing. ## What changes - Rows present at boot are rescanned by id in batches of 5000, read outside the writer lock. Stale rows commit 250 per transaction (the retention bound), so ingest waits for one chunk at most, not a whole batch. - After each scan batch that changed rows, a progress line is logged. - A stale hash is rewritten in place. - A row whose new hash already exists is **merged** into that survivor: - its observations move, except any that would violate the dedup index; - the survivor's `first_seen` and `last_seen` widen to cover both; - the duplicate is deleted with the same child cleanup retention uses (now one shared helper). - Collisions are found by lookup, not by catching a constraint error. - The server copy is deleted. `/api/stats.hashMigrationComplete` keeps its shape and now reports the ingestor's `_async_migrations` status. - `readonly_invariant_test.go` now fails on any `Begin` or `INSERT`/`UPDATE`/`DELETE` against the analyzer DB from `cmd/server`. ## Known limitation: a running server does not follow the pass A server that loaded these rows before the pass keeps the old hashes, the merged duplicates, and survivors without their moved observations until it restarts. That's deliberate: - `tx_rewrite_feed` cannot carry a merge. Its rows cascade away with the deleted duplicate, and the server's tail refreshes only `decoded_json` and `scope_name`. - Removing a transmission from every in-memory index is hot-path code that a one-time pass over old rows does not justify. Instead, when the pass changed anything, its summary log line tells the operator to restart servers. The reasoning is recorded at the top of `cmd/ingestor/hash_migrate.go`. (Raised by a reviewer in #2396.) ## Testing - `cmd/ingestor/hash_migrate_test.go`: in-place rewrite; stale rows of one scan batch commit in write-sized chunks (`TestMigrateContentHashesCommitsInWriteChunks`); merge of a colliding pair (observation move, dedup-index conflicts, span widening, child cleanup); a stale pair split across batches; no-op; stops on cancel; reports a write failure; scheduled on open; and `TestContentHashMigrationSummary` (the restart hint, and silence when nothing changed). - `cmd/server/hash_migration_status_test.go`: `hashMigrationComplete` follows the ingestor's migration row. - `cmd/server/readonly_invariant_test.go`: no write path to the analyzer DB from the server. - Lock hold, measured on 100k stale transmissions (half merging, 5 observations each) with a concurrent writer probing the lock: | write batch | hold p50 | hold max | probe max wait | total | |---|---|---|---|---| | 5000 (before) | 1269 ms | 1356 ms | 1351 ms | 25.3 s | | 250 (now) | 63 ms | 108 ms | 158 ms | 27.5 s | The same run at 10× the size (1M stale transmissions, 5M observations): | write batch | hold p50 | hold max | probe max wait | total | |---|---|---|---|---| | 5000 (before) | 1344 ms | 1808 ms | 1802 ms | 4 m 33 s | | 250 (now) | 66 ms | 132 ms | 199 ms | 4 m 38 s | - Removed `cmd/server/hash_migrate_test.go` and `hash_migrate_eviction_test.go` (#2327) along with the server code they covered. - `go vet` and `go test` pass on all modules, with gofmt clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Merge order These four PRs were split out of #2396 and are **independent — they can merge in any order.** Verified: every pair is conflict-free, and all four merged together pass `go vet`, `go test ./...` (all 16 modules) and `sh test-all.sh`. Suggested order, none of it urgent: 1. **#2408** — never lose rows during startup / a failed poll. 2. **#2407** — explicit `observers.idx` primary key. 3. **#2409** — content-hash migration moved into the ingestor. 4. **#2410** — pin musl thread stacks at 8 MiB. Released images are not affected (zig already links them with 8 MiB thread stacks); it fixes native Alpine builds. After these land, #2396 (Postgres) will be rebased onto master. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
17c1e3d1b3 |
test(live): wait for drag controls to finish initializing (#2423)
Partial fix for #1995. The Live page displays its panel headers before it finishes loading map configuration and registering the drag handlers. The browser test could drag the feed during that interval and then fail its position, storage and reload checks. Its short-click check could also pass before a handler saw the gesture. After each reload, the suite waits for the visible header and the current page’s drag manager. It hovers the header, presses the mouse, checks that the intended panel accepted the press, and measures movement from the actual press coordinates. This removes three fixed waits. TDD exemption: pure test-harness repair. The only altered file is `tests/e2e/test-drag-manager-e2e.js`. All seven scenarios and eleven original assertions remain. The additional ownership check prevents a click on the map canvas from satisfying the legend’s dead-zone test. Browser verified: `http://localhost:15027/#/live`. Delaying the actual map configuration response by 2.5 seconds reproduced three assertion failures on upstream. The final candidate passed 7/7 under normal and delayed loading. Blocking a genuine legend press made the old check pass incorrectly. The strengthened check failed precisely that scenario, with the other six passing. The earlier #2419 CI failure showed the same three symptoms. That run did not record initialization timing. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
cfb38e76a7 |
test(planner): benchmark full-range viewshed work counts (#2426)
Fixes #2424. The server’s default race-and-coverage suite ran full 30 km and 60 km terrain diagnostics on every run. Three workflows reached their 20-minute package deadline while the work-count test was running. The ordinary test now exercises a 5 km grid with 2,012 classified cells and checks the actual lookup reduction. Excluding the initial site lookup, the single terrain walk must use at most half the reference lookups. The missing-site test still allocates and checks the 60 km grid. Full-size diagnostic counts are available from `cmd/server`: ```sh go test -run '^$' -bench '^BenchmarkViewshedWorkCount$' -benchtime=1x ``` TDD exemption: pure test-harness refactor. Only `planner_viewshed_test.go` changes. The benchmark preserves the original grid dimensions, classified cells and lookup counts at both distances. The existing reference-equivalence, parallel-worker, cancellation and race checks remain in the regular suite. Validation on Windows, Go 1.27.2, `GOMAXPROCS=2`: - Default work-count test, native: 17.60 s before, 0.04 s after. With race detection and atomic coverage, the original exceeded the 210 s diagnostic limit. The candidate took 0.63 s. - Independent benchmark reproduced 31,756,821 reference / 15,878,411 single-pass lookups at 30 km, and 62,381,281 / 31,190,641 at 60 km. - Deliberate duplicate traversal and oversized-grid controls both failed their intended assertions. These timings measure the test workload. - The [hosted Go job](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44199958643/job/134451261110) passed the full server race-and-coverage suite in 1,125.875 s under its existing 1,200 s deadline. All browser shards passed. Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
be16173ef2 |
feat(nodes): link one-byte repeaters to Prefix Tool (#2412)
Red commit: `e27f81f705bf757da25c9fcd5858a6c73fcf54ed` ([CI run](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44061866573)). Known one-byte repeaters now offer **Inspect in Prefix Tool** in the shared Hash Prefix metadata. The link carries the full public key so the existing checker can exclude the selected repeater and automatically show its matches. Full detail, wide preview, tablet and mobile use the same helper. Unknown sizes, other roles and incomplete or invalid keys receive no action; existing prefix and variation text remain unchanged. Partial fix for #1154 U11. This adds navigation only; inline collision warnings/counts and the broader tracker remain open. Prefix Tool retains its existing filtered, capped population. No configuration or customizer changes. - GREEN: `f7a67b0d5ed7764a2400d091cee2dd1c35536e9b`; RED tests unchanged. - Helpers: 814/814; standalone frontend: all 244 suites; lint: zero errors. - E2E assertion added: `tests/e2e/test-issue-1147-section-order-e2e.js:227`. Full suite: 90/90, including both themes, responsive layouts, click/Enter navigation, prefill, results and scope checks. Additional real-data click-through passed without API stubs. - Two initial document-loading timeouts passed on the unchanged sequential rerun. Standalone validation used private Windows path/Python adapters; repository infrastructure is unchanged. - Performance: bounded key validation and one anchor; no node-render requests, scans or new retained state. Existing theme link color applies. Parent independently repeated 814 helper checks and the full 90-case browser suite, and inspected mobile/tablet screenshots. [Exact-head GREEN CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44066398243) and three independent reviews pass with no remaining findings. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
71d84ee432 |
fix(nodes): group exact counts in detail views (#2400)
Node details display exact packet, observation and relay counts using browser-locale grouping: for example, `12278×` becomes `12,278×`. This covers full details, wide previews and tablet metadata. Existing numeric validation, zero handling and raw numeric observer sorting are preserved. Partial implementation of #1154 U5. Directory rows, path summaries and analytics-wide formatting remain outside this slice. No compact rounding, settings, dependencies or API changes. Validation: - Tests-only RED `c9902a165` builds and [fails the intended grouping assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43828088770/job/133166683510); GREEN `dbb8f37c0` supplies the production change. - All 230 standalone frontend suites pass, including 812 helper assertions. Two focused browser suites pass 89 cases across English/German, desktop/tablet/mobile, both themes, numeric sorting, edge values and layout bounds. - Follow-ups align an older help fixture with grouped text and wait for completed initial theme loading before sorting. Every original grouping/sort assertion remains intact. Final source still fails four grouping assertions when served the upstream node renderer. - General browser suite: 149 passed, three existing fixture skips; optional/disabled checks remain outside coverage. Parent independently verified all 13 help scenarios and the seven affected location cases, including a deliberately delayed real theme response. - Three independent reviews completed; parent verified the corrections. [Final CI passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43838164835) on `c611492c6`. Constant formatting work per displayed count; existing loops remain linear, without new requests or retained collections. No speedup claim or customizer change. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
7850d77a80 |
fix(nodes): retain timestamp context in status explanations (#2403)
Red commit: `2e65630193311d5b9b13a0ca50355b346cc7a56a` ([failing CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43893211034)). Partial fix for #1154 U10. Active and Uncertain status explanations now use the shared timestamp renderer in full detail and wide preview. Relative dates expose exact-time tooltips; future dates retain their clock-warning icon. Existing timestamp preferences apply. - Keeps certain activity separate from attributed relay activity, preserving status decisions and wording. - Tests cover one Active timestamp, two Uncertain timestamps, missing certain activity, and future dates. Helpers pass 813/813; the full health-status browser suite passes. - Browser acceptance passed 128 combinations across desktop full/wide, 390px mobile, 800px tablet navigation, relative/absolute mode, UTC/local time, and both themes. No horizontal overflow or browser errors; 14 screenshots captured. - Constant work for at most two timestamps per explanation; no added requests, scans, caches, or retained collections. No new settings or colors. All 236 standalone frontend suites pass using process-local Windows file-URL/Python launcher adapters; repository tests are unchanged. ESLint reports zero errors. The broader U10 audit and other #1154 items remain open. Parent independently reran the helper and affected browser suites and checked desktop/mobile screenshots. [Exact-head CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43898474610) and three independent reviews pass, with no remaining findings. The separate compact-preview date-selection discrepancy is tracked in #2405. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
2ecc3d1e72 |
fix(server): never lose rows committed during startup or a failed poll (#2408)
## Summary Two pre-existing ways a running server **permanently** missed data. The database was correct; the server's in-memory store was not, and stayed wrong until the next restart. Found while comparing several server replicas side by side in #2396: two replicas that restarted mid-ingest each lacked one transmission and its 12 observations. Split out of #2396 at review's request, because it affects every SQLite install. ## The bugs 1. **Startup window.** The poller took its starting cursors from a fresh `MAX(id)` *after* the startup load finished. A row committed between the load's last chunk snapshot and that query ended up in neither the store nor any later poll. Its later observations were then dropped as orphans. 2. **Failed poll.** A failed observation read still advanced the cursor to its upper bound, skipping up to 500 observations for good. ## The fix - `NewPacketStore` records the transmission and observation high-water marks **before** the load, as the advert-evidence and rewrite feeds already did. - The loaded max transmission id stays as a floor. - The loaded max observation id no longer is one: a chunk-1 transmission can gain an observation whose id is *below* one that chunk 2 loaded. - Rows read twice are deduplicated and broadcast at most once. - A failed read keeps the cursor and retries on the next tick, logging once per failure streak. - Comments and a test pin why an observation of a transmission whose chunk has not merged yet cannot be skipped: the poller's evidence refresh waits on that chunk's lock. ## Performance - **Startup:** two extra indexed `MAX(id)` queries, which are primary-key lookups. - **Poll loop:** unchanged complexity. The only new per-tick work is a dedupe check on rows read twice, a map lookup by id (O(1)). Re-reads happen only for rows in the startup window, which is bounded by what was committed during the load, and on a retry after a failed read. ## Testing `cmd/server/startup_poll_gap_test.go` seeds the ingestor's own DDL plus `dbschema.Apply`: | Test | What it pins | |------|--------------| | `TestStartupLoadPollerGap_SQLite` | A transmission and observations committed between the load's snapshot and the poller's start reach the store | | `TestStartupLoadPollerGapMultiChunk_SQLite` | Same, across 3 chunks, including an observation id below the max loaded one | | `TestStartupObsDuringChunkMerge_SQLite` | An observation of a transmission whose chunk has not merged yet is not skipped | | `TestPollerRetriesFailedRead_SQLite` | A failed observation read does not advance the cursor | **Red/green:** against master's `store.go`, `websocket.go`, `chunked_load.go` and `db.go`, the first, second and fourth tests fail with the rows missing, e.g.: ``` tx 3, committed between the load's snapshot and the poller's cursor init, is missing from the store (lost forever) obs 3, committed while the observation read failed, never reached the store: the cursor passed it ``` The third depends on test hooks this change adds, so it cannot run against the old code. `go vet` and `go test` pass on all modules, with gofmt clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Merge order These four PRs were split out of #2396 and are **independent — they can merge in any order.** Verified: every pair is conflict-free, and all four merged together pass `go vet`, `go test ./...` (all 16 modules) and `sh test-all.sh`. Recommended **priority** (not a dependency): 1. **#2410** — musl 8 MiB thread stacks. A public, unauthenticated GET can SIGSEGV the whole server on the shipped image, so merge this first. 2. **#2408** — never lose rows during startup / a failed poll. 3. **#2407** — explicit `observers.idx` primary key. 4. **#2409** — content-hash migration moved into the ingestor. After these land, #2396 (Postgres) will be rebased onto master. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
11d25f9f7d |
docs(api): describe configured areas and geometry (#2401)
Partial fix for #1670. Document the two public area configuration endpoints with reusable array response schemas. The descriptions distinguish labeled summaries from geometry entries, specify latitude/longitude order, and explain omitted fields and preserved zero bounds. Known route-description gaps decrease from 15 to 13. This changes generated API metadata only. Handler behavior, authentication, configuration and packet processing are unchanged; no new customizer settings or dependencies. Validation: - Tests-only RED `8f27f8ff0` builds and [fails the intended documentation assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43828097551/job/133166654496). - GREEN `5966b8021` passes the byte-unchanged tests; [final CI passes](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43830879045). - Handler fixtures cover absent/empty config, labeled/unlabeled and underscore keys, whitespace labels, normal/short/empty polygons, partial bounds and zero values without assuming entry order. - All 26 focused checks pass on final GREEN. Full native server/ingestor suites passed at `e19c49e26` (158.088s/160.279s); the final amendment only clarifies four descriptions, and final CI includes both suites. - Parent browser validation rendered both operations in Swagger UI and executed their actual public GETs, checking coordinate shape, omissions and zero values. - Three independent reviews completed with no remaining findings; parent read every report. The remaining 13 route-description gaps and broader parameter/schema work stay open under #1670. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
86448911a1 |
docs(api): document known channels catalogue (#2411)
Partial fix for #1670. Document the existing public `GET /api/known-channels` endpoint with a concrete response schema and its optional catalogue-region filter. The descriptions cover missing-cache sentinels, omitted fields, verbatim upstream strings, unspecified entry order, and the actual 30/300-second cache lifetimes. Catalogue entries do not establish observed traffic or successful decryption. Known route-description gaps decrease from 15 to 14 on this base. This changes generated API metadata only. Handler, parser, cache, authentication and configuration behavior are unchanged. No new dependencies or customizer settings. Validation: - Tests-only RED `fadc0fb5e` compiles and [fails the missing-documentation/schema assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44058546803/job/133954154417); all ten registered-handler cases already pass. - GREEN `5ed158aab` passes the byte-unchanged tests. Fixtures cover nil/unpopulated/empty caches, case and whitespace handling, arbitrary and unknown regions, raw metadata, omissions, public access and headers. - All 21 focused tests pass. Full native server and ingestor suites pass in 287.788s and 245.003s respectively. - Parent verified Swagger rendering and five real public queries against a cached catalogue while its source was offline, plus the disabled-cache response. [Exact-head GREEN CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/44065924428) and three independent reviews pass with no remaining findings. The remaining endpoints and broader parameter/schema work stay open under #1670. This change is independent of the area-configuration documentation in #2401. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
a2417b25a3 |
test: normalize static helper import as a file URL (#2404)
Partial fix for #1995. On native Windows, this suite aborts before its five checks because the absolute module path is treated as an unsupported `f:` URL. Use Node's built-in `pathToFileURL(...).href` for the dynamic import. TDD exemption: pure test-harness refactor/repair. Application code is unchanged. Per-altered-test justification for `tests/unit/test-dev-static.js`: import-path normalization is the sole changed fixture statement, plus the required built-in import. All five existing test bodies, assertions and teardown are byte-identical. The original loading failure is not claimed as assertion RED. Validation: Windows x64, Node v22.18.0. - Untouched suite: `ERR_UNSUPPORTED_ESM_URL_SCHEME`, exit 1 before assertions. - Updated suite: 5 passed, 0 failed; also 5/5 from a directory containing spaces, `#` and `%`. - Byte comparison, `node --check tests/unit/test-dev-static.js` and `git diff --check` pass. Parent independently passed the five native checks and all 236 standalone frontend suites without an ESM adapter. [Exact-head CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43899294545), including all three browser shards, and three independent reviews pass. Runner reconciliation, browser waits, coverage and the rest of #1995 remain open. Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
11eb72f578 |
feat(planner): Coverage tab, reach of a candidate repeater site (#2406)
The repeater planner (#2391) gets a fourth tab, **Coverage**: put a marker where you are thinking of placing a repeater, set its antenna height, and see on the map who would hear it. It answers the siting question "what does a repeater reach if we put it here", with two layers: handheld users and other repeaters. ## What it shows - **Users layer:** the chance that a handheld at 1.5 m hears the site, in 4 bands (10-25%, 25-50%, 50-75%, 75% and up; below 10% is not painted). The chance comes from a distance curve `P(d) = 1 / (1 + exp((ln d - ln d50) / slope))`, fitted on drive-test receptions of 41 Belgian repeaters (d50 2.7 km, slope 0.30). Terrain only lowers a cell by 2 bands where a 1.5 m receiver sits deeper than 1 first-Fresnel radius under the terrain (`cmd/server/planner_viewshed.go:19-21`). Antenna height does not change the curve: in the drive tests, handheld reach barely changed between 1, 7 and 15 m. The panel says so in one line. - **Repeaters layer:** first-Fresnel clearance from the site to every cell, for a receiver at "Repeaters at" height (default `planner.existingAntennaM`). 3 classes: line of sight (clearance 0% or more), slightly blocked (-20% to 0%), blocked (not painted). Every known repeater within range gets its link computed with the existing planner link code, a dot in its class colour, a line for line of sight and slightly blocked, and an entry in a list. - **Compute button:** changing site, height or range moves the marker and updates the URL hash right away, and dims an existing result with an "Outdated" note. Only "Compute coverage" starts a job. A shared link that carries a site computes once on open. The planner has 1 heavy-job slot shared with Area and allows 6 jobs per minute per IP, so recomputing on every change would burn both. State is in the hash: `#/tools/planner?mode=coverage&site=51.06,4.41&h=10&rx=5&range=30&layer=repeaters`. This builds on the repeater planner @zfouts wrote in #2391: the terrain tiles, the first-Fresnel analysis, the link code, the job protocol and the Area module pattern are all reused here. Thanks for that work, it made this tab a small addition instead of a new subsystem. ## Why the Repeaters classes are not the route LOS verdicts The route verdicts (clear 100%+, clear60 60%+, marginal 0%+, blocked) almost never show on Shuttle Radar Topography Mission (SRTM) terrain at mast heights. At 10 m / 5 m, 30 km around Mechelen, 109 of 70,980 cells reach clear60 or clear. SRTM includes trees and roofs, so the line between 2 masts rarely clears 60% of the Fresnel zone. I checked the clearance against one Belgian repeater (28 m, receivers at 5 m) and the repeaters live data shows it actually hears: | Distance | Heard, clearance >= 0 | Heard, clearance < 0 | |---|---|---| | 0-10 km | 9 of 11 | none in range | | 10-20 km | 3 of 4 | 1 of 8 | | 20-30 km | 0 of 6 | 3 of 35 | | 30-60 km | 1 of 2 | 0 of 84 | Every link heard within 16 km had a clearance between 2% and 54%, so "marginal" is where working links live. The map classes follow that (`repeaterClass`, `cmd/server/planner_viewshed.go:75-89`). Chain, Relay and Area keep the shared verdicts; `losVerdict` is unchanged. This rests on 17 heard links from 1 repeater, which is why the -20% band is narrow and the panel says line of sight stops being enough beyond about 20 km. ## Server - `GET /api/planner/viewshed?site=&h=&rx=&range=` (`cmd/server/planner_api.go:603`, route `cmd/server/routes.go:393`). Same job protocol, result cache, rate limits and opt-in as the other planner endpoints: 404 while `planner.enabled` is false. - One byte per cell: low nibble Users class 1-5, high nibble Repeaters class 1-3, 0 for no data. The grid is about 300 x 300 at every range: 0.2 km cells up to 30 km, `range/150` above (`planner_viewshed.go:47`). - Links skip rooms, stale repeaters and a repeater within 100 m of the site (that is the site itself). They carry no `elev_m` profile. - `truncated` is true only when the 300-node cap may have dropped a repeater within range (`viewshedTruncated`, `planner_api.go:694`). - Tile cap: `Viewshed` in the planner tile limits, sized from the largest offered range (60 km). - New config: `planner.userReach.{d50Km,slope}` with a `_comment` in `config.example.json`. Bounds 0.2-50 km and 0.05-2. - No writes, no new `map[string]interface{}`; response types `PlannerViewshedResponse/Params/Link` in `planner_types.go`. ## Frontend - `public/planner-coverage.js` + `.css`, mounted like `planner-area.js`. `planner.js` gets a `MODULES` table (`public/planner.js:31`) so Area and Coverage share one enter/leave path. - Canvas raster in an `L.imageOverlay`; the layer switch repaints locally without a new request. Requests go through `fetchJob` / `CSApi`. - Colours are CSS variables `--pl-cov-user-2..5` and `--pl-cov-rep-1..3` in `:root` and both dark blocks of `style.css`; the Repeaters ones point at `--status-*`, so a theme that changes those changes the map. ## Performance The job is opt-in and runs in the existing planner job slot, never in a hot path. Measured on real SRTM tiles, local build, 304 Belgian repeaters: | Site | Range | Mast | Job time | |---|---|---|---| | Maasvallei (hilly) | 30 km | 10 m | 2.07 s | | Maasvallei | 30 km | 30 m | 2.57 s | | Maasvallei | 60 km | 10 m | 3.98 s | | Mechelen (flat) | 30 km | 10 m | 2.61 s | On a staging instance (ARM64, cold tile cache) a 30 km job took 6.8 s including the tile downloads. Response size at 60 km with 115 links: 239,942 bytes with link elevation profiles, 149,984 without (what this PR ships). `BenchmarkViewshedMaxLeg` covers the 60 km worst case on a synthetic terrain view. ## Verification - Go: 26 viewshed tests and benchmarks in `planner_viewshed_test.go`, `planner_api_test.go`, `planner_config_test.go` and `planner_tiles_test.go`, among them: flat terrain equals the plain curve and `analyzeLOS`, a ridge shadows only its far side, missing terrain is "no data", the `truncated` rule (proven to fail on the old rule), links without `elev_m`, the repeater at the site left out. `go test ./...` in `cmd/server` passes. - Frontend: `tests/unit/test-planner-coverage.js` (23 tests, in `test-all.sh`): grid decode, class-to-colour mapping, unpainted classes, hash round trip, no fetch on parameter change, 1 fetch per click, deep-link autorun once, stale responses ignored, link classes at the boundaries, escaped repeater names. - E2E: 5 Coverage cases in `tests/e2e/test-planner-e2e.js` (stubbed viewshed): click and compute, layer switch without a request, outdated then recompute, deep link, leaving the tab removes the overlay. Planner E2E 25/25 locally. - eslint 0 errors. Browser: light and dark theme, Users and Repeaters on flat, hilly and coastal sites, deep links, no console errors; also on a staging instance with live data. ## Not in this PR - The `--pl-cov-*` colours are not in the customizer's `THEME_CSS_MAP` yet. They follow `--status-*` for the Repeaters layer; the Users colours need their own entries (AGENTS.md rule 8, later milestone). - In Flanders the Users layer is close to a circle: 0.3% of cells differ from the plain curve around Mechelen, 42% in the Maasvallei, 64% in the Ardennes. The terrain correction is untested in hilly areas, the drive tests had no repeater there. - The Repeaters calibration uses 1 repeater. Other operators with drive tests or neighbour data can tighten it. - A link's class comes from `clearance_pct`, rounded to 0.1%, while the grid uses the raw clearance: within 0.05% of a bound a link can land 1 class off the cell under it. List, line and dot always agree with each other. - An OS dark-mode switch without the theme toggle does not repaint the overlay until the next compute, the same as the planner's tile layer. --------- Co-authored-by: zfouts <zachary.fouts@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
12f9c5c686 |
docs(release): v3.18.0 release notes and changelog entry (#2420)
Release notes and changelog entry for v3.18.0, to be tagged on this PR's merge commit. - `docs/release-notes/v3.18.0.md`: the notes, in the same shape as v3.17.0. The CI/CD Pipeline uses this file as the GitHub release body. - `CHANGELOG.md`: a 3.18.0 entry under `[Unreleased]`. 11 commits since v3.17.0: 5 fix, 3 feat, 1 test, 1 refactor, 1 docs. "Read this before upgrading" covers the `users.db` schema v9 migration with user accounts on (#2415), the WebSocket path `/ws` (#2383) and config tile URLs winning on the Map and Live pages (#2386). Documentation only, no code change.v3.18.0 |
||
|
|
63d8ab23ed |
fix(repeater-info): an open edit form survives a node page re-render (#2417)
Follow-up to #2415, see #2414. |
||
|
|
af732d18aa |
fix(repeater-info): log bulk read errors, cap long notes (#2416)
Follow-up to #2415, see #2414. |
||
|
|
0cf22cdd79 |
feat: repeater info, claims and community-maintained details (#2415)
Closes #2414 |
||
|
|
4ebb5ccc11 |
feat: radio line-of-sight on packet routes + repeater planner (#2391)
Both features are OPT-IN ("los" and "planner" config blocks, default
off); with them off nothing is fetched and payloads are unchanged.
Route view
- Multi-path picker discoverability: on-map "N hops shown · tap to see
all" badge, picker pulse (reduced-motion safe), one-time tip, row hover
arrow; "no GPS" hops counted apart from "unresolved".
- Line of sight for every leg of a packet route: clear / marginal /
blocked chips (first-Fresnel clearance, k=4/3), obstruction markers on
the map, docked terrain-profile panel with antenna/frequency inputs and
a &los= deep link. Backed by GET /api/elevation/profile, an Open-Meteo
proxy that only accepts known node positions, with weighted budgets,
back-off and a per-IP limit.
Repeater planner (#/tools/planner)
- Chain: LOS on every leg between any number of points or nodes.
- Relay: best single relay for A→B among known repeaters and open
terrain.
- Area: suggested new repeater sites that best join the existing
repeater backbone in a box up to 100×100 km, picked greedily to merge
clusters, then remove single points of failure, then reach unlinked
repeaters.
- Terrain: SRTM 1° tiles from AWS Open Data, in memory only (~90 m,
bounded LRU); the server stays read-only. Exact-length gunzip, fetch
budget with back-off, missing terrain reported rather than read as sea
level, fuzzed pixel maths.
- GET /api/planner/{profile,relay,coverage}: quantised cache keys, 202 +
progress then cached 200, separate compute slots, per-IP and global
limits, failed jobs remembered, unpolled jobs cancelled. Nodes pass the
same blacklist / hidden-name / geo_filter rules as /api/nodes. The Go
LOS maths matches public/los.js on shared test vectors.
Also: the per-IP client limiter is extracted from elevation.go into
client_limit.go (behaviour-preserving) so both features share it.
Perf (real SRTM): coverage cold 1.5-5.6 s (tile download), warm 10 ms;
worst real compute 545 ms; peak RSS 91 MiB; live /api/nodes and
/api/packets p95 at most 1.6x while a job computes (GOMAXPROCS=2).
Tests: Go suites for elevation, terrain (fuzz), LOS vectors, graph
(brute-force property test), search and planner API; frontend unit
suites in test-all.sh; Playwright e2e for route LOS and the planner.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: efiten <erwin.fiten@gmail.com>
|
||
|
|
138473d8c2 |
test(tile-resolver): load CSApi into the sandbox before roles.js (#2402)
## Problem `tests/unit/test-tile-resolver.js` fails 0/26 on master with `CSApi is not defined`, and master's CI on `e7fd48a0` is red because of it ([run](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43880608436)). Two PRs that each passed CI on their own met on master: - #2386 added this suite, which runs `public/roles.js` in a `vm` sandbox. - #2383 made `roles.js` call `CSApi.fetch('/api/config/client')` as it loads. The sandbox never had `CSApi`. ## Fix The suite now loads `deployment.js` and `api-base.js` into the sandbox first, through `tests/unit/api-base-sandbox.js`, as the other suites that load `roles.js` already do. That's two lines in a test file, with no app code changes. ## Verification - `node tests/unit/test-tile-resolver.js`: 26 passed, 0 failed (it was 0/26). - `sh test-all.sh` on this branch: exits 0. ## Related #2391 needs this. It now includes #2383 from master, so its CI hits the same failure. Once this lands, I'll merge master into #2391 again. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e7fd48a054 |
feat(frontend): serve the UI from a web container or a static host, three deployment modes (#2383)
## What this does The frontend half of serving the UI apart from the server. The server half was #2362. It gives CoreScope three deployment modes. The server container is the same image in all three, and it still serves everything, the classic UI included. | Mode | The UI is served by | API | Server CORS setting | Logins | |---|---|---|---|---| | All-in-one container (today) | the server container | same origin | none | yes | | Split containers (new) | a web container that forwards the API | same origin | none | yes | | Static host or CDN (new) | Cloudflare Pages, a CDN or any web server | another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI | ### Commit 1: the UI as static files, for any host - **One place for API URLs:** `public/deployment.js` says where the API is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket URL. A lint rule keeps raw `/api` fetches out of the other files. The embedded UI keeps `apiBase: ''`, so it behaves as before, except that its WebSocket now uses `/ws` instead of `/`. - **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`, which is `public/` byte for byte except for the cache buster in `index.html` and `deployment.js`. - It is `dist/web` rather than `dist/` because the Go binaries are built into `dist/` and `make clean` removes that folder. - `API_BASE=same-origin` is for a host that forwards `/api` and `/ws` itself, as the web container does. - **Anonymous build (`AUTH=false`):** for another origin. It has no account features: login, account, admin and other writes link to the same page on the classic UI. It sends no cookies, and its three JSON POSTs go as `text/plain`, so there is no preflight. - **Error notices:** a blocked API answer or an HTML answer each show a notice. - **Channel badge:** `beta`, `next` and `dev` builds show a badge that opens a pre-filled issue, plus an "Open in classic" link. - **Runbook:** `deploy/static/README.md` covers Cloudflare Pages, previews, server CORS and rollback. ### Commit 2: the web container and the three modes - **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It packages the UI with `build.sh` when the image is built, serves it, and forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`). `SITE_ADDRESS` set to a domain name turns on automatic HTTPS. - **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules as `deploy/static/_headers`, and compresses responses with zstd or gzip. - **`docker-compose.split.yml`:** the stock server image (its own Caddy off) plus the web container: `CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f docker-compose.split.yml up -d --build`. - `CORESCOPE_IMAGE` is required and must match the checkout's version, so a release server is never paired with a newer UI by default. - The server's built-in broker, which accepts anonymous clients, is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - **Docs:** `docs/deployment.md` has a new "Deployment modes" section, and `DEPLOY.md` points to it. ### Keeping new code on `CSApi` - **Lint rule:** `.eslintrc.json` now rejects every direct network call in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`, `sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in `href`, `src` and `action` (assigned, set with `setAttribute` or written into markup), `window.open`, `location.assign`, `location.replace` and `location =`. `api-base.js` is the one file exempt. - **HTML pages:** `eslint public/*.js` never sees the inline scripts and markup of `public/*.html`, so `test-lint-network-calls.js` applies the same rule to them, at their page lines. `area-map.html`'s two fetches carry an `eslint-disable` with the reason: its Server field can name another server. - **Why it changed:** the earlier rule only caught a literal `/api…` passed straight to `fetch`. - **Check:** CI already fails on lint errors, and `tests/e2e/test-lint-network-calls.js` pins exactly what the rule reports: 25 bad statements reported, 15 good ones not, `api-base.js` exempt. - **AGENTS.md:** a new "API calls" section covers: - which helper to use for each need; - the lint rule; - how anonymous builds gate login-only features; - loading `api-base.js` in vm and jsdom tests (the region quick-picks suite missed that, and failed shard 1 on the first CI run). ### Review follow-ups - **Account tokens:** the channel badge's pre-filled issue drops the query on `#/account/…` routes, so mail-link tokens never reach a GitHub URL. "Open in classic" keeps the full route, since the classic UI needs the token. - **HTML answers:** on a packaged UI (static host, web container), `api()` rejects an HTML answer before status and retry handling. An HTML 404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server returned an unexpected response" after one request. It used to give a generic error, or six "warming up" retries for a 503. JSON errors and the server's JSON 503 warm-up behave as before. The embedded UI keeps its old handling (`CSApi.embedded`). - **Logins:** `docs/deployment.md`, `Dockerfile.web` and `docker-compose.split.yml` say `userManagement.publicBaseUrl` must be the web container's public origin. ### Audit fixes - **No broken logins build:** `build.sh` refuses `AUTH=true` with a cross-origin `API_BASE`. The server allows no credentialed CORS, so every API request of such a build was blocked; in Chromium all 12 were, and the pages stayed empty. - **No accidental deletes:** `build.sh` refuses an `OUT` that is neither empty nor a previous build. It used to wipe whatever folder or file `OUT` named. - **Lint:** the rule now also reports `/api/` paths in `href`, `src` and `action` attributes in single-quoted strings and template literals. `tests/e2e/test-lint-network-calls.js` pins exactly what it reports. - **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published on localhost only. - **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since v3.16.0 and v3.15.0 predate the static build. It also names the server version the static build needs: before #2362 the warm-up headers stay hidden and wildcard previews are refused. ### Audit notes, fixed - **Cache busting on every page:** the tool pages (`area-map.html`, `geofilter-builder.html`) load their scripts with `?v=__BUST__`. The server and `build.sh` replace it in every top-level page, and `_headers` and the Caddyfile make every page revalidate. Behind Caddy, a returning visitor could otherwise get a new tool page with an old `api-base.js`. - **HTTP/3:** the split compose publishes 443 over UDP too, for the HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`. - **WebSocket on `/`:** a Go test pins that an upgrade on `/` still works, for UIs cached from before `/ws`. ### Second review - **Trusted proxies:** the docs said to trust the compose subnet, and also to publish port 3000 if wanted. Together they let a client spoof its address: the subnet holds the gateway, where a client reaching a published 3000 through Docker's proxy arrives from. The compose file now fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web container's address (`172.30.83.200`), and the docs say to trust that address alone. - **Classic UI unchanged:** the HTML-answer notice above applies to packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as before, and an unknown `/api` path fails to parse as before, with no notice. - **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md says what the rule cannot see: a URL assembled elsewhere and passed in a variable. - **AGENTS.md:** `build.sh` is described as packaging (a copy, `deployment.js` and `__BUST__`, never a code transform), and rule 3 says `__BUST__` is filled in every top-level page. - **CI builds the web image:** the image-check job (no publish) builds `Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`, the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It also validates the split compose file. - **Cross-origin logins:** a hand-edited `deployment.js` with `auth: true` and another origin now logs a warning naming the setting, instead of failing with CORS errors alone. ## Upgrade notes - **WebSocket path:** the UI's live feed connects to `/ws` instead of `/`. The server has answered `/ws` since #1793 and still upgrades `/`, so browsers holding the previous UI keep working. - A reverse proxy that forwards WebSocket upgrades only for an exact `/` must forward `/ws` too. - The nginx example in `docs/deployment.md` (`location /`) already covers it. - **Static UI from another origin:** - The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for the warm-up banner and wildcard preview origins. - `build.sh` refuses `AUTH=true` there, since the server allows no credentialed CORS. - **Split containers:** - `CORESCOPE_IMAGE` is required. - With user accounts on, `userManagement.publicBaseUrl` must be the web container's public origin. - The built-in MQTT broker is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - The compose network is `172.30.83.0/24`, with the web container at `.200`. If Docker reports that the pool overlaps another network, set `SPLIT_NET`. ## Testing - `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with new tests for busting every page and the WebSocket upgrade on `/`. - `sh test-all.sh` passes. New unit suites: - `test-static-build.js`: the build and its input checks, `same-origin` included, cross-origin logins refused, and an unrelated `OUT` left intact. - `test-web-container.js`: the Dockerfile, Caddyfile and compose file agree on packaging, forwarded paths, cache rules and the server port; the server image has no default and MQTT stays on localhost. - The `api-base`, anonymous mode, badge and `dev-static` suites. - `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes 47/47, the four pages included; it needs eslint, so CI runs it on E2E shard 1. - **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes 11/11. That is the static build on one origin against an API-only server on another, which CI runs on E2E shard 1. - **Split containers mode:** - The image builds, and `caddy validate` passes (Caddy 2.11.4). The new CI step passes locally, and fails when the Caddyfile stops forwarding `/api`. - Trusted proxies, with the stack up: the web container gets `172.30.83.200`. With only that address trusted, a spoofed `X-Forwarded-For` sent straight to a published 3000 is ignored, and one sent through Caddy is replaced with the client's address. With the subnet trusted, as the docs used to say, the spoofed address was believed: a deny entry for it rejected the upgrade. - With `docker-compose.split.yml` on the E2E fixture, seeded the way CI seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both through the web container and against the server directly. - Headers, checked with curl: - `no-cache` on `/`, `/index.html` and `/deployment.js`; - zstd on static assets; - `/api/*` proxied (`Via: 1.1 Caddy`); - a `101` WebSocket upgrade on `/ws`. - In Chromium: - every API request and the WebSocket stay on the web container's origin; - there are no page errors and no horizontal overflow at 390 px; - no badge shows on the `static` channel. - Logins through the web container: with user management on and `userManagement.publicBaseUrl` set to the web container's origin, a Chromium run passed every step, all on the web container's origin with no page errors: - registration, and activation from the mail link (which uses `publicBaseUrl`); - the session surviving a reload; - a CSRF-protected `PATCH /api/account`; - logout, then login with the form. - The same login with the backend's origin as `Origin` is refused with 403. The server checks `Origin` against `publicBaseUrl`, not `Host`, which is why the docs ask for it. ## Performance There are no hot-path changes. API calls go through `CSApi.url()`, a string concatenation. In the web container, Caddy serves files from disk and compresses them; API responses pass through with one proxy hop, the same as the all-in-one image's built-in Caddy. ## Notes - **Per-IP limits:** behind the web container, as behind the all-in-one image's Caddy, requests reach the server from the proxy's address. The per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`, the web container's fixed address. The docs say so. - **`depends_on`:** the web container starts with the server, not after its healthcheck, like the all-in-one image's Caddy. Gating on health would make every `up` wait for the first check (30 s) and fail outright when a large database loads for longer than the healthcheck allows; until the server listens, Caddy answers `/api` with 502. - **Not in this PR:** - publishing a `corescope-web` image (CI builds and smokes it, without publishing); - Cross-origin login—including a static frontend and API on sibling custom domains—is outside this PR. In the deployments supported by this PR, account support requires the frontend and API to share one browser origin, either served together or through a reverse proxy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d80ce90976 |
fix(store): release evicted packet references from secondary indexes (#2327)
Addresses #2324 for upstream SQLite; PostgreSQL qualification remains separate. Eviction releases actual node memberships, pointer tails and cache roots without SQL reconstruction. Borrowed distance snapshots and bounded hash-migration ownership remain protected. Review correction `0b1049b5` skips each distance-array copy when no record is evicted from that array. When affected, it clones once and filters the suffix beginning at the first match, preserving borrowed snapshots and the surviving prefix. Work remains O(distance records); no new locks, reader bookkeeping, settings or general speedup claim. Tests-only `16509577` reaches [three real assertion failures in CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43200112374/job/131121073845). GREEN leaves those tests unchanged. Cases cover neither/hops-only/paths-only impact, repeated removals, live prefixes, tail removal and snapshot contents. Parent independently reproduced RED and reran GREEN. Full native server/ingestor suites pass. Targeted checks: 167 passed; race: 165 passed, with two existing timing skips that pass in the ordinary run. Current master `a0b4e6cf` is normally merged; original RED/GREEN history is preserved. Historical November 10 measurements: synthetic retained heap 98.00→27.14 MiB with equal live data; path-update cost at 30k hops/10k paths 0.591→3.675ms and 2,824→5,368,584 B/op. The older ~30ms extra eviction lock time at 240k transmissions is unremeasured. Large-window qualification/optimization: #2393. Invalidated migration progress: #2394; no SQL migration redesign here. Browser verified: populated matching revision (500 transmissions/502 observations), 3/3 interactions. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43202356164) and three independent reviews pass. No settings changes. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
0e23c24847 |
fix(scope-stats): report total and split unscoped by route (#2364) (#2368)
Fixes #2364. ## Problem Since #1838, `GetScopeStats` folds every non-transport transmission (FLOOD 1, DIRECT 2) into `summary.unscoped`, while `transportTotal` and `scoped` still cover transport routes (0, 3) only. The documented `transportTotal = scoped + unscoped` no longer holds, and a client computing an unscoped share against `transportTotal` gets >100% (the reporter shows `transportTotal 7620`, `unscoped 14320`). ## Fix — additive, no existing field changes value `summary` gains: | field | meaning | |---|---| | `total` | all observed transmissions, routes 0–3 (= `scoped + unscoped`) | | `unscopedTransport` | route 0/3 with Code1 = 0000 | | `unscopedFlood` | route 1 — unscoped by protocol | | `unscopedDirect` | route 2 — unscoped by protocol | Invariants (now asserted in `TestGetScopeStats`): ``` transportTotal = scoped + unscopedTransport unscoped = unscopedTransport + unscopedFlood + unscopedDirect total = scoped + unscoped ``` The non-transport count query keeps its single scan; it just splits route 1 vs 2 with `CASE`. On the design question in the issue: rather than changing what `unscoped` means (the Scopes tab and #1842 rely on it), the split lets a rollout view use `unscopedFlood` directly: repeaters apply region rejection to flood routes only (`isRouteFlood()` in `examples/simple_repeater/MyMesh.cpp`), so DIRECT traffic is not subject to flood-region filtering. `docs/api-spec.md` now says so, spells out the route mapping, and notes that `timeSeries` is transport-only (its `unscoped` matches `unscopedTransport`). ## Frontend Scopes tab uses `summary.total` when present and shows `N transport, N flood, N direct` on the Unscoped card; falls back unchanged for older servers. The cards are built by `scopeSummaryCardsHtml(summary)` (exposed as `window._analyticsScopeSummaryCardsHtml`). ## Tested - `go vet ./...` (cmd/server) clean, `gofmt -l` clean - `go test -run ScopeStats ./...` — pass (incl. extended `TestGetScopeStats` and `/api/scope-stats` route test) - `node --check public/analytics.js`; `tests/unit/test-issue-1375-scope-stats-fetch.js`, `test-issue-1979-scope-adverts-by-role.js` pass - new `tests/unit/test-issue-2364-scope-summary-cards.js` (vm sandbox, 11 assertions: shares of `total`, route split, fallback for servers without `total`/split, partial split, empty window); full `test-all.sh` green Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
ea87784257 |
refactor(live): reuse shared timestamp renderer (#2395)
Live timestamps duplicated the shared timestamp markup in `app.js`. Keep `formatLiveTimestampHtml` as the input adapter and delegate its final markup to `renderTimestampHtml`, retaining the existing missing-helper fallback and Date/number/string normalization. Partial follow-up to #1154 U10 and the [maintainer note on #2283](https://github.com/Kpa-clawbot/CoreScope/pull/2283#pullrequestreview-5850899652). This addresses only the Live wrapper; the umbrella remains open. Validation: - All 229 existing standalone frontend suites pass. Two shell-dependent suites passed after selecting the installed shell/interpreter for the local test process. - 5,040 exact-base/candidate comparisons match byte-for-byte across timestamp inputs, preferences, escaping, future warnings, and missing-helper fallbacks. - Actual Go fixture and Chromium: node-panel and replay-feed timestamps checked on desktop and phone, with no page exceptions. The existing phone portrait layout hides the feed; its replay DOM remains populated. - ESLint: zero errors. No new settings, requests, dependencies, or caches; constant work per timestamp, with no speedup claimed. Pure-refactor TDD exemption: **tests/ files diff: no changes**. No test files added or modified; existing assertions remain green. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43202375909) and three independent reviews pass. Parent independently repeated the Live unit, parity and post-push browser checks. Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
c2e770ef65 |
docs: archive superseded repeater liveness plans (#2392)
Partial fix for #1385 (PR-E). Move the two April repeater-liveness documents from `docs/superpowers/` into existing `docs/archive/` with `git mv`. Preserve their dated filenames, content and headings; add historical notices clarifying that the proposed `relaying` status/signature is outdated and linking current guidance. Validation: exact commit blobs preserve 41,483 original bytes and 24 headings after removing only notices; four local links resolve; no tracked old-path references; whitespace and PII checks pass. TDD exemption: pure documentation, no behavior change. No runtime, configuration, customizer, test, hierarchy or ignore-rule changes. Application suites were not rerun. Both published Markdown pages and their guidance links render correctly. [Docs-only CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/43199869942) and three independent reviews pass. The umbrella issue stays open pending maintainer reconciliation. Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
a0b4e6cf5f |
fix(nodes): light-mode map follows tile picks, key and theme changes (#1470/#1420) (#2386)
fix(nodes): light-mode map follows tile picks, key and theme changes
(#1470/#1420)
## Problem
The node-detail maps (`#/nodes/<pubkey>` full page + side pane — and
their
reusers `node-reach-map.js`, `rx-coverage.js`) pick a different basemap
in
**light mode** than the rest of the app:
- hardcode Carto Positron: the user's *Light Mode Provider* customizer
pick
and the operator's `map.tiles.lightDefault` are ignored
- render Carto's `API KEY REQUIRED` watermark when `/api/config/client`
lands after the map is created — the tile requests stay keyless until a
remount (they come back HTTP 200, so nothing surfaces as an error)
- never react to a theme flip or provider change while the page is open
Dark mode is unaffected, which is why it only shows up in light mode.
Reproduce on `master`: open `#/nodes/<id>` in light mode with
`mc-light-tile-provider = carto-voyager` — the inset requests
`light_all`
while the main map requests `rastertiles/voyager`.
## Root cause
`_applyTilesToNodeMap` (`public/nodes.js`) asked `window.getTileUrl()`
for a
single URL string, and roles.js's light branch returned the frozen
`TILE_LIGHT` scalar — the dark branch was the only one that consulted
the
runtime registry (`public/map-tile-providers.js`), which is what owns
localStorage picks, server defaults, the Carto key, and per-provider
attribution/maxZoom/refUrl/invertFilter. map.js and live.js keep the
only
other registry-aware resolvers of the same logic.
## What this PR does
- adds one resolver, `MC_resolveTile(isDark)`, in map-tile-providers.js,
returning the full descriptor
`{ type, id, url, attribution, maxZoom, refUrl, invertFilter }`:
- `url` is always a resolved string — function-typed providers are
invoked at call time (contract of #1614)
- one precedence rule everywhere: explicit operator overrides
(`cfg.tiles.dark|light`, `map.tiles.darkUrl|lightUrl`, tracked via
`MC_LEGACY_TILE_OVERRIDE`) > user/server registry pick >
`TILE_DARK`/`TILE_LIGHT` constants — **this is a deliberate behavior
change on map.js/live.js, see below**
- fully lazy: registry rebuilds / pick changes are picked up by the next
resolve, so config landing after page load fixes itself
- roles.js `getTileUrl()` becomes a thin wrapper over it; the public API
and
the #1614 sandbox contract are unchanged; `getActiveTileProvider()`
keeps
its documented null-in-light contract
- `nodes.js _applyTilesToNodeMap` resolves through the descriptor and
stays
reactive: one shared `mc-tile-provider-changed` listener + one
`data-theme`
MutationObserver re-sync every registered map in place — old layers
removed
before the new ones are added, never stacked; maps whose container left
the
DOM are pruned at sync time, so SPA navigation cannot leak listeners
- map.js `_resolveTileUrl` / live.js `_liveResolveTile` delegate to the
shared resolver (their legacy bodies remain only as sandbox fallbacks)
- provider `maxZoom` replaces the hardcoded 18/19 caps (OpenTopoMap 17
and
USGS 16 used to 404 above their max zoom)
- analytics.js subpath minimap gets provider `maxZoom` as well
## Behavior change (deliberate): operator URL overrides now win on every
surface
On `master`, the two tile-selection mechanisms conflicted and each
surface
resolved the conflict differently:
- `map.js` / `live.js`: user/server **registry pick won** — an
operator's
`tiles.dark|light` / `map.tiles.darkUrl|lightUrl` was effectively dead
config on those pages
- the legacy roles.js path (nodes page, reach map, rx-coverage,
analytics
minimap): **operator override won**
This PR unifies on **operator-override-wins**, because that is the only
reading that gives live operator config a consistent meaning:
- these keys are still applied today. The deprecation warning says they
would "be ignored in v3.5.0", and v3.5.0 shipped 2026-04-08 — the
instance line is now at v3.16.0 and the keys still take effect, so they
are live configuration, not dead letters. An explicit operator pin
should not be silently defeated by a per-user pick on some pages and
honored on others; whichever way the mechanism eventually goes, it must
behave identically everywhere until it is actually removed.
- user choice is not reduced in the general case: picks remain
authoritative for every provider the registry offers, and overrides
only exist where an operator explicitly wrote a literal URL.
If maintainers would rather have pick-wins (operator URL acts only as a
fallback default), flipping the precedence in `MC_resolveTile` is a
small,
well-tested change — say so in review and it will be reworked.
Related cleanup, out of scope here: the stale "ignored in v3.5.0"
deprecation warning in `cmd/server/config.go` should either be honored
or
reworded now that the project is past v3.16.0.
## No change without operator overrides or picks
## Verification
Unit (all deterministic, `vm`-sandbox based):
- new `tests/unit/test-tile-resolver.js` — 22 cases: light/dark pick
honored,
operator `lightDefault`, disabled-provider fallback, override
precedence,
lazy function-URLs, key-after-async-config, in-place swap without
stacking,
detached-map prune, multi-map single-listener sync, provider maxZoom;
listed in `test-all.sh`
- existing suites green: `test-issue-1420-tile-providers` (47),
`test-issue-1470-node-tile-helper` (6),
`test-issue-1614-tile-url-function`
(3), `test-frontend-helpers` (79), `test-settings-sync`,
`test-packet-filter` (105), `test-aging`
- every one of the 227 suites listed in `test-all.sh`, run individually:
all green except `test-manage-workflow` (2 failures that are local-env
only — CRLF bootstrap artifact — and pass in CI; `sh test-all.sh` aborts
there locally under `set -e`, which is why the suites were run one by
one this time; the scripted pipeline runs fully in CI)
- `eslint` on all six touched files: 0 errors
Browser (headless Chromium against this branch, `/api/config/client`
delayed
~800 ms to force the config race):
- light + `/#/nodes/<key>`: 12 keyless tile requests before config, 12
keyed
re-requests after — no reload, no stacked layer
- light pick `carto-voyager`: inset requests `rastertiles/voyager` from
the
start (parity with the main map)
- theme flip on an open node page: `dark_all` swaps in in place
- dark defaults unchanged (`dark_all`, keyed after config)
The Playwright E2E suite was not run locally — no Go toolchain on the
workstation to serve the backend — CI will cover it.
## Scope / follow-ups (intentionally not here)
- `home.js` mini-map (OSM only) and the geo-filter mini-maps in the
customizers are untouched on purpose: they are outside this bug's path,
and migrating them to `MC_resolveTile` later is mechanical
## Performance
Not a hot path: one registry lookup + string build per map creation and
per
provider/theme-change event (each is rare); no per-packet or per-render
work;
the listener footprint is O(1) app-wide (single shared listener pair + a
pruned registry), so there is no growth across SPA navigations.
Co-authored-by: OpenCode <git-llm@trix.moe>
Co-authored-by: OpenCode:GLM-5.3-Flash <git-llm@trix.moe>
Assisted-by: OpenCode:GLM-5.3-Flash eslint
---------
Co-authored-by: OpenCode <git-llm@trix.moe>
|
||
|
|
6dd62711bc |
docs(release): v3.17.0 release notes and changelog entry (#2390)
Release notes for v3.17.0: `docs/release-notes/v3.17.0.md` (used as the GitHub release body by `deploy.yml`) and the `CHANGELOG.md` entry. 24 commits since v3.16.0, counted with #2389, which is merged before this PR. Issues closed come from each PR's `closingIssuesReferences`. Docs only. After merge: tag `v3.17.0` on the merge commit once master CI is green and the `:edge` image carries that commit.v3.17.0 |
||
|
|
873e8b0e16 |
feat(rx-coverage): QR code and link to the operator's CoreDrive RX deployment (#2389)
## What An operator who runs a CoreDrive RX deployment (the web and Android companion app that feeds `clientRxCoverage`) can now point visitors of the Coverage page at it. - New optional config key `clientRxCoverage.appUrl`. `/api/config/client` sends it as `clientRxAppUrl`, only while `clientRxCoverage.enabled` is true and the value is an absolute `https` URL with a host name. Any other value is left out and logged once at startup with a `[config]` line. - `#/rx-coverage` shows a panel under the subtitle when the field is present: a QR code of exactly that URL (vendored `qrcode.js`, same call as the node detail QR), the URL as a link, and one line on scanning it in the app or opening it on a phone. Without the field there is no panel. The page still needs no login. - The subtitle named the app "CoreScope-RX" and linked to `github.com/efiten/corescope-rx`, the app's old repository name. It now says "CoreDrive RX" and links to `github.com/efiten/coredrive-rx`, or carries no link when the panel already shows the deployment. ## Why The CoreDrive RX Android app is not tied to one server: on first start it asks for the address of a deployment. Scanning a QR code on the Coverage page is the shortest way for a visitor to get there, including people without an account. Disclosure: CoreDrive RX (`efiten/coredrive-rx`) is my own project. ## Changes - `cmd/server`: `config.go` (`validClientRxAppURL`, startup warning), `types.go`, `routes.go`, `main.go`, `openapi.go`. - `public/rx-coverage.js` (`appPanelHtml`, `appLinkHtml`), `public/roles.js` (`window.MC_CLIENT_RX_APP_URL`), `public/node-reach-coverage.css` (existing CSS variables only). - Docs: `config.example.json`, `docs/client-rx-coverage.md`, `docs/api-spec.md`. ## Tests - 3 new Go tests in `cmd/server/client_rx_app_url_test.go`: URL validation over 15 inputs, the endpoint field in 6 config states, the startup warning. Full `go test ./cmd/server` passes. - `tests/unit/test-rx-coverage-app-qr.js` (real `rx-coverage.js` and `qrcode.js` in a vm): panel present and absent, QR content, HTML escaping, subtitle link in both cases, old names gone. Added to `test-all.sh`. All 6 `tests/unit/test-rx-coverage-*.js` pass. ## Not done - No E2E test. - Browser-checked on a local server with and without `appUrl` in the light theme only. The QR keeps black modules on white in both themes, unlike the node QR, which is inverted in dark mode; dark mode itself was not checked in a browser. - `cmd/ingestor` is unchanged; it does not need the URL. - The same change runs on analyzer.on8ar.eu (a fork build) since 2026-11-10. |
||
|
|
4140028165 |
fix(ui): keep slide-over close controls above navigation (#2304)
Fixes #2302. Navigation covered tablet slide-over close controls and intercepted pointer input. Two CSS substitutions put the panel/backdrop on the existing modal layers. The regression uses actual close/backdrop clicks and taps, retaining the existing keyboard, focus, scroll and resize assertions. Refreshed at `cd834802` by normally merging current master `d1bf4ca0`, with zero conflicts. The exact three-file feature patch and original RED/GREEN commits are preserved. Both feature tests are unchanged by this refresh, including the supported 180-minute fixture window. [Original RED job](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212233915/job/121225789154): 40 passes, 17 pointer assertion failures, zero skips. The job completed with failure before its aggregate run was cancelled. Merged #2289 resolves the old unrelated RF test failure. Validation: four 57-case slide-over runs; three focus/scroll cases; 20 supplemental cases; 224 frontend suites; related navigation/gesture checks; browser smoke with 149 passes/three existing fixture skips. Parent rerun passes 57+3 after an initial-response timeout; tests were unchanged. [Fresh CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42324795657) passes, including the repeat gate. Independent reviews found no introduced must-fixes; Kent Beck Gate: PASS. Existing observer-focus and packet-navigation problems are separately tracked in #2379 and #2380; runtime JavaScript is unchanged. Browser verified: disposable Go fixture at `http://127.0.0.1:14964`. E2E assertion added: `tests/e2e/test-slideover-1056-e2e.js`. No new settings, dependencies, requests or hot-loop work. ## Preflight overrides - check-red-commit: use the completed RED job’s 17 assertion failures; its aggregate run was subsequently cancelled. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
9e78e5c98d |
fix(live): show the newest certain node activity (#2283)
Fixes #2233. Live node details use the newest certain activity from the node and health responses for both status and Last Activity. Repeaters and rooms with only recent shared-prefix relay evidence remain Uncertain. Stale/missing values and a failed health request retain conservative behavior. Refreshed at `4d645dbc` by normally merging current master `d1bf4ca0`, with zero conflicts. The three-file feature patch and original RED/GREEN history are unchanged. [Original RED CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40190088413) remains; the preserved unit regression reaches 15 passes/eight assertion failures locally. Validation: all 224 frontend suites; 23 health-unit cases; eight health browser groups including 12 Live cases; four RF airtime groups; and 48 Live cases across desktop/phone and light/dark. Parent reran the unit and health browser suites. Backend trees are byte-identical to the already-qualified master; no new native Go result is claimed. The old unrelated RF test failure was reproduced during an actual theme refresh. Upstream #2289’s locator waits correctly while retaining its value/warning assertions; it is included in this refresh. Browser verified: disposable Go fixture at `http://127.0.0.1:14963`. Readable screenshots use the existing hidden-legend setting; the default desktop legend overlap is unchanged. E2E assertion added: `tests/e2e/test-health-status-e2e.js:259`. Fixed-size work for one opened panel; no extra requests, dependencies or settings. [Fresh CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42323634522) passes. Independent correctness, UI and test-quality reviews found no introduced must-fixes; Kent Beck Gate: PASS. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
fad303a4f4 |
docs: split the Docker minimum into running and building the image (#2361)
`docs/deployment.md` listed Docker 20.10+ as the minimum, without separating running the published image from building it. The Dockerfile needs BuildKit (`# syntax=docker/dockerfile:1` on line 1, `RUN --mount` from line 71), and BuildKit is the default builder only from Docker 23.0. On 20.10 to 22.x the legacy builder rejects `RUN --mount` unless `DOCKER_BUILDKIT=1` is set. Found while reviewing #2359. Changes: - `docs/deployment.md`: the requirements table gets two Docker rows, 20.10+ to run the published image, and 23.0+ (or 20.10+ with `DOCKER_BUILDKIT=1`) to build it, plus one sentence on why. - `docs/go-migration.md`: that guide builds the image, so its prerequisite becomes 23.0+ (or 20.10+ with `DOCKER_BUILDKIT=1`). Not verified: I did not run a build on Docker 20.10 or 22.x; the legacy-builder behaviour comes from Docker's documentation, not from a test here. Podman and buildah are not covered. Docs only. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9ddb29c946 |
fix/channels-mobile-header-overflow (#2382)
Fixes #2381. Drops the 56px header cap on mobile; title + Add on row 1, region pills on a sideways-scrolling row 2. See the issue for details --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
afbae53b71 |
ui(nodes): add direct map and packet navigation (#2295)
Fixes #2290; addresses U6 of #1154. Full node details, the side preview and tablet slide-over offer Open in Map and View packets through existing deep links. A shared renderer validates coordinates, including zero, and encodes identifiers without extra fetches. Closing a preview preserves newly selected destinations and titles; manual close returns to Nodes. Tablet targets stay separate, and modified clicks retain native behavior. Refreshed at `538cb169` by normally merging current master `d1bf4ca0`, with zero conflicts. The original five commits and exact seven-file feature patch are preserved. The [original RED CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40203064817) remains; the preserved RED also reproduces 39 assertions locally. Upstream #2333 now supplies the correction for the previous unrelated navigation-focus CI failure. Current validation: all 225 frontend suites; 52 focused unit checks; 31 shortcut browser cases in each theme; 24 additional title/native-click cases; navigation 60/60, branding 17/17 and slide-over 27/27. Parent independently reran 52 unit and 31 browser cases. Native server and ingestor suites pass. General browser smoke reports 149 passes and three existing fixture skips. Unchanged serial retries resolved two documented local test-server/navigation setup failures. Browser verified: disposable local Go fixture at `http://127.0.0.1:14954`. E2E assertion added: `tests/e2e/test-node-quick-actions-e2e.js`. Constant work for one detail view; no new requests, dependencies or settings. [Final refreshed CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42318272198) passes. Independent adversarial, UI and test-quality reviews found no must-fixes; Kent Beck Gate: PASS. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
a59b42d8c2 |
fix(observers): keep charts and controls within narrow pages (#2344)
Fixes #2341. Observer chart grids fit narrow screens, the doughnut stays square while resizing, and empty canvases/long comparison choices no longer widen the page. Radio formatting, requests and chart calculations are unchanged. Refreshed at `f2ad231e` with a conflict-free normal merge of current master `1577b57d`. Merged #2343 delayed-focus coverage and #2374 shared payload labels are preserved. In response to review, the layout test now checks one column at 760px and two at 900px, safely away from the exact breakpoint. All original geometry, reload, disposal and request-count assertions remain. Original RED proof is retained: [four chart-width failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42011115036/job/127190202787) and [four empty-data/control failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42016791855/job/127209299782). Current validation: all 226 frontend suites and 68 ownership browser cases pass. Parent independently reran four chart cases covering 32 layouts. Restoring current-base production through browser interception makes all eight layout cases fail their overflow assertions; tests remain sensitive to the bug. Browser verified: disposable current-base Go fixture at `http://127.0.0.1:14973`, desktop/mobile and both themes. E2E assertion added: `tests/e2e/test-observer-detail-ownership-e2e.js`. No new settings, dependencies, API calls or resize code. [Current CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42803853923) and independent adversarial, visualization and Kent/TDD review pass at `f2ad231e`. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
c03147cf0f |
fix(ui): share timestamp cells and retain future-clock warnings (#2387)
Partial fix for #1154 U10; follow-up to the maintainer's markup/warning note on #2377. Peer Interactions Last Contact now retains the same future-clock warning as Nodes and Packets. One shared HTML renderer in app.js owns the escaped timestamp, tooltip and existing warning markup; the three surfaces delegate to it. Existing wrapper names/fallbacks, timestamp preferences, placeholders and date algorithms are preserved. No new settings, dependencies, requests or render triggers; work stays constant per cell. Tests-only RED `28e09e7c` reached [three unit assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42800749635/job/129777366878) and [six browser missing-warning assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42800749635/job/129777366837). GREEN `27f9fa49` leaves those tests unchanged and passes 115 unit assertions, all 226 frontend suites and 10 browser cases. Parent independently reran the unit and browser checks. Twenty-two Nodes/Packets outputs match base exactly across 11 input/preference cases. Browser verified: disposable Go fixture at `http://127.0.0.1:14975`, desktop/phone with UTC/local and relative/absolute preferences. Visible warning/tooltip checks pass. The fixture's missing battery table and existing final-row bottom-navigation overlap are unchanged limitations. E2E assertion added: `tests/e2e/test-issue-2218-node-analytics-stale-e2e.js`. No production use of a test-only export, and no broader timestamp audit claimed. [Current CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42804968476) and independent adversarial, visualization and Kent/TDD review pass at `27f9fa49`. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
6c37fc59e1 |
chore: ignore root local worktree directories (#2388)
Partial fix for #1385 (PR-G). Root `_wt-*` worktrees and `.worktrees/` folders currently appear as untracked content. Add anchored, directory-only ignore rules for those two local worktree conventions. Nested folders and similarly named files remain visible. Validation: - Isolated Git repositories with empty global and local excludes: 21 before/after cases, 42 assertions passed. - No currently tracked path matches either new rule; all other tracked content and `.dockerignore` are unchanged. - Existing planning-file and attachment visibility is preserved; `git diff --check` passed. TDD config-change exemption: no test files modified. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42809414204) passes without test edits. No runtime, browser UI, or customizer behavior changes. Parent reran the Git checks and post-push browser smoke (3/3); served assets match the candidate. Three independent reviews pass. This covers only PR-G's transient worktree ignore audit. Other #1385 work, including PR-E documentation, remains outside this change; the issue stays open. Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
1577b57d67 |
docs(accounts): approved channels are backfilled since #2133 (#2385)
`docs/user-guide/accounts.md` said that messages received before a channel proposal is approved stay encrypted. Since #2133 that is no longer the case: when the key set grows, the ingestor re-decodes the stored rows it could not decrypt (`cmd/ingestor/key_backfill.go`), and the server tails `tx_rewrite_feed`, so no restart is needed. Checked on an instance running #2133: approving four hashtag channels triggered one pass (`[key-backfill] channel: 4 of 410 key(s) not yet applied to history`, then `pass complete, 801 row(s) rewritten`, 13 s), and the 801 rows are exactly the stored messages of those four channels, the oldest from three months before the approval. Docs only, one sentence. Not done: I did not check the other docs pages for the same claim beyond a grep for "stay encrypted" and "received before" in `docs/` and `README.md`, which found only this one. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d66a219bcb |
feat(cors): let a UI on another origin connect, static host previews included (#2362)
## What this does Server changes so the UI can be served from another origin, such as a Cloudflare Pages project and its previews, against an existing backend. It ships before the frontend, which is parked on [`backup/static-frontend-1a`](https://github.com/ppicazo/CoreScope/tree/backup/static-frontend-1a) and will come as its own PR. - **Exposed headers:** allowed origins get `Access-Control-Expose-Headers: Retry-After, X-CoreScope-Load-Status`. The UI reads both, and cross-origin they are `null` without this. - **Wildcard origins:** an entry may be `scheme://*.host[:port]`, such as `https://*.myproject.pages.dev`. It matches exactly one leftmost label (each deploy and branch preview) for both CORS and `/ws`, but not `myproject.pages.dev` itself. The part after `*.` needs at least three labels, so `*.pages.dev` is refused. Refused or malformed wildcards are dropped at startup with a log line. That floor is a sanity check, not a Public Suffix List lookup: `https://*.s3.amazonaws.com` passes and would admit every S3 website endpoint, so a wildcard belongs only over a host the operator controls. - **Case:** exact entries now match case-insensitively for CORS (was `==`), as they already did for `/ws`. - **Test only:** pins that the three anonymous POSTs (`/api/decode`, `/api/packets/observations`, `/api/paths/inspect`) accept a `text/plain` JSON body, which keeps them free of a preflight. Credentials stay off, so nothing a listed origin sends rides on cookies. Beyond GET, a listed origin, and so every deployment under a wildcard entry, can: - call those three anonymous POSTs as simple requests. That predates this PR; they compute an answer and write nothing. `Access-Control-Allow-Methods: GET, HEAD, OPTIONS` only governs preflighted requests. - with `userManagement.enabled`, use the existing bearer tier for companion linking (`/api/auth/device-token`, `/api/auth/me`, `/api/auth/logout`, `/api/account/settings`, `/api/account/companions*`) with POST, PUT and DELETE. Those routes need a device token in `Authorization`, not a cookie, but a preview deployment gets them too. On such a server, list a wildcard only if previews may use them. ## Configuring ```yaml - CORS_ALLOWED_ORIGINS=https://existing-embed.example,https://myproject.pages.dev,https://*.myproject.pages.dev ``` The env var replaces `corsAllowedOrigins`, so keep the origins already listed there. ## Testing - `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`. The new tests were written first and failed first. - In Chromium, the static build from the backup branch, served on `pr-1.proj.preview.localhost`, passed its split E2E (9/9, WebSocket included) against this server. - From the same setup, the project origin connected, a two-level subdomain was blocked, and `X-CoreScope-Load-Status` read `ready` where master gives `null`. ## Performance These run per request in the CORS middleware and the `/ws` handshake, and they allocate nothing. Same-origin requests skip them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
899a3ad0b1 |
fix(map): normalize saved viewport longitudes (#2378)
Partial fix for #2375. Restoring a saved longitude such as -352.7 previously put markers one world-width off-screen while tiles looked correct. Saved positions now use Leaflet’s native wrapping and the existing viewport validator; movement saves canonical coordinates. Invalid saved values fall back to configured defaults. Valid URL viewports and node highlighting keep precedence. This repairs saved state, not live dragging across world copies. No new settings, dependencies, requests, marker loops or render triggers; added work is constant per initialization/existing moveend. The reported-node URL diagnosis is separate: no-slash returns JSON when enabled and intentionally 404 when disabled; trailing-slash returns HTML. See #2372. The endpoint/gate remain unchanged; #2375 stays open for maintainer reconciliation. RED `cb453c0d` produced [14 passes and 10 intended CI assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42317069830/job/128232241526). Production fix `e6b70dfd` passed CI, 224 frontend suites, 24 browser cases, six estimated-marker and three mobile checks. Marker placement changes from x=262754/offscreen to x=610/visible in the same fixture. Review follow-up `99a4746e` changes only test data: configured and saved centers now differ, while invalid storage independently expects the default. Parent rerun passes 24/24; a browser-only mutation discarding saved coordinates now fails seven cases. All previous assertions and production files are retained. The single review finding is resolved. Browser verified: disposable Go fixture at `http://127.0.0.1:14956`, real Leaflet rendering. E2E assertion added: `tests/e2e/test-map-nodes-pagination-e2e.js:129`. Final follow-up CI is blocked by the unchanged observer disposal assertion (#2342; fix #2343). A maintainer must rerun the workflow; this remains draft. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
8420a7ac0f |
feat(map): identify historical foreign-advert flags (#2371)
Partial implementation of #730 M2; the issue remains open. The map marks nodes whose stored `foreign` Boolean is true in dot and hash-label modes. Titles, accessible names and popups explain that an advert was flagged outside the configured area at ingestion. This historical flag does not establish current location, bridging, an observer or a recent count. Analytics/history remain separate. The warning uses existing theme tokens and stays clear of nearby markers and phone controls. Footprint preparation adds O(n) work and numeric metadata to the existing marker list (around 2K nodes), with one typography read per applicable render and no per-marker DOM measurement or requests. Existing bounded-offset placement retains its worst-case quadratic search; this is not a speedup claim. No new settings or render triggers. RED: `4ad11148` produced [44 missing-warning assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42012004213/job/127193151694). Follow-up `c4a157de` produced [11 warning-occlusion assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42023035345/job/127230364384). GREEN `ad465e32`: parent browser check 66/66, 224 frontend suites, 38 existing map/route checks and CSS audit. Incorrect-copy controls also prove the tests reject a current-location inference. Browser verified: local Go fixture, desktop/phone, both themes. E2E assertion added: `tests/e2e/test-map-foreign-advert-e2e.js` covers keyboard access, missing/false flags, no-GPS, refresh, clustering, overlays, zoom and resize. ## Preflight overrides - check-red-commit: initial RED includes the new browser regression plus necessary CI/inventory registration, with no production changes; CI reached the intended assertions. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42026358900) passes. Three independent reviews were consolidated; both findings are fixed and parent-verified. Kent Beck Gate: PASS. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
|
65990873ad |
feat(map): deep-link the route path picker by observer or All view (#2376)
## Summary The map route deep link (`#/map?packet=<hash>[&obs=<id>]`) always opened in the canonical-path view. Two optional query params now choose the path picker's starting view, so a shared link shows exactly what the sender meant. | Param | Effect | |---|---| | `paths=all` | Opens the picker's **All** view (every unique path), the same as clicking **All** | | `observer=<pubkey>` | Full public key or a hex prefix of 8+ characters, case-insensitive. Picks that observer's **earliest** observation (by timestamp, then id), makes it the canonical route and isolates its path row, the same as clicking the row. "Back to packet" then links to that observation. | Examples: ``` #/map?packet=974ad4ecefbe3761&paths=all #/map?packet=974ad4ecefbe3761&observer=a1b2c3d4 ``` When params are combined, `obs=` wins over `observer=`, and `observer=` wins over `paths=all`. An unknown or malformed observer falls back to the current default view and logs a console warning. ## Changes - `public/route-view-utils.js`: two new pure helpers. - `resolveDeepLinkObservation(observations, params)` picks the observation and the initial view. - `pathGroupKey(rawHops, filterHops)` is the path picker's grouping key. It was written out twice in `route-view.js` and now lives in one place, which keeps the deep link and the picker rows in sync. - `public/map.js`: `loadRouteFromDeepLink` uses the helper and passes `initialPathView` through `drawPacketRouteMulti` to the renderer. - `public/route-view.js`: the picker count and rows use `pathGroupKey`. `sidebar._applyPathView` applies the view through the existing All and row click handlers, in the frame after the route's initial fit. ## Performance This doesn't touch a hot path. The work is one pass over a single packet's observations (tens at most) plus a scan of the picker rows, and it runs once per deep-link load. It adds no API calls. ## Config / customizer No new configurable values. ## Tests - New unit suite `tests/unit/test-map-deeplink-path-params.js` (29 checks). It covers prefix vs full key, case, short/non-hex keys, earliest-observation choice with tie-break, the precedence rules, `pathGroupKey`, and the source wiring. - `tests/unit/test-issue-1633-hide-1byte-hops.js` replayed a copy of the key logic. It now tests the real `pathGroupKey`. - New Playwright suite `tests/e2e/test-map-deeplink-path-params-e2e.js` (5 checks) on the seeded #1486 packet. It's registered in `scripts/non-unit-tests.json` and CI shard 3. With the `route-view.js` change removed, 3 of its 5 checks fail. - Local runs against the CI-prepared fixture: `test-all.sh` passes. Route-map a11y E2E 20/20, map mobile E2E 3/3, `test-e2e-playwright.js` 149/152 (3 skipped by the fixture, 0 failed). ## Browser validation Checked in a browser against a local server, using a scratch copy of the fixture that gives the seeded packet multi-hop paths through real GPS nodes. One observer had two observations, and the API listed the later one first. - `observer=<prefix>` picked that observer's earliest observation and drew only its path, with numbered markers and the map fitted to it. - `paths=all` drew the same picture as clicking **All** by hand. |
||
|
|
3100d7a4c9 |
fix(ui): route remaining payload label surfaces through payload-labels.js (#2373) (#2374)
Fixes #2373 ## Summary Finishes the #1799 / #1804 migration: every frontend surface that names a payload type now reads it from `public/payload-labels.js`. Node Analytics, Observer detail and Observer comparison showed GRP_DATA, MULTIPART and RAW_CUSTOM as `Type 6` / `Type 10` / `Type 15`. The map route sidebar had RAW_CUSTOM on id 12; the firmware uses `0x0F`. ## Changes - **`public/payload-labels.js`**: new `PayloadLabels.api.displayName(id)` returns the canonical short label, or `Type N` for ids the firmware does not define (e.g. a corrupted header byte stays visible as `Type 13`). It lives on `.api` only; the legacy root aliases are not extended. - **`public/node-analytics.js`, `public/observer-detail.js`, `public/compare.js`**: drop the local `PAYLOAD_LABELS` maps and use `displayName()`. - **`public/nodes.js`**: drop the local `PAYLOAD_TYPES` map; Recent Adverts uses `shortById()` and keeps its `Packet` fallback. - **`public/app.js`**: drop `PAYLOAD_TYPES`; `payloadTypeName()` uses `shortById()` and keeps its `UNKNOWN` fallback. - **`public/map.js`**: drop `PAYLOAD_TYPE_MAP`; `inferredType` uses `enumNameById()`, which fixes RAW_CUSTOM (15 → `RAW_CUSTOM` instead of `OTHER`). - **`public/packets.js`**: `typeName()` is now `displayName` (same output, one implementation). Consumers resolve labels at render time rather than at file load, so the many unit sandboxes that load these files without `payload-labels.js` keep working. `index.html` already loads `payload-labels.js` before all of them. ## Tests - **New `tests/unit/test-payload-labels-surfaces.js`** (listed in `test-all.sh`): - drift guard: fails if any `public/*.js` other than `payload-labels.js` pairs a numeric id with a short label or firmware enum name (it caught the `map.js` copy, which was not on the original list); - `displayName()` for every enum, string ids, and the `Type N` fallback; - loads the real `node-analytics.js` in a vm sandbox and asserts the doughnut labels for ids 6, 10, 15 and 13. - Before the fix: 11 failures. After: 82 passed. - **`tests/e2e/test-issue-1799-label-vocab-e2e.js`**: two new steps intercept `/api/nodes/:pk/analytics` and `/api/observers/:id/analytics` (the fixture has no GRP_DATA in the store), inject ids 6, 10, 15 and 13, and read the labels from `Chart.getChart()`. Against upstream `master`: 11 passed, 2 failed (`["Type 6","Type 10","Type 15","Type 13"]`). With this branch: 13 passed. - **Sandbox updates**: `test-compare-overlap.js` and `test-observer-detail-ownership.js` now load `payload-labels.js` before the page script, as `index.html` does. `test-issue-1418-raw-hex-extraction.js` asserted the hand-rolled `PAYLOAD_TYPE_MAP` literal; it now asserts that `inferredType` uses the canonical map and that no local map exists. - Unit suites, each `test-all.sh` entry run on its own: 223 pass on this branch vs 222 on `master` (+1 new suite). `test-manage-workflow.js` and `test-preflight-xss-gate.js` fail identically on `master` in my Windows environment (CRLF in the shell fixture, no `python`); they are unrelated to this change. - E2E, run against a local server on the CI fixture (freshened, seeded and migrated as in `deploy.yml`): `test-issue-1799-label-vocab-e2e.js`, `test-charts-fluid-1058-e2e.js`, `test-issue-1147-section-order-e2e.js`, `test-issue-1640-compare-discovery-e2e.js`, `test-issue-1648-m2/m3-icons-e2e.js` and `test-observer-detail-ownership-e2e.js` all pass. `test-e2e-playwright.js` fails one step (`Version info lives on Perf dashboard`, `#navStats` wait timeout) with the same result on `master` locally. ## Performance No hot-path change. Every lookup is still one object-key read per packet: the packets-table sort accessor calls `displayName(t)` instead of `TYPE_NAMES[t] ?? ...`. No new loops, requests or allocations. ## Browser validation Checked against the local fixture server, with the analytics responses patched in the page to contain GRP_DATA: - Node Analytics → Packet Types legend: `Group Data, Advert, Channel Msg, Direct Msg, Multipart, Raw Custom, Type 13` - Observer detail → doughnut and the first Recent Packets row read `Group Data` - Observer comparison → packet table rows read `Group Data` - Map route deep link (`#/map?packet=…`) renders the route panel; no new console errors 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6618820836 |
fix(nodes): show timestamp context for Last Contact (#2377)
Partial fix for #1154 U10. Peer Interactions Last Contact now shows timestamp context through the existing shared formatter: relative text with an absolute tooltip, or the reverse when absolute display is selected. Existing timezone, precision and custom-format preferences apply; missing/invalid values remain a dash. Other U10 surfaces and tracking items stay open. The production change only formats the existing cell. Work stays O(P) over the existing maximum 20 peers; no requests, settings, dependencies, caches or render triggers are added. Tooltip and text are HTML-escaped. RED `c47cce20` reached [11 unit assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42316165311/job/128229216777) and [six browser missing-tooltip assertions](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42316165311/job/128229216764). GREEN `2c29eccb` keeps those tests unchanged and passes 82 unit assertions, 10 focused browser cases (parent rerun), and all 224 frontend suites. Existing browser smoke reports 149 passes and three fixture skips. One separate navigation timeout passed on an unchanged serial retry. Browser verified: disposable Go fixture at `http://127.0.0.1:14953`, desktop/phone; tests retain the real API response and supply deterministic peer contacts. Its unrelated missing battery-metrics table is a fixture limitation. E2E assertion added: `tests/e2e/test-issue-2218-node-analytics-stale-e2e.js:123`. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/42317025356) passes. Independent adversarial, UI and test-quality reviews found no must-fixes; Kent Beck Gate: PASS. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |