mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-26 03:23:47 +00:00
e8f32df4dcef22e5c09c6d58ad9ecbda2d4c1e0a
351
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
e8f32df4dc |
feat(#1784): gate ingestor neighbor-edge creation on the path-trust threshold (rebase of #1863) (#1930)
Continues #1863. Three of the four commits are @Saarlandpower's and @SaarMesh-Bot's, authorship unchanged. The fourth is mine and is explained below. ## Why a rebase was needed #1863 was stacked on #1824, and #1841 merged instead. Both carried the same pathTrust base from different commits, which is why the two conflicted while each reported MERGEABLE against master. Cherry-picking #1863's own three commits onto master applied cleanly with no conflicts, which confirms its actual work was always independent of that duplicated base. ## The fourth commit, and a correction to something I got wrong The three commits do not build on master: ``` cmd/ingestor/main.go:455:23: cfg.GetPathTrust undefined (type *Config has no field or method GetPathTrust) ``` **#1824 added the pathTrust config and helper to both `cmd/server/config.go` and `cmd/ingestor/config.go`. #1841 carried only the server half** — one of its own commits is titled "remove ingestor side". I then closed #1824 as superseded by #1841, which is true for the server side and wrong for the ingestor side. Master has no pathTrust code in `cmd/ingestor/config.go` at all. The fourth commit restores that half, unchanged from `beae2c1c`: the `packetpath` import, the `PathTrust` field, the `PathTrustConfig` alias, `GetPathTrust`, and `cmd/ingestor/config_test.go` verbatim (28 lines covering the default, an explicit value, and a nil `*Config` receiver). That code is @Bjorkan's and @SaarMesh-Bot's from #1824, not mine; I only put it back. ## Verification - All three original commits cherry-picked onto `b3a306b8` with **no conflicts** - `cmd/ingestor` builds, and its `PathTrust|Neighbor|Config` tests pass - `cmd/server` `Neighbor|PathTrust|AnonReq|Edge` tests pass ## Interaction with #1929 #1929 moves `DefaultMinHashBytesForMapping` from 2 to 1. With that in, this PR's ingestor gate is a no-op by default and only takes effect when an operator sets `minHashBytesForMapping` to 2 or 3, which is the opt-in shape #1784 asks for. The two are complementary; merge order between them does not matter. @Saarlandpower @SaarMesh-Bot — your work, your credit. Say the word and I will close this and hand the rebase back, or push it to the #1863 branch if you would rather that stayed the vehicle. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: Saarlandpower <Mail@mathiaskasper.de> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> |
||
|
|
1720060284 |
fix(#1827): avoid per-observation SQL fetch in handleObserverAnalytics hot loop (#1829)
## Summary Fixes the CPU/DoS issue in #1827: observer detail pages were saturating CPU on busy observers — 6-7 concurrently loaded tabs pegged 12 cores for seconds, and auto-refresh made it self-sustaining. ## Root cause `handleObserverAnalytics` iterated every observation in the requested window and called `enrichObs()` per observation just to read `payload_type` and `decoded_json` for the `packetTypes`/`nodesTimeline` aggregates. `enrichObs()` also runs an on-demand SQL `SELECT resolved_path FROM observations WHERE id=?` (`fetchResolvedPathForObs`) and builds a full response map — both of which are unused by this aggregation loop. `resolved_path` is only actually consumed by the `<=20` kept `recentPackets` entries. Per the triage in #1827 (@carmack): *"Replacing `enrichObs(obs)` with a direct `s.store.byTxID[obs.TransmissionID].PayloadType` read (as sketched in the body) drops a map alloc + interface boxes per obs on the loop that saturated the operator's 12 cores. Byte-identical output. That's ~90% of the value."* This PR implements exactly that fast-path. ## Change - Aggregate loop (`packetTypes`, `nodesTimeline`): read `payload_type`/`decoded_json` directly off the transmission via `s.store.byTxID[obs.TransmissionID]` — no SQL, no per-obs map allocation. - `recentPackets` (`<=20` entries): unchanged, still calls `enrichObs()` since it needs `resolved_path`/`raw_hex`/etc. for display. - Output is unchanged: `packetTypes`/`nodesTimeline` are computed from the exact same underlying fields (`tx.PayloadType`, `tx.DecodedJSON`), just without the O(N) SQL round-trips. ## Scope This is the concrete hot-path fix from #1827's triage — not the broader `/api/observers/{id}/analytics` endpoint-split proposal in #1828, which (per that issue's discussion) is a separate P3 follow-up. #1828's own triage converged on this same `byTxID` fast-path as "the ground-work minimum" before any endpoint splitting. ## Testing - Existing `TestObserverAnalytics` passes unchanged. - Extended `TestObserverAnalytics/default` to assert `packetTypes` counts come out correct (`{"4":2,"5":1}` for the seeded fixture) via the new `byTxID` path, and that `recentPackets` still carries `resolved_path` where present (confirming the `enrichObs()` path for those 20 entries is untouched). - `go build ./...` and `go vet ./...` clean in `cmd/server`. - Full `go test ./...` in `cmd/server`: passes except 4 pre-existing test-order-dependent failures in `TestHandleNodePaths_*` (unrelated to this change — reproduced identically on a fresh, unpatched clone of `upstream/master`). --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
1441734991 |
fix(#1901): detectSchema fails loud instead of caching wrong schema mode (#1909)
Fixes #1901. Thanks to @MarekWo for the exceptionally thorough report — root cause, repro, and a prioritised fix checklist in one. This implements it. ## Problem `detectSchema()` swallowed any probe-query error with a bare `return`, so a single transient failure of the first `PRAGMA table_info(observations)` at startup left `isV3` (and the feature flags) at their zero value **for the entire process lifetime**. The server then ran v2 SQL against a v3 DB: Packets page empty, `/api/channels/<name>/messages` → 500, logs full of `no such column: o.observer_id`, while the database was perfectly healthy. Nothing re-checked the flag, so only a manual restart recovered it. ## Fix Works through the report's checklist: - **Don't swallow the error.** `detectSchema` now returns `error` and `OpenDB` aborts on it. `main.go` already `log.Fatalf`s on an `OpenDB` failure, so the supervisord/Docker restart policy retries and a transient cause clears on the next attempt — strictly better than serving a broken read API. - **Log the mode unconditionally** — `[db] schema mode: v3 (observer_idx)` / `v2 (observer_id)`. A clean startup log is now positive evidence detection ran, not just an absence of errors. - **Run detection on a single pinned connection** (`conn.Conn(ctx)`) rather than an arbitrary pooled one, so the startup race in the report's hypothesis can't quietly hand detection a fresh, not-yet-openable handle — if the connection can't be acquired, we fail loud. - **`Close()` no longer checkpoints the read-only handle.** `PRAGMA wal_checkpoint(TRUNCATE)` on a `mode=ro` connection always failed with `disk I/O error (778)` and looked like a storage fault on every shutdown (the report's aside). The ingestor (the writer) owns WAL checkpointing. The three near-identical PRAGMA scan loops are consolidated into one `schemaColumns()` helper that returns errors instead of ignoring `Scan` failures. ### On the "single source of truth" item The report suggests deriving `isV3` from `dbschema.TableHasColumn(...)`. I kept the PRAGMA-scan structure here because `detectSchema` sets six flags from three tables in a single pass; swapping to `TableHasColumn` would mean six separate probe calls and wouldn't actually be cleaner. The goal it was aimed at — never cache a false negative — is met by making the existing scan fail loud. Happy to switch to the single-probe-per-column shape if you'd prefer it. ### Honest note on the connection `conn.Conn(ctx)` pins *a* single connection for all four probes and fails loud if it can't be acquired; it does not guarantee the literal connection `Ping()` validated (`database/sql` doesn't expose that). The fail-fast is what actually closes the bug — a mis-detected schema aborts startup instead of persisting for the process lifetime. ## Tests - `TestDetectSchemaFailsLoudOnProbeError` — injects a probe failure through a `rowQuerier` and asserts the error propagates and `isV3` stays unset (the invariant the old bare-`return` violated). - `TestDetectSchemaV3AndV2` — covers both schema shapes through `OpenDB`. `go vet ./cmd/server` and `go build` are clean; targeted `go test -run 'DetectSchema|OpenDB'` is green. Heads-up on the full `go test ./cmd/server` run: a handful of `TestHandleNodePaths_*` / `TestHandleAnalytics*` tests return `503 index loading`, plus one intentional panic test — these fail identically on pristine `master` (`a06ac8ac`) with this branch stashed, i.e. they're pre-existing/timing-related and untouched by this change. Out of scope (per the issue): frontend behaviour when the API 500s. 🤖 Authored with [Claude](https://claude.com) · Co-Authored-By trailer on the commit. Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
0d6f59ab2d |
fix(#1864): decode ANON_REQ source pubkey instead of treating it like REQUEST (#1866)
Fixes #1864. ## Problem `PAYLOAD_TYPE_ANON_REQ` was effectively treated like `REQUEST`. The two differ on the wire: ``` REQUEST : <dest hash 1B> <source hash 1B> <hmac 2B> <encrypted> ANON_REQ: <dest hash 1B> <source pubkey 32B, full> <hmac 2B> <encrypted> ``` The decoders read the right bytes but surfaced the sender key as `ephemeralPubKey`, which meant: - `store.go`'s node indexer keys on `pubKey`/`destPubKey`/`srcPubKey`, so ANON_REQ packets were **not** indexed — they didn't show up on a node's packet view; and - the packets list "details" rendered a bare `anon → <destHash>`, throwing away the sender identity the packet actually carries. - the detail side-view byte breakdown fell through the REQ catch-all, mislabelling a nonexistent 1-byte "Src Hash" and placing MAC/Encrypted-Data at the wrong offsets (`+2`/`+4` instead of `+33`/`+35`). ## Fix **Backend** (`cmd/ingestor` + `cmd/server` decoders) - Surface the ANON_REQ sender key as `srcPubKey` (json) so it's indexed and resolvable. The frontend keeps a legacy `ephemeralPubKey` reader so packets decoded before this rename still resolve — no DB migration needed. - `TestDecodeAnonReqValid` now asserts the full 32-byte `srcPubKey`. **Frontend** - `hop-resolver.js`: new O(1) `nameForKey(pubkey)` using the existing `pubkeyIdx` (all nodes). - `getDetailPreview`: resolve the source pubkey to a node **name** when known, else show the first 8 hex chars — no more bare `anon`. - Detail side-view: explicit ANON_REQ breakdown — `Dest Hash (1B)` | `Src Public Key (32B)` (node-linked) | `MAC @+33` | `Encrypted Data @+35`. - Detail header `srcLabel` falls back to the resolved ANON_REQ sender. All rendered names are `escapeHtml`-wrapped. ## Testing - `go test ./...` green for both `cmd/ingestor` and `cmd/server` (incl. strengthened `TestDecodeAnonReqValid`). - `node --check` on `packets.js`; brace/paren balance + markers verified on `hop-resolver.js`. - No HTML sink lines added → XSS preflight gate unaffected; every interpolated name is escaped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
52d08214bb |
fix(#1749): decouple watchdog emit from blocking I/O (root cause) (#1853)
Closes the gap left by #1810: that PR added defer/recover around the watchdog per-source work so a **panic** inside emit cannot kill the loop, but the actual production incident is caused by emit **blocking**, not panicking. ## Root cause In production `emit` is `log.Print`. `log.Print`'s underlying `write()` can block indefinitely if the sink is backpressured (Docker JSON-file log driver falling behind under load, a full stderr pipe, journald hiccups, etc.). A blocked syscall is not a panic -- `recover()` does nothing for it. Because emit was called **synchronously** inside the per-source work, a single stuck `write()` froze the entire tick loop forever -- no further source was ever checked and no further tick was ever processed again. This exactly reproduces the original #1749 incident even after #1810 landed: 3 independent MQTT sources going silent within ~60s of each other (one shared dependency -- the watchdog goroutine itself -- died, not 3 independent paho clients), zero WATCHDOG log lines for the rest of the 75-minute window, every other goroutine in the process continuing to run fine (a hang, not a crash), and only a full container restart recovering it. ## Fix `newAsyncEmit` decouples "decide to log" from "perform the write": the watchdog loop now only ever does a non-blocking channel send. A single background goroutine drains the channel and performs the (potentially blocking) write. If that goroutine itself gets stuck, the bounded queue (256) fills and further sends are dropped -- counted via the new `WatchdogLogDropCount`, surfaced through `/api/mqtt/status` and the ingestor stats snapshot alongside `WatchdogLastTickUnix` / `WatchdogPanicCount`. Worst case under a persistent backpressure event is now lost log lines (visible and counted), not a silently dead watchdog (invisible and undetectable -- the actual #1749 failure). ## Tests - `TestNewAsyncEmit_NeverBlocksWhenWriterStuck_1749` -- floods emit() past queue capacity while the writer is permanently blocked; every call must return immediately and drops must be counted. - `TestMQTTStallWatchdog_LoopSurvivesStuckWriter_1749` -- end-to-end, wires `runLivenessWatchdogLoop` exactly as production does (via `newAsyncEmit` around a permanently-blocking `realEmit`) with 3 registered sources, reproducing the incident shape and asserting the loop keeps ticking regardless. - `TestRunLivenessWatchdog_ProductionWiringUsesAsyncEmit_1749` -- smoke-tests the real entrypoint starts, ticks, and stops cleanly. - `WatchdogLogDropCount` round-trip tests in both the ingestor stats snapshot and the server's `/api/mqtt/status` handler, mirroring the existing `WatchdogPanicCount` coverage from #1810. All pre-existing watchdog/liveness tests (#1749, #1810 r1, force-reconnect) continue to pass unmodified; full ingestor suite green (verified 5x consecutive runs for flake-freedom). Note: the server package has pre-existing test-suite-wide flakiness in unrelated `TestHandleNodePaths_*` tests (confirmed reproducible on unmodified master too, non-deterministic which subset fails per run) -- unrelated to this change and out of scope here. --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
6528c7ba3e |
fix(#1854): move relay last_seen touch to the ingestor — server writes have been no-ops since mode=ro (#1855)
Fixes #1854. Refs #1598, #1611, #1845.
## The bug
`cmd/server/db.go:54` opens SQLite `mode=ro` (#1283/#1289).
`touchRelayLastSeen` → `TouchNodeLastSeen` issues `UPDATE nodes SET
last_seen` on that handle. It has failed on every call since, with the
error discarded at the call site:
```go
if err := s.db.TouchNodeLastSeen(pk, ts); err == nil {
s.lastSeenTouched[pk] = now
}
```
`nodes.last_seen` has therefore tracked ADVERT arrivals only. Verified
on live.saarmesh.de (1388 nodes): 1362 have `last_seen` within one
minute of their own most recent ADVERT. Reproduced directly with the
server's DSN in #1854.
Secondary effect: `lastSeenTouched` is populated only in the success
branch, so the debounce never engaged — the server retried the failing
UPDATE for every resolved pubkey in every decode window.
## The fix
The writer moves to `cmd/ingestor`, which owns `nodes` per #1283/#1287
and since #1547 already resolves hop prefixes to full pubkeys for
`observations.resolved_path`. The touch hooks into that existing
resolution point, so there is no new IPC surface and no second resolver.
Only unambiguously resolved hops qualify — a 1-byte prefix collision
cannot keep a silent node alive.
I considered the `internal/mbcapqueue` snapshot handoff used for
#903/#1324 and did not need it: that pattern exists because the
capability computation lives in the server's analytics cycle. Path
resolution already happens in the ingestor, so a file handoff would add
a hop for nothing.
`Store.TouchRelayNodes`:
- monotonic guard in SQL (`last_seen IS NULL OR last_seen < ?`) —
out-of-order ingest never rewinds
- 5-minute debounce keyed on `rxTime`, matching the interval the server
intended
- UPDATE only — unknown pubkeys never create rows
- unparsable `rxTime` is a no-op rather than writing garbage into the
node directory
- `Stats.RelayTouches` for `/api/perf` visibility
- debounce records the *attempt*, not the row match, so an unknown
pubkey is not retried per observation
## Server-side removal
`touchRelayLastSeen`, `DB.TouchNodeLastSeen`, the `lastSeenTouched` map
and the now-unused `allResolvedPKs` decode-window map are deleted.
`readonly_invariant_test.go` gains `UPDATE\s+nodes\s+SET\s+last_seen`.
`cmd/server/touch_last_seen_test.go` and two tests in
`resolved_index_test.go` go with it. Worth stating why they were green
for months: they build their `PacketStore` on `setupTestDB`, which opens
read-write. The production constraint is the one thing they did not
reproduce, which is why the added invariant regex — not a replacement
unit test — is the right guard here.
## Tests
Five tests in `cmd/ingestor/relay_touch_test.go`, committed red first
(
|
||
|
efitenandClaude Opus 5
|
b3a306b81f |
fix(#1888): count only live observers in the store's /api/stats query (#1892)
Fixes #1888. ## The mismatch `/api/stats.totalObservers` and `/api/observers` counted different sets: | Source | Predicate | |---|---| | `cmd/server/store.go:2089` (store path) | `SELECT COUNT(*) FROM observers` — every row | | `cmd/server/db.go:336` (DB fallback) | `WHERE inactive IS NULL OR inactive = 0` | | `db.GetObservers()` → `/api/observers` | `WHERE inactive IS NULL OR inactive = 0` | `handleStats` uses the store path whenever a `PacketStore` exists (`routes.go:785`), which is every normal deployment. So the header count came from the unfiltered query while the Observers page listed the filtered set. The two stats implementations also disagreed with each other for the same database, which is a bug on its own. ## Reproduction The gap is exactly the observers the `observerDays` retention sweep has soft-deleted. On the instance I reproduced against: ``` GET /api/stats → totalObservers: 79 GET /api/observers → observers.length == 51 ``` ```sql SELECT 'all', COUNT(*) FROM observers -- 79 UNION ALL SELECT 'active', COUNT(*) FROM observers WHERE inactive IS NULL OR inactive = 0 -- 51 UNION ALL SELECT 'inactive', COUNT(*) FROM observers WHERE inactive = 1; -- 28 ``` 79 − 28 = 51. Same shape as the 82 vs 51 in the issue. ## The change One line: the store's stats query gets the same predicate the other two already use, so all three agree. ## Deliberately out of scope Two things the issue raises that this does **not** fix, called out so they are not mistaken for done: - **Config blacklist.** `buildObserversDefaultResponse` drops blacklisted observers in the handler loop (`routes.go:2752`), which no SQL count can see. A deployment with a non-empty `observerBlacklist` will still show a stats count higher than the list, by the number of blacklisted-but-live observers. Closing that needs config plumbing into the count and is a separate change — happy to follow up if wanted. - **Map controls.** The third surface named in the issue derives its count from node role aggregates (`roleCounts`), not from the observer set at all. That is a frontend concern and untouched here. ## Tests `cmd/server/observer_count_1888_test.go`, three cases, each watched fail first: 1. `TestStoreStatsTotalObserversExcludesSoftDeleted` — `TotalObservers = 5, want 4` 2. `TestStoreStatsTotalObserversMatchesObserverList` — `stats totalObservers = 5 but /api/observers lists 4` 3. `TestStoreAndDBStatsAgreeOnTotalObservers` — `store path reports 5 observers, DB fallback reports 4` The fixture includes a row with `inactive = NULL` alongside `inactive = 0` and `inactive = 1`, since `GetObservers` treats NULL as live and only the `1` may be excluded. `cd cmd/server && go test ./...` → ok (87s). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
SaarlandpowerandClaude
|
0352c9a287 |
fix(clock-skew): restrict per-node skew to self-originated adverts (#1816, #1818) (#1820)
Closes #1816. Closes #1818 (confirmed duplicate of #1816 by the triage bot). ## Root cause `byNode` is an involvement index (`indexResolvedPathHops`, `store.go:1696-1705`, #1558/#1352): a transmission is indexed under every relay-hop pubkey found in an observation's `resolved_path`, not just its originator. `getNodeClockSkewLocked` (`clock_skew.go:489`) iterated every ADVERT transaction under a pubkey without checking who actually signed it, so a relay inherited the clock skew of every broken-clock node it forwarded as if it were its own. This produced: - Fleet-wide false `no_clock`/`bimodal_clock` classifications on healthy relays whose only "bad" samples were adverts they merely relayed. - Bit-identical `RecentMedianSkewSec` "clusters" across unrelated relays that all forwarded the same broken-clock originator. - Single relays showing a multi-day skew even though their own self-adverts are healthy, because 1-2 relayed adverts from a broken originator landed in the tail of their small recent-window sample (the #1818 "island" repro from @cwichura). ## Fix Add `txOriginatedBy(tx, pubkey)`: ADVERTs are self-signed, so `decoded["pubKey"]` is the originator per protocol (case-insensitive compare as a defensive measure). Apply it as a guard in both the main skew-aggregation loop and the per-hash evidence loop in `getNodeClockSkewLocked`. `byNode` itself is untouched — #1558/#1352 still rely on the broader involvement index for other consumers. ## Tests - Existing `clock_skew_test.go` / `clock_skew_issue1094_test.go` / `clock_skew_issue1285_test.go` fixtures built synthetic ADVERT transactions without a `pubKey` field and seeded `s.byNode` directly, bypassing the normal `indexByNode` path where every real ADVERT carries `pubKey`. Added `pubKey` to each fixture so it reflects a self-originated advert, which is what these tests already intended to represent. All pre-existing tests pass unchanged in behavior. - New `clock_skew_issue1816_test.go`: - `TestTxOriginatedBy` — unit coverage of the new guard (self, foreign, missing pubKey, case-insensitivity). - `TestIssue1816_RelayDoesNotInheritOriginatorSkew` — a relay with healthy self-adverts plus relayed adverts from a broken-clock originator (matching the report's +100.5k s band) must report `ok` severity based only on its own adverts. - `TestIssue1816_PureRelaysReportNoSkew_NoBitIdenticalCluster` — five relay pubkeys that only ever forward a broken originator's advert (never self-advert) must report `nil`, not a bit-identical copy of the originator's skew. - `TestIssue1818_TwoForeignAdvertsDoNotPoisonIslandNode` — reproduces the cwichura island scenario: 8 healthy self-adverts + 2 foreign adverts at ~10 days skew must not flip severity or pollute `RecentMedianSkewSec`. Full suite: `go test ./...` passes (one pre-existing, unrelated flaky test — `TestHandleNodePaths_PrefixCollision_1352`, an index-loading race — reproduces intermittently on unmodified `master` too). Operator context: running CoreScope for SaarMesh (SaarLorLux, DE/FR/LU, 800+ nodes); this bug was surfacing as fleet-wide clock-skew false positives on our infra nodes. Co-Authored-By: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
9ef4179ef1 |
fix(release): report correct version on fast-path retagged images (#1807) (#1814)
Fixes #1807. Implements the fix path from the triage (env → image-version file → baked version, zero rebuild cost): ## Changes **`cmd/server/main.go` — `resolveVersion()` fallback chain** 1. `CORESCOPE_VERSION` env (operator override) 2. `.image-version` file in the working dir (`/app` in the container) — mirrors the existing `.git-commit` pattern in `resolveCommit()` 3. ldflags-baked `Version` 4. `"unknown"` Edge builds are unaffected: no env, no file → baked `"edge"` as before. **`.github/workflows/release-fast-path.yml` — retag step** Instead of a plain `crane tag :edge → :vX.Y.Z`, the fast path now runs `crane mutate` on `:edge` with: - `--append` of a deterministic one-file layer containing `/app/.image-version` = `vX.Y.Z` - `--label org.opencontainers.image.version=vX.Y.Z` - `--tag :vX.Y.Z` `vX.Y`, `vX` and `latest` are then pointed at the mutated image. Still no rebuild — the mutation is a manifest + single ~100-byte layer operation. ## Notes - The release tags no longer share the exact digest with `:edge` (they carry one extra layer); the fallback SHA check is unaffected since it compares the `org.opencontainers.image.revision` label against `github.sha`. - The layer tar uses `--owner=0 --group=0 --mtime='UTC 2020-01-01'` for reproducibility. - Operators can also fix existing deployments immediately with `-e CORESCOPE_VERSION=v3.9.2`, no image change needed. ## Testing - `go build ./cmd/server` + `go vet` clean (golang:1.24) - Workflow YAML validated - Reporter context: running the affected v3.9.2 fast-path image in production (live.saarmesh.de), happy to verify the next tagged release end-to-end. --------- Co-authored-by: Mathias Kasper <fallisaar@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: SaarMesh-Bot <bot@saarmesh.de> |
||
|
Jesper B
|
8c48f2b49a |
feat(#1784): wire display consumers to path trust threshold (#1841)
Fixes #1784 ## Summary Step 4 of the #1784 multi-PR project — wires all frontend display consumers to respect the configured `pathTrust.minHashBytesForMapping` value. **Depends on:** #1840 (must be merged first — provides `MC_meetsPathTrust` / `MC_pathBelowTrust` / `MC_getPathTrustThreshold` helpers and `PATH_TRUST` global). ## Changes ### `map.js` — trust-gated route display - `drawPacketRoute` checks `MC_pathBelowTrust` before drawing polylines - When all path hops are below the configured threshold, shows a "Route not displayed" message with config guidance instead of speculative polylines - Cleans up the trust message control when a new route is drawn ### `analytics.js` — subpath trust filtering - `renderTable` in `renderSubpaths` filters route patterns whose hops are below trust threshold - Combined filter logic: both the 1-byte hide toggle AND trust threshold are applied together - Info line shows which filters are active ("1-byte hide + 2-byte trust", etc.) - No-data message explains which filters caused exclusion ### `route-view.js` — speculative path annotation - Path picker groups tagged with `belowTrust` flag when any hop doesn't meet the configured threshold - Speculative paths show `(speculative, <N-byte hops)` annotation with tooltip explaining the trust threshold - Tooltip includes guidance on changing `pathTrust.minHashBytesForMapping` in config.json ### `live.js` — Paths Through widget - `_pathHopsBelowTrust()` helper checks whether all hops in a path are below trust threshold - Widget fallback message distinguishes between "1-byte filtered" (display toggle) and "N-byte trust threshold" (server config) - Message includes the config key for operators to adjust ### `nodes.js` — confidence weight adjustment - `modeWeight` initialization considers the trust threshold - Hash modes below threshold get zero confidence weight - Bucket-0 (legacy/unknown) excluded at threshold >= 2 per #1784 bucket-0 policy ## Testing `test-issue-1633-hide-1byte-hops.js`: - 5 new source-grep guards verifying each consumer references the trust threshold helpers - All 26 tests passing (21 existing + 5 new) ## Files changed (6 files, +136/-6) ``` public/analytics.js | 28 ++++++++++++++++++--- public/live.js | 19 ++++++++++++++- public/map.js | 21 ++++++++++++++++ public/nodes.js | 8 ++++++ public/route-view.js | 16 +++++++++++- test-issue-1633-hide-1byte-hops.js | 50 ++++++++++++++++++++++++++++++ ``` --- **Depends on:** #1840 **Written by:** DeepSeek V4 Pro in Max Mode |
||
|
efitenandClaude Opus 5
|
4c45dec79f |
fix(#1902): don't attribute transported scopes from the 1-byte hop prefix (#1903)
Fixes #1902. ## The bug `byPathHop` is keyed on the raw hop string from `path_json`, and both relay-info paths look up the full pubkey **and** fold in `key[:2]` — the 1-byte wire prefix. `TransportedScopes` (#1751) accumulated over that folded set, so every node sharing a pubkey first byte reported the same scopes. On the live network all four active nodes with prefix `f7` returned an identical set: ``` f79616... BE repeater ['#be','#be-van','#de','#de-nw','#nl'] f7e718... BE repeater ['#be','#be-van','#de','#de-nw','#nl'] f788ad... BE repeater ['#be','#be-van','#de','#de-nw','#nl'] f752c2... DE/NRW repeat. ['#be','#be-van','#de','#de-nw','#nl'] ``` Their real sets, from unambiguous full-pubkey hops over the same 7 days, are disjoint: ``` f79616... (BE) #be 471, #eu 9, #nl 6, #de 3, #be-van 1 f752c2... (DE/NRW) #de 13, #de-nw 11 f7e718... (BE) (none) ``` A sysop reads a scope badge as a statement about how their repeater is configured, so a Belgian repeater badged `#de-nw` is a wrong answer, not an imprecise one. ## The change The prefix fold stays for the counters — that is the documented #662 trade-off, "a possible over-count for clearly false zeros", and `RelayCount1h/24h`, `LastRelayed` and `UnscopedRelayCount24h` are magnitudes where an over-count is tolerable. Scopes are not a magnitude. A 1-byte hop names one of N nodes and cannot substantiate a categorical claim. Entries reached only through the prefix bucket are now flagged (`relayEntry.viaPrefix` / a `viaPrefix` argument to the bulk `visit` closure) and excluded from scope accumulation only. Both computation paths are changed together so `/api/nodes` (bulk) and the node-detail endpoint (per-node) stay in parity: - `cmd/server/repeater_liveness.go` — `collectRelayEntriesLocked` / `computeRelayInfoFromEntries` - `cmd/server/repeater_enrich_bulk.go` — `computeRepeaterRelayInfoMap` The `public/nodes.js` tooltip is updated to describe what the field now actually means. Attribution does not collapse: `observations.resolved_path` carries full pubkeys for ~27% of observations on the live instance (408k of 1.54M over 7 days), and those rows produce the correct per-node sets above. A node with no resolved hop yet shows no badge rather than a borrowed one. ## Tests `TestTransportedScopes_CrossBucketFold` pinned the old behaviour ("a scope seen only in the prefix bucket must surface on the full key"), which is the bug. It is replaced by `TestTransportedScopes_PrefixBucketNotAttributed`, which asserts on **both** paths that: 1. a scope evidenced only by a 1-byte hop is not attributed; 2. a scope also present under the full key still is; 3. `RelayCount24h` still counts all three packets — narrowing scopes must not narrow the counters, i.e. the #662 fold is untouched. Red before the change, green after. ``` cd cmd/server && go test ./... ok github.com/corescope/server 98.2s node test-packet-filter.js 92 passed, 0 failed node test-aging.js 18 passed, 0 failed node test-frontend-helpers.js 625 passed, 2 failed ``` The two frontend failures (`favStar returns filled star for favorite`, `favStar returns empty star for non-favorite`) and `cmd/ingestor`'s `TestWriteStatsAtomic_SymlinkAtDestIsReplaced` are **pre-existing** — I ran them on a pristine `upstream/master` worktree and got byte-identical results (the ingestor one is a Windows symlink-privilege limitation, not a code failure). ## Perf No new work in any loop. The bulk path gains one bool argument to an existing closure and one `&& !viaPrefix` on a branch that already ran; the per-node path gains one bool field on `relayEntry`, which is stack/slice-local and not retained. Same complexity, same allocations. ## What I could not verify end-to-end, and why I built a fixture from live data (2512 nodes, 17k transmissions, 529k observations, including all eight `f7` nodes) and ran the before/after binaries against it. Neither reproduced the live field — both returned no `transported_scopes` and `relay_count_24h: 0` for every node. That turns out to be a **separate cold-start bug**: `LoadChunked` calls `indexResolvedPathHops` per observation while scanning chunks, which adds full-pubkey keys to `byPathHop`, and then the post-load block at `cmd/server/chunked_load.go:459` calls `buildPathHopIndex()`, which begins with `s.byPathHop = make(...)` and rebuilds from raw hops only. Every resolved full-pubkey key from the scan is discarded: ``` [store] Built path-hop index: 2924 unique keys <- raw hops only [store] LoadChunked: 17056 transmissions (527331 observations) ``` So on a freshly started server the full-pubkey buckets are empty and only refill from live ingestion. That is being filed separately; it is orthogonal to this change, but it does mean `transported_scopes` will be sparse for a while after any restart until it is fixed. This PR is therefore verified by unit tests on both computation paths plus the live-data derivation above, not by a local end-to-end run. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Joel ClawandJoel Claw
|
f49e3fcc26 |
perf: use cached ParsedDecoded() instead of repeated json.Unmarshal (#1871)
## Problem
`StoreTx.ParsedDecoded()` already caches the result of `json.Unmarshal`
on first call via `sync.Once`. However, 13 call sites in `store.go` and
1 in `routes.go` were independently unmarshaling `DecodedJSON` into
local `map[string]interface{}` variables on every access, completely
ignoring the cache.
## Impact
For a store with 50k+ transmissions, each analytics endpoint that
iterates all packets re-parses 50k JSON strings per request. With
multiple endpoints, this means hundreds of thousands of redundant
`json.Unmarshal` calls per page load, each allocating new maps and
slices.
## Fix
Replace each `var d map[string]interface{};
json.Unmarshal([]byte(tx.DecodedJSON), &d)` with `d :=
tx.ParsedDecoded()`, which returns the cached parse result (parsed once,
reused forever).
### Call sites changed (14 total):
- `untrackAdvertPubkey` — advert PK extraction during eviction
- Ingestion path — payload field for API responses
- `evictStaleInternal` — node cleanup during eviction
- `GetAnalyticsTopology` — node PK extraction
- `GetAnalyticsHashCollisions` — advert PK extraction
- `GetAnalyticsDistance` — region node PK building
- `resolveAreaNodes` — node PK extraction
- `GetRecentPackets` — payload field for API response
- `routes.go` byType grouping — type field extraction
### Left unchanged (3 sites):
Three call sites that unmarshal into typed structs (`grpDec`,
`decodedMsg`, `decodedGrp`) cannot use `ParsedDecoded()` since they need
specific struct types.
## Testing
- `go build` passes
- No behavior change — same data, same logic, just avoids redundant
parses
---------
Co-authored-by: Joel Claw <358739783+Joel-Claw@users.noreply.github.com>
|
||
|
Kpa-clawbotandcorescope-bot
|
59cf5130d1 |
fix(1838): fold non-transport routes into scope-stats Unscoped (#1842)
Fixes #1838 ## Problem `/api/scope-stats` reported 100% scoped whenever any region was configured. Reporter noticed on a scopeless instance that "unscoped" was always zero — the pie visual is misleading to operators deciding on `denyf *`. ## Root cause `cmd/server/db.go:22` restricted the entire scope-stats denominator to `route_type IN (0, 3)`. Per firmware `docs/packet_format.md § Route Types`: - `0` = `TRANSPORT_FLOOD` - `1` = `FLOOD` - `2` = `DIRECT` - `3` = `TRANSPORT_DIRECT` Only routes 0 and 3 carry `transport_code_1` (transport-level scope). Routes 1 and 2 are inherently unscoped by protocol. The existing SQL was correct for the "how many transport-scopable routes are actually scoped" question, but the denominator was silently promoted to "all traffic" in the UI. Bonus: the comment on `routeTypeTransportSQL` labelled routes 0+3 as "FLOOD (0) and DIRECT (3)" — wrong on both counts. ## Fix - `cmd/server/db.go` — corrected the `routeTypeTransportSQL` comment; added `routeTypeNonTransportSQL = "route_type IN (1, 2)"` alongside it. - `GetScopeStats` runs a second `COUNT(*)` over `route_type IN (1,2) AND first_seen >= ?` and folds that count into `Summary.Unscoped`. Same index path as the existing query — one extra scan per `/api/scope-stats` call (cached 30s per triage's carmack finding). - `public/analytics.js` — Scopes tab header explains the denominator (all observed transmissions) and which route types carry scope. Card notes now render `X% of all traffic` for Scoped/Unscoped and `X% of scoped` for Unknown Scope so the pie's denominator is explicit. ## TDD - Red: `5554ffe4` — extended `TestGetScopeStats` + `TestHandleScopeStats` with `route_type=1` and `route_type=2` rows and asserted `Unscoped = 3` (1 transport-NULL + 2 non-transport). Ran the tests and confirmed assertion failure (`Unscoped = 1, want 3`). - Green: `ebbb9253` — implementation + label copy. Full `go test ./cmd/server/...` passes (54s). ## Preflight overrides - check-branch-clean: justified — cross-stack fix by design (backend semantics change + matching frontend label copy). All 4 files are exactly the surface the triage comment identified. ## Verification - `go test ./cmd/server/...` — 54s, all pass. - Firmware confirmation: `firmware/docs/packet_format.md:20-24` (route type table). ## Files touched - `cmd/server/db.go` — comment fix + second COUNT query. - `cmd/server/db_test.go` — extended fixture. - `cmd/server/routes_test.go` — extended fixture + isolate from seed data. - `public/analytics.js` — labels and header copy. --------- Co-authored-by: corescope-bot <bot@corescope.dev> |
||
|
|
d60188e481 |
refactor(1828): split handleObserverAnalytics into 5 helpers + byTxID fast-path (#1839)
## Summary Phase A of #1828: extract the 5 aggregate builders in `handleObserverAnalytics` into pure helpers in a new `cmd/server/observer_analytics.go`. Handler becomes a snapshot + filter + 5 composed calls. Also adopts the `byTxID` direct-read in `buildPacketTypes` (issue body's core observation): the payload-type histogram no longer allocates a full `enrichObs` map + interface-boxed fields just to read `tx.PayloadType`. That's the ~90% perf win the triage called out. Scope is exactly Phase A per the second triage comment. Phase B (sub-endpoints, caching, SQL migration) is deferred to a follow-up. ## Byte-identical output - Timeline / NodesTimeline: same key set, same sort, same labels. - PacketTypes: same keys/counts. Both legacy (`enriched["payload_type"].(int)`) and new (`tx.PayloadType == nil` guard) skip obs whose tx is missing or `PayloadType` is `nil`. - SnrDistribution: same 2-unit floor bucketing (negative-side rounding preserved), same ascending sort. - RecentPackets: still the first 20 enriched observations (`enrichObs` kept only here, where the extra fields are actually needed). ## TDD - Red commit: `9dc62f43` — 7 unit tests fail on assertions (not build errors) against stubs. - Green commit: `8d41011d` — implementations + handler rewire. All new tests + existing `TestObserverAnalytics*` handler tests pass. ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → clean (all 8 hard gates + 3 warnings pass). ## Non-goals - No new endpoints. - No SQL migration. - No public API signature change. - Snapshot count unchanged (still one under RLock, per #1481 P0-2). Fixes #1828. --------- Co-authored-by: fix-1828-bot <bot@corescope.local> Co-authored-by: clawbot <bot@corescope> |
||
|
Michael J. ArcanandWaydroid Builder
|
d2ef624c2e |
feat(api): flood_advert_count_7d on the node detail endpoint (#1831)
Adds, per node, how many distinct FLOOD adverts it originated in the last 7 days. Zero-hop adverts (route_type DIRECT) are excluded, so a nearby observer hearing a node's cheap local adverts does not inflate the number - the existing advert_count mixes both kinds and cannot tell a chatty flooder (mesh-wide airtime) from the recommended 240-minute zero-hop cadence (local only). Consumers (the ArcScope repeater advisor) rate advert hygiene against the community practice of one flood advert every ~49h; with the mixed total, a correctly configured repeater looked chatty whenever an observer sat within zero-hop range. Implemented like the relay-liveness fields: a pure, unit-tested counter over (first_seen, route_type, hash) entries with the same timestamp parsing and hash dedup, fed by a from_pubkey-indexed query capped at the 2000 most recent advert rows. The flood route-type constant is named advertRouteTypeFlood so this merges independently of the open unscoped-relay PR (#1823). --------- Co-authored-by: Waydroid Builder <build@waydroid.local> |
||
|
Michael J. ArcanandWaydroid Builder
|
56fe844871 |
test: dedupe the unscoped-relay tests via a shared fixture (#1832)
Follow-up to #1823: TestRepeaterUnscopedRelayCount and its _Bulk twin were ~30 verbatim lines apart (DB, node insert, store seeding, assertions), differing only in the lookup under test - seeding changes had to land twice. Both now use a shared seedUnscopedRelayFixture + assertUnscopedCounts and contain only their respective lookup call. No behaviour change; the relay-liveness suite passes. Co-authored-by: Waydroid Builder <build@waydroid.local> |
||
|
Michael J. ArcanandWaydroid Builder
|
bd0a58e14c |
feat(api): add unscoped_relay_count_24h per-node field (#1823)
## What Adds a per-node API field `unscoped_relay_count_24h` on repeater/room nodes: the number of the node's 24h relay-hops that were unscoped floods (route_type == ROUTE_TYPE_FLOOD). A strict subset of relay_count_24h. ## Why A well-configured repeater runs `flood.max.unscoped 0` and should not rebroadcast unscoped floods — each one is re-sent by every repeater that hears it, so one packet turns into mesh-wide traffic. Exposing this lets clients (the ArcScope repeater advisor) detect and flag that base-config problem from observed packets. ## How Computed like relay_count_24h in both paths (bulk /api/nodes + per-node detail) with a route_type==FLOOD filter; reuses the byPathHop index, no migration. Wired into both handlers + OpenAPI schema + unit tests (per-node and bulk). Co-authored-by: Waydroid Builder <build@waydroid.local> |
||
|
|
096e16409c |
fix(#1741): wrap test-DB insert loops in a single transaction (#1819)
## Fixes #1741 `TestBoundedLoad_OldestLoadedSet` (and any test building a 5000-row fixture) hung/timed out, blocking reliable `go test ./cmd/server` and CI. ## Root cause The four test-DB builders in `cmd/server/bounded_load_test.go` (`createTestDBAt`, `createTestDBWithObs`, `createTestDBWithAgedPackets`) inserted rows in a loop with no `BEGIN`/`COMMIT`. With the pure-Go `modernc.org/sqlite` driver every `Exec` auto-commits → one fsync per row → ~2N fsyncs for N transmissions (tx + obs). At `numTx=5000` that's ~10k fsyncs and the fixture blows past the test timeout. Sibling tests with `numTx<=3000` happened to stay under the timeout, so only the 5000-row cases visibly hung. ## Fix Wrap each insert loop in a single `BEGIN`/`COMMIT` so the whole fixture build becomes one commit. Fixtures now finish in well under a second regardless of `numTx`; the tests' actual assertions (`oldestLoaded` set, newest-first ordering, bounded load) are exercised instead of the timeout masking them. Also made the prepared-statement `Exec` calls check their error (previously discarded) so a failed insert surfaces instead of silently leaving the DB short. No production code changed — test infrastructure only. ## Verified - `TestBoundedLoad_OldestLoadedSet`: **0.18s** (was: 30s timeout / FAIL). - Full `TestBoundedLoad*` + retention group: passes in ~1.2s. - `go test ./...` in `cmd/server`: exit 0 (no longer blocks on this test). Co-authored-by: Waydroid Builder <build@waydroid.local> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
6a32ec2b2d |
fix(#1729): preserve firmware-default Public channel (0x11) in analytics (#1817)
## Fixes #1729 The firmware-default **Public** channel (channel-hash byte `0x11` = 17) was rendered as an opaque **"Encrypted (0x11)"** row at the bottom of the analytics Channels tab, despite the key being well-known and builtin. ## Root cause `computeAnalyticsChannels` applied the #978 rainbow-table validation (`SHA256(SHA256("#name")[:16])[0]`, the **hashtag** hash scheme) to every decoded channel name. The Public channel is a **PSK** channel whose hash byte is key-derived (`SHA256(key)[0]` = 17), not hashtag-derived (`186` for `#Public`). So the ingestor-decoded name `"Public"` failed the hashtag check and was discarded, the row forced to `encrypted=true, name="ch17"`. ## Fix Trust the ingestor's `decryptionStatus`. The ingestor already persists `decryptionStatus:"decrypted"` when it decoded a packet with a real key (PSK), and `"no_key"` / `"decryption_failed"` otherwise. When the packet is `decrypted`, skip the hashtag hash check and keep the name — it came from a key-based decryption, not a rainbow-table lookup. The #978 mismatch rejection still applies to non-decrypted packets, so rainbow-table collisions are still caught. Frontend needs no change: `encrypted=false, name="Public"` lands in the "Network" group (top), not "Encrypted". ## Tests - `makeGrpTx` gains `makeGrpTxWithStatus` companion to set `decryptionStatus`. - `TestComputeAnalyticsChannels_PublicChannelPreserved`: hash 17 / "Public" / `decrypted` → name stays `"Public"`, `encrypted=false`. - `TestComputeAnalyticsChannels_UndecryptedNameStillValidated`: a non-`decrypted` name failing the hashtag check is still downgraded to `ch17` (#978 regression guard). All channel-analytics tests pass; `go build ./...` clean. Co-authored-by: Waydroid Builder <build@waydroid.local> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
750b8742a7 |
fix(staging-compose): decouple in-container mosquitto from standalone broker (#1813)
Red commit: `3898dbc5` (verified locally — CI run URL pending)
## Problem
A standalone `mqtt-broker` container (`eclipse-mosquitto:2`) was
provisioned out-of-band on the staging VM. It now owns MQTT, is attached
to external docker network `meshcore-net`, and binds host port `8883`.
The current `docker-compose.staging.yml` still:
- Publishes `1883:1883` on the host (dead weight; conflicts the moment
the broker moves to that port).
- Defaults `DISABLE_MOSQUITTO=false`, so the in-container mosquitto
burns RAM and briefly contests the `mqtt-broker` docker DNS name on cold
start.
- Doesn't join `meshcore-net`, so the ingestor can't resolve
`mqtt-broker:1883` via docker DNS without manual surgery.
## Fix (`docker-compose.staging.yml` only)
1. Remove the `1883:1883` host port publish from `staging-go`.
2. Flip `DISABLE_MOSQUITTO` default from `false` to `true`. Operators
can opt back in with the env var.
3. Attach `staging-go` to both `default` and `meshcore-net`; declare
`meshcore-net` as `external: true` so the file never tries to
create/destroy operator state.
Healthcheck and Caddy/443 plumbing untouched (out of scope).
## Test added (TDD framing: Option A — Go shape-asserts)
`cmd/server/staging_compose_broker_test.go:1` adds four regex-based
assertions on the compose file shape:
- staging-go does **not** bind port `1883` in ANY form (quoted/unquoted
short form, or long-form `target: 1883` / `published: 1883`).
- `DISABLE_MOSQUITTO` uses the interpolated default form
`${DISABLE_MOSQUITTO:-true}` (preserves operator override). Bare literal
`true`, or a later `=false` override in the same env block, is rejected.
- Top-level `networks:` declares `meshcore-net` as `external: true`.
- `staging-go` attaches to `meshcore-net` via a real
`services.staging-go.networks:` sub-key (comment-stripped so an
in-comment example can't masquerade).
Regex (not YAML byte-equality) so cosmetic edits don't break the guard.
No new go module deps. Red commit `3898dbc5` fails all 4 assertions on
master. Green commit `38297ff4` makes them pass. Round-1 hardening
commit `9f7155e2` tightens the regexes (per adversarial + kent-beck
must-fixes) and was verified against master's YAML shape — all 4 tests
fail on `origin/master`'s compose, pass on branch, proving the tightened
regexes still gate a real regression.
## Risk
Low, with one intentional semantic change.
- **Semantic change (v3.7+):** `DISABLE_MOSQUITTO` in
`docker-compose.staging.yml` now defaults to `true`. This is a
**deliberate flip** — the standalone `mqtt-broker` container is now
authoritative on the staging host, and running the in-container
mosquitto alongside it wastes RAM and races the docker DNS name
`mqtt-broker` on cold start. Operators who want the pre-v3.7 shape
(in-container mosquitto + host-published `1883`) must explicitly opt
back in via env override AND re-add the `1883:1883` port mapping
(concrete snippet is inline in the compose file and in `DEPLOY.md` under
"Standalone MQTT broker (staging)"). This intent is called out in a
`SEMANTIC CHANGE (v3.7+)` header comment at the top of
`docker-compose.staging.yml`.
- **Deploy prereq:** the external `meshcore-net` docker network MUST
already exist on the host before `docker compose up`. If it doesn't,
compose refuses to start `staging-go`. This is documented inline in the
compose file (with the `docker network create meshcore-net` one-liner)
and in `DEPLOY.md`.
- **Only takes effect where the standalone broker is deployed** — which
it already is on staging today. The legacy `DISABLE_MOSQUITTO=false`
path remains reachable via env override; the ingestor's upstream config
is untouched.
Partial fix — no tracking issue; follow-up to operator-side broker
provisioning.
---------
Co-authored-by: corescope-bot <bot@corescope.local>
Co-authored-by: openclaw-bot <bot@openclaw.local>
|
||
|
|
fa15ab0a30 |
fix(#1809): gate background loader on LoadChunked completion (#1811)
Partial fix for #1809. Red commit: |
||
|
|
242c7c609b |
fix(mqtt): escalate persistent paho disconnect + recover from emit panic + expose watchdog tick (#1749) (#1810)
# Partial fix for #1749 — MQTT watchdog escalation + panic recovery +
tick exposure
Red commit:
|
||
|
|
b74a64ccfa |
fix(ui): canonical payload label map across packets/live/packet-filter (#1799) (#1804)
## Summary Replaces the three drifted per-surface payload-type label vocabularies with a single canonical map keyed by firmware enum name. Per the locked triage comment on #1799 ([comment-4823975431](https://github.com/Kpa-clawbot/CoreScope/issues/1799#issuecomment-4823975431)): > Create `public/payload-labels.js` exporting `{GRP_DATA: {short:'Group Data', long:'Group data packet', enumId:6}, ...}`. Migrate `packets.js typeMap`, `packet-filter.js FW_PAYLOAD_TYPES`, `live.js TYPE_COLORS legend` to consume it. E2E that scrapes each surface and asserts label equality. ## Changes - **`public/payload-labels.js`** (new) — canonical map exposed as `window.PayloadLabels` and `window.PayloadLabelsApi`. Keys are firmware enum names; values carry `{short, long, enumId}` plus derived `SHORT_BY_ID` / `FW_PAYLOAD_TYPES` / `TYPE_ALIASES` for legacy callers. - **`public/packets.js`** — `TYPE_NAMES` + `typeMap` now read from `PayloadLabelsApi.SHORT_BY_ID`. Literal kept only as a defensive fallback for the case where the script tag fails to load. - **`public/packet-filter.js`** — `FW_PAYLOAD_TYPES` + `TYPE_ALIASES` now sourced from `PayloadLabelsApi`. Literal fallback retained so `node test-packet-filter.js` still works headlessly. - **`public/live.js`** — legend `<li>` rows are now generated from `window.PayloadLabels` in stable order, killing the third-vocabulary `Message — Group text` / `Direct — Direct message` drift the #1797 review surfaced. - **`public/index.html`** — `<script src="payload-labels.js">` loaded before `roles.js` / `packet-filter.js` / `packets.js`. - **`test-issue-1799-label-vocab-e2e.js`** (new) — Playwright E2E. Scrapes `#liveLegend` rows and the `/packets` type-filter checklist, asserts each label matches `window.PayloadLabels[ENUM].short` for `TXT_MSG`, `GRP_TXT`, `GRP_DATA`. Also verifies `window.PacketFilter` still recognises the enum names. - **`.github/workflows/deploy.yml`** — wired the new E2E into the existing Playwright block. ## TDD trail - Red commit `eb392d4` — adds the failing E2E only (asserts `window.PayloadLabels` exists and labels match; both fail). - Green commit `44e902a` — introduces the canonical map and migrates the three surfaces. ## Verification - `node test-packet-filter.js` — 92/92 pass with the new fallback wiring. - Preflight: `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` — clean. Browser verified: E2E `test-issue-1799-label-vocab-e2e.js` exercises `/live` legend + `/packets` type filter against a Playwright headless Chromium; CI's Playwright block runs it on every push. E2E assertion added: `test-issue-1799-label-vocab-e2e.js:139` — `assert(fromLegend === canon, ...)` and `assert(fromPackets === canon, ...)` per enum. Fixes #1799 --------- Co-authored-by: mc-bot <bot@corescope> Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: clawbot <clawbot@kpa.com> Co-authored-by: clawbot <bot@clawbot.local> |
||
|
Michael J. ArcanandWaydroid Builder
|
9ae547ed7b |
test: de-flake distance-202 and anchor-bias tests (deterministic timing) (#1808)
Two server tests flaked intermittently and reddened CI on unrelated (frontend) PRs that merged master: - TestDistanceConcurrentRequestsDuringBuildReturn202 asserted all 10 concurrent requests get 202 'during the build window', but the lazy distance build on the tiny test DB finishes almost instantly, so on a fast machine some requests raced past it and got 200 (~50% flake). Add a nil-by-default distanceBuildHook seam on PacketStore (zero overhead in prod) that the test uses to hold the build open until all requests have been served — making the window guarantee deterministic. - TestHandleNodePaths_AnchorBiasInconsistency_Issue1278 queried /paths right after store.Load(), racing the path-hop index that Load() builds in a background goroutine (#1008); the membership/canonical result was thus non-deterministic (rarer flake, worse under suite load). Wait for PathHopIndexReady() before querying. Both run 30x green and pass -race. No production behavior change (hook is nil). Co-authored-by: Waydroid Builder <claude@michael.arcan.de> |
||
|
Kpa-clawbotandopenclaw-bot
|
ec0ebeda2f |
fix(#1793): WebSocket CheckOrigin allowlist (block cross-origin scrapers) (#1795)
## Summary Closes the wide-open `/ws` WebSocket upgrader (`CheckOrigin: return true`) that lets any browser origin scrape live packet data. Replaces it with an explicit allowlist consulted from `cfg.CORSAllowedOrigins`, plus an implicit same-origin allowance and an empty-Origin (non-browser client) allowance. Fixes #1793. ## Rules (`Hub.checkOrigin`) - Empty `Origin` header → **allow** (non-browser clients; per-IP rate/deny gating tracked separately in #1794). - `Origin` host == request `Host` (case-insensitive) → **allow** (same-origin). - `Origin` matches an entry in `cfg.CORSAllowedOrigins` by exact case-insensitive match → **allow**. - `"*"` in `cfg.CORSAllowedOrigins` is **deliberately ignored** for `/ws`. A startup `[ws] WARNING:` is logged once when present. - Anything else → **reject** (gorilla returns 403). ### Deliberate divergence from CORS XHR CORS XHR (`corsMiddleware`) still honors `"*"` for read-only cross-origin GETs. The `/ws` upgrade does NOT, per OWASP's WebSocket Security Cheat Sheet: > Use an allowlist, not a denylist. Avoid wildcards or substring matching. — https://cheatsheetseries.owasp.org/cheatsheets/WebSocket_Security_Cheat_Sheet.html `"*"` on the WS path would re-open the exact CSWSH/scraping vector this PR closes, so it is rejected with a startup warning rather than silently honored. This intentional asymmetry is documented in the updated `_comment_corsAllowedOrigins` in `config.example.json`. ## TDD red → green - `e5974c6a` **RED** — adds `cmd/server/websocket_checkorigin_test.go` with five cases; `SetAllowedOrigins` introduced as an enforcement stub so the test compiles and fails on the assertion (CI fails on this commit by design). - `a4791dc3` **GREEN** — implements `Hub.checkOrigin`, wires `SetAllowedOrigins` from `main.go`, updates the config example. All tests pass. ## Tests added (`cmd/server/websocket_checkorigin_test.go`) - `TestCheckOriginRejectsForeignOrigin` — foreign Origin → 403 - `TestCheckOriginAllowsEmptyOrigin` — non-browser client → 101 - `TestCheckOriginAllowsSameHost` — same-origin → 101 - `TestCheckOriginAllowsAllowlistedOrigin` — exact allowlist match → 101 - `TestCheckOriginWildcardDoesNotAllowForeignOrigin` — `"*"` in allowlist still rejects foreign origin → 403 ## Files changed - `cmd/server/websocket.go` — `Hub.allowedOrigins`, `SetAllowedOrigins`, `checkOrigin`, wired into `Upgrader.CheckOrigin`. - `cmd/server/main.go` — `hub.SetAllowedOrigins(cfg.CORSAllowedOrigins)` at the single call site. - `cmd/server/websocket_checkorigin_test.go` — new test file. - `config.example.json` — updated `_comment_corsAllowedOrigins` to document `/ws` gating and the `"*"` divergence. ## Out of scope (follow-up) - **#1794** — per-IP rate limit / deny list / connection cap for non-browser clients (which still bypass Origin because they don't send one). Layered defense; not in this PR. ## Verification - `go test ./cmd/server/...` — all server tests pass locally (574s). - Preflight clean (`bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master`). --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Michael J. Arcan
|
ae2e3933dd |
feat(server): store memory diagnostics + drop redundant obs.RawHex (#1773)
Drops the redundant per-observation RawHex (~98MB on a live store; reader already falls back to tx.RawHex #881) and adds an opt-in /api/perf?mem=1 memory breakdown (flood-forward share + per-component bytes). Profiled against a live instance. **Savings substantiation:** live-instance profiling shows ~1.66M observations in the store, each previously carrying its own per-observation `raw_hex` (avg ≈118 hex chars ≈59 bytes) that exactly duplicates the parent transmission's `raw_hex`. Dropping the duplicate on every load/ingest path eliminates ≈98 MB of redundant in-memory storage plus ~1.66M string allocations, with no data loss — the read path (`enrichObs`) already falls back to `tx.RawHex` when `obs.RawHex` is empty (verified by the new safety-gate test). The patched build cannot be run against the live instance here; instead the new opt-in `/api/perf?mem=1` diagnostic lets operators measure the real before/after (`trackedMB` and the per-component breakdown) directly after deploy. |
||
|
Michael J. ArcanandWaydroid Builder
|
3efa37c46c |
feat(server): complete the #672 4-axis repeater usefulness score (#1762)
Adds Coverage (harmonic reach) + Redundancy (Tarjan articulation) axes + composite & grade. Closes #672. **TDD note (BLOCKER-1):** Community PR delivered as a single squashed commit, so there is no separate pre-fix failing-test commit — please accept as a community-PR exemption. The tests are *gating*, not just thorough: each axis test pins a specific topology outcome (coverage on line/star/disconnected/weight-sensitive; redundancy online/triangle/star/bridged-cliques), and an end-to-end `/api/nodes` surface test drives the whole pipeline and asserts the composite diverges from the Traffic axis. Inverting the `1/weight` distance, dropping the NaN/Inf reject, removing the `redundancyMinWeight` floor, or aliasing `usefulness_score` back onto `traffic_share_score` each break a specific assertion. The axis functions are pure (no hidden state), so the suite fully characterises the behavior without the red anchor. Co-authored-by: Waydroid Builder <build@waydroid.local> |
||
|
|
17654dd090 |
docs(api): document per-node usefulness metrics in OpenAPI (#1769)
Documented Node schema (the four #672 usefulness axes + composite + A-F grade + relay fields) and response schemas on the node endpoints. Documentation-only; no behaviour change. Pairs with #1762 (documents the metrics it adds). Co-authored-by: Waydroid Builder <build@waydroid.local> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Michael J. Arcan
|
fc26fb6b3a |
feat(#1751): show transported region scopes in repeater sidebar (#1752)
Closes #1751. |
||
|
|
f0763aecce |
fix(#1726): clear stale "varies" hash size once a node settles (#1788)
Fixes #1726. ## Problem A MeshCore v1.16.0 repeater configured for 2-byte path hashes (`path.hash.mode=1`) — e.g. `36f6c7c7…` (`DK_3400_RAK_TEST`) — kept showing as **"varies"** / mixed 1-byte + 2-byte for the full 7-day advert window. Per the live data in the issue triage: of the node's ~20 recent adverts, exactly **one** (2026-06-09, across 15 distinct observer paths) was a genuine 1-byte flood advert; every other advert was 2-byte. The flip-flop heuristic in `computeNodeHashSizeInfo` weighs that stale advert equally with recent ones, so an operator who flips `path.hash.mode` mid-flight (or a single old 1-byte advert) stays flagged for the full window with no way to signal "the config is settled now." ## Fix Two coupled changes in `cmd/server/store.go` `computeNodeHashSizeInfo`: 1. **Chronological ordering.** `byPayloadType[4]` iterates in insertion order, not timestamp order, so `HashSize = Seq[last]` could pick the wrong advert under out-of-order MQTT ingest or chunked cold-load (the "carmack" concern from triage). We now collect `(FirstSeen, size)` pairs and **stable-sort by `FirstSeen`**; ties keep insertion order, preserving prior behavior when timestamps are equal. 2. **Recency decay.** After `transitions >= 2` raises the flip-flop flag, clear it when the most recent `hashSizeRecentAgreeCount` (= **3**) non-zero-hop adverts all agree on a single size. A node still flapping (recent adverts disagree) stays flagged. `3` mirrors the existing ≥3-observation threshold used to raise the flag. ## Policy note Triage marked this **needs-operator-input** because the decay is a behavior/policy change. This PR implements the rule the triage proposed ("if the last 3 adverts agree, clear inconsistent"), which matches the reporter's stated expectation. Happy to adjust the threshold or gate it differently per your call. ## Tests `cmd/server/issue1726_hash_decay_test.go`: - `TestIssue1726_SettledNodeNotInconsistent` — reporter's case (`[2,1,2,2,2]` within window) → `Inconsistent=false`, `HashSize=2`. - `TestIssue1726_HashSizeUsesChronologicallyLatest` — out-of-order insertion still reports the chronologically-latest size. - `TestIssue1726_ActiveFlapperStaysInconsistent` — a node whose recent adverts disagree stays flagged. Existing flip-flop / hash-collision tests unchanged and green; full `cmd/server` package suite passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Erwin Fiten <e.fiten@opteco.be> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
d437958474 |
fix(map): pin APC (Napa) and STS (Sonoma) observers (#1786) (#1787)
Fixes the map-coordinate gap in #1786. ## Problem Observers tagged with IATA code **APC** (Napa County) or **STS** (Charles M. Schulz–Sonoma County) render with no location and never pin on the map. ## Root cause `iataCoords` in `cmd/server/routes.go` is a hardcoded `IATA -> lat/lon` lookup used purely for placing observer/region markers on the map. It had no entry for APC or STS, so those observers had no coordinates to render with. This is **display-only**. Ingestion is not gated on these codes: `IsObserverIATAAllowed` (`cmd/ingestor/config.go`) short-circuits to `true` when the observer IATA whitelist is empty — which is the staging configuration. The reporter''s "packets disappear entirely" symptom is therefore **not** explained by this code path (likely an upstream `meshcoretomqtt`/broker topic issue; needs operator `mosquitto_sub` confirmation per triage). ## Fix - Add `APC {38.2132, -122.2807}` and `STS {38.509, -122.8128}` to `iataCoords`, matching the airports'' published coordinates. - Add a regression test (`TestIataCoordsIncludesNapaAndSonoma`) asserting both are present with the expected coordinates. ## Verification - `go test ./cmd/server/` — full package passes (`ok`). - `go vet ./cmd/server/` — clean. ## Scope note Checked the repo for other statically-enumerable region codes (`config.example.json` regions: SJC/SFO/OAK/MRY) — all already covered. The broader "are other in-use codes missing" question can only be answered against the live `cfg.Regions` + `db.GetDistinctIATAs()` set, which is operational, not in-tree. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Erwin Fiten <e.fiten@opteco.be> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
57956712e7 |
fix(#1768): Relay Airtime Share uses LoRa Time-on-Air (preamble-aware) — partial fix (#1776)
Partial fix for #1768 — Relay Airtime Share now uses closed-form LoRa Time-on-Air instead of a payload-bytes-only proxy, removing the ~3-4× bias against small frames (preamble + fixed-symbol intercept). cross-stack: justified — backend score formula needs a frontend caption change (`public/analytics.js` dumbbell preset banner + tooltip) so operators can interpret the assumed PHY block. Both move together or the metric is misleading. ## Red commit `8da57062` — failing test asserts ToA-based score (~83.48 % ADVERT share on the locked acceptance fixture) instead of the byte proxy's 95.24 %. `internal/lora.TimeOnAir` was a zero-returning stub at the red commit; tests failed with assertion errors, not build errors. ## Green commit `dd402edd` — implements `lora.TimeOnAir` (Semtech AN1200.13 / SX126x §6.1.4 closed form, cross-checked against RadioLib), wires `score = TimeOnAir(payloadBytes, preset) × distinctRelays` in `cmd/server/relay_airtime_share.go`, surfaces the preset in the JSON response and analytics caption. ## Config (per AGENTS Config Documentation Rule) New keys under existing `analytics` block: ```json "loraPreset": { "freq": 869600000, "bw": 62.5, "sf": 8, "cr": 5 } ``` Defaults match the deployment's actual `get radio` (869.6 MHz / BW 62.5 kHz / SF 8 / CR 4/5). `CRC=1`, `IH=0`, `DE = (T_sym ≥ 16 ms)`, and the SF-dependent preamble (32 for SF≤8 else 16, per firmware `preambleLengthForSF` / MeshCore PR #1954) are firmware-fixed constants in `internal/lora/toa.go` and intentionally NOT surfaced as config (per re-triage). ## Scope In-scope files (6): - `internal/lora/toa.go` (new package — closed-form ToA) - `internal/lora/toa_test.go` (table-driven preset tests) - `cmd/server/relay_airtime_share.go` (wire ToA into score) - `cmd/server/relay_airtime_share_test.go` (recomputed expected values) - `cmd/server/config.go` + `config.example.json` (preset config keys) - `public/analytics.js` (preset caption on dumbbell chart + tooltip) Plus `cmd/server/go.mod` (replace directive for the new internal module). ## Deferred to v2 (separate issues per re-triage) - Per-observation SF/BW + radio-settings-aware dedup (blocked: ingestor stores SNR/RSSI only, no SF/BW on observations). - CR-per-hop dual-point sensitivity band (CR scales only the payload symbol term `(CR+4)`, not the preamble/header; second-order accuracy gain). - Cross-SF bridge accounting. ## Tests ``` cd internal/lora && go test ./... → PASS cd cmd/server && go test -run RelayAirtime → PASS ``` ## Preflight overrides - `check-branch-clean` (cross-stack): justified above — score formula change requires matching caption update; both files trace to the same issue. --------- Co-authored-by: kpa-clawbot <kpa-clawbot@users.noreply.github.com> Co-authored-by: Kpa-clawbot <bot@openclaw.local> Co-authored-by: bot <bot@meshcore> |
||
|
|
22fe929da2 |
feat: opt-in mobile client-RX coverage (crowdsourced RF reach) + /api/nodes/resolve (#1728)
Implements #1727. ## What this adds **Mobile client-RX coverage** — an opt-in, crowdsourced RF-coverage feature. A roaming MeshCore **companion** radio (driven by the open-source [corescope-rx](https://github.com/efiten/corescope-rx) PWA, GPLv3) reports which nodes it heard directly, tagged with the phone's GPS and the packet's SNR/RSSI. CoreScope ingests these into a new `client_receptions` table and renders per-node **hex coverage** on the Reach page, plus a standalone **Coverage dashboard** (`#/rx-coverage`) with a top-mobile-observers leaderboard. Also includes **`GET /api/nodes/resolve?prefix=<hex>`** — a read-only node-name lookup by pubkey prefix (`{name, pubkey, ambiguous}`), used by the companion app for friendly names. ## Opt-in — default OFF (zero impact on existing deployments) The whole feature is gated behind one config flag, **disabled by default**: ```jsonc "clientRxCoverage": { "enabled": false } ``` When disabled (the default): the ingestor writes **no** `client_receptions`; the three coverage endpoints return a clean **404**; the UI hides the Coverage nav link, the `#/rx-coverage` route, and the Reach-page toggle. `/api/nodes/resolve` is always available (not coverage-specific). ## How it works ``` companion ──BLE 0x88 (snr+rssi+raw)──▶ corescope-rx PWA ──▶ MQTT meshcore/client/{pubkey}/packets │ ingestor (gated) ──▶ client_receptions (GPS + SNR + heard-key) │ server: pure-Go hex grid ──▶ GeoJSON ──▶ Reach hex overlay + Coverage dashboard ``` - **Direct-only capture:** records only what the companion heard itself and directly — a 0-hop advert's pubkey, or `path[last]` (last forwarder) for FLOOD routes; ≥2-byte path-hash required. Upstream hops discarded. - **No new deps:** hexbins are a pure-Go pointy-top grid over Web Mercator (`cmd/server/hexgrid.go`) computed at query time (`CGO_ENABLED=0` / `modernc.org/sqlite` friendly); frontend uses the existing Leaflet. - **Trust:** companion pubkey = identity; an EMQX ACL binds each client to publish only to its own `meshcore/client/{pubkey}/packets` topic. Payload contract in `docs/client-rx-coverage.md`. ## How to enable / try it 1. In `config.json`, set `"clientRxCoverage": { "enabled": true }` and restart server + ingestor. 2. Point an EMQX (or any broker) listener so a client can publish to `meshcore/client/<pubkey>/packets`; the ingestor already subscribes under `meshcore/#`. 3. Run the [corescope-rx](https://github.com/efiten/corescope-rx) PWA on an Android phone paired (BLE) to a MeshCore companion — it captures heard nodes + GPS and publishes. 4. View results: per-node Reach page → toggle **coverage**, or the **Coverage** dashboard at `#/rx-coverage`. ## What's where - **Ingestor:** `cmd/ingestor/client_reception.go` (ingest), `db.go` (`client_receptions` + `client_observers` schema), `main.go` (gated dispatch), `config.go` (flag). - **Server:** `cmd/server/rx_coverage.go` + `rx_dashboard.go` (endpoints, self-guard 404 when off), `hexgrid.go` (pure-Go grid), `node_resolve.go` (resolve), `routes.go` / `types.go` / `config.go` (wiring + flag + `/api/config/client` field). - **Frontend:** `public/rx-coverage.js` (dashboard), `node-reach-coverage.js` + `.css` (overlay), `node-reach.js` (Reach toggle, flag-gated), `roles.js` (reads the flag, hides nav when off). - **Docs:** `docs/client-rx-coverage.md`. ## Testing - Go: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test ./...` — green, including new gate tests (`coverage_gate_test.go` in both: off → no rows / 404, on → works) and the rx-coverage / resolve / hexgrid suites. - JS: `node test-coverage-gate.js`, `node test-node-reach-coverage.js` (wired into CI). The Playwright `test-node-reach-coverage-e2e.js` is wired into the e2e job and **skips when `clientRxCoverage` is disabled**, so it's safe under the default-off config. ## Notes for reviewers - The four new routes are registered in `cmd/server/openapi_known_gaps.json` (the existing OpenAPI-completeness ratchet), matching how other not-yet-spec'd routes are tracked. Happy to write full OpenAPI spec entries instead if you prefer. - Commits are split per layer (ingestor / server endpoints / resolve / frontend / CI) for review. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Erwin Fiten <e.fiten@opteco.be> |
||
|
Kpa-clawbot
|
97833c523b |
fix(post-packets): use v3 observations schema (closes #1196) (#1704)
## Summary `POST /api/packets` is broken on every v3-schema install — which is the default since #1289. The handler issues two writes against legacy v2 column names and silently swallows the observation insert's error, returning `200 OK` with `id>0` while persisting zero observation rows. ## Root cause `cmd/server/routes.go:1225-1235` (pre-fix) used the v2 schema shape: ```go INSERT INTO transmissions (... path_json ...) // path_json removed in v3 INSERT INTO observations (transmission_id, observer_id, observer_name, snr, rssi, timestamp) // v2 columns // timestamp written as RFC3339 text; v3 wants unix INTEGER // second Exec's error was discarded ``` v3 schema (`cmd/ingestor/db.go:289-304`): `observations.observer_idx INTEGER` (FK `observers.rowid`), `observations.timestamp INTEGER` (unix epoch), `path_json` lives here not on `transmissions`. Reporter [@EldoonNemar](https://github.com/EldoonNemar) called this out precisely in #1196 — both the schema mismatch and the divergence between the test harness (which uses the v3 shape) and the handler (v2 shape). ## Fix `cmd/server/routes.go`: - `transmissions` insert: drop `path_json` column. - Observer resolution: `INSERT OR IGNORE INTO observers (id, name, ...)` then `SELECT rowid` — mirrors the ingestor resolver at `cmd/ingestor/db.go:778,906`. - `observations` insert: write `observer_idx INTEGER` + `timestamp = time.Now().Unix()`; `path_json` moved here. - **Propagate both insert errors** (transmission + observation) as `500` instead of swallowing them. ## TDD | Step | Commit | Result | | ----- | ------- | ------ | | RED | `46d25389` | Test fails on master: `id=0` because the transmissions insert references a column not present in v3. | | GREEN | `dae57d67` | Test passes; round-trip persists the observation with `observer_idx` resolved from the seeded `obs1` row and a unix-epoch `timestamp`. | Local repro: ``` # RED on the test commit alone: $ go test -run TestPostPacketPersistsV3Schema -count=1 . --- FAIL: TestPostPacketPersistsV3Schema (0.03s) routes_test.go:4755: expected transmission id > 0, got 0 (body: {"id":0,"decoded":{...}}) FAIL # GREEN on HEAD: $ go test -run TestPostPacketPersistsV3Schema -count=1 . ok github.com/corescope/server 0.037s ``` ## Scope Two files, both in `cmd/server/`: - `cmd/server/routes.go` (+38/-12) — handler rewrite - `cmd/server/routes_test.go` (+66) — round-trip regression test No public API signature changes. No DB schema changes (consumes the existing v3 schema correctly). Closes #1196 |
||
|
Kpa-clawbotandKpa-clawbot
|
76e130b313 |
fix(#1702): grant actions: write to release-fast-path workflow (#1703)
## Summary Fixes the missing `actions: write` permission on `.github/workflows/release-fast-path.yml` so the fallback `gh workflow run deploy.yml` dispatch no longer returns HTTP 403. ## Triage verdict From issue #1702 root-cause section: > Fast-path workflow YAML likely lacks: > ```yaml > permissions: > contents: read > packages: write > actions: write # MISSING — required to dispatch other workflows > ``` > ## Fix > One-line addition to `.github/workflows/release-fast-path.yml` permissions block. ## Root cause `.github/workflows/release-fast-path.yml` lines 16-18 (before this change) only granted `contents: read` and `packages: write`. The fallback step (`gh workflow run deploy.yml` when `:edge`'s `org.opencontainers.image.revision` label doesn't match the tag SHA) calls the GitHub Actions REST API, which requires `actions: write` on `GITHUB_TOKEN`. Without it, the dispatch fails with `Resource not accessible by integration` and the release stalls until an operator manually re-runs the fast-path job after `:edge` rebuilds. ## Change - `.github/workflows/release-fast-path.yml`: add `actions: write` to the workflow-level `permissions:` block. - `cmd/server/release_fast_path_workflow_test.go`: extend the existing config-gate test (issue #1677) to require `actions: write` alongside the previously asserted `contents: read` and `packages: write`. Two commits, red→green: 1. `test(#1702): assert release-fast-path.yml requires actions: write` — extends the assertion. Verified to fail on this commit (`release-fast-path.yml: missing required permission "actions: write"`). 2. `fix(#1702): grant actions: write to release-fast-path workflow` — adds the permission. Test green. ## TDD posture The repo already had a YAML-config gate at `cmd/server/release_fast_path_workflow_test.go` (parses the workflow as text and asserts required permission strings). Strict TDD applied: red commit extends the test, green commit fixes the workflow. No exemption needed. ## Acceptance criteria (from #1702) - [x] `permissions.actions: write` added to the fast-path workflow - [ ] Manual test: tag a scratch SHA where `:edge` is stale; confirm fallback dispatches deploy.yml without 403 — by-design out of CI scope (would require a throwaway tag + race condition); covered by next real release. - [ ] Operator-felt: next release where notes-commit lands AFTER `:edge` build completes works in one pass without manual rerun — verifiable only on next release; in-scope of `Closes #1702` because bullet 1 (the structural defect) is the cause of bullets 2 and 3. ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → **clean** (all hard gates pass, no warnings). Closes #1702 --------- Co-authored-by: Kpa-clawbot <kpa-clawbot@users.noreply.github.com> |
||
|
Kpa-clawbotandKpa-clawbot
|
e96f0f9f9f |
fix(#1694): port extended ACK decoder to server (ackLen/ackAttempt/ackRand parity) (#1695)
## Summary Ports the firmware-1.16.0 extended ACK decoding from the ingestor (PR #1618, issue #1610) into the server-side re-decoder. Previously `cmd/server/decoder.go` silently dropped `ackLen`, `ackAttempt`, and `ackRand` (and the multipart inner equivalents) — the server emitted plain 4-byte ACKs even when the wire carried the 5/6-byte extended form. Now both decoders agree byte-for-byte. Closes #1694. ## What changed - `cmd/server/decoder.go::decodeAck`: sets `AckLen` (capped at 6), `AckAttempt` (`buf[4]` when `len>=5`), `AckRand` (`buf[5]` when `len>=6`). Mirrors `cmd/ingestor/decoder.go:279-305`. - `cmd/server/decoder.go::decodeMultipart` ACK branch: sets `InnerAckLen = len(buf)-1` (capped at 6), `InnerAckAttempt`, `InnerAckRand`. Mirrors `cmd/ingestor/decoder.go:696-714`. - `Payload` struct gains six `*int` fields tagged `omitempty`: `AckLen`, `AckAttempt`, `AckRand`, `InnerAckLen`, `InnerAckAttempt`, `InnerAckRand`. Backward-compatible JSON — legacy 4-byte ACKs leave attempt/rand nil and the fields are omitted from the output. No other decoder consumer is touched. Routes / store auto-surface the new fields via JSON marshaling. ## Test layout `cmd/server/decoder_ack_extended_test.go` drives `decodeAck` table-driven across the three wire shapes: | Buffer | AckLen | AckAttempt | AckRand | |---|---|---|---| | `EF BE AD DE` (CRC only) | 4 | nil | nil | | `EF BE AD DE 07` | 5 | 7 | nil | | `EF BE AD DE 07 42` | 6 | 7 | 0x42 | Plus `TestDecodeMultipartAckExtendedInner` for a 7-byte multipart buffer (`0x33` header + 6-byte inner ACK), asserting `InnerAckLen=6`, `InnerAckAttempt=7`, `InnerAckRand=0x42`. ## TDD trail - **Red commit** (test + struct stubs only, `decodeAck`/`decodeMultipart` unchanged) → assertions fail on `AckLen=nil`. - **Green commit** (port implementation) → all assertions pass. Full `cd cmd/server && go test ./...` passes locally. ## Firmware refs - `firmware/src/helpers/BaseChatMesh.cpp:218-234` (extended ACK layout) - firmware commit `f6e6fdaa` (attempt counter) - firmware commit `a130a95a` (RNG byte) --------- Co-authored-by: Kpa-clawbot <bot@kpa-clawbot> |
||
|
|
a8c99c61fd |
fix(#1659): block analytics endpoint until first pass complete (503 Retry-After) (#1688)
## Summary Fixes #1659 — analytics cards no longer show the post-restart slice when "All data" is selected. ## Root cause After server restart, `s.recompRF` / `s.recompTopology` / `s.recompChannels` cache the FIRST computation, which is the small in-RAM observations slice (background chunk-loader has not yet backfilled history). The recomputer serves that slice through `GetAnalyticsRFWithWindow`'s default shortcut for an entire recompute interval, while the client pins it via `CLIENT_TTL.analyticsRF`. UX: cards show a tiny window even when the user selects "All data". ## Fix shape (option B from the issue body) Server-side per-recomputer warm-up gate: - `cmd/server/analytics_warmup_1659.go` adds a per-recomputer `firstPassDoneNs` atomic timestamp, set ONLY by the first successful `runOnce()` (CAS-guarded for idempotency). `IsWarmingUp_1659()` / `FirstPassDoneAt_1659()` are lock-free reads. - `cmd/server/analytics_recomputer.go` `runOnce()` calls `markFirstPassDone_1659()` after every successful compute. - `cmd/server/routes.go` handlers for RF / Topology / Channels: when the request is the default shape (`region=="" && area=="" && window.IsZero()`) AND the matching recomputer is still warming up, return `503` + `Retry-After: 5` + `{"error":"analytics warming up","retry_after_s":5}`. Windowed / region-filtered requests bypass the gate (they already bypass the recomputer cache, so they are unaffected by the warm-up bug). Client-side: - `public/app.js` `api()` helper retries any 503 response, honoring `Retry-After`, with exponential backoff capped at 30s, max 6 attempts (~63s total). - Small "Computing analytics…" banner appears while any warm-up retry is in flight, dismissed once the request resolves. Pages can override via `window.onWarmup_1659`. ## Tests RED commit `8b2b2d7` ships failing-on-assertion tests + a stub. GREEN commit `2716c23` lands the fix and flips them green. - `cmd/server/analytics_warmup_1659_test.go` — 3 cases: 503 during warmup, 200 after first pass, windowed request bypasses gate. - `test-1659-analytics-warmup.js` — 3 cases: Retry-After honored, retry cap bounded, non-503 errors not retried. Wired into `.github/workflows/deploy.yml`. ## Preflight overrides - cross-stack: justified — server-side 503 contract MUST be paired with client-side retry-and-banner handling; splitting across two PRs would land a half-working fix. Fixes #1659. --------- Co-authored-by: corescope-bot <bot@corescope.local> Co-authored-by: openclaw <openclaw@local> |
||
|
|
048143f54f |
fix(#1690): cold-load uses last_seen (effective recency) instead of first_seen (#1691)
## #1690 — cold-load uses wrong time axis (RED → GREEN) The on-disk DB has thousands of long-lived hashes with recent traffic. Prod's cold-load filter (`transmissions.first_seen >= cutoff`) is bound to a column that is set once at insert time and never updated — so re-observation of an old hash does not move it into the hot window. Result: prod cold-loaded ~0.3% of the on-disk rows and flipped `backgroundLoadComplete=true` without ever walking the retention window (the `retentionHours - hotStartupHours <= 0` short-circuit at line 1353 of `cmd/server/store.go`). ### Three sub-fixes **A) Denormalize `transmissions.last_seen`** so cold-load can window on effective recency. - `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn` adds the column + `idx_tx_last_seen` (single-column INTEGER ALTER + index; both PREFLIGHT-annotated as cheap metadata-only ops). - `cmd/ingestor/db.go::OpenStoreWithInterval` schedules `tx_last_seen_backfill_v1` via `Store.RunAsyncMigration` — `UPDATE transmissions SET last_seen = MAX(observations.timestamp) WHERE last_seen = 0` — non-blocking on boot (1.9M+ obs row scan in prod). - Writer-side: `InsertTransmission` seeds `last_seen` on initial insert, and every observation insert bumps `last_seen = ?` via prepared statement `stmtBumpTxLastSeen` (conditional `last_seen < ?` so out-of-order ingest never goes backwards). - Reader-side: `cmd/server/store.go::Load`, `loadChunk`, and `cmd/server/chunked_load.go::LoadChunked` switch the WHERE/ORDER-BY clauses to `t.last_seen` when the column is present (PRAGMA-detected via `DB.hasLastSeen`). Test/legacy DBs without the column fall back to `first_seen` so existing fixtures stay green. **B) Honest `backgroundLoadComplete` gating.** - Drop the `retentionHours - hotStartupHours <= 0` short-circuit. Prod runs with both at 12h, which flipped Done=true immediately. - After the chunk loop, query `SELECT COUNT(*) FROM transmissions WHERE last_seen >= retentionFloor` and compute `loadCoverageRatio = inMem / inDB`. Done=true only when `ratio >= 0.90` AND no chunk errors. `backgroundLoadFailed=true` + `backgroundLoadError` populated otherwise (e.g. `"loaded 20.0% of 5000 rows (1000 in memory)"`). - `bgErrMu`-guarded `loadCoverageRatio` + `backgroundLoadErr` so the perf endpoint can read them without blocking the writer. **C) Perf exposure.** `PerfPacketStoreStats` gains `RetentionHours`, `OldestLoaded`, `LoadCoverageRatio`, `BackgroundLoadError` — surfaces what fraction of the on-disk DB the in-memory store currently reflects, so operators can see the 0.3% case in `/api/perf` without reading the logs. ### TDD trail - **RED**: `05f0c6dd2bea6dc37324c548a49564d739aca920` — failing tests + 21-line store.go scaffolding. CI on this commit failed on assertions (intended). - **GREEN**: this PR's HEAD commit (8 files, +271/-24). Targeted suite: `Test1690_ColdLoad_TimeAxis`, `Test1690_BackgroundLoadHonesty`, `Test1690_PerfStats_NewFields`, `TestHotStartup_*`, `TestIssue1690_LastSeenUpdatedOnObservation` — all pass. Anti-tautology: locally reverted the `if !s.backgroundLoadFailed.Load()` guard around `backgroundLoadDone.Store(true)` — `Test1690_BackgroundLoadHonesty` fails on the assertion `"backgroundLoadDone=true with only 1000/5000 packets loaded; must be false until coverage ≥ 90%"`. Restored. ### Async-migration preflight - `ensureTransmissionsLastSeenColumn` — ALTER + CREATE INDEX both `// PREFLIGHT: async=true reason="..."` annotated. - `tx_last_seen_backfill_v1` — wrapped in `Store.RunAsyncMigration`. - `stmtBumpTxLastSeen` prepared statement — annotated; it is a row-level UPDATE BY PRIMARY KEY, not a migration. ### Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=5K - check-async-migration: justified for `cmd/server/issue1690_cold_load_test.go` CREATE TABLE/INDEX statements — these build an in-memory test fixture DB (≤5000 rows, runs in <1s in CI), not a prod migration. Fixes #1690. --------- Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: bot <bot@example.com> |
||
|
|
d910ea0208 |
feat(#1638): confidence rating weighted by hash mode (#1687)
Fixes #1638. ## Problem `getConfidenceIndicator` in `public/nodes.js` treats every observation as equal evidence, so a node seen 5 times via 1-byte hash prefixes (which collide ~8-way across a typical mesh) scores the same as a node seen 5 times via 6-byte prefixes (effectively unambiguous). The user asked for confidence to respect ambiguity. ## Change - `cmd/server/neighbor_graph.go` — new `CountsByMode map[int]int` on `NeighborEdge`, bumped in `upsertEdge` / `upsertEdgeWithCandidates` based on the observation's hash-prefix byte length (1/2/4/6). Merged in `resolveEdge` when ambiguous→resolved edges collapse. - `cmd/server/neighbor_api.go` — `NeighborEntry.counts_by_mode` exposed (omitempty), and `dedupPrefixEntries` merges per-mode counts when an unresolved prefix entry collapses into a resolved one. Flat `Count` field preserved for back-compat. - `public/nodes.js::getConfidenceIndicator` — weights observations by mode: 1-byte=0.125, 2-byte=0.5, 4/6-byte=1.0. A single 6-byte sighting counts ~8× a raw 1-byte one. HIGH triggers when EITHER the legacy heuristic clears OR weighted count ≥3. Legacy entries without `counts_by_mode` keep working (default weight 0.5). - Tooltip now shows the per-mode breakdown (e.g. "Observations: 5 (1-byte: 3, 6-byte: 2)"). ## TDD - RED: `cmd/server/neighbor_graph_test.go::TestBuildNeighborGraph_CountsByMode` — fixture with 1/2/4-byte sightings asserts per-mode tally (commit `838965f3`). - RED: `test-confidence-indicator.js` — 6-byte mostly-sighted neighbor must outrank 1-byte mostly-sighted neighbor at equal flat count (commit `4bd5e18e`). - GREEN: implementation in commit `7511606d`. All 4 JS tests pass; new Go test passes; full Go suite passes (two pre-existing flakes unrelated, both pass when isolated). ## Browser verification Synthetic side-by-side of OLD vs NEW classifier against representative inputs — see screenshot. 1-byte-only and 6-byte-only at the same flat count diverge from MEDIUM/MEDIUM to MEDIUM/HIGH, and 3 6-byte sightings now upgrade where 20 1-byte sightings stay MEDIUM. ## Preflight overrides - check-branch-scope: cross-stack: justified — backend exposes the new `counts_by_mode` field and the frontend consumes it; the whole point of the change. ## Compat - `Count` field unchanged in shape and value. - `counts_by_mode` is `omitempty`; legacy persisted edges (loaded from `neighbor_edges` via `neighbor_persist.go`) get no per-mode breakdown and fall back to the default weight (0.5) — no UI regression. --------- Co-authored-by: bot <bot@local> Co-authored-by: corescope-bot <bot@corescope.local> |
||
|
Kpa-clawbotandopenclaw-bot
|
efd66ea3f5 |
feat(mqtt): per-source status endpoint + Observers panel (#1682)
## Summary Adds MQTT source status visibility per #1043 acceptance criteria: - **Ingestor:** per-source counter registry (`cmd/ingestor/source_status.go`) tracking `connected`, `lastConnectUnix`, `lastDisconnectUnix`, `lastPacketUnix`, `connectCount`, `disconnectCount`, `packetsTotal`, `packetsLast5m` (sliding 5-min window via per-second buckets keyed by unix second — no stale-leak), `lastError`. Wired at the existing OnConnect / ConnectionLost / DefaultPublish callsites alongside the liveness watchdog. Idempotent registration so counters survive reconnects. Snapshot emitted in the existing stats file under `source_statuses` (additive, `omitempty`). - **Backend:** new `GET /api/mqtt/status` handler reads the ingestor stats file and returns the per-source list. **Broker passwords are masked** via a regex over the `scheme://user:pass@host` form (covers mqtt/mqtts/tcp/ssl/ws/wss). Mask is also applied to `lastError` as defense-in-depth (broker libs occasionally quote the failing URL). OpenAPI completeness gate satisfied with a `routeDescriptions` entry. - **Frontend:** small self-contained panel (`public/mqtt-status-panel.js`) mounted above the Observers table. Auto-refreshes every 10s, color-codes each row (green = connected + recent packet, yellow = connected idle, red = disconnected), and tears down its timer on SPA route change. ## TDD - Red commit `f19a93b5` — stub `/api/mqtt/status` handler + assertion test that the broker password is `****`-redacted. Test fails on the assertion (handler passes the URL through verbatim). Compile-clean — assertion-fail, not build-fail. - Green commit `77042e41` — `maskBrokerURL` helper + table-driven unit tests across all schemes + handler rewires to mask both `Broker` and `LastError`. - Subsequent commits land the ingestor wiring and the frontend panel. ## Tests ``` $ cd cmd/server && go test -run 'TestMqttStatus|TestMaskBrokerURL' -v ./... PASS: TestMqttStatus_MasksBrokerPassword PASS: TestMqttStatus_EmptyWhenNoStatsFile PASS: TestMaskBrokerURL_Patterns (10 subtests) $ cd cmd/ingestor && go test -run 'TestSourceStatus|TestSnapshotSourceStatuses' -v ./... PASS: TestSourceStatus_BasicLifecycle PASS: TestSourceStatus_Disconnect PASS: TestSnapshotSourceStatuses_ReturnsAll $ node test-mqtt-status-panel.js 7 passed, 0 failed ``` Full `go test ./...` clean in both `cmd/server` and `cmd/ingestor`. ## Preflight overrides - `cross-stack`: justified — issue #1043 is intrinsically full-stack (ingestor stats → server endpoint → observers panel). Per-stack split would land an unreachable endpoint or a fetch with no backend. - `check-xss-sinks` (public/mqtt-status-panel.js:55): justified — the flagged `innerHTML=` is a fully-static literal (empty-state placeholder, no payload data interpolated). All payload-bearing `innerHTML=` sites in this file run through `escapeHTML` (defined in the same file); the test `renderPanel never echoes a plaintext password (defense-in-depth)` exercises the rendered HTML against payload strings. ## Acceptance criteria - [x] `/api/mqtt/status` returns per-source connection state — `cmd/server/mqtt_status.go` - [x] UI panel shows all configured sources with live status — `public/mqtt-status-panel.js` - [x] Connection state updates on reconnect/disconnect events — `MarkConnect` / `MarkDisconnect` wired in `cmd/ingestor/main.go` - [x] Broker URLs don't expose passwords in the API response — `maskBrokerURL` + 13 test cases - [x] Works with 1-N sources — registry is keyed per-source, snapshot iterates the map **Partial fix for #1043** — per-packet `mqtt_source` attribution (the issue's "Follow-up" section) is **deferred** per the `mc-bot-triaged:v1` triage and the autofix comment ("Per-packet attribution deferred to follow-up issue"). That work requires a new observation-row column and DB schema migration, both explicitly out of scope for this PR. Refs #1043 --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbotandKpa-clawbot
|
2ef7d2437d |
fix(ci): release fast-path re-tag :edge → :vX.Y.Z when SHA matches (Fixes #1677) (#1680)
## Summary Adds `.github/workflows/release-fast-path.yml`: a metadata-only re-tag workflow that fires on `push.tags: v[0-9]+.[0-9]+.[0-9]+` and, when `:edge`'s `org.opencontainers.image.revision` label matches the tag SHA, applies `:vX.Y.Z`, `:vX.Y`, `:vX`, `:latest` to the existing edge manifest via `crane tag`. No rebuild, no test re-run — ~seconds vs ~30 min today. If the SHA doesn't match (tag points to an older commit, or `:edge` wasn't built yet), it dispatches the existing `deploy.yml` pipeline as a fallback so validated bytes always ship. To prevent double-fire, `deploy.yml`'s top-level `on:` block drops `tags: ['v*']` — `release-fast-path.yml` is now the sole consumer of `push.tags`. Edge publishing on master push is untouched. ## TDD Red commit adds `cmd/server/release_fast_path_workflow_test.go` (two tests: one asserts the new workflow exists with the required trigger/permissions/markers; the other asserts `deploy.yml`'s `on:` block no longer mentions `tags:`). Both fail on assertions in the red commit. Green commit adds the workflow file + edits `deploy.yml`; both pass. ## Acceptance criteria (from #1677) - Tag-CI completes in <2 min when tag SHA == `:edge` revision → fast-path is metadata-only, single short job - Falls back to full pipeline on SHA mismatch → `gh workflow run deploy.yml --ref ${{ github.ref }}` - `:vX.Y.Z` has same digest as `:edge` → `crane tag` copies the manifest, bytes are byte-identical - No regression on older-SHA tags → fallback path runs the unchanged full validation Fixes #1677 --------- Co-authored-by: Kpa-clawbot <bot@corescope.local> |
||
|
Kpa-clawbotandclawbot
|
653d47e03c |
test(openapi): add CI completeness gate for /api routes (Phase 1 of #1670) (#1678)
## Summary Partial fix for #1670 — **Phase 1 only** (CI completeness gate). Phase 2 (backfilling the 18 currently-undocumented routes into `openapi.go`) is deferred to a separate issue per the triage on #1670 and is explicitly out of scope here. ## What this adds - `cmd/server/openapi_completeness_test.go` — AST-walks every non-`_test.go` file in `cmd/server/`, finds string-literal first args to `*.HandleFunc(...)` calls beginning with `/api/`, and diffs against the paths declared in `routeDescriptions()` in `cmd/server/openapi.go`. - `cmd/server/openapi_known_gaps.json` — seeded allowlist of the **18** `/api/` routes currently registered via `HandleFunc` but not yet documented in `openapi.go`. ## Ratchet pattern From this branch forward, `TestOpenAPICompleteness` fails when: 1. A new `HandleFunc("/api/...")` is added without a matching entry in `openapi.go` **or** the allowlist (regression gate — the main goal of Phase 1). 2. A route in the allowlist is *also* documented in `openapi.go` — the allowlist must shrink as Phase 2 backfills land, never go stale. The two-commit history (red → green) demonstrates the gate works: - **Red commit**: adds only the test. Fails on master with the 18 missing routes listed. - **Green commit**: adds the allowlist seeded with that exact 18-route set. Test passes at the current baseline. ## Local verification - `go test ./cmd/server/ -run TestOpenAPICompleteness -v` → PASS at baseline (`44/62 covered; 18 in allowlist; 18 gaps remain`). - Ratchet validation: temporarily inserted `r.HandleFunc("/api/ratchet-test-route", ...)` into `routes.go` → test FAILED with that exact route name; reverted → test PASSES again. ## Files changed - `cmd/server/openapi_completeness_test.go` (+203 / new) - `cmd/server/openapi_known_gaps.json` (+24 / new) ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → all hard gates pass; no warnings. ## Out of scope - Backfilling the 18 allowlisted routes into `openapi.go` (Phase 2 — tracked separately). - Schema validation of the spec against OpenAPI 3.0 (Phase 3 per the issue). - PR template checkbox update (Phase 2 follow-up). Issue #1670 stays open for Phase 2. --------- Co-authored-by: clawbot <bot@corescope.local> |
||
|
|
938153dd92 |
fix(nodes): rebuild relay-hop history on startup from path_json (#1643)
## Problem A relay node's **activity timeline** — and its per-node `packetsToday` / observer counts — collapses to *"only the hour the server restarted"* after every restart. Before the restart the timeline shows only the node's own adverts (~1–2/hr); all of its relay activity piles into the single post-restart hour. ## Root cause All DB cold-load paths (`Load`, `loadChunk`, `scanAndMergeChunk`) index relay-hop attribution into `byNode` **only** from `observations.resolved_path`. But since #1287 the ingestor persists relay data as aggregate `neighbor_edges` and **never writes `resolved_path`** — it is `NULL` on every deployment (verified on a live DB: 0 of ~440k rows populated). So relay attribution is never reconstructed on startup; it only re-accumulates from live traffic (`IngestNew*`, which re-resolves from `path_json` + the neighbor graph), piling a relay node's whole history into the post-restart window. ## Fix Server read-side only — **no schema / ingestor / migration change**. When `resolved_path` is empty, re-resolve relay hops from the already-persisted `path_json` using the in-memory prefix map + neighbor graph (the same `resolvePathForObs` compute the live ingest path already runs). `main.go` now loads the persisted neighbor graph *before* the packet load so resolution has the graph available. Two correctness details worth a close look: 1. **Fetch the prefix-map/graph snapshot BEFORE opening each load cursor.** `getCachedNodesAndPM` issues its own DB query; doing so while a load cursor is open deadlocks on a single-connection SQLite pool (the test harness uses one). 2. **Index into `byNode` ONLY** — not the `resolved_path` / path-hop indexes. Those are cross-checked by `handleNodePaths` against the persisted `resolved_path` column (NULL here); populating them from an in-memory re-resolution would make that SQL confirmation fail and wrongly drop the tx from paths-through (#1352). ## Tests New coverage asserts a relay pubkey reachable *only* via `path_json` lands in `byNode` after a restart-style load, for both the hot-window (`LoadChunked`) and background-window (`loadChunk`) paths. Existing #1558 (`resolved_path`) and #1352 (paths-through) tests still pass. Full `cd cmd/server && go test ./...` is green under `-race`. ## Perf The fallback runs `resolvePathForObs` per observation with a non-empty `path_json` during cold load — the same per-packet compute the live ingest path already performs, so no new asymptotic cost. The prefix map + graph are snapshotted **once per load** (not per row); `getCachedNodesAndPM` is 30s-cached. In `loadChunk` the resolution runs in the existing lock-free scan and is accumulated locally, matching that function's "build local, merge under lock" design. ## Note on a pre-existing flaky test `TestDistanceConcurrentRequestsDuringBuildReturn202` is timing-fragile (fails ~1/15 on `master` without this change). It relies on the lazy distance build being slow because it's the first caller of `getCachedNodesAndPM` (cold cache). This PR pre-warms that cache during `Load`, narrowing the build window, so the test fails more often in **non-race** local runs. It passes reliably under `-race` (CI mode), where the build stays slow. Flagging in case you want to harden the test separately. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: openclaw-bot <openclaw-bot@users.noreply.github.com> Co-authored-by: openclaw-bot <bot@openclaw> |
||
|
Kpa-clawbot
|
825b26485c |
fix(#1181): hide nodes whose name starts with a configured prefix (#1655)
Fixes #1181. ## Summary Adds operator-configurable name-prefix hiding for nodes. When a node's name starts with any prefix listed in the new `hiddenNamePrefixes` config field (default `["🚫"]`), it is omitted from `/api/nodes`, `/api/nodes/search`, and `/api/nodes/{pubkey}`. DB rows are preserved — the filter runs at the API layer only, so observation history (paths, hops, distances) stays intact and the node simply re-appears if the operator clears the prefix list. This mirrors the convention already in use on other MeshCore map dashboards: an operator who wants their node hidden renames it with the 🚫 prefix and sends an advert; the next advert is then dropped from the dashboard. The node is **not** hidden from the mesh itself — only from this dashboard. This is documented inline in `config.example.json`. Implementation follows the existing `IsBlacklisted` pattern exactly: a new `Config.IsNameHidden(name)` method, and three filters in `routes.go` placed alongside the corresponding blacklist filters. No DB schema, public API, or websocket changes. ## Files changed - `cmd/server/config.go` — new `HiddenNamePrefixes []string` field + `IsNameHidden` method - `cmd/server/routes.go` — filters in `handleNodes`, `handleNodeSearch`, `handleNodeDetail` - `config.example.json` — new field + `_comment_hiddenNamePrefixes` operator doc - `cmd/server/hidden_name_prefix_1181_test.go` — new test file (red → green) ## Test plan Two new subtests in `TestHiddenNamePrefix_1181_*`: 1. `_NodesList` — inserts a node named `🚫 ban me`, asserts it is present when `HiddenNamePrefixes` is empty and absent when set to `["🚫"]`. 2. `_Search` — inserts `🚫 search me`, asserts `/api/nodes/search?q=search` does not surface it when the prefix is configured. Verified red→green: - Red commit `d0903852`: `go test -run TestHiddenNamePrefix_1181` fails on the leak assertion (`hidden_name_prefix_1181_test.go:94`). - Green commit `e79a0d8d`: same command passes. ``` $ cd cmd/server && go test -run TestHiddenNamePrefix_1181 -count=1 . ok github.com/corescope/server 0.060s ``` ## Out of scope - Auto-purging DB rows for hidden nodes — left to existing retention. The triage was explicit: hide, do not delete. - Live websocket broadcast: nodes are not broadcast via websocket (only packets), so no separate emit path needs filtering. Frontend reads nodes via `/api/nodes`, which is filtered. - Frontend customizer for the prefix list — operators configure via `config.json` like every other knob. |
||
|
Kpa-clawbotandKpa-clawbot
|
e04c7113cb |
feat: integrate hashtag channels from meshcore-channels catalogue (#1323) (#1656)
Fixes #1323 ## Summary Adds a small in-memory cache of the community-maintained hashtag-channels catalogue (`marcelverdult/meshcore-channels`) and exposes it as `GET /api/known-channels?region=XX` plus a collapsed sidebar section on the Channels view ("Known channels (catalogue)") with a one-click "+ Add" button per row. Per triage (#1323): new `cmd/server/known_channels_cache.go`, new `GET /api/known-channels?region=…`, frontend section in `public/channels.js`. No new DB tables — cache is in-memory only. ## What changed - `cmd/server/known_channels_cache.go` — `knownChannelsCache` with an atomic snapshot pointer, 24h default refresh, 30s HTTP timeout, 4 MB body cap, custom `User-Agent`. Fail-soft: a failed refresh leaves the last-known snapshot in place. Background goroutine started from `main.go` after the neighbor-graph recomputer; never blocks startup. - `cmd/server/known_channels_route.go` — `GET /api/known-channels?region=` serves the cached snapshot off the atomic pointer (never blocks on upstream). Region filter is case-insensitive ISO 3166-1 alpha-2. Empty/missing cache returns 200 with an empty entries list (fail-soft for the UI). - `cmd/server/config.go` — `KnownChannelsURL` + `KnownChannelsRefreshMs`. - `config.example.json` — example values + `_comment_knownChannels`. - `public/channels.js` — new collapsed sidebar section "Known channels (catalogue)" that lazy-fetches `/api/known-channels` on first render and renders rows with a "+ Add" button. The button calls the existing `addUserChannel(name)` path, so adding catalogue channels reuses the full save-key + decrypt flow that user-typed hashtags already use. - `cmd/server/known_channels_cache_test.go` — failing-first tests: - `TestKnownChannelsParseFixture` asserts the parser populates `GeneratedAt`/`License` and region-stamps every entry while skipping empty countries. - `TestKnownChannelsRouteRegionFilter` asserts the route returns 200 with exactly the filtered subset for `?region=be`. - `TestKnownChannelsFailSoftOn500` asserts a failed upstream fetch leaves the prior snapshot in place and bumps `failCount`. ## Upstream pinning The default URL is pinned to the specific file `channels-by-country.json` on `main`: > https://raw.githubusercontent.com/marcelverdult/meshcore-channels/main/channels-by-country.json Shape (verified 2026-05-24): ```json { "generated_at": "...", "license": "CC0-1.0", "countries": { "be": [{"channel": "#antwerpen", "description": "..."}], ... } } ``` ## Test plan ``` cd cmd/server && go test -run 'TestKnownChannels' -count=1 . ok github.com/corescope/server 0.008s ``` Red commit: 5c43cff3 (all three tests fail on assertions, build clean). Green commit: 54a1080e (parser + cache + route implemented, all three pass). ## TDD evidence (red → green) - **Red commit `5c43cff3427afd8aa2f3cce20c31058190aebc37`** — tests added with stub implementations that compile but return zero/empty so each test fails on an assertion (not a compile/import error). `go test -run TestKnownChannels` output captured in the commit message. - **Green commit `54a1080e45fd2e10da2caa156f376bf4d0212976`** — parser, cache, route, main-wiring, frontend section land; all three tests pass. ## Frontend verification Browser verified: http://analyzer-stg.00id.net/#/channels (with the `/api/known-channels` response stubbed in DevTools to simulate the cache being populated on staging, which is still on master and doesn't have the new endpoint yet). E2E assertion added: cmd/server/known_channels_cache_test.go:71 — asserts the route returns 200 and the response body's `entries` length matches the filtered subset. ## Limitations / follow-ups (not in scope of this PR) - The catalogue only ships PSK keys for a small subset of entries (the upstream schema makes `key` optional). For entries WITHOUT a `key`, the "+ Add" button still wires through `addUserChannel("#name")` — which derives the standard public-channel key from the name (the same path used today when a user types `#foo` into the Add Channel modal). For entries WITH a `key`, a follow-up PR can pass the key through to `addUserChannel` so the UX matches "paste-a-PSK". Today the key is shown in the JSON payload but not yet wired into the FE button. - No deduplication against the in-memory `/api/channels` list — the catalogue section is intentionally separate so the user sees which channels exist worldwide even if their server hasn't seen traffic. - No per-section region selector yet — the section shows the full catalogue regardless of the page-level region filter. Future work: add a dropdown. ## Preflight ``` ═══ Preflight clean. ═══ ``` cross-stack: justified — issue #1323 spans `cmd/server` (cache + route) and `public/channels.js` (sidebar surface); same feature, both halves required. --------- Co-authored-by: Kpa-clawbot <bot@corescope.local> |
||
|
Kpa-clawbotandopenclaw-bot
|
1116801b2f |
M5: emoji → Phosphor Icons — settings & customize (#1648) (#1653)
**Red commit:** `851cc8c3a024b1675558092d772444bf4f1ec625` — failing test on a stub branch (will link CI run after PR opens). Partial fix for #1648 (M5 of 6). **Do NOT close the tracking issue** — M6 (server-side residual emoji sweep + lint gate) still pending. ## Per-file swap counts | File | Phosphor `<use>` refs | Notes | |---|---|---| | `public/customize.js` | 20 | DEFAULTS → `ph:<name>` tokens; render path keeps legacy emoji branch (back-compat) | | `public/customize-v2.js` | 26 | same as v1; cv2 overrides path unchanged | | `public/home.js` | (helpers added) | `_renderHomeGlyph` / `_renderHomeLabel` accept both `ph:<name>` and legacy emoji | | `public/geofilter-builder.html` | 5 | clear / undo / save / load buttons (+inline `.ph-icon` CSS) | | `public/audio.js` | 1 | audio unlock prompt | | `public/filter-ux.js` | 5 (3 new) | help popover star + close, saved-filter delete | | `public/style.css` | 0 | `#chList .ch-share-btn::before { content: '📤' }` removed; JS now renders an inline sprite | | `cmd/server/routes.go` | (6 `ph:` tokens) | onboarding home defaults updated in lockstep with customize-v2.js | ## Operator config back-compat — PROMINENT Per design call #1 (user-locked): existing operator-stored emoji values in `config.json` / `localStorage` are **NOT** touched. The render path supports both: ```js function renderConfigGlyph(value) { var m = String(value || '').match(/^ph:([a-z][a-z0-9-]+)$/); if (m) return '<svg class="ph-icon"><use href="/icons/phosphor-sprite.svg#ph-' + m[1] + '"/></svg>'; return esc(value); // EMOJI-OK-LEGACY-RENDER — operator-stored emoji/text path } ``` Defaults flipped to `ph:<name>` tokens, so new operators (and operators who hit "Reset to Defaults") see Phosphor sprites. Operators with stored emoji values continue to see their emoji exactly as before. Verified end-to-end (see E2E (b) below). ## cmd/server/routes.go — changed in lockstep Per design call #2: the home-defaults `steps` / `footerLinks` mirror the JS DEFAULTS, so they MUST update together. routes.go now emits `ph:<name>` tokens; the frontend home-render path resolves them. Existing tests (`TestConfigThemeHomeDefaults`) still pass — they assert structure, not glyph values. ## E2E assertions added - `test-issue-1648-m5-emoji-scan.js` — per-file zero-emoji + ph-token DEFAULTS + sprite presence - `test-issue-1648-m5-icons-e2e.js`: - (a) customize chrome — tabs/header rendered as sprites; chrome text icon-free - **(b) back-compat — injects fake `🐙` operator step into localStorage, reloads, opens customize, asserts the emoji renders verbatim in both the input value AND the live preview span; asserts the ph-token step renders as a sprite** (design call #1 in action) - (c) `/channels` modal sprite count - (d) `/audio-lab` sprite presence - (e) `geofilter-builder.html` control buttons sprite-driven - (f) every `<use>` resolves to a defined symbol id ## Out of scope (M6 cleanup) - cmd/server/routes.go residual server-rendered emoji **not** tied to customize defaults (none found by my grep — file already audited) - `make lint-no-emoji` CI grep gate (M6 owns it) - `public/icons/README.md` workflow doc cross-stack: justified — design call #2 requires Go + JS update together. --------- Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
|
8295c2115c |
fix(reach): bust response cache on blacklist change (#1629) (#1636)
Red commit: 178617ca7b6a3b34597ae59d83e15801985be949 (CI run: https://github.com/Kpa-clawbot/CoreScope/actions/runs/27191921487 — red-state was verified locally; CI on this branch runs against green HEAD per pull_request triggers) Fixes #1629 ## Summary `/api/nodes/{pubkey}/reach` cached responses survived blacklist mutations for up to the 5-minute TTL. A node added to `NodeBlacklist` after a recent reach request was still served the cached non-blacklisted payload until the entry expired. ## Fix (per triage) Per @Kpa-clawbot's locked fix path on the issue: 1. Add a monotonic `BlacklistGeneration()` counter on `*Config`. 2. `SetNodeBlacklist` (new setter) atomically replaces the slice, rebuilds the lookup set under an `RWMutex`, and bumps the generation via `atomic.AddUint64`. 3. `cmd/server/node_reach.go` folds the generation into the cache key (`"<pubkey>|<days>|g<gen>"`) so any mutation invalidates prior entries on the next request — no callbacks bolted onto the setter, no cache-layer surgery, no TTL change. While here, the latent bug in `blacklistSet()` is also fixed: `sync.Once` locked in the initial set, so a later `SetNodeBlacklist` was invisible to `IsBlacklisted`. The `Once` still gates the lock-free initial build; mutations rebuild under `RWMutex` and reads take an `RLock` around the map handoff. ## Files - `cmd/server/config.go` — `SetNodeBlacklist`, `BlacklistGeneration`, `rebuildBlacklistSetLocked`, `RWMutex`. `IsBlacklisted` reads the rebuilt set (no stale-slice short-circuit). - `cmd/server/node_reach.go` — `cacheKey` includes `|g<gen>`. - `cmd/server/node_reach_blacklist_cache_test.go` — new regression test (the red commit). - `cmd/server/node_reach_endpoint_test.go` — existing cache-hit assertion updated to the generation-suffixed key. ## TDD evidence - Red commit `178617ca` adds the test + a deliberate `SetNodeBlacklist` stub that only reassigns the slice. The test fails on the post-blacklist assertion: `status=200 want 404 (cached payload was served — #1629)`. - Green commit `257c104f` replaces the stub with the real implementation; full `go test ./...` and `go test -race -run "TestNodeReach|TestNodeBlacklist|TestConfig"` pass locally. ## Scope - One narrow PR. Backend only — no frontend or API response-shape change. - No public type signatures touched beyond the new exported `SetNodeBlacklist` / `BlacklistGeneration` on `*Config`. - Preflight: all hard gates pass (PII, branch scope, red commit, CSS, LIKE/JSON, sync/async migration, XSS). --------- Co-authored-by: corescope-bot <bot@corescope.local> Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
|
078225a54e |
perf(neighbor_api): fold first_seen into cached map — fix #1627 r3 regression (#1632)
## TL;DR Post-merge regression introduced by #1627 r3 (commit `e2212f50`): `buildNodeInfoMap` in `cmd/server/neighbor_api.go` ran an uncached `SELECT … FROM nodes` scan on every call. Folded `first_seen` into the already-cached `getCachedNodesAndPM` (30s TTL) so the 4 hot handlers that call `buildNodeInfoMap` no longer pay for a full table scan per request. ## Before / After `buildNodeInfoMap` is called by **4 hot handlers**: - `cmd/server/neighbor_api.go:130` - `cmd/server/neighbor_api.go:297` - `cmd/server/neighbor_debug.go:83` - `cmd/server/node_reach.go:421` | | Before | After | |---|---|---| | `SELECT … FROM nodes` per call | 1 (uncached) | 0 (cache hit) | | `SELECT … FROM observers` per call | 1 (uncached) | 1 (unchanged) | | At Cascadia scale (~2600 nodes) | full scan × 4 handlers × N req/s | one scan / 30s | ## How - Extended the `getAllNodes` schema probe to also `COALESCE(first_seen, '')`. Falls back through the existing richest → leanest ladder if the column is missing. - `nodeInfo.FirstSeen` is therefore populated for every cached entry in `getCachedNodesAndPM`. - `buildNodeInfoMap` drops its second `SELECT` entirely and just copies `nodeInfo` values out of the cached map. - Public signature of `buildNodeInfoMap` is unchanged. `node_reach.go:421` still sees `nodeInfo.FirstSeen` populated, served from cache. `cmd/server/store.go` is touched because `getAllNodes` is the only sensible owner of the `first_seen` SELECT — adding a parallel cache would duplicate the 30s TTL machinery this fix is designed to leverage. ## Test (red → green) - Commit 1 (`test:`): `TestBuildNodeInfoMap_FirstSeenIsCached` — calls `buildNodeInfoMap`, mutates `first_seen` out-of-band via a separate rw connection, calls it again, and asserts both calls return the same (cached) value. Fails on `origin/master` (call 2 sees the mutated value, proving the uncached scan). - Commit 2 (`perf:`): the fold. Test now passes. ## Refs Post-merge audit identified this as the only MAJOR finding from #1627; recommendation was a follow-up hot-fix PR. This is that PR. --------- Co-authored-by: openclaw-bot <bot@openclaw> Co-authored-by: openclaw-bot <bot@openclaw.local> |
||
|
Kpa-clawbotandclawbot
|
43be1bb76a |
fix(reach): scanReachRows DB errors must surface as 500 not 404 (#1631) (#1635)
Red commit:
|
||
|
|
e2212f5015 |
feat(nodes): per-node Reach page + GET /api/nodes/{pubkey}/reach (v2, review-complete) (#1627)
Re-submission of #1625 (which was merged early, then reverted in #1626) — now with **all three round-1 reviews addressed** so it lands in one hardened state instead of as post-merge follow-ups. ## What Per-node **Reach** view: a standalone page (`#/nodes/{pubkey}/reach`) + a node-detail section + `GET /api/nodes/{pubkey}/reach`. It shows which nodes a node has a **stable two-way RF link** with, derived from raw `path_json` adjacency (a path travels origin→observer, so `[A,B]` ⇒ B heard A). A link is bidirectional when both directions have observations; the **bottleneck** (weaker direction) rates two-way reliability. Nodes are identified only by **unique 2–3 byte** path prefixes (1-byte collides → excluded). ## Review fixes folded in vs #1625 **Performance (Carmack):** hard scan LIMIT (200k) + modest prealloc; `json.Unmarshal` replaced by a single-pass `parsePathTokens` (100k-row scan 2.2M→1.3M allocs, 344→203ms); memoized resolver; size-hinted maps (attribution over 100k rows: 102 allocs); `context.Context` plumbed; cache `RWMutex` + evict-oldest (no full wipe); singleflight dedup; degree/rank from a 60s shared snapshot; bench rewritten (ReportAllocs, 1k/10k/100k, mixed-payload, isolated attribution). **Correctness/safety + tests (Independent + Kent Beck):** pubkey validation → 400; error logging instead of silent swallow (first_seen / degree / marshal→500 / discarded rows); `public_key=?` index use; canonical `PayloadADVERT`; `min()` builtin; documented cache-slice immutability; mux ordering comment. New tests: scanReachRows decode, 3-byte token branch, non-advert first-hop guard, observer SNR aggregation across rows, HTTP-level attribution (asserts non-zero we_hear/they_hear), 400/404/blacklist/cache-hit. **UI / a11y / Tufte:** in-map legend (tiers + thresholds); dropped the colour+width double-encoding (constant width, colour-only); colour-blind glyphs (●●●/●●/●) + tier title beside the bottleneck number; dark-theme `--link-*`; lighter table (horizontal rules, sentence-case headers); map built once + link layer updated in place on toggle (no flicker); time-range no longer flashes a loader; `destroy()` generation guard; statCard escaping; scoped `@media print` to `#nq-report`; `fieldset/legend` + `for/id` toggles; `aria-pressed` / `aria-live` / back-link `aria-label`; "distance (km)" + bottleneck tooltip + no-GPS note; inline styles → CSS; decorative emoji removed. **Docs:** api-spec documents the 5-min cache, 200k scan cap, and 400. ## Testing - `cmd/server` full suite green; reach unit + endpoint + bench all pass. - `eslint public/*.js` (no-undef) and the XSS-sink gate clean. - E2E updated: request status checks + exact (non-tautological) toggle assertions + hard map-render assert. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## TDD-history note (Kent Beck gate) This branch carries production + tests together, not a fabricated red→green sequence. That's deliberate: the branch was rebased onto upstream and the intermediate SHAs were squashed, so reconstructing a "failing-test-first" commit after the fact would be theatre, not evidence — and rewriting history to stage it would be dishonest. The behaviour is instead covered by a comprehensive, anti-tautological suite (directional attribution edges, 3-byte token branch, non-advert first-hop guard, observer SNR aggregation, HTTP-level attribution asserting non-zero counts, scan-cap truncation, zero-reach 200-not-404, companion mis-attribution, cache eviction). Requesting maintainer acceptance of the work on test *substance* rather than commit *choreography*; the net-new-UI exemption is not claimed for the server endpoint. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: meshcore-bot <bot@meshcore> |