151 Commits

Include renames

Author	SHA1	Message	Date
n30nex	b9fb3d244c	feat(analytics): split adverts using recorded route evidence (#2088) ... Red commits: `dc7dcbe`, `76e3217` ([assertion failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37059977385)). Fixes #2041; supplies #2085's backend contract. Records at most two route bits per retained advert. Node flood counts now use the same evidence, including mixed and transport-flood adverts. Analytics failures log/count errors while core observation, path and liveness writes continue. The unrelated recent-advert limit clamp is removed. Direct adverts (empty path) describes the observed remaining path across valid hash widths. Firmware can forward an advert down to an empty path; this cannot establish original send mode or RF distance. The `zero_hop` API spelling remains compatible. Successful post-upgrade evidence writes survive observation replacement and restart. Older overwritten frames and failed writes can leave gaps. The UI explains automatic background backfill. ## Measured validation Synthetic Windows workloads; timings are not production guarantees: - 300K retained adverts, one-hour window: median 64.35→39.84 ms; 13.98 MB→145 KB allocated. Seven-day allocations unchanged; five paired samples. - 1M transmissions/11M observations: backfill 5m43s. Simple 10 Hz writer: all 3,423 observations persisted; p99 28 ms, maximum 323 ms. - Simulated pre-backfill cursor: retained evidence ready 4m21s; full catch-up 6m41s, 122 cache invalidations. Normal completed-migration startup bulk-loads current evidence. Portable opt-in scale tests and window benchmarks are included. All 183 frontend suites, full server tests, targeted races, lint and real-API desktop/mobile checks passed. Full [Linux CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/37063210229) passed at `36def9c`; three independent reviews are clear. Local Windows testing hit the unchanged symlink-privilege test. No configuration changes. Preflight overrides: external script/OpenClaw profile unavailable; repository checks and Chromium used.	2026-10-03 12:16:55 +02:00
efitenandClaude Opus 5	6e121ff8e6	perf(db): build planner statistics at startup when there are none (#2058) (#2074) ... Follow-up to #2072, which closed #2058 but left one gap named in its own description: the refresh ticker waits 2 minutes before its first run, and a query arriving in that window against a database with no statistics gets the bad plan. Deployed to staging to measure it rather than reason about it, with `sqlite_stat1` dropped first so the build path actually ran. That changed two of the numbers in #2072, both in the expensive direction. ## The gap is once per database, not once per restart `sqlite_stat1` is an ordinary table, so once `ANALYZE` has written it the statistics stay in the file. Checked four ways: - they survive closing the connection that wrote them - a `mode=ro` handle reads them back, which is how `cmd/server` opens the database - reopening the same path through a second `OpenStore` finds them and skips the rebuild (`TestPlannerStatsSurviveReopen_Issue2058`) - on staging they survived a full redeploy to a different build that has no refresh ticker at all, and that build still gets the good plan So the window opens once, on the first start after this lands, and never again for that database. ## The cost, corrected #2072 said 2.0s. Observed on staging, 9.4 GB, commit `4500cfa6`: ``` 13:51:26 [analyze] planner statistics refresh scheduled every 24h (analysis_limit=10000) 13:55:10 [analyze] planner statistics built in 3m43.874s (analysis_limit=10000, first run against this database) ``` **3m43.9s.** Every `ANALYZE` duration in #2072's ladder was timed warm, run after run; cold, on a freshly started container, the same statement takes nearly four minutes. That is the same warm/cold split #2058 work already established for the query itself, 56.7s against 0.80s, and I then repeated it for the `ANALYZE`. Every `2.0s` in the tree is now marked warm and points at the cold figure. It holds the single write connection throughout, so ingest stalls and buffers. Per minute in `observations`: | minute | rows | |---|---| | 13:49 | 220 | | 13:50 | 106 | | 13:51 | 0 | | 13:52 | 0 | | 13:53 | 0 | | 13:54 | 0 | | 13:55 | **1027** | | 13:56 | 154 | Nothing was dropped. The burst is about four minutes of traffic at the surrounding rate, and the only ingest-buffer line in the log is the startup one reporting `0 dropped`. The cost is a four-minute write stall, once, not data loss. ## This cost is not introduced here The ticker merged in #2072 pays the identical 3m43.9s two minutes later on any database with no statistics. **Live has none, so #2072 as merged will stall live ingest for about four minutes on its first run, with or without this branch.** This only moves it earlier, into the startup burst the ingest buffer is already sized for. Flagging it on #2072 as well. ## The change `Store.EnsurePlannerStats(analysisLimit)` checks before it builds: - database has statistics: one `sqlite_master` query. This is every restart after the first. - database has none: one `ANALYZE`, and a warning first. The warning is the part that earns its place operationally. Four minutes of stalled ingest with no explanation in the log looks exactly like a hang, so `EnsurePlannerStats` now says why the write path is about to pause, what it measured on 9.4 GB, and that it happens once per database. It stays silent on a restart, because a warning on every boot would be worse than none. It runs on the refresh goroutine, not the startup path, so no boot step waits for it. `hasPlannerStats` now gates a decision instead of only wording a log line, so its comment says what the swallowed error costs: a query failure reads as "no stats", which spends one unnecessary `ANALYZE` rather than skipping a necessary one. ## Verification on staging - before: plan drove from `idx_transmissions_payload_type`, no `sqlite_stat1` - after: 50 rows in `sqlite_stat1`, plan drives from `idx_tx_channel_hash` - dropping the table first flipped the plan back, so the causality holds in both directions ## Tests 13 in the file. New here: builds when absent, skips when present, disabled on a negative limit, survives close-and-reopen, warns before building, stays quiet when statistics exist. The reopen test is the guard on the whole design: if statistics ever stopped living in the file, `EnsurePlannerStats` would quietly run a four-minute `ANALYZE` on every restart and nothing else would notice. Run locally: 13/13 on the `Issue2058` tests, `go vet` clean, `gofmt` clean, and the rest of `cmd/ingestor` green apart from `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails on `os.Symlink` with "A required privilege is not held by the client" on Windows without elevation, in a file this branch does not touch. ## Not done - No query rewrite, same as #2058 and #2072. - **No live deploy.** Live still has no statistics, so the four-minute stall is ahead of it whenever #2072 ships there. Worth picking the moment. - Staging has been returned to its own fork build; the statistics it built remain, so its next start exercises the skip path rather than the build path. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_013YAR8fdNTzqjtsggq4xCX6 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-26 10:08:43 +02:00
efitenandClaude Opus 5	a5aa3cdc41	perf(db): refresh SQLite planner statistics with a bounded ANALYZE (#2058) (#2072) ... Closes #2058. `ANALYZE` has never run against these databases, so `sqlite_stat1` does not exist and the planner works from built-in guesses. On the channel queries it guesses wrong: it drives from the plain `idx_transmissions_payload_type` instead of `idx_tx_channel_hash`, the partial index (`WHERE payload_type = 5`) the schema already carries for that exact filter. @anieto's report did the diagnosis and the arithmetic. This adds the maintenance operation that was missing, at a value measured rather than assumed. ## The diagnosis transfers, the remedy needed measuring Measured on our 9.4 GB staging database: 1,250,489 transmissions, 14,169,329 observations, 2.7x and 7x the reported database. Region-filtered `GetChannels` produces the identical plan reported in #2058, down to both temp b-trees, so the problem is the same one. Wall time is the wrong metric here. The same query and the same plan measure **56.7s cold and 0.80s warm** on that file, so the OS page cache dominates. Counting page-cache misses instead: | analysis_limit | ANALYZE | driving index | page misses | |---|---|---|---| | none (no statistics) | - | `idx_transmissions_payload_type` | 143,442 | | 400 | 171 ms | `idx_transmissions_payload_type` | 143,449 | | 1000 | 171 ms | `idx_transmissions_payload_type` | 143,450 | | **10000** | **2.0 s** | **`idx_tx_channel_hash`** | **107,429** | | 0 (unbounded) | 242.9 s | `idx_tx_channel_hash` | 107,429 | 400, the value SQLite's documentation offers for the bounded form, changes nothing on this data: it samples too few rows to separate the 126,336-row partial index from the 920,700-row plain one. 10000 buys the entire plan change for 2.0 s, and the four-minute unbounded `ANALYZE` buys nothing beyond it. ## What it is worth, as measured 25% fewer pages read per query, 143,442 to 107,429, about 147 MB less at a 4 KB page. Warm wall time does not move: 0.80s either way. The gain lands on the cold path, the one that measured 56.7s, so the claim here is fewer pages read, not a warm speedup. This is smaller and differently shaped than the 3-4x in #2058. I cannot reproduce that ratio on a database of this size and am not claiming it. ## The change - `Store.RefreshPlannerStats(analysisLimit)` in `cmd/ingestor/db.go`: `PRAGMA analysis_limit=N` then `ANALYZE`, logging the duration and whether this was the first run. - Wired in `cmd/ingestor/main.go` next to the existing WAL checkpoint ticker: 24h, staggered 2 minutes past startup because it takes the write lock. - `db.analysisLimit` in `internal/dbconfig`, default 10000, negative disables it. It runs in the ingestor, not the server: `cmd/server/db.go:145` opens `mode=ro`, and `ANALYZE` writes. This respects the read/write separation invariant in AGENTS.md. `analysis_limit=0` means *no* limit to SQLite rather than "use a default", so an unset config maps to 10000 and a test covers that specific case. ## Two faults the measurement caught in my own first commit Both are in the history rather than hidden, because the second commit is the one that measured: 1. **`PRAGMA optimize` was the wrong statement.** It analyzes only tables the calling connection has itself queried during the session, and a maintenance call has queried none. Run against staging it wrote nothing and left `sqlite_stat1` absent; `PRAGMA optimize(0x03)` returned no statements at all. Verified on an empty database too (SQLite 3.45.1): `ANALYZE` creates `sqlite_stat1`, `PRAGMA optimize` does not. That difference is what makes the behavioural test a guard instead of a no-op. 2. **`analysis_limit=400` was the wrong value**, per the table above. ## Tests Six cases in `cmd/ingestor/refresh_planner_stats_test.go`: - statistics are actually written (the guard against returning to `PRAGMA optimize`) - the pragma reaches the connection, read back through `PRAGMA analysis_limit` - a negative limit leaves `sqlite_stat1` absent - two consecutive refreshes, since a ticker calls this repeatedly - the config default and the JSON round trip - the default is above the range measured ineffective, so lowering it back to 400 fails ## Not done, and one caveat - **Correction to an earlier version of this description**, which said the Go tests could not run locally because this box has no C compiler. That was wrong: `CGO_ENABLED=0` and `gcc` being absent from `PATH` is not the same as no compiler, and a mingw-w64 toolchain is installed here. Run properly, all six tests pass locally, and so does the rest of `cmd/ingestor` apart from `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails on `os.Symlink` with "A required privilege is not held by the client" on Windows without elevation and lives in `stats_file_test.go`, a file this branch does not touch. CI agrees: Go Build & Test and the ingestor race detector are both green. - The SQLite behaviours above were measured against 3.45.1 on the server, not against the amalgamation `mattn/go-sqlite3` bundles. - **No query rewrite.** #2058 explicitly left that out and so does this; the correctness caveats it lists (per-channel most-recent-message semantics, v2/v3 branches, `enc_` exclusion) are untouched here. - Staging carries limit-10000 statistics, matching what this code produces. Reversible: `DROP TABLE sqlite_stat1` was verified on a scratch database before any of it ran. - Whether a cold-start `ANALYZE` should also run before the 2 minute stagger is not addressed. The first query after a restart is the expensive one, and it can arrive first. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_013YAR8fdNTzqjtsggq4xCX6 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-25 00:08:54 +02:00
3 people	291393dcc0	feat(ingestor): log a throttled warning when the IATA whitelist drops a region (#2067) ... Takes over #2008 by @nullrouten0, as offered there on 2026-09-16 and 2026-09-17. **Their commit is the first of the two here, unchanged and under their authorship**; the second is only the fix for the one blocker. Closing #2008 in favour of this so the rebase and the fix travel together, not to reassign the work. ## The feature, unchanged `observerIATAWhitelist` dropped non-whitelisted regions silently. An allow-list fails in the dangerous direction: a legitimate but unlisted region vanishes with nothing to show for it. One line per dropped region now, re-logged at most every `iataWarnIntervalSec` (new optional key, default 6h) for as long as that region keeps arriving. The periodic re-log rather than a strict log-once is the author's call and it is the right one: a single edge event rolls out of any scrape window, leaving an actively-dropping region indistinguishable from a healthy one. ## The blocker, now fixed `ShouldWarnIATADrop` keyed its throttle map on a topic segment the **publisher** controls and never evicted it — a remote memory sink. Measured on the original branch: 200,000 distinct codes retained 200,000 entries and 15.1 MB of heap. My review offered two shapes. This takes the cap rather than shape-validation, and the reason matters: **nothing in this codebase constrains an IATA code's shape.** It is uppercased and trimmed in `config.go` and `db.go` and never validated. Rejecting by shape would invent a rule operators have not agreed to, and would silently drop the warning for anyone whose code does not fit it — the same failure mode, one level down. So `iataWarnMaxTracked = 512`: far above any real deployment (the reference instance runs 43 observers across a handful of regions) and small enough that a hostile feed gains nothing. **Past the cap the drop is still logged**, throttled on one shared timestamp instead of a per-code one. Swallowing it there would reintroduce exactly the silent failure this feature exists to fix. ## Tests The author's `iata_drop_warn_test.go` plus three: - the map stops growing when fed 2048 distinct codes - a new code past the cap still warns once, is then throttled, and speaks again after the interval elapses - an already-tracked code's throttling is unchanged, so the cap does not alter the normal path ## Verification `gofmt` clean, cherry-picked cleanly onto current master (`cmd/ingestor/main.go` auto-merged). Go tests not run locally: no cgo toolchain here since #1992, and per AGENTS.md `CGO_ENABLED=0` links a stub that proves nothing. CI is their first run. ## Not done The `iataWarnIntervalSec` key is undocumented outside the struct comment. If there is a config reference that should list it, say where and I will add it. --------- Co-authored-by: nullrouten <nullrouten@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-23 10:58:27 +02:00
efitenandClaude Opus 5	50c4d9615b	test(ingestor): wait for the boot migrations before handing over a test store (#2066) ... Closes #2065. Master's `🏁 Race detector (ingestor)` job has been red since the pushes at 2026-09-22 21:55 and 21:56. ## Correcting my own diagnosis The issue says the fault is a goroutine outliving its test and racing a later one, and proposes making it joinable. That was wrong, and it matters because it changes the fix: `Close()` **already** waits on `backfillWg` (`cmd/ingestor/db.go`), and `newTestStore` registers it as `t.Cleanup`. The goroutines are joined before the next test starts. The race is inside a single test. `OpenStore` schedules two async migrations — `obs_observer_ts_idx_v1` and `tx_last_seen_backfill_v1` — whose goroutines log while they run. `TestHandleMessageDecodeErrorLog_PII_Issue1211` then points the standard logger at a `bytes.Buffer` and reads it, so its **own** store's migrations write into the buffer it reads: ``` Write by goroutine 760: RunAsyncMigration.func1 async_migration.go:124 (log.Printf) Read by goroutine 757: ...PII_Issue1211 decode_error_log_test.go:37 (buf.String) ``` ## Why the helper rather than the one test These tests capture the standard logger in **21 places across 7 files**. Any of them that also builds a store is exposed to the same thing; the decode-error test is just the one whose timing lost. So `newTestStore` now waits after `OpenStore` instead of only at cleanup, and no test body can run while a migration is in flight. Checked before touching a shared helper: no test references either boot migration by name, and the `pending_async` assertions in `async_migration_test.go` use their own names with a blocking `fn`, so they are unaffected. Cost is a few milliseconds against an empty temp database. ## Tests The race detector only catches this when the scheduler cooperates — it sat latent from 2026-09-03, when those files were last touched, until it surfaced three weeks later, and a re-run would have made it look like a flake. So both new tests are deterministic: - **`TestNewTestStoreWaitsForBootMigrations`** — `tx_last_seen_backfill_v1` is scheduled unconditionally by `OpenStore`, so on a fresh temp database it is pending at that instant and can only read `done` if something waited. Remove the wait and this fails every run. - **`TestCapturedLogIsFreeOfMigrationOutput`** — asserts a captured buffer holds no `[migration/async]` or `[async-migration]` output, which is the failing test's own situation stated as an assertion. ## Verification `gofmt` clean. Go tests were not run locally: no cgo toolchain on this machine since #1992, and per AGENTS.md `CGO_ENABLED=0` links a stub that proves nothing. CI is their first run, and the race-detector job is the one that matters here. ## Not done The decode-error path still logs through the standard logger, so a future test capturing it while any other goroutine logs will race again. An injectable logger would close that class properly. This closes the store-boot case, which is the one that exists today. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-23 10:23:59 +02:00
efiten	e01565737a	feat(ingestor): store CoreDrive RX region answers, with position, clock and retention (#2047) ... The Scope Audit page said a repeater's declared region list can come from CoreDrive RX while nothing the app sends ever reached it: the client-topic switch handled packets and rf only, so /regions was dropped without a log line, and node_declared_regions is read by region_keys.go and config.go but created by nothing in this tree. #2044 found that gap and proved it against a live instance. This lands the implementation that has been carrying the feature in production on the ON8AR fork since 2026-09-06, the instance CoreDrive RX publishes to. Measured there: 1840 answers about 275 repeaters from 51 collectors, 2026-08-18 to 2026-09-19. Beyond storing the answer it keeps three things the first version did not: position (lat, lon, pos_acc_m, filled on 1263 of 1840 rows, with acc_m dropped when the fix it qualifies was rejected), repeater_clock (filled on all 1840, so a wrong repeater clock cannot make an answer look newer than it is), and retention with per-collector history (pruneOldClientDeclaredRegionsAt bounds by age instead of keeping one row per target). On that dataset 138 of 275 repeaters have answers from more than one collector and 40 have collectors that disagree about the region list, which is the signal the Scope Audit exists to surface and which only survives while more than one answer does. It gates on its own clientRegions block rather than riding on clientRxCoverage, so region answers can be accepted without GPS-tagged reception uploads, and AES block padding is trimmed from region names on ingest. Taken from #2044 with the author credited as co-author: declaredRegionsTablePresent() and its test, a real bug this version lacked (supervisord starts both processes together, so a server that probes first ignores every answer until its next restart), and the docs/client-rx-coverage.md section. Ported by cherry-picking the fork's nine commits rather than retyping, so this is the code that has been running. CI run 35434151026 is green: server ok 80.177s, ingestor ok 99.413s, race detector ok 112.406s, no --- FAIL lines. Merged by the interim maintainer without a second human reviewer: CI and the production figures above are the independent checks.	2026-09-19 11:44:47 +02:00
efiten	5430bc7923	test(ingestor): anchor the RF-sample fixtures to now, not to a calendar date (#2034) ... The three ClientRfDeltas tests seeded 2026-08-17T10:00:00.000Z and queried that window back. resolveRxTimeCore (cmd/ingestor/main.go:1527) replaces timestamps older than 30 days with the ingest time, so from 2026-09-16T10:00Z the seeds landed at time.Now() and every delta fell outside the queried window. Master and every open PR went red on it. Fixtures now derive from a package-level base two hours in the past, computed once per test binary so two seeds cannot straddle a second boundary and break the exact WallMillis assertion. Merged by the interim maintainer without a second human reviewer: CI is the only independent check (run 35222316327, ingestor tests ok in 97.033s, race detector ok, no --- FAIL). Fixed dates elsewhere in the ingestor tests are untouched, they assert row counts rather than querying by the seeded date.	2026-09-17 16:52:20 +02:00
Sylvain Rabot	a2ea18f778	perf(sqlite): swap modernc.org/sqlite for mattn/go-sqlite3, cross-built with zig (#1992) ... Swaps the SQLite driver from `modernc.org/sqlite` (pure Go, SQLite 3.46.0) to `github.com/mattn/go-sqlite3` (cgo, bundled SQLite 3.53.4), and pays the resulting cross-compilation cost with `zig cc`. Draft because the riskiest part of this deletes rows — see [Please review this part first](#please-review-this-part-first) — and because three things remain unverified at the bottom. `modernc.org/sqlite` is a transpilation of the C amalgamation. This repo is read-heavy: `cmd/server` chunk-loads a graph at startup and fans out neighbour/topology/analytics queries per request, and it pays for that transpilation on exactly those paths. Head-to-head on the same 120k-transmission / 240k-observation database, running our own hot-path SQL under both drivers (Apple M4, `-count=5`, medians): | workload | modernc | mattn | | |---|---:|---:|---| | chunk load (`chunked_load.go` v3 join, 20k tx) | 449ms | 196ms | **2.3×** | | aggregate scan (240k-row join + `GROUP BY`) | 276ms | 137ms | **2.0×** | | 1500 prepared-statement lookups | 512ms | 403ms | **1.3×** | Allocations fall with it: 1.12M vs 1.64M allocs and 21MB vs 30MB on the chunk load. **Superseded by a production run.** @efiten measured both drivers on a real instance — 11,077,038 observations, 9.7GB database, 4-core arm64 — as server-only containers against the same live volume, one at a time, with round 2 reversing the order so the page cache favours the old driver: | | audit 7d | audit 24h | background fill (13 chunks) | start → /api/health | |---|---:|---:|---:|---:| | modernc, round 1 | 16.67s | 2.27s | 130.2s | 16.6s | | mattn, round 1 | 7.87s | 1.34s | 93.8s | 13.5s | | mattn, round 2 | 8.15s | 1.35s | 96.4s | 13.0s | | modernc, round 2 | 13.46s | 2.29s | 137.8s | 15.5s | Warm, the old driver improves to 13.46s on the 7d audit and still loses by ~1.8×. Chunk load is ~1.4×. `/api/nodes?limit=500` is 0.039s against 0.037s — nothing. **So the real gain is ~1.4–1.8× on the paths that matter, not 2–2.3×.** The shape the harness predicted holds — scans and joins gain, small lookups do not — which is more reassuring than the magnitude would have been. Quote these numbers. **The counterweight**, cold and native on that machine: a build goes from **52s to 163s**. An instance that builds its own image pays that per deploy. ## The build is cgo now, and one thing about that is a trap **`CGO_ENABLED=0` still builds.** mattn links a stub, and the binary dies on its first query with `go-sqlite3 requires cgo to work. This is a stub`. A green build is not evidence of anything here, which is why `AGENTS.md` now says so explicitly. `GOOS=linux go build` genuinely cannot cross-compile any more. A new root `Makefile` is the entry point. `make crossbuild` uses `zig cc -target {x86_64,aarch64}-linux-musl` and links static, so each artifact stays a single self-contained file and the `alpine:3.20` runtime no longer depends on the base image's libc at all. `-Wl,-s` is load-bearing: Go's own `-s -w` does not reach the musl objects zig links in, and without it the server binary is 19.8MB instead of 12.1MB. The Dockerfile keeps its single `$BUILDPLATFORM` builder — still no QEMU for compilation — and gains a checksum-pinned zig plus BuildKit cache mounts. The mounts are not a nicety: without them an image build recompiles the amalgamation from cold and takes over half an hour. ## Please review this part first `internal/dbschema/dedup_index.go` **deletes observation rows**. It is the one part of this change that can lose data, and it exists because the migration exposed a real bug rather than causing one. `stmtInsertObservation` resolves its `ON CONFLICT` against `idx_observations_dedup`, which `cmd/ingestor/db.go` only ever created inside the branch that creates the `observations` table for the first time. Any database whose table predates that branch never got one, so the UPSERT had no conflict target. modernc failed on the first insert; mattn fails at `OpenStore`. Same bug, found earlier. Creating the index unconditionally repairs it — but the index is what was supposed to prevent duplicates, so a database that never had it can already hold rows violating it. **`test-fixtures/e2e-fixture.db` in this repo holds one.** So duplicates are collapsed first. Refusing is not the safer option: without the index the ingestor cannot prepare its UPSERT, so it cannot start at all. Replaying that UPSERT faithfully is subtler than it looks, and a first cut of this got it wrong twice: - `COALESCE(excluded.x, x)` means the **incoming** value wins, so down a group in id order the survivor keeps the **last** non-NULL value. Taking the first silently discarded newer readings. - The UPSERT names exactly five columns (`snr`, `rssi`, `score`, `raw_hex`, `resolved_path`). Every other column must keep the surviving row's own value; merging those too invents history the ingestor would never have written. Merge, delete and `CREATE UNIQUE INDEX` now share one transaction. Split apart, a writer inserting a duplicate in the gap fails the index creation while leaving the deletions committed — rows destroyed and no index to show for it. Cost, measured on 2.4M synthetic rows holding 5 duplicates: **4.1s**, holding the write lock throughout, once, at ingestor startup before MQTT subscribe. Materialising the duplicate-group scan once rather than per column took that from 9.7s; the pathological case (400k of 600k rows duplicated) is 5.7s, slightly worse than the 4.2s it was before that change. ## Four more behavioural differences Full detail in `docs/sqlite-driver-migration.md`. Briefly: **Statement preparation is eager.** modernc's `newStmt` stored the SQL and compiled lazily; mattn calls `sqlite3_prepare_v2` inside `Prepare`, so SQL naming a missing table fails at *open*. 59 server tests failed on this alone, all fixtures with partial schemas. `OpenDB` keeps failing loudly (#1901; `main.go` gates on `dbschema.AssertReady` anyway) and the fixtures now declare what they are prepared against via `ensurePreparable`. This also exposed nine `nodes(pubkey …)` declarations across seven files, where production has only ever had `public_key` — lazy compilation had hidden the mismatch for as long as it existed. **`synchronous` silently dropped FULL → NORMAL.** mattn defaults it to NORMAL and executes the pragma unconditionally, where SQLite's own default (what modernc left alone) is FULL. In WAL mode that weakens durability under power loss. Pinned in `dbschema.WriterDSN`, which both writers now share — `cmd/migrate` kept a bare path at first and so quietly wrote at NORMAL, which is what a second copy of a DSN buys you. **The DSN dialects are mutually invisible.** modernc understood only `_pragma=name(value)`, mattn only `_`-prefixed parameters, and neither errors on the other's form — a driver-only rename would have dropped every pragma in silence. `_journal_mode=WAL` is also gone from the server's read handle: modernc ignored it, mattn honours it, and setting `journal_mode` on a read-only connection is a write. Dropping `_busy_timeout` with it costs nothing, since mattn already defaults to 5000ms — which means the read handle finally *gets* the busy timeout it had silently lacked. **`mode=ro` survives for a non-obvious reason.** mattn always passes `READWRITE|CREATE` and its amalgamation has `SQLITE_USE_URI=0`; what makes the URI work is its C wrapper ORing `SQLITE_OPEN_URI` in. So the #1283/#1289 invariant holds with no build flags — but it depends on the `file:` prefix. `cmd/decrypt` had been building its DSN without one, so its `mode=ro` had never applied and a missing path was created read-write. Fixed in passing; never a migration regression. ## What did not change No modernc-specific API was in use: no `RegisterFunction`, no `*sqlite.Conn`, no `sqlite/lib` error constants, no `sql.Register`. No `time.Time` is ever bound as a query argument, so driver time handling is not in play. Both drivers convert declared `DATE`/`DATETIME`/`TIMESTAMP` columns to `time.Time`, so `/api/dropped-packets` keeps emitting `dropped_at` as RFC3339 — an earlier draft "fixed" that with a `CAST` and would have been the regression. ## Tests and CI New regression tests, each written because something got through without it: - `TestEnsureObservationsDedupIndexKeepsLatestValues` — the merge ordering. The original test used complementary NULLs, which passes whichever direction you pick, which is why the bug survived it. - `TestCollapseDuplicatesAndIndexIsAtomic` — a failed index creation must roll the deletions back. - `TestOpenStorePragmas` / `TestWriterDSNPragmas` — every writer pragma, read back through the store's own connection. A separate `sqlite3` session or the startup log line would prove nothing. - `TestOpenDBRefusesMissingDatabase` — the read-only invariant, which now rests on a detail of the driver's C wrapper. - `TestEnsurePreparableMatchesPrepareStatements` — fails when a new prepared statement outgrows the fixture helper. CI gains test execution for `cmd/migrate` and `internal/dbschema`, which had none and both open the database. A PR-time two-arch build plus an arm64 QEMU smoke gate is new: the GHCR push is push/tag-only, so without it nothing on a PR would exercise zig, static musl linking or arm64, and the first signal would arrive on master. `cache-dependency-path` widens from 2 of the 5 tracked `go.sum` files to all of them. `make test` passes across all 14 modules, `cmd/server` also under `-race -count=2` with no failures and no races. `gofmt` and `go vet` clean. Release-routing and Dockerfile COPY-invariant gates pass. ## Verified by running - All 8 cross-builds static and correct-architecture; both arches of the container image built, exported and run under QEMU, serving `/api/health` and `/api/nodes` against a 2.9M-observation production snapshot. - The `migrate` binary repairing that snapshot's duplicate on bare Alpine. - `CGO_ENABLED=0` producing a binary that builds and then fails on first query. ## Not verified - ~~The 2–2.3× figures come from a standalone harness, not this load under the old driver.~~ **Closed** by @efiten's production run above, which also corrected the multiplier. - SQLite 3.46.0 → 3.53.4 query-planner differences on queries with no total `ORDER BY`. - Sustained live ingest through the new writer DSN, and the duplicate collapse against a database an ingestor is actively writing to. Verified against a static snapshot only, and the collapse is measured at 4.1s on 2.4M synthetic rows with 5 duplicates — well short of an 11M-row instance. @efiten has offered a staging instance taking real MQTT traffic; **this is the item to close before the PR leaves draft.** An earlier revision of this branch shipped the dedup merge in the wrong direction with a green test suite, and review then found three more things in the same file: the repair gated on an error string, a non-atomic TEMP table drop aimed at the wrong connection, and a deletion whose only record was a row count. All fixed in ac7e8d38. Passing tests did not establish safety here, which is why the deletion path wanted a second pair of eyes rather than a rubber stamp.	2026-09-16 09:02:13 +02:00
efitenandClaude Opus 5	5efe61eef2	fix(ingestor): set an explicit MQTT ClientID per source (#2013) (#2016) ... ## What `buildMQTTOpts` (`cmd/ingestor/main.go:591`) never called `SetClientID`, so with paho.mqtt.golang v1.5.0 every ingestor connected with a zero-length ClientID and `CleanSession=true`. The session identity then depended on the broker. This PR: - adds an optional `clientId` per `mqttSources` entry (`cmd/ingestor/config.go:29`) - when unset, uses `corescope-<name>-<6 hex chars>` (`cmd/ingestor/main.go:653`). The name is reduced to `[0-9A-Za-z-]`, with the broker host as fallback when the name is empty. The suffix comes from `crypto/rand` and changes on every ingestor start. - sets the ID once per source in `buildMQTTOpts` (`cmd/ingestor/main.go:616`). paho copies the options into the client and reuses them for every reconnect, and the watchdog force-reconnect reuses the same client, so the ID is stable for the life of the process. - logs the ID on connect: `MQTT [tag] connected to <broker> as client <id>` (`cmd/ingestor/main.go:150`) - documents the key in `config.example.json:210` as a `_comment_clientId` entry rather than a value, because `docker/entrypoint.sh:6` copies that file as a live config and a literal value would give every default deployment the same ID. Also listed in `cmd/ingestor/README.md:94`. ## paho behaviour - No client-side length limit. `SetClientID` only stores the value; the 65535 check in `packets/connect.go:156` is in `Validate()`, which the client never calls. - The default ID is longer than the MQTT 3.1 limit of 23 characters for most source names. paho falls back to MQTT 3.1 after any refused CONNACK when no protocol version is set (`client.go:412`), so on a broker that refuses the first 3.1.1 attempt, the retry may hit that limit. I did not cap the length because paho does not require it and the 3.1.1 path accepts it (see below). ## Tests `cmd/ingestor/mqtt_opts_test.go:53-107`: - default ID is non-empty, has the sanitized name prefix, and contains only `[0-9A-Za-z-]` - broker host is used when the name is empty - configured `clientId` is used verbatim - two unconfigured sources with the same name get different IDs - the client built from the options reports the same ID Mutation checks: removing the random bytes fails the "different IDs" test; removing sanitization fails the prefix and character-set tests. `go test ./...` in `cmd/ingestor` passes except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails locally on Windows for a symlink privilege reason. `gofmt` and `go vet` are clean. ## Validation against a real broker On a staging instance (build `e84d2da6`) connecting to a Mosquitto bridge: ``` MQTT [lincomatic] connection attempt #1 to tcp://mosquitto-bridge:1883 MQTT [lincomatic] connected to tcp://mosquitto-bridge:1883 as client corescope-lincomatic-71a6eb MQTT [lincomatic] subscribed to meshcore/# ``` The 27-character default was accepted on the first attempt and packets kept arriving afterwards. ## Not verified - Only one broker type (Mosquitto) was tried. - `-race` was not run locally (no cgo toolchain on the test machine). - The case where both the source name and the broker host are empty (ID becomes `corescope-<hex>`) has no test. Fixes #2013 ## Review follow-up (commit `a1d6709e`) An independent review found no bug in the ID handling, but the tests covered less than their names said. Changed, tests only (`cmd/ingestor/mqtt_opts_test.go`): - `TestBuildMQTTOpts_ClientIDSurvivesReconnects` replaces the old stability test, which only checked that paho copies the options. Against a loopback fake broker built on paho's `packets` codec, the first CONNECT, paho's auto-reconnect after the broker drops the socket, and the watchdog force-reconnect (`buildForceReconnectFn`) must all carry the same non-empty ID. It runs in about 0.01 s and passed `-count=30 -cpu 1,2,8`. - `TestBuildMQTTOpts_ClientIDDefaultShape` asserts full IDs: `^corescope-local-feed-1-[0-9a-f]{6}$`, `^corescope-mqtt-example-com-[0-9a-f]{6}$` for the broker host fallback (no port), and `^corescope-[0-9a-f]{6}$` with neither a name nor a host. - Mutations now caught: `SetClientID` removed, `u.Host` instead of `u.Hostname()`, a 1-byte suffix, the name guard dropped, sanitization removed. The "as client" log line has no test because it is logged from a closure inside `main()`. Corrections to the description: - **Fallback to MQTT 3.1.** paho falls back after any failed handshake once the socket is open, not only after a refused CONNACK: also a read error or timeout before any CONNACK, or a first packet that is not a CONNACK (`client.go:401-416`, `net.go:83-97`). A failed dial does not trigger it (`client.go:387-391`), and after the first successful connect the protocol version is locked in (`client.go:422-424`). Without this PR the 3.1 retry sent an empty ID, which MQTT 3.1 forbids as well, so nothing gets worse. - **Broker side.** On the EMQX broker we run, authorization has per-username and all-client rules and no client-ID rules (checked through its REST API). Two per-username rules use `${clientid}` in a topic, but both are publish rules and the ingestor only subscribes, so no rule can match it. A broker that caps IDs at 23 characters but accepted the empty ID before would now reject the default ID for source names of 7 or more characters; I have no evidence such a broker is in use. - The "Not verified" item about the empty name and host case no longer applies. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-13 19:57:37 +02:00
liquidraver	fe37f1060c	fix(ingestor): delete aged packets in bounded batches so prune stops stalling ingest (#2000) ... Reviewed at ecf0b371: query plans dumped and confirmed index-driven for all three statements, termination proven against concurrent ingest (first_seen is always time.Now()), FK child-first ordering required and correct, writer-stats assertions non-racy. Two low findings noted on the PR for follow-up: the dropped RowsAffected error now gates the loop, and ~0.53s batches will trip defaultSlowWriterMs=500.	2026-09-11 11:29:49 +02:00
efiten	02feb2a88e	test(ingestor): join the watchdog loop goroutine instead of only asking it to stop (#2003) ... Verified before merging: on upstream/master `go test ./cmd/ingestor -run TestMQTTStallWatchdog -count=20` fails; on this branch the same command passes. The flake blocked CI on #2000.	2026-09-11 10:58:56 +02:00
efiten	0c7f2306f6	feat(ingestor): keep the scope-match tally across restarts (#2002) ... ## What `scopeMatchCounters` (unique / explicit-over-derived / ambiguous / none) lives only in the ingestor process, and the only way to read it is the periodic log line. A restart zeroes the counters, and recreating the container removes that log with it, so a measurement in progress cannot be recovered afterwards. That happened here on 2026-09-10: a 24h ambiguity measurement completed, two deploys followed before it was read, and nothing on disk held the number. `/var/lib/docker/containers/*/*-json.log` had no earlier copy. The tally gates a real decision (whether the collision tie-break from the `autoRegionKeys` design is worth building), and that needs days of traffic, so it has to survive the process counting it. ## How A single-row table, `scope_match_totals`: - `OpenStore` restores the counters from it, so counting continues instead of restarting. - The 5-minute stats ticker writes them back, and so does the shutdown path, which is what a deploy triggers. - `since_unix` carries the window start across restarts, so the ratio keeps a denominator. The periodic log line now prints it. Saving rides the **stats** ticker, not the region-refresh ticker: matches are recorded for every transport-scoped packet, including on instances that never enable `autoRegionKeys` and so never start the refresh loop. The table is **not** in `internal/dbschema` on purpose. The server neither reads nor PRAGMA-detects it; putting it in `AssertReady` would make an older DB fail the server's startup check over data the server never looks at. A failed restore is logged and ingestion continues. Losing an observability counter is not a reason to stop ingesting. ## Tests Four, all in `cmd/ingestor/scope_match_tally_test.go`: - totals and `since_unix` restored across a close/reopen - a fresh DB gets its anchor row immediately, so the first window has a start time - recording after a restore adds to the carried total instead of counting from zero - repeated saves keep exactly one row Full `cmd/ingestor` suite green locally except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails on Windows only (`os.Symlink` needs a privilege this account lacks) and predates this branch. `go vet` and `gofmt` clean. The race detector was not run locally (no cgo toolchain here); the `race-test` job covers it, since this branch touches `cmd/ingestor`. ## Not done No UI or API surface for the tally. It is still read from the log line or straight from the table.	2026-09-11 00:00:45 +02:00
efiten	6c92a8b612	test(ingestor): make the suite race-clean, and run the detector when it matters (#1994) ... `go test -race ./...` on `cmd/ingestor` reports **six data races** on master. None is in production logic. All six come from test helpers that outlive the test that started them, which is why the detector blames whichever test happens to be running: two different tests failed on two consecutive runs of the same code. ## What was racing **`StartStatsFileWriter` had no way to stop.** Two tests start it at a 50ms interval, and its goroutine then runs for the rest of the process. It reads the package-level `readProcSelfIOFn` hook, which a later test replaces to inject a fake, so the write and the read race. The same leak explains the stray log lines about writing stats into temp directories that were already cleaned up. It now returns a stop function that closes the goroutine and waits for it to exit. Production ignores the return value and runs for the process lifetime exactly as before; the two tests call it through `t.Cleanup`. **The migration test read a log buffer while a goroutine wrote to it.** `log.Logger` serialises its own writes, but `logContains` read `buf.String()` outside that lock while `RunAsyncMigration` kept logging after the call that started it had returned. The capture helper now uses a mutex-protected buffer. That one is worth calling a real race rather than a test artefact: a concurrent read during a buffer grow can panic outright with "concurrent map read and map write"-class behaviour, not merely trip `-race`. ## Measured `go test -race ./...` on linux/arm64 under go1.27.1: **exit 0, zero races, 704s**. ## The CI job, and why it is shaped this way The server has had `-race` since #1208. This closes the same gap for the ingestor, which carries an `atomic.Pointer` snapshot (the region key set from #1989) whose safety has been an argument rather than a measurement. Two deliberate choices, because a check that costs too much gets switched off: - **Its own job, not a step inside "Go Build & Test".** Appending `-race` there puts its ten-odd minutes on the critical path, taking the pipeline from roughly 20 minutes to roughly 32. As a separate job it runs beside the E2E job (15-17 minutes) and hides inside that window. - **Only when `cmd/ingestor/**.go` changed**, decided by the existing change-scope job, which already gates the heavy jobs on documentation-only PRs. A frontend or docs PR cannot introduce a data race in the ingestor. Pushes to master always run it, as they already do for `code`. Nothing `needs:` the new job. Adding it to `build-and-publish` would mean a skipped race job skips everything downstream, which is the opposite of what a conditional check should do. It reports as its own check; whether that blocks a merge is a repository setting rather than workflow logic. ## Scope Test helpers, one production signature (`StartStatsFileWriter` now returns a stop function), and the workflow. No change to what the ingestor does at runtime.	2026-09-10 22:30:27 +02:00
efiten	0e607c1d01	feat(ingestor): derive region keys from what nodes declare, opt-in (#1989) ... Follow-up to #1988, which made an ambiguous match deterministic. This adds the second tier of keys that ambiguity rule was needed for. A transport-scoped packet can only be named by a region key this instance holds. `hashRegions` is a hand-maintained list, so every region a node forwards that nobody typed into the config is stored unmatched, and everything downstream reports that region as **absent** rather than as **unnameable**. The instance already knows the names, though: `nodes.configured_scope` holds what the observer `/neighbors` ingestion (#1865) confirmed each node is configured for. This derives keys from those names, on top of the explicit list rather than instead of it. **Default off.** An absent config block leaves behaviour byte-for-byte unchanged, asserted by tests rather than argued. ## Sources Two, mirroring what the server's `AllCurrentDeclaredRegions` already merges, so the derived tier sees exactly what the Scope Audit sees: | source | availability | |---|---| | `nodes.configured_scope` | always — the column is part of the schema, written by the `/neighbors` path | | `node_declared_regions` | optional, where a deployment fills it by other means | The optional table is probed via `sqlite_master` before it is read. A stock install does not have it, and its absence must not abort a refresh the first source could answer on its own. ## The two spellings The sources spell the same region differently, and both are accepted: - `configured_scope` carries the leading `#` that `normalizeScopeList` adds, because every other stored scope value has one - an OTA answer in the optional table carries the bare name - `loadRegionKeys` already prefixes a missing `#` before hashing So `regionNameAcceptable` canonicalises before judging, and hands back the bare name the caller re-prefixes. What it rejects is what cannot be a region name at all: `*` (the flood wildcard, not a region), a comma (it would split the name on the next round-trip through a comma-separated column), a second `#`, whitespace, non-ASCII, NUL padding from a stale client, and anything past 32 characters. The rules are deliberately structural rather than about meaning. A real declared set contains entries that look like junk, but a blocklist on string values is unmaintainable, and the cost of one bad name is a single slot out of the cap plus a 1-in-65536 collision chance. `*` is skipped in **two** places on purpose: in the filter, so no key is ever derived for it, and in the source count, because nearly every node declares it and counting it would overstate both the cap arithmetic and the refresh log on every deployment. ## Rule 0 Each derived key costs one HMAC per transport-scoped packet and raises the random 2-byte collision rate by 1/65536. So: - the tier is **capped** (default 256) - over the cap, names are kept by how many distinct nodes declare them, so a one-off local name is dropped before a region half the network uses - benchmarked linear at ~0.65µs per key: at 314 keys that is 217µs per packet, 0.0008% of one core at the 0.037 transport-scoped packets/s this network produces There is no indexable shortcut to reach for, for the same reason as in #1988: `code1` is an HMAC over the payload, so nothing is payload-independent. The set is an `atomic.Pointer` to an immutable snapshot. A refresh builds the replacement off to the side and swaps the pointer, so the ingest path never blocks on a rebuild. `refreshDerived` is a load-then-store rather than a CAS loop, which is safe only because exactly one goroutine calls it: the refresh ticker, plus one synchronous call at startup. The comment says so, rather than leaving the type looking as though it tolerates concurrent refreshers. ## Tier 2, and the counters When several keys match one packet and exactly one of them is explicit operator config, the explicit one wins: an operator who typed a region into `hashRegions` outranks a name overheard on the air. Ambiguity between two equally-sourced keys still stores unmatched, unchanged from #1988. Counters tally how each packet was decided (unique / explicit-over-derived / ambiguous / none) and are logged on the refresh tick. They exist to answer one question with data rather than estimation: whether a third tier that breaks ties on path evidence is worth building at all. Measured on a live instance at 159 keys over 41.7 hours and 147,535 packets: **0.253% ambiguous**, with explicit-over-derived at zero because that instance has exactly one derived key. ## Rule 8 `maxDerived` and `refreshMinutes` are configurable values and belong in the customizer eventually. Documented in `config.example.json` for now, flagged here so it is tracked rather than forgotten. ## Tests - default off, and a refresh that is a no-op while disabled - the name filter across both spellings, the wildcard, and each structural rejection - ranking by declarer count, then recency, then name, so the result is deterministic rather than churning between refreshes - `configured_scope` as a source, including that a node counts once per name and the newest answer wins - both sources merged, neither dropped - the explicit-over-derived tie-break - the derived tier replaced rather than merged on refresh, so a region that stops being declared leaves the key set and the cap keeps meaning something `cd cmd/ingestor && go test ./...` passes apart from `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which needs `SeCreateSymbolicLinkPrivilege` and fails on Windows on master too. `go vet` and `gofmt -l` clean, `config.example.json` still parses.	2026-09-09 17:58:04 +02:00
efiten	fd825a779c	fix(ingestor): name a region scope deterministically, or not at all (#1988) ... `matchScope` returns the first configured region whose derived code equals the packet's `code1` and stops there. Go randomises map iteration order per `range`, so when two configured regions collide on a payload, the stored region name depends on which key the runtime happened to visit first. **The same packet can be named differently on two runs of the same binary**, and neither answer is evidence of anything. ## How often this actually happens `code1` is two bytes, so any two configured regions collide on a given payload with probability 1/65536. That is a curiosity at 5 configured regions and routine at 150. Measured on a live instance carrying 159 configured regions, over 41.7 hours and 147,535 transport-scoped packets: **374 collisions, 0.253% of decisions.** They concentrate on four key pairs rather than scattering, because `code1` is an HMAC over the payload: a payload that collides collides every time it is seen, and a flooded packet is seen by many observers. The existing comment sizes the function for "≤ 50 regions". A live BE/NL instance declares 126 distinct region names across its repeaters, so operators are already past that. ## The rule `matchingRegions` returns every match; `matchScope` applies one rule: - exactly one match names the packet - several matches name nothing The candidates are equally sourced, there is no principled winner between them, and storing a wrong region name is worse than storing none. `""` is already the ingestor's "transport-scoped but unnameable" state (`scopeNameForDB`), so an ambiguous packet lands in a state the rest of the system already understands rather than in a new one. Nothing downstream needs to learn a new value. The collision is logged, because it is otherwise invisible: such a packet is stored exactly like one whose region this instance holds no key for. An operator watching an unnameable count grow deserves to see which of their own configured regions are colliding, since the fix is theirs to make. ## Rule 0 Cost is unchanged: the same single pass over the same keys, it just no longer stops early. The early exit was worth nothing on the common path, where zero or one key matches and the loop runs to the end either way. Worst case is unchanged at one HMAC per key per transport-scoped packet. There is no indexable shortcut to reach for. `code1` is an HMAC over the packet payload, so nothing is payload-independent to index on, and the old comment suggesting a "pre-indexed lookup table" is removed rather than left as a false lead for the next reader. ## Tests Three, and the fixture matters: - an unambiguous packet still gets its region name - a genuinely colliding payload stores the unmatched state instead of a coin flip - the ambiguous case run 50 times, because a first-match matcher passes a single iteration roughly half the time The collision is **found by searching payloads** (~65k tries, fractions of a second) rather than asserting on a hand-picked `code1`. The case only exists when the matcher genuinely finds two names for one packet, and a fabricated code would only prove the test agrees with itself. `cd cmd/ingestor && go test ./...` passes apart from `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which needs `SeCreateSymbolicLinkPrivilege` and fails on Windows on master too. `go vet` and `gofmt -l` clean.	2026-09-09 17:18:54 +02:00
efitenandSaarMesh-Bot	9d6f08c144	feat(#1865): ingest observer /neighbors as confirmed scope evidence (#1971) ... Carries SaarMesh-Bot's implementation from the closed #1867 forward onto current master, 67 commits later, and surfaces the result on the per-node Reach report. The declared region list a repeater answers with is now stored on the node as configured_scope, normalised to the same leading-# syntax default_scope already uses so the two are directly comparable. That normalisation is the point @cwichura raised on #1865 and @dborup agreed with before the original PR closed. Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>	2026-09-06 21:06:12 +02:00
3 people	40f664c587	chore(#1859): gofmt sweep + gofmt/go vet CI gate (rebase of #1881) (#1941) ... Rebase of #1881 by @SaarMesh-Bot onto current master. Their three commits are preserved, two of them cherry-picked with authorship intact; the sweep itself had to be regenerated. Opened as a new PR rather than force-pushing their branch. Closes #1881 once merged. Addresses parts 1 and 3 of #1859; part 2 landed as #1937. ## Why regenerated rather than merged The sweep in #1881 was cut on 2026-09-02 07:13 and roughly forty PRs landed after it, so it went `CONFLICTING/DIRTY`. Re-running `gofmt` on current master is cheaper and less error-prone than resolving 72 conflicts that are all whitespace. The drift it fixes also grew in the meantime: 66 files now, against 72 then, but spread differently. ## The three commits 1. **`style(#1859)`** — `gofmt -w` across the 14 modules. 66 files. 2. **`test(#1859)`** — @SaarMesh-Bot's fix for the one `go vet` copylocks finding, `cmd/ingestor/coverage_boost_test.go`: the range variable copied a `Config` embedding `sync.Once`. Cherry-picked unchanged. 3. **`ci(#1859)`** — @SaarMesh-Bot's CI step that fails on gofmt drift or vet findings, plus `.git-blame-ignore-revs`. Cherry-picked with one change, noted in the commit message: the ignore file pointed at `04bc80ee`, the sweep commit on their branch, which does not exist on this base and would make `git blame --ignore-revs-file` error. Repointed at `d3a02599`, the sweep here. ## Verification The claim "formatting only" is checked twice rather than asserted: - Every changed file is byte-identical to `gofmt(previous content)`. 0 of 66 deviate. - With line comments and all whitespace stripped, 0 of 66 files differ, so no code outside comments changed. 14 of the 66 also show doc-comment reflow. Since Go 1.19 `gofmt` re-indents indented comment blocks to tabs and inserts a blank comment line before them; the behavior matrix above `resolveHopWithContext` in `cmd/ingestor/path_resolver.go` is a clear example. That is gofmt's own output, not an edit, but it is worth naming because it makes the diff look larger than "whitespace" suggests. The gate was run locally exactly as the workflow runs it: `gofmt` clean, and `go vet` clean in all 14 modules, including `cmd/ingestor` which is what commit 2 fixes. Suites: `cmd/server` ok (80.7s), `internal/packetpath` ok (2.3s), `cmd/ingestor` passes except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which fails identically on bare master with "A required privilege is not held by the client" (Windows symlink privilege on my host, not code). ## Sequencing This should go last in the queue. The sweep touches 66 files, so merging it before the remaining open Go PRs gives each of them a conflict about nothing but formatting. After it lands the gate is active, and any PR with drift fails CI until it runs `gofmt -w`. Excluded from the sweep: the misnamed `Dockerfile.go`, which is a Dockerfile that gofmt cannot parse (the workflow excludes it too), and `docs/DEPLOYMENT.md`, which a case-insensitive filesystem surfaces as a spurious modification against `docs/deployment.md` and is unrelated. --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-03 18:52:03 +02:00
efitenandClaude Opus 5	9ae3387416	feat(ingestor): RF environment samples from mobile clients (#1906) ... > **Stacked on #1905.** This branch contains #1905's commits; review and merge that one first. The diff unique to this PR is the `client_rf_samples` table, its handler, the delta query and its retention. ## What Everything CoreDrive RX records today is anchored to a *packet*. But a drive also passes through RF conditions that exist whether or not a packet arrives: the noise floor, how busy the channel is, how many receptions fail CRC. The radio measures all three and was never asked. This samples the companion's own counters along the GPS track and stores them, so the server can render a noise-floor map, a channel-utilisation map and a CRC-error-rate map. A fixed observer cannot produce those — it measures one point forever. **Zero airtime:** `CMD_GET_STATS` is a local Bluetooth query to the attached radio. Nothing is transmitted. ## Design points worth knowing - **Absolutes are stored; deltas are derived at query time.** A lost or reordered sample then costs one interval rather than corrupting a running total. `ClientRfDeltas` breaks the chain whenever `uptime_secs` fails to increase — that is the reboot and counter-wrap detector. - **Absent is not zero, end to end.** Firmware predating the `recv_errors` field cannot count CRC errors at all, and a stored `0` would read downstream as "a perfectly clean channel" — the opposite of "we don't know". Presence/absence is preserved through the app parser, the wire payload, a nullable column, and the delta view, which returns `nil` rather than `0` when either endpoint is unknown. Each of those five layers has its own test. - **`sampled_at` is millisecond precision, and it is load-bearing.** SQLite compares these strings lexicographically and `.` (0x2E) sorts before `Z` (0x5A), so a second-resolution retention cutoff would delete rows *inside* the window. The prune formats its cutoff with the same layout. ## Performance justification (touches the ingest hot path) - One INSERT per sample, gated behind an opt-in flag that defaults off. Sample rate is 15 s while moving and 5 min while parked, so roughly 240 rows per hour per active driver. - The delta query is a single `LAG(...) OVER` pass with no nested query inside the loop, so it cannot deadlock the single writer connection. Window functions are already used elsewhere in this codebase. - Retention has its own key and index (`sampled_at`); without it the table would grow unbounded, so `config.example.json` documents it inline. ## Safety for existing deployments Opt-in and default off on both sides (`clientRfSamples.enabled`, and `rfSampler` in the app). The coverage path is untouched — `Publisher.buildPayload` is byte-identical and a record with no `kind` field still routes to `/packets` unchanged. The MQTT dispatch was reshaped so that **anything on `meshcore/client/…` returns from that branch in every config state**, with the enable-gates inside rather than in the topic match. Previously a disabled gate let the message fall through to the observer path, where `parts[1]` — the literal string `client` — was read as a region and the phone's pubkey registered as an observer. The blacklist check now also runs ahead of the sub-topic switch, so it covers every present and future client sub-topic. ## Testing Full ingestor suite green. Notable coverage: a `/rf` message with the gate off writes nothing anywhere and does not fall through; a sample missing `uptime_secs` is rejected rather than stored as an unusable row; two samples 40 ms apart remain two rows; and the retention test seeds a row with a non-zero millisecond component inside the cutoff second, which is the only row that distinguishes a correct cutoff from an RFC3339 one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 21:22:07 +00:00
efitenandClaude Opus 5	9e13e0b05f	feat(ingestor): full-packet RF observations from mobile clients (#1905) ... ## What A CoreDrive RX drive already carries far more RF information than reaches CoreScope, and it was being discarded twice: once in the mobile app (every packet it could not attribute to a directly-heard node was dropped before queueing) and once here (the ingestor decodes the *complete* packet, then keeps only `heard_key`/`snr`/`rssi`/`lat`/`lon`). This captures what was being thrown away, at **zero extra airtime** — nothing new is transmitted. - **`transmissions.code1` / `code2`** — the transport codes were decoded on every packet and used only to derive `scope_name`, then dropped. Storing them turns "which repeater forwards which scope" from a re-parse into a query. - **An async backfill** re-parses the `raw_hex` already on disk, so months of scope history become queryable with no new data collection. - **`client_rx_observations`** — a new diagnostic table holding every decodable packet a phone heard, with route type, transport codes, scope name, path-hash size, the full forwarder chain and the forwarder. ## Why it is safe for existing deployments Both halves are **opt-in and default off** (`clientRxObservations.enabled`, and `fullRfLog` on the app side), so an existing deployment sees no behaviour change and no volume change on upgrade. The coverage invariant is untouched: `client_receptions` keeps its rule — 0-hop advert pubkey or FLOOD `path[last]`, ≥2-byte hash — and an unattributable packet writes **zero** coverage rows. `deriveHeardKey`, `buildClientReception` and `InsertClientReception` are unmodified except for one guard described below. ## Performance justification (touches the ingest hot path) - **Backfill:** keyset-paginated by `id` in 5000-row batches, a single forward scan, `rows.Close()` before `Begin()` so it never deadlocks against `SetMaxOpenConns(1)`, and commits per batch so live ingest interleaves. Termination is driven by rows *scanned*, not rows decoded — an earlier count-based loop would have stopped at the first batch containing an undecodable row and then written its completion guard, permanently stranding the rest. - **Guard row is written if and only if the loop ran to genuine exhaustion.** Every error path leaves it unwritten so the next startup retries. - **Per-packet cost:** one extra INSERT on the client topic when enabled, gated behind an opt-in flag. No new work on the observer path. - **New indexes** cover the prune (`rx_at`), the flood-grouping (`pkt_hash, rx_at`), the per-repeater query (`forwarder, rx_at`) and the scope query (`scope_name, rx_at`). Retention has its own shorter window — this table is diagnostic, not archival. ## Two firmware-derived correctness points - **`pkt_hash` is `ComputeContentHash()`**, byte-identical to `transmissions.hash`, so dark-traffic queries are a plain equality join rather than a translation layer. - **TRACE packets are refused.** TRACE repurposes the header path bytes as per-hop SNR values, so deriving a `heard_key` from them invents a node that never existed. `packetpath.PathBytesAreHops` existed but was never wired into the client path; it became reachable only because the app half now publishes packets it previously dropped locally. ## Testing Full ingestor suite green. Notable coverage: a FLOOD-routed TRACE writes zero coverage rows and NULL `forwarder`; a `direction: "tx"` message writes no observation; a DIRECT route never sets `forwarder`; two forwarder copies of one flood remain two rows; the backfill's multi-batch path is exercised with an undecodable row in the first page; and a forced error asserts the migration guard stays unwritten. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 18:35:20 +02:00
efitenandClaude Opus 5	d821d9a390	feat(retention): add observerPurgeDays hard-delete for long-inactive observers (#1886) ... ## Problem `RemoveStaleObservers` only soft-deletes — it sets `inactive = 1` and the row stays forever. On a long-running deployment those rows just accumulate: on a two-year-old instance roughly 25% of the `observers` table was rows nobody can ever see again. There is currently no way to reclaim them. ## Fix A second retention stage. `PurgeStaleObservers` hard-deletes rows that are: - already `inactive = 1` (so the soft-delete stage owns the decision of *when* an observer goes stale), **and** - older than `retention.observerPurgeDays`, **and** - referenced by nothing. New config field `retention.observerPurgeDays`, default `0` = disabled. Existing deployments are unaffected until they opt in. Set it above both `observerDays` and `packetDays` — below those the reference guards keep every candidate row anyway. ## Why the reference guards are the point `observations.observer_idx` is a bare rowid with no foreign key. Deleting a still-referenced observer silently orphans history — `packets_v` stops resolving the observer and those packets get mis-attributed. Nothing errors; the data just quietly goes wrong. So the statement guards on all three referencing tables: ```sql AND NOT EXISTS (SELECT 1 FROM observations o WHERE o.observer_idx = observers.rowid) AND NOT EXISTS (SELECT 1 FROM observer_metrics m WHERE m.observer_id = observers.id) AND NOT EXISTS (SELECT 1 FROM dropped_packets d WHERE d.observer_id = observers.id) ``` This is correctness, not defensive padding — it was found the hard way, by orphaning 280 observation rows during a manual purge that skipped one of these checks. Each guard has its own test. ## Performance Each `NOT EXISTS` is an index seek per candidate row (`idx_observations_observer_idx`, `idx_dropped_observer`, the `observer_metrics` PK), and `observers` is O(100). It runs on the existing daily retention tick alongside `RemoveStaleObservers`, never on the ingest path. ## Tests Eight tests in `cmd/ingestor/observer_purge_test.go`, written before the implementation: - deletes an unreferenced stale row - keeps a row referenced by `observations` — and asserts zero orphans afterwards - keeps a row referenced by `observer_metrics` - keeps a row referenced by `dropped_packets` - keeps a row that is old enough but still `inactive = 0` - keeps a row inside the retention window - no-ops when disabled (`0` and `-1`) - config accessor table test ## Invariant Writes stay in `cmd/ingestor` per #1283. `cmd/server/readonly_invariant_test.go` now also forbids `PurgeStaleObservers` as a method on the server's `*DB`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:22:30 +02:00
efitenandClaude Opus 5	e5595ad92f	fix: unbreak master — decouple the pathTrust builder test from the default (#1932) ... **master is currently red.** This is the fix. ``` --- FAIL: TestNeighborEdgesBuilderPathTrustExcludesOneByte neighbor_builder_test.go:301: 1-byte hop must not produce an edge under the default threshold, got 1 ``` ## What happened Two PRs that were each green on their own: - **#1929** moved `DefaultMinHashBytesForMapping` from 2 to 1. - **#1930** carries `TestNeighborEdgesBuilderPathTrustExcludesOneByte`, written when the default was 2. Neither pipeline saw the other, because a `pull_request` run tests the merge commit as it stood when that run started. Both merged, and the combination fails. My mistake for merging them in the same batch without re-running one against the other. ## The fix The test passed `nil` for the trust config and leaned on the package default being 2: ```go // nil == package default (MinHashBytesForMapping = 2). store.buildAndPersistNeighborEdges(nil) ``` That coupling is the real defect. The test is about what happens **at threshold 2**, not about what the default happens to be. It now says so: ```go trust := &packetpath.TrustConfig{MinHashBytesForMapping: 2} store.buildAndPersistNeighborEdges(trust) ``` It keeps testing exactly what it was written to test, and stops breaking when the default moves. The sibling `TestNeighborEdgesBuilderPathTrustAllowsTwoByte` already passes its own fixture explicitly, so this brings the two into line. **No production code changed.** `cmd/ingestor` PathTrust and Neighbor tests pass. ## Worth recording This is the failure mode I have been flagging on other PRs all day, and I walked into it myself: green CI on a PR is a statement about the base it was tested against, not about master. Two PRs can each be green and still be red together. Nothing about the review process would have caught it — only re-running one against the other, or a merge queue, would. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 09:31:01 +00:00
4 people	e8f32df4dc	feat(#1784): gate ingestor neighbor-edge creation on the path-trust threshold (rebase of #1863) (#1930) ... Continues #1863. Three of the four commits are @Saarlandpower's and @SaarMesh-Bot's, authorship unchanged. The fourth is mine and is explained below. ## Why a rebase was needed #1863 was stacked on #1824, and #1841 merged instead. Both carried the same pathTrust base from different commits, which is why the two conflicted while each reported MERGEABLE against master. Cherry-picking #1863's own three commits onto master applied cleanly with no conflicts, which confirms its actual work was always independent of that duplicated base. ## The fourth commit, and a correction to something I got wrong The three commits do not build on master: ``` cmd/ingestor/main.go:455:23: cfg.GetPathTrust undefined (type *Config has no field or method GetPathTrust) ``` **#1824 added the pathTrust config and helper to both `cmd/server/config.go` and `cmd/ingestor/config.go`. #1841 carried only the server half** — one of its own commits is titled "remove ingestor side". I then closed #1824 as superseded by #1841, which is true for the server side and wrong for the ingestor side. Master has no pathTrust code in `cmd/ingestor/config.go` at all. The fourth commit restores that half, unchanged from `beae2c1c`: the `packetpath` import, the `PathTrust` field, the `PathTrustConfig` alias, `GetPathTrust`, and `cmd/ingestor/config_test.go` verbatim (28 lines covering the default, an explicit value, and a nil `*Config` receiver). That code is @Bjorkan's and @SaarMesh-Bot's from #1824, not mine; I only put it back. ## Verification - All three original commits cherry-picked onto `b3a306b8` with **no conflicts** - `cmd/ingestor` builds, and its `PathTrust|Neighbor|Config` tests pass - `cmd/server` `Neighbor|PathTrust|AnonReq|Edge` tests pass ## Interaction with #1929 #1929 moves `DefaultMinHashBytesForMapping` from 2 to 1. With that in, this PR's ingestor gate is a no-op by default and only takes effect when an operator sets `minHashBytesForMapping` to 2 or 3, which is the opt-in shape #1784 asks for. The two are complementary; merge order between them does not matter. @Saarlandpower @SaarMesh-Bot — your work, your credit. Say the word and I will close this and hand the rebase back, or push it to the #1863 branch if you would rather that stayed the vehicle. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE --------- Co-authored-by: Saarlandpower <Mail@mathiaskasper.de> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com>	2026-09-02 11:07:18 +02:00
Jonathan Herlin	647841c990	fix: stop watchdog force-reconnect from racing paho's own retry loop (#1897) ... Relates to #1335, which was already closed by PR #1336 shipping the naive `client.Disconnect(250); client.Connect()` force-reconnect. That fix has its own bug: liveness.IsConnectedFn (paho's IsConnected()) reports true for the entire time paho is actively retrying, not just when genuinely connected, so the watchdog's stall check cannot tell a half-open TCP socket (the original #1335 case) from a broker that paho is already correctly reconnecting to. Unconditionally calling Disconnect(250) then Connect() on that second, transitional case races paho's status machine and permanently kills its retry loop, requiring another watchdog trigger to recover, sometimes compounding into 100+ minute outages. This is a different failure mode from #1749/PR #1853: that bug is a blocking log.Print() write freezing the entire watchdog loop before ForceReconnectFn is ever called. This bug only manifests once ForceReconnectFn does fire, so the two fixes are independent and touch disjoint files. buildForceReconnectFn now gates Disconnect() on IsConnectionOpen() (true only when status is strictly connected) so it only tears down a genuinely open connection, and logs Connect()'s error token instead of discarding it.	2026-09-02 10:28:00 +02:00
3 people	0d6f59ab2d	fix(#1864): decode ANON_REQ source pubkey instead of treating it like REQUEST (#1866) ... Fixes #1864. ## Problem `PAYLOAD_TYPE_ANON_REQ` was effectively treated like `REQUEST`. The two differ on the wire: ``` REQUEST : <dest hash 1B> <source hash 1B> <hmac 2B> <encrypted> ANON_REQ: <dest hash 1B> <source pubkey 32B, full> <hmac 2B> <encrypted> ``` The decoders read the right bytes but surfaced the sender key as `ephemeralPubKey`, which meant: - `store.go`'s node indexer keys on `pubKey`/`destPubKey`/`srcPubKey`, so ANON_REQ packets were **not** indexed — they didn't show up on a node's packet view; and - the packets list "details" rendered a bare `anon → <destHash>`, throwing away the sender identity the packet actually carries. - the detail side-view byte breakdown fell through the REQ catch-all, mislabelling a nonexistent 1-byte "Src Hash" and placing MAC/Encrypted-Data at the wrong offsets (`+2`/`+4` instead of `+33`/`+35`). ## Fix **Backend** (`cmd/ingestor` + `cmd/server` decoders) - Surface the ANON_REQ sender key as `srcPubKey` (json) so it's indexed and resolvable. The frontend keeps a legacy `ephemeralPubKey` reader so packets decoded before this rename still resolve — no DB migration needed. - `TestDecodeAnonReqValid` now asserts the full 32-byte `srcPubKey`. **Frontend** - `hop-resolver.js`: new O(1) `nameForKey(pubkey)` using the existing `pubkeyIdx` (all nodes). - `getDetailPreview`: resolve the source pubkey to a node **name** when known, else show the first 8 hex chars — no more bare `anon`. - Detail side-view: explicit ANON_REQ breakdown — `Dest Hash (1B)` | `Src Public Key (32B)` (node-linked) | `MAC @+33` | `Encrypted Data @+35`. - Detail header `srcLabel` falls back to the resolved ANON_REQ sender. All rendered names are `escapeHtml`-wrapped. ## Testing - `go test ./...` green for both `cmd/ingestor` and `cmd/server` (incl. strengthened `TestDecodeAnonReqValid`). - `node --check` on `packets.js`; brace/paren balance + markers verified on `hop-resolver.js`. - No HTML sink lines added → XSS preflight gate unaffected; every interpolated name is escaped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-09-02 10:27:56 +02:00
3 people	52d08214bb	fix(#1749): decouple watchdog emit from blocking I/O (root cause) (#1853) ... Closes the gap left by #1810: that PR added defer/recover around the watchdog per-source work so a **panic** inside emit cannot kill the loop, but the actual production incident is caused by emit **blocking**, not panicking. ## Root cause In production `emit` is `log.Print`. `log.Print`'s underlying `write()` can block indefinitely if the sink is backpressured (Docker JSON-file log driver falling behind under load, a full stderr pipe, journald hiccups, etc.). A blocked syscall is not a panic -- `recover()` does nothing for it. Because emit was called **synchronously** inside the per-source work, a single stuck `write()` froze the entire tick loop forever -- no further source was ever checked and no further tick was ever processed again. This exactly reproduces the original #1749 incident even after #1810 landed: 3 independent MQTT sources going silent within ~60s of each other (one shared dependency -- the watchdog goroutine itself -- died, not 3 independent paho clients), zero WATCHDOG log lines for the rest of the 75-minute window, every other goroutine in the process continuing to run fine (a hang, not a crash), and only a full container restart recovering it. ## Fix `newAsyncEmit` decouples "decide to log" from "perform the write": the watchdog loop now only ever does a non-blocking channel send. A single background goroutine drains the channel and performs the (potentially blocking) write. If that goroutine itself gets stuck, the bounded queue (256) fills and further sends are dropped -- counted via the new `WatchdogLogDropCount`, surfaced through `/api/mqtt/status` and the ingestor stats snapshot alongside `WatchdogLastTickUnix` / `WatchdogPanicCount`. Worst case under a persistent backpressure event is now lost log lines (visible and counted), not a silently dead watchdog (invisible and undetectable -- the actual #1749 failure). ## Tests - `TestNewAsyncEmit_NeverBlocksWhenWriterStuck_1749` -- floods emit() past queue capacity while the writer is permanently blocked; every call must return immediately and drops must be counted. - `TestMQTTStallWatchdog_LoopSurvivesStuckWriter_1749` -- end-to-end, wires `runLivenessWatchdogLoop` exactly as production does (via `newAsyncEmit` around a permanently-blocking `realEmit`) with 3 registered sources, reproducing the incident shape and asserting the loop keeps ticking regardless. - `TestRunLivenessWatchdog_ProductionWiringUsesAsyncEmit_1749` -- smoke-tests the real entrypoint starts, ticks, and stops cleanly. - `WatchdogLogDropCount` round-trip tests in both the ingestor stats snapshot and the server's `/api/mqtt/status` handler, mirroring the existing `WatchdogPanicCount` coverage from #1810. All pre-existing watchdog/liveness tests (#1749, #1810 r1, force-reconnect) continue to pass unmodified; full ingestor suite green (verified 5x consecutive runs for flake-freedom). Note: the server package has pre-existing test-suite-wide flakiness in unrelated `TestHandleNodePaths_*` tests (confirmed reproducible on unmodified master too, non-deterministic which subset fails per run) -- unrelated to this change and out of scope here. --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-09-02 10:27:48 +02:00
3 people	6528c7ba3e	fix(#1854): move relay last_seen touch to the ingestor — server writes have been no-ops since mode=ro (#1855) ... Fixes #1854. Refs #1598, #1611, #1845. ## The bug `cmd/server/db.go:54` opens SQLite `mode=ro` (#1283/#1289). `touchRelayLastSeen` → `TouchNodeLastSeen` issues `UPDATE nodes SET last_seen` on that handle. It has failed on every call since, with the error discarded at the call site: ```go if err := s.db.TouchNodeLastSeen(pk, ts); err == nil { s.lastSeenTouched[pk] = now } ``` `nodes.last_seen` has therefore tracked ADVERT arrivals only. Verified on live.saarmesh.de (1388 nodes): 1362 have `last_seen` within one minute of their own most recent ADVERT. Reproduced directly with the server's DSN in #1854. Secondary effect: `lastSeenTouched` is populated only in the success branch, so the debounce never engaged — the server retried the failing UPDATE for every resolved pubkey in every decode window. ## The fix The writer moves to `cmd/ingestor`, which owns `nodes` per #1283/#1287 and since #1547 already resolves hop prefixes to full pubkeys for `observations.resolved_path`. The touch hooks into that existing resolution point, so there is no new IPC surface and no second resolver. Only unambiguously resolved hops qualify — a 1-byte prefix collision cannot keep a silent node alive. I considered the `internal/mbcapqueue` snapshot handoff used for #903/#1324 and did not need it: that pattern exists because the capability computation lives in the server's analytics cycle. Path resolution already happens in the ingestor, so a file handoff would add a hop for nothing. `Store.TouchRelayNodes`: - monotonic guard in SQL (`last_seen IS NULL OR last_seen < ?`) — out-of-order ingest never rewinds - 5-minute debounce keyed on `rxTime`, matching the interval the server intended - UPDATE only — unknown pubkeys never create rows - unparsable `rxTime` is a no-op rather than writing garbage into the node directory - `Stats.RelayTouches` for `/api/perf` visibility - debounce records the *attempt*, not the row match, so an unknown pubkey is not retried per observation ## Server-side removal `touchRelayLastSeen`, `DB.TouchNodeLastSeen`, the `lastSeenTouched` map and the now-unused `allResolvedPKs` decode-window map are deleted. `readonly_invariant_test.go` gains `UPDATE\s+nodes\s+SET\s+last_seen`. `cmd/server/touch_last_seen_test.go` and two tests in `resolved_index_test.go` go with it. Worth stating why they were green for months: they build their `PacketStore` on `setupTestDB`, which opens read-write. The production constraint is the one thing they did not reproduce, which is why the added invariant regex — not a replacement unit test — is the right guard here. ## Tests Five tests in `cmd/ingestor/relay_touch_test.go`, committed red first (573bbde3) with a stubbed `TouchRelayNodes` so the suite compiles and reds on assertions: ``` --- FAIL: TestTouchRelayNodes_AdvancesLastSeen last_seen = "2026-07-01T00:00:00Z", want "2026-07-10T12:00:00Z" RelayTouches = 0, want 1 --- FAIL: TestTouchRelayNodes_Debounces RelayTouches = 0, want 1 (second touch should be debounced) ``` Coverage: `AdvancesLastSeen` (core regression), `NeverGoesBackwards` (monotonic), `Debounces` (write amplification on the hot path), `IgnoresEmptyAndUnknown` (unresolved hops must not create rows), `MalformedTimestamp`. `cmd/ingestor`: full suite green, 100.7s. `cmd/server`: green for the invariant and the affected packages, but the suite is order-dependent on master today. Unmodified `upstream/master` produced 8 failures on this machine (`TestHandleNodePaths_*`, `TestHandleAnalytics*`, `TestComputeAnalyticsDistanceLockHoldDuration`); this branch produced 5, and the set shifts between runs. All pass in isolation. Untouched by this change — flagging rather than papering over, and happy to open a separate issue if that is not already known. ## Impact on the open threads This is the backend half of #1598. The frontend work there keys on relay recency; that signal was never being written, so the two changes are complementary rather than alternatives. It also removes the eviction problem I raised in #1845 without touching `MoveStaleNodes`: once `last_seen` reflects relay activity, the existing `last_seen < cutoff` predicate stops evicting nodes that are carrying traffic. Not addressed here: the duplicate-row behaviour between `nodes` and `inactive_nodes` (609 keys in both on my deployment), which is an independent defect and wants its own change. ## Verification offer I run a 1100-repeater MeshCore deployment and can run this against production traffic and report `RelayTouches` plus the resulting `last_seen` distribution before/after, if that is useful for review. --------- Co-authored-by: SaarMesh-Bot <300107934+SaarMesh-Bot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-09-02 10:27:40 +02:00
efitenandClaude Opus 5	9bd5f5a3a2	fix(ingestor): a retained status message is not observer liveness (#1885) ... ## Problem The broker replays every retained `status` message on subscribe, so each ingestor restart pushes all of them through the status path in `handleMessage`. That path stamps `last_seen` with `time.Now()` (deliberately, per #1465). Observed on a live deployment on 2026-08-11: **23 observers all carried `last_seen = 2026-08-06T08:20:09Z`** — 15 seconds after container start — and 18 of them had sent no actual packet in over a month. Their retained publish dates lined up almost 1:1 with `last_packet_at`, i.e. the replay was their only sign of "life": | observer | last real packet | retained status published | |---|---|---| | ON8AR - Observer | never | 2026-03-20 | | BE-BGS-RRY120-RES | never | 2026-04-02 | | A3BEF374 | 2026-04-30 | 2026-04-30 | | BE-BGS-RRY120-RUDY | 2026-05-18 | 2026-05-18 | | BE-JBE-ETG-O1 | 2026-06-10 | 2026-06-10 | That makes dead observers immortal, three ways per restart: 1. `last_seen` jumps forward, so `RemoveStaleObservers` can never age them out as long as a restart happens inside `observerDays`. 2. The unconditional `inactive = 0` reactivation at the end of `UpsertObserverAt` undoes any soft-delete that did land. 3. A metrics sample is filed at ingest time, dating a months-old reading as a present-tense measurement. `UpsertObserverAt`'s docstring already claimed retained replays were a no-op for `last_seen` thanks to the `MAX` guard. That held only while the caller passed the envelope timestamp; #1465 switched it to ingest time, which defeats the guard. ## Fix The retained path now updates metadata only, via a new `UpsertObserverRetained`: - no `last_seen` advance - no `inactive = 0` reactivation - no `packet_count` bump - no metrics sample - **no INSERT** — a retained-only observer the analyzer has never heard from live describes a past that may be months old and does not belong in the list. A live message from the same observer creates the row through the normal path moments later. Live status handling is unchanged. ## Tests Seven tests in `cmd/ingestor/retained_status_test.go`, written before the fix: - 4 that failed on the bug: `last_seen` advance, reactivation of a soft-deleted row, creation of a never-seen observer, metrics-sample insert - 2 regression guards pinning live (non-retained) behaviour: `last_seen` still advances, unknown observer still created - 1 asserting retained metadata is still applied — the snapshot is the observer's last known state, only the liveness signal is suppressed `mockMessage` gained a `retained` field so `Retained()` is controllable. Meta flattening is extracted to `observerMetaColumns` so both write paths bind identical args. Full `cmd/ingestor` suite passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 09:11:05 +02:00
4 people	242c7c609b	fix(mqtt): escalate persistent paho disconnect + recover from emit panic + expose watchdog tick (#1749) (#1810) ... # Partial fix for #1749 — MQTT watchdog escalation + panic recovery + tick exposure Red commit: 9912bbb3e9 (CI run: https://github.com/Kpa-clawbot/CoreScope/actions/runs?branch=fix%2Fissue-1749) ## Problem Production CoreScope v3.9.1 (and a more recent prod recurrence on 2026-06-30 with the wcmesh source on `ssl://mqtt2.wcmesh.com:8883`) showed two distinct watchdog failure modes: 1. **Per-source paho machinery dies silently.** `IsConnectedFn` returns false; paho's `SetAutoReconnect(true)` never retries. The watchdog's `processLivenessTransition` deliberately stays silent on `LivenessDisconnected`, trusting paho to recover — so there is no escalation path when that trust is misplaced. 2. **Watchdog goroutine death.** Three sources went silent within ~60s of each other; no `WATCHDOG` log lines for 75 min. The most plausible single point of failure is a panic inside `emit` (e.g. a blocked log pipe) killing the loop with no defer/recover. ## Changes **`cmd/ingestor/mqtt_watchdog.go`** - Added `disconnectedReconnectMultiplier = 5` constant. - Added `DisconnectedSinceUnix int64` (atomic) on `SourceLivenessState`. Stamped on the first tick the source is observed disconnected; cleared on any non-disconnected tick. - `processLivenessTransition` now escalates: when `(now - DisconnectedSinceUnix) > multiplier × threshold`, emits a `WATCHDOG ESCALATION` WARN and calls `maybeForceReconnect` (subject to existing `forceReconnectThrottle`). Distinct from the existing `LivenessStalled` path so operators can grep escalation events independently. - Added package-level `watchdogLastTickUnix atomic.Int64` + `WatchdogLastTickUnix()` getter. The loop stamps it BEFORE per-source processing — a wedged source-handler does not freeze the clock for an external observer. - `runLivenessWatchdogLoop` wraps each per-source `processLivenessTransition` call in `func() { defer recover; ... }()` so a panic in `emit` (or in any per-source code path) is logged and skipped, not fatal. The loop continues to the next source and the next tick. **`cmd/ingestor/stats_file.go`** - Added `WatchdogLastTickUnix int64` field on `IngestorStatsSnapshot` (additive, `omitempty`); populated from `WatchdogLastTickUnix()` each stats tick. **`cmd/server/mqtt_status.go`** - `MqttStatusResponse` gains `WatchdogLastTickUnix int64` (additive, `omitempty`) sourced from the ingestor stats file; surfaced via `GET /api/mqtt/status`. **`config.example.json`** - No new config field added — the multiplier is a code constant (5×) per the issue's "N×threshold (e.g. 5×)" recommendation. The active per-source `threshold` is the 5-minute scan threshold hard-coded at the sole `runLivenessWatchdog` callsite (`cmd/ingestor/main.go:460`: `runLivenessWatchdog(60*time.Second, 5*time.Minute)`), so escalation fires at ~25 minutes (5 × 5min) of continuous disconnect — plus a deterministic per-source jitter of 0..30s (#1810 round-1, see Taleb #4) to avoid synchronized escalation across N sources sharing an upstream broker outage. ## Acceptance (#1749) - [x] Persistent `LivenessDisconnected` > N×threshold → force-reconnect + WARN - [x] Watchdog goroutine liveness clock exposed (`WatchdogLastTickUnix` in `/api/mqtt/status`) - [x] Test: `IsConnectedFn` false for >5×threshold → assert `ForceReconnectFn` invoked at least once - [x] Test: panic in `emit` → assert loop recovers and continues ticking ## Test plan - 4 new tests in `cmd/ingestor/mqtt_watchdog_1749_test.go` (all RED on master, GREEN on this PR). - Existing watchdog tests (`mqtt_watchdog_force_reconnect_test.go`, `mqtt_reconnect_test.go`, r1/r2/m1 suites) continue to pass — the escalation path is additive. ## Preflight - TDD: red commit pushed and asserted to fail BEFORE green commit landed. - PII grep: clean on diff and PR body. - Worktree: `_wt-fix-1749` on branch `fix/issue-1749`. --------- Co-authored-by: corescope-bot <bot@corescope.dev> Co-authored-by: openclaw-bot <bot@openclaw.local> Co-authored-by: bot <bot@local>	2026-06-30 15:16:31 -07:00
Kpa-clawbotandclawbot	b3189c613a	fix(#1802): decode CONTROL DISCOVER_REQ/RESP subtype + body fields (#1806) ... ## Summary Extend CONTROL packet decoding to surface DISCOVER_REQ / DISCOVER_RESP subtype plus body fields in the packet detail view. Previously only the byte0 zero-hop flag was decoded; the body was rendered as opaque hex. ## What changed **Backend** — `cmd/ingestor/decoder.go` `decodeControl()` - New `Payload` fields (all omitempty): `CtrlSubtype`, `CtrlFilter`, `CtrlTag`, `CtrlSince`, `CtrlNodeType`, `CtrlSNR`, `CtrlPubKey`. - Subtype derived from `byte0 & 0xF0`: `0x80` → `DISCOVER_REQ`, `0x90` → `DISCOVER_RESP`, otherwise `UNKNOWN`. - REQ body parsed when `len(buf) >= 6`: `filter:u8 | tag:u32 LE`, plus optional `since:u32 LE` when 4 more bytes remain. - RESP body parsed when `len(buf) >= 6`: `node_type` (low nibble of byte0), `snr:i8`, `tag:u32 LE`, and `pubkey` hex — 32 bytes when full, 8 bytes when prefix-only. - Every field gated on length; short/truncated bodies emit subtype only and never panic. - `CtrlZeroHop` retained for backwards compatibility (rename flagged for follow-up per triage). **Frontend** — `public/packets.js` `getDetailPreview()` - New `decoded.type === 'CONTROL'` branch renders subtype + present body fields (filter / tag / since / node_type / snr / pubkey). Each field shown only when populated, so truncated CONTROL still gets a subtype label. ## Wire format reference - `firmware/src/Mesh.cpp:69` — `CTL_TYPE_NODE_DISCOVER_REQ=0x80`, `CTL_TYPE_NODE_DISCOVER_RESP=0x90`. - `firmware/examples/simple_repeater/MyMesh.cpp:773-820` — body parse / build. ## Tests (red → green, per AGENTS.md STRICT TDD) - `cmd/ingestor/issue1802_test.go` — 6 cases: REQ full body (with since), REQ no-since, RESP 32B pubkey, RESP 8B prefix pubkey, RESP truncated pubkey (no panic, no pubkey emitted), short body (subtype only), unknown subtype. Red commit `43713d3a` → green commit `d4b28180`. Pre-existing CONTROL tests (`TestDecodeControlZeroHop`, `TestDecodeControlMultiHop`) still pass. - `test-packets.js` — 3 cases on `getDetailPreview`: DISCOVER_REQ (filter+tag rendered), DISCOVER_RESP (snr+pubkey rendered), UNKNOWN subtype label. Red commit `be23e349` → green commit `845d6c48`. ## Preflight overrides - `check-branch-clean` (cross-stack): justified — issue #1802 explicitly spans backend decoder (`cmd/ingestor/decoder.go`) and frontend renderer (`public/packets.js`) per triage comment. Tests in both layers. Single-purpose PR. ## Scope discipline Files touched: `cmd/ingestor/decoder.go`, `cmd/ingestor/issue1802_test.go`, `public/packets.js`, `test-packets.js`. No other files. No firmware changes. No `cmd/server/decoder.go` changes. No `CtrlZeroHop` rename (deferred per triage). Fixes #1802 --------- Co-authored-by: clawbot <bot@meshcore.local>	2026-06-28 06:32:07 -07:00
Kpa-clawbotandclawbot	e465e1c6c6	perf(#1740): replace idx_tx_last_seen with partial index WHERE last_seen=0 (#1756) ... Fixes #1740. ## What Replace the full `idx_tx_last_seen` with a partial index `WHERE last_seen=0`. Two ordered, sequential migrations in `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn`: - **(a)** `CREATE INDEX IF NOT EXISTS idx_tx_last_seen_zero ON transmissions(id) WHERE last_seen=0` - **(b)** `DROP INDEX IF EXISTS idx_tx_last_seen` — gated after (a) succeeds (sequential `Exec`; (b) never runs if (a) errors) ## Why The only consumer is `chunkedTxLastSeenBackfill`'s `WHERE last_seen=0` scan + `MAX(id)` lookup. The full index covers ALL rows including the long tail where `last_seen != 0` after backfill converges (71K+ rows in prod per carmack's #1740 note). The partial index degenerates to ~the count of un-backfilled rows (0 in steady state, bounded by ingest rate during ops) and stops competing for page cache. ## Migration cost Both migrations are sync and annotated `PREFLIGHT: async=false`: - (a) `CREATE INDEX` on partial subset `WHERE last_seen=0` is bounded by un-backfilled rows — a small superset of the inflight ingest window, not a full table scan. - (b) `DROP INDEX` is a metadata-only schema rewrite in SQLite — no row scan at any size. `dbschema/` has no access to `Store.RunAsyncMigration` (that helper lives in `cmd/ingestor/` and is the wrong layer for the schema source-of-truth per #1321), so sync is the only path here regardless. ## TDD - **RED** `a529f0f4`: `EXPLAIN QUERY PLAN` test asserting the backfill `MAX(id) WHERE last_seen=0` query uses `idx_tx_last_seen_zero` + a second test asserting the legacy `idx_tx_last_seen` is dropped post-Apply. Both failed on assertion (planner picked the full index; partial index didn't exist). - **GREEN** `208fde8a`: add migrations (a) and (b). Both tests pass. ## Files touched - `internal/dbschema/dbschema.go` — swap the index - `internal/dbschema/dbschema_test.go` — `EXPLAIN QUERY PLAN` pin + DROP assertion - `cmd/ingestor/db.go` — comment refresh only (idx_tx_last_seen → idx_tx_last_seen_zero) ## Acceptance - ✅ New partial index created - ✅ Old full index dropped via gated migration (sequential, order preserved) - ✅ Query plan test asserts partial-index usage - ✅ Existing migration tests still green (`internal/dbschema`, `cmd/ingestor` TestIssue1690 + applySchema) --------- Co-authored-by: clawbot <bot@openclaw>	2026-06-20 15:45:28 -07:00
3 people	22fe929da2	feat: opt-in mobile client-RX coverage (crowdsourced RF reach) + /api/nodes/resolve (#1728) ... Implements #1727. ## What this adds **Mobile client-RX coverage** — an opt-in, crowdsourced RF-coverage feature. A roaming MeshCore **companion** radio (driven by the open-source [corescope-rx](https://github.com/efiten/corescope-rx) PWA, GPLv3) reports which nodes it heard directly, tagged with the phone's GPS and the packet's SNR/RSSI. CoreScope ingests these into a new `client_receptions` table and renders per-node **hex coverage** on the Reach page, plus a standalone **Coverage dashboard** (`#/rx-coverage`) with a top-mobile-observers leaderboard. Also includes **`GET /api/nodes/resolve?prefix=<hex>`** — a read-only node-name lookup by pubkey prefix (`{name, pubkey, ambiguous}`), used by the companion app for friendly names. ## Opt-in — default OFF (zero impact on existing deployments) The whole feature is gated behind one config flag, **disabled by default**: ```jsonc "clientRxCoverage": { "enabled": false } ``` When disabled (the default): the ingestor writes **no** `client_receptions`; the three coverage endpoints return a clean **404**; the UI hides the Coverage nav link, the `#/rx-coverage` route, and the Reach-page toggle. `/api/nodes/resolve` is always available (not coverage-specific). ## How it works ``` companion ──BLE 0x88 (snr+rssi+raw)──▶ corescope-rx PWA ──▶ MQTT meshcore/client/{pubkey}/packets │ ingestor (gated) ──▶ client_receptions (GPS + SNR + heard-key) │ server: pure-Go hex grid ──▶ GeoJSON ──▶ Reach hex overlay + Coverage dashboard ``` - **Direct-only capture:** records only what the companion heard itself and directly — a 0-hop advert's pubkey, or `path[last]` (last forwarder) for FLOOD routes; ≥2-byte path-hash required. Upstream hops discarded. - **No new deps:** hexbins are a pure-Go pointy-top grid over Web Mercator (`cmd/server/hexgrid.go`) computed at query time (`CGO_ENABLED=0` / `modernc.org/sqlite` friendly); frontend uses the existing Leaflet. - **Trust:** companion pubkey = identity; an EMQX ACL binds each client to publish only to its own `meshcore/client/{pubkey}/packets` topic. Payload contract in `docs/client-rx-coverage.md`. ## How to enable / try it 1. In `config.json`, set `"clientRxCoverage": { "enabled": true }` and restart server + ingestor. 2. Point an EMQX (or any broker) listener so a client can publish to `meshcore/client/<pubkey>/packets`; the ingestor already subscribes under `meshcore/#`. 3. Run the [corescope-rx](https://github.com/efiten/corescope-rx) PWA on an Android phone paired (BLE) to a MeshCore companion — it captures heard nodes + GPS and publishes. 4. View results: per-node Reach page → toggle **coverage**, or the **Coverage** dashboard at `#/rx-coverage`. ## What's where - **Ingestor:** `cmd/ingestor/client_reception.go` (ingest), `db.go` (`client_receptions` + `client_observers` schema), `main.go` (gated dispatch), `config.go` (flag). - **Server:** `cmd/server/rx_coverage.go` + `rx_dashboard.go` (endpoints, self-guard 404 when off), `hexgrid.go` (pure-Go grid), `node_resolve.go` (resolve), `routes.go` / `types.go` / `config.go` (wiring + flag + `/api/config/client` field). - **Frontend:** `public/rx-coverage.js` (dashboard), `node-reach-coverage.js` + `.css` (overlay), `node-reach.js` (Reach toggle, flag-gated), `roles.js` (reads the flag, hides nav when off). - **Docs:** `docs/client-rx-coverage.md`. ## Testing - Go: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test ./...` — green, including new gate tests (`coverage_gate_test.go` in both: off → no rows / 404, on → works) and the rx-coverage / resolve / hexgrid suites. - JS: `node test-coverage-gate.js`, `node test-node-reach-coverage.js` (wired into CI). The Playwright `test-node-reach-coverage-e2e.js` is wired into the e2e job and **skips when `clientRxCoverage` is disabled**, so it's safe under the default-off config. ## Notes for reviewers - The four new routes are registered in `cmd/server/openapi_known_gaps.json` (the existing OpenAPI-completeness ratchet), matching how other not-yet-spec'd routes are tracked. Happy to write full OpenAPI spec entries instead if you prefer. - Commits are split per layer (ingestor / server endpoints / resolve / frontend / CI) for review. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Erwin Fiten <e.fiten@opteco.be>	2026-06-19 11:37:16 -07:00
3 people	048143f54f	fix(#1690): cold-load uses last_seen (effective recency) instead of first_seen (#1691) ... ## #1690 — cold-load uses wrong time axis (RED → GREEN) The on-disk DB has thousands of long-lived hashes with recent traffic. Prod's cold-load filter (`transmissions.first_seen >= cutoff`) is bound to a column that is set once at insert time and never updated — so re-observation of an old hash does not move it into the hot window. Result: prod cold-loaded ~0.3% of the on-disk rows and flipped `backgroundLoadComplete=true` without ever walking the retention window (the `retentionHours - hotStartupHours <= 0` short-circuit at line 1353 of `cmd/server/store.go`). ### Three sub-fixes **A) Denormalize `transmissions.last_seen`** so cold-load can window on effective recency. - `internal/dbschema/dbschema.go::ensureTransmissionsLastSeenColumn` adds the column + `idx_tx_last_seen` (single-column INTEGER ALTER + index; both PREFLIGHT-annotated as cheap metadata-only ops). - `cmd/ingestor/db.go::OpenStoreWithInterval` schedules `tx_last_seen_backfill_v1` via `Store.RunAsyncMigration` — `UPDATE transmissions SET last_seen = MAX(observations.timestamp) WHERE last_seen = 0` — non-blocking on boot (1.9M+ obs row scan in prod). - Writer-side: `InsertTransmission` seeds `last_seen` on initial insert, and every observation insert bumps `last_seen = ?` via prepared statement `stmtBumpTxLastSeen` (conditional `last_seen < ?` so out-of-order ingest never goes backwards). - Reader-side: `cmd/server/store.go::Load`, `loadChunk`, and `cmd/server/chunked_load.go::LoadChunked` switch the WHERE/ORDER-BY clauses to `t.last_seen` when the column is present (PRAGMA-detected via `DB.hasLastSeen`). Test/legacy DBs without the column fall back to `first_seen` so existing fixtures stay green. **B) Honest `backgroundLoadComplete` gating.** - Drop the `retentionHours - hotStartupHours <= 0` short-circuit. Prod runs with both at 12h, which flipped Done=true immediately. - After the chunk loop, query `SELECT COUNT(*) FROM transmissions WHERE last_seen >= retentionFloor` and compute `loadCoverageRatio = inMem / inDB`. Done=true only when `ratio >= 0.90` AND no chunk errors. `backgroundLoadFailed=true` + `backgroundLoadError` populated otherwise (e.g. `"loaded 20.0% of 5000 rows (1000 in memory)"`). - `bgErrMu`-guarded `loadCoverageRatio` + `backgroundLoadErr` so the perf endpoint can read them without blocking the writer. **C) Perf exposure.** `PerfPacketStoreStats` gains `RetentionHours`, `OldestLoaded`, `LoadCoverageRatio`, `BackgroundLoadError` — surfaces what fraction of the on-disk DB the in-memory store currently reflects, so operators can see the 0.3% case in `/api/perf` without reading the logs. ### TDD trail - **RED**: `05f0c6dd2bea6dc37324c548a49564d739aca920` — failing tests + 21-line store.go scaffolding. CI on this commit failed on assertions (intended). - **GREEN**: this PR's HEAD commit (8 files, +271/-24). Targeted suite: `Test1690_ColdLoad_TimeAxis`, `Test1690_BackgroundLoadHonesty`, `Test1690_PerfStats_NewFields`, `TestHotStartup_*`, `TestIssue1690_LastSeenUpdatedOnObservation` — all pass. Anti-tautology: locally reverted the `if !s.backgroundLoadFailed.Load()` guard around `backgroundLoadDone.Store(true)` — `Test1690_BackgroundLoadHonesty` fails on the assertion `"backgroundLoadDone=true with only 1000/5000 packets loaded; must be false until coverage ≥ 90%"`. Restored. ### Async-migration preflight - `ensureTransmissionsLastSeenColumn` — ALTER + CREATE INDEX both `// PREFLIGHT: async=true reason="..."` annotated. - `tx_last_seen_backfill_v1` — wrapped in `Store.RunAsyncMigration`. - `stmtBumpTxLastSeen` prepared statement — annotated; it is a row-level UPDATE BY PRIMARY KEY, not a migration. ### Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=5K - check-async-migration: justified for `cmd/server/issue1690_cold_load_test.go` CREATE TABLE/INDEX statements — these build an in-memory test fixture DB (≤5000 rows, runs in <1s in CI), not a prod migration. Fixes #1690. --------- Co-authored-by: meshcore-bot <bot@meshcore.local> Co-authored-by: bot <bot@example.com>	2026-06-12 12:47:53 -07:00
Kpa-clawbotandopenclaw-bot	efd66ea3f5	feat(mqtt): per-source status endpoint + Observers panel (#1682) ... ## Summary Adds MQTT source status visibility per #1043 acceptance criteria: - **Ingestor:** per-source counter registry (`cmd/ingestor/source_status.go`) tracking `connected`, `lastConnectUnix`, `lastDisconnectUnix`, `lastPacketUnix`, `connectCount`, `disconnectCount`, `packetsTotal`, `packetsLast5m` (sliding 5-min window via per-second buckets keyed by unix second — no stale-leak), `lastError`. Wired at the existing OnConnect / ConnectionLost / DefaultPublish callsites alongside the liveness watchdog. Idempotent registration so counters survive reconnects. Snapshot emitted in the existing stats file under `source_statuses` (additive, `omitempty`). - **Backend:** new `GET /api/mqtt/status` handler reads the ingestor stats file and returns the per-source list. **Broker passwords are masked** via a regex over the `scheme://user:pass@host` form (covers mqtt/mqtts/tcp/ssl/ws/wss). Mask is also applied to `lastError` as defense-in-depth (broker libs occasionally quote the failing URL). OpenAPI completeness gate satisfied with a `routeDescriptions` entry. - **Frontend:** small self-contained panel (`public/mqtt-status-panel.js`) mounted above the Observers table. Auto-refreshes every 10s, color-codes each row (green = connected + recent packet, yellow = connected idle, red = disconnected), and tears down its timer on SPA route change. ## TDD - Red commit `f19a93b5` — stub `/api/mqtt/status` handler + assertion test that the broker password is `****`-redacted. Test fails on the assertion (handler passes the URL through verbatim). Compile-clean — assertion-fail, not build-fail. - Green commit `77042e41` — `maskBrokerURL` helper + table-driven unit tests across all schemes + handler rewires to mask both `Broker` and `LastError`. - Subsequent commits land the ingestor wiring and the frontend panel. ## Tests ``` $ cd cmd/server && go test -run 'TestMqttStatus|TestMaskBrokerURL' -v ./... PASS: TestMqttStatus_MasksBrokerPassword PASS: TestMqttStatus_EmptyWhenNoStatsFile PASS: TestMaskBrokerURL_Patterns (10 subtests) $ cd cmd/ingestor && go test -run 'TestSourceStatus|TestSnapshotSourceStatuses' -v ./... PASS: TestSourceStatus_BasicLifecycle PASS: TestSourceStatus_Disconnect PASS: TestSnapshotSourceStatuses_ReturnsAll $ node test-mqtt-status-panel.js 7 passed, 0 failed ``` Full `go test ./...` clean in both `cmd/server` and `cmd/ingestor`. ## Preflight overrides - `cross-stack`: justified — issue #1043 is intrinsically full-stack (ingestor stats → server endpoint → observers panel). Per-stack split would land an unreachable endpoint or a fetch with no backend. - `check-xss-sinks` (public/mqtt-status-panel.js:55): justified — the flagged `innerHTML=` is a fully-static literal (empty-state placeholder, no payload data interpolated). All payload-bearing `innerHTML=` sites in this file run through `escapeHTML` (defined in the same file); the test `renderPanel never echoes a plaintext password (defense-in-depth)` exercises the rendered HTML against payload strings. ## Acceptance criteria - [x] `/api/mqtt/status` returns per-source connection state — `cmd/server/mqtt_status.go` - [x] UI panel shows all configured sources with live status — `public/mqtt-status-panel.js` - [x] Connection state updates on reconnect/disconnect events — `MarkConnect` / `MarkDisconnect` wired in `cmd/ingestor/main.go` - [x] Broker URLs don't expose passwords in the API response — `maskBrokerURL` + 13 test cases - [x] Works with 1-N sources — registry is keyed per-source, snapshot iterates the map **Partial fix for #1043** — per-packet `mqtt_source` attribution (the issue's "Follow-up" section) is **deferred** per the `mc-bot-triaged:v1` triage and the autofix comment ("Per-packet attribution deferred to follow-up issue"). That work requires a new observation-row column and DB schema migration, both explicitly out of scope for this PR. Refs #1043 --------- Co-authored-by: openclaw-bot <bot@openclaw.local>	2026-06-12 08:11:02 -07:00
3 people	a4776557ae	feat(#1290): use firmware repeat:on|off hint to exclude listener-only observers from disambiguator (#1624) ... Closes #1290. cross-stack: justified — backend persists firmware-side `repeat` hint to a new observers column, frontend surfaces the listener/repeater status as a badge on the observers list and node-detail Heard By table per the issue's UI acceptance criterion. ## What Firmware 1.16 publishes a `repeat: on|off` flag in the MQTT `/status` JSON (confirmed by @cwichura on the issue thread — see [`MQTTMessageBuilder.cpp:58`](https://github.com/agessaman/MeshCore/blob/b45373a31f111fb0de98bb3b168226d09ceadc47/src/helpers/MQTTMessageBuilder.cpp#L58) in `agessaman/MeshCore mqtt-bridge-implementation-flex`). Listener-only observers (`repeat:off`) by firmware contract never relay packets, so they cannot legitimately be a hop in someone else's resolved path. This PR plumbs the hint end-to-end so the disambiguator stops considering them. ## How * **`internal/dbschema`**: idempotent `can_relay INTEGER DEFAULT 1` migration on `observers`, plus `AssertReady` probe (server fatal-logs if absent). Mirrored in `cmd/ingestor/db.go` `CREATE TABLE` for fresh DBs. Annotated `PREFLIGHT: async=true` — `DEFAULT 1` is constant so SQLite does this as a metadata-only schema rewrite. * **`cmd/ingestor`**: `extractObserverMeta` accepts `repeat` as bool, case-insensitive string (`on|off|true|false|yes|no`), or numeric `0|1`. Missing field → `nil` → `COALESCE` preserves the existing column value (back-compat with legacy observers). Plumbed through `UpsertObserverAt` and the prepared upsert statement. * **`cmd/server`**: `GetNonRelayObserverPubkeys` + new `prefixMap.markNonRelay` drop matching candidates inside `pm.resolveWithContext` at the top of the resolver, so all 4 tiers see the pruned candidate set. `ObserverResp.CanRelay` is surfaced on `/api/observers` and `/api/observers/{id}`. `GetNodeHealth` enriches per-observer rows with `can_relay` so the node-detail badge renders. Probe-and-fall-back when the `can_relay` column is absent (legacy test fixtures). * **`public/`**: listener vs repeater pill on observers list, observer detail `Relay` stat card, and node-detail `Heard By` table. CSS uses existing theme vars. ## Test Added `TestResolveWithContext_ExcludesNonRelayObservers_Issue1290` in `cmd/server/resolve_non_relay_1290_test.go` covering all three required cases: * `repeat:off` pubkey → not a candidate (assertion failed in red commit `5f7fdb96`, passes after green `f12911dc`) * `repeat:on` pubkey → still a candidate (regression guard) * legacy obs (no field) → still a candidate (back-compat) Red→green proof: ``` $ git log --oneline origin/master..HEAD f12911dc feat(#1290): exclude listener-only observers from path-hop disambiguator 5f7fdb96 test(#1290): red — assert listener-only observers excluded from path-hop candidates ``` Full server + ingestor + dbschema + migrate test suites pass locally. ## Acceptance checklist (from #1290) * [x] Ingestor parses `repeat` field (boolean OR string `on|off`) * [x] Field persisted on `observers` table (new `can_relay BOOLEAN` column, idempotent migration via `internal/dbschema`) * [x] Server's disambiguator (`pm.resolveWithContext`) excludes `can_relay=false` observer-nodes from path-hop candidate set * [x] UI badge on observers list + node detail page indicating "listener" vs "repeater" * [x] Backward compat: legacy observers default to `can_relay=true` * [x] Test: `repeat:off` → NOT a candidate * [x] Test: `repeat:on` → IS a candidate * [x] Test: legacy → IS a candidate ## Out of scope (preserved per issue) Backfilling already-resolved paths is left as a follow-up. No firmware/broker changes. --------- Co-authored-by: Kpa-clawbot <bot@kpa-clawbot.local> Co-authored-by: openclaw-bot <bot@openclaw>	2026-06-08 01:27:13 -07:00
Kpa-clawbotandcorescope-bot	3d12266595	fix(#1608): address PR #1609 follow-up findings — config doc, receipt-time liveness, buffer stop/clamp warn (#1623) ... Follow-up to #1609 / #1608. Addresses the 5 unresolved findings from the PR #1609 round-1 polish review. ## Findings addressed | Tag | Severity | Fix | Commits | |-----|----------|-----|---------| | **B1** | BLOCKER | Document `ingestBufferSize` in `config.example.json` near other ingestor knobs. Default `50000`, comment text from review. | `f0b4e411` | | **M1** | MAJOR (option 1 from review) | Split receipt-time vs post-write liveness: add `SourceLivenessState.LastReceiptUnix` + `MarkReceipt`, stamp at the MQTT receipt callback, leave `LastMessageUnix` post-write only. Drop the double-stamp at receipt that masked write-path stalls. Surface both clocks via the ingestor stats file (`source_liveness`) and the server's `/api/healthz` (`ingest_liveness`, additive — older builds unaffected). | RED `fa78233d` / GREEN `bc81b544` | | **M1 (drop-log)** | MAJOR | Log every drop when buffer is at capacity. Removes the `n==1 \|\| n%1000` throttle that hid the first stall behind 1000 lost packets. The Submit drop branch only fires when the channel is at cap so volume is naturally bounded by the stall, not by an arbitrary modulo. | RED `a468763e` / GREEN `7b24fce5` | | **m1** | MINOR | Add `IngestBuffer.Stop()` and `Done()` so tests stop leaking the consumer goroutine that `Start()` spawns. Existing tests gain `t.Cleanup(b.Stop)`. Drain semantics: stop-before-Ready exits immediately; stop-after-Ready best-effort drains queued jobs. | RED `8430c822` / GREEN `78c9b223` | | **m2** | MINOR | `NewIngestBuffer(<1)` now logs a `[ingest-buffer] WARN` line on clamp so misconfigured `ingestBufferSize` values are visible instead of silently running a 1-slot queue. Test captures log output. | RED `62119ab4` / GREEN `815bfd02` | | **m3** | MINOR | Add godoc to `Submit` and `Ready` documenting the Start-before-Submit / Start-before-Ready ordering invariant. | `564a813b` | ## TDD discipline Each behavioral fix (M1, M1-drop-log, m1, m2) lands as a red-then-green pair. Red commits compile + run + fail on assertion, verified locally before the green commit. Per-finding red→green pairs are visible in the commit graph above. B1 and m3 are docs-only and ship as single commits (preflight script accepts them under the docs/comments exemption). ## Schema compatibility `/api/healthz` change is purely additive: `ingest_liveness` is only included when the ingestor publishes the new `source_liveness` field, so older ingestor + newer server combos are unaffected. Field order in the response stays stable for prior consumers. ## Test output - `go test -count=1 -timeout 180s ./cmd/ingestor/...` → green (160s) - `go test -count=1 -timeout 300s ./cmd/server/...` → green (48s) - Race-mode runs of the touched packages (`IngestBuffer|Liveness|Watchdog|Receipt|Healthz`) → green - Full-package race runs locally exceed the brief's 120s timeout on pre-existing slow integration tests (TestObsTimestampIndexMigration, TestNeighborEdgesBuilderDeltaScan); CI has the headroom. ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → all hard gates pass, no warnings. ## Files changed - `config.example.json` — B1 - `cmd/ingestor/ingest_buffer.go` — m1, m2, M1-drop-log, m3 - `cmd/ingestor/ingest_buffer_test.go` — m1, m2, M1-drop-log - `cmd/ingestor/mqtt_watchdog.go` — M1 - `cmd/ingestor/mqtt_watchdog_m1_test.go` — M1 (new) - `cmd/ingestor/main.go` — M1 (receipt callsite) - `cmd/ingestor/stats_file.go` — M1 (publish `source_liveness`) - `cmd/server/perf_io.go` — M1 (type + reader) - `cmd/server/healthz.go` — M1 (surface `ingest_liveness`) Original review reference: PR #1609 polish review by the M-axis bot. --------- Co-authored-by: corescope-bot <bot@corescope.local>	2026-06-07 09:28:51 -07:00
Kpa-clawbotandcorescope-bot	222bfdf6cf	feat(perf): SQLite writer-lock wait/hold instrumentation per component (#1340) (#1594) ... ## What Per-component SQLite writer-lock instrumentation so the next neighbor-builder-style write-lock starvation (root cause of #1339, invisible to operators for ~3 days) is detectable from `/api/perf`. Adds `Store.WriterExec` / `Store.WriterTx` wrappers that gate every wrapped call on a package-level `writerMu` so the wait the SQLite driver hides becomes Go-visible, and record `wait_ms` + `hold_ms` + `contention_total` (wait_ms > 100ms) under a component tag. Per-component p50/p95/p99 + max are published to `/api/perf/write-sources` under `.writer_perf` via the existing ingestor stats-file path. Slow-writer log line (`[db-slow-writer] component=X duration=Yms query=<200ch>`) fires on `hold_ms > 500ms` (threshold overridable via `CORESCOPE_DB_SLOW_WRITER_MS` env var). ## Tagged call sites | Component | Location | |-----------|----------| | `mqtt_handler` | `InsertTransmission` (db.go) | | `neighbor_builder` | `buildAndPersistNeighborEdges` (neighbor_builder.go) | | `prune_packets` | `PruneOldPackets` (maintenance.go) | | `prune_observers` | `RemoveStaleObservers` + orphan-metrics cleanup (db.go) | | `prune_metrics` | `PruneOldMetrics` (db.go) | | `vacuum` | `RunIncrementalVacuum` + `CheckAutoVacuum`'s full VACUUM (db.go) | ## TDD red→green - **Red commit** `68de585b` — `cmd/ingestor/db_writer_perf_test.go` + `Store.Writer*` stubs at end of `db.go`. Test synthetically blocks the writer for 60s tagged `neighbor_builder`, then asserts `mqtt_handler.wait_ms.p99 > 50000ms` on concurrent inserts. Fails on the assertion (p99 = 0.0ms) with the stub — not a build error. - **Green commit** `6a9be174` — replaces stubs with real wait/hold/contention aggregator + wires every writer call site. Same test passes: ``` 2026/06/05 04:36:47 [db-slow-writer] component=neighbor_builder duration=60059.0ms query=COMMIT --- PASS: TestWriterStarvationVisibleInPerf (60.40s) PASS ok github.com/corescope/ingestor 60.408s ``` ## Scope discipline - **API**: no public `Store`/`DB` signature change. Only additive exports. - **Server**: extends existing `/api/perf/write-sources` JSON with `.writer_perf` — does **not** add a new route, does **not** replace `handlePerf`. Empty `.writer_perf` map when paired with an older ingestor. - **Read/write invariant** (#1283) preserved: all instrumentation lives on the ingestor's writer connection. - **Files touched** (6 total): `cmd/ingestor/db.go`, `cmd/ingestor/db_writer_perf_test.go`, `cmd/ingestor/maintenance.go`, `cmd/ingestor/neighbor_builder.go`, `cmd/ingestor/stats_file.go`, `cmd/server/perf_io.go`, `config.example.json`. ## Deferred (acceptance items NOT in this PR) - **`mbcap_persist` component tag** — `RunMultibyteCapPersist`'s tx is intentionally NOT wrapped in this PR to stay within the implementation brief's 3-files-outside-whitelist budget. One-file follow-up to instrument. - **CI smoke test** asserting "neighbor-builder hold_ms < 1000ms on 100k-obs fixture" — deferred to a separate PR per the brief; this PR is scoped to instrumentation only. ## Preflight overrides PREFLIGHT-MIGRATION-SCALE: <30s N=runtime — the async-migration gate flagged five `instrumentedExec` / wrapped-`tx.Exec` lines on `DELETE FROM observer_metrics`, `UPDATE observers`, `DELETE FROM observer_metrics`, `DELETE FROM observations`, `DELETE FROM transmissions`. These are **not** schema migrations — they are the existing runtime prune / retention queries that already ran sync against `s.db.Exec` / `tx.Exec` on every retention cycle on master. This PR only swapped the surface call (sync → sync, via the wrapper) to record wait/hold timing; no new sync schema work was introduced. Behavior on production data is identical to master. Also: red commit's synthetic `UPDATE nodes SET name = name WHERE 0` is a test-only stub designed to acquire the writer without mutating any row (the `WHERE 0` is a no-op predicate). Fixes #1340 --------- Co-authored-by: corescope-bot <bot@corescope.local>	2026-06-06 21:05:59 -07:00
Kpa-clawbotandcorescope-bot	1b112f0b08	feat(memlimit): GOMEMLIMIT via runtime.maxMemoryMB in server + ingestor (#1010) (#1595) ... Red commit: 929da3c6dcc1b619c27478291125d1c91323db8f — CI: https://github.com/Kpa-clawbot/CoreScope/commit/929da3c6dcc1b619c27478291125d1c91323db8f/checks Fixes #1010. ## What Adds `GOMEMLIMIT` support to both `cmd/server` and `cmd/ingestor` per the locked triage scope on #1010. Precedence (env wins): 1. `GOMEMLIMIT` env var 2. `runtime.maxMemoryMB` config field (new) 3. Server only: implicit `packetStore.maxMemoryMB * 1.5` (existing #836 behavior, unchanged when `runtime.maxMemoryMB` is absent) 4. Otherwise unset — default Go behavior preserved (backwards compatible) Each startup logs a `[memlimit]` line echoing the effective source/limit, or an "unset → default" note when neither is set. ## Changes - `cmd/ingestor/memlimit.go` — new, `applyMemoryLimit(runtimeMaxMB, envSet)`. - `cmd/ingestor/memlimit_test.go` — new, env/config/none/precedence assertions. - `cmd/ingestor/config.go` — new `RuntimeConfig{MaxMemoryMB int}` field. - `cmd/ingestor/main.go` — wires `applyMemoryLimit` into startup right after `LoadConfig`. - `cmd/server/config.go` — new `RuntimeConfig` + `cfg.Runtime` field. - `cmd/server/main.go` — adds explicit `runtime.maxMemoryMB` precedence over packetStore-derived; existing `warnIfMemlimitUnderprovisioned` (#1264) unchanged. - `config.example.json` — new `runtime` block with `_comment_runtime_maxMemoryMB` per the Config Documentation Rule. - `README.md` — sizing-table row with ≥1.5× working set floor + death-spiral warning. ## TDD - Red: `929da3c6` — ingestor `applyMemoryLimit` stub returns `(0,"none")`; four tests fail on assertions (`expected source=env, got "none"`, etc.) — no compile errors. - Green: `953ec9d8` — implements ingestor `applyMemoryLimit`, wires startup, threads `runtime.maxMemoryMB` through server too. ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → clean (all gates pass, all warnings pass). ## Out of scope - `pprof`-verified GC-trigger acceptance criterion from the original issue — requires production tracing; the triage scope is the operator-tunable plumbing. - Container auto-detection of cgroup memory limit (already covered by #1264's `warnIfMemlimitUnderprovisioned`). --------- Co-authored-by: corescope-bot <bot@corescope>	2026-06-06 21:05:56 -07:00
efitenandClaude Opus 4.8	18810b5c13	fix(ingestor): subscribe to MQTT before startup maintenance, buffer until writer is free (#1608) (#1609) ... ## Summary Closes #1608. The ingestor's MQTT connect/subscribe loop ran **last** in `main()`, after the synchronous startup-maintenance block. Because all writes share a single SQLite writer (#1283), that maintenance — and the connect loop after it — serialize behind any long-running async migration. The subscription therefore came up minutes late (observed ~4.5 min after the v3.8.3 `obs_observer_ts_idx_v1` index build over ~4.9M rows), and QoS-0 packets published in that window were dropped. This decouples **receipt** from **write**: - New `IngestBuffer` — a bounded FIFO drained by a **single** gated consumer goroutine. - The MQTT subscription is brought up first; its publish handler stamps source liveness at receipt and enqueues a `handleMessage` closure. - Startup maintenance runs, then `WaitForAsyncMigrations()`, then `IngestBuffer.Ready()` opens the gate and the backlog drains. A single consumer preserves the single-writer invariant (#1283); buffering replays the original messages, so it introduces **no duplicates** (unlike a QoS-1 broker queue). Broker-agnostic — helps direct-connect and bridged operators alike. ## Changes - `cmd/ingestor/ingest_buffer.go` — `IngestBuffer` (`Submit`/`Start`/`Ready`/`Dropped`/`Pending`); non-blocking submit with drop-on-full counter; single consumer. - `cmd/ingestor/config.go` — `ingestBufferSize` knob (default 50000). - `cmd/ingestor/main.go` — reorder boot: connect/subscribe **before** startup maintenance; stamp liveness at receipt; `Ready()` after maintenance + `WaitForAsyncMigrations()`; periodic stats log buffer `pending`/`dropped`. ## Test plan - [x] `go test ./...` in `cmd/ingestor` — `IngestBuffer` suite covers gating-until-ready, FIFO order, drop-on-full, serial execution (single-writer), and concurrent-submit. - [ ] `go test -race` in CI (concurrency on `IngestBuffer`). - [ ] Manual: restart with a pending heavy migration → `subscribed to meshcore/#` appears within seconds; `[ingest-buffer] write path ready` after the migration; packets received during the window are written after `Ready()` (0 dropped under normal traffic); stall watchdog stays quiet (liveness stamped at receipt). ## Out of scope A hard crash while messages sit in the in-memory buffer still loses them; crash-durability requires broker-side persistence, which is topology-specific. This PR closes the startup-migration and deploy loss windows. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-06 21:05:53 -07:00
Kpa-clawbotandcorescope-bot	9612f08e46	fix(#1610): decode firmware 1.16.0 extended ACK (5/6-byte payloads) (#1618) ... ## Summary Firmware 1.16.0 (`companion-v1.16.0`) ships variable-length `PAYLOAD_TYPE_ACK` payloads: 4 bytes (legacy) → 5 bytes (4-byte CRC + 1-byte attempt, commit `f6e6fdaa`) → 6 bytes (+ 1-byte RNG, commit `a130a95a`). CoreScope's decoder previously truncated past the 4-byte CRC and discarded the attempt + RNG bytes. This PR teaches `cmd/ingestor/decoder.go` to surface the extended bytes on the decoded payload so the DB/UI can distinguish v1.15 vs v1.16 senders, with no schema or wire-compat changes. Partial fix for #1610 — top-level ACK + multipart-inner ACK are covered. PATH-extra ACK parsing (`decodePathPayload`) is deferred to #1612 per triage. ## Changes - `decodeAck` reads 4/5/6-byte payloads. Keeps `extraHash` (4-byte CRC) for compat; adds optional `ackLen`, `ackAttempt`, `ackRand` JSON fields. Legacy 4-byte ACKs leave attempt/rand `nil`. - `decodeMultipart` ACK branch relaxes the `len >= 5` floor so the inner blob can be 4/5/6 bytes (multipart `payload_len` 5/6/7). Adds `innerAckLen`, `innerAckAttempt`, `innerAckRand`. - All additions are `omitempty` — backwards-compatible JSON only. No DB column, no schema migration, no frontend change. ## Out of scope (per issue triage) - `decodePathPayload` PATH-extra parsing — tracked separately in #1612. - Frontend rendering of attempt counter — leave for a follow-up if the DB/UI eventually wants to display it. ## TDD - **Red commit `3fce0465`** adds `cmd/ingestor/issue1610_test.go` with 6 new assertions (legacy 4-byte, extended 5/6-byte, multipart variants of each). New fields are declared on `Payload` so the test compiles, but no decoder populates them yet — tests fail on `ackLen=<nil> want 4` etc. Verified isolation with `git stash` of decoder.go + re-run. - **Green commit `5165c202`** implements the decoder changes. `go test ./...` in `cmd/ingestor` passes. ## Fixtures Synthetic wire vectors built by hand against the firmware spec — the issue did not provide real captures. Each test cites the firmware ref + commit it derives from (`BaseChatMesh.cpp:218-234`, commits `f6e6fdaa` and `a130a95a`). ## References - Issue #1610 - Firmware tag `companion-v1.16.0` @ `07a3ca9e` - Upstream PR meshcore-dev/MeshCore#2594 - Blog: https://blog.meshcore.io/2026/06/06/release-1-16-0 --------- Co-authored-by: corescope-bot <bot@corescope.local>	2026-06-06 21:05:50 -07:00
Kpa-clawbotandmeshcore-bot	af669438ff	docs+test(ingestor): document writeStatsAtomic symlink-replace semantics + regression test (#1170) (#1588) ... Fixes #1170. ## What 1. **Doc comment** on `writeStatsAtomic` (`cmd/ingestor/stats_file.go`) spelling out the two-sided symlink story: - tmp side (`path+".tmp"`): protected by `O_NOFOLLOW` (existing behavior, already noted). - rename side (`path` itself): NOT protected by `O_NOFOLLOW`; instead `os.Rename` semantics are relied upon — rename atomically replaces any existing entry at `path` (including a symlink) with the new regular file. The symlink target is never written through because all writes happened to the unrelated tmp file before rename. 2. **Regression guardrail test** `TestWriteStatsAtomic_SymlinkAtDestIsReplaced` in `cmd/ingestor/stats_file_test.go` that pre-plants a symlink at the destination path pointing to an unrelated target file, calls `writeStatsAtomic`, and asserts: - (a) `os.Lstat(path).Mode()&os.ModeSymlink == 0` (post-write path is a regular file, not a symlink) - (b) the original symlink target's sentinel bytes are unchanged. If a future refactor swaps `os.Rename` for a destination-symlink-following primitive (e.g. `open(path, O_WRONLY)` without `O_NOFOLLOW`, or a copy-then-truncate), the test fails loudly. ## TDD note (red-commit exemption) The current `writeStatsAtomic` ALREADY satisfies the new test's assertions — `os.Rename` does the right thing today. Per the fix-issue skill's exemption for pure-documentation / guardrail tests on already-correct behavior, no fabricated red commit was constructed; the test stands as a pinning regression guard. The two commits are therefore: (1) test addition, (2) doc comment. ## Scope - `cmd/ingestor/stats_file.go` — doc comment only - `cmd/ingestor/stats_file_test.go` — one new test function No production behavior change. No public API change. No new dependencies. No CI workflow changes. `O_NOFOLLOW` and the existing tmp-side behavior are untouched. ## Preflight All hard gates pass (PII, branch scope, red commit, CSS vars, LIKE-on-JSON, sync/async migration, XSS sinks). No warnings. --------- Co-authored-by: meshcore-bot <bot@meshcore.local>	2026-06-04 18:48:23 -07:00
Kpa-clawbotandKpa-clawbot	cd19285f7f	fix(ingestor): defense-in-depth empty-scope guard in UpdateNodeDefaultScope (#1534) (#1575) ... ## Summary Follow-up to PR #1569 (merged). Adds defense-in-depth at the DB layer for the #1534 default_scope-overwrite class of bug. PR #1569 fixed #1534 by guarding the call site in `handleMessage` with `if shouldUpdateDefaultScope(pktData)`. Adversarial review of #1569 flagged this as one-layer defense: a future refactor that drops the call-site `if` and calls `store.UpdateNodeDefaultScope(pubkey, pktData.ScopeName)` unconditionally would silently re-introduce the bug — overwriting a previously-correct `default_scope` (e.g. `#belgium`) with the empty string. This PR adds the belt-and-braces guard recommended by that review: - `Store.UpdateNodeDefaultScope(pk, "")` is now a silent no-op (early `return nil`) - New DB-layer regression test that fails on `master` and proves the DB function used to write `""` straight through - Two new call-site anchor tests that drive a transport-scoped ADVERT end-to-end through `handleMessage` (matched + unmatched region key) so the existing call-site guard from #1569 can't be deleted without a test going red Net production change: 8 lines in `cmd/ingestor/db.go`. No behavior change for any non-empty scope. ## Why this is a follow-up, not a re-fix Issue #1534 is already closed by #1569 and `master` no longer regresses for users (the call-site guard is in place). This PR is purely belt-and-braces — it adds the second layer of defense the adversarial reviewer asked for and the test coverage that anchors both layers. ## Files changed | File | Change | |------|--------| | `cmd/ingestor/db.go` | +8 — empty-scope early return in `UpdateNodeDefaultScope` | | `cmd/ingestor/db_test.go` | +43 — `TestUpdateNodeDefaultScope_EmptyScopeIsNoop` | | `cmd/ingestor/main_test.go` | +97 — `TestHandleMessageAdvert_EmptyScopeSkipsDefaultScopeUpdate` + `TestHandleMessageAdvert_MatchedScopeUpdatesDefaultScope` | ## Red → green commits - **red** `c062af59` — `test(ingestor): red — DB-layer empty-scope guard regression test for #1534` - Adds three tests; `TestUpdateNodeDefaultScope_EmptyScopeIsNoop` fails on assertion (`default_scope` overwritten with `""`) - Two call-site tests pass already (call-site guard merged in #1569) — they anchor that behavior against future refactors - **green** `7ab12d53` — `fix(ingestor): defense-in-depth empty-scope guard in UpdateNodeDefaultScope (#1534)` - Adds the early-return; all three tests green ## Operator remediation (from issue #1534) Operators whose production DB still has rows where `default_scope` was overwritten with the empty string before #1569 deployed can clean up with: ```sql -- Inspect affected rows first SELECT public_key, name, default_scope FROM nodes WHERE default_scope = ''; SELECT public_key, name, default_scope FROM inactive_nodes WHERE default_scope = ''; -- Convert empty-string default_scope back to NULL so the next valid -- matched-scope advert can re-populate it cleanly. UPDATE nodes SET default_scope = NULL WHERE default_scope = ''; UPDATE inactive_nodes SET default_scope = NULL WHERE default_scope = ''; ``` After #1569 + this PR are deployed, no new rows can be created with `default_scope = ''` from this code path. ## Test plan ```bash cd cmd/ingestor && go test ./... -count=1 # ok github.com/corescope/ingestor ~98s ``` ## Preflight Clean — PII, branch scope, red commit, CSS-var defined, CSS self-fallback, LIKE-on-JSON, sync migration, async-migration gate, XSS sinks all pass. No warnings. --------- Co-authored-by: Kpa-clawbot <bot@meshcore-analyzer>	2026-06-04 10:35:46 -07:00
Kpa-clawbotandcorescope-bot	05af6c6ee5	fix(ingestor): skip default_scope update when ScopeName is empty (#1534) (#1569) ... Red commit: e5668585da Fixes #1534 ## Problem `cmd/ingestor/main.go:720` called `UpdateNodeDefaultScope` whenever a packet was transport-scoped (`IsTransportScoped == true`), without checking whether `matchScope()` actually returned a region match. Transport-scoped adverts from non-matching regions carry `ScopeName=""`, which then overwrote previously-correct `nodes.default_scope` values with the empty string — surfacing as "unknown scope" / "--" in the node sidebar. ## Fix Extracted the guard into `shouldUpdateDefaultScope(pktData)` and added the non-empty `ScopeName` check: ```go return pktData.IsTransportScoped && pktData.ScopeName != "" ``` ## TDD - Red commit (`e5668585`): adds `TestBuildPacketDataScopeMatchingNoMatch` + helper that mirrors the buggy guard. CI must fail on assertion. - Green commit (`aab7f5d7`): adds the `ScopeName != ""` check. Test passes. ## Out of scope (deferred) - The optional one-time backfill / migration marker removal described in the issue — new matching adverts will self-correct existing rows. - Refactor of `IsTransportScoped` + `ScopeName` into a typed wrapper. ## Files - `cmd/ingestor/main.go` — guard + new helper - `cmd/ingestor/main_test.go` — regression test ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` — clean. --------- Co-authored-by: corescope-bot <bot@corescope.local>	2026-06-04 10:06:13 -07:00
4 people	3feb97f16f	fix(ingestor): write resolved_path on new observations (regression from #1289) (#1548) ... # fix(ingestor): write resolved_path on new observations (full restore — closes #1547 + #1560) Fixes #1547. Closes #1560. ## Root cause PR #1289 (the "ingestor owns the neighbor graph; server is read-only" refactor, ~2026-05-21) moved the neighbor graph + schema writes to the ingestor, and as a side-effect removed the server-side writer that populated `observations.resolved_path` AND the context-aware `pm.resolveWithContext` that disambiguated 1-byte prefix collisions. Result: every observation inserted after the deploy has `resolved_path = NULL` (3.1M/6.3M NULL on staging; 100% NULL on fresh deploys; symptom on Cascadia: hops fail to resolve because the small-mesh client-side fallback breaks on prefix collisions). ## Full restore This PR resolves both single-byte and multi-byte prefix paths. Single-byte disambiguation uses NeighborGraph adjacency and ADVERT `from_pubkey` anchoring, ported from pre-#1289 `pm.resolveWithContext` logic (last good at cmd/server/store.go @ commit 450236d5) and the #1144 / #1352 fixes. New file `cmd/ingestor/path_resolver.go`: - `NeighborGraph` + `neighborGraphHolder` — in-memory adjacency snapshot, atomic-published. - `loadNeighborGraph(db)` — one-shot SELECT from `neighbor_edges`. - `resolveHopWithContext(hop, anchor, graph, idx, exclude) *string` — single-hop, tier-1 disambiguator. - `resolvePathWithContext(hops, fromPubkey, graph, idx) []*string` — walks the path, anchoring hop 0 on `from_pubkey` (ADVERTs) and each subsequent hop on the previous resolved hop, excluding already-resolved pubkeys. - `Store.RefreshNeighborGraph()` — called on warm-up and every 60s tick in the neighbor-edges builder alongside `RefreshPrefixIndex`. Existing file `cmd/ingestor/resolved_path.go` (PR #1547 base) is untouched: `resolvePath` + `marshalResolvedPath` + the all-nil → empty-string clobber-guard contract are preserved verbatim. `cmd/ingestor/db.go` — `InsertTransmission` now calls `resolvePathWithContext` instead of the naive `resolvePath`. ## Algorithm (per hop) 1. Look up candidate pubkeys by prefix-match (existing `prefixIndex`). 2. `len==0 → nil`; `len==1 → that pubkey`. 3. `len>1` → filter by `NeighborGraph` adjacency to the anchor. Anchor is `from_pubkey` for hop 0 on ADVERTs, the previous resolved hop otherwise. Exactly 1 surviving candidate → use it; else nil. 4. Previously resolved hops (and the originator) are excluded from downstream candidate pools — a packet does not revisit a node. Tier-2/3/4 from pre-#1289 (geo proximity, GPS preference, observation-count fallback) are intentionally NOT ported — those were noisy in practice and belong in a separate enhancement, not in this regression restore. ## Out of scope - The ~3.1M existing NULL rows from the regression window. Filed as a follow-up backfill task — too risky to bundle here (touches a 6M-row table). - The dead-flag bug #1546 — separate concern. ## TDD red → green - Red commit `80b0f476` — adds five new context-resolver tests; stub `resolvePathWithContext` falls back to naive `resolvePath`. CI run 26946935615 → **failure** with assertion errors on the three collision tests (`TestResolveHopWithContext_OneByteCollision_AdjacencyResolves`, `TestResolvePathWithContext_TwoHopChainAnchoredOnFromNode`, `TestResolvePathWithContext_AdvertAnchoring`); the two regression tests (multi-byte still works + all-nil contract) stayed green. - Green commit `7b4950ce` — real algorithm + InsertTransmission wiring + RefreshNeighborGraph in the builder tick. All five new tests pass; original four `resolved_path` tests stay green. ## Verification - `go test -race ./cmd/ingestor/...` for the 11 affected tests — pass. - `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` — exit 0 (all gates clean). - PII grep on body + diff: clean. Tested with: existing `TestInsertTransmissionWritesResolvedPath` + `TestInsertTransmissionDoesNotClobberResolvedPathOnAllNil` (PR #1547 base) plus the new collision-resolution suite: - `TestResolveHopWithContext_OneByteCollision_AdjacencyResolves` — 3-of-5 nodes share `0x5c`, chain A↔B↔C↔D↔E; anchored on A, hop `5c` → B. - `TestResolvePathWithContext_TwoHopChainAnchoredOnFromNode` — path `[5c, 5c]` from_node A → `[B, C]`. - `TestResolveHopWithContext_NoAdjacencyContext_ReturnsNil` — 3 ambiguous candidates, no anchor / non-adjacent anchor → nil. - `TestResolvePathWithContext_AdvertAnchoring` — ADVERT, `from_pubkey=A`, path `[5c]` → only-adjacent neighbor B. - `TestResolvePathWithContext_RegressionMultiByteStillWorks` — unique-prefix path with no graph context still resolves. - `TestResolvePathWithContext_AllNilContractPreserved` — unresolvable path → `marshalResolvedPath==""` (clobber-guard from PR #1548 untouched). ## Browser-validated N/A — backend-only change. Frontend already handles populated `resolved_path` via `getResolvedPath` in `cmd/server/db.go` and `public/packets.js`. ## Round-1 fixes addressed - **MUST-FIX #1 (data-loss clobber on all-nil resolution):** when every hop fails to resolve, `marshalResolvedPath` returns `""` instead of `"[null,null,...]"`, so `nilIfEmpty` → SQL NULL and the `COALESCE(excluded.resolved_path, resolved_path)` UPSERT preserves any previously stored good value on re-ingest. Regression test asserts: insert a transmission, observe `resolved_path` populated, wipe the prefix index, re-ingest the same packet, assert the existing `resolved_path` is unchanged. --------- Co-authored-by: corescope-bot <bot@corescope> Co-authored-by: openclaw-bot <bot@openclaw> Co-authored-by: openclaw-bot <bot@openclaw.local>	2026-06-04 07:35:13 -07:00
Kpa-clawbotandclawbot	7b43045043	fix(security): sanitize 3 more log-injection sites missed by #1540 (#1544) ... Follow-up to merged #1540. Self-review of #1540 found 3 additional `log.Printf` sites interpolating MQTT-controlled strings without `sanitizeLogString` — fixing here for completeness. ## Sites fixed | File:line | Format | MQTT-controlled fields | Attacker scenario | |---|---|---|---| | `cmd/ingestor/main.go:531` | `status: %s (%s)` | `name`, `iata` | Hostile node sends status with `name="evil\r\n[security] forged-line"` — appears as a fake log line in operator dashboards / journalctl. | | `cmd/ingestor/main.go:854` | `channel message: ch%s from %s` | `channelIdx`, `sender` | Attacker spoofs `sender="evil\r\n[security] backdoor-installed"` on any channel message — same forged-line outcome. | | `cmd/ingestor/main.go:940` | `direct message from %s` | `sender` | DM injection via crafted sender field, same outcome. | All three now route through `sanitizeLogString` from `cmd/ingestor/sanitize_log.go` (added by #1540) which replaces CR/LF/control bytes with `?`. ## TDD Red commit (`8b3ad398`) adds 3 testable format helpers (`formatStatusLog`, `formatChannelMessageLog`, `formatDirectMessageLog`) plus tests pinning CR/LF stripping. Helpers return raw `fmt.Sprintf` output, so tests fail on assertion (not build). Green commit applies `sanitizeLogString` inside the helpers and swaps the 3 call sites in `main.go` to use them. Tests red-on-revert (verified locally). ## Scope Strictly the 3 sites above. No other refactors. No changes to `sanitizeLogString` itself. --------- Co-authored-by: clawbot <clawbot@users.noreply.github.com>	2026-06-03 15:01:51 -07:00
3 people	e438451dc9	feat(preflight): hard-fail gate on sync schema migrations + async runner (#1541) ... Closes the recurring "sync migration on large table" regression class (#791-style, #1483-style). ## Problem Pattern that keeps repeating: 1. A perf/feature PR adds `CREATE INDEX` / `ALTER TABLE` / `UPDATE ... WHERE` in a migration file (typically `cmd/ingestor/db.go`). 2. Local dev DB has ~100 rows. Migration returns in milliseconds. CI is green. 3. Reviewers approve on plan correctness; nobody knows what the prod table size is. 4. First prod boot at scale (Cascadia: ~2600 nodes, 80K+ obs; previous prod: 1.9M+ obs) pins the ingestor at `[migration] Adding index...` for minutes. 5. Healthcheck times out → container restart → loop. Operator pages. Hotfix. Most recent case: `obs_observer_ts_idx_v1` in v3.8.3 — release notes already document an "expect a longer first boot" warning because we knew it would hit prod hard. ## What this PR adds **Async helper (`cmd/ingestor/async_migration.go`):** - `Store.RunAsyncMigration(ctx, name, fn)` — registers the migration as `pending_async` in a new `_async_migrations` bookkeeping table, returns to caller immediately, schedules `fn` in a goroutine on the shared backfill `WaitGroup`, transitions to `done` (or `failed` with error captured) on completion. - `Store.AsyncMigrationStatus(name)` and `Store.WaitForAsyncMigrations()` for tests/shutdown. - Idempotent: `done` rows short-circuit; `pending_async`/`failed` rows are retried on next boot. **Retroactive #1483 conversion (`cmd/ingestor/db.go`):** - `obs_observer_ts_idx_v1` (the composite `(observer_idx, timestamp)` index build on `observations`) is now scheduled via `RunAsyncMigration` from `OpenStore()` so the ingestor accepts packets immediately while the index builds in the background. - Legacy `_migrations` gate is preserved by the async fn → DBs that already completed the sync build stay no-op. **Annotation convention (`MIGRATIONS.md`):** Every new `CREATE INDEX` / `ALTER TABLE` / data-rewrite in a migration file must do ONE of: 1. Run via `Store.RunAsyncMigration(...)` (preferred for backfills). 2. Carry a `// PREFLIGHT: async=true reason="..."` comment directly above the migration block. 3. Include a `PREFLIGHT-MIGRATION-SCALE: <30s N=<scale>` line in the PR body. **TDD pair:** - Red commit `2c6744cc` — `TestRunAsyncMigration_PendingThenDone` against a stub helper. Build passes, assertion fails (`async migration fn did not start within 2s`). - Green commit `38354f32` — real helper + retroactive fix + docs. Test green. **Fixtures (`cmd/ingestor/testdata/preflight-migrations/`):** - `bad_sync_migration.go` — known-bad sample with no annotation. - `good_annotated_migration.go` — known-good sample with annotation. The preflight gate script can be unit-tested against these. ## Gate location (NOT in this PR) The actual `check-async-migrations.sh` lives in the OpenClaw skills directory at `~/.openclaw/skills/pr-preflight/scripts/` (separate from the repo) and is wired into `run-all.sh`. It greps the diff for new/modified migration blocks and hard-fails (exit 1) on any sync schema mutation lacking one of the three opt-outs above. The fixtures in this PR give maintainers a reproducible target. ## Why annotation-discipline, not size detection You cannot determine table size from a diff. The gate enforces that every author who adds a schema migration must consciously decide which bucket it falls into and write that down. That is the cheapest possible intervention that breaks the cycle. ## Testing - `go test ./...` in `cmd/ingestor` — all tests pass including the new `TestRunAsyncMigration_PendingThenDone`. - Manual: red commit fails on assertion (not build), green commit passes — verifiable by `git checkout 2c6744cc -- cmd/ingestor/async_migration.go && go test -run TestRunAsync ./cmd/ingestor` from the green commit. ## Preflight overrides None — clean run after the convention is applied. --------- Co-authored-by: clawbot <bot@openclaw.local> Co-authored-by: clawbot <bot@openclaw>	2026-06-03 21:03:59 +00:00
Kpa-clawbotandclawbot	800d61c382	fix(security): uniform limit-clamp, log-injection sanitization, SPA path validation (#1540) ... Follow-up to v3.8.3 security train. Found by non-XSS input-validation audit. Three findings closed in one PR — all defense-in-depth: medium is genuinely DoS-only (no data exposure), lows tighten log hygiene and SPA path handling so future router changes can't silently expose the filesystem. ## Findings addressed ### MEDIUM — unbounded `limit` on list endpoints - **What:** four list endpoints accepted `limit=999999999` and passed the value straight to SQL `LIMIT ?` and Go `make(..., 0, limit)`. - **Where:** `cmd/server/routes.go` — handlePackets (incl. multi-node branch), handleNodes, handleChannelMessages, handleAnalyticsSubpaths, handleAnalyticsSubpathsBulk per-group lim, handleDroppedPackets. - **Fix:** new `clampLimit(raw, def, max)` helper in `cmd/server/clamp_limit.go` plus `queryLimit(r, def, max)` HTTP wrapper. Caps: packets/nodes/channels/dropped = 500, analytics buckets / bulk-health = 200. Already-clamped endpoints (handleBulkHealth) migrated to the helper for uniformity. Silent clamp — no response-shape change. Negative / zero / non-numeric → default. ### LOW — log injection via newline in advert name - **What:** advert `name` field allows `\n` / `\t` (sanitizeName intentionally preserves them for display). Logged at two MQTT-ingest sites, an attacker with publish ACL could forge log lines. - **Where:** `cmd/ingestor/main.go:659,690`. - **Fix:** new `sanitizeLogString` in `cmd/ingestor/sanitize_log.go` strips control bytes < 0x20 and DEL with `?`. Wrapped at the two log call sites that interpolate `name=` and `observer=`. Stored display values untouched. ### LOW — SPA static handler depends on default mux path-cleaning - **What:** `cmd/server/main.go:469` joins `r.URL.Path` to root; safe today only because gorilla/mux runs `path.Clean` and `http.FileServer` rejects `..`. A future `SkipClean(true)` or router swap would silently expose the filesystem. - **Where:** `cmd/server/main.go` (spaHandler). - **Fix:** new `isSafeStaticPath` rejects requests whose decoded or raw path contains `..`, `%2e%2e`, `\\`, or `%5c` with a 400. Legit asset names with dots (`/app.js`, `/customize-v2.js`, `/themes/dark.css`) are unaffected. ## TDD - Commit 1 (red): adds `TestClampLimit`, `TestSpaHandlerPathTraversal`, `TestSanitizeLogString` with stub helpers — tests fail on assertions (not build errors), proving they gate the change. - Commit 2 (green): production fix. Revert the green commit and the red commit's assertions fail. ## Audit reference Source: non-XSS input-validation audit dated 2026-06-03 (workspace). Sibling PR `fix/xss-r2-trace-obs-anl` owns the XSS findings — not included here. --------- Co-authored-by: clawbot <clawbot@users.noreply.github.com>	2026-06-03 13:58:04 -07:00
3 people	13bdee57d4	perf: P0 hot-path fixes (observers, neighbor-graph, observer-analytics) (#1481) (#1483) ... ## What Three of the four P0s from #1481's scale-test findings. Each cuts a distinct hot path; together they target /api/observers, /api/analytics/neighbor-graph, and /api/observers/{id}/analytics — the top three live offenders. ### P0-1: 5-min atomic-pointer cache for default neighbor-graph response - Live p95 10.8s on the most-trafficked organic endpoint. - Background recomputer (5-min cadence per operator directive) builds the default-filter (`minCount=5 minScore=0.1`, no region, no role) `NeighborGraphResponse` and stores it via `atomic.Pointer`. - `handleNeighborGraph` short-circuits on the default shape; non-default filters take the extracted `computeNeighborGraphResponse` path (identical semantics to the previous inline build). ### P0-2: cache parsed `StoreObs.Timestamp` + drop RLock window - `handleObserverAnalytics` re-parsed the RFC3339 timestamp three times per observation, for 60k+ observations per active observer, under `s.store.mu.RLock` — blocking writers for the full scan. - `StoreObs.ParsedTime()` parses once via `sync.Once` (mirrors `StoreTx.ParsedDecoded`). - Handler snapshots the `byObserver[id]` pointer slice, releases the RLock immediately, then iterates locally. ### P0-3: 30s cache for `/api/observers` + sargable `IN` + covering index - Three SQL queries on every request → ~1.7s p50 at 50-concurrent. - Atomic-pointer 30s cache for the default (no-filter) query. - `GetNodeLocationsByKeys` drops `LOWER(public_key) IN (...)` (non-sargable); callers pre-lowercase in Go and the plain `IN` matches the existing `public_key` index. - New ingestor migration `obs_observer_ts_idx_v1` adds composite index `idx_observations_observer_idx_timestamp(observer_idx, timestamp)` so `GetObserverPacketCounts` can resolve its GROUP-BY + range filter from the index without scanning the 1.9M-row observations table. ### P0-4: deferred `perfMiddleware`'s global mutex was claimed to serialize every API request. A direct test (`50 concurrent requests through the middleware, handler sleeps 20ms each`) shows total elapsed ≈ 25ms, not 1s — the lock is held only for the post-handler bookkeeping (a few µs). Real impact is below measurement noise. Skipping to avoid invasive churn on PerfStats consumers without a demonstrable win. ## Test plan Red → green per P0: - `observers_cache_test.go` — handler reads `s.observersCache` before SQL, TTL boundary, atomic.Pointer (no mutex contention). - `storeobs_parsedtime_test.go` — parses three timestamp shapes, caches result, no race under concurrent readers. - `neighbor_graph_cache_test.go` — handler serves from atomic pointer when set, bypasses cache when `?region=` (or any non-default filter) is passed. Full server + ingestor suites pass: `go test -count=1 ./...`. ## Perf proof Before/after p50/p95/p99 (50 requests × 50 concurrent) against prod (before) and staging once CI deploys (after) will be posted as a PR comment per the operator's "no merge without proof of improvement" gate. Closes #1481 ## TDD exemption — P0-1 and P0-2 (net-new surfaces, AGENTS.md) Per CoreScope `AGENTS.md` § "Exemptions": **net-new code surfaces with no prior tests to break** may land tests in the same PR without a strict test-first → impl commit split. - **P0-1 (neighbor-graph atomic-pointer cache)** — `neighborGraphCache`, `recomputeNeighborGraphCache`, `loadNeighborGraphCacheBytes`, `startNeighborGraphRecomputer` and the default-shape short-circuit in `handleNeighborGraph` were brand-new code with no pre-existing assertions covering them. There was no green test to first turn red. - **P0-2 (cached `StoreObs.Timestamp` + RLock window drop)** — `StoreObs.ParsedTime()` and the snapshot+release pattern in `handleObserverAnalytics` were new surfaces; the prior code did the parse inline per call with no behavioural test to break. P0-3 was authored properly red-then-green (commit `6e63ec6a` red, then `83ae129b` green) and does NOT use this exemption. ## Default-filter detection vs frontend reality (#1483 follow-up) The Neighbor Graph analytics tab in `public/analytics.js` fetches `/analytics/neighbor-graph?min_count=1&min_score=0` because the client-side sliders need the full edge set to filter from. That shape did NOT match the `(5, 0.1)` cached default, so the UI tab still paid the cold compute cost despite #1481 P0-1. The #1483 follow-up commit caches BOTH shapes in the same recomputer pass: - `(minCount=5, minScore=0.1, no region, no role)` — `live.js` affinity-scoring consumer. - `(minCount=1, minScore=0, no region, no role)` — analytics tab. Both are served from `atomic.Pointer` with an `X-Cache-Age-Seconds` header. The per-shape cost in the background goroutine is roughly linear in edge count; total recompute time stays well under the 5-minute cadence on prod-scale graphs. --------- Co-authored-by: openclaw-bot <bot@openclaw.dev> Co-authored-by: mc-bot <mc-bot@users.noreply.github.com>	2026-05-29 02:42:21 -07:00
Kpa-clawbotandopenclaw	43b93c6bb9	feat(observers): surface naive-clock observers as ⚠️ chip + detail banner (#1478) (#1480) ... ## Summary Issue #1478 — surface observers whose envelope timestamps are being clamped because they're emitting zone-less local-time strings (UTC-N observers showed up perpetually as "Stale" before #1466, and per-packet rxTime is still clamped to ingest time for them, muddying propagation-delay analytics). Now the UI tells operators which observers are misconfigured + how to fix it. ## What changed ### Ingestor (cmd/ingestor) - New `observers_clock_naive_v1` migration adds three columns to `observers`: - `clock_skew_seconds INTEGER` (signed: negative = behind UTC, positive = ahead) - `clock_skew_count_24h INTEGER` (rolling 24h event count) - `clock_last_naive_at TEXT` (RFC3339 timestamp of last clamp) - `resolveRxTime` now returns `(rxTime, naiveSkewSec)`. The packet-handler call site invokes `store.RecordNaiveSkew(observerID, deltaSec)` whenever a naive envelope is clamped (the existing >15 min naive-tolerance path). The counter resets to 1 if no event in the prior 24h, else increments. Single INSERT-or-UPDATE round trip per clamp. ### Server (cmd/server) - `Observer` struct + `GetObservers` / `GetObserverByID` extended to scan the three new columns. - `ObserverResp` gains four JSON fields exposed by `/api/observers` and `/api/observers/{id}`: - `clock_naive` (bool, derived from `clock_last_naive_at` being within 24h) - `clock_skew_seconds`, `clock_skew_count_24h`, `clock_last_naive_at` - Decay is **read-side**: a stale event yields `clock_naive=false` with zero counts. No background sweep, no writes from the read-only server, no race with the ingestor. ### Frontend (public) - `window.ObserversNaiveChip.render(o)` — total render helper, returns ⚠️ chip HTML when `o.clock_naive===true`, `""` otherwise. Used inline in the observers-list `name` cell and in the row-detail slide-over. Tooltip explains magnitude + direction + count + fix. - `window.ObserverDetailNaiveBanner.render(obs)` — yellow alert banner at the top of the observer-detail page with the skew magnitude, last-event timestamp, and the actionable fix ("Set host clock to UTC, OR emit Z-suffixed/offset-aware timestamps from the observer script"). ## TDD trail - `5ddd5b42` red: backend `cmd/server/observer_naive_clock_1478_test.go` (3 tests asserting JSON fields + 24h decay) + frontend `test-observer-naive-clock-1478.js` (8 jsdom-style tests asserting helpers exist and render correctly). Both failed on master with field-missing / export-missing assertions. - `4ecc79c8` green backend: schema + Observer / GetObservers / ObserverResp / handler decay. - `2137ab81` green frontend: chip + banner helpers and call sites. ## Tests - `cd cmd/server && go test ./...` → all green (full suite, 46s) - `cd cmd/ingestor && go test ./...` → all green (full suite, 98s) - `node test-observer-naive-clock-1478.js` → 8/8 pass - `node test-frontend-helpers.js` → unchanged from master (pre-existing failures only) ## Acceptance (issue #1478) - ✅ Observer running with `python datetime.now().isoformat()` (naive, off by N hours) → `clock_naive=true` after the next clamp → UI shows ⚠️ chip + banner. - ✅ Observer with `datetime.now(timezone.utc).isoformat()` (Z-suffixed) → never clamped → never flagged. - ✅ Observer that fixed its clock → `clock_naive` returns to `false` 24h after the last clamp event (read-side decay). Closes #1478. --------- Co-authored-by: openclaw <bot@openclaw.local>	2026-05-29 01:08:12 -07:00
Kpa-clawbotandopenclaw-bot	0a58aa146a	fix(ingestor): silence per-message naive-timestamp log (#1478 followup) (#1479) ... Operator on prod reports the per-message naive-timestamp warning drowns the log when an observer's local clock isn't UTC. Since observer.last_seen already uses ingest time regardless of envelope (#1466), and per-packet rxTime is already clamped (#1464), the per-message console log adds nothing actionable. This PR silences the log. #1478 tracks the proper followup: surface broken observers in the UI (chip + banner on observer detail). Backend-only, hot-deployable via image pull (no API/schema change). Co-authored-by: openclaw-bot <bot@openclaw.local>	2026-05-29 06:27:58 +00:00
efitenandClaude Sonnet 4.6	196f1c6720	fix(ingestor): don't stamp timestamp in procIO snapshot on os.Open failure (#1428) ... ## Summary - `readProcSelfIO()` stamped `at=time.Now()` before attempting to open `/proc/self/io` - On non-Linux hosts or when the kernel file is unavailable, it returned a snapshot with `ok=false` but a fresh timestamp - The rate calculator used `prevIO.at` for delta computation, so the next successful read produced a phantom rate spike spanning the entire failure interval - Fix: move the timestamp stamp to after successful `os.Open`, so failed opens return a zero-value snapshot with no timestamp — `procIORate` short-circuits on `prev.ok=false` and returns nil ## Test plan - [ ] `go test ./...` in `cmd/ingestor` — both new unit tests pass: - `TestProcIORate_ZeroValuePrevSuppressesRate` — asserts nil rate when prev is zero-value - `TestProcIORate_NormalPath` — asserts correct rate for valid prev/cur pair - [ ] On Linux: confirm `procIO` block still appears in the stats file after 2 ticks Closes #1169 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-28 22:50:23 -07:00