4 Commits

Include renames

Author	SHA1	Message	Date
efiten	e01565737a	feat(ingestor): store CoreDrive RX region answers, with position, clock and retention (#2047) ... The Scope Audit page said a repeater's declared region list can come from CoreDrive RX while nothing the app sends ever reached it: the client-topic switch handled packets and rf only, so /regions was dropped without a log line, and node_declared_regions is read by region_keys.go and config.go but created by nothing in this tree. #2044 found that gap and proved it against a live instance. This lands the implementation that has been carrying the feature in production on the ON8AR fork since 2026-09-06, the instance CoreDrive RX publishes to. Measured there: 1840 answers about 275 repeaters from 51 collectors, 2026-08-18 to 2026-09-19. Beyond storing the answer it keeps three things the first version did not: position (lat, lon, pos_acc_m, filled on 1263 of 1840 rows, with acc_m dropped when the fix it qualifies was rejected), repeater_clock (filled on all 1840, so a wrong repeater clock cannot make an answer look newer than it is), and retention with per-collector history (pruneOldClientDeclaredRegionsAt bounds by age instead of keeping one row per target). On that dataset 138 of 275 repeaters have answers from more than one collector and 40 have collectors that disagree about the region list, which is the signal the Scope Audit exists to surface and which only survives while more than one answer does. It gates on its own clientRegions block rather than riding on clientRxCoverage, so region answers can be accepted without GPS-tagged reception uploads, and AES block padding is trimmed from region names on ingest. Taken from #2044 with the author credited as co-author: declaredRegionsTablePresent() and its test, a real bug this version lacked (supervisord starts both processes together, so a server that probes first ignores every answer until its next restart), and the docs/client-rx-coverage.md section. Ported by cherry-picking the fork's nine commits rather than retyping, so this is the code that has been running. CI run 35434151026 is green: server ok 80.177s, ingestor ok 99.413s, race detector ok 112.406s, no --- FAIL lines. Merged by the interim maintainer without a second human reviewer: CI and the production figures above are the independent checks.	2026-09-19 11:44:47 +02:00
Sylvain Rabot	a2ea18f778	perf(sqlite): swap modernc.org/sqlite for mattn/go-sqlite3, cross-built with zig (#1992) ... Swaps the SQLite driver from `modernc.org/sqlite` (pure Go, SQLite 3.46.0) to `github.com/mattn/go-sqlite3` (cgo, bundled SQLite 3.53.4), and pays the resulting cross-compilation cost with `zig cc`. Draft because the riskiest part of this deletes rows — see [Please review this part first](#please-review-this-part-first) — and because three things remain unverified at the bottom. `modernc.org/sqlite` is a transpilation of the C amalgamation. This repo is read-heavy: `cmd/server` chunk-loads a graph at startup and fans out neighbour/topology/analytics queries per request, and it pays for that transpilation on exactly those paths. Head-to-head on the same 120k-transmission / 240k-observation database, running our own hot-path SQL under both drivers (Apple M4, `-count=5`, medians): | workload | modernc | mattn | | |---|---:|---:|---| | chunk load (`chunked_load.go` v3 join, 20k tx) | 449ms | 196ms | **2.3×** | | aggregate scan (240k-row join + `GROUP BY`) | 276ms | 137ms | **2.0×** | | 1500 prepared-statement lookups | 512ms | 403ms | **1.3×** | Allocations fall with it: 1.12M vs 1.64M allocs and 21MB vs 30MB on the chunk load. **Superseded by a production run.** @efiten measured both drivers on a real instance — 11,077,038 observations, 9.7GB database, 4-core arm64 — as server-only containers against the same live volume, one at a time, with round 2 reversing the order so the page cache favours the old driver: | | audit 7d | audit 24h | background fill (13 chunks) | start → /api/health | |---|---:|---:|---:|---:| | modernc, round 1 | 16.67s | 2.27s | 130.2s | 16.6s | | mattn, round 1 | 7.87s | 1.34s | 93.8s | 13.5s | | mattn, round 2 | 8.15s | 1.35s | 96.4s | 13.0s | | modernc, round 2 | 13.46s | 2.29s | 137.8s | 15.5s | Warm, the old driver improves to 13.46s on the 7d audit and still loses by ~1.8×. Chunk load is ~1.4×. `/api/nodes?limit=500` is 0.039s against 0.037s — nothing. **So the real gain is ~1.4–1.8× on the paths that matter, not 2–2.3×.** The shape the harness predicted holds — scans and joins gain, small lookups do not — which is more reassuring than the magnitude would have been. Quote these numbers. **The counterweight**, cold and native on that machine: a build goes from **52s to 163s**. An instance that builds its own image pays that per deploy. ## The build is cgo now, and one thing about that is a trap **`CGO_ENABLED=0` still builds.** mattn links a stub, and the binary dies on its first query with `go-sqlite3 requires cgo to work. This is a stub`. A green build is not evidence of anything here, which is why `AGENTS.md` now says so explicitly. `GOOS=linux go build` genuinely cannot cross-compile any more. A new root `Makefile` is the entry point. `make crossbuild` uses `zig cc -target {x86_64,aarch64}-linux-musl` and links static, so each artifact stays a single self-contained file and the `alpine:3.20` runtime no longer depends on the base image's libc at all. `-Wl,-s` is load-bearing: Go's own `-s -w` does not reach the musl objects zig links in, and without it the server binary is 19.8MB instead of 12.1MB. The Dockerfile keeps its single `$BUILDPLATFORM` builder — still no QEMU for compilation — and gains a checksum-pinned zig plus BuildKit cache mounts. The mounts are not a nicety: without them an image build recompiles the amalgamation from cold and takes over half an hour. ## Please review this part first `internal/dbschema/dedup_index.go` **deletes observation rows**. It is the one part of this change that can lose data, and it exists because the migration exposed a real bug rather than causing one. `stmtInsertObservation` resolves its `ON CONFLICT` against `idx_observations_dedup`, which `cmd/ingestor/db.go` only ever created inside the branch that creates the `observations` table for the first time. Any database whose table predates that branch never got one, so the UPSERT had no conflict target. modernc failed on the first insert; mattn fails at `OpenStore`. Same bug, found earlier. Creating the index unconditionally repairs it — but the index is what was supposed to prevent duplicates, so a database that never had it can already hold rows violating it. **`test-fixtures/e2e-fixture.db` in this repo holds one.** So duplicates are collapsed first. Refusing is not the safer option: without the index the ingestor cannot prepare its UPSERT, so it cannot start at all. Replaying that UPSERT faithfully is subtler than it looks, and a first cut of this got it wrong twice: - `COALESCE(excluded.x, x)` means the **incoming** value wins, so down a group in id order the survivor keeps the **last** non-NULL value. Taking the first silently discarded newer readings. - The UPSERT names exactly five columns (`snr`, `rssi`, `score`, `raw_hex`, `resolved_path`). Every other column must keep the surviving row's own value; merging those too invents history the ingestor would never have written. Merge, delete and `CREATE UNIQUE INDEX` now share one transaction. Split apart, a writer inserting a duplicate in the gap fails the index creation while leaving the deletions committed — rows destroyed and no index to show for it. Cost, measured on 2.4M synthetic rows holding 5 duplicates: **4.1s**, holding the write lock throughout, once, at ingestor startup before MQTT subscribe. Materialising the duplicate-group scan once rather than per column took that from 9.7s; the pathological case (400k of 600k rows duplicated) is 5.7s, slightly worse than the 4.2s it was before that change. ## Four more behavioural differences Full detail in `docs/sqlite-driver-migration.md`. Briefly: **Statement preparation is eager.** modernc's `newStmt` stored the SQL and compiled lazily; mattn calls `sqlite3_prepare_v2` inside `Prepare`, so SQL naming a missing table fails at *open*. 59 server tests failed on this alone, all fixtures with partial schemas. `OpenDB` keeps failing loudly (#1901; `main.go` gates on `dbschema.AssertReady` anyway) and the fixtures now declare what they are prepared against via `ensurePreparable`. This also exposed nine `nodes(pubkey …)` declarations across seven files, where production has only ever had `public_key` — lazy compilation had hidden the mismatch for as long as it existed. **`synchronous` silently dropped FULL → NORMAL.** mattn defaults it to NORMAL and executes the pragma unconditionally, where SQLite's own default (what modernc left alone) is FULL. In WAL mode that weakens durability under power loss. Pinned in `dbschema.WriterDSN`, which both writers now share — `cmd/migrate` kept a bare path at first and so quietly wrote at NORMAL, which is what a second copy of a DSN buys you. **The DSN dialects are mutually invisible.** modernc understood only `_pragma=name(value)`, mattn only `_`-prefixed parameters, and neither errors on the other's form — a driver-only rename would have dropped every pragma in silence. `_journal_mode=WAL` is also gone from the server's read handle: modernc ignored it, mattn honours it, and setting `journal_mode` on a read-only connection is a write. Dropping `_busy_timeout` with it costs nothing, since mattn already defaults to 5000ms — which means the read handle finally *gets* the busy timeout it had silently lacked. **`mode=ro` survives for a non-obvious reason.** mattn always passes `READWRITE|CREATE` and its amalgamation has `SQLITE_USE_URI=0`; what makes the URI work is its C wrapper ORing `SQLITE_OPEN_URI` in. So the #1283/#1289 invariant holds with no build flags — but it depends on the `file:` prefix. `cmd/decrypt` had been building its DSN without one, so its `mode=ro` had never applied and a missing path was created read-write. Fixed in passing; never a migration regression. ## What did not change No modernc-specific API was in use: no `RegisterFunction`, no `*sqlite.Conn`, no `sqlite/lib` error constants, no `sql.Register`. No `time.Time` is ever bound as a query argument, so driver time handling is not in play. Both drivers convert declared `DATE`/`DATETIME`/`TIMESTAMP` columns to `time.Time`, so `/api/dropped-packets` keeps emitting `dropped_at` as RFC3339 — an earlier draft "fixed" that with a `CAST` and would have been the regression. ## Tests and CI New regression tests, each written because something got through without it: - `TestEnsureObservationsDedupIndexKeepsLatestValues` — the merge ordering. The original test used complementary NULLs, which passes whichever direction you pick, which is why the bug survived it. - `TestCollapseDuplicatesAndIndexIsAtomic` — a failed index creation must roll the deletions back. - `TestOpenStorePragmas` / `TestWriterDSNPragmas` — every writer pragma, read back through the store's own connection. A separate `sqlite3` session or the startup log line would prove nothing. - `TestOpenDBRefusesMissingDatabase` — the read-only invariant, which now rests on a detail of the driver's C wrapper. - `TestEnsurePreparableMatchesPrepareStatements` — fails when a new prepared statement outgrows the fixture helper. CI gains test execution for `cmd/migrate` and `internal/dbschema`, which had none and both open the database. A PR-time two-arch build plus an arm64 QEMU smoke gate is new: the GHCR push is push/tag-only, so without it nothing on a PR would exercise zig, static musl linking or arm64, and the first signal would arrive on master. `cache-dependency-path` widens from 2 of the 5 tracked `go.sum` files to all of them. `make test` passes across all 14 modules, `cmd/server` also under `-race -count=2` with no failures and no races. `gofmt` and `go vet` clean. Release-routing and Dockerfile COPY-invariant gates pass. ## Verified by running - All 8 cross-builds static and correct-architecture; both arches of the container image built, exported and run under QEMU, serving `/api/health` and `/api/nodes` against a 2.9M-observation production snapshot. - The `migrate` binary repairing that snapshot's duplicate on bare Alpine. - `CGO_ENABLED=0` producing a binary that builds and then fails on first query. ## Not verified - ~~The 2–2.3× figures come from a standalone harness, not this load under the old driver.~~ **Closed** by @efiten's production run above, which also corrected the multiplier. - SQLite 3.46.0 → 3.53.4 query-planner differences on queries with no total `ORDER BY`. - Sustained live ingest through the new writer DSN, and the duplicate collapse against a database an ingestor is actively writing to. Verified against a static snapshot only, and the collapse is measured at 4.1s on 2.4M synthetic rows with 5 duplicates — well short of an 11M-row instance. @efiten has offered a staging instance taking real MQTT traffic; **this is the item to close before the PR leaves draft.** An earlier revision of this branch shipped the dedup merge in the wrong direction with a green test suite, and review then found three more things in the same file: the repair gated on an error string, a non-atomic TEMP table drop aimed at the wrong connection, and a deletion whose only record was a row count. All fixed in ac7e8d38. Passing tests did not establish safety here, which is why the deletion path wanted a second pair of eyes rather than a rubber stamp.	2026-09-16 09:02:13 +02:00
efitenandClaude Opus 5	9e13e0b05f	feat(ingestor): full-packet RF observations from mobile clients (#1905) ... ## What A CoreDrive RX drive already carries far more RF information than reaches CoreScope, and it was being discarded twice: once in the mobile app (every packet it could not attribute to a directly-heard node was dropped before queueing) and once here (the ingestor decodes the *complete* packet, then keeps only `heard_key`/`snr`/`rssi`/`lat`/`lon`). This captures what was being thrown away, at **zero extra airtime** — nothing new is transmitted. - **`transmissions.code1` / `code2`** — the transport codes were decoded on every packet and used only to derive `scope_name`, then dropped. Storing them turns "which repeater forwards which scope" from a re-parse into a query. - **An async backfill** re-parses the `raw_hex` already on disk, so months of scope history become queryable with no new data collection. - **`client_rx_observations`** — a new diagnostic table holding every decodable packet a phone heard, with route type, transport codes, scope name, path-hash size, the full forwarder chain and the forwarder. ## Why it is safe for existing deployments Both halves are **opt-in and default off** (`clientRxObservations.enabled`, and `fullRfLog` on the app side), so an existing deployment sees no behaviour change and no volume change on upgrade. The coverage invariant is untouched: `client_receptions` keeps its rule — 0-hop advert pubkey or FLOOD `path[last]`, ≥2-byte hash — and an unattributable packet writes **zero** coverage rows. `deriveHeardKey`, `buildClientReception` and `InsertClientReception` are unmodified except for one guard described below. ## Performance justification (touches the ingest hot path) - **Backfill:** keyset-paginated by `id` in 5000-row batches, a single forward scan, `rows.Close()` before `Begin()` so it never deadlocks against `SetMaxOpenConns(1)`, and commits per batch so live ingest interleaves. Termination is driven by rows *scanned*, not rows decoded — an earlier count-based loop would have stopped at the first batch containing an undecodable row and then written its completion guard, permanently stranding the rest. - **Guard row is written if and only if the loop ran to genuine exhaustion.** Every error path leaves it unwritten so the next startup retries. - **Per-packet cost:** one extra INSERT on the client topic when enabled, gated behind an opt-in flag. No new work on the observer path. - **New indexes** cover the prune (`rx_at`), the flood-grouping (`pkt_hash, rx_at`), the per-repeater query (`forwarder, rx_at`) and the scope query (`scope_name, rx_at`). Retention has its own shorter window — this table is diagnostic, not archival. ## Two firmware-derived correctness points - **`pkt_hash` is `ComputeContentHash()`**, byte-identical to `transmissions.hash`, so dark-traffic queries are a plain equality join rather than a translation layer. - **TRACE packets are refused.** TRACE repurposes the header path bytes as per-hop SNR values, so deriving a `heard_key` from them invents a node that never existed. `packetpath.PathBytesAreHops` existed but was never wired into the client path; it became reachable only because the app half now publishes packets it previously dropped locally. ## Testing Full ingestor suite green. Notable coverage: a FLOOD-routed TRACE writes zero coverage rows and NULL `forwarder`; a `direction: "tx"` message writes no observation; a DIRECT route never sets `forwarder`; two forwarder copies of one flood remain two rows; the backfill's multi-batch path is exercised with an undecodable row in the first page; and a forced error asserts the migration guard stays unwritten. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 18:35:20 +02:00
3 people	22fe929da2	feat: opt-in mobile client-RX coverage (crowdsourced RF reach) + /api/nodes/resolve (#1728) ... Implements #1727. ## What this adds **Mobile client-RX coverage** — an opt-in, crowdsourced RF-coverage feature. A roaming MeshCore **companion** radio (driven by the open-source [corescope-rx](https://github.com/efiten/corescope-rx) PWA, GPLv3) reports which nodes it heard directly, tagged with the phone's GPS and the packet's SNR/RSSI. CoreScope ingests these into a new `client_receptions` table and renders per-node **hex coverage** on the Reach page, plus a standalone **Coverage dashboard** (`#/rx-coverage`) with a top-mobile-observers leaderboard. Also includes **`GET /api/nodes/resolve?prefix=<hex>`** — a read-only node-name lookup by pubkey prefix (`{name, pubkey, ambiguous}`), used by the companion app for friendly names. ## Opt-in — default OFF (zero impact on existing deployments) The whole feature is gated behind one config flag, **disabled by default**: ```jsonc "clientRxCoverage": { "enabled": false } ``` When disabled (the default): the ingestor writes **no** `client_receptions`; the three coverage endpoints return a clean **404**; the UI hides the Coverage nav link, the `#/rx-coverage` route, and the Reach-page toggle. `/api/nodes/resolve` is always available (not coverage-specific). ## How it works ``` companion ──BLE 0x88 (snr+rssi+raw)──▶ corescope-rx PWA ──▶ MQTT meshcore/client/{pubkey}/packets │ ingestor (gated) ──▶ client_receptions (GPS + SNR + heard-key) │ server: pure-Go hex grid ──▶ GeoJSON ──▶ Reach hex overlay + Coverage dashboard ``` - **Direct-only capture:** records only what the companion heard itself and directly — a 0-hop advert's pubkey, or `path[last]` (last forwarder) for FLOOD routes; ≥2-byte path-hash required. Upstream hops discarded. - **No new deps:** hexbins are a pure-Go pointy-top grid over Web Mercator (`cmd/server/hexgrid.go`) computed at query time (`CGO_ENABLED=0` / `modernc.org/sqlite` friendly); frontend uses the existing Leaflet. - **Trust:** companion pubkey = identity; an EMQX ACL binds each client to publish only to its own `meshcore/client/{pubkey}/packets` topic. Payload contract in `docs/client-rx-coverage.md`. ## How to enable / try it 1. In `config.json`, set `"clientRxCoverage": { "enabled": true }` and restart server + ingestor. 2. Point an EMQX (or any broker) listener so a client can publish to `meshcore/client/<pubkey>/packets`; the ingestor already subscribes under `meshcore/#`. 3. Run the [corescope-rx](https://github.com/efiten/corescope-rx) PWA on an Android phone paired (BLE) to a MeshCore companion — it captures heard nodes + GPS and publishes. 4. View results: per-node Reach page → toggle **coverage**, or the **Coverage** dashboard at `#/rx-coverage`. ## What's where - **Ingestor:** `cmd/ingestor/client_reception.go` (ingest), `db.go` (`client_receptions` + `client_observers` schema), `main.go` (gated dispatch), `config.go` (flag). - **Server:** `cmd/server/rx_coverage.go` + `rx_dashboard.go` (endpoints, self-guard 404 when off), `hexgrid.go` (pure-Go grid), `node_resolve.go` (resolve), `routes.go` / `types.go` / `config.go` (wiring + flag + `/api/config/client` field). - **Frontend:** `public/rx-coverage.js` (dashboard), `node-reach-coverage.js` + `.css` (overlay), `node-reach.js` (Reach toggle, flag-gated), `roles.js` (reads the flag, hides nav when off). - **Docs:** `docs/client-rx-coverage.md`. ## Testing - Go: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test ./...` — green, including new gate tests (`coverage_gate_test.go` in both: off → no rows / 404, on → works) and the rx-coverage / resolve / hexgrid suites. - JS: `node test-coverage-gate.js`, `node test-node-reach-coverage.js` (wired into CI). The Playwright `test-node-reach-coverage-e2e.js` is wired into the e2e job and **skips when `clientRxCoverage` is disabled**, so it's safe under the default-off config. ## Notes for reviewers - The four new routes are registered in `cmd/server/openapi_known_gaps.json` (the existing OpenAPI-completeness ratchet), matching how other not-yet-spec'd routes are tracked. Happy to write full OpenAPI spec entries instead if you prefer. - Commits are split per layer (ingestor / server endpoints / resolve / frontend / CI) for review. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Erwin Fiten <e.fiten@opteco.be>	2026-06-19 11:37:16 -07:00