3 Commits

Include renames

Author	SHA1	Message	Date
efitenandClaude Opus 5.5	f3004d977e	feat(rx-coverage): show where a companion drove and received nothing (#2172) ... The Mobile RX coverage page (`#/rx-coverage`) shows only cells where a companion received something. Where a companion drove and received nothing is just as useful for operators, and the data is already there: the RF sample track (`client_rf_samples`, #1905/#1906) and the receptions (`client_receptions`) use the same hex grid (`hexCellAt`, `zoomToHexRes`). This PR adds those cells as a hatched layer under the reception cells. ## Changes in this PR **Server** - `GET /api/rx-coverage?...&gaps=1` adds a `gaps` member to the FeatureCollection: cells with moving (non-stationary) track samples and no reception in the same `days` window and `bbox`, each with `samples`. `gaps_truncated` is set when more than 5000 cells qualify (the most-sampled are kept, same pattern as `coverageFeatureCap` and `rfNoiseFeatureCap`). - Gaps are only returned when `clientRfSamples` is on and there is no `node` filter: with `?node=` the question would be "this node was not heard", which these cells do not answer. `?rx=` limits both the track and the receptions to that companion. - Without `gaps=1` the payload is unchanged: the `gaps` member is omitted. - New code lives in `cmd/server/rx_coverage_gaps.go` (pure `aggregateGaps` plus `queryTrackPoints`); the handler change in `rx_dashboard.go` is 14 lines. - `/api/rx-coverage` is now documented in `openapi.go` and removed from `openapi_known_gaps.json` (#1670). **Frontend** (`public/rx-coverage.js`, `public/node-reach-coverage.css`) - A "Nothing received" toggle on the signal layer, on by default, only rendered with `MC_CLIENT_RF_SAMPLES`. Hidden on the noise layer. - Gap cells use a 45-degree hatch (an SVG pattern in a dedicated renderer, so they sit under the reception cells). The hatch is denser for cells driven through more often (1 / 2-3 / 4+ samples). Tooltip: sample count. Legend entry: "driven, nothing received". - Off state is deep-linkable as `gaps=0`. - New token `--nq-cov-gap` for light and dark. Plain grey (`--nq-cov-grey`) already means "received, no SNR", so the hatch keeps the two apart. ## Threshold: one sample Measured on a live network (analyzer.on8ar.eu, Belgium, September to October 2026), with cells of about 250 m: a cell with track samples and no reception in one half of a month, driven again in the other half. How often did it receive something the second time? | Minimum samples | Cells "nothing received" | Driven again | Received something later | |---|---|---|---| | 1 | 3971 | 1263 | 29% | | 2 | 1565 | 592 | 32% | | 3 | 619 | 243 | 35% | | 5 | 142 | 63 | 40% | A higher threshold hides cells without making the rest more reliable, so the threshold is 1 and the wording is "nothing received in this period", never "no coverage". Using every decoded packet as reception evidence (including 1-byte path hashes) gave 25% to 32%, so the capture rule for 1-byte hashes is not what drives this. The median cell has 2 samples. ## Verification - Go: 7 new tests in `rx_coverage_gaps_test.go` (gap/no gap, stationary excluded, single sample, cap keeps the most-sampled, endpoint, `?rx=` filter, no `gaps` member when not asked / with `?node=` / with RF samples off). The endpoint tests fail without the handler change. Full `cmd/server` suite green locally, `go vet` and `gofmt` clean. - E2E: `tests/e2e/test-rx-coverage-gaps-e2e.js` (4 steps: no toggle with RF samples off, on by default with hatched cells under the reception cells, toggle off with `gaps=0` and deep link, noise layer hides it). 2 steps fail without the frontend change. Existing `test-rx-coverage-noise-e2e.js` and `test-rx-coverage-viewport-e2e.js` still pass. Listed in `scripts/non-unit-tests.json`. - Frontend unit suite (`test-all.sh`) green. - Real data on a staging instance (7 days, bbox 50.8,4.9,51.3,5.8, z=12): 478 reception cells and 379 gap cells, no cell in both, reception features identical with and without `gaps=1`, 0.48 s to 0.52 s per request both ways. - Browser: checked on staging with real data (light theme), and light, dark and 390 px wide with mocked data. ## Not in this PR - A per-node variant on the Reach page ("this repeater transmitted while a companion was nearby, and was not heard"). That needs the transmit moments from `observations` and is a heavier query. - `deploy.yml` does not get a line for the new E2E file, like the other entries that are only in `non-unit-tests.json` (#2037). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 17:39:33 +02:00
efitenandClaude Opus 5	00ee4f9e23	fix(coverage): attribute node-discover replies to the responder (#2111) ... ## Problem A CoreDrive RX companion that runs node-discover gets no coverage on an upstream instance. Reported today by an operator on v3.13.0: the app logged 32 discover replies from one repeater in 16 minutes (`heard 751a49f0c5dadc70 (8B, discover)`), all published, and `client_receptions` stayed at 0 rows for that companion while `client_rx_observations` held 93. Two gaps, both on the upstream side: 1. **Ingestor.** `deriveHeardKey` attributes a FLOOD `path[last]` and a 0-hop advert, and drops a 0-hop `CONTROL/DISCOVER_RESP` (firmware `CTL_TYPE_NODE_DISCOVER_RESP`). The reply carries the responder's own pubkey at offset 6 of the payload, which `decoder.go` already parses into `CtrlPubKey` (#1802), but the coverage path never used it. 2. **Server.** CoreDrive RX asks for `DISCOVER_PREFIX_ONLY`, so the firmware answers with an 8-byte prefix (`simple_repeater/MyMesh.cpp:799-805`). `coverageHeardKeyCandidates` only built the 64, 6 and 4 hex candidates, so a 16-hex `heard_key` would be stored and then matched by no per-node coverage query. ## Change - `cmd/ingestor/client_reception.go`: a third branch in `deriveHeardKey` for a discover response with no hops. Accepts exactly 8 or 32 bytes, nothing truncated, stored with `src='discover'`. - `cmd/server/rx_coverage.go`: adds the 16-hex prefix to `coverageHeardKeyCandidates`. - `docs/client-rx-coverage.md`: documents the `discover` source, the 8-byte keylen and the four-candidate lookup. The leaderboard and `/api/rx-coverage` read `client_receptions` without a key filter, so they pick the rows up without a change. Name resolution goes through `batchResolveHeardKeys`, which is a prefix lookup and handles 16 hex as is. ## Evidence from a deployment that has had this since 2026-08-19 On analyzer.on8ar.eu, `client_receptions` over the last 7 days by `src`: discover 4232, rxlog 4909, geo 548, advert 34. Discover replies are 44% of all coverage rows there (4232 of 9723); on an upstream instance those rows are not written. They cannot be backfilled afterwards either: `client_rx_observations` keeps no raw bytes, so the responder pubkey is gone. ## Tests - `TestDeriveHeardKey` and `TestBuildClientReception` gain discover cases: 8-byte and 32-byte keys accepted (32-byte uppercase input lowercased), a 3-byte and an empty key rejected, a non-discover CONTROL rejected, a discover response with hops not attributed. - `TestHandleClientPacketDiscoverRespWritesReception`: end to end, a raw 0-hop DISCOVER_RESP on the client topic writes one `client_receptions` row with `src='discover'`. - `TestCoverageHeardKeyCandidatesIncludesDiscoverPrefix`: the 16-hex prefix is among the per-node candidates. - Ran locally on Windows: `go test ./...` in `cmd/server` passes; in `cmd/ingestor` everything passes except `TestWriteStatsAtomic_SymlinkAtDestIsReplaced`, which needs the symlink privilege on Windows and fails on clean master too. Not done: no browser validation, as the change is ingestor and server only and the frontend reads the same endpoints. Not included: geographic resolution of 1-byte hops (`src='geo'`) and the RF noise layer, which are separate changes. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-10-04 17:15:50 +02:00
3 people	22fe929da2	feat: opt-in mobile client-RX coverage (crowdsourced RF reach) + /api/nodes/resolve (#1728) ... Implements #1727. ## What this adds **Mobile client-RX coverage** — an opt-in, crowdsourced RF-coverage feature. A roaming MeshCore **companion** radio (driven by the open-source [corescope-rx](https://github.com/efiten/corescope-rx) PWA, GPLv3) reports which nodes it heard directly, tagged with the phone's GPS and the packet's SNR/RSSI. CoreScope ingests these into a new `client_receptions` table and renders per-node **hex coverage** on the Reach page, plus a standalone **Coverage dashboard** (`#/rx-coverage`) with a top-mobile-observers leaderboard. Also includes **`GET /api/nodes/resolve?prefix=<hex>`** — a read-only node-name lookup by pubkey prefix (`{name, pubkey, ambiguous}`), used by the companion app for friendly names. ## Opt-in — default OFF (zero impact on existing deployments) The whole feature is gated behind one config flag, **disabled by default**: ```jsonc "clientRxCoverage": { "enabled": false } ``` When disabled (the default): the ingestor writes **no** `client_receptions`; the three coverage endpoints return a clean **404**; the UI hides the Coverage nav link, the `#/rx-coverage` route, and the Reach-page toggle. `/api/nodes/resolve` is always available (not coverage-specific). ## How it works ``` companion ──BLE 0x88 (snr+rssi+raw)──▶ corescope-rx PWA ──▶ MQTT meshcore/client/{pubkey}/packets │ ingestor (gated) ──▶ client_receptions (GPS + SNR + heard-key) │ server: pure-Go hex grid ──▶ GeoJSON ──▶ Reach hex overlay + Coverage dashboard ``` - **Direct-only capture:** records only what the companion heard itself and directly — a 0-hop advert's pubkey, or `path[last]` (last forwarder) for FLOOD routes; ≥2-byte path-hash required. Upstream hops discarded. - **No new deps:** hexbins are a pure-Go pointy-top grid over Web Mercator (`cmd/server/hexgrid.go`) computed at query time (`CGO_ENABLED=0` / `modernc.org/sqlite` friendly); frontend uses the existing Leaflet. - **Trust:** companion pubkey = identity; an EMQX ACL binds each client to publish only to its own `meshcore/client/{pubkey}/packets` topic. Payload contract in `docs/client-rx-coverage.md`. ## How to enable / try it 1. In `config.json`, set `"clientRxCoverage": { "enabled": true }` and restart server + ingestor. 2. Point an EMQX (or any broker) listener so a client can publish to `meshcore/client/<pubkey>/packets`; the ingestor already subscribes under `meshcore/#`. 3. Run the [corescope-rx](https://github.com/efiten/corescope-rx) PWA on an Android phone paired (BLE) to a MeshCore companion — it captures heard nodes + GPS and publishes. 4. View results: per-node Reach page → toggle **coverage**, or the **Coverage** dashboard at `#/rx-coverage`. ## What's where - **Ingestor:** `cmd/ingestor/client_reception.go` (ingest), `db.go` (`client_receptions` + `client_observers` schema), `main.go` (gated dispatch), `config.go` (flag). - **Server:** `cmd/server/rx_coverage.go` + `rx_dashboard.go` (endpoints, self-guard 404 when off), `hexgrid.go` (pure-Go grid), `node_resolve.go` (resolve), `routes.go` / `types.go` / `config.go` (wiring + flag + `/api/config/client` field). - **Frontend:** `public/rx-coverage.js` (dashboard), `node-reach-coverage.js` + `.css` (overlay), `node-reach.js` (Reach toggle, flag-gated), `roles.js` (reads the flag, hides nav when off). - **Docs:** `docs/client-rx-coverage.md`. ## Testing - Go: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test ./...` — green, including new gate tests (`coverage_gate_test.go` in both: off → no rows / 404, on → works) and the rx-coverage / resolve / hexgrid suites. - JS: `node test-coverage-gate.js`, `node test-node-reach-coverage.js` (wired into CI). The Playwright `test-node-reach-coverage-e2e.js` is wired into the e2e job and **skips when `clientRxCoverage` is disabled**, so it's safe under the default-off config. ## Notes for reviewers - The four new routes are registered in `cmd/server/openapi_known_gaps.json` (the existing OpenAPI-completeness ratchet), matching how other not-yet-spec'd routes are tracked. Happy to write full OpenAPI spec entries instead if you prefer. - Commits are split per layer (ingestor / server endpoints / resolve / frontend / CI) for review. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Erwin Fiten <e.fiten@opteco.be>	2026-06-19 11:37:16 -07:00