mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-11-11 18:57:45 +00:00
d083f9ade91dbcbc7aa577bbf7b2248ce21ebf09
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 n30nexandn30nex
|
f8c3b32633 |
docs(api): describe readiness responses and ingestor liveness (#2358)
Partial follow-up for #1670. Documents public `GET /api/healthz`: 503/loading, current in-memory counts, the fixed legacy backfill compatibility object, and optional ingestor liveness timestamps. Receipt and handler-entry clocks have different reset semantics; handler entry does not prove a completed database write. The legacy object does not track real migration progress. The 200 response has a typed schema, including its dynamic source-name map. Only `/api/healthz` leaves the known-gap list (16 → 15). Handlers, configuration and frontend behavior are unchanged; formal non-200 response-schema work remains separate. Validation: - The new documentation assertion failed before metadata was added; actual handler-response checks passed. - Full native server/ingestor suites passed in 257.438 s / 452.613 s. - After review corrections, focused checks passed in 2.757 s and the parent rerun in 2.097 s, including exact compatibility values and retained-receipt/zero-handler timestamps. - Independent adversarial, correctness and TDD reviews completed; the parent verified both semantic corrections against actual call sites. - [Exact-head CI passed](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40979351261), including browser shards. TDD exemption: documentation/API metadata only. Local standalone browser smoke was blocked; no local-browser or deployment claim is made. Production untyped-map count is unchanged. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
n30nexandn30nex
|
ef17d78909 |
docs(api): document node resolution and battery history (#2338)
Partial follow-up to #1670; the broader documentation issue stays open. Documents the existing node-prefix resolution and node battery-history GET APIs: parameters, normalization and lookup rules, defaults, ambiguity/visibility, ordered samples, nullable latest values and error media types. The generator now has concrete success schemas for both. Error statuses remain documented in prose under its existing convention. Removes exactly these two entries from the known-gap list: 18 → 16, with 115 of 131 routes described. No handler, query, configuration or dependency changes. [RED CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40426035729/job/121931981058) reaches the missing-documentation assertions. GREEN `521fb3ef` passes both full native CGO suites and focused OpenAPI/real-handler contract tests. Parent browser acceptance passed 28 checks through the rendered Swagger UI and actual endpoints, including defaults, errors and empty/null history. All three independent review roles passed. [Final CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40427081025) passes on this exact head, including all browser shards and the two-architecture image check. --------- Co-authored-by: n30nex <6686206+n30nex@users.noreply.github.com> |
||
|
n30nex
|
041daecd1e |
docs(api): describe performance and clock diagnostics (#2303)
Partial follow-up to #1670. Six existing read-only diagnostics endpoints lacked usable OpenAPI descriptions: process I/O, SQLite metrics, ingestor write counters, and fleet/node/observer clock skew. This documents their parameters, empty/error behavior and response root shapes. The known-gap list falls from 24 to 18. Descriptions preserve current behavior, including unknown/empty area fallback, the PacketStore cache ratio, stale write-counter snapshots, and the single-node 404. Schemas remain minimal; the existing generator still declares only the generic 200 response, with 404 explained in prose. No runtime handler, authentication, configuration or customizer changes. No added generic response maps. Validation: regression tests exercise the served spec and real router, including missing/malformed data, area selection and fallback, and calibration responses. Full server and ingestor tests pass locally; post-push Chromium verifies the served spec, six routes and Swagger rendering. Three independent reviews passed. Test-only RED `218a3bf3` produced the [six expected documentation failures](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212159661/job/121225534416). GREEN `1e2d8c08` adds the documentation and [passes CI](https://github.com/Kpa-clawbot/CoreScope/actions/runs/40212678062), including all browser shards and architecture builds. |
||
|
efitenandClaude Opus 5.5
|
f3004d977e |
feat(rx-coverage): show where a companion drove and received nothing (#2172)
The Mobile RX coverage page (`#/rx-coverage`) shows only cells where a companion received something. Where a companion drove and received nothing is just as useful for operators, and the data is already there: the RF sample track (`client_rf_samples`, #1905/#1906) and the receptions (`client_receptions`) use the same hex grid (`hexCellAt`, `zoomToHexRes`). This PR adds those cells as a hatched layer under the reception cells. ## Changes in this PR **Server** - `GET /api/rx-coverage?...&gaps=1` adds a `gaps` member to the FeatureCollection: cells with moving (non-stationary) track samples and no reception in the same `days` window and `bbox`, each with `samples`. `gaps_truncated` is set when more than 5000 cells qualify (the most-sampled are kept, same pattern as `coverageFeatureCap` and `rfNoiseFeatureCap`). - Gaps are only returned when `clientRfSamples` is on and there is no `node` filter: with `?node=` the question would be "this node was not heard", which these cells do not answer. `?rx=` limits both the track and the receptions to that companion. - Without `gaps=1` the payload is unchanged: the `gaps` member is omitted. - New code lives in `cmd/server/rx_coverage_gaps.go` (pure `aggregateGaps` plus `queryTrackPoints`); the handler change in `rx_dashboard.go` is 14 lines. - `/api/rx-coverage` is now documented in `openapi.go` and removed from `openapi_known_gaps.json` (#1670). **Frontend** (`public/rx-coverage.js`, `public/node-reach-coverage.css`) - A "Nothing received" toggle on the signal layer, on by default, only rendered with `MC_CLIENT_RF_SAMPLES`. Hidden on the noise layer. - Gap cells use a 45-degree hatch (an SVG pattern in a dedicated renderer, so they sit under the reception cells). The hatch is denser for cells driven through more often (1 / 2-3 / 4+ samples). Tooltip: sample count. Legend entry: "driven, nothing received". - Off state is deep-linkable as `gaps=0`. - New token `--nq-cov-gap` for light and dark. Plain grey (`--nq-cov-grey`) already means "received, no SNR", so the hatch keeps the two apart. ## Threshold: one sample Measured on a live network (analyzer.on8ar.eu, Belgium, September to October 2026), with cells of about 250 m: a cell with track samples and no reception in one half of a month, driven again in the other half. How often did it receive something the second time? | Minimum samples | Cells "nothing received" | Driven again | Received something later | |---|---|---|---| | 1 | 3971 | 1263 | 29% | | 2 | 1565 | 592 | 32% | | 3 | 619 | 243 | 35% | | 5 | 142 | 63 | 40% | A higher threshold hides cells without making the rest more reliable, so the threshold is 1 and the wording is "nothing received in this period", never "no coverage". Using every decoded packet as reception evidence (including 1-byte path hashes) gave 25% to 32%, so the capture rule for 1-byte hashes is not what drives this. The median cell has 2 samples. ## Verification - Go: 7 new tests in `rx_coverage_gaps_test.go` (gap/no gap, stationary excluded, single sample, cap keeps the most-sampled, endpoint, `?rx=` filter, no `gaps` member when not asked / with `?node=` / with RF samples off). The endpoint tests fail without the handler change. Full `cmd/server` suite green locally, `go vet` and `gofmt` clean. - E2E: `tests/e2e/test-rx-coverage-gaps-e2e.js` (4 steps: no toggle with RF samples off, on by default with hatched cells under the reception cells, toggle off with `gaps=0` and deep link, noise layer hides it). 2 steps fail without the frontend change. Existing `test-rx-coverage-noise-e2e.js` and `test-rx-coverage-viewport-e2e.js` still pass. Listed in `scripts/non-unit-tests.json`. - Frontend unit suite (`test-all.sh`) green. - Real data on a staging instance (7 days, bbox 50.8,4.9,51.3,5.8, z=12): 478 reception cells and 379 gap cells, no cell in both, reception features identical with and without `gaps=1`, 0.48 s to 0.52 s per request both ways. - Browser: checked on staging with real data (light theme), and light, dark and 390 px wide with mocked data. ## Not in this PR - A per-node variant on the Reach page ("this repeater transmitted while a companion was nearby, and was not heard"). That needs the transmit moments from `observations` and is a heavier query. - `deploy.yml` does not get a line for the new E2E file, like the other entries that are only in `non-unit-tests.json` (#2037). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
22fe929da2 |
feat: opt-in mobile client-RX coverage (crowdsourced RF reach) + /api/nodes/resolve (#1728)
Implements #1727. ## What this adds **Mobile client-RX coverage** — an opt-in, crowdsourced RF-coverage feature. A roaming MeshCore **companion** radio (driven by the open-source [corescope-rx](https://github.com/efiten/corescope-rx) PWA, GPLv3) reports which nodes it heard directly, tagged with the phone's GPS and the packet's SNR/RSSI. CoreScope ingests these into a new `client_receptions` table and renders per-node **hex coverage** on the Reach page, plus a standalone **Coverage dashboard** (`#/rx-coverage`) with a top-mobile-observers leaderboard. Also includes **`GET /api/nodes/resolve?prefix=<hex>`** — a read-only node-name lookup by pubkey prefix (`{name, pubkey, ambiguous}`), used by the companion app for friendly names. ## Opt-in — default OFF (zero impact on existing deployments) The whole feature is gated behind one config flag, **disabled by default**: ```jsonc "clientRxCoverage": { "enabled": false } ``` When disabled (the default): the ingestor writes **no** `client_receptions`; the three coverage endpoints return a clean **404**; the UI hides the Coverage nav link, the `#/rx-coverage` route, and the Reach-page toggle. `/api/nodes/resolve` is always available (not coverage-specific). ## How it works ``` companion ──BLE 0x88 (snr+rssi+raw)──▶ corescope-rx PWA ──▶ MQTT meshcore/client/{pubkey}/packets │ ingestor (gated) ──▶ client_receptions (GPS + SNR + heard-key) │ server: pure-Go hex grid ──▶ GeoJSON ──▶ Reach hex overlay + Coverage dashboard ``` - **Direct-only capture:** records only what the companion heard itself and directly — a 0-hop advert's pubkey, or `path[last]` (last forwarder) for FLOOD routes; ≥2-byte path-hash required. Upstream hops discarded. - **No new deps:** hexbins are a pure-Go pointy-top grid over Web Mercator (`cmd/server/hexgrid.go`) computed at query time (`CGO_ENABLED=0` / `modernc.org/sqlite` friendly); frontend uses the existing Leaflet. - **Trust:** companion pubkey = identity; an EMQX ACL binds each client to publish only to its own `meshcore/client/{pubkey}/packets` topic. Payload contract in `docs/client-rx-coverage.md`. ## How to enable / try it 1. In `config.json`, set `"clientRxCoverage": { "enabled": true }` and restart server + ingestor. 2. Point an EMQX (or any broker) listener so a client can publish to `meshcore/client/<pubkey>/packets`; the ingestor already subscribes under `meshcore/#`. 3. Run the [corescope-rx](https://github.com/efiten/corescope-rx) PWA on an Android phone paired (BLE) to a MeshCore companion — it captures heard nodes + GPS and publishes. 4. View results: per-node Reach page → toggle **coverage**, or the **Coverage** dashboard at `#/rx-coverage`. ## What's where - **Ingestor:** `cmd/ingestor/client_reception.go` (ingest), `db.go` (`client_receptions` + `client_observers` schema), `main.go` (gated dispatch), `config.go` (flag). - **Server:** `cmd/server/rx_coverage.go` + `rx_dashboard.go` (endpoints, self-guard 404 when off), `hexgrid.go` (pure-Go grid), `node_resolve.go` (resolve), `routes.go` / `types.go` / `config.go` (wiring + flag + `/api/config/client` field). - **Frontend:** `public/rx-coverage.js` (dashboard), `node-reach-coverage.js` + `.css` (overlay), `node-reach.js` (Reach toggle, flag-gated), `roles.js` (reads the flag, hides nav when off). - **Docs:** `docs/client-rx-coverage.md`. ## Testing - Go: `cd cmd/server && go test ./...` and `cd cmd/ingestor && go test ./...` — green, including new gate tests (`coverage_gate_test.go` in both: off → no rows / 404, on → works) and the rx-coverage / resolve / hexgrid suites. - JS: `node test-coverage-gate.js`, `node test-node-reach-coverage.js` (wired into CI). The Playwright `test-node-reach-coverage-e2e.js` is wired into the e2e job and **skips when `clientRxCoverage` is disabled**, so it's safe under the default-off config. ## Notes for reviewers - The four new routes are registered in `cmd/server/openapi_known_gaps.json` (the existing OpenAPI-completeness ratchet), matching how other not-yet-spec'd routes are tracked. Happy to write full OpenAPI spec entries instead if you prefer. - Commits are split per layer (ingestor / server endpoints / resolve / frontend / CI) for review. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Erwin Fiten <e.fiten@opteco.be> |
||
|
Kpa-clawbotandclawbot
|
653d47e03c |
test(openapi): add CI completeness gate for /api routes (Phase 1 of #1670) (#1678)
## Summary Partial fix for #1670 — **Phase 1 only** (CI completeness gate). Phase 2 (backfilling the 18 currently-undocumented routes into `openapi.go`) is deferred to a separate issue per the triage on #1670 and is explicitly out of scope here. ## What this adds - `cmd/server/openapi_completeness_test.go` — AST-walks every non-`_test.go` file in `cmd/server/`, finds string-literal first args to `*.HandleFunc(...)` calls beginning with `/api/`, and diffs against the paths declared in `routeDescriptions()` in `cmd/server/openapi.go`. - `cmd/server/openapi_known_gaps.json` — seeded allowlist of the **18** `/api/` routes currently registered via `HandleFunc` but not yet documented in `openapi.go`. ## Ratchet pattern From this branch forward, `TestOpenAPICompleteness` fails when: 1. A new `HandleFunc("/api/...")` is added without a matching entry in `openapi.go` **or** the allowlist (regression gate — the main goal of Phase 1). 2. A route in the allowlist is *also* documented in `openapi.go` — the allowlist must shrink as Phase 2 backfills land, never go stale. The two-commit history (red → green) demonstrates the gate works: - **Red commit**: adds only the test. Fails on master with the 18 missing routes listed. - **Green commit**: adds the allowlist seeded with that exact 18-route set. Test passes at the current baseline. ## Local verification - `go test ./cmd/server/ -run TestOpenAPICompleteness -v` → PASS at baseline (`44/62 covered; 18 in allowlist; 18 gaps remain`). - Ratchet validation: temporarily inserted `r.HandleFunc("/api/ratchet-test-route", ...)` into `routes.go` → test FAILED with that exact route name; reverted → test PASSES again. ## Files changed - `cmd/server/openapi_completeness_test.go` (+203 / new) - `cmd/server/openapi_known_gaps.json` (+24 / new) ## Preflight `bash ~/.openclaw/skills/pr-preflight/scripts/run-all.sh origin/master` → all hard gates pass; no warnings. ## Out of scope - Backfilling the 18 allowlisted routes into `openapi.go` (Phase 2 — tracked separately). - Schema validation of the spec against OpenAPI 3.0 (Phase 3 per the issue). - PR template checkbox update (Phase 2 follow-up). Issue #1670 stays open for Phase 2. --------- Co-authored-by: clawbot <bot@corescope.local> |