diff --git a/README.md b/README.md index 35bb921..6fd7969 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,8 @@ This repo is where you deploy and operate it. The code lives in The proposed paired backend, basic Atlas and Topology release is described in [Core performance candidate](docs/performance-release-candidate.md), including validation, operational budgets and remaining release gates. +The separate [optional collection guide](docs/telemetry-collection.md) covers +mobile/headless clients, intake, sensor cards and their qualification gates. > **Coming from 1.x?** 2.0.0 needs a fresh database and a few config changes. Read > [Upgrading](docs/upgrading.md) before you pull new images. diff --git a/docs/telemetry-collection.md b/docs/telemetry-collection.md new file mode 100644 index 0000000..3c00b88 --- /dev/null +++ b/docs/telemetry-collection.md @@ -0,0 +1,138 @@ +# Optional companion telemetry collection + +Enable this alongside the core performance, basic Atlas and Topology candidate +only when the intake service and client qualification are ready. Basic Atlas, +Topology and ordinary observer telemetry continue to work without it. + +## Components + +| Component | Responsibility | +| --- | --- | +| Beacon Flutter app | User-selected BLE or Wi-Fi/TCP companion, saved repeater selection, optional local password/admin requirement, polling budgets and signed delivery | +| Beacon server | Optional bounded proxy and `/info` capability for the selected instance | +| Telemetry intake | Automatic enrollment validation, sample validation/deduplication, quotas, separate storage, retention, history and availability | +| Beacon web | Repeaters-with-telemetry filter, attributed battery/environmental series and explicit gaps/partial history | +| Headless Collector | Alternate polling client for unattended USB/BLE use; it uses the same intake contract | + +The app replaces the user-operated polling client, not the intake/storage service. +The current intake is packaged in the separate Collector project. The org +`beacon-agent` scaffold is not a deployable replacement for that implementation. +Software distribution/access for the alpha is separate from client enrollment: +there is no per-user operator approval list. + +## Enable an instance + +Run intake with its own PostgreSQL database. Its canonical `COLLECTOR_INSTANCE` +must equal the HTTPS Beacon URL the app selects, including any installation +subpath and excluding a trailing slash, query string or credentials. A session +issued for the Canadaverse preview is not a session for another Beacon instance. + +Point the optional server setting at the intake over the private service network: + +```yaml +node_telemetry: + gateway_url: http://collector:8081 +``` + +Replace `collector` with that deployment's actual private service hostname. The +public client uses the Beacon instance's existing HTTPS API. The core proxy +accepts only the enrollment/report/read routes, bounds bodies/responses/deadlines, +does not follow redirects and does not forward caller cookies or authorization +headers. Set the intake's trusted proxy CIDR to the core app's actual private +peer address; never trust a caller-provided forwarding header unconditionally. + +When configured, `/api/v1/info` includes `collectedNodeTelemetry: true`. This is +a capability statement, not an assertion that every companion or sensor is +currently healthy. If intake is disabled, optional history is unavailable and +basic cards still work. A configured service failure must remain a visible, +retryable error rather than an empty successful collection. + +## Client workflow + +In the app, select the Beacon instance, open Telemetry sharing, connect the +companion, choose saved repeaters and an interval from 1 to 72 hours. Optional +repeater/admin passwords stay on the client. The companion private key is never +exported. First use starts with flood discovery; subsequently learned routes are +used within the client's hop/congestion policy. Environmental decoding preserves +channel numbers, including channel 2, and explicit zero values. + +The mobile client reserves attempts before transmitting, applies a shared +two-attempt/six-command hourly companion budget and checks airtime/queue state. +An interval is a minimum; selecting more nodes can make the effective interval +longer. Upload retries resend the same saved report and do not poll again. +There is no automatic burst to catch up missed RF intervals. + +For Android, background collection is an explicit remembered choice. The +connected-device service has an ongoing notification with Stop/Disconnect +actions and optional counts. Expired saved reports are counted separately from +accepted reports. Loss of the connection/process stops collection; a foreground +service is not an exact-time or uninterrupted-work guarantee. iOS collection is +foreground-only in this draft. + +No global cross-instance RF lease or hardware attestation is claimed. Cooperating +clients still need coordination before a broad rollout. The server validates +key possession and report structure; public data remains attributed community +measurements rather than independently verified sensor truth. + +## HTTP and MQTT acceptance + +The version-1 routes are under `/api/v1/node-telemetry`: + +| Method/path | Contract | +| --- | --- | +| POST `/challenge` | Short-lived instance/radio/client-bound challenge | +| POST `/enroll` | Both companion and client sign the prefixed challenge; returns an intake session | +| POST `/reports` | Validates the signed sample and returns acceptance only after storage/deduplication | +| GET `/availability` | Full node keys with retained data, 30-day retention and explicit directory truncation | +| GET `/{full-public-key}` | At most 500 retained samples, chronological output, cap/coverage metadata | + +Sample IDs are idempotent only for identical validated samples. A reused ID with +different data is rejected. Session secrets currently rotate when intake restarts; +clients re-enroll and retry the same queued sample, preserving RF cooldowns. + +The app uses HTTPS. The headless client's optional MQTT publish is followed by +idempotent HTTPS confirmation of the same signed sample. Broker PUBACK alone +does not remove the outbox. This protects against an unavailable consumer or an +intake rejection without adding an RF request. MQTT still needs the existing +enrollment API and appropriate topic/TLS credentials. + +## History, capacity and retention + +History responses include `limit`, `truncated`, `retentionDays`, `windowStart` +and `windowEnd`. A cap is not a complete 30-day history. The web labels partial +history and keeps missing measurements distinct from zero; dashed bridges are +visual estimates, not new samples. Full node keys separate identities and +collector/radio attribution keeps different sources from silently joining. + +Availability is coalesced and cached for at most a minute, invalidated by new +accepted reports and retention maintenance. Concurrent first reads share one +bounded aggregation. Default intake pool capacity is four connections unless an +operator explicitly overrides it; budget these alongside core pools. + +Retention runs in short minute-spaced passes, with a five-second context and at +most four indexed batches of 5,000 expired rows. Monitor the oldest retained +timestamp/backlog on larger deployments. The separate +`collector_schema_migrations` ledger records intake initialization; it is not a +core Beacon migration. Keep core and intake backup/recovery identities distinct. + +## Qualification and deployment gates + +Use the exact paired optional server/web commits above the accepted core stack, +the matching app branch and the identified intake build. Do not merge an older +telemetry PR wholesale over newer config, read-model or virtualization changes. + +Validate signatures, instance binding, null/range handling, replay/different-data +IDs, concurrent quotas, expiry, history/directory caps, cache invalidation and +outage recovery on an isolated database. Synthetic signed samples belong in that +isolated test instance, not in the shared preview's public repeater history. + +Then verify the preview's actual retained data, API proxy, web filter/cards and +normal collector acceptance without resetting RF cooldowns. Preserve the +exclusive radio lease and unrelated services during updates. Check before/after +source/binary/asset hashes, database/configuration identity and recovery copies. + +Physical qualification remains separate: a real selected companion/repeater, +status plus channel-2 readings, intake acceptance, matching Atlas display, locked +screen intervals, notification actions and transport/network failure. The current +consolidation found no Android device attached, so these new physical runs are +not claimed. Build/unit/API success does not replace them.