> **Stacked on #1905.** This branch contains #1905's commits; review and merge that one first. The diff unique to this PR is the `client_rf_samples` table, its handler, the delta query and its retention. ## What Everything CoreDrive RX records today is anchored to a *packet*. But a drive also passes through RF conditions that exist whether or not a packet arrives: the noise floor, how busy the channel is, how many receptions fail CRC. The radio measures all three and was never asked. This samples the companion's own counters along the GPS track and stores them, so the server can render a noise-floor map, a channel-utilisation map and a CRC-error-rate map. A fixed observer cannot produce those — it measures one point forever. **Zero airtime:** `CMD_GET_STATS` is a local Bluetooth query to the attached radio. Nothing is transmitted. ## Design points worth knowing - **Absolutes are stored; deltas are derived at query time.** A lost or reordered sample then costs one interval rather than corrupting a running total. `ClientRfDeltas` breaks the chain whenever `uptime_secs` fails to increase — that is the reboot and counter-wrap detector. - **Absent is not zero, end to end.** Firmware predating the `recv_errors` field cannot count CRC errors at all, and a stored `0` would read downstream as "a perfectly clean channel" — the opposite of "we don't know". Presence/absence is preserved through the app parser, the wire payload, a nullable column, and the delta view, which returns `nil` rather than `0` when either endpoint is unknown. Each of those five layers has its own test. - **`sampled_at` is millisecond precision, and it is load-bearing.** SQLite compares these strings lexicographically and `.` (0x2E) sorts before `Z` (0x5A), so a second-resolution retention cutoff would delete rows *inside* the window. The prune formats its cutoff with the same layout. ## Performance justification (touches the ingest hot path) - One INSERT per sample, gated behind an opt-in flag that defaults off. Sample rate is 15 s while moving and 5 min while parked, so roughly 240 rows per hour per active driver. - The delta query is a single `LAG(...) OVER` pass with no nested query inside the loop, so it cannot deadlock the single writer connection. Window functions are already used elsewhere in this codebase. - Retention has its own key and index (`sampled_at`); without it the table would grow unbounded, so `config.example.json` documents it inline. ## Safety for existing deployments Opt-in and default off on both sides (`clientRfSamples.enabled`, and `rfSampler` in the app). The coverage path is untouched — `Publisher.buildPayload` is byte-identical and a record with no `kind` field still routes to `/packets` unchanged. The MQTT dispatch was reshaped so that **anything on `meshcore/client/…` returns from that branch in every config state**, with the enable-gates inside rather than in the topic match. Previously a disabled gate let the message fall through to the observer path, where `parts[1]` — the literal string `client` — was read as a region and the phone's pubkey registered as an observer. The blacklist check now also runs ahead of the sub-topic switch, so it covers every present and future client sub-topic. ## Testing Full ingestor suite green. Notable coverage: a `/rf` message with the gate off writes nothing anywhere and does not fall through; a sample missing `uptime_secs` is rejected rather than stored as an unusable row; two samples 40 ms apart remain two rows; and the retention test seeds a row with a non-zero millisecond component inside the cutoff second, which is the only row that distinguishes a correct cutoff from an RFC3339 one. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
7.7 KiB
Client RF Environment Samples
Crowdsourced RF environment data from mobile clients: the same phone that samples client RX coverage also reads its attached MeshCore companion radio's own counters — noise floor, RX/TX airtime, CRC errors, packet totals — along the GPS track, and publishes them on a separate topic. Unlike a fixed observer, which measures one point forever, a roaming companion builds a noise-floor map, a channel-utilisation map and a CRC-error-rate map as it moves.
Enabling RF samples (operators)
Off by default. To turn it on:
- In CoreScope's
config.json, set"clientRfSamples": { "enabled": true }and restart the ingestor. Independent ofclientRxCoverage— a deployment can run one, both, or neither. - Required: an ACL-capable broker, same as coverage. Bind
meshcore/client/{PUBLIC_KEY}/rfso each client may publish only under its own pubkey. The ingestor already subscribes undermeshcore/#. - Optionally set
retention.clientRfDaysto bound the table.
MQTT topic & payload
Topic: meshcore/client/{PUBLIC_KEY}/rf — {PUBLIC_KEY} is the companion's pubkey.
{
"type": "RF_SAMPLE",
"timestamp": "2026-08-17T10:00:00.000Z",
"gps": { "lat": 51.2, "lon": 4.4, "acc_m": 8.0 },
"stationary": false,
"uptime_secs": 84213,
"battery_mv": 3950,
"queue_len": 0,
"errors": 0,
"noise_floor": -119,
"last_rssi": -92,
"last_snr": -7.0,
"tx_air_secs": 120,
"rx_air_secs": 20877,
"recv": 4310,
"sent": 12,
"flood_rx": 3980,
"direct_rx": 330,
"flood_tx": 10,
"direct_tx": 2,
"recv_errors": 5
}
- The discriminator is the
gpsobject, same as coverage: a sample withoutgps(or with an out-of-rangelat/lon) is dropped. uptime_secsis required. It is the only reboot/wrap detector the delta query has — see Deltas below. A sample missing it is dropped and logged, the same as a missing GPS fix, rather than silently defaulting to 0.stationaryis optional, defaults tofalsewhen absent.- Every counter besides
uptime_secsis optional. Absent stays SQLNULL, never0— see recv_errors below. errorsis not a counter despite sitting among them here: per the firmware stats frame it is an error-flags bitmask, so it is stored as reported but deliberately excluded fromClientRfDeltas.- Subscription: the ingestor's default subscription (
meshcore/#) already covers this topic. Sources configured with an explicit topic list must addmeshcore/client/+/rf.
Trust
Identity = the companion pubkey, taken from the {PUBLIC_KEY} topic segment — never from a
payload-supplied origin_id, which would defeat the ACL trust model. The ingestor rejects any topic
pubkey that is not lowercase hex before writing (same clientPubkeyRe used for coverage), and the
observer blacklist is enforced before any write, so a blacklisted operator cannot contribute RF
samples any more than it can contribute coverage.
recv_errors: absent vs. zero
recv_errors counts CRC errors. Firmware predating the counter cannot report it at all, and that
case must stay NULL — storing 0 would read as "a perfectly clean channel," the opposite of "we
don't know." The app is expected to omit the field entirely (not send a fabricated 0) on firmware
that doesn't support it, and the handler never collapses an absent optional field to a zero.
Storage — client_rf_samples (ingestor-owned)
client_rf_samples(
id, rx_pubkey, sampled_at, ingested_at, lat, lon, pos_acc_m, stationary,
uptime_secs, battery_mv, queue_len, errors, noise_floor, last_rssi, last_snr,
tx_air_secs, rx_air_secs, recv, sent, flood_rx, direct_rx, flood_tx, direct_tx,
recv_errors,
UNIQUE(rx_pubkey, sampled_at)) -- idempotent re-ingest
Every counter is stored as an absolute cumulative value, exactly as the radio reports it — the
handler does no arithmetic. errors is the exception: per the firmware stats frame it is an
error-flags bitmask, not a counter, so it is deliberately excluded from ClientRfDeltas — a future
consumer must not wire it into a rate calculation. sampled_at is stored at millisecond precision (the
rxTimeMillisLayout format shared with client_rx_observations.rx_at), not the second-resolution
RFC3339 used by client_receptions.rx_at. This matters for two reasons: samples can arrive faster
than once per second along a moving track, and Task 6's retention prune compares the cutoff
lexicographically against these stored strings — a second-resolution cutoff against
millisecond-resolution values would delete rows it should keep.
Retention: retention.clientRfDays bounds the table by sampled_at; 0 disables it (Task 6).
Deltas — reboot and wrap handling
Store.ClientRfDeltas(rxPubkey, from, to) derives consecutive-sample deltas (RX/TX airtime, recv,
recv_errors, wall-clock milliseconds) for one radio over a time range, via a LAG(...) OVER (ORDER BY sampled_at) window query. Every counter in the table is cumulative, so a delta is only meaningful
between two samples from the same uninterrupted uptime:
- A pair is skipped whenever
uptime_secsdoes not strictly increase between consecutive samples. That is a reboot (uptime reset near 0) or a counter wrap, and subtracting across it would produce a large negative or a bogus spike.uptime_secshas whole-second granularity, so sampling faster than 1 Hz means roughly every other pair gets skipped this way too (equal, not decreased,uptime_secs) — that is expected, not a bug: strict-increase is deliberately correct for reboot detection, but it meansClientRfDeltaseffectively assumes callers sample at ≥1 s intervals if they want every interval represented. - The absolute values always stay in
client_rf_samples; only the delta view drops the cross-reboot pair. No row is ever deleted or modified because of this check. - Each
*Deltafield onClientRfDelta(RxAirDelta,TxAirDelta,RecvDelta,RecvErrDelta) is a*int64, nil when either endpoint of the pair is NULL — i.e. the underlying counter is unsupported on that firmware. A delta of0there would be indistinguishable from a measured zero, which forRecvErrDeltain particular would render as "clean channel" on a radio that simply can't count CRC errors. Consumers must treat nil as unknown, not zero. - When both endpoints of a pair are present, the per-metric delta additionally guards against
cur < prev(covers a wrapped individual counter even whenuptime_secsitself looks monotonic) by clamping to0rather than going negative. from/toare compared lexicographically against millisecond-precisionsampled_atstrings, so callers must pass bounds in the samerxTimeMillisLayoutformat (2006-01-02T15:04:05.000Z07:00). A bound like10:00:00Zwould lexicographically exclude a sample stored as10:00:00.500Z(.sorts beforeZ), silently narrowing the range.- A phone clock running more than 14h ahead has its
timestampclamped to ingest time byresolveRxTimeCore(same clamp used across the client-RX paths), which compressesWallMillisfor a buffered batch uploaded in one burst — several samples with distinct on-device timestamps can collapse onto ingest-time values seconds apart.sampled_atcan never be zero-spaced (the UNIQUE constraint drops an exact collision), so this is a wrong-rate hazard, not a divide-by-zero: a consumer computing a rate from*Delta / WallMillismust sanity-boundWallMillisbefore dividing.
Configurable values (future customizer)
retention.clientRfDays is the only tunable so far; no color/threshold customization applies to this
feature yet (no map rendering built on it in this task).