Files
meshcore-analyzer/docs/client-rf-samples.md
T
efitenandClaude Opus 5 9ae3387416 feat(ingestor): RF environment samples from mobile clients (#1906)
> **Stacked on #1905.** This branch contains #1905's commits; review and
merge that one first. The diff unique to this PR is the
`client_rf_samples` table, its handler, the delta query and its
retention.

## What

Everything CoreDrive RX records today is anchored to a *packet*. But a
drive also passes through RF conditions that exist whether or not a
packet arrives: the noise floor, how busy the channel is, how many
receptions fail CRC. The radio measures all three and was never asked.

This samples the companion's own counters along the GPS track and stores
them, so the server can render a noise-floor map, a channel-utilisation
map and a CRC-error-rate map. A fixed observer cannot produce those — it
measures one point forever.

**Zero airtime:** `CMD_GET_STATS` is a local Bluetooth query to the
attached radio. Nothing is transmitted.

## Design points worth knowing

- **Absolutes are stored; deltas are derived at query time.** A lost or
reordered sample then costs one interval rather than corrupting a
running total. `ClientRfDeltas` breaks the chain whenever `uptime_secs`
fails to increase — that is the reboot and counter-wrap detector.
- **Absent is not zero, end to end.** Firmware predating the
`recv_errors` field cannot count CRC errors at all, and a stored `0`
would read downstream as "a perfectly clean channel" — the opposite of
"we don't know". Presence/absence is preserved through the app parser,
the wire payload, a nullable column, and the delta view, which returns
`nil` rather than `0` when either endpoint is unknown. Each of those
five layers has its own test.
- **`sampled_at` is millisecond precision, and it is load-bearing.**
SQLite compares these strings lexicographically and `.` (0x2E) sorts
before `Z` (0x5A), so a second-resolution retention cutoff would delete
rows *inside* the window. The prune formats its cutoff with the same
layout.

## Performance justification (touches the ingest hot path)

- One INSERT per sample, gated behind an opt-in flag that defaults off.
Sample rate is 15 s while moving and 5 min while parked, so roughly 240
rows per hour per active driver.
- The delta query is a single `LAG(...) OVER` pass with no nested query
inside the loop, so it cannot deadlock the single writer connection.
Window functions are already used elsewhere in this codebase.
- Retention has its own key and index (`sampled_at`); without it the
table would grow unbounded, so `config.example.json` documents it
inline.

## Safety for existing deployments

Opt-in and default off on both sides (`clientRfSamples.enabled`, and
`rfSampler` in the app). The coverage path is untouched —
`Publisher.buildPayload` is byte-identical and a record with no `kind`
field still routes to `/packets` unchanged.

The MQTT dispatch was reshaped so that **anything on `meshcore/client/…`
returns from that branch in every config state**, with the enable-gates
inside rather than in the topic match. Previously a disabled gate let
the message fall through to the observer path, where `parts[1]` — the
literal string `client` — was read as a region and the phone's pubkey
registered as an observer. The blacklist check now also runs ahead of
the sub-topic switch, so it covers every present and future client
sub-topic.

## Testing

Full ingestor suite green. Notable coverage: a `/rf` message with the
gate off writes nothing anywhere and does not fall through; a sample
missing `uptime_secs` is rejected rather than stored as an unusable row;
two samples 40 ms apart remain two rows; and the retention test seeds a
row with a non-zero millisecond component inside the cutoff second,
which is the only row that distinguishes a correct cutoff from an
RFC3339 one.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 21:22:07 +00:00

7.7 KiB

Client RF Environment Samples

Crowdsourced RF environment data from mobile clients: the same phone that samples client RX coverage also reads its attached MeshCore companion radio's own counters — noise floor, RX/TX airtime, CRC errors, packet totals — along the GPS track, and publishes them on a separate topic. Unlike a fixed observer, which measures one point forever, a roaming companion builds a noise-floor map, a channel-utilisation map and a CRC-error-rate map as it moves.

Enabling RF samples (operators)

Off by default. To turn it on:

  1. In CoreScope's config.json, set "clientRfSamples": { "enabled": true } and restart the ingestor. Independent of clientRxCoverage — a deployment can run one, both, or neither.
  2. Required: an ACL-capable broker, same as coverage. Bind meshcore/client/{PUBLIC_KEY}/rf so each client may publish only under its own pubkey. The ingestor already subscribes under meshcore/#.
  3. Optionally set retention.clientRfDays to bound the table.

MQTT topic & payload

Topic: meshcore/client/{PUBLIC_KEY}/rf{PUBLIC_KEY} is the companion's pubkey.

{
  "type": "RF_SAMPLE",
  "timestamp": "2026-08-17T10:00:00.000Z",
  "gps": { "lat": 51.2, "lon": 4.4, "acc_m": 8.0 },
  "stationary": false,
  "uptime_secs": 84213,
  "battery_mv": 3950,
  "queue_len": 0,
  "errors": 0,
  "noise_floor": -119,
  "last_rssi": -92,
  "last_snr": -7.0,
  "tx_air_secs": 120,
  "rx_air_secs": 20877,
  "recv": 4310,
  "sent": 12,
  "flood_rx": 3980,
  "direct_rx": 330,
  "flood_tx": 10,
  "direct_tx": 2,
  "recv_errors": 5
}
  • The discriminator is the gps object, same as coverage: a sample without gps (or with an out-of-range lat/lon) is dropped.
  • uptime_secs is required. It is the only reboot/wrap detector the delta query has — see Deltas below. A sample missing it is dropped and logged, the same as a missing GPS fix, rather than silently defaulting to 0.
  • stationary is optional, defaults to false when absent.
  • Every counter besides uptime_secs is optional. Absent stays SQL NULL, never 0 — see recv_errors below.
  • errors is not a counter despite sitting among them here: per the firmware stats frame it is an error-flags bitmask, so it is stored as reported but deliberately excluded from ClientRfDeltas.
  • Subscription: the ingestor's default subscription (meshcore/#) already covers this topic. Sources configured with an explicit topic list must add meshcore/client/+/rf.

Trust

Identity = the companion pubkey, taken from the {PUBLIC_KEY} topic segment — never from a payload-supplied origin_id, which would defeat the ACL trust model. The ingestor rejects any topic pubkey that is not lowercase hex before writing (same clientPubkeyRe used for coverage), and the observer blacklist is enforced before any write, so a blacklisted operator cannot contribute RF samples any more than it can contribute coverage.

recv_errors: absent vs. zero

recv_errors counts CRC errors. Firmware predating the counter cannot report it at all, and that case must stay NULL — storing 0 would read as "a perfectly clean channel," the opposite of "we don't know." The app is expected to omit the field entirely (not send a fabricated 0) on firmware that doesn't support it, and the handler never collapses an absent optional field to a zero.

Storage — client_rf_samples (ingestor-owned)

client_rf_samples(
  id, rx_pubkey, sampled_at, ingested_at, lat, lon, pos_acc_m, stationary,
  uptime_secs, battery_mv, queue_len, errors, noise_floor, last_rssi, last_snr,
  tx_air_secs, rx_air_secs, recv, sent, flood_rx, direct_rx, flood_tx, direct_tx,
  recv_errors,
  UNIQUE(rx_pubkey, sampled_at))   -- idempotent re-ingest

Every counter is stored as an absolute cumulative value, exactly as the radio reports it — the handler does no arithmetic. errors is the exception: per the firmware stats frame it is an error-flags bitmask, not a counter, so it is deliberately excluded from ClientRfDeltas — a future consumer must not wire it into a rate calculation. sampled_at is stored at millisecond precision (the rxTimeMillisLayout format shared with client_rx_observations.rx_at), not the second-resolution RFC3339 used by client_receptions.rx_at. This matters for two reasons: samples can arrive faster than once per second along a moving track, and Task 6's retention prune compares the cutoff lexicographically against these stored strings — a second-resolution cutoff against millisecond-resolution values would delete rows it should keep.

Retention: retention.clientRfDays bounds the table by sampled_at; 0 disables it (Task 6).

Deltas — reboot and wrap handling

Store.ClientRfDeltas(rxPubkey, from, to) derives consecutive-sample deltas (RX/TX airtime, recv, recv_errors, wall-clock milliseconds) for one radio over a time range, via a LAG(...) OVER (ORDER BY sampled_at) window query. Every counter in the table is cumulative, so a delta is only meaningful between two samples from the same uninterrupted uptime:

  • A pair is skipped whenever uptime_secs does not strictly increase between consecutive samples. That is a reboot (uptime reset near 0) or a counter wrap, and subtracting across it would produce a large negative or a bogus spike. uptime_secs has whole-second granularity, so sampling faster than 1 Hz means roughly every other pair gets skipped this way too (equal, not decreased, uptime_secs) — that is expected, not a bug: strict-increase is deliberately correct for reboot detection, but it means ClientRfDeltas effectively assumes callers sample at ≥1 s intervals if they want every interval represented.
  • The absolute values always stay in client_rf_samples; only the delta view drops the cross-reboot pair. No row is ever deleted or modified because of this check.
  • Each *Delta field on ClientRfDelta (RxAirDelta, TxAirDelta, RecvDelta, RecvErrDelta) is a *int64, nil when either endpoint of the pair is NULL — i.e. the underlying counter is unsupported on that firmware. A delta of 0 there would be indistinguishable from a measured zero, which for RecvErrDelta in particular would render as "clean channel" on a radio that simply can't count CRC errors. Consumers must treat nil as unknown, not zero.
  • When both endpoints of a pair are present, the per-metric delta additionally guards against cur < prev (covers a wrapped individual counter even when uptime_secs itself looks monotonic) by clamping to 0 rather than going negative.
  • from/to are compared lexicographically against millisecond-precision sampled_at strings, so callers must pass bounds in the same rxTimeMillisLayout format (2006-01-02T15:04:05.000Z07:00). A bound like 10:00:00Z would lexicographically exclude a sample stored as 10:00:00.500Z (. sorts before Z), silently narrowing the range.
  • A phone clock running more than 14h ahead has its timestamp clamped to ingest time by resolveRxTimeCore (same clamp used across the client-RX paths), which compresses WallMillis for a buffered batch uploaded in one burst — several samples with distinct on-device timestamps can collapse onto ingest-time values seconds apart. sampled_at can never be zero-spaced (the UNIQUE constraint drops an exact collision), so this is a wrong-rate hazard, not a divide-by-zero: a consumer computing a rate from *Delta / WallMillis must sanity-bound WallMillis before dividing.

Configurable values (future customizer)

retention.clientRfDays is the only tunable so far; no color/threshold customization applies to this feature yet (no map rendering built on it in this task).