Port the qualified server preview at 84ac3c8781 onto
org dev c7209b7043. Preserve upstream changes and
branch policies; keep the independent private Collector service separate.
Preview-Version: 2.2.0-n30nex.1
Preview-Revision: 84ac3c8781
A short topic pubkey panicked a lane worker and killed the process; keys must
now be 32 bytes and lane workers recover. Also reject non-finite radio values
before airtime, strip NULs from advert names and neighbour scopes, guard the
MQTT client pointer, and never stream a broker copy of a hearing as a repeat.
Stats endpoints read hourly rollup tables filled once per closed hour by one
advisory-locked task, instead of materialized views that re-scanned every
observation on each refresh. IATA-set tables give exact distinct counts for any
IATA set or region. Cleanup holds raw rows for unrolled hours (up to 24h).
Adds /stats/series; ingest keeps packet observation counts, observer last IATA,
channel message counts and trace tag summaries. 001_baseline.sql is rewritten,
so pre-release 2.0 databases must be recreated. See docs/historical-stats.md.
An observer sending NUL-padded status strings had every status update rejected
by Postgres, freezing its status and telemetry. Strip them from the text fields
and the stored JSON, and warn with the fields affected.
Also drop the timestamp-clamp, empty-timestamp and malformed-IATA logs to debug;
they're expected and outnumbered inserted packets several times over.
MarkSent ran after the side effects, so a broker copy on another worker could win it and be streamed as a repeat of the first hearing. It now runs before any further DB work, unconditionally so the set is warm before anyone opts in, and never for TRACE, whose path bytes are per-hop SNR.
Duplicate observations are dropped before the hub today, so a live map only sees each packet's first hearing per observer. Clients that configure includeRepeats also get later hearings whose path is new; nothing is stored, repeats skip the count query, and they are dropped first when the broadcast channel is half full.
Storing a full candidate snapshot on every observation was the wrong call. The same node lists got copied thousands of times, about 22 GB on prod. Resolve endpoints live like path hops and drop the column.
Refresh last_heard on duplicate observations too, capped at hourly, so
steady traffic can't age a trace out of the filter (dedup key has no
heard_at). Drop the unused last_heard index column so upserts stay HOT,
and share one retention cutoff across the cleanup deletes.
The trace list joined every trace packet to ~50M observations to apply
the IATA filter; the parallel hash join overran docker's 64MB /dev/shm
and the filtered list errored. Keep a small trace_iatas table at ingest
(like channel_iatas) and group packets by tag instead. Filtered stats
are now per tag rather than per matching packet.
Refresh last_heard on duplicate observations too, capped at hourly, so
steady traffic can't age a channel out of the filter while its packets
stay retained (dedup key has no heard_at). Guard the seed join against
the docker /dev/shm cap like the trace one, and drop the unused
last_heard index column so the upserts stay HOT.
The IATA filter ran a correlated EXISTS over packets/observations with
ILIKE, which skipped the iata index and took ~7s live. Keep a small
channel_iatas table at ingest (like node_iatas) and filter against it.
Also honor the iatas= param so multi-site regions stop getting the
global list. Filter now ages out with packet retention rather than
matching any retained packet.
any route observer where all hops are high confidence
are now stored and available at /routes
you can also search for a src dest route and it will
return any recored route subset that can link the src
and dest hash
n30nex