The payload-breakdown and top-observers matviews stored a single 7-day
total per group, so the 24h/7d/30d selector did nothing: payload ignored
the window entirely (the store discarded `since`) and top-observers only
varied which rows cleared the last_heard cutoff, never the counts.
Bucket both views by hour over a 30-day horizon and sum the buckets in
range, mirroring mv_hourly_iata_stats. The stats endpoints now honour the
requested window while still reading a small precomputed table.
GetScopeStats left-joined packets, observer_scopes and nodes at once,
multiplying rows into the millions before COUNT(DISTINCT) deduped them
(~10s per call). Count each table on its own and index packets(scope_id).
Refresh last_heard on duplicate observations too, capped at hourly, so
steady traffic can't age a trace out of the filter (dedup key has no
heard_at). Drop the unused last_heard index column so upserts stay HOT,
and share one retention cutoff across the cleanup deletes.
The trace list joined every trace packet to ~50M observations to apply
the IATA filter; the parallel hash join overran docker's 64MB /dev/shm
and the filtered list errored. Keep a small trace_iatas table at ingest
(like channel_iatas) and group packets by tag instead. Filtered stats
are now per tag rather than per matching packet.
Refresh last_heard on duplicate observations too, capped at hourly, so
steady traffic can't age a channel out of the filter while its packets
stay retained (dedup key has no heard_at). Guard the seed join against
the docker /dev/shm cap like the trace one, and drop the unused
last_heard index column so the upserts stay HOT.
The IATA filter ran a correlated EXISTS over packets/observations with
ILIKE, which skipped the iata index and took ~7s live. Keep a small
channel_iatas table at ingest (like node_iatas) and filter against it.
Also honor the iatas= param so multi-site regions stop getting the
global list. Filter now ages out with packet retention rather than
matching any retained packet.
Quiet sites probed hundreds of thousands of packets to fill a page.
Walk idx_observations_iata_heard per requested site and dedup instead.
Filtered lists now order by site-local recency; the cursor follows.
text[] from string_to_array can't match the CHAR(3) indexes and hides
the values from the planner. Handlers already have a slice; stop
joining to CSV in the store layer and pass a typed array everywhere.
The packet list sorts by last_heard_at, so every page was a seq scan
plus sort over all packets. Cheap to maintain now that presence writes
are coalesced.
this was causing duplicates and each obersver heard a given
packet hash or it didn't, there are not more observations of
the same hash from one observer
Enot (ded) Skelly