Storing a full candidate snapshot on every observation was the wrong call. The same node lists got copied thousands of times, about 22 GB on prod. Resolve endpoints live like path hops and drop the column.
A packet repeats once per observer in the site scan, so grouping can
collapse a scan_depth window to fewer packets than the page asked for.
That short page set hasMore=false and stranded all older history: YOW
dead-ended after 45 packets, ~6 minutes back.
Report scan saturation and the floor the scans covered, and treat a
saturated short page as more data. Clamp the cursor to the floor so
paging on cannot skip the band the scan never read.
The payload-breakdown and top-observers matviews stored a single 7-day
total per group, so the 24h/7d/30d selector did nothing: payload ignored
the window entirely (the store discarded `since`) and top-observers only
varied which rows cleared the last_heard cutoff, never the counts.
Bucket both views by hour over a 30-day horizon and sum the buckets in
range, mirroring mv_hourly_iata_stats. The stats endpoints now honour the
requested window while still reading a small precomputed table.
GetScopeStats left-joined packets, observer_scopes and nodes at once,
multiplying rows into the millions before COUNT(DISTINCT) deduped them
(~10s per call). Count each table on its own and index packets(scope_id).
Refresh last_heard on duplicate observations too, capped at hourly, so
steady traffic can't age a trace out of the filter (dedup key has no
heard_at). Drop the unused last_heard index column so upserts stay HOT,
and share one retention cutoff across the cleanup deletes.
The trace list joined every trace packet to ~50M observations to apply
the IATA filter; the parallel hash join overran docker's 64MB /dev/shm
and the filtered list errored. Keep a small trace_iatas table at ingest
(like channel_iatas) and group packets by tag instead. Filtered stats
are now per tag rather than per matching packet.
Refresh last_heard on duplicate observations too, capped at hourly, so
steady traffic can't age a channel out of the filter while its packets
stay retained (dedup key has no heard_at). Guard the seed join against
the docker /dev/shm cap like the trace one, and drop the unused
last_heard index column so the upserts stay HOT.
The IATA filter ran a correlated EXISTS over packets/observations with
ILIKE, which skipped the iata index and took ~7s live. Keep a small
channel_iatas table at ingest (like node_iatas) and filter against it.
Also honor the iatas= param so multi-site regions stop getting the
global list. Filter now ages out with packet retention rather than
matching any retained packet.
Quiet sites probed hundreds of thousands of packets to fill a page.
Walk idx_observations_iata_heard per requested site and dedup instead.
Filtered lists now order by site-local recency; the cursor follows.
text[] from string_to_array can't match the CHAR(3) indexes and hides
the values from the planner. Handlers already have a slice; stop
joining to CSV in the store layer and pass a typed array everywhere.
n30nex