The payload-breakdown and top-observers matviews stored a single 7-day
total per group, so the 24h/7d/30d selector did nothing: payload ignored
the window entirely (the store discarded `since`) and top-observers only
varied which rows cleared the last_heard cutoff, never the counts.
Bucket both views by hour over a 30-day horizon and sum the buckets in
range, mirroring mv_hourly_iata_stats. The stats endpoints now honour the
requested window while still reading a small precomputed table.
GetScopeStats left-joined packets, observer_scopes and nodes at once,
multiplying rows into the millions before COUNT(DISTINCT) deduped them
(~10s per call). Count each table on its own and index packets(scope_id).
Refresh last_heard on duplicate observations too, capped at hourly, so
steady traffic can't age a trace out of the filter (dedup key has no
heard_at). Drop the unused last_heard index column so upserts stay HOT,
and share one retention cutoff across the cleanup deletes.
The trace list joined every trace packet to ~50M observations to apply
the IATA filter; the parallel hash join overran docker's 64MB /dev/shm
and the filtered list errored. Keep a small trace_iatas table at ingest
(like channel_iatas) and group packets by tag instead. Filtered stats
are now per tag rather than per matching packet.
Refresh last_heard on duplicate observations too, capped at hourly, so
steady traffic can't age a channel out of the filter while its packets
stay retained (dedup key has no heard_at). Guard the seed join against
the docker /dev/shm cap like the trace one, and drop the unused
last_heard index column so the upserts stay HOT.
The IATA filter ran a correlated EXISTS over packets/observations with
ILIKE, which skipped the iata index and took ~7s live. Keep a small
channel_iatas table at ingest (like node_iatas) and filter against it.
Also honor the iatas= param so multi-site regions stop getting the
global list. Filter now ages out with packet retention rather than
matching any retained packet.
The packet list sorts by last_heard_at, so every page was a seq scan
plus sort over all packets. Cheap to maintain now that presence writes
are coalesced.
this was causing duplicates and each obersver heard a given
packet hash or it didn't, there are not more observations of
the same hash from one observer
any route observer where all hops are high confidence
are now stored and available at /routes
you can also search for a src dest route and it will
return any recored route subset that can link the src
and dest hash
new config for transport scope keys seeded to db
new transport keystore to lookup matching keys and tag
- packets with scope key
- nodes with default scope
- observers with all supported scopes it forwards
allow filtering by region scope key on: packets, nodes, observers
changes channel storage to include the full key and a hashtag if isHashtag
this prevents some collision issues and will improve message lookup accuracy as well
as allow message search by hashtag channel, name or single byte hash in future
MrAlders0n