Commit Graph
278 Commits
Author SHA1 Message Date
MrAlders0n b30668bac4 fix(ws): keep the newest event on overflow and apply route/observer filters
A full send buffer dropped the oldest event and then the new one too while
reporting one drop. Subscribe accepted routeTypes and observerIds but never
applied them.
2026-10-02 16:53:32 -04:00
MrAlders0n 2f4f92e31e fix(db): tie-safe trace paging, atomic migrations and honest stats windows
GET /traces takes an optional cursorTag so tags sharing a millisecond survive
paging. Migrations run under an advisory lock and record each file in the
same implicit transaction, so a crash can't strand a fresh database.
Overview windowHours counts the hours it covers, hourly observer activity
fills unrolled hours from raw rows, and integration fixtures are UTC.
2026-10-02 16:53:32 -04:00
MrAlders0n ec0037860c fix(meshmapper): count cut-short requests and qualify groups by heard IATAs
Record each attempt before sending so a request lost to a deadline still
waits out MeshMapper's limit, and stop holding the directory lock across
HTTP. Groups now qualify only through heard or configured IATAs, and one
missing country list only skips pruning.
2026-10-02 16:53:32 -04:00
MrAlders0n 52fc3e1e48 fix(ingest): validate topic keys and harden against malformed input
A short topic pubkey panicked a lane worker and killed the process; keys must
now be 32 bytes and lane workers recover. Also reject non-finite radio values
before airtime, strip NULs from advert names and neighbour scopes, guard the
MQTT client pointer, and never stream a broker copy of a hearing as a repeat.
2026-10-02 16:53:32 -04:00
MrAlders0n 5ccb1a2ce3 fix(api): warn on untrusted proxies and tighten config validation
Warn at startup when limits are on without trusted_proxies, and once when a
request carries forwarding headers Beacon ignores. Reject short or whitespace
admin keys and negative durations. Key default-window stats by hour, return
500 for scope lookup failures, and validate IATA path params.
2026-10-02 16:53:32 -04:00
MrAlders0n 2042c1617f feat(stats): roll hourly scope observers and advertising nodes 2026-10-02 15:24:33 -04:00
MrAlders0n 53d9f86dd4 feat(stats): split /stats/scopes packet counts by hour 2026-10-02 13:57:54 -04:00
MrAlders0n 3e0b6c074c feat(stats): report the longest path per hour in /stats/paths 2026-10-02 13:57:54 -04:00
MrAlders0n 98006a9364 feat(stats): serve historical stats from hourly rollups
Stats endpoints read hourly rollup tables filled once per closed hour by one
advisory-locked task, instead of materialized views that re-scanned every
observation on each refresh. IATA-set tables give exact distinct counts for any
IATA set or region. Cleanup holds raw rows for unrolled hours (up to 24h).

Adds /stats/series; ingest keeps packet observation counts, observer last IATA,
channel message counts and trace tag summaries. 001_baseline.sql is rewritten,
so pre-release 2.0 databases must be recreated. See docs/historical-stats.md.
2026-10-02 07:43:56 -04:00
MrAlders0n 119be44c0d feat(channels): filter channel list by keyKnown
Also seed the cursor integration test with MeshMapper membership; it still
used channel_iatas after channel lists moved to region scoping.
2026-10-01 09:06:50 -04:00
MrAlders0n 27ca84f51e fix(meshmapper): allow MeshMapper's 60s request timeout 2026-10-01 06:05:42 -04:00
MrAlders0n 7392fdd2d6 fix(meshmapper): create group member IATAs and import zone names and centers 2026-09-30 23:21:44 -04:00
MrAlders0n f8ee1ac884 feat(meshmapper): import regional channels and scope channel lists by region 2026-09-30 23:09:40 -04:00
MrAlders0n 38b1a206b1 fix(meshmapper): respect MeshMapper's per-endpoint rate limits 2026-09-30 22:56:10 -04:00
MrAlders0n a970484b00 feat(meshmapper): import zone groups as regions 2026-09-30 22:56:10 -04:00
MrAlders0n bfda0d5281 feat(meshmapper): discover scope sources from the zone list 2026-09-30 22:46:00 -04:00
MrAlders0n 7267b8bca3 feat(meshmapper): accept scope sources outside configured regions 2026-09-30 22:27:36 -04:00
MrAlders0n 9a4e17b270 feat(meshmapper): import zones for every known IATA 2026-09-30 22:23:37 -04:00
MrAlders0n 2f49ffb3a3 feat(api): filter scope lists by region and catalogue membership
Region-filtered /scopes and /stats/scopes now list manual scopes configured
for the region and imported scopes whose MeshMapper catalogue includes the
IATA, instead of inferring membership from observed traffic. Filtered
/scopes returns string[] like the unfiltered call.

Manual scopes now require a region in config.
2026-09-30 22:05:04 -04:00
MrAlders0n 3ac4035942 refactor(db): flatten migrations into a 2.0.0 baseline
2.0.0 is a fresh deploy, so the 1.x chain becomes one schema file. Startup now refuses a pre-2.0.0 database instead of applying the baseline on top.
2026-09-30 21:37:28 -04:00
n30nex 765408e7fc test: drain readiness probes before packet assertions 2026-09-30 19:34:14 -04:00
MrAlders0n 5b177d6fc9 fix(ingest): strip NUL bytes from observer status
An observer sending NUL-padded status strings had every status update rejected
by Postgres, freezing its status and telemetry. Strip them from the text fields
and the stored JSON, and warn with the fields affected.

Also drop the timestamp-clamp, empty-timestamp and malformed-IATA logs to debug;
they're expected and outnumbered inserted packets several times over.
2026-09-30 18:31:42 -04:00
MrAlders0n 14354b03dc fix(telemetry): stop partial and bogus stats inflating bucketed counters
Bucketed airtime and receive errors used MAX - MIN per bucket, so one zeroed,
replayed or garbage row counted the whole running total (e.g. 47,737 s RX in a
6 h bucket). Buckets are now summed from per-row increments with dips, stale
replays and physically impossible jumps rejected. Ingest skips uptime-only
stats and a migration deletes the rows already stored.
2026-09-30 16:32:11 -04:00
MrAlders0n 0e242574fb feat(meshmapper): import region boundaries from the Zones API
Imported boundaries override borderFile on the border map and for foreign marking, falling back to the manual border when MeshMapper has none.
2026-09-30 12:38:36 -04:00
MrAlders0n 689bc2326a fix(ingest): clear a node's location on an explicit 0/0 advert
A repeater that advertises 0/0 is hiding its position. It was stored as a literal 0/0 that every consumer had to special-case; now the stored latitude, longitude and location source are cleared, an omitted location still preserves the stored one, and the nodeUpdate event sends explicit null lat/lng so clients can clear too. Migration 043 drops rows stored as 0/0 before this change.
2026-09-30 11:29:01 -04:00
MrAlders0n 50d35d8626 fix(api): emit observer statusMetadata as JSON 2026-09-30 10:31:08 -04:00
MrAlders0n 8a488fde95 fix(meshmapper): retry failed catalogue fetches sooner
A transient failure pushed the next attempt out by the full refresh interval. Failures now retry in five minutes, with a server Retry-After still honoured as a ceiling.
2026-09-30 10:31:08 -04:00
MrAlders0n 9ce4e8d10c fix(cache): invalidate derived scope keys on catalogue import 2026-09-30 10:31:08 -04:00
MrAlders0n 3b548900a0 fix(api): tolerate client clock skew and drop the dead route evidence cursor 2026-09-30 10:31:08 -04:00
MrAlders0n 1c2841a3c9 perf(db): compute route ambiguity once per reconfirm run
The four-way GROUP BY over node_short_ids ran once per 1000-route batch, up to 750 times a run, and could eat the whole 5s batch budget. The task now fetches the set once and passes it to each batch.
2026-09-30 10:31:08 -04:00
MrAlders0n ab27334f15 fix(backup): report trailing archive data as corruption 2026-09-30 10:31:08 -04:00
MrAlders0n e1b78b6458 fix(profiling): offset periodic captures and harden the recorder
Periodic ticks now start five minutes off the task tickers so the hourly reconfirm trigger stops colliding with the cooldown, any capture clears a deferred periodic, subdirectories no longer stop the recorder, and the metadata sidecar is published before the profile.
2026-09-30 10:31:08 -04:00
MrAlders0n e7061f5e10 fix(api): log the cause of account 500s 2026-09-30 10:31:08 -04:00
MrAlders0n c952afdd29 fix(ingest): let paho auto-ack MQTT deliveries
Sessions are clean and client IDs random, so deferring acks never bought redelivery; it only capped delivery at the broker's inflight window and pushed backlog into the broker queue, where drops are silent.
2026-09-30 10:31:08 -04:00
MrAlders0n 6852074180 fix(ingest): record hearing paths right after the insert
MarkSent ran after the side effects, so a broker copy on another worker could win it and be streamed as a repeat of the first hearing. It now runs before any further DB work, unconditionally so the set is warm before anyone opts in, and never for TRACE, whose path bytes are per-hop SNR.
2026-09-30 10:31:08 -04:00
MrAlders0n d733975165 fix(backup): abort exports on shutdown and sweep stale staging dirs
srv.Shutdown never cancelled the request context, so a SIGTERM mid-export left the plaintext dump and pg_service.conf in TMPDIR. The handler now derives its context from the app context and the router removes staging dirs older than 20 minutes at startup.
2026-09-30 10:31:08 -04:00
MrAlders0n 5cf060114a feat: add bounded CPU profiling (#182)
* feat: add bounded CPU profiling

* fix(profiling): defer cooldown-skipped samples and configure triggers
2026-09-30 07:52:59 -04:00
MrAlders0n fd57c8b00c fix(ingest): accept space-separated and colonless-offset timestamps 2026-09-30 07:48:21 -04:00
n30nex 89a376c291 fix(ingest): accept RFC3339 timestamp offsets and precision 2026-09-30 07:33:53 -04:00
n30nex 97cc22a5a0 fix: address maintainer review for #166 2026-09-30 07:33:45 -04:00
n30nex 3b806c1c39 fix(ingest): resolve advert endpoints after node updates 2026-09-30 07:33:45 -04:00
n30nex 6bf58acb80 fix: address maintainer review for #176 2026-09-30 07:19:03 -04:00
n30nex 2abed6c195 feat(channels): expose recorded packet scope in message delivery 2026-09-30 07:19:03 -04:00
n30nex 83cbaff1cf fix: address release review follow-ups for #174 2026-09-30 07:19:03 -04:00
n30nex 574bba4e13 fix: address maintainer review for #174 2026-09-30 07:19:03 -04:00
n30nex 986a6c0788 test(scopes): synchronize the timeout fixture state 2026-09-30 07:19:03 -04:00
n30nex c18e059e0d feat(scopes): import cached regional MeshMapper catalogues 2026-09-30 07:19:03 -04:00
n30nex 85c9123e9a fix: address maintainer review for #172 2026-09-30 07:19:03 -04:00
n30nex cd20bc5c34 test(routes): cover server-anchored evidence windows 2026-09-30 07:19:03 -04:00
n30nex c448a04ea0 feat(routes): add bounded retained observation evidence 2026-09-30 07:19:03 -04:00