Commit Graph
490 Commits
Author SHA1 Message Date
MrAlders0n 41baee9197 Packets: one statement per filter shape, exact pageCursor on packets, nodes and observers (2.1.0) (#241)
* feat(db): packet type domains, row-cursor indexes and the inverted-heard index

* feat(api): exact page cursors for packets, nodes and observers

* feat(db): normalise packet list filters before choosing a statement

* feat(db): one packets statement per filter shape with an exact row cursor

* fix(db): let the window statements decide since after until

* feat(db): exact keyset cursors for nodes and observers; observers page by last_seen

* feat(api): pageCursor on packets, nodes and observers; it wins over cursor

* test(planguard): packets statements and node/observer keysets leave pending; diff and storediff cases

* chore: bump version to 2.1.0

* test(db): turn off JIT in the keyset cursor tests
2026-10-19 15:59:55 -04:00
MrAlders0n 94b4b40f9b planguard: plan and result regression checks, fixture tier in CI, read-only mode (#240)
* feat(planguard): read sqlc statement text and load a strict plan corpus

* fix(planguard): fail on unextracted statements, keep fragments mode-neutral

* feat(planguard): plan assertions, scoped sort exemptions and narrow retired-index verdicts

* fix(planguard): an errored run always fails

* fix(planguard): keep subplan conds off their parent scan; fail on unanalyzed budgets

* feat(planguard): read-only plan runner, reports and the run command

* fix(planguard): validate identifiers in emitted scripts; prove read-only runs

* feat(planguard): end-state corpus, planted regressions and the generated cached-plan policy

* fix(planguard): derive observer cases from data; mark legacy generic plans

* fix(planguard): bound the dense-region route resolvers; test policy refusals

* fix(planguard): re-measure the dense-region route cases; cover every policy refusal

* feat(planguard): synthetic fixture built through the migrations, checked in CI

* ci(planguard): build the fixture once; guard fixture data for identifiers

* fix(planguard): hash every fixture generator input; harden the fixture guard

* feat(planguard): snapshot diff over every column, named result classes and cursor walks

* fix(planguard): bound the relaxed diff classes; derive the cursor tie from data

* fix(planguard): refuse multi-statement text in the corpus runner

* feat: read-only server mode and an HTTP page diff between two builds

* fix(planguard): refuse escape strings and psql meta-commands in corpus text

* fix: storediff follows each build's own cursor; prove read-only gates via the log

* fix(planguard): match the PostgreSQL lexer on high-bit bytes; check every emitted line

* feat(admin): daily read-only plan check on the maintenance pool, its metric and report

* feat(planguard): daily prod check behind planguard.daily_check

* fix(planguard): cancel fresh runs at shutdown; expose a never-ran check

* ci(planguard): pin the fixture server and collation; refresh baselines

* fix(planguard): calibrate route budgets on one dataset; skip entries whose index is absent
2026-10-19 13:45:39 -04:00
MrAlders0n dba66cada7 Small tables: drop HOT-blocking indexes, channel write-through (#239)
* perf(db): drop small-table indexes that block HOT updates

* perf(db): touch known channels by id instead of upserting

* feat: preload channel ids at boot for ingest and maintenance
2026-10-19 08:28:15 -04:00
MrAlders0n 03f66d1ae4 Database pools by workload, plan router, observe-only guards, metrics (rest of #237) (#238)
* feat(db): five pgx pools with fixed session settings and Prometheus tracing

* feat(guards): per-family observe/enforce guards with client slots, deadlines and a shadow budget

* fix(guards): attribute only guard deadlines and classify like the handlers

* feat(db): route API statements by sqlc name; class C always unnamed

* feat(admin): statement statistics by sqlc name; create pg_stat_statements where allowed

* fix(db): skip pg_stat_statements when no schema is creatable

* feat(background): index health against post-rebuild baselines, Postgres activity, job and ingest metrics

* feat: separate API, ingest and maintenance stores; guards on the public API; metrics listener

* fix(db): keep ingest writes waiting, size the ingest pool to the host, tidy

* test(db): give the statements probe a query id of its own
2026-10-19 07:52:51 -04:00
MrAlders0n 3ae3c250d8 feat(config): plan layout, guard and admin listener settings; plan policy classes (#237) 2026-10-19 07:43:49 -04:00
MrAlders0n c8595424e9 Drop three route indexes no statement reads (#236)
* perf(db): drop three route indexes no statement reads

* docs(db): reword the retention residual comment
2026-10-19 05:46:20 -04:00
MrAlders0n 8e0fd4b535 Routes: one statement per filter shape, exact bounded retention, stamp-free reconfirm (#235)
* feat(db): background_state table and the (hop_count, iata) route index

* perf(db): one /routes statement per filter shape, hop pair on custom plans

* feat(db): keyset route retention and grace slices at exact cutoffs

* feat(background): hourly route retention with a bounded grace walk and watermark

* fix(background): bound the grace walk at microsecond cutoffs

* feat(db): stamp-free reconfirm sweep over the route primary key

* feat(background): pace route reconfirm at routes.reconfirm_rows_per_hour, no stamps

* test(routes): drive route retention end to end and tidy comments

* fix(profiling): sample during the hourly route retention run
2026-10-19 05:39:38 -04:00
MrAlders0n c27f721dd2 Route search hop order, node-types split, bounded migration lock waits (#234)
* perf(routes): keep hop_count indexes out of route search order

* perf(stats): count node types without node_iatas when unfiltered

* perf(db): drop the unused idx_known_routes_iata_last_seen

* fix(db): bound boot migration lock waits and name the blockers

* test(db): pin the lock-retry rebuild and tidy comments
2026-10-19 03:58:37 -04:00
MrAlders0n d5aacce3c7 fix(db): bigint id cursors before observation ids pass 2^31 (#233)
* fix(db): type id cursors as bigint

* chore: bump version to 2.0.8
2026-10-19 03:33:40 -04:00
MrAlders0n 127fc76092 Raise the default per-client API limit to 600 a minute (#232)
Opening Map and Nodes on a region of a few thousand nodes could spend the
whole 300-request minute and leave the map half loaded. 600 still sits
under the fail2ban flood jail's 900, so the server's 429 answers first.
2026-10-18 23:12:57 -04:00
MrAlders0n 202acc6cc8 Merge pull request #228 from MeshCore-Beacon/dev
release: v2.0.7
v2.0.7
2026-10-14 17:19:52 -04:00
MrAlders0n b43da0c3a9 chore: bump version to 2.0.7 2026-10-14 17:10:50 -04:00
MrAlders0n 632b83c4b9 feat(api): search packets by hash prefix (#227) 2026-10-14 17:04:22 -04:00
MrAlders0n bc84be577c Merge pull request #222 from MeshCore-Beacon/dev
release: v2.0.6
v2.0.6
2026-10-12 13:32:55 -04:00
MrAlders0n 354dad6925 chore: bump version to 2.0.6 2026-10-12 13:31:33 -04:00
MrAlders0n 7297215555 feat(meshmapper): import repeater site details for node detail (#221) 2026-10-11 14:18:26 -04:00
MrAlders0n 570a72b614 release: v2.0.5 (#220)
release: v2.0.5
v2.0.5
2026-10-10 10:06:17 -04:00
MrAlders0n 8df99c641e chore: bump version to 2.0.5 2026-10-10 09:51:33 -04:00
MrAlders0n caaa6a5974 chore(api): raise app floor to 2.0.0 (#219)
The app now versions alongside web and server.
2026-10-10 09:49:44 -04:00
MrAlders0n d788e82ab4 feat(stats): rank top advertisers by flood or direct (#218)
Top advertisers ranked by flood + direct, so chatty companions with few flood adverts topped the list. sort=flood|direct is now required and applied before the limit; client floors raised to app 1.0.0 / web 2.0.4.
2026-10-10 09:38:36 -04:00
MrAlders0n 9bbf8efee5 release: v2.0.4 (#217)
* perf(routes): batch cross-IATA and hash route search (#216)

One search fired a /routes/cross request per IATA pair (138k on 2026-10-08). Route search now takes an optional iatas list and runs a fixed number of index-backed queries. /info advertises built-in minimum client versions.

* chore: bump version to 2.0.4
v2.0.4
2026-10-09 22:17:23 -04:00
MrAlders0n 768889243c release: v2.0.3 (#215)
* chore: add .github/CODEOWNERS for main branch policy

Signed-off-by: MrAlders0n <MrAlders0n@smal.ca>

* chore: add .github/workflows/main-source-guard.yml for main branch policy

Signed-off-by: MrAlders0n <MrAlders0n@smal.ca>

* feat(meshmapper): authenticate API imports (#211)

* feat(observers): add traffic-sorted directory (#212)

* fix(observers): rank available traffic over rolling seven days

* chore: bump version to 2.1.0

* chore: set release version to 2.0.2

* feat(api): add /info with mobile min app version (#214)

* chore: bump version to 2.0.3

---------

Signed-off-by: MrAlders0n <MrAlders0n@smal.ca>
v2.0.3
2026-10-06 21:53:30 -04:00
MrAlders0n 041d9c1f45 release: v2.0.2
Release the changes on `dev` as v2.0.2: authenticated MeshMapper imports and the traffic-sorted observer directory, including rolling seven-day counts from available history. Bump the API version and regenerate Swagger.

MeshMapper integration requires an API key. Regional admins’ existing per-region or grouped-region coverage keys also work for these APIs. Deployments should set `MESHMAPPER_API_KEY` and restart Beacon.

Validation: `go build ./...`, `gofmt -l .` (empty), `go vet ./...`, `go test ./...`, `govulncheck ./...`, and race tests for MeshMapper and scopestore passed in a clean checkout. PostgreSQL integration tests require CI because Docker is unavailable locally.
v2.0.2
2026-10-05 18:35:41 -04:00
MrAlders0n 3828e689ed chore: add .github/workflows/main-source-guard.yml for main branch policy
Signed-off-by: MrAlders0n <MrAlders0n@smal.ca>
2026-10-03 20:05:59 -04:00
MrAlders0n 21b526969b chore: add .github/CODEOWNERS for main branch policy
Signed-off-by: MrAlders0n <MrAlders0n@smal.ca>
2026-10-03 20:05:58 -04:00
MrAlders0n 0dca03cd4d release: v2.0.1 (#209)
- fix(meshmapper): refresh the longest-waiting scope catalogue first
- feat(config): default scope catalogue refresh to 24h
- chore: bump version
v2.0.1
2026-10-03 16:51:11 -04:00
MrAlders0n 001543032f Merge pull request #207 from MeshCore-Beacon/dev
release: v2.0.0
v2.0.0
2026-10-02 22:52:26 -04:00
MrAlders0n 1f83df8583 docs: move operator docs to beacon-docs (#206)
* docs: keep only server-specific contributor notes

* docs: trim README to the code and link to beacon-docs

* chore: strip tracking parameters from two airport website URLs
2026-10-02 21:43:03 -04:00
MrAlders0n be35b36679 docs: condense backup, README and contributor docs (#205) 2026-10-02 21:42:17 -04:00
MrAlders0n af20bebfe5 docs: document route cursorId and activity coverage fields 2026-10-02 19:03:46 -04:00
MrAlders0n ff8758b932 fix(db): read observer activity raw rows back to the rollup watermark
Hourly activity read raw rows only 3h back, so a longer rollup backlog left
hours drawn as zero. The raw tail now starts at the newest complete rollup
hour, capped at the 24h raw holdback, and the response reports rolledUntil
and rawFrom so clients can show anything still uncovered.
2026-10-02 19:03:46 -04:00
MrAlders0n 4ad869d711 fix(db): page known routes by (last_seen, id)
The cursor was the last row's last_seen in ms, so routes sharing that
millisecond (common within one upsert batch) never came back. GET /routes
takes an optional cursorId, backed by keyset indexes; cursor alone keeps
working as before.
2026-10-02 19:03:46 -04:00
MrAlders0n a1aff810e9 fix(api): answer 404 only for missing rows
Lookups turned every error into a 404, and region filters into a 400, so a
transient database failure looked final to clients that don't retry 4xx.
Missing rows stay 404/400; anything else is a 500. Empty /regions and
/iatas lists are [] rather than null.
2026-10-02 19:03:46 -04:00
MrAlders0n 5b49356437 docs: update for 2.0.0 2026-10-02 16:53:32 -04:00
MrAlders0n 951e0b0de6 docs: regenerate swagger 2026-10-02 16:53:32 -04:00
MrAlders0n ec0ecd321b refactor(db): move the heard-IATA lookup into sqlc 2026-10-02 16:53:32 -04:00
MrAlders0n b30668bac4 fix(ws): keep the newest event on overflow and apply route/observer filters
A full send buffer dropped the oldest event and then the new one too while
reporting one drop. Subscribe accepted routeTypes and observerIds but never
applied them.
2026-10-02 16:53:32 -04:00
MrAlders0n 2f4f92e31e fix(db): tie-safe trace paging, atomic migrations and honest stats windows
GET /traces takes an optional cursorTag so tags sharing a millisecond survive
paging. Migrations run under an advisory lock and record each file in the
same implicit transaction, so a crash can't strand a fresh database.
Overview windowHours counts the hours it covers, hourly observer activity
fills unrolled hours from raw rows, and integration fixtures are UTC.
2026-10-02 16:53:32 -04:00
MrAlders0n ec0037860c fix(meshmapper): count cut-short requests and qualify groups by heard IATAs
Record each attempt before sending so a request lost to a deadline still
waits out MeshMapper's limit, and stop holding the directory lock across
HTTP. Groups now qualify only through heard or configured IATAs, and one
missing country list only skips pruning.
2026-10-02 16:53:32 -04:00
MrAlders0n 52fc3e1e48 fix(ingest): validate topic keys and harden against malformed input
A short topic pubkey panicked a lane worker and killed the process; keys must
now be 32 bytes and lane workers recover. Also reject non-finite radio values
before airtime, strip NULs from advert names and neighbour scopes, guard the
MQTT client pointer, and never stream a broker copy of a hearing as a repeat.
2026-10-02 16:53:32 -04:00
MrAlders0n 5ccb1a2ce3 fix(api): warn on untrusted proxies and tighten config validation
Warn at startup when limits are on without trusted_proxies, and once when a
request carries forwarding headers Beacon ignores. Reject short or whitespace
admin keys and negative durations. Key default-window stats by hour, return
500 for scope lookup failures, and validate IATA path params.
2026-10-02 16:53:32 -04:00
MrAlders0n 2042c1617f feat(stats): roll hourly scope observers and advertising nodes 2026-10-02 15:24:33 -04:00
MrAlders0n 53d9f86dd4 feat(stats): split /stats/scopes packet counts by hour 2026-10-02 13:57:54 -04:00
MrAlders0n 3e0b6c074c feat(stats): report the longest path per hour in /stats/paths 2026-10-02 13:57:54 -04:00
MrAlders0n 98006a9364 feat(stats): serve historical stats from hourly rollups
Stats endpoints read hourly rollup tables filled once per closed hour by one
advisory-locked task, instead of materialized views that re-scanned every
observation on each refresh. IATA-set tables give exact distinct counts for any
IATA set or region. Cleanup holds raw rows for unrolled hours (up to 24h).

Adds /stats/series; ingest keeps packet observation counts, observer last IATA,
channel message counts and trace tag summaries. 001_baseline.sql is rewritten,
so pre-release 2.0 databases must be recreated. See docs/historical-stats.md.
2026-10-02 07:43:56 -04:00
MrAlders0n 6cd03e73da fix(db): refresh saved-route hash prefix on re-hearing 2026-10-01 12:14:36 -04:00
MrAlders0n 119be44c0d feat(channels): filter channel list by keyKnown
Also seed the cursor integration test with MeshMapper membership; it still
used channel_iatas after channel lists moved to region scoping.
2026-10-01 09:06:50 -04:00
MrAlders0n 27ca84f51e fix(meshmapper): allow MeshMapper's 60s request timeout 2026-10-01 06:05:42 -04:00
MrAlders0n 7392fdd2d6 fix(meshmapper): create group member IATAs and import zone names and centers 2026-09-30 23:21:44 -04:00
MrAlders0n f8ee1ac884 feat(meshmapper): import regional channels and scope channel lists by region 2026-09-30 23:09:40 -04:00