Commit Graph
520 Commits
Author SHA1 Message Date
MrAlders0n 826fa942e6 Count every timeout 503 in beacon_api_unavailable_total (#281)
* Count every timeout 503 in beacon_api_unavailable_total

* Check each model's timeout count; clarify respondReaderError
2026-10-24 15:29:11 -04:00
MrAlders0n 1b7539f2f3 Seek region and message pages from their window, drop the slow pool (#279)
* Seek region and message pages from their window, drop the slow pool

* Bound typed and scoped region pages and message since pages, migration 059

* Count the late rows in the zero-hit type pair read bound

* Cap typed window ranges, broader bounded cmi verification
2026-10-24 15:09:09 -04:00
MrAlders0n 23aa42badf Wait for every background job on shutdown, with bounded waits (#286)
* Wait for every background job on shutdown, with bounded waits; detach the observer cleanup flush

* Cleanup flush bound matches the ticker flush; no catch-up error for a shutdown
2026-10-24 14:48:26 -04:00
MrAlders0n c3350953e5 Re-hearing a hash-only channel with a nil fingerprint updates its row (#284) 2026-10-24 14:41:14 -04:00
MrAlders0n 3d5df344c1 Finish presence flushes before the pools close on shutdown (#285)
* Finish presence flushes before the pools close on shutdown

* Ticker flushes outlive a shutdown that starts mid-flush; test and comment fixes
2026-10-24 14:28:23 -04:00
MrAlders0n 2b6b01661d Store a node's full clock drift: widen it to bigint (#283) 2026-10-24 14:23:44 -04:00
MrAlders0n f845b137d0 Make the decrypt backfill merge test deterministic (#282) 2026-10-24 14:10:30 -04:00
MrAlders0n 1967573201 Gate the global hop page on the hop group's estimate (#278)
* Gate the global hop page on the hop group's estimate, keeping today's statement for large groups

* Decide the global hop branch in the store from cached hop statistics

* Planguard test corpora pass the global hop page's read_group

* Read hop statistics outside the lock on their own context; planner clamps; both branches in tests and corpus
2026-10-24 13:52:03 -04:00
MrAlders0n 4483c6de3d Sample the audit and verification slices by row; count payload-type progress in rows (#280)
* Sample the audit and verification slices by row; count payload-type progress in rows

* Read sample statistics outside the verify transaction; anchor payload-type progress at id 0
2026-10-24 13:28:15 -04:00
MrAlders0n 02bcd3c4c3 Leave room on nodes pages for HOT last_seen updates (migration 058) (#277)
* Leave room on nodes pages for HOT last_seen updates

* Budgets for node reads on half-full nodes pages

* Rebudget the remaining node reads; overlay header names 058
2026-10-24 12:55:56 -04:00
MrAlders0n 6c812c2b5c Slice the packet_sites fill by observations and count progress in rows (#274)
* Slice the packet_sites fill by observations and count progress in rows

* Keep one set of id statistics per fill; count new ids one row each
2026-10-24 12:45:11 -04:00
MrAlders0n 8c89b62691 Run the channel message backfill in the background after startup (#276)
* Run the channel message backfill in the background after startup

* Finish each backfilled packet; mark packets whose message exists; log a shutdown stop

* Cap each backfilled packet's statements at 30 s
2026-10-24 12:33:38 -04:00
MrAlders0n 4bd1432d81 Run all-region top nodes as its own statement with two parallel workers (#275)
* Run all-region top nodes as its own statement with two parallel workers

* Check the parallel statement returns its own connection to the pool

* Rebaseline the pnw top-nodes entry for the probe without its all-region branch
2026-10-24 12:28:20 -04:00
MrAlders0n 1c89ba19a8 Write hourly rollup rows in primary-key order (#273)
* Write hourly rollup rows in primary-key order

* Cover talker sets in the rollup order test; local baseline for the pnw entry
2026-10-24 11:55:16 -04:00
MrAlders0n 70f3ac552f Run on Alpine 3.24 and test backups in the runtime image (#272) 2026-10-24 11:55:07 -04:00
MrAlders0n 885b7e60cb Answer statement timeouts with 503 on every read endpoint (#271)
* Answer statement timeouts with 503 on every read endpoint

* Keep the plan check on 500; log the cause on read-model pages too
2026-10-24 11:54:58 -04:00
MrAlders0n 080141145e Read-only scope lists, 503 on statement timeouts, log read-model restores (#264)
Read-only servers now restore the saved MeshMapper scope catalogues (a read-only
step), so region scope lists are no longer empty. A statement that hits the
pool's statement_timeout answers 503 with Retry-After on route evidence and the
read-model pages instead of 500. Read-model trust logs when it comes back after
a drop, and when repair runs drop it.
2026-10-21 11:28:55 -04:00
MrAlders0n d4b97cf440 Require Beacon Web 2.1 (#261) 2026-10-21 03:50:44 -04:00
MrAlders0n ff2a6b9e23 Build with Go 1.26.10 (#262) 2026-10-21 03:37:28 -04:00
MrAlders0n 433ad0234a Observer comparison: read each observer's receptions on their own (#260)
* Read each observer's receptions on their own in the observer comparison

* Tighten the observer comparison plan checks
2026-10-20 18:47:08 -04:00
MrAlders0n a51d950e1f Pace the read-model backfills so they cannot starve a small server's disk (#257)
* Pace the one-off backfills by duty cycle so they cannot starve a small server's disk

* Plan the read-model fills with custom plans and analyze new read-model tables

* Analyze only never-analyzed read-model tables, bound the fill pause, and reset fill progress
2026-10-20 12:16:31 -04:00
MrAlders0n 3dd1dc1cd0 Log 503s that carry Retry-After at warn, not error (#251) 2026-10-20 07:31:42 -04:00
MrAlders0n 100e35cb51 Drop idx_observations_observer_id: advert pages read the partial adverts index (#250)
* perf(db): drop idx_observations_observer_id; advert pages read the partial adverts index

* test(planguard): retired-index rule, end-state overlay for the local tier, budgets kept on degraded entries
2026-10-20 06:23:40 -04:00
MrAlders0n 2b1ad1d6c6 feat(api): region, scope and advert pages answer 503 until their read model is ready, instead of falling back (#249) 2026-10-20 03:33:34 -04:00
MrAlders0n 0a41d935a0 End-state cleanup: slow pool, lookup-index and column drops, sequence headroom (#248)
* feat(db): drop six lookup-table indexes that no plan reads

* feat(db): drop known_routes.last_reconfirmed_at, which nothing reads

* feat(metrics): report how much of each database sequence is used

* refactor(db): retire the legacy API pool; slow region and message statements get their own two connections

* feat(config): database.slow_pool_size sizes the slow pool

* refactor(db): route the two former read-model fallbacks to the slow pool

* fix(metrics): sequence headroom reads any schema name and marks cycling sequences

* feat(metrics): count the 503s pages answer, by reason and model; warn about a busy slow pool once a minute

* refactor(db): expose the adverts page mapping and the original channel message pages for oracles

* refactor(db): name the read-model fallbacks the slow pool runs; count only what this tree refuses

* feat(config): the slow pool defaults to 8 connections while it also serves untrusted read-model pages
2026-10-20 03:19:49 -04:00
MrAlders0n 16fb706d17 Packet sites: a read model that serves region packet pages while trusted (#247)
* feat(db): packet_sites table and indexes

* feat(ingest): keep each packet's site row in the observation insert

* feat(db): packet_sites backfill, reconcile and audit

* feat(db): region packet pages from packet_sites when trusted, with an exact page cursor

* feat(db): packet_sites front sweeper with plan and liveness metrics

* feat: boot repair, reconcile, audit, sweep and first-time verification for packet_sites

* fix(db): read no site range when no packet was first heard in the window

* fix(db): read the window guard's maxima through indexes

* test(planguard): packet_sites statements active, data-derived cases, diff and storediff

* docs(api): pageCursor applies to region packet pages

* fix(db): drop packet_sites trust again after the unsited fallback write

* fix(db): the packet_sites audit refills a pair at its newest hearing

* fix(db): bounded region reads fill the page within the request, else today's statement answers

* fix(planguard): tie_cut needs both pages at the limit and members of other identities

* fix: packet_sites sweep without JIT, one verification VACUUM, read-only checks to the newest id

* docs(api): cursor and pageCursor on region pages served without nextPageCursor

* test: drop local baselines of entries the copy skips; plain race schedule names

* test(planguard): storediff walks of the bounded region shapes; local baselines after the overlay rebuild

* fix(db): drop message-site trust again after the unseeded insert

* fix(db): a region page cursor's fallback to today's statement continues from its exact position
2026-10-20 00:18:33 -04:00
MrAlders0n 5ded8d925f Channel messages by site: a read model that serves region and scope pages while trusted (#246)
* feat(db): channel_message_iatas table and indexes

* feat(ingest): record channel message sites at insert and on every later hearing

* feat(db): channel message site backfill and two-range reconcile

* feat(db): serve region and scope message pages from the site read model when trusted

* feat: boot repair, periodic reconcile and first-time verification for the message site read model

* test(planguard): channel message site statements, diff and storediff cases

* feat(background): log when a read-only server trusts the message site read model

* test(planguard): storediff vars in query values, and message cases read from the data

* fix(ingest): record a stored hearing's channel message site right after the observation insert

* fix: bounded message dispatch for filtered all-channel pages, fresher read-only trust, earlier invalidation

* fix: record a hearing's site after its path is marked and skip the backstop when the message was found; read-only trust ends at the writer's deadline
2026-10-19 21:20:21 -04:00
MrAlders0n 232588c361 fix(api): reject list params with more than 1000 values (#245) 2026-10-19 19:47:01 -04:00
MrAlders0n d4e85cbf90 feat(background): mark the payload-type backfill done at once when no observation is untyped (#244) 2026-10-19 19:44:22 -04:00
MrAlders0n ded8708fec Stats, adverts, traces and telemetry: bounded statements, same results (#243)
* feat(db): observer adverts on a partial index once observations carry their payload type

* fix(db): keep today's adverts statement unguarded and rate-limit the gate's read warning

* feat(background): batched, resumable payload-type backfill for untyped observations

* fix(background): cap each payload-type backfill statement at 50,000 rows

* fix(db): reject a non-positive backfill row cap

* perf(db): probe rollups per hour and IATA for region-sized stats lists

* fix(db): keep the stats probes inside each rollup table's hour range

* test(db): push the far-future until to year 9999 so a missing path-stats end bound times out

* perf(db): row-comparison trace cursor and a since-only telemetry statement

* test(db): pin the empty trace cursor tag and run the trace pagination tests in CI

* test(planguard): stats, adverts, traces and telemetry statements leave pending; diff cases

* test(planguard): window-bounded advert roles, short quiet page, per-observer telemetry cursors
2026-10-19 19:18:45 -04:00
MrAlders0n 41baee9197 Packets: one statement per filter shape, exact pageCursor on packets, nodes and observers (2.1.0) (#241)
* feat(db): packet type domains, row-cursor indexes and the inverted-heard index

* feat(api): exact page cursors for packets, nodes and observers

* feat(db): normalise packet list filters before choosing a statement

* feat(db): one packets statement per filter shape with an exact row cursor

* fix(db): let the window statements decide since after until

* feat(db): exact keyset cursors for nodes and observers; observers page by last_seen

* feat(api): pageCursor on packets, nodes and observers; it wins over cursor

* test(planguard): packets statements and node/observer keysets leave pending; diff and storediff cases

* chore: bump version to 2.1.0

* test(db): turn off JIT in the keyset cursor tests
2026-10-19 15:59:55 -04:00
MrAlders0n 94b4b40f9b planguard: plan and result regression checks, fixture tier in CI, read-only mode (#240)
* feat(planguard): read sqlc statement text and load a strict plan corpus

* fix(planguard): fail on unextracted statements, keep fragments mode-neutral

* feat(planguard): plan assertions, scoped sort exemptions and narrow retired-index verdicts

* fix(planguard): an errored run always fails

* fix(planguard): keep subplan conds off their parent scan; fail on unanalyzed budgets

* feat(planguard): read-only plan runner, reports and the run command

* fix(planguard): validate identifiers in emitted scripts; prove read-only runs

* feat(planguard): end-state corpus, planted regressions and the generated cached-plan policy

* fix(planguard): derive observer cases from data; mark legacy generic plans

* fix(planguard): bound the dense-region route resolvers; test policy refusals

* fix(planguard): re-measure the dense-region route cases; cover every policy refusal

* feat(planguard): synthetic fixture built through the migrations, checked in CI

* ci(planguard): build the fixture once; guard fixture data for identifiers

* fix(planguard): hash every fixture generator input; harden the fixture guard

* feat(planguard): snapshot diff over every column, named result classes and cursor walks

* fix(planguard): bound the relaxed diff classes; derive the cursor tie from data

* fix(planguard): refuse multi-statement text in the corpus runner

* feat: read-only server mode and an HTTP page diff between two builds

* fix(planguard): refuse escape strings and psql meta-commands in corpus text

* fix: storediff follows each build's own cursor; prove read-only gates via the log

* fix(planguard): match the PostgreSQL lexer on high-bit bytes; check every emitted line

* feat(admin): daily read-only plan check on the maintenance pool, its metric and report

* feat(planguard): daily prod check behind planguard.daily_check

* fix(planguard): cancel fresh runs at shutdown; expose a never-ran check

* ci(planguard): pin the fixture server and collation; refresh baselines

* fix(planguard): calibrate route budgets on one dataset; skip entries whose index is absent
2026-10-19 13:45:39 -04:00
MrAlders0n dba66cada7 Small tables: drop HOT-blocking indexes, channel write-through (#239)
* perf(db): drop small-table indexes that block HOT updates

* perf(db): touch known channels by id instead of upserting

* feat: preload channel ids at boot for ingest and maintenance
2026-10-19 08:28:15 -04:00
MrAlders0n 03f66d1ae4 Database pools by workload, plan router, observe-only guards, metrics (rest of #237) (#238)
* feat(db): five pgx pools with fixed session settings and Prometheus tracing

* feat(guards): per-family observe/enforce guards with client slots, deadlines and a shadow budget

* fix(guards): attribute only guard deadlines and classify like the handlers

* feat(db): route API statements by sqlc name; class C always unnamed

* feat(admin): statement statistics by sqlc name; create pg_stat_statements where allowed

* fix(db): skip pg_stat_statements when no schema is creatable

* feat(background): index health against post-rebuild baselines, Postgres activity, job and ingest metrics

* feat: separate API, ingest and maintenance stores; guards on the public API; metrics listener

* fix(db): keep ingest writes waiting, size the ingest pool to the host, tidy

* test(db): give the statements probe a query id of its own
2026-10-19 07:52:51 -04:00
MrAlders0n 3ae3c250d8 feat(config): plan layout, guard and admin listener settings; plan policy classes (#237) 2026-10-19 07:43:49 -04:00
MrAlders0n c8595424e9 Drop three route indexes no statement reads (#236)
* perf(db): drop three route indexes no statement reads

* docs(db): reword the retention residual comment
2026-10-19 05:46:20 -04:00
MrAlders0n 8e0fd4b535 Routes: one statement per filter shape, exact bounded retention, stamp-free reconfirm (#235)
* feat(db): background_state table and the (hop_count, iata) route index

* perf(db): one /routes statement per filter shape, hop pair on custom plans

* feat(db): keyset route retention and grace slices at exact cutoffs

* feat(background): hourly route retention with a bounded grace walk and watermark

* fix(background): bound the grace walk at microsecond cutoffs

* feat(db): stamp-free reconfirm sweep over the route primary key

* feat(background): pace route reconfirm at routes.reconfirm_rows_per_hour, no stamps

* test(routes): drive route retention end to end and tidy comments

* fix(profiling): sample during the hourly route retention run
2026-10-19 05:39:38 -04:00
MrAlders0n c27f721dd2 Route search hop order, node-types split, bounded migration lock waits (#234)
* perf(routes): keep hop_count indexes out of route search order

* perf(stats): count node types without node_iatas when unfiltered

* perf(db): drop the unused idx_known_routes_iata_last_seen

* fix(db): bound boot migration lock waits and name the blockers

* test(db): pin the lock-retry rebuild and tidy comments
2026-10-19 03:58:37 -04:00
MrAlders0n d5aacce3c7 fix(db): bigint id cursors before observation ids pass 2^31 (#233)
* fix(db): type id cursors as bigint

* chore: bump version to 2.0.8
2026-10-19 03:33:40 -04:00
MrAlders0n 127fc76092 Raise the default per-client API limit to 600 a minute (#232)
Opening Map and Nodes on a region of a few thousand nodes could spend the
whole 300-request minute and leave the map half loaded. 600 still sits
under the fail2ban flood jail's 900, so the server's 429 answers first.
2026-10-18 23:12:57 -04:00
MrAlders0n 202acc6cc8 Merge pull request #228 from MeshCore-Beacon/dev
release: v2.0.7
v2.0.7
2026-10-14 17:19:52 -04:00
MrAlders0n b43da0c3a9 chore: bump version to 2.0.7 2026-10-14 17:10:50 -04:00
MrAlders0n 632b83c4b9 feat(api): search packets by hash prefix (#227) 2026-10-14 17:04:22 -04:00
MrAlders0n bc84be577c Merge pull request #222 from MeshCore-Beacon/dev
release: v2.0.6
v2.0.6
2026-10-12 13:32:55 -04:00
MrAlders0n 354dad6925 chore: bump version to 2.0.6 2026-10-12 13:31:33 -04:00
MrAlders0n 7297215555 feat(meshmapper): import repeater site details for node detail (#221) 2026-10-11 14:18:26 -04:00
MrAlders0n 570a72b614 release: v2.0.5 (#220)
release: v2.0.5
v2.0.5
2026-10-10 10:06:17 -04:00
MrAlders0n 8df99c641e chore: bump version to 2.0.5 2026-10-10 09:51:33 -04:00
MrAlders0n caaa6a5974 chore(api): raise app floor to 2.0.0 (#219)
The app now versions alongside web and server.
2026-10-10 09:49:44 -04:00
MrAlders0n d788e82ab4 feat(stats): rank top advertisers by flood or direct (#218)
Top advertisers ranked by flood + direct, so chatty companions with few flood adverts topped the list. sort=flood|direct is now required and applied before the limit; client floors raised to app 1.0.0 / web 2.0.4.
2026-10-10 09:38:36 -04:00