From 3775747d0346e595d8d30ed729ac7713c768f8a6 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 11:55:46 -0400 Subject: [PATCH 01/21] Document 503 on statement timeouts for every read endpoint (#46) * Document 503 on statement timeouts for every read endpoint * Name the admin routes that keep their own errors --- docs/api-contract.md | 4 ++-- docs/configuration.md | 2 +- docs/operations.md | 4 ++-- docs/upgrading.md | 15 +++++++++++++-- 4 files changed, 18 insertions(+), 7 deletions(-) diff --git a/docs/api-contract.md b/docs/api-contract.md index 930885f..3b7e4e7 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -60,7 +60,7 @@ Retry-After: 5 The body is the usual error envelope. Wait at least `Retry-After` seconds before retrying; a `deadline_exceeded` request will likely run as long again, so narrow it (a shorter window, a region) rather than repeat it. Clients are keyed by IP after the trusted-proxy step above. -Separately from the guards, and on every server, a page whose database statement runs past its pool's statement limit (35 s; 120 s for the slow region and message statements) answers `503` with code `deadline_exceeded` and `Retry-After: 10`. This covers route evidence (`/routes/{iata}/{pathKey}/observations`) and the [background-data pages](#pages-that-wait-for-server-data) (region packets, region and scope messages, observer adverts). Other endpoints still answer `500` in that case. +Separately from the guards, and on every server, a page whose database statement runs past its pool's statement limit (35 s; 120 s for the slow region and message statements) answers `503` with code `deadline_exceeded` and `Retry-After: 10`. This covers every `/api/v1` endpoint that reads the database; account, backup and plan-check admin routes keep their own errors. The heavy stats endpoints (`series`, `signal`, `paths`, `observer-comparison`) and route searches (`/routes/search`, `/routes/cross`) answer the same `503` when their own request time limit runs out, with a message that names the query. Any other failed read is a `500` without `Retry-After`. Each request falls in one family and one class: @@ -117,7 +117,7 @@ All errors use one shape: { "error": { "code": "not_found", "message": "packet not found" } } ``` -`code` is the HTTP status text in snake_case, so match on it or on the status. Codes in use: `bad_request`, `unauthorized`, `not_found`, `conflict`, `request_entity_too_large`, `unsupported_media_type`, `rate_limited`, `internal_server_error`, `service_unavailable`, `gateway_timeout`, `insufficient_storage`, and `deadline_exceeded` (a `503` from the [cost guards](#cost-guards)). A `503` `service_unavailable` with `Retry-After` can also mean a page is [waiting for server data](#pages-that-wait-for-server-data). The heavy stats endpoints (`series`, `signal`, `paths`, `observer-comparison`) return `503` when the query times out; try a shorter window. +`code` is the HTTP status text in snake_case, so match on it or on the status. Codes in use: `bad_request`, `unauthorized`, `not_found`, `conflict`, `request_entity_too_large`, `unsupported_media_type`, `rate_limited`, `internal_server_error`, `service_unavailable`, `gateway_timeout`, `insufficient_storage`, and `deadline_exceeded` (a `503` from the [cost guards](#cost-guards)). A `503` `service_unavailable` with `Retry-After` can also mean a page is [waiting for server data](#pages-that-wait-for-server-data). A `503` `deadline_exceeded` with `Retry-After` from any endpoint means its query ran out of time; retry later or ask for a shorter window. A `404` means the requested thing doesn't exist. A database or other server-side failure is always a `500` (or `503` above), never a `404`, so clients can retry 5xx and treat 4xx as final. Empty lists are `200` with `[]`. diff --git a/docs/configuration.md b/docs/configuration.md index 1820cf0..dce6994 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -155,7 +155,7 @@ of budget, and the all-channel message statement for `iatas` combined with `sinc Two connections by default (`database.slow_pool_size`), a 120 s `statement_timeout`, no parallel workers. A request that finds every connection busy waits at most 5 s and then answers `503` with `Retry-After: 5`; it never takes a connection from `api_custom`. A statement that runs past -120 s ends with a `500`. +120 s answers `503` `deadline_exceeded` with `Retry-After: 10`. ```yaml database: diff --git a/docs/operations.md b/docs/operations.md index 0f9db40..a6ca387 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -964,8 +964,8 @@ They run on their own pool, `api_slow`: 2 connections by default connection is busy a request waits at most 5 s and then answers `503` with `Retry-After: 5`, without touching the other pools; `beacon_api_unavailable_total{reason="slow_pool_busy"}` counts those, and the server logs `api: slow requests refused, every slow connection is busy` at most -once a minute. A statement that passes 120 s ends with a `500`; narrowing the request (fewer -sites, a wider window) avoids it. +once a minute. A statement that passes 120 s answers `503` `deadline_exceeded` with +`Retry-After: 10`; narrowing the request (fewer sites, a wider window) avoids it. Before this release the region statement ran on `api_legacy` without a time limit for a millisecond cursor, and on `api_custom` (35 s, gated by `packets_region`) for an exact page diff --git a/docs/upgrading.md b/docs/upgrading.md index 1b2ec52..f20c4e7 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -318,10 +318,21 @@ Region `/packets` pages, channel message pages filtered by `iatas` (or by `scope `since`), and observer advert pages are served only from their read model or backfilled data. While that is not ready they answer `503` with `Retry-After: 30`; there is no fallback to the previous statements. The kill switches above turn those pages off instead of serving them from -the previous statements. A page whose statement runs past its pool's limit answers `503` with -`Retry-After: 10`. To get the fallbacks back, revert their one server commit; see +the previous statements. To get the fallbacks back, revert their one server commit; see [Reverting to the read-model fallbacks](operations.md#reverting-to-the-read-model-fallbacks). +#### Statement timeouts answer 503 + +Every `/api/v1` read whose statement runs past its pool's limit (35 s; 120 s on `api_slow`) +now answers `503` `deadline_exceeded` with `Retry-After: 10` instead of a `500`. The heavy +stats endpoints and route searches already answered `503` when their own time limit ran out; +that `503` now carries `Retry-After: 10` and code `deadline_exceeded` (was +`service_unavailable`), and also covers a statement timeout. Web and app retry it like the other +`503`s. In the server log a timeout is one `request complete` line at `WARN` with `status=503` +instead of an `ERROR` pair; a real `500` still logs `api: error 500: ` at `ERROR`, now +with the database error in `error`. Only pages that read a read model count timeouts in +`beacon_api_unavailable_total{reason="timeout"}`. + ## Changes that need a restart Beacon reads `config.yaml` once at startup. After editing it, run From 0646dd581b526ffc38c62f5ae98a125e0a9b5439 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 11:56:22 -0400 Subject: [PATCH 02/21] Explain why some observations outlive packets.retention (#47) --- docs/operations.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/operations.md b/docs/operations.md index a6ca387..d20ff8b 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -76,6 +76,15 @@ minutes after it closes, and packet cleanup holds back up to 24 hours of raw row hours are rolled, so raw data lives a little longer than `packets.retention` says. An hour whose raw rows were deleted first is reported as partial, with no values. +`packets.retention` counts from the last time any observer heard a packet, and a packet goes +together with all its observations and its channel message. A packet that is still being heard +keeps its older observations, so `packet_observations` always holds a small tail of rows older +than the retention, with ids far below the current range. Most of it is byte-identical packets +that nodes send again and again, such as adverts from nodes whose clock was never set (their +advert timestamp stays the same, so every advert has the same hash). A later hearing by an +observer that already reported the packet stores no new observation, but it still counts as +heard. On a 72 h retention this tail was about 0.5 % of observations. + ## Backing up Two options. From f3318c12c6439e109c83ec4004c81c8b10aa00a4 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 12:04:46 -0400 Subject: [PATCH 03/21] Document the Alpine 3.24 runtime image (#48) --- docs/backup-export.md | 6 ++++-- docs/upgrading.md | 7 +++++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/backup-export.md b/docs/backup-export.md index 991ca3e..cdb73a9 100644 --- a/docs/backup-export.md +++ b/docs/backup-export.md @@ -29,8 +29,10 @@ changes, PostgreSQL roles and cluster settings, Redis and deployment files are n - `pg_dump` must be in the **same runtime** as the exporter (the CLI's environment, or the server's PATH for the download), with a major version at least as new as the database's. A client on the Docker host or in another container doesn't count. -- The Docker image ships the PostgreSQL 16 client. Its Alpine 3.19 base has no 17/18 - packages, so a newer database needs a different runtime, not just `POSTGRES_CLIENT_MAJOR`. +- The published Docker image ships the PostgreSQL 16 client on an Alpine 3.24 base. For a + PostgreSQL 17 or 18 database, build your own image from a beacon-server checkout, e.g. + `docker build -f .build/Dockerfile --build-arg POSTGRES_CLIENT_MAJOR=17 -t beacon:pg17 .`, + and point the stack's `image:` at that tag. - The temp directory (`TMPDIR`) needs about twice the SQL limit free. A small tmpfs won't do. - The output filesystem must support hard links (used to publish atomically). Existing destinations, symlinks included, are never replaced. Files are 0600 and staging dirs 0700; diff --git a/docs/upgrading.md b/docs/upgrading.md index f20c4e7..006e835 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -333,6 +333,13 @@ instead of an `ERROR` pair; a real `500` still logs `api: error 500: ` with the database error in `error`. Only pages that read a read model count timeouts in `beacon_api_unavailable_total{reason="timeout"}`. +#### Runtime image on Alpine 3.24 + +The server image moved from Alpine 3.19, which is out of support, to Alpine 3.24 with the +PostgreSQL 16 client tools. Nothing to change for a PostgreSQL 16 database. A self-built image +with `POSTGRES_CLIENT_MAJOR=17` or `18` now works too; see +[Backup export](backup-export.md). + ## Changes that need a restart Beacon reads `config.yaml` once at startup. After editing it, run From 4fe3a2827c8a06f361ea90373a7b2370e39bc365 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 12:29:41 -0400 Subject: [PATCH 04/21] Top nodes over every IATA use two parallel workers; rollup row order; node type pages trade (#50) * Document parallel workers for all-region top nodes and the node type page trade * Document the rollup row order and its optional one-off CLUSTER --- docs/operations.md | 43 ++++++++++++++++++++++++++++++++++++++++--- docs/upgrading.md | 4 ++++ 2 files changed, 44 insertions(+), 3 deletions(-) diff --git a/docs/operations.md b/docs/operations.md index d20ff8b..cdbcb8c 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -478,9 +478,13 @@ They are also guarded, but a guard's deadline applies only if its family is list For top nodes, top advertisers and paths, requests with at most 32 distinct IATAs and whole-hour bounds use new statements that read the hourly rollups one hour and IATA at a time. -Only wider lists, such as every IATA at once, still run the previous statements, now without -parallel workers. Measured over all 625 IATAs for 7 days on a copy of a large production -database, they stay far inside the timeout: +Top nodes with no IATA filter run their own statement over the whole window: the only API +statement allowed parallel workers. The server raises `max_parallel_workers_per_gather` to 2 for +that statement alone, inside its own transaction (with 2 workers it took 300-400 ms instead of +620-850 ms over 7 days on the same copy; it is cached for an hour). If the server has no free +worker, the statement runs without one, at the speed shown below. Lists of more than 32 IATAs +still run the previous statements, without parallel workers. Measured over all 625 IATAs for +7 days on a copy of a large production database, they stay far inside the timeout: | Statement | With parallel workers | Without | |---|---:|---:| @@ -491,6 +495,32 @@ database, they stay far inside the timeout: Responses are unchanged. Under `split`, `ListObserverAdverts` and `GetObserverTelemetrySince` run on cached generic plans; see [Configuration](configuration.md#database-pools-and-cost-guards). +### After upgrading: stats rollup row order + +2.1 writes each hourly rollup in its primary-key order, so one hour's rows for an IATA sit on a +few adjacent pages. Older servers wrote them in hash order: one page per row. The per-(hour, IATA) +reads (top nodes, top advertisers and paths with up to 32 IATAs) read about a tenth of the pages +on the new layout: for a 13-IATA region over 7 days, about 11,000 buffers instead of about 116,000. + +Rows written before the upgrade keep the old layout, so these pages get faster as old hours leave +the requested window: fully within 7 days for the default 7-day view, and within 30 days for a +30-day one. Nothing needs to be done. + +**Optional: heal at once.** Rewrite the two tables those reads use, off-peak: + +```sql +CLUSTER analytics_hourly_advert_hearings USING analytics_hourly_advert_hearings_pkey; +CLUSTER analytics_hourly_paths USING analytics_hourly_paths_pkey; +``` + +- Each `CLUSTER` holds an `ACCESS EXCLUSIVE` lock for its whole run. Stats pages that read the + table and the hourly rollup wait for it. +- It needs free disk for a full copy of the table and its indexes until it finishes. +- Measured: 9.6 s for 2.0 million rows (226 MB heap) on a fast workstation. Rollups are kept for + 90 days, so these tables grow by about 0.3 million rows a day on a large deployment. At about + 35 days, expect around 10 million rows (1.3 GB heap) for `analytics_hourly_advert_hearings` and + 1 to 3 minutes per table on a busy server. + ### After upgrading: compacting observers and channels The small-table release (migrations `011` to `024`) drops the indexes that made every observer @@ -553,6 +583,13 @@ snapshots lower it somewhat. **Rollback.** None needed for the rewrite; it changes no data. +Type-filtered node pages (`/nodes` with a node type) are slower in this release on purpose. +They now read the whole `nodes` table like the unfiltered page: about 5-12 ms on a large +deployment's 68,000 nodes, against 1.4 ms with the dropped `(node_type, last_seen)` index. That +index held `last_seen`, which every node advert rewrites (about 1.1 million times a day on that +deployment). In a benchmark of 100,000 `last_seen` updates it added 31 % WAL and 22 % time, more +than doubled the table's index size, and stopped every one of those updates from being HOT. + To go back to the previous server image, just deploy it: it works without the dropped indexes, but type-filtered node pages (`/nodes` with a node type) are slower until the indexes are rebuilt. It ignores the channel id cache and goes back to using a sequence value per diff --git a/docs/upgrading.md b/docs/upgrading.md index 006e835..009f2bd 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -238,6 +238,10 @@ see [Pages that wait for a read model](operations.md#pages-that-wait-for-a-read- - **Optional, large deployments, once, off-peak:** [compact `observers` and `channels`](operations.md#after-upgrading-compacting-observers-and-channels) and [rebuild the route keyset indexes](operations.md#rebuilding-the-route-keyset-indexes). +- **Stats pages.** Top nodes, top advertisers and paths for a region get faster as hours rolled + before the upgrade leave the window: fully within 7 days for the default view. See + [After upgrading: stats rollup row order](operations.md#after-upgrading-stats-rollup-row-order) + for an optional one-off `CLUSTER` that heals it at once. - **Plan checks.** Turn on the [daily plan check](operations.md#turning-on-the-daily-check) only after the upgrade, if you want it. From 2d2b90a61ecf8cd64aa84f76ab53cdce26b981fe Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 12:33:48 -0400 Subject: [PATCH 05/21] Document the background channel message backfill (#51) --- docs/configuration.md | 4 +++- docs/high-level-design.md | 2 +- docs/operations.md | 17 +++++++++++++++++ docs/troubleshooting.md | 6 +++--- docs/upgrading.md | 17 +++++++++++++++-- 5 files changed, 39 insertions(+), 7 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index dce6994..3f78ae4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -298,7 +298,9 @@ writes the tables; it trusts each one only while the writing server keeps it up `readmodels.fill` paces the one-off backfills: the `packet_sites` and `channel_message_iatas` fills after an upgrade or a rebuild, and the -[payload-type backfill](operations.md#after-upgrading-observation-payload-types). After each +[payload-type backfill](operations.md#after-upgrading-observation-payload-types). The +[channel message backfill](operations.md#channel-message-backfill) after each start uses `duty` +only, with no `min_pause`. After each slice or batch a fill sleeps `max(min_pause, elapsed × (1 − duty) / duty)`, where `elapsed` is how long that slice took. A slow disk makes each slice take longer, so it stretches the pauses on its own. Pacing changes only when the fills run, never what they write. diff --git a/docs/high-level-design.md b/docs/high-level-design.md index 268635b..7b70650 100644 --- a/docs/high-level-design.md +++ b/docs/high-level-design.md @@ -761,7 +761,7 @@ retention under 24h) fail startup rather than being ignored. IATAs are still auto-created when packets arrive from unrecognized codes. The config file is only needed to override the default display name/coordinates, or to assign an IATA to a super-region. -Channel hashes also auto-populate as messages arrive; adding a key in the config decrypts the stored hash-only rows for that channel on the next startup. +Channel hashes also auto-populate as messages arrive; adding a key in the config decrypts the stored hash-only rows for that channel after the next startup, in the background. ### Admin API and access control diff --git a/docs/operations.md b/docs/operations.md index cdbcb8c..76212af 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -57,6 +57,23 @@ Everything persistent is under `data/` next to `docker-compose.yml`: - `data/app/config.yaml` and `.env` are your configuration. Keep copies of both somewhere else; no backup below includes `.env`. +## Channel message backfill + +After every start, once the server listens, Beacon retries every stored channel message it +could not decrypt against the configured keys, so a key added to `config.yaml` surfaces its +channel's history. Keys a MeshMapper import adds queue their channels on the same background +job. It runs one pass at a time, in batches of 500 packets, paced by +[`readmodels.fill.duty`](#pacing-the-backfills) alone (no `min_pause`), and stops after 30 +minutes, pauses included; whatever it did not reach is retried after the next start. A packet it +has started is finished (within 30 s), so a stop does not leave a message half stored. The log shows `channel message backfill started` +with `packets`, then `channel message backfill done: N previously-undecrypted channel +message(s)` with `decrypted` and `duration` (or a `WARN` that it stopped after 30 minutes, or +`channel message backfill stopped by shutdown`). +Messages it stores are not broadcast over the WebSocket, and their hours are re-rolled for the +historical stats. A packet heard live before the backfill reaches it is decrypted by ingest +instead: its message is stored once either way, its hours are re-rolled too, and, as a live +message, it is broadcast over the WebSocket with its original `sentAt`. + ## Retention Everything below is a duration in `config.yaml`. Shorten them if disk is tight, lengthen them diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 43b773b..83081f4 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -89,9 +89,9 @@ advertised. This fills in on its own as the mesh is observed. ## Channel messages show a hash but no text The channel's key is not configured, or it was added without a restart. Add it under -`channel_keys:` in `config.yaml` and run `docker compose restart app`. On start, Beacon decrypts -the stored messages for the new key and logs -`config: backfilled N previously-undecrypted channel message(s)`. +`channel_keys:` in `config.yaml` and run `docker compose restart app`. Once it listens, Beacon +decrypts the stored messages for the new key in the background and logs +`channel message backfill done: N previously-undecrypted channel message(s)`. ## MeshMapper imports are unconfigured or stale diff --git a/docs/upgrading.md b/docs/upgrading.md index 009f2bd..05245c1 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -344,13 +344,26 @@ PostgreSQL 16 client tools. Nothing to change for a PostgreSQL 16 database. A se with `POSTGRES_CLIENT_MAJOR=17` or `18` now works too; see [Backup export](backup-export.md). +#### Channel message backfill after startup + +The retry of stored, undecrypted channel messages against the configured keys no longer +delays startup: it runs in the background once the server listens. On a slow disk that was +most of the boot time (reading every undecrypted packet), so MQTT ingest and the API start +sooner. Its log lines changed: `config: backfilled N previously-undecrypted channel +message(s)` is now `channel message backfill started` (with `packets`) and `channel message +backfill done: N previously-undecrypted channel message(s)`, logged on every run. Keys that a +MeshMapper import adds queue on the same job. A message a live hearing decrypts first, before +the backfill reaches its packet, is stored once and its earlier hours are re-rolled, as the +backfill does. + ## Changes that need a restart Beacon reads `config.yaml` once at startup. After editing it, run `docker compose restart app`. On the next start: -- A newly added channel key decrypts the matching stored messages. Look for - `config: backfilled N previously-undecrypted channel message(s)` in the log. +- A newly added channel key decrypts the matching stored messages, in the background once + the server listens. Look for + `channel message backfill done: N previously-undecrypted channel message(s)` in the log. - Border file and MeshMapper import changes take effect. - Changed admin or MeshMapper keys in the YAML file are picked up. From b3f6c70d16df3a8155ceda0e7a4f7569b8dfe559 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 12:45:20 -0400 Subject: [PATCH 06/21] Document packet_sites fill slices by observations (#49) --- docs/configuration.md | 2 +- docs/operations.md | 21 +++++++++++++-------- docs/upgrading.md | 8 ++++++++ 3 files changed, 22 insertions(+), 9 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 3f78ae4..0ea278d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -309,7 +309,7 @@ on its own. Pacing changes only when the fills run, never what they write. |---|---|---|---| | `duty` | `0.5` | 0.01 to 1 | The share of time a fill may be busy. `0.5` waits as long as each slice took; `0.25` waits three times as long; `1` waits only `min_pause`. No single pause is longer than 10 minutes. | | `min_pause` | `100ms` | `0s` to `1m` | The shortest pause after a slice or batch. | -| `packet_sites_slice` | `500000` | 1 to 10,000,000 | Observation ids one `packet_sites` backfill slice covers. | +| `packet_sites_slice` | `500000` | 1 to 10,000,000 | Observations one `packet_sites` backfill slice covers, however far apart their ids are. | | `cmi_batch` | `10000` | 1 to 1,000,000 | Message ids one `channel_message_iatas` backfill batch covers. | The defaults suit a small server. On a slow (rotational or shared) disk, set `duty: 0.25`. A diff --git a/docs/operations.md b/docs/operations.md index 76212af..795422c 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -817,14 +817,19 @@ On the first start the server fills the table in the background: 1. It waits until every transaction that started before boot and holds a transaction id has finished (`packet_sites boot repair waits for transactions from before boot` while it waits), then runs one reconcile from the saved watermark. -2. It backfills the table newest observation first, in slices of 500,000 observation ids +2. It backfills the table newest observation first, in slices of 500,000 observations (`readmodels.fill.packet_sites_slice`), pausing after each slice as [`readmodels.fill`](#pacing-the-backfills) sets: at least 0.1 s, and by default as long as - the slice took, so the fill is busy at most half the time. Once a minute the log shows a - `packet_sites backfill progress` line with `ids_done`, `ids_remaining` and `eta`, then - `packet_sites backfill complete`. On a large deployment (about 766 million ids, 29.4 million - observations) that is about 1,530 slices; expect roughly twice the fill work, about 10 to 40 - minutes, longer on a slow disk or a lower `duty`, and a few GB of WAL. + the slice took, so the fill is busy at most half the time. A slice counts observations, not + ids: the few old observations [retention keeps](#retention) spread over a long id range cost + a few slices, not one per 500,000 ids. Once a minute the log shows a + `packet_sites backfill progress` line with `ids_done`, `ids_remaining`, + `observations_done`, `observations_remaining` and `eta`, then + `packet_sites backfill complete`. The observation counts are the planner's estimates from + the table statistics, and `eta` follows them, so it no longer counts empty ids as work. + On a large deployment (29.4 million observations over about 766 million ids) that is about + 60 slices; expect roughly twice the fill work, about 10 to 40 minutes, longer on a slow disk + or a lower `duty`, and a few GB of WAL. 3. It runs a one-time verification. It first runs `VACUUM (ANALYZE) packet_sites` (a few minutes on a large deployment), then an exact check of two slices of 500,000 observation ids, the newest and one random older one, and a page-by-page comparison of region pages of @@ -991,7 +996,7 @@ logged (`read-model ANALYZE failed; the fill goes on`) and the fill continues; a already has statistics is left to autovacuum. Measured on a laptop Postgres with 3 million synthetic observations (`packet_sites`, slices of -500,000 ids): +500,000 observations, dense ids): | `duty` | Fill time | Busy in fill statements | |---|---|---| @@ -999,7 +1004,7 @@ Measured on a laptop Postgres with 3 million synthetic observations (`packet_sit | `0.5` (default) | 153 s | 50 % | | `0.25` | 316 s | 25 % | -Slices of 100,000 ids cost about 7 % more fill work (85 s at `duty: 1`) but cut the longest +Slices of 100,000 observations cost about 7 % more fill work (85 s at `duty: 1`) but cut the longest burst from about 14 s to about 3 s. Settings, under [`readmodels.fill`](configuration.md#backfill-pacing) in `config.yaml`: diff --git a/docs/upgrading.md b/docs/upgrading.md index 05245c1..48b80f1 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -325,6 +325,14 @@ previous statements. The kill switches above turn those pages off instead of ser the previous statements. To get the fallbacks back, revert their one server commit; see [Reverting to the read-model fallbacks](operations.md#reverting-to-the-read-model-fallbacks). +#### Packet sites fill counts observations + +The `packet_sites` first fill slices by observations instead of observation ids, so the few +old observations far below the live ids no longer cost a paced slice per 500,000 ids, and its +progress line adds `observations_done` and `observations_remaining` (planner estimates) and +bases `eta` on them. `readmodels.fill.packet_sites_slice` keeps its default and range but now +counts observations. + #### Statement timeouts answer 503 Every `/api/v1` read whose statement runs past its pool's limit (35 s; 120 s on `api_slow`) From 56d1fd2e0836b4ed5d96a53f786a74326610bf10 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 12:56:05 -0400 Subject: [PATCH 07/21] Fillfactor 50 on nodes (migration 058) and its one-off compaction (#52) * Document fillfactor 50 on nodes and its one-off compaction * Note migration 058 with the small-table migrations --- docs/operations.md | 30 ++++++++++++++++++++++-------- docs/upgrading.md | 2 +- 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/docs/operations.md b/docs/operations.md index 795422c..e975210 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -191,6 +191,10 @@ four tables during the upgrade; a blocked drop stops startup after about two min described below, and the next start resumes where it stopped. Afterwards, compact the two tables once; see [Compacting observers and channels](#after-upgrading-compacting-observers-and-channels). +Migration `058` sets `fillfactor = 50` on `nodes`. It only changes the table's settings (no +rewrite, a lock that does not block reads or writes), and applies to pages written from then on; +the same compaction section covers `nodes`. + Migrations `025` to `033` (2.1.0) change `packets`: - `025` and `027` add the checks `packets_payload_type_domain` (`payload_type` 0 to 15) and @@ -547,23 +551,32 @@ only applies to pages written from then on. A table that bloated before the upgr size until it is rewritten once: on a large deployment `observers` held about 4,000 rows in 52 MB, and about 6 MB after the rewrite. -**When.** Once, off-peak, after the release is live and its last migration is recorded. This -must return `1`: +2.1 also sets `fillfactor = 50` on `nodes` (migration `058`). Every node advert rewrites its +node's `last_seen` (about 1.1 million updates a day on a large deployment), and with packed pages +almost none of those updates could stay on their page. In a benchmark of 343,000 `last_seen` +updates over 68,700 nodes, fillfactor 50 made 60 % of them HOT instead of 0.5 %, with 47 % less +WAL and 37 % less time. Node page times did not change: the table is about twice its packed size +(about 23 MB instead of 11 MB for 68,700 nodes), a few more cached pages per read. Like the others, +the setting only applies to pages written from then on; the rewrite below compacts `nodes` too. + +**When.** Once, off-peak, after the release is live and its last migration is recorded: ```sql SELECT count(*) FROM schema_migrations WHERE filename LIKE '024_%'; +SELECT count(*) FROM schema_migrations WHERE filename LIKE '058_%'; ``` -Not before: without the fillfactor and the dropped indexes, the tables bloat again. +Both must return `1`. Not before: without the fillfactor and the dropped indexes, the tables +bloat again. **Is it worth it?** Check the sizes in `psql` as above: ```sql SELECT relname, relpages, pg_size_pretty(pg_relation_size(oid)) AS heap, reltuples::bigint AS rows -FROM pg_class WHERE relname IN ('observers', 'channels'); +FROM pg_class WHERE relname IN ('observers', 'channels', 'nodes'); ``` -If both are already a few MB, skip it. +If they are already a few MB (`nodes` about twice its packed size), skip it. **Run** as the table owner (the `beacon` user) or a superuser, each statement on its own in autocommit mode, not inside a transaction block: @@ -572,12 +585,13 @@ autocommit mode, not inside a transaction block: SET lock_timeout = '3s'; VACUUM FULL observers; VACUUM FULL channels; +VACUUM FULL nodes; ``` - Each `VACUUM FULL` rewrites the table and its indexes under an `ACCESS EXCLUSIVE` lock, which blocks reads and writes on that table while it runs. On tables of this size that is a few - seconds, so ingest's observer and channel updates and the observer and channel API pages - wait that long. + seconds, so ingest's observer, channel and node updates and the observer, channel and node + API pages wait that long. - If it reports `canceling statement due to lock timeout`, another session held the table; run the same statement again. - It needs free disk equal to the table's new size plus its indexes, a few MB here. The old @@ -591,7 +605,7 @@ apart and compare the change: ```sql SELECT relname, n_tup_upd, n_tup_hot_upd, round(100.0 * n_tup_hot_upd / nullif(n_tup_upd, 0), 1) AS hot_pct -FROM pg_stat_user_tables WHERE relname IN ('observers', 'channels'); +FROM pg_stat_user_tables WHERE relname IN ('observers', 'channels', 'nodes'); ``` Between the two samples, nearly all new updates (`n_tup_hot_upd` growth over `n_tup_upd` diff --git a/docs/upgrading.md b/docs/upgrading.md index 48b80f1..348c982 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -236,7 +236,7 @@ see [Pages that wait for a read model](operations.md#pages-that-wait-for-a-read- large deployment it takes 12 to 14 minutes in short statements. See [After upgrading: route maintenance](operations.md#after-upgrading-route-maintenance). - **Optional, large deployments, once, off-peak:** - [compact `observers` and `channels`](operations.md#after-upgrading-compacting-observers-and-channels) + [compact `observers`, `channels` and `nodes`](operations.md#after-upgrading-compacting-observers-and-channels) and [rebuild the route keyset indexes](operations.md#rebuilding-the-route-keyset-indexes). - **Stats pages.** Top nodes, top advertisers and paths for a region get faster as hours rolled before the upgrade leave the window: fully within 7 days for the default view. See From 514ac8689e9868a36931b2e864836ab1b84d11e5 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 13:28:24 -0400 Subject: [PATCH 08/21] Document audit and verification slices sampled by row (#54) --- docs/operations.md | 13 +++++++++---- docs/upgrading.md | 5 ++++- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/docs/operations.md b/docs/operations.md index e975210..f40f320 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -439,7 +439,10 @@ typed, each statement reads its span and updates nothing. Until a server process reads `done`, `/observers/{observerId}/adverts` answers `503` with `Retry-After: 30` (the statement that served those pages before `done` is gone). The job logs `observation payload type backfill progress: observer advert pages answer 503 until it is done` -with `next_id`, `max_id`, `ids_done`, `ids_remaining` and `eta` once a minute while it walks. Once a server process reads `done`, +with `next_id`, `max_id`, `ids_done`, `ids_remaining`, `observations_done`, +`observations_remaining` and `eta` once a minute while it walks. The observation counts are the +planner's estimates from the table statistics, and `eta` follows them, so a long, nearly empty +old id range does not count as work. Once a server process reads `done`, it serves the pages from `idx_observations_adverts` and keeps that until restart. On a database whose observations already all carry a payload type (every deployment that started on a release with the column), the job's first run finds no untyped row and records `done` at once @@ -773,7 +776,8 @@ Run this release for at least a week before upgrading to that one. latest hearing. Each row also copies the packet's payload type, route type, scope and first hearing. The observation insert writes the row in the same statement, so a stored observation always has its row. A background reconcile repairs the newest rows every 5 minutes, an hourly -audit refills one random older slice, and a front sweep every 30 seconds keeps the newest +audit refills a random older slice of 500,000 observations (picked by row, so every +observation is as likely to be audited however sparse the old ids are), and a front sweep every 30 seconds keeps the newest index entries dense. On a large deployment (about 7.4 million rows) it takes about 1.6 GB: heap 484 MB, primary key 285 MB, `idx_packet_sites_site` 419 MB, `idx_packet_sites_site_type` 353 MB, `idx_packet_sites_site_scope` 53 MB. @@ -845,8 +849,9 @@ On the first start the server fills the table in the background: 60 slices; expect roughly twice the fill work, about 10 to 40 minutes, longer on a slow disk or a lower `duty`, and a few GB of WAL. 3. It runs a one-time verification. It first runs `VACUUM (ANALYZE) packet_sites` (a few - minutes on a large deployment), then an exact check of two slices of 500,000 observation - ids, the newest and one random older one, and a page-by-page comparison of region pages of + minutes on a large deployment), then an exact check of the newest 500,000 observation ids + and of a random 500,000 older observations (picked by row, like the audit), and a page-by-page + comparison of region pages of the busiest recent sites with pages read straight from the observations. On success it logs `packet_sites first-time verification passed` and records the time, so later starts skip it. 4. It logs `packet_sites trusted: region packet pages read packet_sites`. diff --git a/docs/upgrading.md b/docs/upgrading.md index 348c982..deed42d 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -331,7 +331,10 @@ The `packet_sites` first fill slices by observations instead of observation ids, old observations far below the live ids no longer cost a paced slice per 500,000 ids, and its progress line adds `observations_done` and `observations_remaining` (planner estimates) and bases `eta` on them. `readmodels.fill.packet_sites_slice` keeps its default and range but now -counts observations. +counts observations. The hourly audit and the first-time verification pick their random older +slice by observations too, so on a database with a long, sparse old id range they check real +rows. The observation payload-type backfill's progress line gains the same `observations_*` +estimates and bases its `eta` on them. #### Statement timeouts answer 503 From e154873debcf92b39a86f113f16cd0e4ce661708 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 14:24:41 -0400 Subject: [PATCH 09/21] Clock drift range, migration 060 and its rollback step (#56) * Document the clock drift range and migration 060 * Rollback step for wide clock drifts; exact drift range --- docs/api-contract.md | 3 +++ docs/operations.md | 7 +++++++ docs/upgrading.md | 9 +++++++++ 3 files changed, 19 insertions(+) diff --git a/docs/api-contract.md b/docs/api-contract.md index 3b7e4e7..79240a4 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -707,6 +707,9 @@ clamped to the oldest hour the rollups still hold. An empty region returns zeros `{ observerA, observerB, since, until, totalPackets, onlyA, onlyB, both }`. - `clock-drift` lists repeaters and room servers whose last advert clock is off by more than `nodes.clock_drift_threshold`, worst first. It is current state, not windowed. + `clockDriftSeconds` (here and on node detail) is the advert's clock minus the server's, in seconds. + A device clock decades off gives values from about -1.8 × 10⁹ to +2.5 × 10⁹ (the negative bound + grows over time); every value fits a JSON number exactly. - `radio-presets` is `[{ preset, iata, sourceType, count }]`, where `preset` is `freqMhz,bwKhz,sf` and `sourceType` is `observer` or `node`; it is rebuilt every `background.view_refresh` (default `1h`). `node-types` is `[{ nodeType, nodeTypeName, count }]` over all known nodes. diff --git a/docs/operations.md b/docs/operations.md index f40f320..561ae7f 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -195,6 +195,13 @@ Migration `058` sets `fillfactor = 50` on `nodes`. It only changes the table's s rewrite, a lock that does not block reads or writes), and applies to pages written from then on; the same compaction section covers `nodes`. +Migration `060` widens `nodes.device_clock_drift_seconds` to `bigint`. Before it, a device clock +more than about 68 years ahead wrapped to a large negative drift. It rewrites `nodes` once under +an `ACCESS EXCLUSIVE` lock at startup: about 0.1 s for 70,000 nodes, so no pre-build is needed. +A read-only server already connected to the same database may log one error per cached statement +that reads `nodes` while the column changes type; it recovers by itself. Rolling back past `060` +needs one `UPDATE` first; see [Rolling back from 2.1](upgrading.md#rolling-back-from-21). + Migrations `025` to `033` (2.1.0) change `packets`: - `025` and `027` add the checks `packets_payload_type_domain` (`payload_type` 0 to 15) and diff --git a/docs/upgrading.md b/docs/upgrading.md index deed42d..fa71146 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -264,6 +264,15 @@ CREATE INDEX CONCURRENTLY idx_observations_observer_id ON packet_observations US SELECT indisvalid AND indisready FROM pg_index WHERE indexrelid = 'idx_observations_observer_id'::regclass; ``` +Also clear clock drifts an older server cannot read: `060` lets drifts exceed the 32-bit range, and +2.0.7 fails node detail and the clock-drift stats on such a row. It is a few rows at most, and +each node measures its drift again on its next advert: + +```sql +UPDATE nodes SET device_clock_drift_seconds = NULL +WHERE device_clock_drift_seconds NOT BETWEEN -2147483648 AND 2147483647; +``` + The `ALTER` is a catalog change (milliseconds, brief lock on `known_routes`). Run the `CREATE` on its own in autocommit mode; on a large deployment it takes about a minute or two, and the `SELECT` must print `t`. Then deploy the 2.0.7 server and a matching 2.0 web build. The other From c3f5333cfa00d6e8175f9932cb78d07bfb8ee6d7 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 14:48:34 -0400 Subject: [PATCH 10/21] Stop grace period of 75 s for the app; document the shutdown order (#57) * Give the app a 75 s stop grace period; document the shutdown order * SIGKILL can lose queued MQTT messages; upgrade note for the stop grace period --- docker-deployment-type1/docker-compose.yml | 2 ++ .../server/docker-compose.yml | 2 ++ docs/operations.md | 20 +++++++++++++++++++ docs/upgrading.md | 3 +++ 4 files changed, 27 insertions(+) diff --git a/docker-deployment-type1/docker-compose.yml b/docker-deployment-type1/docker-compose.yml index 9f445d2..d8ea609 100644 --- a/docker-deployment-type1/docker-compose.yml +++ b/docker-deployment-type1/docker-compose.yml @@ -12,6 +12,8 @@ services: redis: condition: service_healthy restart: unless-stopped + # Room for a clean shutdown (docs/operations.md, Stopping the server); Docker's default is 10 s. + stop_grace_period: 75s db: image: postgres:16-alpine diff --git a/docker-deployment-type2/server/docker-compose.yml b/docker-deployment-type2/server/docker-compose.yml index 2a69d51..fc9831c 100644 --- a/docker-deployment-type2/server/docker-compose.yml +++ b/docker-deployment-type2/server/docker-compose.yml @@ -12,6 +12,8 @@ services: redis: condition: service_healthy restart: unless-stopped + # Room for a clean shutdown (docs/operations.md, Stopping the server); Docker's default is 10 s. + stop_grace_period: 75s db: image: postgres:16-alpine diff --git a/docs/operations.md b/docs/operations.md index 561ae7f..dd06690 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -74,6 +74,26 @@ historical stats. A packet heard live before the backfill reaches it is decrypte instead: its message is stored once either way, its hours are re-rolled too, and, as a live message, it is broadcast over the WebSocket with its original `sentAt`. +## Stopping the server + +On `SIGTERM` (`docker compose stop`, a redeploy) the server stops in order, so nothing it has +accepted is lost: + +1. It stops taking HTTP requests and lets the ones in flight finish (up to 30 s). At the same + time, ingest stops reading MQTT and stores what it has queued (up to 20 s). +2. Background jobs (rollups, retention, read-model upkeep, the channel message backfill, imports) + stop on their next check; a statement in flight finishes or is cancelled. Shutdown waits up + to 15 s for them, then logs `shutdown: background jobs still running` and goes on. +3. Observer and packet presence held in memory is written out (up to 15 s), then the database + connections close. + +A normal stop takes a few seconds; the worst case is about 60 s. Docker sends `SIGKILL` after the +service's `stop_grace_period`, 10 s by default, which can cut any step short. The example compose +files set `stop_grace_period: 75s` on the `app` service; set the same on your own. A server killed +before step 1 ends loses the MQTT messages still queued for storage (the broker does not resend +them). One killed later loses at most the last presence interval (`last_seen` freshness). Either +way, a background job resumes where it stopped. + ## Retention Everything below is a duration in `config.yaml`. Shorten them if disk is tight, lengthen them diff --git a/docs/upgrading.md b/docs/upgrading.md index fa71146..cc41bca 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -242,6 +242,9 @@ see [Pages that wait for a read model](operations.md#pages-that-wait-for-a-read- before the upgrade leave the window: fully within 7 days for the default view. See [After upgrading: stats rollup row order](operations.md#after-upgrading-stats-rollup-row-order) for an optional one-off `CLUSTER` that heals it at once. +- **Stop grace period.** If you run your own compose file, give the `app` service + `stop_grace_period: 75s` (Docker's default is 10 s), so a stop finishes storing what ingest has + queued; see [Stopping the server](operations.md#stopping-the-server). - **Plan checks.** Turn on the [daily plan check](operations.md#turning-on-the-daily-check) only after the upgrade, if you want it. From 5cc875ca20d756eb1f9abe407ce4e92a10b78f4c Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 15:09:17 -0400 Subject: [PATCH 11/21] Deep region and message pages, slow pool removed (#53) * Deep region and message pages, slow pool removed * Deep pages: soften the sparse-filter claim, name commits by content * Deep pages: typed ranges, the bounded message since, migration 059 * Deep pages: the typed range cap and its worst case, old since --- docs/api-contract.md | 6 +- docs/configuration.md | 35 +++----- docs/operations.md | 193 ++++++++++++++++++++++++++++++------------ docs/upgrading.md | 41 ++++++--- 4 files changed, 184 insertions(+), 91 deletions(-) diff --git a/docs/api-contract.md b/docs/api-contract.md index 79240a4..9597d83 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -60,7 +60,7 @@ Retry-After: 5 The body is the usual error envelope. Wait at least `Retry-After` seconds before retrying; a `deadline_exceeded` request will likely run as long again, so narrow it (a shorter window, a region) rather than repeat it. Clients are keyed by IP after the trusted-proxy step above. -Separately from the guards, and on every server, a page whose database statement runs past its pool's statement limit (35 s; 120 s for the slow region and message statements) answers `503` with code `deadline_exceeded` and `Retry-After: 10`. This covers every `/api/v1` endpoint that reads the database; account, backup and plan-check admin routes keep their own errors. The heavy stats endpoints (`series`, `signal`, `paths`, `observer-comparison`) and route searches (`/routes/search`, `/routes/cross`) answer the same `503` when their own request time limit runs out, with a message that names the query. Any other failed read is a `500` without `Retry-After`. +Separately from the guards, and on every server, a page whose database statement runs past its pool's statement limit (35 s) answers `503` with code `deadline_exceeded` and `Retry-After: 10`. This covers every `/api/v1` endpoint that reads the database; account, backup and plan-check admin routes keep their own errors. The heavy stats endpoints (`series`, `signal`, `paths`, `observer-comparison`) and route searches (`/routes/search`, `/routes/cross`) answer the same `503` when their own request time limit runs out, with a message that names the query. Any other failed read is a `500` without `Retry-After`. Each request falls in one family and one class: @@ -91,10 +91,10 @@ Some pages are served only from data the server prepares in the background. Whil | Pages | Ready when | |---|---| | `/packets` with `iata`, `iatas`, `region` or `regionId`, without `hashPrefix` (region pages) | the server trusts its `packet_sites` read model | -| `/channels/{id}/messages` with `iatas`, or with `scope` and no `since`; `/messages` with `iatas` and neither `since` nor `scope`, or with `scope`, no `since` and no `iatas` | the server trusts its `channel_message_iatas` read model | +| `/channels/{id}/messages` with `iatas`, or with `scope` and no `since`; `/messages` with `iatas` (with or without `since` or `scope`), or with `scope`, no `since` and no `iatas` | the server trusts its `channel_message_iatas` read model | | `/observers/{observerId}/adverts` | every observation carries its payload type | -That happens during the first fill after an upgrade, during a repair, after a failed verification, until the observation payload-type backfill is done, and for as long as the operator turns a read model off (see [Operations](operations.md#pages-that-wait-for-a-read-model)). `/messages` with `iatas` and `since` or `scope` reads no read model and is always served, on a small pool of its own. The same pool answers trusted region `/packets` pages whose filter is too sparse for the read model's bounded reads. When that pool is busy, those requests answer `503` `service_unavailable` with `Retry-After: 5`. +That happens during the first fill after an upgrade, during a repair, after a failed verification, until the observation payload-type backfill is done, and for as long as the operator turns a read model off (see [Operations](operations.md#pages-that-wait-for-a-read-model)). `/messages` with `iatas` and `since` or `scope` is in the table since 2.1: development builds of 2.1 served it from the observations whatever the read model's state, and answered `503` with `Retry-After: 5` when that pool was busy; that `503` is gone. ``` HTTP/1.1 503 Service Unavailable diff --git a/docs/configuration.md b/docs/configuration.md index 0ea278d..27e1112 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -126,11 +126,11 @@ cannot honour. These are the ones people hit: Beacon opens one PostgreSQL connection pool per workload. Pools open connections only when needed, so these are peaks: -| `database.api_plan_layout` | `api_legacy` | `api_slow` | `api_custom` | `api_cached` | `ingest` | `maint` | Total | -|---|---:|---:|---:|---:|---:|---:|---:| -| `custom_all` (default) | - | 2 (`slow_pool_size`) | 8 | - | 6 | 4 | 20 | -| `split` | - | 2 (`slow_pool_size`) | 7 | 5 | 6 | 4 | 24 | -| `single` (rollback) | 8 | - | - | - | 6 | 4 | 18 | +| `database.api_plan_layout` | `api_legacy` | `api_custom` | `api_cached` | `ingest` | `maint` | Total | +|---|---:|---:|---:|---:|---:|---:| +| `custom_all` (default) | - | 8 | - | 6 | 4 | 18 | +| `split` | - | 7 | 5 | 6 | 4 | 22 | +| `single` (rollback) | 8 | - | - | 6 | 4 | 18 | `ingest` has one connection per CPU on hosts with more than 6 CPUs, which adds to each total. The startup log line `database pools open` shows the real figure. Postgres `max_connections` @@ -149,24 +149,13 @@ layout sets the sizes. - **`single`** is the rollback: one API pool (`api_legacy`) on the server's own settings, and every guard family only observes, whatever `guards.enforce` says. -`api_slow` runs only the statements that may read a large part of the observations in one call: -the region `/packets` statement that answers when the site read model's bounded reads run out -of budget, and the all-channel message statement for `iatas` combined with `since` or `scope`. -Two connections by default (`database.slow_pool_size`), a 120 s `statement_timeout`, no parallel -workers. A request that finds every connection busy waits at most 5 s and then answers `503` -with `Retry-After: 5`; it never takes a connection from `api_custom`. A statement that runs past -120 s answers `503` `deadline_exceeded` with `Retry-After: 10`. - -```yaml -database: - slow_pool_size: 2 # api_slow's connections under custom_all and split; at least 1, at most 1000 -``` - -Omit the key to take the default; set it only to override it. Raise it if -`beacon_api_unavailable_total{reason="slow_pool_busy"}` keeps growing, and count it in -`max_connections`. The default was 8 while untrusted read-model pages also ran here; reverting the -change that removed those fallbacks restores that default by itself (see -[Operations](operations.md#reverting-to-the-read-model-fallbacks)). +`database.slow_pool_size` is no longer read. It sized the `api_slow` pool of the 2.1 development +builds, which ran region and message statements that could read whole site histories; those +pages now seek from their window (see +[Deep region and message pages](operations.md#deep-region-and-message-pages)) and run on the +pools above. A config that still sets the key loads as before, and the server logs +`config: database.slow_pool_size is no longer used and is ignored; remove it` at `WARN` at +startup. Remove the key. The layout changes at a restart. What each pool sets is in [Operations](operations.md#database-pools-and-connections). diff --git a/docs/operations.md b/docs/operations.md index dd06690..c98e75f 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -287,7 +287,8 @@ Migrations `042` to `045` add the region packet read model; see `idx_packet_sites_site_scope` with `CREATE INDEX CONCURRENTLY` on the empty table, which takes seconds. No pre-build is needed. -After the deploy, check the table's four indexes: +After the deploy, check the table's seven indexes (four from `042` to `045`, three from `054`, +`055` and `059` below): ```sql SELECT c.relname, i.indisvalid AND i.indisready AS ok @@ -295,7 +296,7 @@ FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid WHERE i.indrelid = 'packet_sites'::regclass ORDER BY 1; ``` -It must return 4 rows, all `ok = t`. +It must return 7 rows, all `ok = t`. Migrations `047` to `053` finish the cleanup; see [Lookup-table indexes and the route stamp column](#lookup-table-indexes-and-the-route-stamp-column). @@ -312,6 +313,44 @@ Migrations `047` to `053` finish the cleanup; see carry it. **It is a one-way door for older images**; see [Upgrading](upgrading.md). +### Indexes for deep region and message pages (`054` to `057`, `059`) + +Migrations `054` to `057` and `059` add five indexes for region and message pages that reach far +back or filter sparsely; see [Deep region and message pages](#deep-region-and-message-pages). +Each is one `CREATE INDEX CONCURRENTLY`: + +- `054` `idx_packet_sites_late` and `055` `idx_packet_sites_early` on `packet_sites`: partial + indexes on the rows heard at a site more than 30 minutes after, or before, their packet's + first hearing (about 0.8 % of the rows, 4 MB on a large deployment; the second stays about + empty). +- `056` `idx_cmi_site_scope` on `channel_message_iatas` (6 MB on a large deployment). +- `057` `idx_channel_messages_sent` on `channel_messages` (4 MB on a large deployment). +- `059` `idx_packet_sites_site_scope_type` on `packet_sites`: one range per site, scope and type + pair, on the scoped rows (61 MB on a large deployment). + +Upgrading from 2.0.x, `054` to `056` and `059` build on tables the same startup created and +filled nothing into yet, so they are instant, and `057` reads `channel_messages` (about 200,000 +rows on a large deployment) in well under a second. On a server that already runs a 2.1 +development build, the tables are full: on a production-size copy `054` took 2.0 s, `055` 0.5 s, +`056` 0.3 s, `057` 0.2 s and `059` 1.9 s. No pre-build is needed. To pre-build them anyway, run +the pre-deploy check above, then each statement on its own in autocommit mode, exactly as written +(`054`, `055` and `059` only where `packet_sites` exists, `056` only where +`channel_message_iatas` exists): + +```sql +CREATE INDEX CONCURRENTLY idx_packet_sites_late ON packet_sites USING btree (iata, site_heard_at DESC, packet_hash DESC) INCLUDE (first_heard_at, payload_type, route_type, scope_id) WHERE ((site_heard_at - first_heard_at) > '00:30:00'::interval); +CREATE INDEX CONCURRENTLY idx_packet_sites_early ON packet_sites USING btree (iata, first_heard_at) INCLUDE (site_heard_at, packet_hash, payload_type, route_type, scope_id) WHERE ((first_heard_at - site_heard_at) > '00:30:00'::interval); +CREATE INDEX CONCURRENTLY idx_cmi_site_scope ON channel_message_iatas USING btree (iata, scope_id, message_id DESC) WHERE (scope_id IS NOT NULL); +CREATE INDEX CONCURRENTLY idx_channel_messages_sent ON channel_messages USING btree (sent_at); +CREATE INDEX CONCURRENTLY idx_packet_sites_site_scope_type ON packet_sites USING btree (iata, scope_id, payload_type, route_type, site_heard_at DESC, packet_hash DESC) INCLUDE (first_heard_at) WHERE (scope_id IS NOT NULL); +SELECT c.relname, i.indisvalid AND i.indisready FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid +WHERE c.relname IN ('idx_packet_sites_late', 'idx_packet_sites_early', 'idx_cmi_site_scope', 'idx_channel_messages_sent', + 'idx_packet_sites_site_scope_type'); +``` + +Every row of the last statement must read `t`. When the server finds a valid index with one of +these names, it records that migration as applied without building anything. + ### Pre-building the route hop index Migration `006_known_routes_hop_iata.sql` builds `idx_known_routes_hop_iata`, which `/routes` @@ -669,20 +708,33 @@ of the same name on `(iata)`; leave it in place. `channel_message_iatas` is a small table with one row for each channel message and each site that heard it. Each row also copies the message's channel and its packet's scope. Ingest keeps it up to date, and a background reconcile repairs it every 5 minutes. On a large deployment it -takes about 120-150 MB with its five indexes. +takes about 125-155 MB with its six indexes. While the server process trusts the table, these channel message pages read it: - `/channels/{id}/messages` with `iatas`, with `iatas` and `scope`, with `scope` alone, and with `since` and `iatas`; -- `/messages` with `iatas`, and with `scope` alone. +- `/messages` with `iatas`, with `iatas` and `scope`, with `iatas` and `since`, and with + `scope` alone. Each such page reads about as many index entries as it returns, where the previous -statement read every observation of the channel. While the process does not trust the table, -those pages answer `503` with `Retry-After: 30`. Two shapes always use the previous all-channel -statement, trusted or not, on the two-connection `api_slow` pool: `/messages` with `iatas` and -`since`, and `/messages` with `iatas` and `scope`. Pages without `iatas` or `scope` never read -the table. Responses are the same as the previous statements' in every case. +statement read every observation of the channel. `/messages` with `iatas` and `scope` reads one +range per site and scope (`idx_cmi_site_scope`). `/messages` with `iatas` and `since` first +counts the messages sent since then, up to 5,001 (`idx_channel_messages_sent`): up to 5,000, it +reads exactly those and keeps the ones heard at a requested site. More, and it reads the newest +5,000 messages first, which fills the page for sites heard often; a page that leaves short is read +from all the messages sent since then, which is what it asked for. Sender clocks put +`sentAt` anywhere, so neither the message order nor a site's range bounds `since` by itself. +While the process does not trust the table, those pages answer `503` with `Retry-After: 30`; +until this release `/messages` with `iatas` and `since` or `scope` ran the previous statement +instead, trusted or not. Pages without `iatas` or `scope` never read the table. Responses are +the same as the previous statements' in every case. + +On a production-size copy (warm): `/messages` with one busy site and a scope it never had went +from 241,633 buffers (916 ms) to 6 buffers; a busy site and its busiest scope, five pages, from +1,560,771 buffers to 3,606; the 13 busiest sites with a scope and `since` 3 days back, two +pages, from 438,827 buffers to 29,461; one busy site with `since` an hour back from 48,664 +buffers to 3,255. ### Installing it @@ -807,7 +859,8 @@ audit refills a random older slice of 500,000 observations (picked by row, so ev observation is as likely to be audited however sparse the old ids are), and a front sweep every 30 seconds keeps the newest index entries dense. On a large deployment (about 7.4 million rows) it takes about 1.6 GB: heap 484 MB, primary key 285 MB, `idx_packet_sites_site` 419 MB, `idx_packet_sites_site_type` -353 MB, `idx_packet_sites_site_scope` 53 MB. +353 MB, `idx_packet_sites_site_scope` 53 MB, `idx_packet_sites_late` 4 MB and +`idx_packet_sites_early` (about empty) 8 kB and `idx_packet_sites_site_scope_type` 61 MB. While the server process trusts the table, `/packets` pages with `iatas`, `iata`, `region` or `regionId` (region pages) read it: @@ -815,10 +868,12 @@ While the server process trusts the table, `/packets` pages with `iatas`, `iata` - one site: one range of the site's index; - several sites: one range per site; - payload and route types, up to 8 (payload, route) pairs: one exact range per site and pair; -- a `since`/`until` window, scopes, or more than 8 type pairs: bounded reads of each site range - that continue, within the same request, until the page is full. If 8 reads do not fill the - page, the previous release's statement answers that request, on the two-connection - `api_slow` pool with a 120 s statement limit (see [Slow region and message statements](#slow-region-and-message-statements)). +- a `since`/`until` window, or scopes: each site's range (each site and scope's, with scopes; + with a type filter, each type pair's range in it) from the page start down to the window; see + [Deep region and message pages](#deep-region-and-message-pages); +- more than 8 type pairs without a window: bounded reads of each site range that continue, + within the same request, until the page is full; if 8 reads do not fill it, one exact range + per site and type pair answers it. Unknown site codes are dropped by a cheap check first. Requests with `hashPrefix` never read the table. A region page served from the table now returns `nextPageCursor`; see @@ -833,14 +888,18 @@ skips nothing. Packets tied on their site time come out by packet hash, descendi On a production-size copy, the heaviest region pages went from seconds to under a second: all 561 sites for the last hour from 19.5 s to 0.3 s, and the 13 busiest sites for the last 75 minutes at limit 200 from 4.5 s to 31 ms. A filter of rare types on a very sparse site can read a few -buffers more than before. An `until` well in the past often runs out of the read budget, so -the previous statement answers many of those pages, at the previous cost. +buffers more than before. ### Ingest cost The site row makes each observation insert about 15-21% slower (mean 104 µs to 119-125 µs) and adds about 37-53% WAL per stored observation (1,425-1,470 B to 2,008-2,188 B), measured over 100,000 inserts on a scratch database. Re-sends of an observation still write nothing. +The two partial indexes from `054` and `055` add about 1-6 % to the insert time and no +measurable WAL (three runs of 150,000 inserts with 1 % of the hearings arriving hours late: +within 0.4 %): only a row heard more than 30 minutes away from its first hearing enters them. +`059` adds no measurable insert time (-1.2 to +1.4 %) and 0.5-0.9 % WAL, with one packet in +eight scoped: only scoped rows enter it. After the deploy, compare `InsertObservation` in `pg_stat_statements` (`mean_exec_time`, and `wal_bytes / calls`) for the hour before and the hour after, and watch @@ -1067,59 +1126,86 @@ values in force. While a fill runs, check `/proc/pressure/io` on the host: `full stay well below 50 %. Lower `duty` further if it does not. The reconcile and the hourly audit are not paced: they cover only the observations since the last run, or one slice an hour. -## Slow region and message statements +## Deep region and message pages -Two kinds of request still read a large part of the observations in one statement: a region -`/packets` page whose bounded reads ran out of budget (a sparse filter such as a narrow window -or rare payload types over busy sites), and `/messages` with `iatas` and `since` or `scope`. -They run on their own pool, `api_slow`: 2 connections by default -([`database.slow_pool_size`](configuration.md#database-pools-and-cost-guards)), -`statement_timeout=120s`, no parallel workers, not gated by the cost guards. When every -connection is busy a request waits at most 5 s and then answers `503` with `Retry-After: 5`, -without touching the other pools; `beacon_api_unavailable_total{reason="slow_pool_busy"}` counts -those, and the server logs `api: slow requests refused, every slow connection is busy` at most -once a minute. A statement that passes 120 s answers `503` `deadline_exceeded` with -`Retry-After: 10`; narrowing the request (fewer sites, a wider window) avoids it. +Region `/packets` pages with a window, scopes or types, and `/messages` pages with `iatas` and +`since` or `scope`, read at most their page per index range (site, scope, type pair) or, for +`/messages` with `since`, at most the messages sent since then. The worst case is a quiet site +with `since` near the oldest retained message: it reads every message sent since then, which for an +old `since` is effectively the whole message history. On a production-size copy (196,000 +messages) that took 2,907 buffers and 148 ms. Until this release a region page whose bounded reads ran out of budget (a sparse filter such as +a narrow window, an `until` well in the past, or rare payload types over busy sites), and +`/messages` with `iatas` and `since` or `scope`, ran the previous release's statements on a +two-connection `api_slow` pool with a 120 s statement limit. That pool and its +`database.slow_pool_size` key are gone; those pages run on the pool their plan class picks +(`api_custom`, 35 s; `api_legacy` under the `single` layout), like every other page. -Before this release the region statement ran on `api_legacy` without a time limit for a -millisecond cursor, and on `api_custom` (35 s, gated by `packets_region`) for an exact page -cursor; the all-channel message statement ran on `api_legacy` without a limit. +A region page with a window seeks from it. Its rows are first heard in `[since, until]`, so +almost all of them were heard at a site within 30 minutes of that: each site's range is read +from `until` + 30 minutes (or the page cursor, when lower) down to `since` − 30 minutes, and +stops at the page. Two partial indexes hold the only rows that can lie outside those bounds: +`idx_packet_sites_late`, the rows heard at a site more than 30 minutes after their packet was +first heard (a re-hearing hours or days later; about 0.8 % of the rows), and +`idx_packet_sites_early`, the rows heard more than 30 minutes before it (ingest clamps hearing +times to 30 minutes of its own clock, so it stays about empty). A page reads about its own rows, +the 30 minutes above `until` and the late rows above that; a sparse filter reads at most its +window. Scoped pages do the same per site and scope. With a type filter, each site's (or site and +scope's) range is read per (payload type, route type) pair instead, on `idx_packet_sites_site_type` +or `idx_packet_sites_site_scope_type`, so a rare type reads at most a page per pair: a page costs +at most the pairs times its size, however sparse the filter. That holds up to 4,096 ranges (sites +times scopes times type pairs; a region page with up to 8 pairs is always under it). Above it the +page reads each site or site-and-scope range with the types as a filter, as an untyped page does: +cheap while the filter is dense, but a sparse filter then reads each range down to the window. +Only requests naming thousands of combinations get there; the worst measured, every known site +(562) with every scope (1,491) and one rare payload type, took 2.5 million buffers and 2.1 s +without a window and 5.0 million buffers and 3.0 s with `until` a day back, on a warm +production-size copy, nearly all of it probing the 838,000 mostly empty site-and-scope ranges. +Pages are full until the last, and the +rows, their order and `nextPageCursor` are those of the observations; where the previous +statement had ended a page short at its scan floor, the new page is full and the walk returns +the same rows in the same order. -Measured on a copy of a large production database (29.4 million observations, 561 IATAs), with -the pool's settings: +Measured on a copy of a large production database (29.4 million observations; warm; every region +and message statement a request runs, page by page): -| Request | Warm | Cold cache | +| Request | Before: buffers, time | After | |---|---:|---:| -| The busiest region (13 IATAs), three rare payload types | 3.1-3.3 s | 22.9 s | -| The 13 busiest IATAs, a one-hour window 60 hours back | 5.5 s | 23.3 s (first run) | -| Every IATA (561), a one-minute window | 50 s | 34 s | -| Every IATA (561), a one-hour window 120 hours back | 115 s | not measured | +| The busiest site, `until` 24 hours back, page 1 | 131,591, 2.1 s | 2,442, 47 ms | +| The same, 20 pages | 383,754, 3.8 s | 34,830, 434 ms | +| The 13 busiest IATAs, `until` 24 hours back, 10 pages | 4,431,750, 3.6 s | 45,140, 117 ms | +| The 13 busiest IATAs, a one-hour window 60 hours back | 3,694,758, 16.5 s | 2,326, 25 ms | +| Three sites and a scope, `until` 24 hours back, 3 pages | 1,294,780, 3.7 s | 6,766, 52 ms | +| The 13 busiest IATAs, the last 75 minutes, limit 200, 2 pages | 65,485, 826 ms | 14,361, 119 ms | -Requests naming every IATA are not something the web or app send; the largest region on that -database has 22 IATAs. `beacon_db_time_seconds{pool="api_slow"}` and -`beacon_pool_acquire_seconds{pool="api_slow"}` show how busy the pool is. +On production the busiest-site `until` page took 48-51 s with a cold cache before this change +(about 181,000 blocks read per call). Message pages are in +[Read models: channel messages by site](#read-models-channel-messages-by-site). + +A server whose `config.yaml` still sets `database.slow_pool_size` starts as before and logs +`config: database.slow_pool_size is no longer used and is ignored; remove it` at `WARN`. ### Reverting to the read-model fallbacks If the read-model pages must be served while a model is not trusted, as in earlier releases, -revert the server change that removed the fallbacks rather than rolling back the image. It is a -single commit on the server's `dev` branch ("region, scope and advert pages answer 503 until -their read model is ready, instead of falling back"); reverting it gives back exactly the -previous release's tree: +revert two server commits on `dev` rather than rolling back the image, newest first: the one +that removed the `api_slow` pool (it adds migrations `054` to `057`), then the one that removed +the fallbacks (region, scope and advert pages answer `503` until their read model is ready). +Find them with `git log --oneline -- db/pools.go`: ```bash -git revert --no-edit +git revert --no-edit ``` -Then build, deploy and restart; no configuration change is needed, because the revert also -restores `slow_pool_size`'s default of 8 (an explicit `slow_pool_size` in `config.yaml` still -wins). What to expect: +Then build, deploy and restart; no configuration change is needed, because the reverts also +restore the pool and its default of 8 connections (an explicit `slow_pool_size` in `config.yaml` +still wins). Migrations `054` to `057` stay applied; the restored statements don't read their +indexes. What to expect: - Untrusted region `/packets` pages, channel message pages and observer advert pages are served by the previous statements again, with no `503` from a read model. - All four previous statements (region, per-channel and all-channel messages, adverts) run on `api_slow`: 8 connections, like the old `api_legacy`, ungated, with the 5 s connection wait and - a 120 s statement limit (the old pool had no limit). The total connection count grows by 6. + a 120 s statement limit (the old pool had no limit). The total connection count grows by 8. - If migration `046` (the drops release) has run, rebuild `idx_observations_observer_id` first, because the restored adverts statement reads it: @@ -1306,7 +1392,6 @@ connection belongs to: | Pool | `application_name` | Size | Session settings | |---|---|---:|---| | `api_legacy` | `beacon-api_legacy` | 8, `single` only | Server defaults. The rollback layout's only API pool. | -| `api_slow` | `beacon-api_slow` | 2 (`slow_pool_size`), not under `single` | Server defaults, with `statement_timeout=120s` and `max_parallel_workers_per_gather=0`. Only the [slow region and message statements](#slow-region-and-message-statements). | | `api_custom` | `beacon-api_custom` | 8 (7 under `split`) | `plan_cache_mode=force_custom_plan`, `jit=off`, `max_parallel_workers_per_gather=0`, `statement_timeout=35s`, `idle_in_transaction_session_timeout=30s` | | `api_cached` | `beacon-api_cached` | 5, `split` only | As `api_custom`, with `plan_cache_mode=force_generic_plan` | | `ingest` | `beacon-ingest` | 6, or one per CPU above 6 | `plan_cache_mode=auto`, `jit=off`, no statement timeout: an ingest write waits out a lock rather than drop a packet | @@ -1315,7 +1400,7 @@ connection belongs to: The `single` layout opens only `api_legacy`, `ingest` and `maint`; the other layouts never open `api_legacy`. -- **Headroom.** `SHOW max_connections;` must leave room for the total (20 by default on up to +- **Headroom.** `SHOW max_connections;` must leave room for the total (18 by default on up to 6 CPUs) plus every other client: `psql` sessions, backups, monitoring. If your deploy starts the new server before the old one stops, count both. - **Checking.** Parallel workers carry their leader's `application_name`, so count client @@ -1438,7 +1523,7 @@ scrape_configs: | `beacon_packet_sites_front_live` | gauge | `index` | Index entries written in the last two minutes that the last front sweep visited | | `beacon_packet_sites_bytes` | gauge | | Total size of `packet_sites`, heap and indexes, hourly | | `beacon_db_sequence_used_ratio` | gauge | `sequence`, `cycle` | Share of a sequence's usable values already handed out, 0 to 1, each minute. The usable range ends at the sequence's own limit or its column's type limit (`integer` 2,147,483,647), whichever is lower. `cycle="true"` marks a `CYCLE` sequence, whose ratio is its position, not used-up headroom. See [Sequence headroom](#sequence-headroom). | -| `beacon_api_unavailable_total` | counter | `reason`, `model` | `503`s a page answered: `not_ready` while its read model or backfill is not ready, `slow_pool_busy` when `api_slow` had no free connection, `timeout` when its statement ran past its pool's statement limit or an enforced guard's deadline. `model` is `packet_sites`, `cmi` or `observation_payload_types`. | +| `beacon_api_unavailable_total` | counter | `reason`, `model` | `503`s a page answered: `not_ready` while its read model or backfill is not ready, `timeout` when its statement ran past its pool's statement limit or an enforced guard's deadline. `model` is `packet_sites`, `cmi` or `observation_payload_types`. | `guard` is `deadline`, `cap`, `admission` or `budget`. The Postgres counters are read once a minute; the index gauges first appear an hour after startup. The two `beacon_planguard_*` @@ -1508,8 +1593,6 @@ groups: - alert: BeaconAdvertsNotReady # observer advert pages answering 503 for over 2 h expr: sum(increase(beacon_api_unavailable_total{reason="not_ready",model="observation_payload_types"}[15m])) > 0 for: 2h - - alert: BeaconSlowPoolBusy # region or message requests refused for want of a slow connection - expr: sum(increase(beacon_api_unavailable_total{reason="slow_pool_busy"}[15m])) > 10 # The sequence rules leave out CYCLE sequences, which wrap instead of running out. - alert: BeaconSequenceHalfUsed expr: beacon_db_sequence_used_ratio{cycle="false"} > 0.5 diff --git a/docs/upgrading.md b/docs/upgrading.md index cc41bca..c154243 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -159,7 +159,7 @@ then show an error with a retry button. `readmodels.fill.duty: 0.25` and `readmodels.fill.packet_sites_slice: 100000` in `config.yaml` before upgrading. At full speed the first fill can stall such a machine; paced, it takes longer but the server stays responsive. See [Pacing the backfills](operations.md#pacing-the-backfills). -4. **Connections.** The server now opens one pool per workload: 20 connections by default, plus +4. **Connections.** The server now opens one pool per workload: 18 connections by default, plus one for each CPU above 6. Check that Postgres `max_connections` leaves room for that and your own sessions; see [Database pools and connections](operations.md#database-pools-and-connections). 5. **Large deployments only** (tens of millions of routes, hundreds of millions of observations): @@ -170,7 +170,12 @@ then show an error with a retry button. - [the four packets indexes](operations.md#pre-building-the-packets-indexes), - [the observer adverts index](operations.md#pre-building-the-observer-adverts-index). - Everyone else can skip this: the server builds them at startup. + Everyone else can skip this: the server builds them at startup. The five indexes of + [`054` to `057` and `059`](operations.md#indexes-for-deep-region-and-message-pages-054-to-057-059) + need no pre-build: four build on tables the same startup creates, and + `idx_channel_messages_sent` reads `channel_messages` in well under a second. A server already + on a 2.1 development build builds all five on full tables in a few seconds; pre-build them + there only if you want, with the text in that section. 6. Run the [pre-deploy check](operations.md#before-deploying-a-release) (no long transactions, no anti-wraparound vacuum, no `pg_dump` running), and pick a quiet hour. @@ -229,8 +234,8 @@ see [Pages that wait for a read model](operations.md#pages-that-wait-for-a-read- - **Ingest cost.** Each stored observation now costs about 15 to 21 % more insert time and 37 to 53 % more WAL. Watch `beacon_ingest_queue_depth` for the first hours; see [Ingest cost](operations.md#ingest-cost). -- **Alerts.** Load the read-model `Untrusted` rules, `BeaconAdvertsNotReady`, - `BeaconSlowPoolBusy` and the sequence rules from +- **Alerts.** Load the read-model `Untrusted` rules, `BeaconAdvertsNotReady` and the sequence + rules from [Example alert rules](operations.md#example-alert-rules). - **Route maintenance.** The first route retention run after the upgrade is a full pass; on a large deployment it takes 12 to 14 minutes in short statements. See @@ -320,21 +325,37 @@ See [Read models: packets by site](operations.md#read-models-packets-by-site). #### Pools, dropped indexes and a dropped column (`046` to `053`) -- `api_legacy` opens only under the `single` layout. `api_slow` (2 connections by default, - `database.slow_pool_size`, 120 s statement limit) runs the few statements that may read whole - site histories; see - [Slow region and message statements](operations.md#slow-region-and-message-statements). +- `api_legacy` opens only under the `single` layout. The 2.1 development builds also had an + `api_slow` pool and a `database.slow_pool_size` key; both are gone (see below). - `046` drops `idx_observations_observer_id`, `047` to `052` drop six cold lookup-table indexes, and `053` drops `known_routes.last_reconfirmed_at`. Every 2.0.x image reads or writes that column, hence the rollback steps above. +#### Deep region and message pages (`054` to `057`, `059`) + +Region `/packets` pages with a `since`/`until` window or scopes seek from the window instead of +reading each site from its newest row, and `/messages` with `iatas` and `since` or `scope` read +`channel_message_iatas`; no page reads a whole site history in one statement any more. Two +partial indexes on `packet_sites` (`054`, `055`), one on `channel_message_iatas` (`056`), one on +`channel_messages` (`057`), and one on the scoped rows of `packet_sites` for a scope with a type +filter (`059`, 61 MB on a large deployment). Pages are full until the last; rows, order and cursors +are unchanged. On production the busiest region with `until` 24 hours back took 48-51 s before; +on a production-size copy it went from 131,591 buffers to 2,442. See +[Deep region and message pages](operations.md#deep-region-and-message-pages). + +The 2.1 development builds ran those pages on a two-connection `api_slow` pool. It is gone, and +so is `beacon_api_unavailable_total{reason="slow_pool_busy"}` (and the `BeaconSlowPoolBusy` +alert in the examples). A `database.slow_pool_size` left in `config.yaml` is ignored, with a +`WARN` at startup; remove it. The default connection total drops from 20 to 18 (22 under +`split`). + #### Read-model pages without fallback Region `/packets` pages, channel message pages filtered by `iatas` (or by `scope` without `since`), and observer advert pages are served only from their read model or backfilled data. While that is not ready they answer `503` with `Retry-After: 30`; there is no fallback to the previous statements. The kill switches above turn those pages off instead of serving them from -the previous statements. To get the fallbacks back, revert their one server commit; see +the previous statements. To get the fallbacks back, revert their two server commits; see [Reverting to the read-model fallbacks](operations.md#reverting-to-the-read-model-fallbacks). #### Packet sites fill counts observations @@ -350,7 +371,7 @@ estimates and bases its `eta` on them. #### Statement timeouts answer 503 -Every `/api/v1` read whose statement runs past its pool's limit (35 s; 120 s on `api_slow`) +Every `/api/v1` read whose statement runs past its pool's limit (35 s) now answers `503` `deadline_exceeded` with `Retry-After: 10` instead of a `500`. The heavy stats endpoints and route searches already answered `503` when their own time limit ran out; that `503` now carries `Retry-After: 10` and code `deadline_exceeded` (was From a91e553baba707a875034bd72d0fce165254bcf3 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 15:29:20 -0400 Subject: [PATCH 12/21] Document timeout 503s counted for every page (#55) --- docs/operations.md | 6 +++++- docs/upgrading.md | 5 +++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/operations.md b/docs/operations.md index c98e75f..b93897d 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1523,7 +1523,7 @@ scrape_configs: | `beacon_packet_sites_front_live` | gauge | `index` | Index entries written in the last two minutes that the last front sweep visited | | `beacon_packet_sites_bytes` | gauge | | Total size of `packet_sites`, heap and indexes, hourly | | `beacon_db_sequence_used_ratio` | gauge | `sequence`, `cycle` | Share of a sequence's usable values already handed out, 0 to 1, each minute. The usable range ends at the sequence's own limit or its column's type limit (`integer` 2,147,483,647), whichever is lower. `cycle="true"` marks a `CYCLE` sequence, whose ratio is its position, not used-up headroom. See [Sequence headroom](#sequence-headroom). | -| `beacon_api_unavailable_total` | counter | `reason`, `model` | `503`s a page answered: `not_ready` while its read model or backfill is not ready, `timeout` when its statement ran past its pool's statement limit or an enforced guard's deadline. `model` is `packet_sites`, `cmi` or `observation_payload_types`. | +| `beacon_api_unavailable_total` | counter | `reason`, `model` | `503`s a page answered: `not_ready` while its read model or backfill is not ready, `timeout` when its statement ran past its pool's statement limit, an enforced guard's deadline or the endpoint's own time limit. `model` is the read model the page reads (`packet_sites`, `cmi` or `observation_payload_types`), or `none` for every other `/api/v1` page. A request an enforced guard deadline stops is also counted in `beacon_guard_refused_total{guard="deadline"}`; admission and cap refusals happen before the page runs and are not counted here. | `guard` is `deadline`, `cap`, `admission` or `budget`. The Postgres counters are read once a minute; the index gauges first appear an hour after startup. The two `beacon_planguard_*` @@ -1593,6 +1593,10 @@ groups: - alert: BeaconAdvertsNotReady # observer advert pages answering 503 for over 2 h expr: sum(increase(beacon_api_unavailable_total{reason="not_ready",model="observation_payload_types"}[15m])) > 0 for: 2h + - alert: BeaconQueryTimeouts # pages answering 503 because their statement ran out of time + # One client retrying a slow page adds a few dozen in 15 minutes; tune the floor to your traffic. + expr: sum by (model) (increase(beacon_api_unavailable_total{reason="timeout"}[15m])) > 100 + for: 30m # The sequence rules leave out CYCLE sequences, which wrap instead of running out. - alert: BeaconSequenceHalfUsed expr: beacon_db_sequence_used_ratio{cycle="false"} > 0.5 diff --git a/docs/upgrading.md b/docs/upgrading.md index c154243..cba00ac 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -378,8 +378,9 @@ that `503` now carries `Retry-After: 10` and code `deadline_exceeded` (was `service_unavailable`), and also covers a statement timeout. Web and app retry it like the other `503`s. In the server log a timeout is one `request complete` line at `WARN` with `status=503` instead of an `ERROR` pair; a real `500` still logs `api: error 500: ` at `ERROR`, now -with the database error in `error`. Only pages that read a read model count timeouts in -`beacon_api_unavailable_total{reason="timeout"}`. +with the database error in `error`. Every timeout `503` is counted in +`beacon_api_unavailable_total{reason="timeout"}`: under the read model a page reads, or +`model="none"` for the other pages. #### Runtime image on Alpine 3.24 From ddff03114cb389d93d3ef0188edaa9dfa12b88c1 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 15:29:53 -0400 Subject: [PATCH 13/21] Upgrading to 2.1: migrations 004 to 060, nodes changes, fill note for the release (#58) --- docs/upgrading.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/upgrading.md b/docs/upgrading.md index cba00ac..976da18 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -136,7 +136,7 @@ the flood jail's `maxretry` of 900 still sits above the new limit. ## Upgrading to 2.1 -Server 2.1.0 is one upgrade from any 2.0.x release. It runs migrations `004` to `053` at startup, +Server 2.1.0 is one upgrade from any 2.0.x release. It runs migrations `004` to `060` at startup, adds two read models that region and scope pages are served from, and drops indexes and one column nothing reads any more. No config change is required. Beacon Web must move to 2.1 at the same time (`minWebVersion` is 2.1.0; an older web build shows the reload screen). The minimum app @@ -206,16 +206,18 @@ routes, 30 million observations and 3 million packets; default pacing): | Step | Time | |---|---| | Pre-building the eight indexes, old version still serving | 1 min 50 s (route hop index 72 s, adverts index 16 s, the rest under 8 s each) | -| Migrations `004` to `053` at startup | 7.4 s (the two `packets` checks 3.7 s and 1.8 s) | +| Migrations `004` to `053` at startup (a 2.1 development build; the release adds `054` to `060`, a few seconds more) | 7.4 s (the two `packets` checks 3.7 s and 1.8 s) | | API and ingest down | about 15 s | | Channel message pages by site back | 8 min after startup | | Observer advert pages | at once (no untyped observations) | | Region packet pages back | 50 min after startup | | Database size | 91 GB before, 71 GB after | -The region packet fill walks the whole observation id range, newest to oldest, so a database that -still holds a few very old observations fills for longer than its row count suggests, and the -`eta` in its progress lines overstates the time left. +That development build walked the whole observation id range in fixed id slices, so the few very +old observations the database still held stretched its fill and its `eta`. The release slices the +fill by observations instead and reports progress and `eta` in rows (see +[Pacing the backfills](operations.md#pacing-the-backfills)); on a copy of the same data it needs +about 60 slices instead of 1,533. Watch the log for these lines; each one ends a phase: @@ -349,6 +351,16 @@ alert in the examples). A `database.slow_pool_size` left in `config.yaml` is ign `WARN` at startup; remove it. The default connection total drops from 20 to 18 (22 under `split`). +#### Nodes table (`058`, `060`) + +`058` sets `fillfactor = 50` on `nodes` so the `last_seen` update of every advert can stay on its +page (fewer index writes, about half the WAL); it changes settings only and applies to pages +written from then on. `060` widens `nodes.device_clock_drift_seconds` to `bigint`, so device +clocks more than about 68 years off are stored as they are instead of wrapping; it rewrites +`nodes` once at startup (about 0.1 s for 70,000 nodes). See +[How startup migrations wait for locks](operations.md#how-startup-migrations-wait-for-locks) and, +before going back to 2.0.x, the extra step in [Rolling back from 2.1](#rolling-back-from-21). + #### Read-model pages without fallback Region `/packets` pages, channel message pages filtered by `iatas` (or by `scope` without From 82bf29ec40e0d096cbbb3fd591491a0d226da0b3 Mon Sep 17 00:00:00 2001 From: MrAlders0n Date: Mon, 26 Oct 2026 13:00:55 -0400 Subject: [PATCH 14/21] docs(api): note WebSocket permessage-deflate --- docs/api-contract.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/api-contract.md b/docs/api-contract.md index 9597d83..f3568c1 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -899,6 +899,8 @@ The upgrade is checked before anything else: - **Origin.** A browser `Origin` must match the request's host, or one of `websocket.allowed_origins`. Anything else gets `403`. Set `allowed_origins` when the web app is served from a different host than the API. - **Concurrent connections.** At most `websocket.max_connections_per_ip` (default 5) open sockets per IP. Past that, the socket is accepted and immediately closed with code `1013` (try again later), reason `connection limit reached`, so the browser can back off. +**Compression.** When the client offers `permessage-deflate` (browsers and `dart:io` do by default), the server accepts it with context takeover, unless the client asks for `no_context_takeover`, and compresses messages of 128 bytes or more. The live feed shrinks about 3× on the wire. Clients that don't offer the extension get uncompressed frames, as before. Message content is the same either way. + On connect the server sends a `hello`: ```json From 8a00362ed09c23873456cf43bdcddffbf11931f7 Mon Sep 17 00:00:00 2001 From: MrAlders0n Date: Mon, 26 Oct 2026 13:53:19 -0400 Subject: [PATCH 15/21] Publish UDP 443 so the edge Caddy serves HTTP/3 --- README.md | 8 ++++--- app_config/fail2ban/README.md | 9 +++++--- .../fail2ban/caddy/jail.d/beacon-http.conf | 5 ++-- docker-deployment-type1/docker-compose.yml | 1 + docker-deployment-type2/README.md | 3 ++- .../server/docker-compose.yml | 1 + .../web/docker-compose.yml | 1 + docs/reverse-proxy.md | 23 +++++++++++++++++++ docs/upgrading.md | 20 ++++++++++++++++ 9 files changed, 62 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index d6231b4..7717c7f 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,8 @@ This repo is where you deploy and operate it. The code lives in ## Deploy the all-in-one stack One server runs everything: the API, Postgres, Redis, the web frontend and Caddy with automatic -HTTPS. You need a domain pointed at the server, ports 80 and 443 open, Docker with the compose -plugin, and a subscriber account on a MeshCore MQTT broker +HTTPS. You need a domain pointed at the server, TCP ports 80 and 443 and UDP port 443 open, +Docker with the compose plugin, and a subscriber account on a MeshCore MQTT broker ([where packets come from](docs/getting-packets-in.md)). If you would rather run the API and the web frontend on different hosts, see @@ -101,7 +101,9 @@ Every other key, with its default, is documented in ### 4. Point DNS at the server Create an `A` or `AAAA` record for `DOMAIN` pointing at the server's public IP. Caddy needs -ports 80 and 443 reachable from the internet to get its certificate. +ports 80 and 443 reachable from the internet to get its certificate. Open UDP 443 too, in the +host firewall and any cloud security group, so browsers can use +[HTTP/3](docs/reverse-proxy.md#http3). ### 5. Bring it up diff --git a/app_config/fail2ban/README.md b/app_config/fail2ban/README.md index db6499d..431189f 100644 --- a/app_config/fail2ban/README.md +++ b/app_config/fail2ban/README.md @@ -73,9 +73,12 @@ there must be an obvious scraper, otherwise raise `maxretry`. **Enforce.** Caddy box: traffic to Docker-published ports is DNAT'ed through the FORWARD path, which the default nftables action (input hook) never sees, so switch to -`banaction = nftables[type=allports, chain=f2b-forward, chain_hook=forward]` (a second chain in the -existing `f2b-table`; `iptables-allports[chain=DOCKER-USER]` is equivalent). Apache box: remove the -`dummy` line, the inherited nftables input-hook action works because Apache listens on the host. +`banaction = nftables[type=allports, protocol="tcp,udp", chain=f2b-forward, chain_hook=forward]` +(a second chain in the existing `f2b-table`; +`iptables-allports[chain=DOCKER-USER, protocol="tcp,udp"]` is equivalent). The action bans TCP +only by default; `protocol="tcp,udp"` also stops a banned address reaching Caddy over HTTP/3 on +UDP 443. Apache box: remove the `dummy` line, the inherited nftables input-hook action works +because Apache listens on the host. Then: ```bash diff --git a/app_config/fail2ban/caddy/jail.d/beacon-http.conf b/app_config/fail2ban/caddy/jail.d/beacon-http.conf index cf3cf13..4577f22 100644 --- a/app_config/fail2ban/caddy/jail.d/beacon-http.conf +++ b/app_config/fail2ban/caddy/jail.d/beacon-http.conf @@ -6,8 +6,9 @@ # - banaction: traffic to Docker-published 80/443 is DNAT'ed through the FORWARD path, so the # default nftables action (input hook) never sees it. Start in observe mode (dummy), then switch # to a forward-hook chain in the existing f2b-table: -# nftables[type=allports, chain=f2b-forward, chain_hook=forward] -# (equivalent: iptables-allports[chain=DOCKER-USER]) +# nftables[type=allports, protocol="tcp,udp", chain=f2b-forward, chain_hook=forward] +# (equivalent: iptables-allports[chain=DOCKER-USER, protocol="tcp,udp"]) +# protocol: the default is tcp only, which leaves HTTP/3 on UDP 443 open to a banned address. # - ignoreip: loopback, RFC1918 (Docker bridges), Tailscale. Add the operator's home IP if wanted. # - bantime is a flat 10m: a banned CGNAT address hurts every user behind it. diff --git a/docker-deployment-type1/docker-compose.yml b/docker-deployment-type1/docker-compose.yml index d8ea609..7e6789f 100644 --- a/docker-deployment-type1/docker-compose.yml +++ b/docker-deployment-type1/docker-compose.yml @@ -50,6 +50,7 @@ services: ports: - "80:80" - "443:443" + - "443:443/udp" # HTTP/3 (QUIC) environment: - DOMAIN=${DOMAIN:?DOMAIN environment variable is required} env_file: .env diff --git a/docker-deployment-type2/README.md b/docker-deployment-type2/README.md index ec620a3..7bae505 100644 --- a/docker-deployment-type2/README.md +++ b/docker-deployment-type2/README.md @@ -12,7 +12,8 @@ docker-deployment-type2/ ## What you need -- Two hosts with Docker and ports 80 and 443 open. +- Two hosts with Docker, TCP ports 80 and 443 and UDP port 443 open (UDP is for + [HTTP/3](../docs/reverse-proxy.md#http3)). - Two DNS names pointing at them, for example `api.example.com` and `beacon.example.com`. - A subscriber account on a MeshCore MQTT broker. See [Getting packets in](../docs/getting-packets-in.md). diff --git a/docker-deployment-type2/server/docker-compose.yml b/docker-deployment-type2/server/docker-compose.yml index fc9831c..913eaa3 100644 --- a/docker-deployment-type2/server/docker-compose.yml +++ b/docker-deployment-type2/server/docker-compose.yml @@ -50,6 +50,7 @@ services: ports: - "80:80" - "443:443" + - "443:443/udp" # HTTP/3 (QUIC) environment: - DOMAIN=${DOMAIN:?DOMAIN environment variable is required} - WEB_DOMAIN=${WEB_DOMAIN:?WEB_DOMAIN environment variable is required} diff --git a/docker-deployment-type2/web/docker-compose.yml b/docker-deployment-type2/web/docker-compose.yml index bf9a013..0632c69 100644 --- a/docker-deployment-type2/web/docker-compose.yml +++ b/docker-deployment-type2/web/docker-compose.yml @@ -4,6 +4,7 @@ services: ports: - "80:80" - "443:443" + - "443:443/udp" # HTTP/3 (QUIC) environment: - DOMAIN=${DOMAIN:?DOMAIN environment variable is required} volumes: diff --git a/docs/reverse-proxy.md b/docs/reverse-proxy.md index d868472..b1822b0 100644 --- a/docs/reverse-proxy.md +++ b/docs/reverse-proxy.md @@ -45,6 +45,29 @@ server: trusted_proxies: [172.30.0.0/24] ``` +### HTTP/3 + +Caddy serves HTTP/3 (QUIC) by default and announces it in an `alt-svc` header. QUIC runs on +UDP, so the compose files publish `443:443/udp` next to `443:443`. Open UDP 443 wherever TCP +443 is open: the host firewall and the cloud provider's security group. HTTP/3 helps most on +lossy mobile links: a lost packet no longer holds up every request on the connection, and +connections start faster and survive a change of network. + +After `docker compose up -d`, responses carry the header: + +```bash +curl -sI https://beacon.example.com/ | grep -i alt-svc +# alt-svc: h3=":443"; ma=2592000 +``` + +Caddy sends it even when UDP is blocked, so it only shows that HTTP/3 is on. To see HTTP/3 get +through, open the browser's network panel and reload the page, more than once if needed: the +Protocol column shows `h3`. A curl built with HTTP/3 (`curl -V` lists `HTTP3`) checks it +directly with `curl --http3-only -sI https://beacon.example.com/`. + +Nothing breaks where UDP is blocked, by your firewall or the visitor's network: browsers fall +back to HTTP/2. The nginx and Apache setups below have no HTTP/3. + ## Behind a CDN Behind Cloudflare or another CDN, `{remote_host}` is the CDN, not the visitor. The comment at diff --git a/docs/upgrading.md b/docs/upgrading.md index 976da18..3d7d454 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -134,6 +134,26 @@ ratelimit: Then run `docker compose restart app`. If you run the [fail2ban jails](../app_config/fail2ban/README.md), the flood jail's `maxretry` of 900 still sits above the new limit. +## HTTP/3 needs UDP 443 + +The compose files now publish UDP 443 so Caddy can serve HTTP/3. An existing deployment keeps +its own `docker-compose.yml`, so add the UDP line to the `caddy` service by hand (on both hosts +of a split deployment): + +```yaml + caddy: + ports: + - "80:80" + - "443:443" + - "443:443/udp" +``` + +Open UDP 443 in the host firewall and any cloud security group, then run +`docker compose up -d caddy`, which recreates the Caddy container. [HTTP/3](reverse-proxy.md#http3) +shows how to check it. If you enforce the [fail2ban jails](../app_config/fail2ban/README.md), +add `protocol="tcp,udp"` to the ban action as shown there: the default bans TCP only, so a +banned address could still reach the site over HTTP/3. + ## Upgrading to 2.1 Server 2.1.0 is one upgrade from any 2.0.x release. It runs migrations `004` to `060` at startup, From da58ec15254b73dd0ea65a72c9ec2d076b94937e Mon Sep 17 00:00:00 2001 From: MrAlders0n Date: Mon, 26 Oct 2026 13:59:32 -0400 Subject: [PATCH 16/21] docs(api): /regions items carry their IATAs and detail --- docs/api-contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/api-contract.md b/docs/api-contract.md index f3568c1..94a70f2 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -567,7 +567,7 @@ Channel keys come from the server config file and, optionally, MeshMapper. | `GET /iatas` | All IATAs, including ones created from traffic. `[]` when there are none. | | `GET /iatas/{iata}` | One IATA. `404` if unknown or not three letters. | | `GET /iatas/{iata}/border` | GeoJSON Feature (Polygon or MultiPolygon, with `bbox`); `204` when the IATA has no border, `404` if the IATA is unknown. | -| `GET /regions` | Regions in display order. `[]` when none are configured. | +| `GET /regions` | Regions in display order, each with the same fields as `GET /regions/{regionId}` (`iatas` always present, `[]` for a region with none), so one call resolves every region. `[]` when none are configured. Older servers list only `id`, `slug` and `name`: when `iatas` is missing, fetch each region. | | `GET /regions/{regionId}` | One region with its IATAs. `400` for a non-numeric ID, `404` if unknown. | | `GET /scopes` | Array of scope names. Location filters narrow it to manual scopes configured for a matching region and imported scopes whose MeshMapper catalogue includes a matching IATA. | | `GET /scopes/{name}` | Scope detail (`%23bc` for `#bc`). | From 57e13735aa7507d6e9be99f2c7c389b5a402b56c Mon Sep 17 00:00:00 2001 From: MrAlders0n Date: Tue, 27 Oct 2026 08:08:33 -0400 Subject: [PATCH 17/21] docs(api): /nodes pages go up to 2000 --- docs/api-contract.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/api-contract.md b/docs/api-contract.md index 94a70f2..e0779ee 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -129,7 +129,7 @@ Most lists return a page: { "items": [ ... ], "nextCursor": 1747665456000, "hasMore": true } ``` -Pass `nextCursor` back as `cursor` for the next page; it is omitted on the last page. What the cursor holds depends on the list (a timestamp in epoch ms, or an ID), as noted per endpoint. `limit` has a per-endpoint default, must be positive, and is clamped to 200. +Pass `nextCursor` back as `cursor` for the next page; it is omitted on the last page. What the cursor holds depends on the list (a timestamp in epoch ms, or an ID), as noted per endpoint. `limit` has a per-endpoint default, must be positive, and is clamped to 200 (2000 for `GET /nodes`). ID cursors are 64-bit integers: the observation ID `cursor` of `/observers/{id}/adverts` and `/nodes/{id}/observations`, telemetry `afterId`, the message ID `cursor` of `/messages`, and the backfill `afterObservationId` and `afterId` ([Backpressure and reconnection](#backpressure-and-reconnection)). The same goes for the IDs they come from. Store and send them as full integers; never narrow them to 32 bits, since a busy server passes 2^31. As JSON numbers they stay exact up to 2^53, so a JavaScript `number` or a Dart `int` holds them. @@ -352,7 +352,7 @@ Adverts with a bad signature are not decoded and have no `parsedPayload`. | Endpoint | Notes | |---|---| -| `GET /nodes` | Page of node summaries, newest `lastSeen` first, ties by `id` descending. Cursor is `lastSeen` epoch ms; `pageCursor` is exact. Default limit 50. | +| `GET /nodes` | Page of node summaries, newest `lastSeen` first, ties by `id` descending. Cursor is `lastSeen` epoch ms; `pageCursor` is exact. Default limit 50, at most 2000 (older servers clamp to 200, so ask for 2000 and keep paging): clients that load a whole region, like the map, use big pages. | | `GET /nodes/{nodeId}` | Node detail: capability flags, `minFirmwareVersion`, `iatas` (`[{iata, lastHeard}]`), `neighbors`, clock drift for repeaters and room servers, and `meshmapper` when imported (see [MeshMapper repeater details](#meshmapper-repeater-details)). | | `GET /nodes/{nodeId}/observations` | Page of observations of packets the node originated. Cursor is the observation ID. | | `GET /nodes/{nodeId}/neighbors` | Plain array of neighbours: `{ id, name, publicKey, nodeType, nodeTypeName, lat, lng, iata, observationCount, firstSeen, lastSeen, snr }`. | From 9365235e08154b420f2e43591476da01f1dd6c74 Mon Sep 17 00:00:00 2001 From: MrAlders0n Date: Tue, 27 Oct 2026 15:31:30 -0400 Subject: [PATCH 18/21] docs(api): /routes takes several IATAs --- docs/api-contract.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/api-contract.md b/docs/api-contract.md index e0779ee..8ad4e77 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -581,14 +581,18 @@ Known routes are fully resolved multi-hop paths distilled from packet history. | Endpoint | Notes | |---|---| -| `GET /routes` | Plain array, newest `lastSeen` first. Params: `iata`, `hopCount`, `cursor` (last item's `lastSeen`, epoch ms), `cursorId` (last item's `id`), `limit` (default 50). | +| `GET /routes` | Plain array, newest `lastSeen` first. Params: `iatas` (comma list, at most 100), `iata`, `hopCount`, `cursor` (last item's `lastSeen`, epoch ms), `cursorId` (last item's `id`), `limit` (default 50, larger values clamp to 200). With neither `iatas` nor `iata`, every IATA. | | `GET /routes/search?from=&to=[&iatas=]` | Routes within one IATA between two node hash prefixes. `from`, `to` required (hex, 1–4 bytes; widths may differ). `iatas` optional: comma list, at most 100, omitted = every IATA; the single `iata` is still accepted. At most 500 results, shortest first. `hops` is trimmed to source…destination and `hopCount` = `hops.length`. | | `GET /routes/cross?fromHash=&toHash=[&iatas=]` | Routes that cross from one IATA into another. `fromHash`, `toHash` required. `iatas` optional: 2–100 codes, omitted = every IATA. The legacy `fromIata`+`toIata` pair is still accepted, but not together with `iatas`. At most 500 results, sorted by `totalHops`, then `crossHop.lastSeen` newest first. | | `GET /routes/{iata}/{pathKey}/observations` | Retained reports that match a saved route exactly. `pathKey` comes from a route response. Window: `range` (default `24h`, max `720h`) or `since`+`until` (max 30 days); `pageCursor` for the next page; `limit` default 50. A lookup past the statement limit answers `503` + `Retry-After` ([Cost guards](#cost-guards)). | To page `/routes`, send the last item's `lastSeen` as `cursor` and its `id` as `cursorId`. Routes share a millisecond often (one batch of upserts), and `cursorId` returns the ones that didn't fit on the previous page. `cursor` alone still works but can skip those ties. `cursorId` must be a positive integer and requires `cursor`; otherwise `400`. -An `iata` that ends in a blank (`YYZ` followed by a space) matches no region and returns `200 []`. +`iatas` returns the routes of every listed IATA, in the same order and with the same cursors as one IATA's list, so a region pages like a single IATA. Codes are trimmed, uppercased and deduplicated; one that isn't three characters, more than 100 codes, or a list with no codes (`iatas=,`) is a `400`. `iatas` takes precedence over `iata`. An empty `iatas=` counts as absent. + +The single `iata` is matched exactly as sent, so lowercase matches nothing. An `iata` that ends in a blank (`YYZ` followed by a space) matches no region and returns `200 []`. + +`iatas` is new in 2.1.0: older servers, and 2.1.0 development builds from before it, ignore it and answer with every IATA's routes. A client that asks for a region checks each route's `iata`; when one is outside the list, it pages without `iatas` and filters, as before. `/routes` and `/routes/search` return a plain array of routes: From d61676a11ee936d484d04dbc8b3b777b18c3efb3 Mon Sep 17 00:00:00 2001 From: MrAlders0n Date: Tue, 27 Oct 2026 15:51:44 -0400 Subject: [PATCH 19/21] docs(api): sorted /nodes lists --- docs/api-contract.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/api-contract.md b/docs/api-contract.md index 8ad4e77..f6ebc0d 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -139,8 +139,8 @@ A few lists return a plain array instead: the backfill endpoints, `/routes*`, `/ A numeric `cursor` holds a millisecond, so rows that share the last row's millisecond can fall between two pages. `/packets`, `/nodes`, `/observers` and `/channels` also return -`nextPageCursor` on a page with `hasMore`: an opaque string of at most 160 characters that marks -the exact last row. Send it back as `pageCursor`. A walk by `pageCursor` returns every row once, +`nextPageCursor` on a page with `hasMore`: an opaque string of at most 160 characters (512 on a +sorted `/nodes` list) that marks the exact last row. Send it back as `pageCursor`. A walk by `pageCursor` returns every row once, including those ties. Don't parse or build it; today it reads `v1::` on `/packets` and `v1::` on `/nodes` and `/observers`. @@ -352,12 +352,20 @@ Adverts with a bad signature are not decoded and have no `parsedPayload`. | Endpoint | Notes | |---|---| -| `GET /nodes` | Page of node summaries, newest `lastSeen` first, ties by `id` descending. Cursor is `lastSeen` epoch ms; `pageCursor` is exact. Default limit 50, at most 2000 (older servers clamp to 200, so ask for 2000 and keep paging): clients that load a whole region, like the map, use big pages. | +| `GET /nodes` | Page of node summaries, newest `lastSeen` first, ties by `id` descending, or in a `sort` order (see below). Cursor is `lastSeen` epoch ms; `pageCursor` is exact. Default limit 50, at most 2000 (older servers clamp to 200, so ask for 2000 and keep paging): clients that load a whole region, like the map, use big pages. | | `GET /nodes/{nodeId}` | Node detail: capability flags, `minFirmwareVersion`, `iatas` (`[{iata, lastHeard}]`), `neighbors`, clock drift for repeaters and room servers, and `meshmapper` when imported (see [MeshMapper repeater details](#meshmapper-repeater-details)). | | `GET /nodes/{nodeId}/observations` | Page of observations of packets the node originated. Cursor is the observation ID. | | `GET /nodes/{nodeId}/neighbors` | Plain array of neighbours: `{ id, name, publicKey, nodeType, nodeTypeName, lat, lng, iata, observationCount, firstSeen, lastSeen, snr }`. | -`/nodes` params: `type` (1 companion, 2 repeater, 3 room server, 4 sensor) or `typeName`, the location filters, `name` (partial, case-insensitive), `scope`, `pubkey` (exact hex), `pubkeyPrefix`, `supportsMultibytePaths`, `supportsMultibyteTraces`, `neighbors` (adds `neighborIds`), `cursor`, `pageCursor`, `limit`. Summaries include `stale` (not seen within `nodes.stale_threshold`) and, when `nodes.mark_foreign` is on, `possiblyForeign`. +`/nodes` params: `type` (1 companion, 2 repeater, 3 room server, 4 sensor) or `typeName`, the location filters, `name` (partial, case-insensitive), `scope`, `pubkey` (exact hex), `pubkeyPrefix`, `supportsMultibytePaths`, `supportsMultibyteTraces`, `neighbors` (adds `neighborIds`), `sort`, `order`, `cursor`, `pageCursor`, `limit`. Summaries include `stale` (not seen within `nodes.stale_threshold`) and, when `nodes.mark_foreign` is on, `possiblyForeign`. + +Sorted node lists (2.1.0): `sort` is `lastSeen` (the default), `name`, `type` or `neighbors`, and `order` is `asc` (the default) or `desc`; `lastSeen` is `desc` only. Every page says which order it is in with `sort` and `order` (`lastSeen` and `desc` by default). The orders match what the web's node table shows: + +- `name`: natural order, as `Intl.Collator` with `numeric: true` sorts (symbols and emoji before digits, digits before letters, `Node 9` before `Node 10`, lowercase before uppercase). Nodes without a name come last in both directions. +- `type`: companion, repeater, room server, sensor, then any other type. +- `neighbors`: `knownNeighborCount`. + +Ties go newest `lastSeen` first, then by `id`. Filters work the same as without `sort`. Page a sorted list with `pageCursor` only: a numeric `cursor`, or a `pageCursor` from another sort or order, is a `400`. A sorted page has no `nextCursor`. An unknown `sort` or `order` is a `400`. A server older than 2.1.0 ignores both and returns pages in `lastSeen` order with no `sort` field; a client that sees no `sort` must sort for itself. #### MeshMapper repeater details From 90ae38035e4c07ba61827e4ca868900a5c60b4c0 Mon Sep 17 00:00:00 2001 From: MrAlders0n Date: Tue, 27 Oct 2026 16:17:36 -0400 Subject: [PATCH 20/21] docs(api): node list rows carry defaultScope --- docs/api-contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/api-contract.md b/docs/api-contract.md index f6ebc0d..38b6a21 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -357,7 +357,7 @@ Adverts with a bad signature are not decoded and have no `parsedPayload`. | `GET /nodes/{nodeId}/observations` | Page of observations of packets the node originated. Cursor is the observation ID. | | `GET /nodes/{nodeId}/neighbors` | Plain array of neighbours: `{ id, name, publicKey, nodeType, nodeTypeName, lat, lng, iata, observationCount, firstSeen, lastSeen, snr }`. | -`/nodes` params: `type` (1 companion, 2 repeater, 3 room server, 4 sensor) or `typeName`, the location filters, `name` (partial, case-insensitive), `scope`, `pubkey` (exact hex), `pubkeyPrefix`, `supportsMultibytePaths`, `supportsMultibyteTraces`, `neighbors` (adds `neighborIds`), `sort`, `order`, `cursor`, `pageCursor`, `limit`. Summaries include `stale` (not seen within `nodes.stale_threshold`) and, when `nodes.mark_foreign` is on, `possiblyForeign`. +`/nodes` params: `type` (1 companion, 2 repeater, 3 room server, 4 sensor) or `typeName`, the location filters, `name` (partial, case-insensitive), `scope`, `pubkey` (exact hex), `pubkeyPrefix`, `supportsMultibytePaths`, `supportsMultibyteTraces`, `neighbors` (adds `neighborIds`), `sort`, `order`, `cursor`, `pageCursor`, `limit`. Summaries include `stale` (not seen within `nodes.stale_threshold`), `defaultScope` (the transport scope last matched from the node's adverts, omitted when none; list rows carry it from 2.1.0, node detail always has) and, when `nodes.mark_foreign` is on, `possiblyForeign`. To list one scope's nodes, send `scope` rather than filtering rows: a server before 2.1.0 leaves `defaultScope` off list rows. Sorted node lists (2.1.0): `sort` is `lastSeen` (the default), `name`, `type` or `neighbors`, and `order` is `asc` (the default) or `desc`; `lastSeen` is `desc` only. Every page says which order it is in with `sort` and `order` (`lastSeen` and `desc` by default). The orders match what the web's node table shows: From 0159855d48d89e992d7d5e0c4afb0ba29e4e2a97 Mon Sep 17 00:00:00 2001 From: MrAlders0n Date: Thu, 29 Oct 2026 11:07:24 -0400 Subject: [PATCH 21/21] docs(upgrading): 2.1 runs migration 061 and needs app 2.1 --- docs/upgrading.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/upgrading.md b/docs/upgrading.md index 3d7d454..4a72d6c 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -156,11 +156,12 @@ banned address could still reach the site over HTTP/3. ## Upgrading to 2.1 -Server 2.1.0 is one upgrade from any 2.0.x release. It runs migrations `004` to `060` at startup, +Server 2.1.0 is one upgrade from any 2.0.x release. It runs migrations `004` to `061` at startup, adds two read models that region and scope pages are served from, and drops indexes and one column nothing reads any more. No config change is required. Beacon Web must move to 2.1 at the -same time (`minWebVersion` is 2.1.0; an older web build shows the reload screen). The minimum app -version stays 2.0.0, so installed apps keep working. +same time (`minWebVersion` is 2.1.0; an older web build shows the reload screen). BEACON Mobile +needs 2.1 too (`minAppVersion` is 2.1.0; an older release build shows the update screen). Beta +builds are never blocked, so installed betas keep working. The upgrade has two phases. First the server runs its migrations while the API and ingest are down. Then it serves the API again and fills the read models in the background; until each one @@ -226,7 +227,7 @@ routes, 30 million observations and 3 million packets; default pacing): | Step | Time | |---|---| | Pre-building the eight indexes, old version still serving | 1 min 50 s (route hop index 72 s, adverts index 16 s, the rest under 8 s each) | -| Migrations `004` to `053` at startup (a 2.1 development build; the release adds `054` to `060`, a few seconds more) | 7.4 s (the two `packets` checks 3.7 s and 1.8 s) | +| Migrations `004` to `053` at startup (a 2.1 development build; the release adds `054` to `061`, a few seconds more) | 7.4 s (the two `packets` checks 3.7 s and 1.8 s) | | API and ingest down | about 15 s | | Channel message pages by site back | 8 min after startup | | Observer advert pages | at once (no untyped observations) |