From e2b59e9fd5d603b1e1a1def019779a98f23c3005 Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Wed, 21 Oct 2026 11:29:09 -0400 Subject: [PATCH] Document statement-timeout 503s, read-only scope restore, read-model restore logs (#40) --- docs/api-contract.md | 4 +++- docs/configuration.md | 4 ++++ docs/operations.md | 12 +++++++++--- 3 files changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/api-contract.md b/docs/api-contract.md index 8d86aad..930885f 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -60,6 +60,8 @@ Retry-After: 5 The body is the usual error envelope. Wait at least `Retry-After` seconds before retrying; a `deadline_exceeded` request will likely run as long again, so narrow it (a shorter window, a region) rather than repeat it. Clients are keyed by IP after the trusted-proxy step above. +Separately from the guards, and on every server, a page whose database statement runs past its pool's statement limit (35 s; 120 s for the slow region and message statements) answers `503` with code `deadline_exceeded` and `Retry-After: 10`. This covers route evidence (`/routes/{iata}/{pathKey}/observations`) and the [background-data pages](#pages-that-wait-for-server-data) (region packets, region and scope messages, observer adverts). Other endpoints still answer `500` in that case. + Each request falls in one family and one class: | Path | Condition | Family | Class | @@ -582,7 +584,7 @@ Known routes are fully resolved multi-hop paths distilled from packet history. | `GET /routes` | Plain array, newest `lastSeen` first. Params: `iata`, `hopCount`, `cursor` (last item's `lastSeen`, epoch ms), `cursorId` (last item's `id`), `limit` (default 50). | | `GET /routes/search?from=&to=[&iatas=]` | Routes within one IATA between two node hash prefixes. `from`, `to` required (hex, 1–4 bytes; widths may differ). `iatas` optional: comma list, at most 100, omitted = every IATA; the single `iata` is still accepted. At most 500 results, shortest first. `hops` is trimmed to source…destination and `hopCount` = `hops.length`. | | `GET /routes/cross?fromHash=&toHash=[&iatas=]` | Routes that cross from one IATA into another. `fromHash`, `toHash` required. `iatas` optional: 2–100 codes, omitted = every IATA. The legacy `fromIata`+`toIata` pair is still accepted, but not together with `iatas`. At most 500 results, sorted by `totalHops`, then `crossHop.lastSeen` newest first. | -| `GET /routes/{iata}/{pathKey}/observations` | Retained reports that match a saved route exactly. `pathKey` comes from a route response. Window: `range` (default `24h`, max `720h`) or `since`+`until` (max 30 days); `pageCursor` for the next page; `limit` default 50. | +| `GET /routes/{iata}/{pathKey}/observations` | Retained reports that match a saved route exactly. `pathKey` comes from a route response. Window: `range` (default `24h`, max `720h`) or `since`+`until` (max 30 days); `pageCursor` for the next page; `limit` default 50. A lookup past the statement limit answers `503` + `Retry-After` ([Cost guards](#cost-guards)). | To page `/routes`, send the last item's `lastSeen` as `cursor` and its `id` as `cursorId`. Routes share a millisecond often (one batch of upserts), and `cursorId` returns the ones that didn't fit on the previous page. `cursor` alone still works but can skip those ties. `cursorId` must be a positive integer and requires `cursor`; otherwise `400`. diff --git a/docs/configuration.md b/docs/configuration.md index 260c982..1820cf0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -217,6 +217,10 @@ MeshMapper imports. Every database session it opens is read-only `POST /api/v1/admin/accounts`, fails with SQLSTATE `25006` instead of changing data. The server logs a warning at startup saying so. The one thing it still runs is a read-only check of each read model (channel messages and packets by site) every 5 minutes; see [Read models](#read-models). +When `meshmapper.scopes.enabled` is on, it also loads the writer's saved MeshMapper scope +catalogues once at boot, reading only, so region and IATA scope lists (`/scopes?region=`, +`/stats/scopes?region=`) match the writer's as of that boot. They don't refresh until the next +restart. The REST API and `/ws` still serve what is in the database, but nothing new arrives. Because no migrations run, the database must already be at the schema the build expects. Its use is diff --git a/docs/operations.md b/docs/operations.md index a2e0f4a..0f9db40 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -619,7 +619,8 @@ Trust is held in memory by each server process; nothing in the database marks th exact. A process earns it at every start, once the boot reconcile has run, the backfill is complete and the one-time verification has passed. It drops trust: -- after 3 reconciles in a row that each repaired at least one row. Two clean reconciles in a +- after 3 reconciles in a row that each repaired at least one row. It logs + `read model untrusted: repair runs fixed rows the writers missed`. Two clean reconciles in a row restore it; - at once when ingest leaves a message without its sites: a failed site insert, a message stored without its sites, or any message insert error. It logs @@ -628,6 +629,8 @@ complete and the one-time verification has passed. It drops trust: one interval; - when the stored state is no longer complete and verified, for example after a rebuild. +Each restore after a drop logs `read model trusted again: its pages are served from it`. + A reconcile that repairs rows logs `cmi reconcile repaired N pair(s): a writer bypassed the read model`. Something stored messages without recording their sites. On a single server running this release that should not happen. Running two server versions that both write to one database at @@ -791,7 +794,8 @@ complete and the one-time verification has passed. It drops trust: - at once when a reconcile or the hourly audit repairs any row: something bypassed the writer. Two clean reconciles in a row restore it. A repair logs `packet_sites reconcile repaired N row(s): a writer bypassed the read model` or - `packet_sites audit repaired N row(s) below the watermark`; + `packet_sites audit repaired N row(s) below the watermark`, and the drop logs + `read model untrusted: repair runs fixed rows the writers missed`; - at once when an observation insert with its site row fails. Ingest never loses the observation: it logs `observation site row failed; storing the observation without it` and stores it without its row. Trust drops with @@ -801,6 +805,8 @@ complete and the one-time verification has passed. It drops trust: - after 3 failed reconciles in a row, until one succeeds; - when the saved state is no longer complete and verified, for example after a rebuild. +Each restore after a drop logs `read model trusted again: its pages are served from it`. + On a single server running this release, repairs should not happen. Running two server versions that both write to one database at the same time is not supported; if it happens, the rows the older one leaves out are repaired within one reconcile interval, and trust returns two clean @@ -1318,7 +1324,7 @@ scrape_configs: | `beacon_packet_sites_front_live` | gauge | `index` | Index entries written in the last two minutes that the last front sweep visited | | `beacon_packet_sites_bytes` | gauge | | Total size of `packet_sites`, heap and indexes, hourly | | `beacon_db_sequence_used_ratio` | gauge | `sequence`, `cycle` | Share of a sequence's usable values already handed out, 0 to 1, each minute. The usable range ends at the sequence's own limit or its column's type limit (`integer` 2,147,483,647), whichever is lower. `cycle="true"` marks a `CYCLE` sequence, whose ratio is its position, not used-up headroom. See [Sequence headroom](#sequence-headroom). | -| `beacon_api_unavailable_total` | counter | `reason`, `model` | `503`s a page answered: `not_ready` while its read model or backfill is not ready, `slow_pool_busy` when `api_slow` had no free connection. `model` is `packet_sites`, `cmi` or `observation_payload_types`. | +| `beacon_api_unavailable_total` | counter | `reason`, `model` | `503`s a page answered: `not_ready` while its read model or backfill is not ready, `slow_pool_busy` when `api_slow` had no free connection, `timeout` when its statement ran past its pool's statement limit or an enforced guard's deadline. `model` is `packet_sites`, `cmi` or `observation_payload_types`. | `guard` is `deadline`, `cap`, `admission` or `budget`. The Postgres counters are read once a minute; the index gauges first appear an hour after startup. The two `beacon_planguard_*`