diff --git a/app_config/config.yaml.example b/app_config/config.yaml.example index 6e8c2c3..b34207b 100644 --- a/app_config/config.yaml.example +++ b/app_config/config.yaml.example @@ -37,6 +37,9 @@ server: # (0.0.0.0:9090 inside Docker, published on the host's 127.0.0.1 only). Off when empty. # It has no authentication: bind it to a private interface. admin_listen: "" + # read_only: true boots a server that never writes: no migrations, no ingest, no background jobs, no + # MeshMapper imports, and every database session is read-only. Used to compare two builds on one database. + read_only: false # Applies to /api/v1/* only. Both counters are local to this Beacon instance. # Set trusted_proxies correctly before using a reverse proxy, or clients share @@ -55,8 +58,8 @@ ratelimit: # Keep max_connections well above that. A pool_max_conns in POSTGRES_DSN is overridden. #database: # # custom_all (default): every API statement gets a custom plan. -# # split: statements admitted to the generic-plan class run on cached generic plans. No statement -# # is admitted yet, so split has no effect; keep custom_all. +# # split: statements admitted to the generic-plan class (GetNodeByID, ListKnownRoutesByIATA, +# # ListKnownRoutesGlobal) run on cached generic plans. Keep custom_all unless the plan check is clean. # # single: rollback. One API pool on the previous settings; every guard family observes. # api_plan_layout: custom_all @@ -70,6 +73,12 @@ ratelimit: # enforce: [] # budget: shadow # the per-client DB-time budget only records; enforcement is not available +# Daily read-only plan regression check (GET /api/v1/admin/db/planguard, beacon_planguard_* metrics). +# Enable on large deployments; plan expectations assume production-sized tables. +# Cannot be enabled together with server.read_only. +#planguard: +# daily_check: false + # MeshMapper integrations (optional, require a regional or grouped-region API key). # Your local MeshMapper regional or grouped-region admin can generate the key. # A grouped-region key covers multiple regions, including imported group members. diff --git a/docs/api-contract.md b/docs/api-contract.md index c587da4..2972108 100644 --- a/docs/api-contract.md +++ b/docs/api-contract.md @@ -706,6 +706,7 @@ All under `/api/v1/admin/`, bearer key required (see [Auth](#auth)). | `GET /admin/config` | Running CORS settings with defaults applied, `auth.configured`, `ingest.broker_count` (configured broker workers, not connection status), `mobile.min_app_version` and `web.min_web_version` (read-only effective floors; `""` when there is none), and `guards` (below). No credentials, broker addresses, channel material or database connection settings. | | `PUT /admin/config` | Replaces `cors.allowed_origins`, `guards.enforce` or both until the next restart. JSON body up to 16 KiB. | | `GET /admin/db/statements` | `pg_stat_statements` for this database, summed by statement name. `503` when the extension is missing or not preloaded. | +| `GET /admin/db/planguard` | The last report of the daily plan check. `?fresh=true` runs it now, at most one run per 5 minutes. `404` while `planguard.daily_check` is off or before the first run; `500` when no run has succeeded. Details below. | | `GET /admin/accounts`, `POST /admin/accounts` | List or create operator account records (name only; not logins). | | `GET /admin/accounts/{id}`, `DELETE /admin/accounts/{id}` | Fetch or deactivate one. | | `GET /admin/backup` | Streams a private `.tar.gz` of the database and saved config. Off unless `backup.enabled: true`; needs `pg_dump` in the server's runtime. One export at a time (`409`), `504` on timeout, `507` over the size limit. Contains secrets. | @@ -739,6 +740,58 @@ Times are milliseconds rounded to microseconds. `plan_ms` stays 0 unless `pg_sta Counters run from the last `pg_stat_statements` reset. Setup is in [Operations](operations.md#statement-statistics). +`GET /admin/db/planguard` reports the daily plan check (see +[Configuration](configuration.md#daily-plan-check)). It is admin-only and changes nothing in the +public API. Like the other `/admin/db/` reports, its fields are snake_case, and `started_at` is +an RFC 3339 timestamp rather than epoch milliseconds: + +```json +{ + "tier": "prod", + "started_at": "2026-10-19T14:02:11.482Z", + "duration_ms": 41873.2, + "clean": true, + "counts": { "ok": 120, "RETIRED_INDEX": 2, "SKIP": 640 }, + "failures": [], + "runs": [ + { + "entry": "routes.byiata.R01_SEA_first_l50", + "stmt": "ListKnownRoutesByIATA", + "mode": "custom", + "verdict": "ok", + "drivers": ["known_routes:Index Scan:idx_known_routes_iata_keyset"] + } + ] +} +``` + +- `clean` is true when no run is `FALSE_POSITIVE`, `MISSED` or `ERROR`. +- `counts` maps each verdict to its number of runs. +- `runs` has one item per corpus entry and plan mode (`custom` or `generic`): `entry`, `stmt`, + `mode`, `verdict`, and when they apply `failures` (the plan rules the run broke), `drivers` + (`relation:node:index` of each driving scan) and `skip` (why the run was not made). +- `failures` holds the failing runs in the same shape. It is `[]`, never null, when there are + none. +- `verdict` is one of `ok`, `DETECTED`, `MISSED`, `FALSE_POSITIVE`, `RETIRED_INDEX`, + `KNOWN_ISSUE`, `SKIP` or `ERROR`. What each means is in + [Operations](operations.md#plan-checks-planguard). + +Without `fresh`, it returns the last successful report, or `404` with +`planguard has not run yet` (the first run is about an hour after startup) or +`planguard daily check is disabled`. A read-only server cannot turn the check on, so it always +answers the latter. + +`?fresh=true` runs the check now and waits for it, which can take minutes. A request during a run +joins that run. Within 5 minutes of the end of the last run it starts nothing and returns the last +successful report (`started_at` gives its age), or `500` when no run has succeeded yet. A run +that fails is a `500` too. A run started this way keeps going if the client gives up, for at most +15 minutes. A `fresh` value that is not a boolean is a `400`. + +The check also feeds two metrics on the [metrics listener](operations.md#metrics): +`beacon_planguard_failures{entry,mode}` (1 for each failing run of the last check) and +`beacon_planguard_last_run_timestamp_seconds` (when the last successful check finished; 0 until +the first one, and absent while the check is off). + `POST /admin/accounts` takes `{"name": "..."}` in a body of at most 4 KiB. The name is trimmed, case-sensitive, 1 to 128 characters with no control characters, and unique among active accounts. `DELETE` deactivates: `204`, or `404` if missing, `409` if already inactive. The name diff --git a/docs/configuration.md b/docs/configuration.md index 6198622..816bf55 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -67,9 +67,10 @@ which ones to read. |---|---|---| | `auth` | The admin API key, if you would rather keep it in the file than in `BEACON_API_KEY`. | You want the admin endpoints. | | `log` | Log level and format. | You want JSON logs for a collector. | -| `server` | `trusted_proxies`: the proxy addresses allowed to tell Beacon the real client IP. `admin_listen`: an optional private listener for Prometheus `/metrics`. | Always, if Beacon sits behind a proxy. See [Reverse proxy](reverse-proxy.md). | +| `server` | `trusted_proxies`: the proxy addresses allowed to tell Beacon the real client IP. `admin_listen`: an optional private listener for Prometheus `/metrics`. `read_only`: boot a server that never writes, for comparing two builds (default `false`). | Always, if Beacon sits behind a proxy. See [Reverse proxy](reverse-proxy.md) and [Read-only mode](#read-only-mode). | | `ratelimit` | Per-client REST limits for `/api/v1/*`. On by default at 600 requests a minute. | You get 429s you did not expect. | | `database` | `api_plan_layout`: how API statements are spread over database connection pools. | You check `max_connections`, or need the rollback layout. See [Database pools and cost guards](#database-pools-and-cost-guards). | +| `planguard` | `daily_check`: a daily read-only check that the database still plans Beacon's queries the way they were designed. Off by default. | You run a large deployment. See [Daily plan check](#daily-plan-check). | | `guards` | Which request families the cost guards may refuse. All of them only observe by default. | After a week of observing. See [Database pools and cost guards](#database-pools-and-cost-guards). | | `meshmapper` | Import transport scopes, IATA borders, region groups and public channels from MeshMapper instead of listing them by hand, and add repeater site details to node detail. Requires a [MeshMapper API key](#meshmapper-api-key). | You are in a region MeshMapper covers. | | `iatas` | Display names, coordinates and optional border files for airport codes. IATAs are created automatically when traffic arrives; this block only decorates them. A `borderFile` path is relative to `config.yaml`; in Docker the compose file mounts only `config.yaml`, so add a mount for the folder too (`- ./data/app/borders:/app/borders:ro` under the `app` service). | You want names on the map or a border drawn. | @@ -112,6 +113,8 @@ cannot honour. These are the ones people hit: - `database.api_plan_layout` other than `custom_all`, `split` or `single`; a `guards.enforce` entry that is not a family name; `guards.budget` other than `shadow`; `server.admin_listen` that is not `host:port` with a port from 1 to 65535. +- `planguard.daily_check: true` together with `server.read_only: true`. A read-only server runs + no background jobs, so it cannot run the check. - A negative `ratelimit.requests_per_minute`, `ratelimit.burst`, `websocket.max_connects_per_minute` or `routes.reconfirm_rows_per_hour`. A negative `websocket.max_connections_per_ip` is not caught at startup and rejects every @@ -135,8 +138,10 @@ layout sets the sizes. - **`custom_all`** plans every API statement for its own parameters on every call, so no statement can switch to a slow generic plan. Use it. -- **`split`** would run a list of proven statements on cached generic plans. That list is empty - in this release, so `split` only opens a pool nothing uses. Keep `custom_all`. +- **`split`** runs the statements the plan corpus admits as safe on cached generic plans, in + the `api_cached` pool. Today that is `GetNodeByID`, `ListKnownRoutesByIATA` and + `ListKnownRoutesGlobal`. Keep `custom_all` unless the [plan check](operations.md#plan-checks-planguard) + has run clean against this deployment's database. - **`single`** is the rollback: one API pool on the previous session settings, and every guard family only observes, whatever `guards.enforce` says. @@ -174,6 +179,49 @@ second per second) only counts; it never refuses. loopback or a private network and never publish it through the proxy. See [Operations](operations.md#metrics). +## Read-only mode + +```yaml +server: + read_only: false +``` + +`server.read_only: true` boots a server that never writes to the database. It skips the +migrations, the seeding of configured IATAs, regions and scopes, the boot backfill of channel +messages, MQTT ingest, the presence writer, the rollup catch-up, every background job and the +MeshMapper imports. Every database session it opens is read-only +(`default_transaction_read_only`), so anything that would still write, such as +`POST /api/v1/admin/accounts`, fails with SQLSTATE `25006` instead of changing data. The server +logs a warning at startup saying so. + +The REST API and `/ws` still serve what is in the database, but nothing new arrives. Because no +migrations run, the database must already be at the schema the build expects. Its use is +comparing two builds side by side against one copy of a database; see +[Operations](operations.md#comparing-two-builds). Do not run it as a production instance. + +## Daily plan check + +```yaml +planguard: + daily_check: false +``` + +`planguard.daily_check: true` makes the server check, once a day, that PostgreSQL still plans +each of Beacon's statements the way it was designed: the expected driving index and no +forbidden scans, in both custom and generic plan mode. The first run is about an hour after +startup. It plans only, with plain `EXPLAIN` on the maintenance pool, inside read-only +transactions with a 1 s lock timeout and 5 s per plan, so it never executes a statement or +changes data. + +It is off by default because the expected plans assume production-sized tables. On a small +database the planner legitimately picks other plans, and the check would report failures that +are not real. Turn it on for a large deployment; see +[Operations](operations.md#turning-on-the-daily-check). The report is served at +`GET /api/v1/admin/db/planguard` ([API contract](api-contract.md#admin)) and exported as +metrics. + +It cannot be combined with `server.read_only: true`; the server refuses to start with both. + ## MeshMapper API key MeshMapper APIs require either a regional API key or a grouped-region API key covering diff --git a/docs/operations.md b/docs/operations.md index 7b53876..f6b4a61 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -132,6 +132,10 @@ does not give way to a lock request the way an ordinary autovacuum does, so a mi that table waits until the vacuum ends. Let either finish, or end a stuck client session with `SELECT pg_cancel_backend();`, then deploy. +If you run the [daily plan check](#turning-on-the-daily-check), also confirm its report is clean +before and after the deploy. Maintainers run the plan checks before every server release; see +[Before a release](#before-a-release). + ### How startup migrations wait for locks Each migration statement runs with `lock_timeout = 30s`. When a statement times out on a lock, @@ -579,9 +583,12 @@ scrape_configs: | `beacon_pg_wal_bytes_total` | counter | | WAL written, cluster-wide | | `beacon_pg_checkpoints_total` | counter | `kind` | Checkpoints, `timed` or `requested` | | `beacon_pg_xid_age` | gauge | | Age of the oldest transaction ID in the database | +| `beacon_planguard_failures` | gauge | `entry`, `mode` | 1 for each corpus entry and plan mode that failed the last [plan check](#turning-on-the-daily-check) | +| `beacon_planguard_last_run_timestamp_seconds` | gauge | | Unix time the last successful plan check finished; 0 until the first one | `guard` is `deadline`, `cap`, `admission` or `budget`. The Postgres counters are read once a -minute; the index gauges first appear an hour after startup. Go runtime and process metrics +minute; the index gauges first appear an hour after startup. The two `beacon_planguard_*` +gauges exist only with `planguard.daily_check: true`. Go runtime and process metrics are included too. ### Example alert rules @@ -610,6 +617,11 @@ groups: expr: rate(beacon_pg_wal_bytes_total[1h]) > - alert: BeaconXIDAge expr: beacon_pg_xid_age > 300e6 + # The two plan check rules only matter with planguard.daily_check: true. + - alert: BeaconPlanguardFailures # the last plan check found a regression + expr: sum(beacon_planguard_failures) > 0 + - alert: BeaconPlanguardStale # no successful plan check for 26 h + expr: time() - beacon_planguard_last_run_timestamp_seconds > 26*3600 and on() time() - process_start_time_seconds > 26*3600 ``` Replace `` with your normal WAL rate before loading the rules. The route @@ -683,6 +695,172 @@ When `BeaconKeysetBloat` fires, rebuild that index as in [Rebuilding the route keyset indexes](#rebuilding-the-route-keyset-indexes). The rebuild resets its baseline, and the ratio drops back to about 1 on the next hourly run. +## Plan checks (planguard) + +planguard checks that PostgreSQL still plans each of Beacon's statements the way it was designed. +Its corpus, in the server repo under `db/planguard/`, lists every API and job statement with +parameters and what its plan must look like: the driving index and its index conditions, the +scans it must never use, and buffer, row and time budgets. It runs at three tiers against three +databases: + +| Tier | Database | How it plans | Checks | Who runs it | +|---|---|---|---|---| +| `fixture` | A synthetic database built through the migrations | Plain `EXPLAIN` | Plan shapes, against the committed `baseline/fixture` | CI, on every server pull request | +| `local` | A copy of a large production database, or a restored dump | `EXPLAIN (ANALYZE, BUFFERS)`, read-only | Plan shapes and budgets, against your own `baseline/local` | Maintainers, before a release | +| `prod` | A live production database | Plain `EXPLAIN`, read-only, 5 s per plan | Plan shapes only, no baseline | The server's [daily check](#turning-on-the-daily-check), or by hand through psql | + +Every tier runs each statement inside a `READ ONLY` transaction with a 1 s lock timeout and rolls +it back. Only the `local` tier executes statements, and then only reads; writes are only ever +planned. + +Each run gets a verdict: + +| Verdict | Meaning | Blocks a release | +|---|---|---| +| `ok` | The plan matches its expectations. | No | +| `FALSE_POSITIVE` | A statement that should pass broke a rule: a real regression, or an expectation that no longer holds on this database. | Yes | +| `MISSED` | A planted regression (an entry that must fail) passed, so the check itself is not catching what it should. | Yes | +| `ERROR` | The statement could not be planned. | Yes | +| `DETECTED` | A planted regression failed, as it should. | No | +| `RETIRED_INDEX` | The only failures name an index that a pending change drops. Expected until that change ships. | No | +| `KNOWN_ISSUE` | A failure the corpus already records as known. | No | +| `SKIP` | The run was not made, for example a statement whose feature is not built yet or whose parameters found no value. | No | + +The command line tool is `cmd/planguard` in the server repo. Run it from a checkout of the commit +you are about to release. It exits 0 when no run fails and 1 when one does, and writes its +results to `planguard-out/`. + +### Fixture tier + +CI builds the fixture and runs this tier on every pull request. To run it yourself you need an +empty database with C collation on PostgreSQL 16 (CI pins 16.15): + +```sql +CREATE DATABASE planguard TEMPLATE template0 LC_COLLATE 'C' LC_CTYPE 'C' ENCODING 'UTF8'; +``` + +```bash +go run ./cmd/planguard run --tier fixture --dsn 'postgres://...:5432/planguard' +``` + +The first run builds the fixture through the migrations, which takes a minute or two; later runs +reuse it while its inputs are unchanged. The builder refuses a database that already holds data +or is not C-collated. + +### Local tier + +The local tier is the release gate. Run it against a copy of a large production database, or a +restored dump, never against the live one: it executes every read statement. The copy needs its +own baseline (written to `db/planguard/baseline/local/` in your checkout), so write one on the +first run, review it, then check against it: + +```bash +go run ./cmd/planguard run --tier local --dsn "$COPY_DSN" --baseline write +go run ./cmd/planguard run --tier local --dsn "$COPY_DSN" # --baseline check is the default +``` + +A plan that changes against the baseline fails the check run. When the change is intended, write +the baseline again and keep it with the copy. `--only` and `--skip` take a regular expression of +entry ids to narrow a run. + +### Prod tier through psql + +The server's daily check runs the prod tier itself. To get the same evidence from a database the +server does not check, for example before turning the check on, emit read-only psql scripts and +run them yourself. Nothing connects from the tool; you run each script and hand its output back: + +```bash +go run ./cmd/planguard run --tier prod --emit resolvers > resolvers.sql +docker compose exec -T db psql -X -U beacon -d beacon < resolvers.sql > resolvers.out + +go run ./cmd/planguard run --tier prod --emit plans --params resolvers.out > plans.sql +docker compose exec -T db psql -X -U beacon -d beacon < plans.sql > plans.out + +go run ./cmd/planguard run --tier prod --params resolvers.out --plans plans.out +``` + +The first script looks up parameter values (20 s statement timeout); the second runs plain +`EXPLAIN`s with them. Both set `default_transaction_read_only` and a 1 s lock timeout. Each +script carries a random marker, and the last step rejects output that is truncated or does not +match the script. + +### Before a release + +Before tagging a server release, maintainers confirm: + +1. CI's fixture tier is clean. +2. The local tier, against your copy and its baseline, is clean. +3. On a deployment with the daily check on, `GET /api/v1/admin/db/planguard?fresh=true` returns + `"clean": true`, before and after the deploy. + +`RETIRED_INDEX` runs are expected while an index drop is pending; each names the index that will +go. + +### Turning on the daily check + +The daily check is for large deployments only. Its expectations assume production-sized +tables, and on a small database the planner rightly picks other plans, which the check reports +as failures. Before turning it on, run the prod tier through psql once and look at the result. +Then set: + +```yaml +planguard: + daily_check: true +``` + +and restart the server. It cannot be combined with `server.read_only`. The first check runs +about an hour after startup and then once a day; a failed run is retried every hour. Read the +last report, or run one now (at most one new run per 5 minutes), with the admin key: + +```bash +curl -s https:///api/v1/admin/db/planguard \ + -H "Authorization: Bearer $BEACON_API_KEY" +curl -s 'https:///api/v1/admin/db/planguard?fresh=true' \ + -H "Authorization: Bearer $BEACON_API_KEY" +``` + +`clean` must be true. `failures` lists each failing run with the rules it broke and the indexes +it used. The response is described in the [API contract](api-contract.md#admin). + +With [metrics](#metrics) on, add the two plan check alerts from the +[example rules](#example-alert-rules): + +- `BeaconPlanguardFailures` fires when the last check had a failing run. + `beacon_planguard_failures` names the entry and mode. +- `BeaconPlanguardStale` fires when no check has succeeded for 26 hours. The last-run gauge is + 0 from startup until the first success, so a check that never succeeds fires it too. The + `process_start_time_seconds` condition keeps a recent restart from firing it before the first + run has had time to happen. + +To turn the check off, set `daily_check: false` and restart. Nothing is stored in the database. + +### Comparing two builds + +`server.read_only: true` (see [Configuration](configuration.md#read-only-mode)) lets two server +builds serve the same database copy side by side, so a change to a query can be checked against +real data before it ships. `planguard storediff` then walks the same API pages on both and +compares the whole JSON of every page. + +1. Build the base and the candidate server binaries. +2. Give each a `config.yaml` with `server.read_only: true` and `ratelimit.enabled: false`, and + leave `REDIS_ADDR` unset so neither serves cached responses. Point both `POSTGRES_DSN`s at + the same copy, already migrated to the schema both builds expect. Indexes or other objects + only the candidate needs can live in a scratch schema on that copy, put first on the + candidate's search path with `?search_path=,public` in its DSN. +3. Start them on two ports, for example 8481 and 8482. Each logs a read-only warning at startup + and no migration lines. +4. Compare them: + + ```bash + go run ./cmd/planguard storediff --old http://127.0.0.1:8481 --new http://127.0.0.1:8482 \ + --cases db/planguard/storediff/smoke.yaml --walk 1,2,3,web --out storediff-out + ``` + +`--walk` lists the page sizes to walk; `web` is each case's web page size. Small page sizes catch +pagination that skips or repeats rows. The tool exits non-zero when any page differs, and with +`--out` writes `storediff_report.md` there. Never use read-only mode on your live instance: it +ingests nothing. + ## Rolling back the pool layout If the new pools or plans misbehave, set this and restart the server: