dandri/Beacon-Docs
1
0 Fork 0
mirror of https://github.com/MeshCore-Beacon/beacon-docs.git synced 2026-10-23 13:22:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Document planguard, read-only mode and the daily plan check (#30)

Browse Source
This commit is contained in:
MrAlders0n authored and GitHub committed 2026-10-19 13:50:36 -04:00
1 parent 096a463fcc
commit ca805e739b
4 files changed
+294 -6
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

app_config/config.yaml.example
+11 -2
View File
@@ -37,6 +37,9 @@ server:
# (0.0.0.0:9090 inside Docker, published on the host's 127.0.0.1 only). Off when empty.
# It has no authentication: bind it to a private interface.
admin_listen: ""
# read_only: true boots a server that never writes: no migrations, no ingest, no background jobs, no
# MeshMapper imports, and every database session is read-only. Used to compare two builds on one database.
read_only: false
# Applies to /api/v1/* only. Both counters are local to this Beacon instance.
# Set trusted_proxies correctly before using a reverse proxy, or clients share
@@ -55,8 +58,8 @@ ratelimit:
# Keep max_connections well above that. A pool_max_conns in POSTGRES_DSN is overridden.
#database:
# # custom_all (default): every API statement gets a custom plan.
# # split: statements admitted to the generic-plan class run on cached generic plans. No statement
# # is admitted yet, so split has no effect; keep custom_all.
# # split: statements admitted to the generic-plan class (GetNodeByID, ListKnownRoutesByIATA,
# # ListKnownRoutesGlobal) run on cached generic plans. Keep custom_all unless the plan check is clean.
# # single: rollback. One API pool on the previous settings; every guard family observes.
# api_plan_layout: custom_all
@@ -70,6 +73,12 @@ ratelimit:
# enforce: []
# budget: shadow # the per-client DB-time budget only records; enforcement is not available
# Daily read-only plan regression check (GET /api/v1/admin/db/planguard, beacon_planguard_* metrics).
# Enable on large deployments; plan expectations assume production-sized tables.
# Cannot be enabled together with server.read_only.
#planguard:
# daily_check: false
# MeshMapper integrations (optional, require a regional or grouped-region API key).
# Your local MeshMapper regional or grouped-region admin can generate the key.
# A grouped-region key covers multiple regions, including imported group members.
docs/api-contract.md
+53
View File
@@ -706,6 +706,7 @@ All under `/api/v1/admin/`, bearer key required (see [Auth](#auth)).
| `GET /admin/config` | Running CORS settings with defaults applied, `auth.configured`, `ingest.broker_count` (configured broker workers, not connection status), `mobile.min_app_version` and `web.min_web_version` (read-only effective floors; `""` when there is none), and `guards` (below). No credentials, broker addresses, channel material or database connection settings. |
| `PUT /admin/config` | Replaces `cors.allowed_origins`, `guards.enforce` or both until the next restart. JSON body up to 16 KiB. |
| `GET /admin/db/statements` | `pg_stat_statements` for this database, summed by statement name. `503` when the extension is missing or not preloaded. |
| `GET /admin/db/planguard` | The last report of the daily plan check. `?fresh=true` runs it now, at most one run per 5 minutes. `404` while `planguard.daily_check` is off or before the first run; `500` when no run has succeeded. Details below. |
| `GET /admin/accounts`, `POST /admin/accounts` | List or create operator account records (name only; not logins). |
| `GET /admin/accounts/{id}`, `DELETE /admin/accounts/{id}` | Fetch or deactivate one. |
| `GET /admin/backup` | Streams a private `.tar.gz` of the database and saved config. Off unless `backup.enabled: true`; needs `pg_dump` in the server's runtime. One export at a time (`409`), `504` on timeout, `507` over the size limit. Contains secrets. |
@@ -739,6 +740,58 @@ Times are milliseconds rounded to microseconds. `plan_ms` stays 0 unless `pg_sta
Counters run from the last `pg_stat_statements` reset. Setup is in
[Operations](operations.md#statement-statistics).
`GET /admin/db/planguard` reports the daily plan check (see
[Configuration](configuration.md#daily-plan-check)). It is admin-only and changes nothing in the
public API. Like the other `/admin/db/` reports, its fields are snake_case, and `started_at` is
an RFC 3339 timestamp rather than epoch milliseconds:
```json
{
"tier": "prod",
"started_at": "2026-10-19T14:02:11.482Z",
"duration_ms": 41873.2,
"clean": true,
"counts": { "ok": 120, "RETIRED_INDEX": 2, "SKIP": 640 },
"failures": [],
"runs": [
{
"entry": "routes.byiata.R01_SEA_first_l50",
"stmt": "ListKnownRoutesByIATA",
"mode": "custom",
"verdict": "ok",
"drivers": ["known_routes:Index Scan:idx_known_routes_iata_keyset"]
}
]
}
```
- `clean` is true when no run is `FALSE_POSITIVE`, `MISSED` or `ERROR`.
- `counts` maps each verdict to its number of runs.
- `runs` has one item per corpus entry and plan mode (`custom` or `generic`): `entry`, `stmt`,
`mode`, `verdict`, and when they apply `failures` (the plan rules the run broke), `drivers`
(`relation:node:index` of each driving scan) and `skip` (why the run was not made).
- `failures` holds the failing runs in the same shape. It is `[]`, never null, when there are
none.
- `verdict` is one of `ok`, `DETECTED`, `MISSED`, `FALSE_POSITIVE`, `RETIRED_INDEX`,
`KNOWN_ISSUE`, `SKIP` or `ERROR`. What each means is in
[Operations](operations.md#plan-checks-planguard).
Without `fresh`, it returns the last successful report, or `404` with
`planguard has not run yet` (the first run is about an hour after startup) or
`planguard daily check is disabled`. A read-only server cannot turn the check on, so it always
answers the latter.
`?fresh=true` runs the check now and waits for it, which can take minutes. A request during a run
joins that run. Within 5 minutes of the end of the last run it starts nothing and returns the last
successful report (`started_at` gives its age), or `500` when no run has succeeded yet. A run
that fails is a `500` too. A run started this way keeps going if the client gives up, for at most
15 minutes. A `fresh` value that is not a boolean is a `400`.
The check also feeds two metrics on the [metrics listener](operations.md#metrics):
`beacon_planguard_failures{entry,mode}` (1 for each failing run of the last check) and
`beacon_planguard_last_run_timestamp_seconds` (when the last successful check finished; 0 until
the first one, and absent while the check is off).
`POST /admin/accounts` takes `{"name": "..."}` in a body of at most 4 KiB. The name is trimmed,
case-sensitive, 1 to 128 characters with no control characters, and unique among active
accounts. `DELETE` deactivates: `204`, or `404` if missing, `409` if already inactive. The name
docs/configuration.md
+51 -3
View File
@@ -67,9 +67,10 @@ which ones to read.
|---|---|---|
| `auth` | The admin API key, if you would rather keep it in the file than in `BEACON_API_KEY`. | You want the admin endpoints. |
| `log` | Log level and format. | You want JSON logs for a collector. |
| `server` | `trusted_proxies`: the proxy addresses allowed to tell Beacon the real client IP. `admin_listen`: an optional private listener for Prometheus `/metrics`. | Always, if Beacon sits behind a proxy. See [Reverse proxy](reverse-proxy.md). |
| `server` | `trusted_proxies`: the proxy addresses allowed to tell Beacon the real client IP. `admin_listen`: an optional private listener for Prometheus `/metrics`. `read_only`: boot a server that never writes, for comparing two builds (default `false`). | Always, if Beacon sits behind a proxy. See [Reverse proxy](reverse-proxy.md) and [Read-only mode](#read-only-mode). |
| `ratelimit` | Per-client REST limits for `/api/v1/*`. On by default at 600 requests a minute. | You get 429s you did not expect. |
| `database` | `api_plan_layout`: how API statements are spread over database connection pools. | You check `max_connections`, or need the rollback layout. See [Database pools and cost guards](#database-pools-and-cost-guards). |
| `planguard` | `daily_check`: a daily read-only check that the database still plans Beacon's queries the way they were designed. Off by default. | You run a large deployment. See [Daily plan check](#daily-plan-check). |
| `guards` | Which request families the cost guards may refuse. All of them only observe by default. | After a week of observing. See [Database pools and cost guards](#database-pools-and-cost-guards). |
| `meshmapper` | Import transport scopes, IATA borders, region groups and public channels from MeshMapper instead of listing them by hand, and add repeater site details to node detail. Requires a [MeshMapper API key](#meshmapper-api-key). | You are in a region MeshMapper covers. |
| `iatas` | Display names, coordinates and optional border files for airport codes. IATAs are created automatically when traffic arrives; this block only decorates them. A `borderFile` path is relative to `config.yaml`; in Docker the compose file mounts only `config.yaml`, so add a mount for the folder too (`- ./data/app/borders:/app/borders:ro` under the `app` service). | You want names on the map or a border drawn. |
@@ -112,6 +113,8 @@ cannot honour. These are the ones people hit:
- `database.api_plan_layout` other than `custom_all`, `split` or `single`; a `guards.enforce`
entry that is not a family name; `guards.budget` other than `shadow`; `server.admin_listen`
that is not `host:port` with a port from 1 to 65535.
- `planguard.daily_check: true` together with `server.read_only: true`. A read-only server runs
no background jobs, so it cannot run the check.
- A negative `ratelimit.requests_per_minute`, `ratelimit.burst`, `websocket.max_connects_per_minute`
or `routes.reconfirm_rows_per_hour`.
A negative `websocket.max_connections_per_ip` is not caught at startup and rejects every
@@ -135,8 +138,10 @@ layout sets the sizes.
- **`custom_all`** plans every API statement for its own parameters on every call, so no
statement can switch to a slow generic plan. Use it.
- **`split`** would run a list of proven statements on cached generic plans. That list is empty
in this release, so `split` only opens a pool nothing uses. Keep `custom_all`.
- **`split`** runs the statements the plan corpus admits as safe on cached generic plans, in
the `api_cached` pool. Today that is `GetNodeByID`, `ListKnownRoutesByIATA` and
`ListKnownRoutesGlobal`. Keep `custom_all` unless the [plan check](operations.md#plan-checks-planguard)
has run clean against this deployment's database.
- **`single`** is the rollback: one API pool on the previous session settings, and every guard
family only observes, whatever `guards.enforce` says.
@@ -174,6 +179,49 @@ second per second) only counts; it never refuses.
loopback or a private network and never publish it through the proxy. See
[Operations](operations.md#metrics).
## Read-only mode
```yaml
server:
read_only: false
```
`server.read_only: true` boots a server that never writes to the database. It skips the
migrations, the seeding of configured IATAs, regions and scopes, the boot backfill of channel
messages, MQTT ingest, the presence writer, the rollup catch-up, every background job and the
MeshMapper imports. Every database session it opens is read-only
(`default_transaction_read_only`), so anything that would still write, such as
`POST /api/v1/admin/accounts`, fails with SQLSTATE `25006` instead of changing data. The server
logs a warning at startup saying so.
The REST API and `/ws` still serve what is in the database, but nothing new arrives. Because no
migrations run, the database must already be at the schema the build expects. Its use is
comparing two builds side by side against one copy of a database; see
[Operations](operations.md#comparing-two-builds). Do not run it as a production instance.
## Daily plan check
```yaml
planguard:
daily_check: false
```
`planguard.daily_check: true` makes the server check, once a day, that PostgreSQL still plans
each of Beacon's statements the way it was designed: the expected driving index and no
forbidden scans, in both custom and generic plan mode. The first run is about an hour after
startup. It plans only, with plain `EXPLAIN` on the maintenance pool, inside read-only
transactions with a 1 s lock timeout and 5 s per plan, so it never executes a statement or
changes data.
It is off by default because the expected plans assume production-sized tables. On a small
database the planner legitimately picks other plans, and the check would report failures that
are not real. Turn it on for a large deployment; see
[Operations](operations.md#turning-on-the-daily-check). The report is served at
`GET /api/v1/admin/db/planguard` ([API contract](api-contract.md#admin)) and exported as
metrics.
It cannot be combined with `server.read_only: true`; the server refuses to start with both.
## MeshMapper API key
MeshMapper APIs require either a regional API key or a grouped-region API key covering
docs/operations.md
+179 -1
View File
@@ -132,6 +132,10 @@ does not give way to a lock request the way an ordinary autovacuum does, so a mi
that table waits until the vacuum ends. Let either finish, or end a stuck client session with
`SELECT pg_cancel_backend(<pid>);`, then deploy.
If you run the [daily plan check](#turning-on-the-daily-check), also confirm its report is clean
before and after the deploy. Maintainers run the plan checks before every server release; see
[Before a release](#before-a-release).
### How startup migrations wait for locks
Each migration statement runs with `lock_timeout = 30s`. When a statement times out on a lock,
@@ -579,9 +583,12 @@ scrape_configs:
| `beacon_pg_wal_bytes_total` | counter | | WAL written, cluster-wide |
| `beacon_pg_checkpoints_total` | counter | `kind` | Checkpoints, `timed` or `requested` |
| `beacon_pg_xid_age` | gauge | | Age of the oldest transaction ID in the database |
| `beacon_planguard_failures` | gauge | `entry`, `mode` | 1 for each corpus entry and plan mode that failed the last [plan check](#turning-on-the-daily-check) |
| `beacon_planguard_last_run_timestamp_seconds` | gauge | | Unix time the last successful plan check finished; 0 until the first one |
`guard` is `deadline`, `cap`, `admission` or `budget`. The Postgres counters are read once a
minute; the index gauges first appear an hour after startup. Go runtime and process metrics
minute; the index gauges first appear an hour after startup. The two `beacon_planguard_*`
gauges exist only with `planguard.daily_check: true`. Go runtime and process metrics
are included too.
### Example alert rules
@@ -610,6 +617,11 @@ groups:
expr: rate(beacon_pg_wal_bytes_total[1h]) > <baseline bytes/s>
- alert: BeaconXIDAge
expr: beacon_pg_xid_age > 300e6
# The two plan check rules only matter with planguard.daily_check: true.
- alert: BeaconPlanguardFailures # the last plan check found a regression
expr: sum(beacon_planguard_failures) > 0
- alert: BeaconPlanguardStale # no successful plan check for 26 h
expr: time() - beacon_planguard_last_run_timestamp_seconds > 26*3600 and on() time() - process_start_time_seconds > 26*3600
```
Replace `<baseline bytes/s>` with your normal WAL rate before loading the rules. The route
@@ -683,6 +695,172 @@ When `BeaconKeysetBloat` fires, rebuild that index as in
[Rebuilding the route keyset indexes](#rebuilding-the-route-keyset-indexes). The rebuild resets
its baseline, and the ratio drops back to about 1 on the next hourly run.
## Plan checks (planguard)
planguard checks that PostgreSQL still plans each of Beacon's statements the way it was designed.
Its corpus, in the server repo under `db/planguard/`, lists every API and job statement with
parameters and what its plan must look like: the driving index and its index conditions, the
scans it must never use, and buffer, row and time budgets. It runs at three tiers against three
databases:
| Tier | Database | How it plans | Checks | Who runs it |
|---|---|---|---|---|
| `fixture` | A synthetic database built through the migrations | Plain `EXPLAIN` | Plan shapes, against the committed `baseline/fixture` | CI, on every server pull request |
| `local` | A copy of a large production database, or a restored dump | `EXPLAIN (ANALYZE, BUFFERS)`, read-only | Plan shapes and budgets, against your own `baseline/local` | Maintainers, before a release |
| `prod` | A live production database | Plain `EXPLAIN`, read-only, 5 s per plan | Plan shapes only, no baseline | The server's [daily check](#turning-on-the-daily-check), or by hand through psql |
Every tier runs each statement inside a `READ ONLY` transaction with a 1 s lock timeout and rolls
it back. Only the `local` tier executes statements, and then only reads; writes are only ever
planned.
Each run gets a verdict:
| Verdict | Meaning | Blocks a release |
|---|---|---|
| `ok` | The plan matches its expectations. | No |
| `FALSE_POSITIVE` | A statement that should pass broke a rule: a real regression, or an expectation that no longer holds on this database. | Yes |
| `MISSED` | A planted regression (an entry that must fail) passed, so the check itself is not catching what it should. | Yes |
| `ERROR` | The statement could not be planned. | Yes |
| `DETECTED` | A planted regression failed, as it should. | No |
| `RETIRED_INDEX` | The only failures name an index that a pending change drops. Expected until that change ships. | No |
| `KNOWN_ISSUE` | A failure the corpus already records as known. | No |
| `SKIP` | The run was not made, for example a statement whose feature is not built yet or whose parameters found no value. | No |
The command line tool is `cmd/planguard` in the server repo. Run it from a checkout of the commit
you are about to release. It exits 0 when no run fails and 1 when one does, and writes its
results to `planguard-out/`.
### Fixture tier
CI builds the fixture and runs this tier on every pull request. To run it yourself you need an
empty database with C collation on PostgreSQL 16 (CI pins 16.15):
```sql
CREATE DATABASE planguard TEMPLATE template0 LC_COLLATE 'C' LC_CTYPE 'C' ENCODING 'UTF8';
```
```bash
go run ./cmd/planguard run --tier fixture --dsn 'postgres://...:5432/planguard'
```
The first run builds the fixture through the migrations, which takes a minute or two; later runs
reuse it while its inputs are unchanged. The builder refuses a database that already holds data
or is not C-collated.
### Local tier
The local tier is the release gate. Run it against a copy of a large production database, or a
restored dump, never against the live one: it executes every read statement. The copy needs its
own baseline (written to `db/planguard/baseline/local/` in your checkout), so write one on the
first run, review it, then check against it:
```bash
go run ./cmd/planguard run --tier local --dsn "$COPY_DSN" --baseline write
go run ./cmd/planguard run --tier local --dsn "$COPY_DSN" # --baseline check is the default
```
A plan that changes against the baseline fails the check run. When the change is intended, write
the baseline again and keep it with the copy. `--only` and `--skip` take a regular expression of
entry ids to narrow a run.
### Prod tier through psql
The server's daily check runs the prod tier itself. To get the same evidence from a database the
server does not check, for example before turning the check on, emit read-only psql scripts and
run them yourself. Nothing connects from the tool; you run each script and hand its output back:
```bash
go run ./cmd/planguard run --tier prod --emit resolvers > resolvers.sql
docker compose exec -T db psql -X -U beacon -d beacon < resolvers.sql > resolvers.out
go run ./cmd/planguard run --tier prod --emit plans --params resolvers.out > plans.sql
docker compose exec -T db psql -X -U beacon -d beacon < plans.sql > plans.out
go run ./cmd/planguard run --tier prod --params resolvers.out --plans plans.out
```
The first script looks up parameter values (20 s statement timeout); the second runs plain
`EXPLAIN`s with them. Both set `default_transaction_read_only` and a 1 s lock timeout. Each
script carries a random marker, and the last step rejects output that is truncated or does not
match the script.
### Before a release
Before tagging a server release, maintainers confirm:
1. CI's fixture tier is clean.
2. The local tier, against your copy and its baseline, is clean.
3. On a deployment with the daily check on, `GET /api/v1/admin/db/planguard?fresh=true` returns
`"clean": true`, before and after the deploy.
`RETIRED_INDEX` runs are expected while an index drop is pending; each names the index that will
go.
### Turning on the daily check
The daily check is for large deployments only. Its expectations assume production-sized
tables, and on a small database the planner rightly picks other plans, which the check reports
as failures. Before turning it on, run the prod tier through psql once and look at the result.
Then set:
```yaml
planguard:
daily_check: true
```
and restart the server. It cannot be combined with `server.read_only`. The first check runs
about an hour after startup and then once a day; a failed run is retried every hour. Read the
last report, or run one now (at most one new run per 5 minutes), with the admin key:
```bash
curl -s https://<your-domain>/api/v1/admin/db/planguard \
-H "Authorization: Bearer $BEACON_API_KEY"
curl -s 'https://<your-domain>/api/v1/admin/db/planguard?fresh=true' \
-H "Authorization: Bearer $BEACON_API_KEY"
```
`clean` must be true. `failures` lists each failing run with the rules it broke and the indexes
it used. The response is described in the [API contract](api-contract.md#admin).
With [metrics](#metrics) on, add the two plan check alerts from the
[example rules](#example-alert-rules):
- `BeaconPlanguardFailures` fires when the last check had a failing run.
`beacon_planguard_failures` names the entry and mode.
- `BeaconPlanguardStale` fires when no check has succeeded for 26 hours. The last-run gauge is
0 from startup until the first success, so a check that never succeeds fires it too. The
`process_start_time_seconds` condition keeps a recent restart from firing it before the first
run has had time to happen.
To turn the check off, set `daily_check: false` and restart. Nothing is stored in the database.
### Comparing two builds
`server.read_only: true` (see [Configuration](configuration.md#read-only-mode)) lets two server
builds serve the same database copy side by side, so a change to a query can be checked against
real data before it ships. `planguard storediff` then walks the same API pages on both and
compares the whole JSON of every page.
1. Build the base and the candidate server binaries.
2. Give each a `config.yaml` with `server.read_only: true` and `ratelimit.enabled: false`, and
leave `REDIS_ADDR` unset so neither serves cached responses. Point both `POSTGRES_DSN`s at
the same copy, already migrated to the schema both builds expect. Indexes or other objects
only the candidate needs can live in a scratch schema on that copy, put first on the
candidate's search path with `?search_path=<scratch>,public` in its DSN.
3. Start them on two ports, for example 8481 and 8482. Each logs a read-only warning at startup
and no migration lines.
4. Compare them:
```bash
go run ./cmd/planguard storediff --old http://127.0.0.1:8481 --new http://127.0.0.1:8482 \
--cases db/planguard/storediff/smoke.yaml --walk 1,2,3,web --out storediff-out
```
`--walk` lists the page sizes to walk; `web` is each case's web page size. Small page sizes catch
pagination that skips or repeats rows. The tool exits non-zero when any page differs, and with
`--out` writes `storediff_report.md` there. Never use read-only mode on your live instance: it
ingests nothing.
## Rolling back the pool layout
If the new pools or plans misbehave, set this and restart the server: