Document the packet sites read model (#35)

* Document the packet sites read model

* List the region page differences; scope the sweep alerts to the writer

app_config/config.yaml.example

@@ -86,6 +86,10 @@ ratelimit:
 #readmodels:
 #  cmi:
 #    enabled: true
+#  packet_sites:
+#    # Serve region packet pages (iatas, region) from packet_sites once this process has repaired and verified
+#    # it. false serves them from the original statement; the table is still maintained and swept.
+#    enabled: true
 
 # MeshMapper integrations (optional, require a regional or grouped-region API key).
 # Your local MeshMapper regional or grouped-region admin can generate the key.

docs/api-contract.md

@@ -124,7 +124,8 @@ on `/packets` and `v1:<epoch µs>:<id>` on `/nodes` and `/observers`.
 | Endpoint | `pageCursor` | `nextPageCursor` |
 |---|---|---|
 | `/packets` without `iata`, `iatas`, `region`, `regionId` or `hashPrefix` | Wins over `cursor` when both are sent. | When `hasMore` |
-| `/packets` with an IATA or region filter, or `hashPrefix` | Checked, then ignored; `cursor` pages. | Not returned |
+| `/packets` with `iata`, `iatas`, `region` or `regionId` (a region page), without `hashPrefix` | While the server trusts its site read model, wins over `cursor` when both are sent. A region page served without `nextPageCursor` pages by `cursor`: there `cursor` wins, and a `pageCursor` sent alone pages by its millisecond. | When `hasMore`, while the server trusts its site read model |
+| `/packets` with `hashPrefix` | Checked, then ignored; `cursor` pages. | Not returned |
 | `/nodes`, `/observers` | Wins over `cursor` when both are sent. | When `hasMore` |
 | `/channels` | Can't be combined with a positive `cursor` (`400`). | When `hasMore` |
 
@@ -135,6 +136,10 @@ on `/packets` and `v1:<epoch µs>:<id>` on `/nodes` and `/observers`.
   `pageCursor` set to `nextPageCursor` when the page has both, `cursor` alone otherwise. A server
   older than 2.1.0 ignores `pageCursor` and still moves forward by `cursor`. Treat
   `nextPageCursor` as optional on every list. `/channels` takes only one of the two.
+- Region pages of `/packets` return `nextPageCursor` only while the server trusts its
+  `packet_sites` read model, which it checks again after every start and drops on any doubt.
+  Otherwise they page by `cursor` exactly as before, so one walk can mix pages with and without
+  it. Keep sending both whenever a page has both.
 - A malformed `pageCursor` is a `400` with message `invalid pageCursor` on those three
   endpoints, on every path, including the `/packets` paths that ignore it. Start again from
   the first page without it. An empty `pageCursor` counts as absent.
@@ -166,6 +171,14 @@ Comma-separated list params (`iatas`, `scopes`, `payloadTypes`, `routeTypes`) ta
 
 `/packets` params: `payloadType` or `payloadTypes` (comma-separated integers), `payloadTypeName` (`advert`, `group_text`, `trace`, ...; used when no number is given), `routeType` or `routeTypes`, the location filters, `scope` or `scopes` (comma-separated), `since`, `until`, `hashPrefix` (4 to 16 hex characters of the packet hash), `cursor`, `pageCursor`, `limit`. Filters OR within themselves and AND across each other. The cursor is epoch ms: global `lastHeardAt` order without an IATA filter, and the time heard at the requested sites when one is set, so a filtered list stays bounded to those IATAs. With `hashPrefix` the order and cursor use global `lastHeardAt` even when an IATA filter is set.
 
+With an IATA or region filter and no `hashPrefix` (a region page), the list is ordered by the time the packet was last heard at the requested sites, newest first. While the server trusts its site read model (see [Operations](operations.md#read-models-packets-by-site)), `pageCursor` applies and those pages are the same as before with three differences:
+
+- Packets heard at the same site time are ordered by `packetHash`, descending.
+- A `pageCursor` page returns the rows in the cursor's millisecond that a millisecond `cursor` skips, as on `/packets` without a region (see [Exact page cursors](#exact-page-cursors)).
+- Where many observers heard a site's packets, the previous statement could end a page short, end a walk early or move `nextCursor` past rows it never returned; the new page returns those rows.
+
+Otherwise region pages page by `cursor` as before.
+
 Without an IATA or region filter or `hashPrefix`, the list is ordered by `lastHeardAt`, newest first, and packets heard in the same instant by `packetHash`, descending. `pageCursor` applies there (see [Exact page cursors](#exact-page-cursors)). Repeated values in the type and scope lists count once, and type values outside the header's range (payload type 0 to 15, route type 0 to 3) are dropped; a type list left with no valid value returns an empty page, not a `400`. When several observations of a packet share the newest time, `latestObserver` comes from the one with the highest observation ID, so the choice is stable.
 
 `/packets/backfill` takes single `payloadType`/`payloadTypeName`, `routeType`, the location filters, `scope` and `limit`.

docs/configuration.md

@@ -71,7 +71,7 @@ which ones to read.
 | `ratelimit` | Per-client REST limits for `/api/v1/*`. On by default at 600 requests a minute. | You get 429s you did not expect. |
 | `database` | `api_plan_layout`: how API statements are spread over database connection pools. | You check `max_connections`, or need the rollback layout. See [Database pools and cost guards](#database-pools-and-cost-guards). |
 | `planguard` | `daily_check`: a daily read-only check that the database still plans Beacon's queries the way they were designed. Off by default. | You run a large deployment. See [Daily plan check](#daily-plan-check). |
-| `readmodels` | `cmi.enabled`: whether region and scope channel message pages may read the `channel_message_iatas` read model (default `true`). | You suspect the read model serves wrong pages. See [Read models](#read-models). |
+| `readmodels` | `cmi.enabled`: whether region and scope channel message pages may read the `channel_message_iatas` read model. `packet_sites.enabled`: whether region packet pages may read the `packet_sites` read model. Both default to `true`. | You suspect a read model serves wrong pages. See [Read models](#read-models). |
 | `guards` | Which request families the cost guards may refuse. All of them only observe by default. | After a week of observing. See [Database pools and cost guards](#database-pools-and-cost-guards). |
 | `meshmapper` | Import transport scopes, IATA borders, region groups and public channels from MeshMapper instead of listing them by hand, and add repeater site details to node detail. Requires a [MeshMapper API key](#meshmapper-api-key). | You are in a region MeshMapper covers. |
 | `iatas` | Display names, coordinates and optional border files for airport codes. IATAs are created automatically when traffic arrives; this block only decorates them. A `borderFile` path is relative to `config.yaml`; in Docker the compose file mounts only `config.yaml`, so add a mount for the folder too (`- ./data/app/borders:/app/borders:ro` under the `app` service). | You want names on the map or a border drawn. |
@@ -196,8 +196,8 @@ messages, MQTT ingest, the presence writer, the rollup catch-up, every backgroun
 MeshMapper imports. Every database session it opens is read-only
 (`default_transaction_read_only`), so anything that would still write, such as
 `POST /api/v1/admin/accounts`, fails with SQLSTATE `25006` instead of changing data. The server
-logs a warning at startup saying so. The one thing it still runs is a read-only check of the
-channel message read model every 5 minutes; see [Read models](#read-models).
+logs a warning at startup saying so. The one thing it still runs is a read-only check of each
+read model (channel messages and packets by site) every 5 minutes; see [Read models](#read-models).
 
 The REST API and `/ws` still serve what is in the database, but nothing new arrives. Because no
 migrations run, the database must already be at the schema the build expects. Its use is
@@ -233,6 +233,8 @@ It cannot be combined with `server.read_only: true`; the server refuses to start
 readmodels:
   cmi:
     enabled: true
+  packet_sites:
+    enabled: true
 ```
 
 `channel_message_iatas` holds one row for each channel message and each site that heard it.
@@ -245,8 +247,18 @@ pages. Every channel message page is then served by the statements of the previo
 The server still maintains and repairs the table, so setting it back to `true` needs only a
 restart. Omitting the key means `true`. Like every key in this file, a change needs a restart.
 
-On a [read-only server](#read-only-mode) this key works the same way. Such a server never
-writes the table; it trusts it only while the writing server keeps it up to date.
+`packet_sites` holds one row for each packet and each site that heard it. `/packets` pages
+filtered by `iatas`, `iata`, `region` or `regionId` read it instead of every observation, but
+only while the server process trusts the table; see
+[Operations](operations.md#read-models-packets-by-site).
+
+`readmodels.packet_sites.enabled: false` is its kill switch. Every region packet page is then
+served by the statement of the previous release, without `nextPageCursor`. The server still
+maintains and sweeps the table, so setting it back to `true` needs only a restart. Omitting the
+key means `true`.
+
+On a [read-only server](#read-only-mode) both keys work the same way. Such a server never
+writes the tables; it trusts each one only while the writing server keeps it up to date.
 
 ## MeshMapper API key
 

docs/operations.md

@@ -221,6 +221,25 @@ WHERE c.relname IN ('channel_message_iatas_pkey', 'idx_cmi_channel_site', 'idx_c
 
 It must return 7 rows, all `t`.
 
+Migrations `042` to `045` add the region packet read model; see
+[Read models: packets by site](#read-models-packets-by-site).
+
+- `042` creates the table `packet_sites`. Its foreign key to `packets` takes a brief lock on
+  `packets`, so run the pre-deploy check above first.
+- `043` to `045` build `idx_packet_sites_site`, `idx_packet_sites_site_type` and
+  `idx_packet_sites_site_scope` with `CREATE INDEX CONCURRENTLY` on the empty table, which
+  takes seconds. No pre-build is needed.
+
+After the deploy, check the table's four indexes:
+
+```sql
+SELECT c.relname, i.indisvalid AND i.indisready AS ok
+FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
+WHERE i.indrelid = 'packet_sites'::regclass ORDER BY 1;
+```
+
+It must return 4 rows, all `ok = t`.
+
 ### Pre-building the route hop index
 
 Migration `006_known_routes_hop_iata.sql` builds `idx_known_routes_hop_iata`, which `/routes`
@@ -640,6 +659,184 @@ Then restart the server. It backfills and verifies again as on the first install
 A later release will drop `idx_channel_messages_channel` and `idx_channel_messages_sent_brin`.
 Run this release for at least a week before upgrading to that one.
 
+## Read models: packets by site
+
+`packet_sites` holds one row for each packet and each site (IATA) that heard it, at that site's
+latest hearing. Each row also copies the packet's payload type, route type, scope and first
+hearing. The observation insert writes the row in the same statement, so a stored observation
+always has its row. A background reconcile repairs the newest rows every 5 minutes, an hourly
+audit refills one random older slice, and a front sweep every 30 seconds keeps the newest
+index entries dense. On a large deployment (about 7.4 million rows) it takes about 1.6 GB:
+heap 484 MB, primary key 285 MB, `idx_packet_sites_site` 419 MB, `idx_packet_sites_site_type`
+353 MB, `idx_packet_sites_site_scope` 53 MB.
+
+While the server process trusts the table, `/packets` pages with `iatas`, `iata`, `region` or
+`regionId` (region pages) read it:
+
+- one site: one range of the site's index;
+- several sites: one range per site;
+- payload and route types, up to 8 (payload, route) pairs: one exact range per site and pair;
+- a `since`/`until` window, scopes, or more than 8 type pairs: bounded reads of each site range
+  that continue, within the same request, until the page is full. If 8 reads do not fill the
+  page, the previous release's statement answers that request.
+
+Unknown site codes are dropped by a cheap check first. Requests with `hashPrefix` never read
+the table. A region page served from the table now returns `nextPageCursor`; see
+[Exact page cursors](api-contract.md#exact-page-cursors). While the process does not trust the
+table, region pages are served by the previous release's statement exactly as before.
+
+Each page is the previous statement's page, except where that statement loses rows: when a
+site's packets are heard by many observers, it can end a page short, end a walk early (an
+empty page with `hasMore: false`) or jump `nextCursor` past rows. The new page is full and
+skips nothing. Packets tied on their site time come out by packet hash, descending.
+
+On a production-size copy, the heaviest region pages went from seconds to under a second: all
+561 sites for the last hour from 19.5 s to 0.3 s, and the 13 busiest sites for the last 75 minutes at
+limit 200 from 4.5 s to 31 ms. A filter of rare types on a very sparse site can read a few
+buffers more than before. An `until` well in the past often runs out of the read budget, so
+the previous statement answers many of those pages, at the previous cost.
+
+### Ingest cost
+
+The site row makes each observation insert about 15-21% slower (mean 104 µs to 119-125 µs)
+and adds about 37-53% WAL per stored observation (1,425-1,470 B to 2,008-2,188 B), measured
+over 100,000 inserts on a scratch database. Re-sends of an observation still write nothing.
+
+After the deploy, compare `InsertObservation` in `pg_stat_statements` (`mean_exec_time`, and
+`wal_bytes / calls`) for the hour before and the hour after, and watch
+`beacon_ingest_queue_depth`. `pg_stat_statements` is cumulative, so take this snapshot an hour
+before the deploy, at the deploy and an hour after, and divide the differences:
+
+```sql
+SELECT now(), sum(calls) AS calls, sum(total_exec_time) AS exec_ms, sum(wal_bytes) AS wal_bytes
+FROM pg_stat_statements WHERE query LIKE '-- name: InsertObservation :one%';
+```
+
+A growing ingest queue, or WAL beyond what you planned for, means rolling back.
+
+### Installing it
+
+The release adds migrations `042` to `045`; see [Before deploying a release](#before-deploying-a-release).
+On the first start the server fills the table in the background:
+
+1. It waits until every transaction that started before boot and holds a transaction id has
+   finished (`packet_sites boot repair waits for transactions from before boot` while it
+   waits), then runs one reconcile from the saved watermark.
+2. It backfills the table newest observation first, in slices of 500,000 observation ids with
+   a 0.1 s pause between them. The log shows `packet_sites backfill progress` lines, then
+   `packet_sites backfill complete`. On a large deployment (about 766 million ids, 29.4 million
+   observations) that is about 1,530 slices: about 3 minutes of pauses plus the fill work, so
+   expect roughly 5 to 20 minutes and a few GB of WAL.
+3. It runs a one-time verification. It first runs `VACUUM (ANALYZE) packet_sites` (a few
+   minutes on a large deployment), then an exact check of two slices of 500,000 observation
+   ids, the newest and one random older one, and a page-by-page comparison of region pages of
+   the busiest recent sites with pages read straight from the observations. On success it logs
+   `packet_sites first-time verification passed` and records the time, so later starts skip it.
+4. It logs `packet_sites trusted: region packet pages read packet_sites`.
+
+Until step 4, every region page is served by the previous release's statement, and users see
+no difference. If the verification fails, the server logs
+`packet_sites verification failed; region pages stay on the original statement` and tries again
+every 5 minutes.
+
+Check progress, read-only:
+
+```sql
+SELECT value, updated_at FROM background_state WHERE key = 'packet_sites';
+SELECT pg_size_pretty(pg_total_relation_size('packet_sites'));
+```
+
+`complete` turns `true`, then `first_verified_at` appears. After that, `updated_at` advances
+every 5 minutes.
+
+### When pages use it
+
+Trust is held in memory by each server process; nothing in the database marks the table as
+exact. A process earns it at every start, once the boot reconcile has run, the backfill is
+complete and the one-time verification has passed. It drops trust:
+
+- at once when a reconcile or the hourly audit repairs any row: something bypassed the
+  writer. Two clean reconciles in a row restore it. A repair logs
+  `packet_sites reconcile repaired N row(s): a writer bypassed the read model` or
+  `packet_sites audit repaired N row(s) below the watermark`;
+- at once when an observation insert with its site row fails. Ingest never loses the
+  observation: it logs `observation site row failed; storing the observation without it` and
+  stores it without its row. Trust drops with
+  `read model untrusted until the next reconcile` and the reason, and returns after the next
+  reconcile that started after the failure, so region pages use the previous statement for at
+  most about one interval;
+- after 3 failed reconciles in a row, until one succeeds;
+- when the saved state is no longer complete and verified, for example after a rebuild.
+
+On a single server running this release, repairs should not happen. Running two server versions
+that both write to one database at the same time is not supported; if it happens, the rows the
+older one leaves out are repaired within one reconcile interval, and trust returns two clean
+reconciles later.
+
+A [read-only server](configuration.md#read-only-mode) cannot repair anything. Every 5 minutes it
+checks, read-only, that the writing server's saved state is complete and verified, that the
+writer saved it within the last 10 minutes (two reconcile intervals), and that the newest
+20,000 observations pass the exact check. While all three hold it logs
+`packet_sites trusted on this read-only server: region packet pages read packet_sites` and
+serves region pages from the table. Its trust ends on its own 10 minutes after the writer's last
+save, unless a later check finds a newer one. Otherwise it logs one of:
+
+- `packet_sites not trusted on this read-only server: backfill or first verification not done`
+- `packet_sites not trusted on this read-only server: the writer has not saved its state recently`
+- `packet_sites not trusted on this read-only server: the newest observations fail the exact check`
+
+### Turning it off
+
+Set `readmodels.packet_sites.enabled: false` in `config.yaml` and restart; see
+[Configuration](configuration.md#read-models). Every region page is then served by the previous
+statement. The table is still maintained and swept, so turning it back on needs only a
+restart. A later release removes the previous statement once the read model has run in
+production.
+
+### Metrics and alerts
+
+On the [admin listener](#metrics):
+
+- `beacon_readmodel_trusted{model="packet_sites"}` is 1 while this process serves region pages
+  from the table.
+- `beacon_readmodel_repaired_rows_total{model="packet_sites"}` should stay flat.
+- `beacon_readmodel_watermark_age_seconds{model="packet_sites"}` is the time since the last
+  finished reconcile, or on a read-only server since the writer last saved its state.
+- `beacon_packet_sites_sweep_age_seconds` is the time since the last successful front sweep,
+  and `beacon_packet_sites_sweep_plan_ok` is 1 while the sweep plans as index scans on the three
+  time-ordered indexes. Without that plan the sweep marks no dead entries and the indexes bloat;
+  the server logs `packet_sites sweep plan sets no dead-entry hints; the indexes will bloat`.
+  Both are written only by the server that runs background jobs.
+- `beacon_packet_sites_front_live{index}` is the entries the last sweep visited, and
+  `beacon_packet_sites_bytes` the table's size with its indexes, hourly.
+- `beacon_index_bloat_ratio` covers the four `packet_sites` indexes. Their baselines are the
+  sizes of a fresh build, applied once the table holds 100,000 rows, so the ratio reads against
+  a compact index rather than the size the server first saw.
+
+Suggested rules are in [Example alert rules](#example-alert-rules): repaired rows in the last
+10 minutes, trusted at 0 for 40 minutes, a watermark older than 15 minutes, a sweep older than
+300 s or a sweep plan at 0, a bloat ratio of the three time-ordered indexes over 2.5 (warn) or
+3, and the table over 2.5 GiB (warn; scale it for your deployment).
+
+### Rolling back
+
+Deploy the previous image. It ignores `packet_sites`, still has the previous release's
+indexes, and the table's rows are deleted with their packets, so it does not grow stale rows.
+Leave the table and its indexes in place.
+
+To roll forward again later, deploy the new image. Before it serves any region page from the
+table, the boot reconcile adds the rows written meanwhile, starting from the saved watermark;
+the previous statement serves until then. Nothing to do by hand.
+
+Rebuild the table from scratch only if the verification keeps failing:
+
+```sql
+DELETE FROM background_state WHERE key = 'packet_sites';
+TRUNCATE packet_sites;
+```
+
+Then restart the server. It backfills and verifies again as on the first install.
+
 ## Postgres settings
 
 The stacks in this repo run stock `postgres:16-alpine` with no tuning. The settings below suit
@@ -938,9 +1135,13 @@ scrape_configs:
 | `beacon_pg_xid_age` | gauge | | Age of the oldest transaction ID in the database |
 | `beacon_planguard_failures` | gauge | `entry`, `mode` | 1 for each corpus entry and plan mode that failed the last [plan check](#turning-on-the-daily-check) |
 | `beacon_planguard_last_run_timestamp_seconds` | gauge | | Unix time the last successful plan check finished; 0 until the first one |
-| `beacon_readmodel_trusted` | gauge | `model` | 1 while this process serves pages from the [read model](#read-models-channel-messages-by-site), 0 while it serves the previous statements |
+| `beacon_readmodel_trusted` | gauge | `model` | 1 while this process serves pages from the read model ([`cmi`](#read-models-channel-messages-by-site) or [`packet_sites`](#read-models-packets-by-site)), 0 while it serves the previous statements |
 | `beacon_readmodel_repaired_rows_total` | counter | `model` | Rows the read-model reconcile repaired; above zero, a writer bypassed the model |
 | `beacon_readmodel_watermark_age_seconds` | gauge | `model` | Seconds since the last finished reconcile of the read model (on a read-only server, since the writer last saved its state) |
+| `beacon_packet_sites_sweep_age_seconds` | gauge | | Seconds since the last successful `packet_sites` front sweep; absent before the first |
+| `beacon_packet_sites_sweep_plan_ok` | gauge | | 1 while the front sweep plans as index scans on its three indexes, 0 otherwise; checked at the first sweep, then hourly |
+| `beacon_packet_sites_front_live` | gauge | `index` | Index entries written in the last two minutes that the last front sweep visited |
+| `beacon_packet_sites_bytes` | gauge | | Total size of `packet_sites`, heap and indexes, hourly |
 
 `guard` is `deadline`, `cap`, `admission` or `budget`. The Postgres counters are read once a
 minute; the index gauges first appear an hour after startup. The two `beacon_planguard_*`
@@ -986,6 +1187,27 @@ groups:
         for: 20m
       - alert: BeaconReadModelStale            # no finished reconcile (or writer save) for 15 min
         expr: beacon_readmodel_watermark_age_seconds{model="cmi"} > 900
+      # The packet_sites trust rules don't apply with readmodels.packet_sites.enabled: false.
+      # Install the two sweep rules only for the server that runs background jobs, never for a
+      # read-only server: it never sweeps, so its sweep_plan_ok stays 0 and the plan rule would fire.
+      - alert: BeaconPacketSitesRepairing      # a reconcile or audit repaired rows: a writer bypassed the model
+        expr: increase(beacon_readmodel_repaired_rows_total{model="packet_sites"}[10m]) > 0
+      - alert: BeaconPacketSitesUntrusted      # region packet pages on the previous statement
+        expr: beacon_readmodel_trusted{model="packet_sites"} == 0
+        for: 40m
+      - alert: BeaconPacketSitesStale          # no finished reconcile (or writer save) for 15 min
+        expr: beacon_readmodel_watermark_age_seconds{model="packet_sites"} > 900
+      - alert: BeaconPacketSitesSweepStale
+        expr: beacon_packet_sites_sweep_age_seconds > 300
+      - alert: BeaconPacketSitesSweepPlan      # the sweep sets no dead-entry hints; the indexes will bloat
+        expr: beacon_packet_sites_sweep_plan_ok == 0
+        for: 5m
+      - alert: BeaconPacketSitesBloatWarn
+        expr: beacon_index_bloat_ratio{index=~"idx_packet_sites_site(_type|_scope)?"} > 2.5
+      - alert: BeaconPacketSitesBloat
+        expr: beacon_index_bloat_ratio{index=~"idx_packet_sites_site(_type|_scope)?"} > 3
+      - alert: BeaconPacketSitesSize           # warn; scale 2.5 GiB for your deployment
+        expr: beacon_packet_sites_bytes > 2.5 * 1024^3
 ```
 
 Replace `<baseline bytes/s>` with your normal WAL rate before loading the rules. The route
@@ -1051,6 +1273,11 @@ server sees the index and again whenever it is rebuilt, so the ratio is growth s
 rebuild, not absolute bloat: 1 means as compact as then, 4 means four times the bytes per
 row. An index that has never been analyzed has no ratio.
 
+The four `packet_sites` indexes are never built fresh on a full table, so their first-seen size
+would make a poor baseline. They start from the bytes per row of a fresh build instead, once
+the table holds 100,000 rows; below that they have no ratio. A rebuild of one of them
+re-baselines it from its measured size, as for any index.
+
 The first run, and then one run a week, also logs `indexes with no scans since the last
 statistics reset` with each unused non-unique index and its size. An index that stays on that
 list for weeks is a candidate to drop; check that no rare job reads it first.

docs/upgrading.md

@@ -193,6 +193,34 @@ what to watch, the metrics and alerts, and how read-only servers decide.
 - Run this release for at least a week before upgrading to the release that drops
   `idx_channel_messages_channel` and `idx_channel_messages_sent_brin`.
 
+## Packets by site
+
+The release that adds migrations `042` to `045` adds `packet_sites`, a read model that `/packets`
+pages filtered by `iatas`, `iata`, `region` or `regionId` read once the server trusts it. It
+keeps the minimum web and app versions. Those pages gain `nextPageCursor`, which current web and
+app versions already send back as `pageCursor`; see
+[Exact page cursors](api-contract.md#exact-page-cursors). Run the
+[pre-deploy check](operations.md#before-deploying-a-release) first: migration `042` takes a brief
+lock on `packets`. No pre-build is needed: the new indexes build on an empty table in seconds.
+
+After the deploy, the server fills the table in the background and verifies it once; on a large
+deployment expect roughly 5 to 20 minutes and a few GB of WAL, then a few minutes of
+`VACUUM (ANALYZE)`. Until the log shows
+`packet_sites trusted: region packet pages read packet_sites`, every region page is served by
+the previous release's statement, so users see no change. See
+[Read models: packets by site](operations.md#read-models-packets-by-site) for what to watch, the
+metrics and alerts, and how read-only servers decide.
+
+- Each stored observation now costs about 15-21% more insert time and 37-53% more WAL. After
+  the deploy, compare `InsertObservation` in `pg_stat_statements` for the hour before and after
+  and watch `beacon_ingest_queue_depth`; see
+  [Ingest cost](operations.md#ingest-cost). A growing queue or WAL beyond your budget means
+  rolling back.
+- To turn it off without rolling back, set `readmodels.packet_sites.enabled: false` and
+  restart; see [Configuration](configuration.md#read-models).
+- To roll back, deploy the previous image. Rolling forward again later repairs the table at
+  startup before any region page reads it; nothing to do by hand.
+
 ## Changes that need a restart
 
 Beacon reads `config.yaml` once at startup. After editing it, run