Document the background channel message backfill (#51)

docs/configuration.md

@@ -298,7 +298,9 @@ writes the tables; it trusts each one only while the writing server keeps it up
 
 `readmodels.fill` paces the one-off backfills: the `packet_sites` and `channel_message_iatas`
 fills after an upgrade or a rebuild, and the
-[payload-type backfill](operations.md#after-upgrading-observation-payload-types). After each
+[payload-type backfill](operations.md#after-upgrading-observation-payload-types). The
+[channel message backfill](operations.md#channel-message-backfill) after each start uses `duty`
+only, with no `min_pause`. After each
 slice or batch a fill sleeps `max(min_pause, elapsed × (1 − duty) / duty)`, where `elapsed` is
 how long that slice took. A slow disk makes each slice take longer, so it stretches the pauses
 on its own. Pacing changes only when the fills run, never what they write.

docs/high-level-design.md

@@ -761,7 +761,7 @@ retention under 24h) fail startup rather than being ignored.
 
 IATAs are still auto-created when packets arrive from unrecognized codes. The config file is only needed to override the default display name/coordinates, or to assign an IATA to a super-region.
 
-Channel hashes also auto-populate as messages arrive; adding a key in the config decrypts the stored hash-only rows for that channel on the next startup.
+Channel hashes also auto-populate as messages arrive; adding a key in the config decrypts the stored hash-only rows for that channel after the next startup, in the background.
 
 ### Admin API and access control
 

docs/operations.md

@@ -57,6 +57,23 @@ Everything persistent is under `data/` next to `docker-compose.yml`:
 - `data/app/config.yaml` and `.env` are your configuration. Keep copies of both somewhere
   else; no backup below includes `.env`.
 
+## Channel message backfill
+
+After every start, once the server listens, Beacon retries every stored channel message it
+could not decrypt against the configured keys, so a key added to `config.yaml` surfaces its
+channel's history. Keys a MeshMapper import adds queue their channels on the same background
+job. It runs one pass at a time, in batches of 500 packets, paced by
+[`readmodels.fill.duty`](#pacing-the-backfills) alone (no `min_pause`), and stops after 30
+minutes, pauses included; whatever it did not reach is retried after the next start. A packet it
+has started is finished (within 30 s), so a stop does not leave a message half stored. The log shows `channel message backfill started`
+with `packets`, then `channel message backfill done: N previously-undecrypted channel
+message(s)` with `decrypted` and `duration` (or a `WARN` that it stopped after 30 minutes, or
+`channel message backfill stopped by shutdown`).
+Messages it stores are not broadcast over the WebSocket, and their hours are re-rolled for the
+historical stats. A packet heard live before the backfill reaches it is decrypted by ingest
+instead: its message is stored once either way, its hours are re-rolled too, and, as a live
+message, it is broadcast over the WebSocket with its original `sentAt`.
+
 ## Retention
 
 Everything below is a duration in `config.yaml`. Shorten them if disk is tight, lengthen them

docs/troubleshooting.md

@@ -89,9 +89,9 @@ advertised. This fills in on its own as the mesh is observed.
 ## Channel messages show a hash but no text
 
 The channel's key is not configured, or it was added without a restart. Add it under
-`channel_keys:` in `config.yaml` and run `docker compose restart app`. On start, Beacon decrypts
-the stored messages for the new key and logs
-`config: backfilled N previously-undecrypted channel message(s)`.
+`channel_keys:` in `config.yaml` and run `docker compose restart app`. Once it listens, Beacon
+decrypts the stored messages for the new key in the background and logs
+`channel message backfill done: N previously-undecrypted channel message(s)`.
 
 ## MeshMapper imports are unconfigured or stale
 

docs/upgrading.md

@@ -344,13 +344,26 @@ PostgreSQL 16 client tools. Nothing to change for a PostgreSQL 16 database. A se
 with `POSTGRES_CLIENT_MAJOR=17` or `18` now works too; see
 [Backup export](backup-export.md).
 
+#### Channel message backfill after startup
+
+The retry of stored, undecrypted channel messages against the configured keys no longer
+delays startup: it runs in the background once the server listens. On a slow disk that was
+most of the boot time (reading every undecrypted packet), so MQTT ingest and the API start
+sooner. Its log lines changed: `config: backfilled N previously-undecrypted channel
+message(s)` is now `channel message backfill started` (with `packets`) and `channel message
+backfill done: N previously-undecrypted channel message(s)`, logged on every run. Keys that a
+MeshMapper import adds queue on the same job. A message a live hearing decrypts first, before
+the backfill reaches its packet, is stored once and its earlier hours are re-rolled, as the
+backfill does.
+
 ## Changes that need a restart
 
 Beacon reads `config.yaml` once at startup. After editing it, run
 `docker compose restart app`. On the next start:
 
-- A newly added channel key decrypts the matching stored messages. Look for
-  `config: backfilled N previously-undecrypted channel message(s)` in the log.
+- A newly added channel key decrypts the matching stored messages, in the background once
+  the server listens. Look for
+  `channel message backfill done: N previously-undecrypted channel message(s)` in the log.
 - Border file and MeshMapper import changes take effect.
 - Changed admin or MeshMapper keys in the YAML file are picked up.