From 2d2b90a61ecf8cd64aa84f76ab53cdce26b981fe Mon Sep 17 00:00:00 2001 From: MrAlders0n <55921894+MrAlders0n@users.noreply.github.com> Date: Sat, 24 Oct 2026 12:33:48 -0400 Subject: [PATCH] Document the background channel message backfill (#51) --- docs/configuration.md | 4 +++- docs/high-level-design.md | 2 +- docs/operations.md | 17 +++++++++++++++++ docs/troubleshooting.md | 6 +++--- docs/upgrading.md | 17 +++++++++++++++-- 5 files changed, 39 insertions(+), 7 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index dce6994..3f78ae4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -298,7 +298,9 @@ writes the tables; it trusts each one only while the writing server keeps it up `readmodels.fill` paces the one-off backfills: the `packet_sites` and `channel_message_iatas` fills after an upgrade or a rebuild, and the -[payload-type backfill](operations.md#after-upgrading-observation-payload-types). After each +[payload-type backfill](operations.md#after-upgrading-observation-payload-types). The +[channel message backfill](operations.md#channel-message-backfill) after each start uses `duty` +only, with no `min_pause`. After each slice or batch a fill sleeps `max(min_pause, elapsed × (1 − duty) / duty)`, where `elapsed` is how long that slice took. A slow disk makes each slice take longer, so it stretches the pauses on its own. Pacing changes only when the fills run, never what they write. diff --git a/docs/high-level-design.md b/docs/high-level-design.md index 268635b..7b70650 100644 --- a/docs/high-level-design.md +++ b/docs/high-level-design.md @@ -761,7 +761,7 @@ retention under 24h) fail startup rather than being ignored. IATAs are still auto-created when packets arrive from unrecognized codes. The config file is only needed to override the default display name/coordinates, or to assign an IATA to a super-region. -Channel hashes also auto-populate as messages arrive; adding a key in the config decrypts the stored hash-only rows for that channel on the next startup. +Channel hashes also auto-populate as messages arrive; adding a key in the config decrypts the stored hash-only rows for that channel after the next startup, in the background. ### Admin API and access control diff --git a/docs/operations.md b/docs/operations.md index cdbcb8c..76212af 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -57,6 +57,23 @@ Everything persistent is under `data/` next to `docker-compose.yml`: - `data/app/config.yaml` and `.env` are your configuration. Keep copies of both somewhere else; no backup below includes `.env`. +## Channel message backfill + +After every start, once the server listens, Beacon retries every stored channel message it +could not decrypt against the configured keys, so a key added to `config.yaml` surfaces its +channel's history. Keys a MeshMapper import adds queue their channels on the same background +job. It runs one pass at a time, in batches of 500 packets, paced by +[`readmodels.fill.duty`](#pacing-the-backfills) alone (no `min_pause`), and stops after 30 +minutes, pauses included; whatever it did not reach is retried after the next start. A packet it +has started is finished (within 30 s), so a stop does not leave a message half stored. The log shows `channel message backfill started` +with `packets`, then `channel message backfill done: N previously-undecrypted channel +message(s)` with `decrypted` and `duration` (or a `WARN` that it stopped after 30 minutes, or +`channel message backfill stopped by shutdown`). +Messages it stores are not broadcast over the WebSocket, and their hours are re-rolled for the +historical stats. A packet heard live before the backfill reaches it is decrypted by ingest +instead: its message is stored once either way, its hours are re-rolled too, and, as a live +message, it is broadcast over the WebSocket with its original `sentAt`. + ## Retention Everything below is a duration in `config.yaml`. Shorten them if disk is tight, lengthen them diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 43b773b..83081f4 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -89,9 +89,9 @@ advertised. This fills in on its own as the mesh is observed. ## Channel messages show a hash but no text The channel's key is not configured, or it was added without a restart. Add it under -`channel_keys:` in `config.yaml` and run `docker compose restart app`. On start, Beacon decrypts -the stored messages for the new key and logs -`config: backfilled N previously-undecrypted channel message(s)`. +`channel_keys:` in `config.yaml` and run `docker compose restart app`. Once it listens, Beacon +decrypts the stored messages for the new key in the background and logs +`channel message backfill done: N previously-undecrypted channel message(s)`. ## MeshMapper imports are unconfigured or stale diff --git a/docs/upgrading.md b/docs/upgrading.md index 009f2bd..05245c1 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -344,13 +344,26 @@ PostgreSQL 16 client tools. Nothing to change for a PostgreSQL 16 database. A se with `POSTGRES_CLIENT_MAJOR=17` or `18` now works too; see [Backup export](backup-export.md). +#### Channel message backfill after startup + +The retry of stored, undecrypted channel messages against the configured keys no longer +delays startup: it runs in the background once the server listens. On a slow disk that was +most of the boot time (reading every undecrypted packet), so MQTT ingest and the API start +sooner. Its log lines changed: `config: backfilled N previously-undecrypted channel +message(s)` is now `channel message backfill started` (with `packets`) and `channel message +backfill done: N previously-undecrypted channel message(s)`, logged on every run. Keys that a +MeshMapper import adds queue on the same job. A message a live hearing decrypts first, before +the backfill reaches its packet, is stored once and its earlier hours are re-rolled, as the +backfill does. + ## Changes that need a restart Beacon reads `config.yaml` once at startup. After editing it, run `docker compose restart app`. On the next start: -- A newly added channel key decrypts the matching stored messages. Look for - `config: backfilled N previously-undecrypted channel message(s)` in the log. +- A newly added channel key decrypts the matching stored messages, in the background once + the server listens. Look for + `channel message backfill done: N previously-undecrypted channel message(s)` in the log. - Border file and MeshMapper import changes take effect. - Changed admin or MeshMapper keys in the YAML file are picked up.