# Operations

## Live host Compose overlay

The long-running production host shares its historical
`meshcore-analytics_default` bridge with separately managed beacon and Discord
services. Use `docker-compose.live.yml` for every production `config`, `up`,
`ps`, `exec`, and controlled replacement command:

```bash
docker compose -f docker-compose.yml -f docker-compose.live.yml \
  --profile dev --profile tunnel config --quiet
```

The overlay treats that bridge as external and reserves
`172.18.30.10`–`172.18.30.30` for the analytics proxies and tunnel. This keeps
the backend trusted-proxy allowlist exact without attempting to delete or
renumber unrelated endpoints. It also publishes this stack's Prometheus on
`127.0.0.1:9092` because the host's independent isolated-network Prometheus
owns port 9090. Fresh, CI, and isolated restore environments must use only
`docker-compose.yml` and its dedicated `172.30.0.0/24` network.

## Browser operator access

The operator dashboard exchanges `OPERATOR_SITE_TOKEN` for a 30-minute,
in-memory browser session. The session cookie is always `HttpOnly`, `Secure`,
and `SameSite=Strict`; the token is not stored in browser storage. Login,
logout, expiry, and failed authentication are audited without logging the
token.

Use either the deployment's local HTTPS endpoint or an SSH tunnel that opens
the dashboard as `https://localhost`. Plain HTTP to a private IP is not a
supported operator transport and the server will not weaken the cookie to make
it work. Example tunnel:

```bash
ssh -L 8443:127.0.0.1:443 operator@mesh-host
```

Then open `https://localhost:8443/`, enter the operator token, and use the
dashboard normally. Logout requires the session CSRF token. Existing automation
may continue to send `Authorization: Bearer <OPERATOR_SITE_TOKEN>` for one
release; migrate it to a purpose-specific operator credential before that
compatibility path is removed.

## Automated checks

Pull requests and branch pushes run `.github/workflows/ci.yml`. The workflow:

- installs dependencies from lockfiles
- type-checks, tests, and builds the backend
- builds the frontend and runs Playwright desktop/mobile smoke tests
- compiles the observed-link Python worker and tests HopReach with Go 1.23
- enforces the UK-4,600 HopReach speed/memory release gate
- validates the Docker Compose model with required secrets represented by CI-only placeholders

Dependabot groups weekly dependency updates for the backend, frontend, and GitHub Actions.

## Clean bootstrap

Set every required value in `.env`, including the immutable
`HEALTHCHECK_SOURCE_REF` and Anubis signing key. Before starting Mosquitto, load
the environment and run `scripts/bootstrap-mosquitto.sh`. The helper creates a
backend-only read ACL and password file, refuses partial or symlinked state, and
does not replace an existing credential set.

## Legacy schema-16 compatibility cutover

An existing database that has `016_stale_mqtt_observer_cleanup.sql` but not
`016_private_prefixes.sql` deliberately fails closed. After a signed backup and
isolated restore preflight, run the one-time reviewed compatibility path:

```bash
docker compose run --rm \
  -e MIGRATION_016_PRIVATE_PREFIXES_APPROVAL=supersede-016-and-017-with-authoritative-privacy-and-026 \
  db-migrate
```

The runner requires the exact sibling migration and both replacement files,
records `superseded-existing` for the unsafe historical rewrites, and installs
the prefix schema/trigger replacement before a new backend starts.
Authoritative public predicates validate path framing and consult the current
private-prefix table for historical as well as new packets. This avoids
decompressing or rewriting Timescale history. It also does not rewrite every
historical packet merely to materialize a topic prefix; public query predicates
safely derive the prefix from `topic` when the legacy column is empty. Read back
the decision before continuing:

```sql
SELECT migration_name, disposition, replacement_name
FROM schema_migration_compatibility
WHERE migration_name = '016_private_prefixes.sql';
```

Never set this approval on a fresh database or for any other missing migration.

## Metrics

Prometheus scrapes the backend registry from `backend:9091/metrics` on the
internal Compose network. Port 9091 is deliberately not published by Compose or
proxied by Nginx; `/metrics` on a public website origin must return 404.

## Health endpoints

- `GET /healthz` is a process liveness check. It deliberately remains healthy during an MQTT outage so Docker does not restart an otherwise functioning API in a loop.
- `GET /readyz` checks database access and MQTT connectivity. It returns `503` with per-dependency state when the real-time platform is not ready.
- `GET /api/health` returns worker history and a top-level `status` (`healthy`, `degraded`, or `critical`) plus machine-readable `problems`.

Current problem codes cover stale public ingest, worker queue backlog, disk pressure, and frontend error spikes. External monitoring should alert on `readyz` failures and critical `/api/health` problems, while allowing a short deploy/reconnect grace period.

The `synthetic-monitor` service independently checks liveness, dependency readiness, the scoped stats API, and delivery of a WebSocket `initial_state`. Results and latency are retained for 14 days in `operational_check_results`. Three consecutive failures trigger a structured log alert and, when `ALERT_WEBHOOK_URL` is configured, a JSON webhook. A successful check after an alert emits a recovery notification.

## Load checks

Run bounded HTTP load locally with:

```bash
cd backend
npm run load:realtime -- --duration 30 --concurrency 25 --max-p95-ms 1500
```

Use `--mqtt-messages 5000` to exercise MQTT ingest backpressure with isolated, rejected test envelopes, and `--slow-ws-clients 10` to hold non-reading WebSocket clients during the run. MQTT mode uses `MQTT_BROKER_URL`, `MQTT_USERNAME`, and `MQTT_PASSWORD`; never point it at a broker outside the deployment under test.

The internal HopReach compatibility load path is covered by
`backend/src/api/hopreachCompatibility.test.ts`. Its UK-size test performs ten
pages over 4,600 nodes, 32 concurrent calibration requests, a cold pass, and
verified cache hits/coalescing without exposing the adapter publicly.

## Statistics aggregate rollout

`packet_hourly_stats` is maintained atomically with accepted packet batches.
Populate historical hours with a resumable, one-hour-per-transaction tool:

```bash
docker compose exec backend node dist/tools/backfillStatsRollups.js --hourly-days 8
docker compose exec backend node dist/tools/backfillStatsRollups.js --apply --hourly-days 8
```

The first command is a read-only inventory. The checkpoint and the rollup
slice commit together, so interruption resumes without double-counting. The
`STATS_AGGREGATE_READS_ENABLED` and `STATS_AGGREGATE_SHADOW_ENABLED` settings
accept comma-separated network names; `true` retains the all-network mode.
Start with `STATS_AGGREGATE_SHADOW_ENABLED=ukmesh` while reads remain disabled.
The bounded aggregate query and its legacy comparison run off the response path
even in this shadow-only mode. Set `STATS_AGGREGATE_READS_ENABLED=ukmesh` only
after backfill and a clean parity window. Once the
aggregate-writing backend is live, reconcile the partial cutover hour:

```bash
docker compose exec backend node dist/tools/backfillStatsRollups.js \
  --apply --catch-up-current-hour
```

The reconciliation closes the bounded gap between the completed historical
checkpoint and now one hour at a time, including the current partial hour. Each
slice takes a writer lock with a five-second lock timeout and fails closed; a
gap above 48 hours requires a newly reviewed historical backfill. Transactions
that arrive during a rebuild increment the reconstructed rows after its lock is
released.

A short validation window may set
`STATS_AGGREGATE_SHADOW_ENABLED=ukmesh`. The comparison pins one cutoff, scans
the recent source rows once, and runs off the response path with one in-flight
comparison per scope and a five-minute minimum interval. Dimension keys must
match exactly. Count differences may be at most five packets or 0.1%,
whichever is larger, to allow a transaction that commits between the two
snapshots. Treat any `[stats-aggregate-shadow] mismatch` or
`[stats-aggregate-shadow] failed` as a failed rollout and turn aggregate reads
back off. Shadow mode deliberately adds a raw validation scan and must not
remain enabled.

Chart queries also share one `asOf` cutoff. `STATS_QUERY_DURATION_BUDGET_MS`
defaults to 60 seconds, `STATS_CHART_QUERY_TIMEOUT_MS` defaults to 120 seconds,
and `STATS_CHART_MAX_RESULT_ROWS` defaults to 100,000. A query that exceeds a
budget fails the chart refresh and is logged with a `[stats-query-budget]`
record; the existing stale snapshot remains available. The result-row limit is
an output-size guard. It does not measure rows scanned inside PostgreSQL, so
keep reviewing query plans and database scan telemetry before enabling reads on
additional networks.

Production aggregate reads combine maintained full-hour rows with raw boundary
fragments, preserving the legacy exact rolling 24-hour/seven-day windows while
keeping raw work bounded to partial hours.

Canonical chart responses are also stored in `stats_chart_snapshots` after
completeness, scope, age, privacy-generation, and 2 MiB payload checks. Schema
version 2 snapshots survive a backend restart: a fresh snapshot is returned
directly, and a stale snapshot remains available for up to six hours while one
refresh runs in the background. Newer snapshots cannot be overwritten by an
older concurrent refresh. A public/private visibility change immediately
invalidates both memory and durable entries, and a generation publication fence
prevents an in-flight old computation from being cached. Observer-filtered chart
responses are never persisted.

For the first migration to durable snapshots, capture a current successful
canonical response from the still-running release and pipe it to the candidate
backend tool after migration 031 has been applied:

```bash
npm run stats:seed-chart-snapshot -- --network ukmesh < verified-ukmesh-charts.json
```

The tool reads JSON only from stdin and rejects incomplete, cross-scope,
future-dated, more-than-six-hour-old, oversized, or pre-visibility-change input.
It binds the response to the current generation and fails if that generation
changes during publication. Verify the stored `scope_key`, schema version,
visibility generation, `generated_at`, and payload byte count before restarting
the serving backend. Do not seed an observer-filtered response.

## HopReach RF coverage

The app serves only `meta.json`, `progress.json`, and numeric PNG tile paths
under `/rf-coverage/`. The private compatibility adapter is mounted at
`/hopreach` on the backend container and rejects forwarded/public traffic.
Legacy `/api/coverage` and `/api/coverage/planned` contracts return `410 Gone`
without querying `node_coverage` or creating work.

Use [the HopReach rollout and recovery runbook](rf-coverage-rollout.md) for
source publication, benchmark gates, deployment order, progressive live
verification, checkpoint restart tests, and rollback. Do not delete the
`rf_coverage_data` volume or DEM cache to recover a failed run.

## Topology API

`GET /api/topology?network=ukmesh&limit=300` returns recent viable repeater relationships, bounded to 500 links and a 30-day observation window. It reports connected components, recently active isolated repeaters, and articulation points labelled as likely bridges because the bounded graph is evidence rather than a complete routing model. Private-node names and positions pass through the same redaction policy used by the node APIs.

## Public API and exports

`GET /api/v1` is the stable discovery endpoint and links to `/api/v1/openapi.yaml`. Positioned nodes can be exported from `/api/v1/exports/nodes.csv` and `/api/v1/exports/nodes.geojson`; both formats are read-only, rate-limited, capped at 5,000 rows, network-scoped, and use the public-node redaction policy.

`GET /api/activity/timeline` is bounded to a 24-hour window and 250 active node IDs per bucket. `GET /api/rf-validation` compares stored terrain-model viability with recent observed evidence and distinguishes likely mismatches, weak evidence, and explicit operator overrides.

## Public status and maintenance

The public `/health` page polls the aggregated `/api/health` contract every minute. Set `MAINTENANCE_ACTIVE=1` and `MAINTENANCE_MESSAGE` during a planned window; the API and page show the notice without leaking hostnames, credentials, addresses, or private node identities.

## Operational runbooks

- [Immutable release and rollback](runbook-release-rollback.md)
- [Encrypted backup, isolated restore, and disaster recovery](runbook-backup-restore.md)
- [Bounded queue and dead-letter recovery](runbook-queue-recovery.md)
- [SRTM cache, alert delivery, observer review, and planned publication](runbook-srtm-alerts-observers.md)
- [Database compression and retention gates](db-lifecycle.md)

### Owner authorization baseline refresh

`/readyz` fails closed when the current owner accounts, grants, configured grants,
or ACL generations differ from `OWNER_AUTH_INVENTORY_BASELINE_PATH`. This means
an owner grant or `OWNER_MQTT_USERNAME_MAP` entry changed after the baseline was
recorded. Drift is expected after every approved grant change and is not itself
an incident, but it must be reviewed before the baseline is refreshed.

1. Identify the change from the audit trail and confirm it was approved:

   ```bash
   docker exec -e PGOPTIONS='-c default_transaction_read_only=on' \
     meshcore-infra-timescaledb-1 psql -U meshcore -d meshcore_owner_auth -X \
     -c "SELECT occurred_at, event_type, source, actor FROM owner_grant_audit \
         WHERE occurred_at > now() - interval '30 days' ORDER BY occurred_at DESC"
   ```

2. Generate and validate a fresh baseline with the running image (read-only
   against the database; it only writes the new baseline file):

   ```bash
   scripts/refresh-owner-baseline.sh
   ```

3. Activate it after reviewing the printed counts/generations:

   ```bash
   scripts/refresh-owner-baseline.sh --apply --recreate
   curl -fsS http://127.0.0.1:3000/readyz
   ```

The script refuses to overwrite an existing baseline, backs up `.env` before
editing, keeps the previous baseline for rollback, and validates the new file
against the live inventory before activation. To roll back, point
`OWNER_AUTH_INVENTORY_BASELINE_PATH` at the previous file and recreate the
backend.

Use the `/operations`, `/observer-registrations`, and operator audit pages for
normal mutations. They enforce authorization, CSRF, typed confirmation,
idempotency, capacity, and audit rules that direct Redis/SQL edits bypass.

## Alert first response

Start every alert response by recording the firing time, checking Prometheus
target state, and preserving the relevant ten-minute logs. Acknowledge only
after assigning an operator. The following headings are stable targets for the
Prometheus `runbook_url` annotations.

### BackendDown

Check `docker compose ps backend`, then backend logs and the container OOM/exit
state. If the process is stopped, restart only `backend`; if it repeatedly
fails, keep ingest stopped and roll back to the prior signed digest.

### BackendNotReady

Read `/readyz` and act on the named dependency. Check PostgreSQL, Redis, and
Mosquitto before restarting the API. Liveness remaining green is expected
during a dependency outage.

### SyntheticProbeFailed

Compare the failing blackbox target with direct localhost liveness, stats, and
WebSocket checks. If localhost succeeds, inspect Nginx, Anubis, Cloudflare and
DNS in that order; do not restart the database.

### DependencyExporterDown

Check the named exporter and its dependency independently. An exporter-only
failure loses visibility but does not prove PostgreSQL or Redis is down.
Restore telemetry, then verify the underlying dependency metric.

### MosquittoUnavailable

Check broker health, its WebSocket listener, password/ACL file permissions and
the reloader logs. Do not regenerate credentials over an existing set. If ACL
reconciliation caused the alert, keep the last verified ACL and roll back that
change.

### Denied PUBLISH on `/raw` (#269 / #213)

**Disposition:** keep these denials. They are expected log noise from publishers
using a topic outside the supported observer contract; the evidence does not
show a packet-ingest gap.

The B4 review brief reports about 233 Mosquitto `Denied PUBLISH` lines from
`ESP32_a806F8` for `meshcore/LBA/8CA23B78…/raw`, associated with a granted node,
plus a separate `meshmonitor-observer EXT` `/raw` family. The count and client
identities above come from that supplied log review; they were not re-counted
against the live broker during this code change.

The renderer in `backend/src/mqtt/aclManager.ts` grants owner writes to
`packets`, `status`, `neighbors`, and `neighbours`. `backend/src/mqtt/topic.ts`
accepts those suffixes and rejects `/raw`. Packet bytes are carried in the
`raw` JSON field on the `/packets` envelope and decoded in
`backend/src/mqtt/client.ts`; there is no `/raw` topic consumer. Granting the
extra suffix would authorize a broker topic that the backend does not ingest.

Reopen this ruling if a supported publisher is denied on `/packets`, or if the
ingest contract is deliberately extended with a `/raw` consumer. The topic
parser test keeps the current rejection explicit.

### MeshIngestSilent

Check backend MQTT readiness, broker connected-client events, public topic
rates, and whether this is a genuine quiet period. Compare independent
observers before changing credentials or subscriptions.

### QueueAgeHigh

Open `/operations`, identify the oldest queue and heartbeat, then follow the
queue recovery runbook. Fix the worker/dependency cause before requeueing a
single retained dead job.

### QueueCapacityHigh

Disable the affected optional link producer if growth continues. Confirm the worker is consuming and the byte/job counters are
consistent. Never make Redis eviction-based or raise the cap without measuring
host memory.

### DeadJobsPresent

Inspect each bounded reason and attempt count in `/operations`. Requeue only
transient failures after the cause is fixed; purge one permanently invalid job
with the exact confirmation string.

### AnalysisLeaseLost

Preserve both worker logs and the analysis run record. The publication fence
prevents a stale writer from committing. Let the current lease expire, resume
from its checkpoint, and investigate clock, database, or long-step latency.

### HealthWorkerHeartbeatStale

Check the `health-worker` container, database reachability, and its last
successful snapshot. Restart only that worker; public health may be stale but
the API should remain available.

### WorkerMetricsDown

Identify the `role` label, inspect that worker's health endpoint and container
state, and restart only the affected worker. If work is queued, treat a stale
heartbeat as the higher-priority symptom.

### ActiveQueueWorkerHeartbeatStale

Stop new optional work, confirm the queue retains its leased payloads, and
restart the affected link worker. Allow lease recovery to requeue;
do not manually duplicate the job.

### DatabaseUnavailable

Check TimescaleDB container state, disk, memory, and recent logs. Stop writers
if storage errors appear. Never recreate or delete the volume; escalate to the
signed backup/restore runbook.

If a scheduled Timescale policy logs `failed to start a background worker`,
compare `SHOW max_worker_processes` with
`SHOW timescaledb.max_background_workers`. The Compose default reserves 24
PostgreSQL worker slots for the image's current 16-worker Timescale setting,
its launcher/database schedulers, and logical-replication headroom. Correct the
configuration and restart only TimescaleDB; then require the policy's next
manual or scheduled run to succeed before closing the incident.

### DatabasePoolSaturated

Inspect waiting/active connection metrics, slow route latency and PostgreSQL
activity. Shed optional analysis/load, identify the bounded slow query, and
avoid increasing pool size until connection and memory headroom are measured.

### DatabaseVacuumBacklog

Inspect the `relname` table label, `n_dead_tup`, `n_live_tup`, and recent
autovacuum activity in PostgreSQL. A table with more than 50,000 dead tuples
is actionable even when its live-row estimate is small (as can happen for a
TimescaleDB chunk after deletes); run the reviewed bounded vacuum procedure and
confirm that both the alert and `/api/health` clear. Do not drop chunks or
delete the database volume as a response.

### DatabaseWalRateHigh

Identify current maintenance, backfill, compression or ingest activity. Pause
the optional writer and check replica/backup capacity. A short known migration
burst may be observed; sustained unexplained WAL is an incident.

### HostDiskSpaceLow

Stop optional RF/backfill work and identify the mount from the alert. Preserve
databases and signed backups. Prune only explicitly identified unused build
cache/images or let the bounded SRTM/log policies converge; never run a broad
volume prune.

### HostDiskExhaustionForecast

Treat a sustained forecast of disk exhaustion within 24 hours as an incident
before the low-space threshold fires. Identify the filesystem and its largest
consumers, compare the database hypertable and chunk growth gauges, pause
optional backfills, and preserve backup capacity. Do not remove database
chunks or volumes to make space; follow the signed backup and lifecycle gates.

### CoreTelemetryGrowthHigh

Compare the per-table byte and chunk gauges with the retention and compression
policies, then run the read-only `npm run db:lifecycle` inventory during a low
traffic window. Check for missed retention/compression jobs and forecast the
next 30 days of disk use. Changes to destructive retention require the current
signed backup/restore receipt and the per-table approval in `docs/db-lifecycle.md`.

### CoreTelemetryRetentionOverdue

Use read-only queries against `timescaledb_information.jobs` and
`timescaledb_information.chunks` to identify the missing or failing policy for
the labelled table. Verify backup and isolated restore evidence before running
the gated per-table lifecycle command. Keep expired forensic access isolated
and read-only; never expose restored expired rows through production APIs.

### CoreTelemetryCompressionOverdue

Inspect the labelled table's compression policy and Timescale job history, as
well as worker capacity and recent database/WAL pressure. Apply compression
only through the per-table, receipt-gated command in `docs/db-lifecycle.md`.

### PacketPathsCapacity

Compare the health-worker `meshcore_packet_paths_rows_30d`,
`meshcore_packet_paths_bytes_per_row`, and overdue-chunk gauges with the
reviewed 10,000–23,000 rows/day and roughly 1 kB/row baseline. If compression
is overdue, inspect the `packet_paths` compression job and Timescale worker
headroom; never add a retention job for this table. For genuine growth, forecast
the next 12 months against free disk before changing storage or ingest policy.

### Durable MQTT ingestion

The MQTT client uses the library's inbound message callback for backpressure;
QoS 1 PUBACK follows successful database persistence. A database failure is
returned as a handler error, leaving the broker delivery unacknowledged for
redelivery and keeping readiness failed until a durable write succeeds. Inspect
the `failure`, `packet_persist_failure`, `status_persist_failure`, or
`neighbors_persist_failure` ingest outcomes and resolve the database issue;
do not clear the broker session or discard its inflight delivery.

### BackupReceiptMissing

Check the receipt mount, signature/key permissions, and backup job result.
Create a new encrypted backup and complete an isolated restore drill. Do not
enable destructive retention while evidence is missing.

### BackupStale

Run the backup immediately, verify the signature/checksum, and perform the
isolated drill. Investigate target capacity or scheduling after a fresh
verified receipt restores the recovery guarantee.

### TelemetryIntegrityFailures

Inspect aggregate rejection reasons and a bounded redacted sample. Treat
private/cross-network leakage as an immediate rollback condition. Otherwise
identify the malformed observer/client version without weakening validation.

## Spring-clean additions (2026-08-06)

### MQTT ingest resilience

The MQTT client keeps a stable `clean=false` QoS-1 session and uses MQTT.js
message handling as the backpressure boundary. It sends PUBACK only after
packet, status, or neighbour persistence completes. Persistence failures are
propagated to MQTT.js and close the connection without PUBACK so the persistent
broker session redelivers the message. The process-local packet dedupe marker
is written only after packet-batch commit. The broker session is
`meshcore-analytics-ingest` (Mosquitto reports `c0`).

### Feed history contract

`GET /api/feed/messages?channel=<scope>&limit<=50` — up to 50 unique
historical messages per channel, 90-day bound, dedup by packet hash. Uses an
ordered keyset-paginated packet scan (500-row pages, in-memory hash dedup,
early stop at 50) so it stays fast under TimescaleDB load; observer-stat
aggregation is bounded on the history path only (live WS path keeps full
aggregation). UI merges history under live WS messages; `initial_state`
still delivers 200. See `docs/decryption.md` for the join.

### Canonical node identity

Evidence-based identity merging is live (`docs/node-identity.md`,
migrations 036/037). Owner dashboard shows one entry per canonical identity
with the merged member key list.

### Spring-clean UI removals

- Site `Health` page removed (StatusPage + `/health` route + legacy
  `/status → /health` redirect + nav entry/flag). The separate
  `meshcore-health-check` app (healthcheck.ukmesh.com) is unaffected.
- Owner dashboard collapsed to a single page; alert-settings tab and the
  unused "coming next" roadmap payload removed end-to-end (backend, types,
  CSS).
- Live map feed cards are message-first with compact metadata; stale
  repeaters (no recent advert/status AND no multibyte-hop evidence) are
  filtered from the map; MQTT repeaters render like normal repeaters.
- Spam transparency page verified working (live API) and kept.