ukmesh/docs/operations.md

Operations

Live host Compose overlay

The long-running production host shares its historical meshcore-analytics_default bridge with separately managed beacon and Discord services. Use docker-compose.live.yml for every production config, up, ps, exec, and controlled replacement command:

docker compose -f docker-compose.yml -f docker-compose.live.yml \
  --profile dev --profile tunnel config --quiet

The overlay treats that bridge as external and reserves 172.18.30.10–172.18.30.30 for the analytics proxies and tunnel. This keeps the backend trusted-proxy allowlist exact without attempting to delete or renumber unrelated endpoints. It also publishes this stack's Prometheus on 127.0.0.1:9092 because the host's independent isolated-network Prometheus owns port 9090. Fresh, CI, and isolated restore environments must use only docker-compose.yml and its dedicated 172.30.0.0/24 network.

Browser operator access

The operator dashboard exchanges OPERATOR_SITE_TOKEN for a 30-minute, in-memory browser session. The session cookie is always HttpOnly, Secure, and SameSite=Strict; the token is not stored in browser storage. Login, logout, expiry, and failed authentication are audited without logging the token.

Use either the deployment's local HTTPS endpoint or an SSH tunnel that opens the dashboard as https://localhost. Plain HTTP to a private IP is not a supported operator transport and the server will not weaken the cookie to make it work. Example tunnel:

ssh -L 8443:127.0.0.1:443 operator@mesh-host

Then open https://localhost:8443/, enter the operator token, and use the dashboard normally. Logout requires the session CSRF token. Existing automation may continue to send Authorization: Bearer <OPERATOR_SITE_TOKEN> for one release; migrate it to a purpose-specific operator credential before that compatibility path is removed.

Automated checks

Pull requests and branch pushes run .github/workflows/ci.yml. The workflow:

• installs dependencies from lockfiles
• type-checks, tests, and builds the backend
• builds the frontend and runs Playwright desktop/mobile smoke tests
• compiles the observed-link Python worker and tests HopReach with Go 1.23
• enforces the UK-4,600 HopReach speed/memory release gate
• validates the Docker Compose model with required secrets represented by CI-only placeholders

Dependabot groups weekly dependency updates for the backend, frontend, and GitHub Actions.

Clean bootstrap

Set every required value in .env, including the immutable HEALTHCHECK_SOURCE_REF and Anubis signing key. Before starting Mosquitto, load the environment and run scripts/bootstrap-mosquitto.sh. The helper creates a backend-only read ACL and password file, refuses partial or symlinked state, and does not replace an existing credential set.

Legacy schema-16 compatibility cutover

An existing database that has 016_stale_mqtt_observer_cleanup.sql but not 016_private_prefixes.sql deliberately fails closed. After a signed backup and isolated restore preflight, run the one-time reviewed compatibility path:

docker compose run --rm \
  -e MIGRATION_016_PRIVATE_PREFIXES_APPROVAL=supersede-016-and-017-with-authoritative-privacy-and-026 \
  db-migrate

The runner requires the exact sibling migration and both replacement files, records superseded-existing for the unsafe historical rewrites, and installs the prefix schema/trigger replacement before a new backend starts. Authoritative public predicates validate path framing and consult the current private-prefix table for historical as well as new packets. This avoids decompressing or rewriting Timescale history. It also does not rewrite every historical packet merely to materialize a topic prefix; public query predicates safely derive the prefix from topic when the legacy column is empty. Read back the decision before continuing:

SELECT migration_name, disposition, replacement_name
FROM schema_migration_compatibility
WHERE migration_name = '016_private_prefixes.sql';

Never set this approval on a fresh database or for any other missing migration.

Metrics

Prometheus scrapes the backend registry from backend:9091/metrics on the internal Compose network. Port 9091 is deliberately not published by Compose or proxied by Nginx; /metrics on a public website origin must return 404.

Health endpoints

• GET /healthz is a process liveness check. It deliberately remains healthy during an MQTT outage so Docker does not restart an otherwise functioning API in a loop.
• GET /readyz checks database access and MQTT connectivity. It returns 503 with per-dependency state when the real-time platform is not ready.
• GET /api/health returns worker history and a top-level status (healthy, degraded, or critical) plus machine-readable problems.

Current problem codes cover stale public ingest, worker queue backlog, disk pressure, and frontend error spikes. External monitoring should alert on readyz failures and critical /api/health problems, while allowing a short deploy/reconnect grace period.

The synthetic-monitor service independently checks liveness, dependency readiness, the scoped stats API, and delivery of a WebSocket initial_state. Results and latency are retained for 14 days in operational_check_results. Three consecutive failures trigger a structured log alert and, when ALERT_WEBHOOK_URL is configured, a JSON webhook. A successful check after an alert emits a recovery notification.

Load checks

Run bounded HTTP load locally with:

cd backend
npm run load:realtime -- --duration 30 --concurrency 25 --max-p95-ms 1500

Use --mqtt-messages 5000 to exercise MQTT ingest backpressure with isolated, rejected test envelopes, and --slow-ws-clients 10 to hold non-reading WebSocket clients during the run. MQTT mode uses MQTT_BROKER_URL, MQTT_USERNAME, and MQTT_PASSWORD; never point it at a broker outside the deployment under test.

The internal HopReach compatibility load path is covered by backend/src/api/hopreachCompatibility.test.ts. Its UK-size test performs ten pages over 4,600 nodes, 32 concurrent calibration requests, a cold pass, and verified cache hits/coalescing without exposing the adapter publicly.

Statistics aggregate rollout

packet_hourly_stats is maintained atomically with accepted packet batches. Populate historical hours with a resumable, one-hour-per-transaction tool:

docker compose exec backend node dist/tools/backfillStatsRollups.js --hourly-days 8
docker compose exec backend node dist/tools/backfillStatsRollups.js --apply --hourly-days 8

The first command is a read-only inventory. The checkpoint and the rollup slice commit together, so interruption resumes without double-counting. The STATS_AGGREGATE_READS_ENABLED and STATS_AGGREGATE_SHADOW_ENABLED settings accept comma-separated network names; true retains the all-network mode. Start with STATS_AGGREGATE_SHADOW_ENABLED=ukmesh while reads remain disabled. The bounded aggregate query and its legacy comparison run off the response path even in this shadow-only mode. Set STATS_AGGREGATE_READS_ENABLED=ukmesh only after backfill and a clean parity window. Once the aggregate-writing backend is live, reconcile the partial cutover hour:

docker compose exec backend node dist/tools/backfillStatsRollups.js \
  --apply --catch-up-current-hour

The reconciliation closes the bounded gap between the completed historical checkpoint and now one hour at a time, including the current partial hour. Each slice takes a writer lock with a five-second lock timeout and fails closed; a gap above 48 hours requires a newly reviewed historical backfill. Transactions that arrive during a rebuild increment the reconstructed rows after its lock is released.

A short validation window may set STATS_AGGREGATE_SHADOW_ENABLED=ukmesh. The comparison pins one cutoff, scans the recent source rows once, and runs off the response path with one in-flight comparison per scope and a five-minute minimum interval. Dimension keys must match exactly. Count differences may be at most five packets or 0.1%, whichever is larger, to allow a transaction that commits between the two snapshots. Treat any [stats-aggregate-shadow] mismatch or [stats-aggregate-shadow] failed as a failed rollout and turn aggregate reads back off. Shadow mode deliberately adds a raw validation scan and must not remain enabled.

Chart queries also share one asOf cutoff. STATS_QUERY_DURATION_BUDGET_MS defaults to 60 seconds, STATS_CHART_QUERY_TIMEOUT_MS defaults to 120 seconds, and STATS_CHART_MAX_RESULT_ROWS defaults to 100,000. A query that exceeds a budget fails the chart refresh and is logged with a [stats-query-budget] record; the existing stale snapshot remains available. The result-row limit is an output-size guard. It does not measure rows scanned inside PostgreSQL, so keep reviewing query plans and database scan telemetry before enabling reads on additional networks.

Production aggregate reads combine maintained full-hour rows with raw boundary fragments, preserving the legacy exact rolling 24-hour/seven-day windows while keeping raw work bounded to partial hours.

Canonical chart responses are also stored in stats_chart_snapshots after completeness, scope, age, privacy-generation, and 2 MiB payload checks. Schema version 2 snapshots survive a backend restart: a fresh snapshot is returned directly, and a stale snapshot remains available for up to six hours while one refresh runs in the background. Newer snapshots cannot be overwritten by an older concurrent refresh. A public/private visibility change immediately invalidates both memory and durable entries, and a generation publication fence prevents an in-flight old computation from being cached. Observer-filtered chart responses are never persisted.

For the first migration to durable snapshots, capture a current successful canonical response from the still-running release and pipe it to the candidate backend tool after migration 031 has been applied:

npm run stats:seed-chart-snapshot -- --network ukmesh < verified-ukmesh-charts.json

The tool reads JSON only from stdin and rejects incomplete, cross-scope, future-dated, more-than-six-hour-old, oversized, or pre-visibility-change input. It binds the response to the current generation and fails if that generation changes during publication. Verify the stored scope_key, schema version, visibility generation, generated_at, and payload byte count before restarting the serving backend. Do not seed an observer-filtered response.

HopReach RF coverage

The app serves only meta.json, progress.json, and numeric PNG tile paths under /rf-coverage/. The private compatibility adapter is mounted at /hopreach on the backend container and rejects forwarded/public traffic. Legacy /api/coverage and /api/coverage/planned contracts return 410 Gone without querying node_coverage or creating work.

Use the HopReach rollout and recovery runbook for source publication, benchmark gates, deployment order, progressive live verification, checkpoint restart tests, and rollback. Do not delete the rf_coverage_data volume or DEM cache to recover a failed run.

Topology API

GET /api/topology?network=ukmesh&limit=300 returns recent viable repeater relationships, bounded to 500 links and a 30-day observation window. It reports connected components, recently active isolated repeaters, and articulation points labelled as likely bridges because the bounded graph is evidence rather than a complete routing model. Private-node names and positions pass through the same redaction policy used by the node APIs.

Public API and exports

GET /api/v1 is the stable discovery endpoint and links to /api/v1/openapi.yaml. Positioned nodes can be exported from /api/v1/exports/nodes.csv and /api/v1/exports/nodes.geojson; both formats are read-only, rate-limited, capped at 5,000 rows, network-scoped, and use the public-node redaction policy.

GET /api/activity/timeline is bounded to a 24-hour window and 250 active node IDs per bucket. GET /api/rf-validation compares stored terrain-model viability with recent observed evidence and distinguishes likely mismatches, weak evidence, and explicit operator overrides.

Public status and maintenance

The public /health page polls the aggregated /api/health contract every minute. Set MAINTENANCE_ACTIVE=1 and MAINTENANCE_MESSAGE during a planned window; the API and page show the notice without leaking hostnames, credentials, addresses, or private node identities.

Operational runbooks

• Immutable release and rollback
• Encrypted backup, isolated restore, and disaster recovery
• Bounded queue and dead-letter recovery
• SRTM cache, alert delivery, observer review, and planned publication
• Database compression and retention gates

Owner authorization baseline refresh

/readyz fails closed when the current owner accounts, grants, configured grants, or ACL generations differ from OWNER_AUTH_INVENTORY_BASELINE_PATH. This means an owner grant or OWNER_MQTT_USERNAME_MAP entry changed after the baseline was recorded. Drift is expected after every approved grant change and is not itself an incident, but it must be reviewed before the baseline is refreshed.

1. Identify the change from the audit trail and confirm it was approved:

   docker exec -e PGOPTIONS='-c default_transaction_read_only=on' \
     meshcore-infra-timescaledb-1 psql -U meshcore -d meshcore_owner_auth -X \
     -c "SELECT occurred_at, event_type, source, actor FROM owner_grant_audit \
         WHERE occurred_at > now() - interval '30 days' ORDER BY occurred_at DESC"
   

2. Generate and validate a fresh baseline with the running image (read-only against the database; it only writes the new baseline file):

   scripts/refresh-owner-baseline.sh
   

3. Activate it after reviewing the printed counts/generations:

   scripts/refresh-owner-baseline.sh --apply --recreate
   curl -fsS http://127.0.0.1:3000/readyz
   

The script refuses to overwrite an existing baseline, backs up .env before editing, keeps the previous baseline for rollback, and validates the new file against the live inventory before activation. To roll back, point OWNER_AUTH_INVENTORY_BASELINE_PATH at the previous file and recreate the backend.

Use the /operations, /observer-registrations, and operator audit pages for normal mutations. They enforce authorization, CSRF, typed confirmation, idempotency, capacity, and audit rules that direct Redis/SQL edits bypass.

Alert first response

Start every alert response by recording the firing time, checking Prometheus target state, and preserving the relevant ten-minute logs. Acknowledge only after assigning an operator. The following headings are stable targets for the Prometheus runbook_url annotations.

BackendDown

Check docker compose ps backend, then backend logs and the container OOM/exit state. If the process is stopped, restart only backend; if it repeatedly fails, keep ingest stopped and roll back to the prior signed digest.

BackendNotReady

Read /readyz and act on the named dependency. Check PostgreSQL, Redis, and Mosquitto before restarting the API. Liveness remaining green is expected during a dependency outage.

SyntheticProbeFailed

Compare the failing blackbox target with direct localhost liveness, stats, and WebSocket checks. If localhost succeeds, inspect Nginx, Anubis, Cloudflare and DNS in that order; do not restart the database.

DependencyExporterDown

Check the named exporter and its dependency independently. An exporter-only failure loses visibility but does not prove PostgreSQL or Redis is down. Restore telemetry, then verify the underlying dependency metric.

MosquittoUnavailable

Check broker health, its WebSocket listener, password/ACL file permissions and the reloader logs. Do not regenerate credentials over an existing set. If ACL reconciliation caused the alert, keep the last verified ACL and roll back that change.

Denied PUBLISH on /raw (#269 / #213)

Disposition: keep these denials. They are expected log noise from publishers using a topic outside the supported observer contract; the evidence does not show a packet-ingest gap.

The B4 review brief reports about 233 Mosquitto Denied PUBLISH lines from ESP32_a806F8 for meshcore/LBA/8CA23B78…/raw, associated with a granted node, plus a separate meshmonitor-observer EXT /raw family. The count and client identities above come from that supplied log review; they were not re-counted against the live broker during this code change.

The renderer in backend/src/mqtt/aclManager.ts grants owner writes to packets, status, neighbors, and neighbours. backend/src/mqtt/topic.ts accepts those suffixes and rejects /raw. Packet bytes are carried in the raw JSON field on the /packets envelope and decoded in backend/src/mqtt/client.ts; there is no /raw topic consumer. Granting the extra suffix would authorize a broker topic that the backend does not ingest.

Reopen this ruling if a supported publisher is denied on /packets, or if the ingest contract is deliberately extended with a /raw consumer. The topic parser test keeps the current rejection explicit.

MeshIngestSilent

Check backend MQTT readiness, broker connected-client events, public topic rates, and whether this is a genuine quiet period. Compare independent observers before changing credentials or subscriptions.

QueueAgeHigh

Open /operations, identify the oldest queue and heartbeat, then follow the queue recovery runbook. Fix the worker/dependency cause before requeueing a single retained dead job.

QueueCapacityHigh

Disable the affected optional link producer if growth continues. Confirm the worker is consuming and the byte/job counters are consistent. Never make Redis eviction-based or raise the cap without measuring host memory.

DeadJobsPresent

Inspect each bounded reason and attempt count in /operations. Requeue only transient failures after the cause is fixed; purge one permanently invalid job with the exact confirmation string.

AnalysisLeaseLost

Preserve both worker logs and the analysis run record. The publication fence prevents a stale writer from committing. Let the current lease expire, resume from its checkpoint, and investigate clock, database, or long-step latency.

HealthWorkerHeartbeatStale

Check the health-worker container, database reachability, and its last successful snapshot. Restart only that worker; public health may be stale but the API should remain available.

WorkerMetricsDown

Identify the role label, inspect that worker's health endpoint and container state, and restart only the affected worker. If work is queued, treat a stale heartbeat as the higher-priority symptom.

ActiveQueueWorkerHeartbeatStale

Stop new optional work, confirm the queue retains its leased payloads, and restart the affected link worker. Allow lease recovery to requeue; do not manually duplicate the job.

DatabaseUnavailable

Check TimescaleDB container state, disk, memory, and recent logs. Stop writers if storage errors appear. Never recreate or delete the volume; escalate to the signed backup/restore runbook.

If a scheduled Timescale policy logs failed to start a background worker, compare SHOW max_worker_processes with SHOW timescaledb.max_background_workers. The Compose default reserves 24 PostgreSQL worker slots for the image's current 16-worker Timescale setting, its launcher/database schedulers, and logical-replication headroom. Correct the configuration and restart only TimescaleDB; then require the policy's next manual or scheduled run to succeed before closing the incident.

DatabasePoolSaturated

Inspect waiting/active connection metrics, slow route latency and PostgreSQL activity. Shed optional analysis/load, identify the bounded slow query, and avoid increasing pool size until connection and memory headroom are measured.

DatabaseWalRateHigh

Identify current maintenance, backfill, compression or ingest activity. Pause the optional writer and check replica/backup capacity. A short known migration burst may be observed; sustained unexplained WAL is an incident.

HostDiskSpaceLow

Stop optional RF/backfill work and identify the mount from the alert. Preserve databases and signed backups. Prune only explicitly identified unused build cache/images or let the bounded SRTM/log policies converge; never run a broad volume prune.

HostDiskExhaustionForecast

Treat a sustained forecast of disk exhaustion within 24 hours as an incident before the low-space threshold fires. Identify the filesystem and its largest consumers, compare the database hypertable and chunk growth gauges, pause optional backfills, and preserve backup capacity. Do not remove database chunks or volumes to make space; follow the signed backup and lifecycle gates.

CoreTelemetryGrowthHigh

Compare the per-table byte and chunk gauges with the retention and compression policies, then run the read-only npm run db:lifecycle inventory during a low traffic window. Check for missed retention/compression jobs and forecast the next 30 days of disk use. Changes to destructive retention require the current signed backup/restore receipt and the per-table approval in docs/db-lifecycle.md.

CoreTelemetryRetentionOverdue

Use read-only queries against timescaledb_information.jobs and timescaledb_information.chunks to identify the missing or failing policy for the labelled table. Verify backup and isolated restore evidence before running the gated per-table lifecycle command. Keep expired forensic access isolated and read-only; never expose restored expired rows through production APIs.

CoreTelemetryCompressionOverdue

Inspect the labelled table's compression policy and Timescale job history, as well as worker capacity and recent database/WAL pressure. Apply compression only through the per-table, receipt-gated command in docs/db-lifecycle.md.

PacketPathsCapacity

Compare the health-worker meshcore_packet_paths_rows_30d, meshcore_packet_paths_bytes_per_row, and overdue-chunk gauges with the reviewed 10,000–23,000 rows/day and roughly 1 kB/row baseline. If compression is overdue, inspect the packet_paths compression job and Timescale worker headroom; never add a retention job for this table. For genuine growth, forecast the next 12 months against free disk before changing storage or ingest policy.

Durable MQTT ingestion

The MQTT client uses the library's inbound message callback for backpressure; QoS 1 PUBACK follows successful database persistence. A database failure is returned as a handler error, leaving the broker delivery unacknowledged for redelivery and keeping readiness failed until a durable write succeeds. Inspect the failure, packet_persist_failure, status_persist_failure, or neighbors_persist_failure ingest outcomes and resolve the database issue; do not clear the broker session or discard its inflight delivery.

BackupReceiptMissing

Check the receipt mount, signature/key permissions, and backup job result. Create a new encrypted backup and complete an isolated restore drill. Do not enable destructive retention while evidence is missing.

BackupStale

Run the backup immediately, verify the signature/checksum, and perform the isolated drill. Investigate target capacity or scheduling after a fresh verified receipt restores the recovery guarantee.

TelemetryIntegrityFailures

Inspect aggregate rejection reasons and a bounded redacted sample. Treat private/cross-network leakage as an immediate rollback condition. Otherwise identify the malformed observer/client version without weakening validation.

Spring-clean additions (2026-08-06)

MQTT ingest resilience

The MQTT client keeps a stable clean=false QoS-1 session and uses MQTT.js message handling as the backpressure boundary. It sends PUBACK only after packet, status, or neighbour persistence completes. Persistence failures are propagated to MQTT.js and close the connection without PUBACK so the persistent broker session redelivers the message. The process-local packet dedupe marker is written only after packet-batch commit. The broker session is meshcore-analytics-ingest (Mosquitto reports c0).

Feed history contract

GET /api/feed/messages?channel=<scope>&limit<=50 — up to 50 unique historical messages per channel, 90-day bound, dedup by packet hash. Uses an ordered keyset-paginated packet scan (500-row pages, in-memory hash dedup, early stop at 50) so it stays fast under TimescaleDB load; observer-stat aggregation is bounded on the history path only (live WS path keeps full aggregation). UI merges history under live WS messages; initial_state still delivers 200. See docs/decryption.md for the join.

Canonical node identity

Evidence-based identity merging is live (docs/node-identity.md, migrations 036/037). Owner dashboard shows one entry per canonical identity with the merged member key list.

Spring-clean UI removals

• Site Health page removed (StatusPage + /health route + legacy /status → /health redirect + nav entry/flag). The separate meshcore-health-check app (healthcheck.ukmesh.com) is unaffected.
• Owner dashboard collapsed to a single page; alert-settings tab and the unused "coming next" roadmap payload removed end-to-end (backend, types, CSS).
• Live map feed cards are message-first with compact metadata; stale repeaters (no recent advert/status AND no multibyte-hop evidence) are filtered from the map; MQTT repeaters render like normal repeaters.
• Spam transparency page verified working (live API) and kept.