ukmesh/docs/operations.md

Operations

Automated checks

Pull requests and branch pushes run .github/workflows/ci.yml. The workflow:

• installs dependencies from lockfiles
• type-checks, tests, and builds the backend
• builds the frontend and runs Playwright desktop/mobile smoke tests
• compiles Python workers
• validates the Docker Compose model with required secrets represented by CI-only placeholders

Dependabot groups weekly dependency updates for the backend, frontend, and GitHub Actions.

Health endpoints

• GET /healthz is a process liveness check. It deliberately remains healthy during an MQTT outage so Docker does not restart an otherwise functioning API in a loop.
• GET /readyz checks database access and MQTT connectivity. It returns 503 with per-dependency state when the real-time platform is not ready.
• GET /api/health returns worker history and a top-level status (healthy, degraded, or critical) plus machine-readable problems.

Current problem codes cover stale public ingest, worker queue backlog, disk pressure, and frontend error spikes. External monitoring should alert on readyz failures and critical /api/health problems, while allowing a short deploy/reconnect grace period.

The synthetic-monitor service independently checks liveness, dependency readiness, the scoped stats API, and delivery of a WebSocket initial_state. Results and latency are retained for 14 days in operational_check_results. Three consecutive failures trigger a structured log alert and, when ALERT_WEBHOOK_URL is configured, a JSON webhook. A successful check after an alert emits a recovery notification.

Load checks

Run bounded HTTP load locally with:

cd backend
npm run load:realtime -- --duration 30 --concurrency 25 --max-p95-ms 1500

Use --mqtt-messages 5000 to exercise the bounded MQTT ingest queue with isolated, rejected test envelopes, and --slow-ws-clients 10 to hold non-reading WebSocket clients during the run. MQTT mode uses MQTT_BROKER_URL, MQTT_USERNAME, and MQTT_PASSWORD; never point it at a broker outside the deployment under test.

Coverage API safety

The legacy unbounded GET /api/coverage response has been replaced by a bounded viewport API:

GET /api/coverage?bbox=minLon,minLat,maxLon,maxLat&limit=12&cursor=<node-id>

• bbox is required and may span at most 20 degrees on either axis.
• limit defaults to 12 and is capped at 25.
• Each serialized page has a 5 MiB safety budget.
• A single geometry exceeding that budget is represented by {node_id, truncated: true} and can be retrieved through the per-node endpoint.
• page.nextCursor continues a partial result when page.hasMore is true.
• GET /api/coverage/:nodeId remains the preferred interactive-map endpoint.

Topology API

GET /api/topology?network=ukmesh&limit=300 returns recent viable repeater relationships, bounded to 500 links and a 30-day observation window. It reports connected components, recently active isolated repeaters, and articulation points labelled as likely bridges because the bounded graph is evidence rather than a complete routing model. Private-node names and positions pass through the same redaction policy used by the node APIs.

Public API and exports

GET /api/v1 is the stable discovery endpoint and links to /api/v1/openapi.yaml. Positioned nodes can be exported from /api/v1/exports/nodes.csv and /api/v1/exports/nodes.geojson; both formats are read-only, rate-limited, capped at 5,000 rows, network-scoped, and use the public-node redaction policy.

GET /api/activity/timeline is bounded to a 24-hour window and 250 active node IDs per bucket. GET /api/rf-validation compares stored terrain-model viability with recent observed evidence and distinguishes likely mismatches, weak evidence, and explicit operator overrides.

Public status and maintenance

The public /health page polls the aggregated /api/health contract every minute. Set MAINTENANCE_ACTIVE=1 and MAINTENANCE_MESSAGE during a planned window; the API and page show the notice without leaking hostnames, credentials, addresses, or private node identities.