ukmesh/README.md

# MeshCore Analytics

A real-time analytics platform for [MeshCore](https://meshcore.io) networks. It ingests MQTT packets from `mctomqtt`-compatible observers, decodes them with `@michaelhart/meshcore-decoder`, stores them in TimescaleDB, and serves interactive dashboards plus public-facing site pages with live mapping, link intelligence, coverage modelling, packet analytics, and worker/system health.

---

## Features

- Real-time node map with animated packet arcs and live WebSocket updates
- RF coverage viewshed polygons per repeater using SRTM terrain data
- Link intelligence overlay with directional observations and path-loss viability
- Public repeater-topology explorer with hub ranking, graph components, likely bridge repeaters, isolated nodes, and multibyte-evidence filtering
- Map modes, shareable viewport/filter URLs, node detail drawer, and bounded activity replay
- Planned-repeater comparison with saved scenarios, share links, and overlap estimates
- Regional health scoring and predicted-versus-observed RF validation
- Local saved searches/watchlists for nodes, observers, regions, packet types, and incidents
- Privacy-filtered CSV/GeoJSON exports with a versioned OpenAPI contract
- Beta path prediction model with concurrent worker pool and hourly path-learning prior rebuilds
- Multibyte path-hash support (1-byte, 2-byte, 3-byte) throughout the live ingest and pathing stack
- Decoded live packet feed (Advert, GroupText, DM, ACK, Path, Trace)
- Stats pages and chart endpoints for packet rates, radios, hops, and activity
- Public Health page with worker status/history + server resource metrics
- UK site Feed page for public MQTT observer traffic visibility
- Repeater owner portal with MQTT username/password login and encrypted cookie session
- Owner dashboard with repeater summary, direct sender map, live packets, advert trend, heard-by list, link health, and alerts
- Multi-network ingestion (`meshcore/*` and `ukmesh/*`) with per-site filtering
- Isolated test-feed support via `meshcore-test/*` and `test.ukmesh.com`
- Multi-observer deduplication by packet hash
- MQTT connection monitor with Mosquitto log parsing and reconnect tracking

---

## Roadmap

### Phase 1 - Core platform (complete)
- MQTT ingestion via `mctomqtt` with multi-observer support
- Packet decoding with `@michaelhart/meshcore-decoder`
- TimescaleDB storage and live WebSocket fan-out
- React dashboard: node map, animated packet arcs, decoded live feed
- Packet deduplication by hash across observers

### Phase 2 - RF coverage and link intelligence (complete)
- Viewshed worker: SRTM terrain-aware radio horizon computation per repeater
- Coverage polygons served as GeoJSON and rendered on the map
- Link worker: observed relay-path processing into node-to-node link intelligence
- Directional link counts and path-loss viability modelling
- UK mainland clipping to remove sea coverage artifacts

### Phase 3 - Path learning and predictions (beta)
- Hourly path-learning prior rebuild worker
- Beta path overlays and confidence scoring
- Historical calibration using observed packet behavior
- Multibyte path-hash aware path resolution
- Concurrent resolve worker pool for high-throughput path matching

### Phase 4 - Public website and operations (complete)
- Separate public-facing website pages (install, MQTT, packets, stats)
- Public Health page with worker/system status and history
- Click-to-explain worker cards
- UK Feed page for live public observer traffic

### Phase 5 - Repeater owner portal (complete)
- MQTT username/password owner login with encrypted cookie session
- Dedicated owner auth database for username → repeater ownership mapping
- Owner-facing dashboard: repeater summary, packet history, advert counts, direct sender map, heard-by list, link health, and alerts
- Planned node placement tool: drop a marker on the map, preview estimated RF coverage before deploying hardware
- Planned repeater registration/claim workflow improvements
- In-app owner alerts for low battery, stale links, advert drops, silence, and missing observers

### Phase 6 - Network intelligence expansion (complete)
- Bounded topology graph with hubs, graph components, isolated repeaters, and likely bridge nodes
- Regional health scoring based on traffic freshness and observer redundancy
- Timeline replay, map modes, shareable views, node details, and planning comparisons
- Path explanations with confidence, evidence, alternatives, and limitations
- Saved searches/watchlists plus versioned CSV and GeoJSON exports

### Phase 7 - Predicted vs observed RF model validation (complete)
- Compare terrain-predicted links against real observed relay behavior
- Highlight high-confidence mismatches for network tuning
- Separate operator overrides and weak evidence from likely model mismatches

### Phase 8 - Reliability and operations (complete)
- CI for backend, frontend, browser journeys, Python workers, and Compose configuration
- Independent synthetic HTTP/WebSocket monitoring with failure and recovery webhooks
- Liveness/readiness split, public status page, DB maintenance telemetry, and bounded load tooling
- Restartable, audited production-network label migration with snapshot-based rollback guidance

---

## Current State

- Split worker architecture for resilience:
  - `viewshed-worker` (coverage compute)
  - `link-worker` (link/path-loss processing)
  - `path-learning-worker` (hourly model rebuild)
  - `path-history-worker` (historical path resolution backfill)
  - `health-worker` (health snapshots)
  - `link-backfill-worker` (one-shot historical backfill)
- Path resolver runs a concurrent worker pool (`resolveWorker`, `resolvePool`, `resolveCache`) to handle high packet volumes without blocking the main ingest loop.
- Nginx frontend proxies use Docker DNS resolver-based upstreams to avoid stale backend IP issues after container recreates.
- Owner authentication uses MQTT credentials plus a separate owner-auth mapping database rather than public-key login.
- Live coverage is currently served by an RF radial model calibrated against observed repeater links and terrain data.
- Public/test feeds are isolated at the topic level, with `meshcore-test/*` excluded from the public sites.
- MQTT connection state is tracked via Mosquitto log parsing — connect/disconnect events are available in the health feed.

---

## Quick Start

```bash
# 1. Clone and enter the project
git clone https://github.com/gadgethd/ukmesh.git
cd ukmesh

# 2. Copy and configure environment
cp .env.example .env
# Edit .env — at minimum set POSTGRES_PASSWORD, JWT_SECRET, MQTT_PASSWORD,
# and REDIS_PASSWORD.

# 3. Start everything
docker compose up -d

# 4. Check logs
docker compose logs -f backend
```

RF coverage and planned-repeater analysis are opt-in because terrain processing
is resource-intensive. Set `VIEWSHED_ENABLED=true` and start its worker only
when needed:

```bash
docker compose --profile viewshed up -d viewshed-worker
```

Local endpoints:

- Backend API/WS: `http://localhost:3000`
- App (ukmesh): `http://localhost:3003`
- Website (ukmesh): `http://localhost:3004`
- Dev/test site: `http://localhost:3006`
- Liveness/readiness: `http://localhost:3000/healthz` and `http://localhost:3000/readyz`
- API discovery/OpenAPI: `http://localhost:3000/api/v1` and `http://localhost:3000/api/v1/openapi.yaml`

To expose it publicly, configure a Cloudflare Tunnel (see below) or reverse proxy of your choice.

---

## Environment Variables

Copy `.env.example` to `.env` and fill in your values. All variables used by the app:

| Variable | Default | Description |
|---|---|---|
| `POSTGRES_DB` | `meshcore` | TimescaleDB database name |
| `POSTGRES_USER` | `meshcore` | TimescaleDB user |
| `POSTGRES_PASSWORD` | *(required)* | TimescaleDB password |
| `MQTT_BROKER_URL` | `ws://mosquitto:9001` | Mosquitto WebSocket URL (internal) |
| `MQTT_USERNAME` | `backend` | MQTT client username |
| `MQTT_PASSWORD` | *(required)* | MQTT client password |
| `REDIS_PASSWORD` | *(required)* | Password for the bundled Redis service; passed separately from the URL so reserved characters are safe |
| `REDIS_URL` | `redis://redis:6379` | Redis URL for WebSocket pub/sub |
| `JWT_SECRET` | *(required)* | Secret for JWT verification |
| `ALLOWED_ORIGINS` | `http://localhost:3001,http://localhost:3002` | Comma-separated browser origins allowed for CORS and WebSocket |
| `VITE_APP_HOSTNAME` | *(blank — always shows dashboard)* | If set, only this hostname serves the analytics dashboard; all others serve the public website layout |
| `MESHCORE_CHANNEL_SECRETS` | *(blank)* | Comma-separated channel secrets for decrypting GroupText packets. Format: `name:hex` or bare hex. The default MeshCore public channel key is always included. |
| `OPENTOPODATA_API` | `https://api.opentopodata.org` | Elevation API endpoint for viewshed computation |
| `OWNER_DATABASE_URL` | *(optional)* | Separate Postgres database URL for owner portal username → repeater mappings |
| `OWNER_COOKIE_SECRET` | *(optional but recommended)* | Secret used to encrypt/sign the owner session cookie |
| `OWNER_MQTT_USERNAME_MAP` | *(empty)* | Operator-managed owner grants in the format `user=nodeId1|nodeId2,...` |
| `OWNER_AUTHORIZATION_MODE` | `shadow` | `shadow` preserves read-only legacy ACL compatibility; `enforce` accepts verified database/config grants only |
| `OWNER_ACL_MODE` | `shadow` | `shadow` renders and validates without changing Mosquitto; `apply` atomically installs and verifies the canonical ACL |
| `OWNER_ACL_UNMANAGED_USERS` | `backend,test,test2` | Exact broker accounts intentionally preserved outside owner grant management |
| `OWNER_ACL_ALLOW_EMPTY_USERS` | *(empty)* | Explicitly reviewed owner accounts allowed to render with no publish grants |
| `VIEWSHED_ENABLED` | `false` | Enable coverage/planned-repeater API only when the profile-gated viewshed worker is running |
| `COVERAGE_MODEL` | `rf_radial_100m` | Coverage model used by `viewshed-worker` |
| `COVERAGE_MODEL_VERSION` | `5` | Coverage schema/version gate used to trigger recomputation |
| `CLOUDFLARE_TUNNEL_TOKEN` | *(optional)* | Cloudflare Zero Trust tunnel token |
| `PORT` | `3000` | Internal app port |

---

## Mosquitto Setup

Mosquitto is configured for WebSocket-only access with password authentication.
After first starting the stack, add the backend service password:

```bash
# Add the backend client password (must match MQTT_PASSWORD in .env)
docker exec meshcore-analytics-mosquitto-1 \
  mosquitto_passwd -b /mosquitto/config/passwd backend your_password

docker compose restart mosquitto
```

Do not create observer passwords manually. Provision every observer with
`~/bin/newuser`; it installs and verifies the publish ACL before enabling the
credentials.

The host helper at `~/bin/newuser` requires one or more full 64-character node
public keys. It validates the keys, writes and verifies exact per-key publish
ACLs, and only then creates the MQTT password. This ordering prevents an account
from authenticating successfully while all of its publishes are silently denied.
In MeshCore-HA, keep `{PUBLIC_KEY}` in the topic template; the helper requires the
actual key only to provision the server-side ACL.

---

## Cloudflare Tunnel (optional)

To expose the app and MQTT broker publicly without opening firewall ports:

1. Go to [Cloudflare Zero Trust](https://one.dash.cloudflare.com/) → Networks → Tunnels
2. Create a tunnel and copy the token
3. Add to `.env`: `CLOUDFLARE_TUNNEL_TOKEN=<token>`
4. Start with the tunnel profile: `docker compose --profile tunnel up -d`
5. Configure public hostnames in the Cloudflare dashboard (example):
   - `app.example.com` → `http://app-ukmesh:80`
   - `www.example.com` → `http://website-ukmesh:80`
   - `mqtt.example.com` → `http://mosquitto:9001`
   - `healthcheck.example.com` → `http://mesh-health-check:3090`

For UKMesh health checks, point `healthcheck.ukmesh.com` at
`http://mesh-health-check:3090` in the same tunnel. The container uses the
internal Mosquitto WebSocket listener and persists observer/result state in the
`mesh_health_check_data` Docker volume. By default it uses
`HEALTHCHECK_TEST_CHANNEL_NAME=ukmeshtest` and reuses the existing `test:...`
entry from `MESHCORE_CHANNEL_SECRETS` via
`HEALTHCHECK_TEST_CHANNEL_SECRET_SOURCE_NAME=test`.

---

## MQTT Topic Structure

The backend subscribes to `meshcore/#`, `ukmesh/#`, and `meshcore-test/#`. MeshCore observers publish `mctomqtt`-compatible JSON envelopes to topics of the form:

```
meshcore/<IATA>/<observer-public-key>/packets   # received/transmitted packets
meshcore/<IATA>/<observer-public-key>/status    # node status advertisement
ukmesh/<IATA>/<observer-public-key>/packets
ukmesh/<IATA>/<observer-public-key>/status
meshcore-test/<IATA>/<observer-public-key>/packets
meshcore-test/<IATA>/<observer-public-key>/status
```

Payloads are JSON envelopes containing a `raw` hex field (the MeshCore packet) plus metadata such as RSSI, SNR, direction, and hash. The ingest path supports 1-byte, 2-byte, and 3-byte path hashes carried inside the raw packet.

---

## Architecture

```
MeshCore Devices
     │ LoRa RF
     ▼
 mctomqtt-compatible observer
     │ MQTT over WebSocket/TLS
     ▼
 Mosquitto ─────────────────────────────── (optional Cloudflare Tunnel)
     │ subscribe meshcore/# + ukmesh/#
     ▼
 Backend (Node.js/TypeScript)
     │
     ├─ meshcore-decoder → TimescaleDB (packets, nodes, coverage, priors, health snapshots)
     │
     ├─ Path resolver worker pool (concurrent resolve workers + LRU cache)
     │
     ├─ Redis pub/sub
     │
     ├─ WebSocket → frontend live updates
     └─ REST API /api/*

 App/Web Frontends (Nginx + React)
     └─ app-ukmesh / website-ukmesh / website-dev (interactive dashboard + public site + owner portal)

 Python Workers
     ├─ viewshed-worker (meshcore:viewshed_jobs)
     ├─ link-worker (meshcore:link_jobs)
     ├─ SRTM terrain tiles (auto-downloaded)
     └─ node_coverage + node_links updates

 Backend Workers (Node.js)
     ├─ path-learning-worker (hourly prior rebuild)
     ├─ path-history-worker (historical path resolution)
     ├─ health-worker (minute snapshots)
     └─ link-backfill-worker (one-shot historical backfill)

 Owner Auth
     └─ separate Postgres DB for MQTT username → repeater ownership mapping
```

---

## Services

| Service | Image | Purpose |
|---|---|---|
| `timescaledb` | `timescale/timescaledb:latest-pg16` | Time-series and relational data storage |
| `mosquitto` | `eclipse-mosquitto:2` | MQTT broker (WebSocket only) |
| `redis` | `redis:7-alpine` | WebSocket fan-out pub/sub and job queue |
| `backend` | Built from `Dockerfile.backend` | MQTT ingest, decoding, API, WebSocket |
| `path-learning-worker` | Built from `Dockerfile.backend` | Hourly path-learning model rebuilds |
| `path-history-worker` | Built from `Dockerfile.backend` | Historical path resolution backfill |
| `health-worker` | Built from `Dockerfile.backend` | Periodic health snapshot capture |
| `link-backfill-worker` | Built from `Dockerfile.backend` | One-shot historical link backfill |
| `synthetic-monitor` | Built from `Dockerfile.backend` | Independent HTTP/WebSocket journey checks and alert delivery |
| `viewshed-worker` | Built from `viewshed-worker/Dockerfile` | Terrain-aware RF coverage computation |
| `link-worker` | Built from `viewshed-worker/Dockerfile` | Link/path-loss processing from observed paths |
| `app-ukmesh` | Built from `Dockerfile.app` | Interactive dashboard frontend |
| `website-ukmesh` | Built from `Dockerfile.website` | Public website frontend |
| `mesh-health-check` | Built from the configured `gadgethd/meshcore-health-check` ref | MeshCore observer coverage health-check app |
| `website-dev` | Built from `Dockerfile.website` | Isolated test/status site for `meshcore-test/*` traffic |
| `cloudflared` | `cloudflare/cloudflared` | Optional Cloudflare Tunnel (use `--profile tunnel`) |

---

## Data Retention

- Packet retention policy is currently disabled. Historical data is kept indefinitely unless explicitly pruned.
- Node/link/coverage/path-learning/health tables are also retained indefinitely by default.
- Synthetic journey results are pruned to 14 days by the monitor.

---

## Acknowledgements

This project is built on the following open source libraries and tools:

### Frontend
| Package | License |
|---|---|
| [React](https://react.dev) | MIT |
| [Vite](https://vitejs.dev) | MIT |
| [TypeScript](https://www.typescriptlang.org) | Apache 2.0 |
| [MapLibre GL JS](https://maplibre.org/maplibre-gl-js/docs/) | BSD 3-Clause |
| [deck.gl](https://deck.gl) | MIT |
| [react-router-dom](https://reactrouter.com) | MIT |
| [Recharts](https://recharts.org) | MIT |
| [polygon-clipping](https://github.com/mfogel/polygon-clipping) | MIT |

### Backend
| Package | License |
|---|---|
| [Express](https://expressjs.com) | MIT |
| [MQTT.js](https://github.com/mqttjs/MQTT.js) | MIT |
| [ws](https://github.com/websockets/ws) | MIT |
| [ioredis](https://github.com/redis/ioredis) | MIT |
| [node-postgres](https://node-postgres.com) | MIT |
| [cors](https://github.com/expressjs/cors) | MIT |
| [express-rate-limit](https://github.com/express-rate-limit/express-rate-limit) | MIT |
| [@michaelhart/meshcore-decoder](https://www.npmjs.com/package/@michaelhart/meshcore-decoder) | MIT |

### Viewshed worker (Python)
| Package | License |
|---|---|
| [NumPy](https://numpy.org) | BSD 3-Clause |
| [SciPy](https://scipy.org) | BSD 3-Clause |
| [Shapely](https://shapely.readthedocs.io) | BSD 3-Clause |
| [GDAL](https://gdal.org) | MIT/X |
| [psycopg2](https://www.psycopg.org) | LGPL v3 |
| [redis-py](https://github.com/redis/redis-py) | MIT |
| [Requests](https://requests.readthedocs.io) | Apache 2.0 |

### Infrastructure
| Tool | License |
|---|---|
| [TimescaleDB](https://www.timescale.com) | Apache 2.0 (Community) |
| [Redis](https://redis.io) | BSD 3-Clause |
| [Eclipse Mosquitto](https://mosquitto.org) | EPL 2.0 / EDL 1.0 |
| [Docker](https://www.docker.com) | Apache 2.0 |

### Data
| Source | License |
|---|---|
| [SRTM Elevation Data](https://registry.opendata.aws/terrain-tiles) | Public Domain (NASA) |
| [Natural Earth / world-atlas](https://www.naturalearthdata.com) | Public Domain |

---

## License

This project is licensed under MIT — see [LICENSE](LICENSE).

**Note on dependencies:** Eclipse Mosquitto (EPL 2.0) is used as a dependency but not modified. Other runtime dependencies use MIT, BSD, or Apache 2.0 licenses.