mirror of
https://github.com/MeshCore-Beacon/beacon-server.git
synced 2026-09-01 16:48:19 +00:00
427 lines
20 KiB
Markdown
427 lines
20 KiB
Markdown
# MeshCore Beacon
|
|
|
|
MeshCore Beacon is a MeshCore network observation backend. It connects to one or
|
|
more MeshCore MQTT brokers, ingests LoRa packet traffic in real time, stores it
|
|
in PostgreSQL, and streams live events to WebSocket clients.
|
|
|
|
[](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/ci.yml)
|
|
[](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/codeql.yml)
|
|
|
|
[](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/docker-publish.yml)
|
|
|
|
## What it does
|
|
|
|
- Subscribes to MeshCore MQTT brokers and decodes incoming LoRa packets using
|
|
[meshcore-go](https://github.com/meshcore-go/meshcore-go)
|
|
- Stores packets, observations, nodes, observers, traces, routes and channel
|
|
messages in PostgreSQL (more backends to come)
|
|
- Deduplicates observations across multiple brokers (same packet heard by two
|
|
brokers is one observation per observer)
|
|
- Decrypts group text messages for known channel keys
|
|
- Detects firmware capability flags from path hash sizes
|
|
- Streams live events to WebSocket clients with subscription filtering by IATA,
|
|
region, payload type, and event type
|
|
- Serves a REST API for querying stored data
|
|
- Seeds regions, IATA display names, and channel keys from a YAML config file on
|
|
startup
|
|
|
|
For deployment instructions including the frontend app, see the deployment docs.
|
|
|
|
---
|
|
|
|
## Stack
|
|
|
|
| Component | Technology |
|
|
| ------------- | --------------------------------------------------------------- |
|
|
| Language | Go 1.26 |
|
|
| Router | [Chi v5](https://github.com/go-chi/chi) |
|
|
| Database | PostgreSQL 16 |
|
|
| Caching | Redis 7 |
|
|
| DB queries | [sqlc](https://sqlc.dev) + pgx/v5 |
|
|
| MQTT | [paho.mqtt.golang](https://github.com/eclipse/paho.mqtt.golang) |
|
|
| WebSocket | [coder/websocket](https://github.com/coder/websocket) |
|
|
| Packet decode | [meshcore-go](https://github.com/meshcore-go/meshcore-go) |
|
|
| Config | YAML via gopkg.in/yaml.v3 |
|
|
| Env | godotenv |
|
|
|
|
---
|
|
|
|
## Getting started
|
|
|
|
### Prerequisites
|
|
|
|
- Go 1.26+
|
|
- Docker and Docker Compose
|
|
|
|
### 1. Clone and configure
|
|
|
|
```bash
|
|
git clone https://github.com/MeshCore-Beacon/beacon-server.git
|
|
cd beacon-server
|
|
cp env.example .env
|
|
cp config.yaml.example config.yaml
|
|
```
|
|
|
|
Edit `.env` with your broker credentials and database DSN. Edit `config.yaml` to
|
|
define your regions, IATA display names, channel keys, and retention settings.
|
|
|
|
### 2. Start PostgreSQL
|
|
|
|
```bash
|
|
docker compose up postgres -d
|
|
```
|
|
|
|
Database migrations are applied automatically on startup.
|
|
|
|
### 3. Run
|
|
|
|
```bash
|
|
go run ./cmd/beacon
|
|
```
|
|
|
|
Or pull and run the Docker image:
|
|
|
|
```bash
|
|
docker pull ghcr.io/meshcore-beacon/beacon-server:latest
|
|
```
|
|
|
|
The image is public on GitHub Container Registry — no `docker login` required.
|
|
|
|
Beacon will:
|
|
|
|
- Load `.env` and `config.yaml`
|
|
- Connect to PostgreSQL and seed config data
|
|
- Connect to the configured MQTT brokers
|
|
- Start the HTTP server on `LISTEN_ADDR` (default `:8080`)
|
|
|
|
### Cold start and path resolution
|
|
|
|
Path resolution, firmware capability detection, and known route storage all
|
|
depend on nodes having advertised at least once to a local observer. On a fresh
|
|
deployment `resolvedPath` will show `"confidence": "none"` for all hops and
|
|
`supportsMultibytePaths` will be `false` for all nodes until advert traffic
|
|
arrives and populates `node_short_ids`. This is expected behaviour — resolution
|
|
improves automatically as the mesh is observed over time.
|
|
|
|
Similarly, GRP_TXT packets whose channel key isn't yet known at ingest time are
|
|
stored as hash-only, undecrypted rows. Adding the channel's key to `config.yaml`
|
|
doesn't retroactively decrypt that history immediately — it's picked up
|
|
automatically on the next restart, when Beacon scans for undecrypted packets
|
|
matching a now-known channel and decrypts them. Watch the startup log for
|
|
`backfilled N previously-undecrypted channel message(s)`.
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
### Environment variables (`.env`)
|
|
|
|
| Variable | Default | Description |
|
|
| ------------------------ | ------------- | ------------------------------------------------------------ |
|
|
| `LISTEN_ADDR` | `:8080` | HTTP listen address |
|
|
| `POSTGRES_DSN` | — | PostgreSQL connection string |
|
|
| `REDIS_ADDR` | — | Redis address (`host:port`). Leave unset to disable caching. |
|
|
| `REDIS_PASSWORD` | — | Redis password (optional) |
|
|
| `REDIS_DB` | `0` | Redis database index |
|
|
| `CONFIG_PATH` | `config.yaml` | Path to YAML config file |
|
|
| `MQTT_BROKER_1_URL` | — | Broker 1 WebSocket URL (e.g. `wss://mqtt1.example.com:443`) |
|
|
| `MQTT_BROKER_1_USERNAME` | — | Broker 1 username |
|
|
| `MQTT_BROKER_1_PASSWORD` | — | Broker 1 password |
|
|
| `MQTT_BROKER_2_URL` | — | Broker 2 WebSocket URL |
|
|
| `MQTT_BROKER_2_USERNAME` | — | Broker 2 username |
|
|
| `MQTT_BROKER_2_PASSWORD` | — | Broker 2 password |
|
|
|
|
### Config file (`config.yaml`)
|
|
|
|
```yaml
|
|
# Optional IATA overrides — auto-created on first packet arrival,
|
|
# only needed if you want to customise display name or coordinates.
|
|
# borderFile points to a GeoJSON Feature (Polygon or MultiPolygon) for the
|
|
# region border map feature; relative paths resolve against this config
|
|
# file's own directory. Validated at boot — invalid geometry fails startup.
|
|
iatas:
|
|
YVR:
|
|
name: Vancouver International
|
|
lat: 49.1967
|
|
lng: -123.1815
|
|
borderFile: borders/yvr.geojson # optional
|
|
|
|
# Super-regions grouping multiple IATAs.
|
|
regions:
|
|
- slug: western-canada
|
|
name: Western Canada
|
|
display_order: 1
|
|
center_lat: 51.0
|
|
center_lng: -114.0
|
|
zoom_level: 5
|
|
iatas: [YVR, YYJ, YYC, YEG]
|
|
|
|
# Channel keys for decrypting group messages.
|
|
channel_keys:
|
|
# Hashtag channels: Beacon derives the PSK from the tag name automatically.
|
|
# secret = SHA256("#tag")[:16], channel_hash = SHA256(secret)[0]
|
|
# Tag names should be provided without the # prefix.
|
|
hashtags:
|
|
- meshcore
|
|
|
|
# Explicit keys: channel hash (hex) and key (hex), with optional display name.
|
|
# The public MeshCore channel key is included in config.yaml.example.
|
|
keys:
|
|
"11":
|
|
key: "8b3387e9c5cdea6ac9e5edbaa115cd72"
|
|
name: "Public"
|
|
|
|
# Regional transport scopes for matching TRANSPORT_FLOOD packets.
|
|
# Plain names have # prepended automatically (e.g. "bc" → "#bc").
|
|
scopes:
|
|
- name: bc
|
|
- name: "#west"
|
|
|
|
# Observer telemetry storage settings.
|
|
telemetry:
|
|
retention: 672h # how long to keep telemetry snapshots (default: 4 weeks)
|
|
resolution: 1h # snapshot frequency per observer; duplicates within window are dropped (default: 1h)
|
|
|
|
# Packet and observation retention.
|
|
packets:
|
|
retention: 720h # how long to keep packets and observations (default: 30 days)
|
|
|
|
# Presence write coalescing.
|
|
# Observer last_seen and packet last_heard_at bumps are batched in memory and
|
|
# flushed on an interval instead of writing one row per observation. An
|
|
# unclean shutdown loses at most one interval of presence freshness.
|
|
presence:
|
|
flush_interval: 30s # how often coalesced bumps are flushed (default: 30s)
|
|
packet_ttl: 30s # how long a quiet packet stays coalesced before writing through again (default: 30s)
|
|
|
|
# WebSocket settings.
|
|
websocket:
|
|
max_connections_per_ip: 5 # default: 5
|
|
|
|
# Node staleness, deletion, and clock-drift thresholds.
|
|
nodes:
|
|
stale_threshold: 24h # mark a node "stale" in the API after this long unseen (default: 24h)
|
|
delete_after: 720h # delete a node entirely after this long unseen (default: 30 days, same default as packets.retention)
|
|
clock_drift_threshold: 5m # |device clock - server clock| above which clockOutOfSync=true for a repeater/room server (default: 5m)
|
|
|
|
# Redis caching layer (optional).
|
|
# Caches read-heavy, slow-changing responses to reduce PostgreSQL load.
|
|
# Connection details (address, password, database) are set via environment
|
|
# variables. Leave REDIS_ADDR unset to disable caching entirely.
|
|
# TTLs are duration strings e.g. "30m", "1h". Per-category TTLs override
|
|
# the global ttl. Any unset category inherits ttl. Default: 1h.
|
|
cache:
|
|
ttl: "1h"
|
|
ttls:
|
|
stats: "1h" # stats endpoints (backed by materialized views)
|
|
reference: "1h" # IATAs, regions, scopes
|
|
nodes: "1h" # node detail (also explicitly invalidated on upsert)
|
|
observers: "1h" # observer detail (also explicitly invalidated on upsert)
|
|
|
|
# Geographic ingest filter (optional).
|
|
# Drop packets from observers outside the specified area.
|
|
# Country codes are ISO 3166-1 alpha-2. Continent codes: AF AN AS EU NA OC SA.
|
|
# If both are set an IATA passes if it matches either (OR semantics).
|
|
# Omit entirely to accept all IATAs (default).
|
|
ingest:
|
|
allow_countries: [CA, US] # only store packets from these countries
|
|
allow_continents: [NA] # or: accept all of North America
|
|
```
|
|
|
|
IATAs are auto-created on first packet arrival. The config file adds display
|
|
names, coordinates, and optional region borders. Regions and channel keys
|
|
must be defined here — they are not auto-created.
|
|
|
|
---
|
|
|
|
## Authentication
|
|
|
|
API authentication is not yet implemented. Beacon is intended for trusted
|
|
internal network or reverse-proxy deployments. Do not expose it directly to the
|
|
public internet without an authentication layer in front of it.
|
|
|
|
---
|
|
|
|
## WebSocket API
|
|
|
|
Connect to `ws://host:8080/ws`.
|
|
|
|
On connect the server sends a `hello`:
|
|
|
|
```json
|
|
{ "v": 1, "type": "hello", "serverTime": 1234567890000, "connectionId": "uuid" }
|
|
```
|
|
|
|
The connection closes after 90 seconds of inactivity. Clients should send a
|
|
`ping` every 30 seconds.
|
|
|
|
### Client → Server messages
|
|
|
|
**Subscribe** — add a filter to this connection. Multiple subscriptions are
|
|
unioned (OR semantics): an event matches if it satisfies any active
|
|
subscription. The server replies with a `subscriptionId` to use for
|
|
unsubscribing.
|
|
|
|
```json
|
|
{
|
|
"v": 1,
|
|
"type": "subscribe",
|
|
"id": "sub-1",
|
|
"scope": {
|
|
"iatas": ["YOW", "YYZ"],
|
|
"regionIds": ["1"],
|
|
"regionSlugs": ["western-canada"],
|
|
"payloadTypes": [4, 5],
|
|
"channelHashes": ["11"],
|
|
"events": ["packetObservation", "channelMessage"]
|
|
}
|
|
}
|
|
```
|
|
|
|
All scope fields are optional. Omitted means no filter on that dimension (match
|
|
everything). Empty array means match nothing on that dimension. `regionIds` and
|
|
`regionSlugs` are both expanded to their member IATAs server-side.
|
|
|
|
**Unsubscribe** — remove a specific subscription by ID.
|
|
|
|
```json
|
|
{
|
|
"v": 1,
|
|
"type": "unsubscribe",
|
|
"id": "unsub-1",
|
|
"subscriptionId": "<uuid from subscribed reply>"
|
|
}
|
|
```
|
|
|
|
**Configure** — toggle connection-wide settings. Currently just `resolvePath`,
|
|
which enables per-hop node resolution on `packetObservation` events (see
|
|
below). Unlike `subscribe`, this is a single flag for the whole connection,
|
|
not additive across calls — each `configure` sets it to exactly the value
|
|
sent, and it can be flipped on or off as many times as you like for the life
|
|
of the connection. Default is `false`.
|
|
|
|
```json
|
|
{ "v": 1, "type": "configure", "id": "cfg-1", "resolvePath": true }
|
|
```
|
|
|
|
The server replies:
|
|
|
|
```json
|
|
{ "v": 1, "type": "configured", "id": "cfg-1", "resolvePath": true }
|
|
```
|
|
|
|
**Ping**
|
|
|
|
```json
|
|
{ "v": 1, "type": "ping", "id": "ping-1" }
|
|
```
|
|
|
|
### Server → Client events
|
|
|
|
| Type | Description |
|
|
| ------------------- | --------------------------------------------------- |
|
|
| `packetObservation` | New observation written to DB |
|
|
| `observerStatus` | Observer status update |
|
|
| `nodeUpdate` | Node upserted from advert |
|
|
| `channelMessage` | Decrypted channel message (scope must include hash) |
|
|
|
|
When `resolvePath` is enabled via `configure`, `packetObservation` events
|
|
include a `resolvedPath` array: one entry per hop in the packet's path, each
|
|
with a `confidence` (`"high"` for exactly one candidate node, `"ambiguous"`
|
|
for multiple, `"none"` for zero) and the matching node(s)' id, name,
|
|
lat/lng, and public key. Without `resolvePath` enabled, `resolvedPath` is
|
|
`null` — this keeps the default event payload small; only opt in if you're
|
|
actually rendering path connections (e.g. drawing hops on a map).
|
|
|
|
### Backpressure
|
|
|
|
The server write buffer per connection is bounded at 256 events. If a client
|
|
falls behind, the server drops the oldest queued events and sends a `lagged`
|
|
notice:
|
|
|
|
```json
|
|
{ "v": 1, "type": "lagged", "droppedCount": 12, "since": 1234567890000 }
|
|
```
|
|
|
|
Clients should respond by re-fetching the relevant REST endpoint using `afterId`
|
|
to backfill missed events, then resume streaming.
|
|
|
|
### Reconnection
|
|
|
|
Subscriptions are not persisted — they exist only for the lifetime of the
|
|
connection. On any disconnect the client should reconnect with backoff, re-issue
|
|
all subscriptions, and backfill via REST using
|
|
`afterId=<last seen observation id>`.
|
|
|
|
### Connection limits
|
|
|
|
By default a maximum of 5 concurrent WebSocket connections are allowed per IP
|
|
address. Connections beyond this limit receive `HTTP 429`. The limit is
|
|
configurable via `websocket.max_connections_per_ip` in `config.yaml`.
|
|
|
|
---
|
|
|
|
## REST API
|
|
|
|
Base path: `/api/v1`
|
|
|
|
All list endpoints support cursor-based pagination via `cursor` and `limit`
|
|
query params. See the Swagger UI at `http://localhost:8080/swagger/index.html`
|
|
for full parameter documentation.
|
|
|
|
### Authentication
|
|
|
|
Not yet implemented — see the Authentication section above.
|
|
|
|
### Endpoints
|
|
|
|
| Method | Path | Description |
|
|
| ------ | ----------------------------------- | -------------------------------------------------------------------------------------------------- |
|
|
| `GET` | `/brokers` | List MQTT brokers and connection status |
|
|
| `GET` | `/channels` | List channels (optional: `?hash=<hex>&iata=<code>&limit=50`) |
|
|
| `GET` | `/channels/{id}` | Get channel detail by integer ID |
|
|
| `GET` | `/channels/{id}/messages` | List messages for a channel (optional: `?since=<ms>&iata=<code>&limit=50`) |
|
|
| `GET` | `/iatas` | List all known IATA codes |
|
|
| `GET` | `/iatas/{iata}` | Get a single IATA code |
|
|
| `GET` | `/iatas/{iata}/border` | Get an IATA's GeoJSON region border, if configured (204 if not) |
|
|
| `GET` | `/messages` | List all messages (optional: `?channelId=<int>&channelHash=<hex>&iata=<code>&since=<ms>&limit=50`) |
|
|
| `GET` | `/messages/backfill` | Backfill messages after a given message ID |
|
|
| `GET` | `/nodes` | List nodes (optional: `?pubkeyPrefix=<hex>&neighbors&iatas=<codes>&pubkey=<hex>`) |
|
|
| `GET` | `/nodes/{nodeId}` | Get node detail |
|
|
| `GET` | `/nodes/{nodeId}/neighbors` | List neighboring nodes observed in the mesh |
|
|
| `GET` | `/nodes/{nodeId}/observations` | List observations for a node |
|
|
| `GET` | `/observers` | List observers (optional: `?iata=<code>&type=<str>&broker=<name>&status=online\|offline`) |
|
|
| `GET` | `/observers/{observerId}` | Get observer detail including broker last-seen timestamps |
|
|
| `GET` | `/observers/{observerId}/adverts` | Adverts heard by observer |
|
|
| `GET` | `/observers/{observerId}/telemetry` | Observer telemetry history (optional: `?range=24h&interval=1h\|6h\|24h`) |
|
|
| `GET` | `/packets` | List packets (optional: `?payloadTypes=<csv>&routeTypes=<csv>&scopes=<csv>` accept plural, comma-separated values alongside the singular params) |
|
|
| `GET` | `/packets/backfill` | Backfill packets after a given observation ID |
|
|
| `GET` | `/packets/{packetHash}` | Get packet with all observations |
|
|
| `GET` | `/regions` | List all regions (summary) |
|
|
| `GET` | `/regions/{id}` | Get a single region with IATA list |
|
|
| `GET` | `/routes` | List known routes (all hops high confidence) |
|
|
| `GET` | `/routes/search` | Search routes by source and destination hash |
|
|
| `GET` | `/routes/cross` | Search for routes crossing IATA boundaries |
|
|
| `GET` | `/scopes` | List transport scopes |
|
|
| `GET` | `/scopes/{name}` | Get scope detail |
|
|
| `GET` | `/stats/observations` | Hourly observation time series (last 7 days by default) |
|
|
| `GET` | `/stats/overview` | Network overview stats |
|
|
| `GET` | `/stats/payload-breakdown` | Observation counts by payload type (last 24h by default) |
|
|
| `GET` | `/stats/scopes` | Configured region scopes and breakdown of packets, nodes, observers |
|
|
| `GET` | `/stats/top-advertisers` | Top N nodes by distinct ADVERT packet count (last 24h by default, from materialized view) |
|
|
| `GET` | `/stats/top-nodes` | Top N nodes by observation count (from materialized view) |
|
|
| `GET` | `/stats/top-observers` | Top N observers by observation count (last 24h by default) |
|
|
| `GET` | `/stats/top-talkers` | Top N companion names by decrypted channel message count (last 24h by default, from materialized view) |
|
|
| `GET` | `/traces` | List trace tags with filters (optional: ?type=TRACE\|PING) |
|
|
| `GET` | `/traces/{tag}` | Get full trace detail with resolved routes |
|
|
|
|
---
|
|
|
|
## Acknowledgements
|
|
|
|
See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the people who have helped build
|
|
Beacon.
|
|
|
|
Beacon stands on the shoulders of giants. See [SHOULDERS.md](SHOULDERS.md) for
|
|
the full list of open source projects that make this possible.
|