mirror of
https://github.com/MeshCore-Beacon/beacon-server.git
synced 2026-10-16 07:19:58 +00:00
MrAlders0n
98006a9364
Stats endpoints read hourly rollup tables filled once per closed hour by one advisory-locked task, instead of materialized views that re-scanned every observation on each refresh. IATA-set tables give exact distinct counts for any IATA set or region. Cleanup holds raw rows for unrolled hours (up to 24h). Adds /stats/series; ingest keeps packet observation counts, observer last IATA, channel message counts and trace tag summaries. 001_baseline.sql is rewritten, so pre-release 2.0 databases must be recreated. See docs/historical-stats.md.
599 lines
31 KiB
Markdown
599 lines
31 KiB
Markdown
# MeshCore Beacon
|
||
|
||
MeshCore Beacon is a MeshCore network observation backend. It connects to one or
|
||
more MeshCore MQTT brokers, ingests LoRa packet traffic in real time, stores it
|
||
in PostgreSQL, and streams live events to WebSocket clients.
|
||
|
||
[](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/ci.yml)
|
||
[](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/codeql.yml)
|
||
|
||
[](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/docker-publish.yml)
|
||
|
||
## What it does
|
||
|
||
- Subscribes to MeshCore MQTT brokers and decodes incoming LoRa packets using
|
||
[meshcore-go](https://github.com/meshcore-go/meshcore-go)
|
||
- Stores packets, observations, nodes, observers, traces, routes and channel
|
||
messages in PostgreSQL (more backends to come)
|
||
- Deduplicates observations across multiple brokers (same packet heard by two
|
||
brokers is one observation per observer)
|
||
- Decrypts group text messages for known channel keys
|
||
- Detects firmware capability flags from path hash sizes
|
||
- Streams live events to WebSocket clients with subscription filtering by IATA,
|
||
region, payload type, and event type
|
||
- Serves a REST API for querying stored data
|
||
- Seeds regions, IATA display names, and channel keys from a YAML config file on
|
||
startup
|
||
|
||
For deployment instructions including the frontend app, see the deployment docs.
|
||
|
||
For a bounded private database and saved-config bundle, see
|
||
[backup export](docs/backup-export.md). A standalone export tool is available; the
|
||
protected download API is opt-in; browser login and import remain separate follow-ups.
|
||
|
||
---
|
||
|
||
## Stack
|
||
|
||
| Component | Technology |
|
||
| ------------- | --------------------------------------------------------------- |
|
||
| Language | Go 1.26 |
|
||
| Router | [Chi v5](https://github.com/go-chi/chi) |
|
||
| Database | PostgreSQL 16 |
|
||
| Caching | Redis 7 |
|
||
| DB queries | [sqlc](https://sqlc.dev) + pgx/v5 |
|
||
| MQTT | [paho.mqtt.golang](https://github.com/eclipse/paho.mqtt.golang) |
|
||
| WebSocket | [coder/websocket](https://github.com/coder/websocket) |
|
||
| Packet decode | [meshcore-go](https://github.com/meshcore-go/meshcore-go) |
|
||
| Config | YAML via gopkg.in/yaml.v3 |
|
||
| Env | godotenv |
|
||
|
||
---
|
||
|
||
## Getting started
|
||
|
||
### Prerequisites
|
||
|
||
- Go 1.26+
|
||
- Docker and Docker Compose
|
||
|
||
### 1. Clone and configure
|
||
|
||
```bash
|
||
git clone https://github.com/MeshCore-Beacon/beacon-server.git
|
||
cd beacon-server
|
||
cp env.example .env
|
||
cp config.yaml.example config.yaml
|
||
```
|
||
|
||
Edit `.env` with your broker credentials and database DSN. Edit `config.yaml` to
|
||
define your regions, IATA display names, channel keys, and retention settings.
|
||
|
||
### 2. Start PostgreSQL
|
||
|
||
```bash
|
||
docker compose up postgres -d
|
||
```
|
||
|
||
Database migrations are applied automatically on startup.
|
||
|
||
### 3. Run
|
||
|
||
```bash
|
||
go run ./cmd/beacon
|
||
```
|
||
|
||
Or pull and run the Docker image:
|
||
|
||
```bash
|
||
docker pull ghcr.io/meshcore-beacon/beacon-server:latest
|
||
```
|
||
|
||
The image is public on GitHub Container Registry — no `docker login` required.
|
||
|
||
Beacon will:
|
||
|
||
- Load `.env` and `config.yaml`
|
||
- Connect to PostgreSQL and seed config data
|
||
- Connect to the configured MQTT brokers
|
||
- Start the HTTP server on `LISTEN_ADDR` (default `:8080`)
|
||
|
||
### Cold start and path resolution
|
||
|
||
Path resolution, firmware capability detection, and known route storage all
|
||
depend on nodes having advertised at least once to a local observer. On a fresh
|
||
deployment `resolvedPath` will show `"confidence": "none"` for all hops and
|
||
`supportsMultibytePaths` will be `false` for all nodes until advert traffic
|
||
arrives and populates `node_short_ids`. This is expected behaviour — resolution
|
||
improves automatically as the mesh is observed over time.
|
||
|
||
Similarly, GRP_TXT packets whose channel key isn't yet known at ingest time are
|
||
stored as hash-only, undecrypted rows. Adding the channel's key to `config.yaml`
|
||
doesn't retroactively decrypt that history immediately — it's picked up
|
||
automatically on the next restart, when Beacon scans for undecrypted packets
|
||
matching a now-known channel and decrypts them. Watch the startup log for
|
||
`backfilled N previously-undecrypted channel message(s)`.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
### Admin authentication
|
||
|
||
The `/api/v1/admin` subtree requires `Authorization: Bearer <key>`. Set the
|
||
operator key with `BEACON_API_KEY` or `auth.api_key` in YAML. A set environment
|
||
variable overrides YAML; an explicitly empty value disables admin access.
|
||
With no key, admin requests return JSON 503 while public reads and WebSockets
|
||
continue normally. With a key, missing, incorrect or duplicate Authorization
|
||
headers return JSON 401 with `WWW-Authenticate: Bearer`.
|
||
|
||
`GET /api/v1/admin/config` returns selected running settings: CORS options with
|
||
Beacon defaults applied, `auth.configured`, and `ingest.broker_count` (configured
|
||
broker workers, not connection status or a tunable processing-worker pool).
|
||
The CORS lists are the options supplied to the middleware; its normal matching
|
||
normalization still applies. The response excludes
|
||
credential fields, broker addresses, channel material, database settings and
|
||
other configuration. Unknown admin paths return 404 and unsupported
|
||
methods on the config endpoint return 405 after authentication.
|
||
Global CORS preflights remain public. Use a long, randomly generated key, keep
|
||
it out of source control and logs, and send it only in the Authorization header,
|
||
never the URL or request body. Require HTTPS at the reverse proxy and restrict
|
||
direct access to Beacon's HTTP listener to that proxy or a private connection.
|
||
Changing the key requires a restart. No API key is issued automatically.
|
||
|
||
`PUT /api/v1/admin/config` accepts only
|
||
`{"cors":{"allowed_origins":["https://example.org"]}}`. It replaces the entire
|
||
origin list immediately and updates the reported configuration with the same
|
||
policy. Requests already in progress may use the previous policy. Concurrent
|
||
valid updates are serialized; updates take effect one at a time. The response
|
||
contains `config`, `persisted: false` and `requires_restart: false`.
|
||
|
||
Updates are **runtime-only**: no file or database is written, and restarting
|
||
reloads the saved configuration. Keep 1–32 ASCII HTTP(S) origins, at most 512 bytes
|
||
each, with an optional single hostname wildcard; a sole `*` permits all origins.
|
||
Empty/null lists, URL paths/queries/credentials, control characters and unknown
|
||
fields are rejected. Requests must be JSON, at most 16 KiB. Other CORS options,
|
||
auth/credential fields and broker count cannot be changed here; there is no
|
||
configurable `ingest.worker_count`. Cross-origin admin clients need PUT allowed
|
||
in the saved CORS methods. CORS controls browser access, not authentication.
|
||
|
||
Operator accounts are available at `GET/POST /api/v1/admin/accounts` and
|
||
`GET/DELETE /api/v1/admin/accounts/{id}`. POST accepts a JSON `name` field in a
|
||
body up to 4 KiB; names are trimmed, case-sensitive and limited to 128 Unicode
|
||
characters without control characters. Active names are unique. DELETE soft
|
||
deactivates the record (204); missing IDs return 404 and an already inactive
|
||
record returns 409. A deactivated name may be reused by a new account.
|
||
Lists include active and inactive records, newest first, without pagination.
|
||
These are operator-defined records; no login, session or API token is created.
|
||
Cross-origin account clients need both `POST` and `DELETE` in the saved
|
||
`cors.allowed_methods`; the default `GET, HEAD, OPTIONS` is read-only. For an
|
||
admin UI that also updates configuration, use `[GET, HEAD, OPTIONS, POST, PUT,
|
||
DELETE]`, restrict `cors.allowed_origins` to that UI, and allow `Authorization`
|
||
and `Content-Type` headers. Otherwise browser preflight blocks these requests
|
||
even when the same bearer-authenticated request works with curl.
|
||
|
||
### Environment variables (`.env`)
|
||
|
||
| Variable | Default | Description |
|
||
| ------------------------ | ------------- | ------------------------------------------------------------ |
|
||
| `LISTEN_ADDR` | `:8080` | HTTP listen address |
|
||
| `POSTGRES_DSN` | — | PostgreSQL connection string |
|
||
| `REDIS_ADDR` | — | Redis address (`host:port`). Leave unset to disable caching. |
|
||
| `REDIS_PASSWORD` | — | Redis password (optional) |
|
||
| `REDIS_DB` | `0` | Redis database index |
|
||
| `CONFIG_PATH` | `config.yaml` | Path to YAML config file |
|
||
| `MQTT_BROKER_1_URL` | — | Broker 1 WebSocket URL (e.g. `wss://mqtt1.example.com:443`) |
|
||
| `MQTT_BROKER_1_USERNAME` | — | Broker 1 username |
|
||
| `MQTT_BROKER_1_PASSWORD` | — | Broker 1 password |
|
||
| `MQTT_BROKER_2_URL` | — | Broker 2 WebSocket URL |
|
||
| `MQTT_BROKER_2_USERNAME` | — | Broker 2 username |
|
||
| `MQTT_BROKER_2_PASSWORD` | — | Broker 2 password |
|
||
|
||
### Config file (`config.yaml`)
|
||
|
||
```yaml
|
||
# MeshMapper integrations (optional, no API key). Import transport scopes and
|
||
# IATA border outlines from MeshMapper instead of maintaining scopes: and
|
||
# iatas.*.borderFile by hand. Both cover every known IATA that MeshMapper lists;
|
||
# no sources to configure. See config.yaml.example for limits and behaviour.
|
||
#meshmapper:
|
||
# scopes:
|
||
# enabled: true
|
||
# zones:
|
||
# enabled: true
|
||
# import_groups: true # MeshMapper zone groups become regions
|
||
# channels:
|
||
# enabled: true # import each region's public hashtag channels
|
||
|
||
# Optional IATA overrides — auto-created on first packet arrival,
|
||
# only needed if you want to customise display name or coordinates.
|
||
# borderFile points to a GeoJSON Feature (Polygon or MultiPolygon) for the
|
||
# region border map feature; relative paths resolve against this config
|
||
# file's own directory. Validated at boot — invalid geometry fails startup.
|
||
iatas:
|
||
YVR:
|
||
name: Vancouver International
|
||
lat: 49.1967
|
||
lng: -123.1815
|
||
borderFile: borders/yvr.geojson # optional; or use meshmapper.zones
|
||
|
||
# Super-regions grouping multiple IATAs.
|
||
regions:
|
||
- slug: western-canada
|
||
name: Western Canada
|
||
display_order: 1
|
||
center_lat: 51.0
|
||
center_lng: -114.0
|
||
zoom_level: 5
|
||
iatas: [YVR, YYJ, YYC, YEG]
|
||
|
||
# Channel keys for decrypting group messages.
|
||
channel_keys:
|
||
# Hashtag channels: Beacon derives the PSK from the tag name automatically.
|
||
# secret = SHA256("#tag")[:16], channel_hash = SHA256(secret)[0]
|
||
# Tag names should be provided without the # prefix. Plain names show under
|
||
# every region; {name, region: <slug>} limits a channel to one region.
|
||
hashtags:
|
||
- meshcore
|
||
- name: vancouver-mesh
|
||
region: western-canada
|
||
|
||
# Explicit keys: channel hash (hex) and key (hex), with optional display name.
|
||
# The public MeshCore channel key is included in config.yaml.example.
|
||
keys:
|
||
"11":
|
||
key: "8b3387e9c5cdea6ac9e5edbaa115cd72"
|
||
name: "Public"
|
||
|
||
# Regional transport scopes for matching TRANSPORT_FLOOD packets.
|
||
# Plain names have # prepended automatically (e.g. "bc" → "#bc").
|
||
# region (required) is a configured region slug; region-filtered scope lists
|
||
# and scope stats show the scope under that region's IATAs. Matching stays global.
|
||
# Optional when meshmapper.scopes covers your regions.
|
||
scopes:
|
||
- name: bc
|
||
region: western-canada
|
||
- name: "#west"
|
||
region: western-canada
|
||
|
||
# Observer telemetry storage settings.
|
||
telemetry:
|
||
retention: 744h # how long to keep telemetry snapshots (default: 31 days)
|
||
resolution: 1h # snapshot frequency per observer; duplicates within window are dropped (default: 1h)
|
||
|
||
# Packet and observation retention.
|
||
packets:
|
||
retention: 168h # packets, observations, channel messages (default: 7 days, minimum 24h)
|
||
|
||
# Hourly analytics rollups behind the stats endpoints; they outlive raw packets.
|
||
analytics:
|
||
rollup_retention: 2160h # also the longest /stats/series window (default: 90 days, minimum 24h)
|
||
|
||
# Presence write coalescing.
|
||
# Observer last_seen and packet last_heard_at bumps are batched in memory and
|
||
# flushed on an interval instead of writing one row per observation. An
|
||
# unclean shutdown loses at most one interval of presence freshness.
|
||
presence:
|
||
flush_interval: 30s # how often coalesced bumps are flushed (default: 30s)
|
||
packet_ttl: 30s # how long a quiet packet stays coalesced before writing through again (default: 30s)
|
||
|
||
# WebSocket settings.
|
||
websocket:
|
||
max_connections_per_ip: 5 # default: 5
|
||
|
||
# Node staleness, deletion, and clock-drift thresholds.
|
||
nodes:
|
||
mark_foreign: false # optional indication for repeaters outside configured IATA borders
|
||
stale_threshold: 24h # mark a node "stale" in the API after this long unseen (default: 24h)
|
||
delete_after: 720h # delete a node entirely after this long unseen (default: 30 days)
|
||
clock_drift_threshold: 5m # |device clock - server clock| above which clockOutOfSync=true for a repeater/room server (default: 5m)
|
||
|
||
# Optional observer age-out (disabled by default; set e.g. 720h to opt in).
|
||
# Enable only when one Beacon ingest process owns the database: presence-cache
|
||
# preparation is local to that process, including both of its MQTT workers.
|
||
# Deletes at most 1000 observers per background.cleanup interval, only when
|
||
# last_seen/last_status_at are old and no observations, telemetry or ownership remain.
|
||
# Broker/location/scope metadata cascades; a returning observer receives a new ID.
|
||
observers:
|
||
delete_after: 0s # omitted or nonpositive disables deletion
|
||
|
||
# Redis caching layer (optional).
|
||
# Caches read-heavy, slow-changing responses to reduce PostgreSQL load.
|
||
# Connection details (address, password, database) are set via environment
|
||
# variables. Leave REDIS_ADDR unset to disable caching entirely.
|
||
# TTLs are duration strings e.g. "30m", "1h". Per-category TTLs override
|
||
# the global ttl. Any unset category inherits ttl. Default: 1h.
|
||
cache:
|
||
ttl: "1h"
|
||
ttls:
|
||
stats: "1h" # stats endpoints (hourly rollups; keys change when a new hour is rolled)
|
||
reference: "1h" # IATAs, regions, scopes
|
||
nodes: "1h" # node detail (also explicitly invalidated on upsert)
|
||
observers: "1h" # observer detail (also explicitly invalidated on upsert)
|
||
|
||
# Geographic ingest filter (optional).
|
||
# Drop packets from observers outside the specified area.
|
||
# Country codes are ISO 3166-1 alpha-2. Continent codes: AF AN AS EU NA OC SA.
|
||
# If both are set an IATA passes if it matches either (OR semantics).
|
||
# Omit entirely to accept all IATAs (default).
|
||
ingest:
|
||
allow_countries: [CA, US] # only store packets from these countries
|
||
allow_continents: [NA] # or: accept all of North America
|
||
```
|
||
|
||
IATAs are auto-created on first packet arrival. The config file adds display
|
||
names, coordinates, and optional region borders. Regions and channel keys
|
||
must be defined here — they are not auto-created.
|
||
|
||
---
|
||
|
||
### Foreign repeater indication
|
||
|
||
Set `nodes.mark_foreign: true` to expose `possiblyForeign` on repeater nodes.
|
||
The local operating area is the union of **all configured**
|
||
`iatas.<code>.borderFile` GeoJSON Polygon/MultiPolygon features. IATAs without
|
||
border files do not add an area; airport coordinates and the current API region
|
||
filter are not boundaries. Enabling this with no borders, missing files or
|
||
invalid geometry fails startup. Border changes require a restart.
|
||
|
||
Inside any polygon (including its edges) means `false`; outside the entire union
|
||
means `true`. Hole interiors are outside; hole edges are local. Other node roles,
|
||
missing or invalid positions have no classification (an explicit 0/0 advert
|
||
clears the stored position).
|
||
This is a hint based on reported position, not proof of a repeater's origin.
|
||
Packet ingestion, heard-in IATAs and route matching remain unchanged.
|
||
|
||
Node list/detail reads apply the current geometry after cache reads, so existing
|
||
historical nodes need no backfill. A `nodeUpdate` includes `possiblyForeign`
|
||
when its advert provides a position: a boolean for known positions, `null` to
|
||
clear it when the advert reports 0/0 or an invalid position. Omission retains the
|
||
previous value when a repeater's advert omits its position. A change to another
|
||
role also sends `null`. `lat`/`lng` follow the same rule: omitted keeps the
|
||
client's position, explicit `null` clears it.
|
||
The field is omitted everywhere when the feature is disabled (the default).
|
||
|
||
Use longitude/latitude coordinate order and split antimeridian-crossing borders
|
||
into MultiPolygons as described in [RFC 7946 section 3.1.9](https://www.rfc-editor.org/rfc/rfc7946#section-3.1.9).
|
||
Classification rejects unsplit edges spanning more than 180 degrees rather than
|
||
silently treating them as the complementary global area.
|
||
|
||
## Authentication
|
||
|
||
API authentication is not yet implemented. Beacon is intended for trusted
|
||
internal network or reverse-proxy deployments. Do not expose it directly to the
|
||
public internet without an authentication layer in front of it.
|
||
|
||
---
|
||
|
||
## WebSocket API
|
||
|
||
Connect to `ws://host:8080/ws`.
|
||
|
||
On connect the server sends a `hello`:
|
||
|
||
```json
|
||
{ "v": 1, "type": "hello", "serverTime": 1234567890000, "connectionId": "uuid" }
|
||
```
|
||
|
||
The connection closes after 90 seconds of inactivity. Clients should send a
|
||
`ping` every 30 seconds.
|
||
|
||
### Client → Server messages
|
||
|
||
**Subscribe** — add a filter to this connection. Multiple subscriptions are
|
||
unioned (OR semantics): an event matches if it satisfies any active
|
||
subscription. The server replies with a `subscriptionId` to use for
|
||
unsubscribing.
|
||
|
||
```json
|
||
{
|
||
"v": 1,
|
||
"type": "subscribe",
|
||
"id": "sub-1",
|
||
"scope": {
|
||
"iatas": ["YOW", "YYZ"],
|
||
"regionIds": ["1"],
|
||
"regionSlugs": ["western-canada"],
|
||
"payloadTypes": [4, 5],
|
||
"channelHashes": ["11"],
|
||
"events": ["packetObservation", "channelMessage"]
|
||
}
|
||
}
|
||
```
|
||
|
||
All scope fields are optional. Omitted means no filter on that dimension (match
|
||
everything). Empty array means match nothing on that dimension. `regionIds` and
|
||
`regionSlugs` are both expanded to their member IATAs server-side.
|
||
|
||
**Unsubscribe** — remove a specific subscription by ID.
|
||
|
||
```json
|
||
{
|
||
"v": 1,
|
||
"type": "unsubscribe",
|
||
"id": "unsub-1",
|
||
"subscriptionId": "<uuid from subscribed reply>"
|
||
}
|
||
```
|
||
|
||
**Configure** — toggle connection-wide settings. Currently just `resolvePath`,
|
||
which enables per-hop node resolution on `packetObservation` events (see
|
||
below). Unlike `subscribe`, this is a single flag for the whole connection,
|
||
not additive across calls — each `configure` sets it to exactly the value
|
||
sent, and it can be flipped on or off as many times as you like for the life
|
||
of the connection. Default is `false`.
|
||
|
||
```json
|
||
{ "v": 1, "type": "configure", "id": "cfg-1", "resolvePath": true }
|
||
```
|
||
|
||
The server replies:
|
||
|
||
```json
|
||
{ "v": 1, "type": "configured", "id": "cfg-1", "resolvePath": true }
|
||
```
|
||
|
||
**Ping**
|
||
|
||
```json
|
||
{ "v": 1, "type": "ping", "id": "ping-1" }
|
||
```
|
||
|
||
### Server → Client events
|
||
|
||
| Type | Description |
|
||
| ------------------- | --------------------------------------------------- |
|
||
| `packetObservation` | New observation written to DB |
|
||
| `observerStatus` | Observer status update |
|
||
| `nodeUpdate` | Node upserted from advert |
|
||
| `channelMessage` | Decrypted channel message (scope must include hash) |
|
||
|
||
When `resolvePath` is enabled via `configure`, `packetObservation` events
|
||
include a `resolvedPath` array: one entry per hop in the packet's path, each
|
||
with a `confidence` (`"high"` for exactly one candidate node, `"ambiguous"`
|
||
for multiple, `"none"` for zero) and the matching node(s)' id, name,
|
||
lat/lng, and public key. Without `resolvePath` enabled, `resolvedPath` is
|
||
`null` — this keeps the default event payload small; only opt in if you're
|
||
actually rendering path connections (e.g. drawing hops on a map).
|
||
|
||
### Backpressure
|
||
|
||
The server write buffer per connection is bounded at 256 events. If a client
|
||
falls behind, the server drops the oldest queued events and sends a `lagged`
|
||
notice:
|
||
|
||
```json
|
||
{ "v": 1, "type": "lagged", "droppedCount": 12, "since": 1234567890000 }
|
||
```
|
||
|
||
Clients should respond by re-fetching the relevant REST endpoint using `afterId`
|
||
to backfill missed events, then resume streaming.
|
||
|
||
### Reconnection
|
||
|
||
Subscriptions are not persisted — they exist only for the lifetime of the
|
||
connection. On any disconnect the client should reconnect with backoff, re-issue
|
||
all subscriptions, and backfill via REST using
|
||
`afterId=<last seen observation id>`.
|
||
|
||
### Connection limits
|
||
|
||
By default a maximum of 5 concurrent WebSocket connections are allowed per IP
|
||
address. Connections beyond this limit receive `HTTP 429`. The limit is
|
||
configurable via `websocket.max_connections_per_ip` in `config.yaml`.
|
||
|
||
---
|
||
|
||
## REST API
|
||
|
||
Base path: `/api/v1`
|
||
|
||
All list endpoints support cursor-based pagination via `cursor` and `limit`
|
||
query params. See the Swagger UI at `http://localhost:8080/swagger/index.html`
|
||
for full parameter documentation.
|
||
|
||
### Authentication
|
||
|
||
Not yet implemented — see the Authentication section above.
|
||
|
||
### Endpoints
|
||
|
||
| Method | Path | Description |
|
||
| ------ | ----------------------------------- | -------------------------------------------------------------------------------------------------- |
|
||
| `GET` | `/brokers` | List MQTT brokers and connection status |
|
||
| `GET` | `/channels` | List channels (optional: `?hash=<hex>&iata=<code>&limit=50`) |
|
||
| `GET` | `/channels/{id}` | Get channel detail by integer ID |
|
||
| `GET` | `/channels/{id}/messages` | List messages for a channel (optional: `?since=<ms>&iata=<code>&limit=50`) |
|
||
| `GET` | `/iatas` | List all known IATA codes |
|
||
| `GET` | `/iatas/{iata}` | Get a single IATA code |
|
||
| `GET` | `/iatas/{iata}/border` | Get an IATA's GeoJSON region border, if configured (204 if not) |
|
||
| `GET` | `/messages` | List all messages (optional: `?channelId=<int>&channelHash=<hex>&iata=<code>&since=<ms>&limit=50`) |
|
||
| `GET` | `/messages/backfill` | Backfill messages after a given message ID |
|
||
| `GET` | `/nodes` | List nodes (optional: `?pubkeyPrefix=<hex>&neighbors&iatas=<codes>&pubkey=<hex>`) |
|
||
| `GET` | `/nodes/{nodeId}` | Get node detail |
|
||
| `GET` | `/nodes/{nodeId}/neighbors` | List neighboring nodes observed in the mesh |
|
||
| `GET` | `/nodes/{nodeId}/observations` | List observations for a node |
|
||
| `GET` | `/observers` | List observers (optional: `?iata=<code>&type=<str>&broker=<name>&status=online\|offline`) |
|
||
| `GET` | `/observers/{observerId}` | Get observer detail including broker last-seen timestamps |
|
||
| `GET` | `/observers/{observerId}/adverts` | Adverts heard by observer |
|
||
| `GET` | `/observers/{observerId}/telemetry` | Observer telemetry history (optional: `?range=24h&interval=1h\|6h\|24h`) |
|
||
| `GET` | `/packets` | List packets (optional: `?payloadTypes=<csv>&routeTypes=<csv>&scopes=<csv>` accept plural, comma-separated values alongside the singular params) |
|
||
| `GET` | `/packets/backfill` | Backfill packets after a given observation ID |
|
||
| `GET` | `/packets/{packetHash}` | Get packet with all observations |
|
||
| `GET` | `/regions` | List all regions (summary) |
|
||
| `GET` | `/regions/{id}` | Get a single region with IATA list |
|
||
| `GET` | `/routes` | List known routes (all hops high confidence) |
|
||
| `GET` | `/routes/search` | Search routes by source and destination hash |
|
||
| `GET` | `/routes/cross` | Search for routes crossing IATA boundaries |
|
||
| `GET` | `/scopes` | List transport scope names; IATA/region filters use configured regions and MeshMapper catalogues |
|
||
| `GET` | `/scopes/{name}` | Get scope detail |
|
||
| `GET` | `/stats/observations` | Hourly observation counts per IATA (last 7 days by default) |
|
||
| `GET` | `/stats/overview` | Network overview over the last 24 rolled hours |
|
||
| `GET` | `/stats/payload-breakdown` | Observation counts by payload type (last 24h by default) |
|
||
| `GET` | `/stats/scopes` | Configured region scopes and breakdown of packets, nodes, observers (last 7 days by default) |
|
||
| `GET` | `/stats/series` | Hourly card metrics and window summary from the rollups (`since`/`until` required) |
|
||
| `GET` | `/stats/top-advertisers` | Top N nodes by distinct ADVERT packet count (last 24h by default) |
|
||
| `GET` | `/stats/top-nodes` | Top N nodes by advert hearings (last 7 days by default) |
|
||
| `GET` | `/stats/top-observers` | Top N observers by observation count (last 24h by default) |
|
||
| `GET` | `/stats/top-talkers` | Top N companion names by decrypted channel message count (last 24h by default) |
|
||
| `GET` | `/traces` | List trace tags with filters (optional: ?type=TRACE\|PING) |
|
||
| `GET` | `/traces/{tag}` | Get full trace detail with resolved routes |
|
||
|
||
---
|
||
|
||
## Acknowledgements
|
||
|
||
See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the people who have helped build
|
||
Beacon.
|
||
|
||
Beacon stands on the shoulders of giants. See [SHOULDERS.md](SHOULDERS.md) for
|
||
the full list of open source projects that make this possible.
|
||
|
||
## Application logging
|
||
|
||
Beacon writes application logs to stderr. Configure the minimum level and output format in `config.yaml`:
|
||
|
||
```yaml
|
||
log:
|
||
level: info # debug, info, warn, error
|
||
format: text # text or json
|
||
```
|
||
|
||
`LOG_LEVEL` and `LOG_FORMAT` override file settings; empty settings use `info` and `text`. Invalid values prevent startup. Configuration-loading failures can use the bootstrap text logger before file settings are available; failures after initialization retain error severity at every supported level.
|
||
|
||
Records include a component field. Ingest workers also include their broker name, and HTTP completion records include the validated client address, route, status and duration. Query strings and protocol hello payloads are excluded. Expected ingest skips and routine WebSocket lifecycle details are debug-level. Changing the application's format does not change Caddy/Apache access logs or their fail2ban configuration. Collect/rotate stderr through Docker or systemd.
|
||
|
||
### Analytics retention
|
||
|
||
Historical stats (traffic, payload, observer activity, talker, advertiser, node,
|
||
scope, Signal and Paths) are read from hourly rollup tables kept for
|
||
`analytics.rollup_retention` (default 90 days), independently of `packets.retention`.
|
||
One background task rolls each UTC hour about 95 minutes after it closes;
|
||
packet cleanup waits for unrolled hours (up to 24h) so they can be rolled in full.
|
||
An hour whose raw rows were deleted first is marked partial and reported without values.
|
||
Packet drill-down, sub-hour observer activity and observer comparison still read
|
||
retained raw data. See [docs/historical-stats.md](docs/historical-stats.md).
|
||
|
||
### Observer monitoring metrics
|
||
|
||
Observer activity returns buckets in `[windowStart, windowEnd)`, including the current partial bucket for live requests, plus
|
||
`generatedAt`, `source` (`raw` or `hourly`) and `summary`. `recordedPackets` is the
|
||
sum of stored observations in those buckets; repeated broker delivery of the
|
||
same retained packet/observer pair counts once. Unknown payload types appear as
|
||
`-1` rather than disappearing. Freshness fields are measured at `generatedAt` even for historical `until` requests; `recordedPackets` alone follows the selected window. `lastCompleteHour` uses the previous complete UTC
|
||
hour and includes its own start/end; `latestRecordedAt` is the latest retained
|
||
reception timestamp. Missing records do not prove downtime. The optional `until`
|
||
(epoch milliseconds, within the last 30 days) aligns two observers' charts.
|
||
|
||
The existing observer `observationCount` remains a legacy cumulative presence
|
||
counter for compatibility, including status/neighbour events. It is not a
|
||
period packet total. Broker presence and packet-arrival timestamps are now
|
||
updated separately; this cannot reconstruct previously overwritten timestamps.
|
||
Hourly activity comes from the analytics rollups and includes unknown-type activity.
|
||
|
||
### Saved-route evidence
|
||
|
||
`GET /api/v1/routes/{iata}/{pathKey}/observations` returns retained observations that match a saved
|
||
route's full path bytes, hash size and hop count within its IATA. `range` defaults to `24h` (max `720h`),
|
||
`limit` to 50 (max 200); page with `nextPageCursor` → `pageCursor`. Evidence expires with raw packets.
|