mirror of
https://github.com/MeshCore-Beacon/beacon-server.git
synced 2026-10-20 14:32:54 +00:00
Merge remote-tracking branch 'origin/main' into codex/post20-audit-server-20261003
This commit is contained in:
n30nex
@@ -34,5 +34,5 @@ paste logs here
|
||||
|
||||
## Additional context
|
||||
|
||||
Any other context — config snippets (redact credentials), MQTT broker details,
|
||||
Any other context: config snippets (redact credentials), MQTT broker details,
|
||||
mesh topology, etc.
|
||||
|
||||
@@ -15,7 +15,7 @@ related to an existing issue or limitation, link it here.
|
||||
|
||||
## Proposed solution
|
||||
|
||||
How you'd like it to work. API shape, config options, WS event format — as much
|
||||
How you'd like it to work. API shape, config options, WS event format, as much
|
||||
detail as you have.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
@@ -23,7 +23,7 @@ issues: `Closes #123` or `Related to #456`.
|
||||
- [ ] API changes include swagger annotations and `swag init` has been run
|
||||
- [ ] `docs/` is committed if swagger was regenerated
|
||||
- [ ] New dependencies are added to `SHOULDERS.md`
|
||||
- [ ] I have read [CONTRIBUTING.md](CONTRIBUTING.md)
|
||||
- [ ] I have read [CONTRIBUTING.md](../CONTRIBUTING.md)
|
||||
|
||||
## Testing notes
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ jobs:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
labels: |
|
||||
org.opencontainers.image.title=beacon-server
|
||||
org.opencontainers.image.description=Beacon — MeshCore LoRa mesh observation server
|
||||
org.opencontainers.image.description=Beacon: MeshCore LoRa mesh observation server
|
||||
org.opencontainers.image.licenses=AGPL-3.0-or-later
|
||||
flavor: |
|
||||
latest=false
|
||||
|
||||
+5
-5
@@ -25,7 +25,7 @@ Examples of behaviour that contributes to a positive environment:
|
||||
|
||||
Examples of unacceptable behaviour:
|
||||
|
||||
- Hate speech or discriminatory language or jokes of any kind — including but
|
||||
- Hate speech or discriminatory language or jokes of any kind, including but
|
||||
not limited to those targeting race, ethnicity, nationality, gender identity
|
||||
or expression, sexual orientation, disability, religion, age, or socioeconomic
|
||||
background
|
||||
@@ -34,7 +34,7 @@ Examples of unacceptable behaviour:
|
||||
- Trolling, insulting or derogatory comments, and personal or political attacks
|
||||
- Public or private harassment of any kind
|
||||
- Deliberate misgendering or use of rejected names
|
||||
- Publishing others' private information — such as a physical or email address —
|
||||
- Publishing others' private information, such as a physical or email address,
|
||||
without their explicit permission
|
||||
- Threats of violence or incitement of violence toward any individual or group
|
||||
- Dismissing or minimising reports of harassment or discrimination
|
||||
@@ -55,15 +55,15 @@ when appropriate.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all project spaces — GitHub issues, pull
|
||||
requests, discussions, and the MeshCore Canada Discord server — and also when an
|
||||
This Code of Conduct applies within all project spaces (GitHub issues, pull
|
||||
requests, discussions, and the MeshCore Canada Discord server) and also when an
|
||||
individual is representing the project in public spaces.
|
||||
|
||||
## Reporting
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behaviour may be
|
||||
reported by contacting the maintainers directly on the MeshCore Canada Discord
|
||||
server at https://discord.gg/Gz3KvJx2hf — reach out to **dedskelly** privately.
|
||||
server at https://discord.gg/Gz3KvJx2hf; reach out to **dedskelly** privately.
|
||||
|
||||
All reports will be handled with discretion and confidentiality. Maintainers are
|
||||
obligated to respect the privacy and safety of the reporter.
|
||||
|
||||
+68
-126
@@ -6,40 +6,16 @@ before opening a PR.
|
||||
|
||||
---
|
||||
|
||||
## Before you start
|
||||
|
||||
**Open or comment on an issue before starting work.** This avoids duplicate
|
||||
effort and lets maintainers flag if something is already in progress or out of
|
||||
scope. For small bug fixes a brief comment is fine; for larger features please
|
||||
discuss the approach first.
|
||||
|
||||
**One thing per PR.** Each pull request should cover one logical change — a bug
|
||||
fix, a new endpoint, a refactor, a new test. PRs that touch many unrelated parts
|
||||
of the codebase are hard to review and hard to revert if something goes wrong.
|
||||
|
||||
**No fully AI-generated contributions.** We welcome developers who use AI tools
|
||||
to assist their work, but PRs should reflect the author's own understanding and
|
||||
judgement. PRs that appear to be unreviewed AI output may be closed without
|
||||
further comment.
|
||||
|
||||
---
|
||||
|
||||
## Branches
|
||||
|
||||
- `main` — stable releases only, protected. Never target this directly.
|
||||
- `dev` — active development. All PRs target `dev`.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Fork or create a branch from `dev`
|
||||
2. Make your changes
|
||||
3. Run the checklist below
|
||||
4. Open a pull request against `dev` with a clear description of what changed
|
||||
and why, referencing any related issues
|
||||
Branches, the pull request flow, commit message format and the release process are the same
|
||||
for every Beacon repo and live in
|
||||
[beacon-docs/CONTRIBUTING.md](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/CONTRIBUTING.md).
|
||||
In short: open or comment on an issue first, branch from `dev`, keep each PR to one change,
|
||||
write conventional commit messages, and open the PR against `dev`. Running beacon-web against
|
||||
a local server is described there too.
|
||||
|
||||
---
|
||||
|
||||
## Checklist before opening a PR
|
||||
Before opening a PR here:
|
||||
|
||||
```
|
||||
go build ./... # must compile
|
||||
@@ -55,8 +31,6 @@ swag init -g cmd/beacon/main.go -o docs --parseInternal --parseDependency
|
||||
|
||||
## Code style
|
||||
|
||||
- Run `gofmt -w .` before committing — CI will fail on unformatted files
|
||||
- Run `go vet ./...` — no warnings
|
||||
- Follow the existing patterns in each package before introducing new ones
|
||||
- Keep functions small and single-purpose
|
||||
- Prefer explicit error handling over panic
|
||||
@@ -69,20 +43,39 @@ swag init -g cmd/beacon/main.go -o docs --parseInternal --parseDependency
|
||||
no side effects) should have unit tests. See `internal/hub/hub_test.go`,
|
||||
`internal/api/nodes_test.go`, and `internal/keystore/keystore_test.go` for
|
||||
examples of the style we use.
|
||||
- Integration tests (requiring a real DB) are not yet required but are welcome.
|
||||
They will be gated before release.
|
||||
- Run `go test ./...` before opening a PR. All tests must pass.
|
||||
- Integration tests (`*_integration_test.go`) need a real PostgreSQL 16 and skip
|
||||
unless `BEACON_TEST_POSTGRES_DSN` is set. CI runs a selected set; run the ones
|
||||
touching your change locally:
|
||||
`BEACON_TEST_POSTGRES_DSN=postgres://... go test ./db/ -run Postgres`
|
||||
- If you are fixing a bug, add a test that would have caught it.
|
||||
|
||||
---
|
||||
|
||||
## Backup export tests
|
||||
|
||||
CI runs export, restore round-trip, offline verify and truncated-archive rejection against
|
||||
PostgreSQL 16. Locally, with `PG*` pointing at a throwaway server whose role can create
|
||||
databases:
|
||||
|
||||
```sh
|
||||
go build -o "$PWD/beacon-backup" ./cmd/beacon-backup
|
||||
BEACON_BACKUP_TEST_POSTGRES=1 BEACON_BACKUP_TEST_BINARY="$PWD/beacon-backup" \
|
||||
go test ./internal/backup -run '^TestExportPostgres$' -v
|
||||
```
|
||||
|
||||
The test creates, migrates, restores and drops two randomly named databases. The export
|
||||
format and operator usage are documented in
|
||||
[Backup and export](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/backup-export.md).
|
||||
|
||||
---
|
||||
|
||||
## Database changes
|
||||
|
||||
All schema changes must include a proper migration path:
|
||||
|
||||
- Add a new migration file to `db/migrations/` following the existing naming
|
||||
convention (e.g. `002_add_observation_count.sql`). `001_baseline.sql` is the
|
||||
2.0.0 schema; do not modify existing migration files — append only via new files.
|
||||
2.0.0 schema; do not modify existing migration files; append only via new files.
|
||||
- Update `db/queries/queries.sql` with any new or modified queries
|
||||
- Re-run `sqlc generate` to regenerate `db/sqlc/`
|
||||
- Update the store layer in `db/` to expose the new functionality
|
||||
@@ -90,7 +83,7 @@ All schema changes must include a proper migration path:
|
||||
- Historical stats read hourly rollups, never raw observations; follow
|
||||
[docs/historical-stats.md](docs/historical-stats.md) to add a metric
|
||||
|
||||
Never edit files under `db/sqlc/` by hand — they are generated by sqlc and will
|
||||
Never edit files under `db/sqlc/` by hand; they are generated by sqlc and will
|
||||
be overwritten. If you need to work around a sqlc limitation, document it
|
||||
clearly in the query comment.
|
||||
|
||||
@@ -118,23 +111,18 @@ that boundary. `/config` remains owned by the shared admin router. No global
|
||||
registration or `init` side effects are needed.
|
||||
|
||||
When concurrent PRs touch the same startup wiring, stack them in an explicit
|
||||
merge order and link each feature-only comparison. Rebase the stack together
|
||||
after prerequisite merges, regenerate Swagger rather than hand-merging it, and
|
||||
compile the combined result even when GitHub reports it as mergeable.
|
||||
merge order, regenerate Swagger rather than hand-merging it, and build the
|
||||
combined result even when GitHub says it merges cleanly.
|
||||
|
||||
Any new or modified REST endpoint must have swagger annotations and regenerated
|
||||
docs:
|
||||
|
||||
- Add or update `// @Summary`, `// @Param`, `// @Success`, `// @Failure`, and
|
||||
`// @Router` comments on the handler function
|
||||
- Response types are defined in `internal/api/` — add new types there, not
|
||||
- Response types are defined in `internal/api/`; add new types there, not
|
||||
inline in handlers
|
||||
- After changing any handler or API type, regenerate the swagger docs and commit
|
||||
the updated `docs/` directory alongside your changes:
|
||||
|
||||
```bash
|
||||
swag init -g cmd/beacon/main.go -o docs --parseInternal --parseDependency
|
||||
```
|
||||
- After changing any handler or API type, rerun the `swag init` command from the
|
||||
checklist and commit the updated `docs/` directory
|
||||
|
||||
Install swag if you don't have it:
|
||||
|
||||
@@ -220,44 +208,37 @@ AIRPORTS_CSV=/path/to/airports.csv go run ./internal/iatadb/gen
|
||||
|
||||
---
|
||||
|
||||
## Commit messages
|
||||
|
||||
Use the conventional commits format:
|
||||
|
||||
```
|
||||
feat(routes): add observation_count to known routes response
|
||||
fix(ingest): correct lat/lon divisor for advert payloads
|
||||
refactor(hub): collapse RegionIATAs into IATAs in Scope
|
||||
test(api): add unit tests for NodeTypeName and NodeTypeFromString
|
||||
chore: update airports.csv
|
||||
docs: expand CONTRIBUTING.md
|
||||
```
|
||||
|
||||
Scopes are optional but helpful for larger codebases. Common scopes: `api`,
|
||||
`db`, `ingest`, `hub`, `ws`, `handlers`, `config`, `keystore`.
|
||||
|
||||
---
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
cmd/beacon/ — main entry point, wiring, startup
|
||||
db/ — store layer: sqlc-generated code + thin mapping layer
|
||||
migrations/ — SQL schema (2.0.0 baseline + numbered files, append only)
|
||||
queries/ — SQL queries (input to sqlc)
|
||||
sqlc/ — generated Go code (do not edit by hand)
|
||||
cmd/beacon/ main entry point, wiring, startup
|
||||
cmd/beacon-backup/ standalone backup export/verify command
|
||||
db/ store layer: sqlc-generated code + thin mapping layer
|
||||
migrations/ SQL schema (2.0.0 baseline + numbered files, append only)
|
||||
queries/ SQL queries (input to sqlc)
|
||||
sqlc/ generated Go code (do not edit by hand)
|
||||
internal/
|
||||
api/ — response types and Reader interface
|
||||
handlers/ — HTTP handlers (validation, routing, response)
|
||||
router/ — chi router wiring
|
||||
config/ — config loading and scope key derivation
|
||||
hub/ — WebSocket fan-out broker
|
||||
ingest/ — MQTT packet ingestion and side effects
|
||||
iatadb/ — in-memory IATA airport lookup
|
||||
keystore/ — channel key lookup
|
||||
scopestore/ — transport scope key lookup
|
||||
ws/ — WebSocket connection handling
|
||||
docs/ — generated swagger docs (do not edit by hand)
|
||||
api/ response types and Reader interface
|
||||
handlers/ HTTP handlers (validation, routing, response)
|
||||
middleware/ auth, rate limiting, logging
|
||||
router/ chi router wiring
|
||||
background/ periodic maintenance (retention, rollups, age-out)
|
||||
backup/ database + config bundle export and verification
|
||||
borders/ foreign-repeater classification against IATA borders
|
||||
cache/ Redis caching layer over the Reader
|
||||
config/ config loading and scope key derivation
|
||||
hub/ WebSocket fan-out broker
|
||||
iatadb/ in-memory IATA airport lookup
|
||||
ingest/ MQTT packet ingestion and side effects
|
||||
keystore/ channel key lookup
|
||||
logging/ slog setup
|
||||
lora/ LoRa airtime calculation
|
||||
meshmapper/ MeshMapper scope and zone imports
|
||||
presence/ coalesced last-seen writes
|
||||
profiling/ opt-in CPU profiling
|
||||
scopestore/ transport scope key lookup
|
||||
ws/ WebSocket connection handling
|
||||
docs/ generated swagger and design notes (swagger: do not edit by hand)
|
||||
```
|
||||
|
||||
Key patterns to understand before contributing:
|
||||
@@ -267,7 +248,7 @@ Key patterns to understand before contributing:
|
||||
put business logic here.
|
||||
- **Ingest layer** (`internal/ingest/`): processes raw MQTT packets, calls the
|
||||
store, and broadcasts hub events. The `DB` interface in `ingest.go` defines
|
||||
exactly what the ingest layer needs from the store — keep it minimal.
|
||||
exactly what the ingest layer needs from the store; keep it minimal.
|
||||
- **Hub** (`internal/hub/`): pure fan-out broker. Events are pre-serialised JSON
|
||||
before entering the hub so the broadcast loop never touches encoding.
|
||||
- **Reader interface** (`internal/api/reader.go`): defines everything the API
|
||||
@@ -277,54 +258,15 @@ Key patterns to understand before contributing:
|
||||
|
||||
## Releases
|
||||
|
||||
Merges from `dev` to `main` are done by maintainers and represent a versioned
|
||||
release. Do not open PRs directly against `main`.
|
||||
|
||||
### Cutting a release
|
||||
|
||||
> NOTE: release are done manually, not from GH. Commits for releases should be
|
||||
> signed.
|
||||
|
||||
1. Ensure all changes are committed and CI is green on `dev`
|
||||
2. Bump the version string in the `@version` swagger annotation in
|
||||
`cmd/beacon/main.go`
|
||||
3. Regenerate swagger docs:
|
||||
The flow is in
|
||||
[Releases and versioning](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/releases.md#cutting-a-release).
|
||||
The server-specific step is bumping `@version` in `cmd/beacon/main.go` and regenerating Swagger:
|
||||
|
||||
```bash
|
||||
swag init -g cmd/beacon/main.go -o docs --parseInternal --parseDependency
|
||||
```
|
||||
|
||||
4. Commit the version bump and updated docs:
|
||||
|
||||
```
|
||||
chore: bump version to vX.Y.Z
|
||||
```
|
||||
|
||||
5. Merge `dev` into `main` (fast-forward only):
|
||||
|
||||
```bash
|
||||
git checkout main
|
||||
git merge --ff-only dev
|
||||
```
|
||||
|
||||
6. Tag the release and push:
|
||||
|
||||
```bash
|
||||
git tag vX.Y.Z
|
||||
git push origin main --tags
|
||||
```
|
||||
|
||||
Pushing the tag triggers the release CI workflow, which builds binaries for all
|
||||
supported platforms and attaches them to a draft GitHub release.
|
||||
|
||||
7. Open the draft release on GitHub, paste the release notes, and publish.
|
||||
|
||||
8. Rebase `dev` on `main` to keep histories in sync for future releases:
|
||||
|
||||
```bash
|
||||
git checkout dev
|
||||
git rebase main
|
||||
```
|
||||
---
|
||||
|
||||
## Recognition
|
||||
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ Beacon is built by its contributors. Thank you to everyone who has helped.
|
||||
|
||||
## Core
|
||||
|
||||
- [ded](https://hackers.town/@ded) — co-founder, lead developer
|
||||
- [ded](https://hackers.town/@ded), co-founder, lead developer
|
||||
|
||||
## Contributors
|
||||
|
||||
|
||||
@@ -1,83 +0,0 @@
|
||||
# Production CPU profiling
|
||||
|
||||
CPU profiling is supported on Linux and macOS and is disabled by default. It writes private files inside the container;
|
||||
there is no HTTP endpoint or published profiling port. Profiling failures stop the
|
||||
recorder and log a warning without stopping ingest.
|
||||
|
||||
## Enable a bounded session
|
||||
|
||||
Create a dedicated directory writable by the container user with mode `0700`.
|
||||
Mount it into the app container and set both environment variables:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
environment:
|
||||
BEACON_CPU_PROFILE_DIR: /profiles
|
||||
BEACON_CPU_PROFILE_UNTIL: "2026-10-02T12:00:00Z" # Replace with your fixed deadline.
|
||||
volumes:
|
||||
- ./profiles:/profiles
|
||||
```
|
||||
|
||||
Choose an RFC3339 deadline no more than 72 hours in the future. The same deadline
|
||||
remains in effect after restarts; expired settings are inactive. Do not generate a
|
||||
fresh deadline automatically on each startup. Restarting the app is required to
|
||||
change these settings. Preserve the rest of the deployment configuration.
|
||||
|
||||
Start with a deadline about one minute away to assess the first capture's overhead.
|
||||
Check process CPU, packet freshness, queue drops and database timeouts against a
|
||||
comparable period with profiling disabled. After reviewing that capture, an operator
|
||||
can enable a longer session. Profiling adds overhead while a capture is active.
|
||||
|
||||
## Capture behavior
|
||||
|
||||
- One 30-second sample immediately, then every 30 minutes, offset five minutes past
|
||||
the half hour so a maintenance trigger due on the hour is not lost to the cooldown.
|
||||
- Route reconfirmation requests an additional sample when the maintenance task starts.
|
||||
This includes the retention step before route validation. A five-minute cooldown
|
||||
between capture starts prevents overlap and repeated triggers from increasing load;
|
||||
a periodic sample due during the cooldown runs when it ends. A triggered sample
|
||||
during the cooldown is skipped, and any sample cancels a periodic one waiting on
|
||||
the cooldown.
|
||||
- Background task stacks carry a `task` label while profiling is enabled.
|
||||
- Shutdown or expiry stops the active sample and saves the shorter profile.
|
||||
- Each profile is limited to 8 MiB. The dedicated directory is limited to 256 MiB
|
||||
and 512 files, including metadata and files left by interrupted runs. The recorder
|
||||
reserves space for a full capture before starting and stops when a limit is reached.
|
||||
Files are never automatically deleted. Existing unrelated regular files consume the budget; subdirectories are ignored.
|
||||
- Profiles and metadata are written with mode `0600`. A `.partial` file indicates
|
||||
an interrupted capture and is not a completed profile.
|
||||
|
||||
Only one Beacon process should write to a profiling directory. Keep the mount
|
||||
private and out of web roots, backups intended for public download, and source control.
|
||||
Do not run another CPU profiler in the same process during a session.
|
||||
|
||||
## Interpret the output
|
||||
|
||||
Each `.pprof` has a `.pprof.json` sidecar containing UTC start/end times, the trigger,
|
||||
Go version, process CPU counters (Linux/macOS), goroutine count, and database-pool
|
||||
counters before and after the capture. Pool acquire duration and counts are cumulative;
|
||||
use differences between the two snapshots. They do not contain SQL text or credentials.
|
||||
|
||||
Copy completed files privately off the host. Record the container's image revision
|
||||
and digest alongside them. On a workstation with Go installed:
|
||||
|
||||
```sh
|
||||
go tool pprof -top capture.pprof
|
||||
go tool pprof -top -cum capture.pprof
|
||||
go tool pprof -tags capture.pprof
|
||||
```
|
||||
|
||||
CPU samples show work inside Beacon, not CPU used by PostgreSQL or time waiting on
|
||||
locks. Correlate each UTC interval with separately collected host/container CPU and
|
||||
I/O, PostgreSQL wait states, observation throughput, and Beacon timeout/drop logs.
|
||||
Do not enable SQL text logging or run full-table counts for this purpose. The recorder
|
||||
makes no diagnostic database queries and does not change the ingest path.
|
||||
|
||||
Review captures from quiet periods, bursts, and maintenance separately before
|
||||
combining them. Additional maintenance samples deliberately bias an aggregate profile.
|
||||
Thirty-second windows can miss brief stalls; the absence of a stack is not proof that
|
||||
it never consumes CPU.
|
||||
|
||||
At the deadline, check for `CPU profiling stopped` in the app log. Export the files,
|
||||
then remove the environment variables and mount during the next planned deployment.
|
||||
@@ -14,8 +14,8 @@ in PostgreSQL, and streams live events to WebSocket clients.
|
||||
- Subscribes to MeshCore MQTT brokers and decodes incoming LoRa packets using
|
||||
[meshcore-go](https://github.com/meshcore-go/meshcore-go)
|
||||
- Stores packets, observations, nodes, observers, traces, routes and channel
|
||||
messages in PostgreSQL (more backends to come)
|
||||
- Deduplicates observations across multiple brokers (same packet heard by two
|
||||
messages in PostgreSQL
|
||||
- Deduplicates observations across multiple brokers (the same packet heard by two
|
||||
brokers is one observation per observer)
|
||||
- Decrypts group text messages for known channel keys
|
||||
- Detects firmware capability flags from path hash sizes
|
||||
@@ -25,14 +25,9 @@ in PostgreSQL, and streams live events to WebSocket clients.
|
||||
- Seeds regions, IATA display names, and channel keys from a YAML config file on
|
||||
startup
|
||||
|
||||
For deployment instructions including the frontend app and reverse proxy
|
||||
examples, see [beacon-docs](https://github.com/MeshCore-Beacon/beacon-docs).
|
||||
|
||||
For a bounded private database and saved-config bundle, see
|
||||
[backup export](docs/backup-export.md). A standalone export tool is available; the
|
||||
protected download API is opt-in; browser login and import remain separate follow-ups.
|
||||
|
||||
---
|
||||
This repo is the code. Deploying, configuring and operating Beacon is documented in
|
||||
[beacon-docs](https://github.com/MeshCore-Beacon/beacon-docs); see
|
||||
[Documentation](#documentation) below.
|
||||
|
||||
## Stack
|
||||
|
||||
@@ -49,659 +44,70 @@ protected download API is opt-in; browser login and import remain separate follo
|
||||
| Config | YAML via gopkg.in/yaml.v3 |
|
||||
| Env | godotenv |
|
||||
|
||||
---
|
||||
## Running it locally
|
||||
|
||||
## Getting started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Go 1.26+
|
||||
- Docker and Docker Compose
|
||||
|
||||
### 1. Clone and configure
|
||||
You need Go 1.26+ and a PostgreSQL 16 database. Docker is the easy way to get the database.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/MeshCore-Beacon/beacon-server.git
|
||||
cd beacon-server
|
||||
git clone https://github.com/MeshCore-Beacon/beacon-server.git && cd beacon-server
|
||||
cp env.example .env
|
||||
cp config.yaml.example config.yaml
|
||||
```
|
||||
|
||||
Edit `.env` with your broker credentials and database DSN. Edit `config.yaml` to
|
||||
define your regions, IATA display names, channel keys, and retention settings.
|
||||
|
||||
### 2. Start PostgreSQL
|
||||
|
||||
Any PostgreSQL 16 database works. For a local throwaway one:
|
||||
|
||||
```bash
|
||||
docker run -d --name beacon-postgres -p 5432:5432 \
|
||||
-e POSTGRES_USER=beacon -e POSTGRES_PASSWORD=beacon -e POSTGRES_DB=beacon \
|
||||
postgres:16-alpine
|
||||
```
|
||||
|
||||
Database migrations are applied automatically on startup.
|
||||
|
||||
### Upgrading from 1.x
|
||||
|
||||
2.0.0 starts from a new schema baseline and **needs a fresh database**. Pointed
|
||||
at a 1.x database it refuses to start with `database schema predates Beacon
|
||||
2.0.0; 2.0.0 needs a fresh database`. 1.x history does not carry over: create a
|
||||
new, empty database (Docker: stop the stack and move or remove the Postgres
|
||||
data directory) before starting 2.0.0.
|
||||
|
||||
Config changes to review when upgrading:
|
||||
|
||||
- Every manually configured `scopes:` entry needs a `region`.
|
||||
- `meshmapper.scopes.sources` is ignored; MeshMapper refresh intervals must be
|
||||
within 1h–24h for scopes and 24h–168h for zones, or startup fails.
|
||||
- REST rate limiting is on by default. Behind a reverse proxy, set
|
||||
`server.trusted_proxies` (see [Reverse proxies and rate limits](#reverse-proxies-and-rate-limits)).
|
||||
- `packets.retention` now defaults to 7 days.
|
||||
|
||||
### 3. Run
|
||||
|
||||
```bash
|
||||
-e POSTGRES_USER=beacon -e POSTGRES_PASSWORD=beacon -e POSTGRES_DB=beacon postgres:16-alpine
|
||||
go run ./cmd/beacon
|
||||
```
|
||||
|
||||
Or pull and run the Docker image:
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/meshcore-beacon/beacon-server:latest
|
||||
```
|
||||
|
||||
The image is public on GitHub Container Registry — no `docker login` required.
|
||||
`latest` tracks stable releases; pin a version tag (e.g. `2.0.0`) for
|
||||
predictable upgrades, and `dev` follows the development branch.
|
||||
|
||||
Beacon will:
|
||||
|
||||
- Load `.env` and `config.yaml`
|
||||
- Connect to PostgreSQL and seed config data
|
||||
- Connect to the configured MQTT brokers
|
||||
- Start the HTTP server on `LISTEN_ADDR` (default `:8080`)
|
||||
|
||||
### Cold start and path resolution
|
||||
|
||||
Path resolution, firmware capability detection, and known route storage all
|
||||
depend on nodes having advertised at least once to a local observer. On a fresh
|
||||
deployment `resolvedPath` will show `"confidence": "none"` for all hops and
|
||||
`supportsMultibytePaths` will be `false` for all nodes until advert traffic
|
||||
arrives and populates `node_short_ids`. This is expected behaviour — resolution
|
||||
improves automatically as the mesh is observed over time.
|
||||
|
||||
Similarly, GRP_TXT packets whose channel key isn't yet known at ingest time are
|
||||
stored as hash-only, undecrypted rows. Adding the channel's key to `config.yaml`
|
||||
doesn't retroactively decrypt that history immediately — it's picked up
|
||||
automatically on the next restart, when Beacon scans for undecrypted packets
|
||||
matching a now-known channel and decrypts them. Watch the startup log for
|
||||
`backfilled N previously-undecrypted channel message(s)`.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Admin authentication
|
||||
|
||||
The `/api/v1/admin` subtree requires `Authorization: Bearer <key>`. Set the
|
||||
operator key with `BEACON_API_KEY` or `auth.api_key` in YAML. A set environment
|
||||
variable overrides YAML; an explicitly empty value disables admin access.
|
||||
With no key, admin requests return JSON 503 while public reads and WebSockets
|
||||
continue normally. With a key, missing, incorrect or duplicate Authorization
|
||||
headers return JSON 401 with `WWW-Authenticate: Bearer`.
|
||||
|
||||
`GET /api/v1/admin/config` returns selected running settings: CORS options with
|
||||
Beacon defaults applied, `auth.configured`, and `ingest.broker_count` (configured
|
||||
broker workers, not connection status or a tunable processing-worker pool).
|
||||
The CORS lists are the options supplied to the middleware; its normal matching
|
||||
normalization still applies. The response excludes
|
||||
credential fields, broker addresses, channel material, database settings and
|
||||
other configuration. Unknown admin paths return 404 and unsupported
|
||||
methods on the config endpoint return 405 after authentication.
|
||||
Global CORS preflights remain public. Use a long, randomly generated key, keep
|
||||
it out of source control and logs, and send it only in the Authorization header,
|
||||
never the URL or request body. Require HTTPS at the reverse proxy and restrict
|
||||
direct access to Beacon's HTTP listener to that proxy or a private connection.
|
||||
Changing the key requires a restart. No API key is issued automatically. The
|
||||
key must be at least 16 characters with no inner whitespace (surrounding
|
||||
whitespace is trimmed); an unusable key prevents startup.
|
||||
|
||||
`PUT /api/v1/admin/config` accepts only
|
||||
`{"cors":{"allowed_origins":["https://example.org"]}}`. It replaces the entire
|
||||
origin list immediately and updates the reported configuration with the same
|
||||
policy. Requests already in progress may use the previous policy. Concurrent
|
||||
valid updates are serialized; updates take effect one at a time. The response
|
||||
contains `config`, `persisted: false` and `requires_restart: false`.
|
||||
|
||||
Updates are **runtime-only**: no file or database is written, and restarting
|
||||
reloads the saved configuration. Keep 1–32 ASCII HTTP(S) origins, at most 512 bytes
|
||||
each, with an optional single hostname wildcard; a sole `*` permits all origins.
|
||||
Empty/null lists, URL paths/queries/credentials, control characters and unknown
|
||||
fields are rejected. Requests must be JSON, at most 16 KiB. Other CORS options,
|
||||
auth/credential fields and broker count cannot be changed here; there is no
|
||||
configurable `ingest.worker_count`. Cross-origin admin clients need PUT allowed
|
||||
in the saved CORS methods. CORS controls browser access, not authentication.
|
||||
|
||||
Operator accounts are available at `GET/POST /api/v1/admin/accounts` and
|
||||
`GET/DELETE /api/v1/admin/accounts/{id}`. POST accepts a JSON `name` field in a
|
||||
body up to 4 KiB; names are trimmed, case-sensitive and limited to 128 Unicode
|
||||
characters without control characters. Active names are unique. DELETE soft
|
||||
deactivates the record (204); missing IDs return 404 and an already inactive
|
||||
record returns 409. A deactivated name may be reused by a new account.
|
||||
Lists include active and inactive records, newest first, without pagination.
|
||||
These are operator-defined records; no login, session or API token is created.
|
||||
Cross-origin account clients need both `POST` and `DELETE` in the saved
|
||||
`cors.allowed_methods`; the default `GET, HEAD, OPTIONS` is read-only. For an
|
||||
admin UI that also updates configuration, use `[GET, HEAD, OPTIONS, POST, PUT,
|
||||
DELETE]`, restrict `cors.allowed_origins` to that UI, and allow `Authorization`
|
||||
and `Content-Type` headers. Otherwise browser preflight blocks these requests
|
||||
even when the same bearer-authenticated request works with curl.
|
||||
|
||||
### Environment variables (`.env`)
|
||||
|
||||
| Variable | Default | Description |
|
||||
| ------------------------ | ------------- | ------------------------------------------------------------ |
|
||||
| `LISTEN_ADDR` | `:8080` | HTTP listen address |
|
||||
| `POSTGRES_DSN` | — | PostgreSQL connection string |
|
||||
| `REDIS_ADDR` | — | Redis address (`host:port`). Leave unset to disable caching. |
|
||||
| `REDIS_PASSWORD` | — | Redis password (optional) |
|
||||
| `REDIS_DB` | `0` | Redis database index |
|
||||
| `CONFIG_PATH` | `config.yaml` | Path to YAML config file |
|
||||
| `MQTT_BROKER_1_URL` | — | Broker 1 WebSocket URL (e.g. `wss://mqtt1.example.com:443`) |
|
||||
| `MQTT_BROKER_1_USERNAME` | — | Broker 1 username |
|
||||
| `MQTT_BROKER_1_PASSWORD` | — | Broker 1 password |
|
||||
| `MQTT_BROKER_2_URL` | — | Broker 2 WebSocket URL |
|
||||
| `MQTT_BROKER_2_USERNAME` | — | Broker 2 username |
|
||||
| `MQTT_BROKER_2_PASSWORD` | — | Broker 2 password |
|
||||
| `BEACON_API_KEY` | — | Admin bearer key; overrides `auth.api_key` when set |
|
||||
| `LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error`; overrides `log.level` |
|
||||
| `LOG_FORMAT` | `text` | `text` or `json`; overrides `log.format` |
|
||||
| `BEACON_CPU_PROFILE_DIR` | — | Private CPU capture directory (see [PROFILING.md](PROFILING.md)) |
|
||||
| `BEACON_CPU_PROFILE_UNTIL` | — | RFC 3339 deadline for CPU captures, at most 72h ahead |
|
||||
|
||||
### Config file (`config.yaml`)
|
||||
|
||||
```yaml
|
||||
# MeshMapper integrations (optional, no API key). Import transport scopes and
|
||||
# IATA border outlines from MeshMapper instead of maintaining scopes: and
|
||||
# iatas.*.borderFile by hand. Both cover every known IATA that MeshMapper lists;
|
||||
# no sources to configure. See config.yaml.example for limits and behaviour.
|
||||
#meshmapper:
|
||||
# scopes:
|
||||
# enabled: true
|
||||
# zones:
|
||||
# enabled: true
|
||||
# import_groups: true # MeshMapper zone groups become regions
|
||||
# channels:
|
||||
# enabled: true # import each region's public hashtag channels
|
||||
|
||||
# Optional IATA overrides — auto-created on first packet arrival,
|
||||
# only needed if you want to customise display name or coordinates.
|
||||
# borderFile points to a GeoJSON Feature (Polygon or MultiPolygon) for the
|
||||
# region border map feature; relative paths resolve against this config
|
||||
# file's own directory. Validated at boot — invalid geometry fails startup.
|
||||
iatas:
|
||||
YVR:
|
||||
name: Vancouver International
|
||||
lat: 49.1967
|
||||
lng: -123.1815
|
||||
borderFile: borders/yvr.geojson # optional; or use meshmapper.zones
|
||||
|
||||
# Super-regions grouping multiple IATAs.
|
||||
regions:
|
||||
- slug: western-canada
|
||||
name: Western Canada
|
||||
display_order: 1
|
||||
center_lat: 51.0
|
||||
center_lng: -114.0
|
||||
zoom_level: 5
|
||||
iatas: [YVR, YYJ, YYC, YEG]
|
||||
|
||||
# Channel keys for decrypting group messages.
|
||||
channel_keys:
|
||||
# Hashtag channels: Beacon derives the PSK from the tag name automatically.
|
||||
# secret = SHA256("#tag")[:16], channel_hash = SHA256(secret)[0]
|
||||
# Tag names should be provided without the # prefix. Plain names show under
|
||||
# every region; {name, region: <slug>} limits a channel to one region.
|
||||
hashtags:
|
||||
- meshcore
|
||||
- name: vancouver-mesh
|
||||
region: western-canada
|
||||
|
||||
# Explicit keys: channel hash (hex) and key (hex), with optional display name.
|
||||
# The public MeshCore channel key is included in config.yaml.example.
|
||||
keys:
|
||||
"11":
|
||||
key: "8b3387e9c5cdea6ac9e5edbaa115cd72"
|
||||
name: "Public"
|
||||
|
||||
# Regional transport scopes for matching TRANSPORT_FLOOD packets.
|
||||
# Plain names have # prepended automatically (e.g. "bc" → "#bc").
|
||||
# region (required) is a configured region slug; region-filtered scope lists
|
||||
# and scope stats show the scope under that region's IATAs. Matching stays global.
|
||||
# Optional when meshmapper.scopes covers your regions.
|
||||
scopes:
|
||||
- name: bc
|
||||
region: western-canada
|
||||
- name: "#west"
|
||||
region: western-canada
|
||||
|
||||
# Observer telemetry storage settings.
|
||||
telemetry:
|
||||
retention: 744h # how long to keep telemetry snapshots (default: 31 days)
|
||||
resolution: 1h # snapshot frequency per observer; duplicates within window are dropped (default: 1h)
|
||||
|
||||
# Packet and observation retention.
|
||||
packets:
|
||||
retention: 168h # packets, observations, channel messages (default: 7 days, minimum 24h)
|
||||
|
||||
# Hourly analytics rollups behind the stats endpoints; they outlive raw packets.
|
||||
analytics:
|
||||
rollup_retention: 2160h # also the longest /stats/series window (default: 90 days, minimum 24h)
|
||||
|
||||
# Presence write coalescing.
|
||||
# Observer last_seen and packet last_heard_at bumps are batched in memory and
|
||||
# flushed on an interval instead of writing one row per observation. An
|
||||
# unclean shutdown loses at most one interval of presence freshness.
|
||||
presence:
|
||||
flush_interval: 30s # how often coalesced bumps are flushed (default: 30s)
|
||||
packet_ttl: 30s # how long a quiet packet stays coalesced before writing through again (default: 30s)
|
||||
|
||||
# Direct proxy peers allowed to set the client IP through X-Real-IP.
|
||||
server:
|
||||
trusted_proxies: [172.30.0.0/24] # default: [] (direct access)
|
||||
|
||||
# Per-client REST limits for /api/v1/*.
|
||||
ratelimit:
|
||||
enabled: true # default: true
|
||||
requests_per_minute: 300 # default: 300
|
||||
|
||||
# WebSocket settings.
|
||||
websocket:
|
||||
max_connections_per_ip: 5 # default: 5
|
||||
max_connects_per_minute: 10 # upgrade attempts per IP (default: 10)
|
||||
allowed_origins: [https://beacon.example.com] # other sites allowed to open /ws (default: same host only)
|
||||
|
||||
# Node staleness, deletion, and clock-drift thresholds.
|
||||
nodes:
|
||||
mark_foreign: false # optional indication for repeaters outside configured IATA borders
|
||||
stale_threshold: 24h # mark a node "stale" in the API after this long unseen (default: 24h)
|
||||
delete_after: 720h # delete a node entirely after this long unseen (default: 30 days)
|
||||
clock_drift_threshold: 5m # |device clock - server clock| above which clockOutOfSync=true for a repeater/room server (default: 5m)
|
||||
|
||||
# Optional observer age-out (disabled by default; set e.g. 720h to opt in).
|
||||
# Enable only when one Beacon ingest process owns the database: presence-cache
|
||||
# preparation is local to that process, including both of its MQTT workers.
|
||||
# Deletes at most 1000 observers per background.cleanup interval, only when
|
||||
# last_seen/last_status_at are old and no observations, telemetry or ownership remain.
|
||||
# Broker/location/scope metadata cascades; a returning observer receives a new ID.
|
||||
observers:
|
||||
delete_after: 0s # omitted or nonpositive disables deletion
|
||||
|
||||
# Redis caching layer (optional).
|
||||
# Caches read-heavy, slow-changing responses to reduce PostgreSQL load.
|
||||
# Connection details (address, password, database) are set via environment
|
||||
# variables. Leave REDIS_ADDR unset to disable caching entirely.
|
||||
# TTLs are duration strings e.g. "30m", "1h". Per-category TTLs override
|
||||
# the global ttl. Any unset category inherits ttl. Default: 1h.
|
||||
cache:
|
||||
ttl: "1h"
|
||||
ttls:
|
||||
stats: "1h" # stats endpoints (hourly rollups; keys change when a new hour is rolled)
|
||||
reference: "1h" # IATAs, regions, scopes
|
||||
nodes: "1h" # node detail (also explicitly invalidated on upsert)
|
||||
observers: "1h" # observer detail (also explicitly invalidated on upsert)
|
||||
|
||||
# Geographic ingest filter (optional).
|
||||
# Drop packets from observers outside the specified area.
|
||||
# Country codes are ISO 3166-1 alpha-2. Continent codes: AF AN AS EU NA OC SA.
|
||||
# If both are set an IATA passes if it matches either (OR semantics).
|
||||
# Omit entirely to accept all IATAs (default).
|
||||
ingest:
|
||||
allow_countries: [CA, US] # only store packets from these countries
|
||||
allow_continents: [NA] # or: accept all of North America
|
||||
```
|
||||
|
||||
IATAs are auto-created on first packet arrival. The config file adds display
|
||||
names, coordinates, and optional region borders. Regions and channel keys
|
||||
must be defined here unless MeshMapper imports them (`meshmapper.zones.import_groups`,
|
||||
`meshmapper.channels`).
|
||||
|
||||
---
|
||||
|
||||
### Reverse proxies and rate limits
|
||||
|
||||
REST requests under `/api/v1` are rate limited per client IP (300/min by
|
||||
default; IPv6 clients share a /64). Exhausted clients get `429` with
|
||||
`rate_limited` and `Retry-After`. WebSocket upgrades are limited separately by
|
||||
`websocket.max_connects_per_minute`.
|
||||
|
||||
Beacon never trusts `X-Forwarded-For` or `True-Client-IP`. Behind a reverse
|
||||
proxy, list the proxy's address in `server.trusted_proxies` (CIDR, e.g.
|
||||
`127.0.0.1/32` or the Docker network subnet) and have the proxy overwrite
|
||||
`X-Real-IP` with the connecting client's address:
|
||||
|
||||
```nginx
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
```
|
||||
|
||||
```caddy
|
||||
reverse_proxy app:8080 {
|
||||
header_up X-Real-IP {remote_host}
|
||||
}
|
||||
```
|
||||
|
||||
Without this every visitor shares the proxy's budget and the site gets 429s
|
||||
under normal load. Beacon logs a warning at startup when limits are on and no
|
||||
proxy is trusted, and once at runtime when a request carries forwarding headers
|
||||
it is ignoring. Full nginx and Caddy examples are in
|
||||
[beacon-docs](https://github.com/MeshCore-Beacon/beacon-docs).
|
||||
|
||||
---
|
||||
|
||||
### Foreign repeater indication
|
||||
|
||||
Set `nodes.mark_foreign: true` to expose `possiblyForeign` on repeater nodes.
|
||||
The local operating area is the union of **all configured**
|
||||
`iatas.<code>.borderFile` GeoJSON Polygon/MultiPolygon features, with
|
||||
`meshmapper.zones` boundaries replacing them where MeshMapper has one. IATAs
|
||||
without a border do not add an area; airport coordinates and the current API
|
||||
region filter are not boundaries. Enabling this with no border source, missing
|
||||
files or invalid geometry fails startup. Border file changes require a restart.
|
||||
|
||||
Inside any polygon (including its edges) means `false`; outside the entire union
|
||||
means `true`. Hole interiors are outside; hole edges are local. Other node roles,
|
||||
missing or invalid positions have no classification (an explicit 0/0 advert
|
||||
clears the stored position).
|
||||
This is a hint based on reported position, not proof of a repeater's origin.
|
||||
Packet ingestion, heard-in IATAs and route matching remain unchanged.
|
||||
|
||||
Node list/detail reads apply the current geometry after cache reads, so existing
|
||||
historical nodes need no backfill. A `nodeUpdate` includes `possiblyForeign`
|
||||
when its advert provides a position: a boolean for known positions, `null` to
|
||||
clear it when the advert reports 0/0 or an invalid position. Omission retains the
|
||||
previous value when a repeater's advert omits its position. A change to another
|
||||
role also sends `null`. `lat`/`lng` follow the same rule: omitted keeps the
|
||||
client's position, explicit `null` clears it.
|
||||
The field is omitted everywhere when the feature is disabled (the default).
|
||||
|
||||
Use longitude/latitude coordinate order and split antimeridian-crossing borders
|
||||
into MultiPolygons as described in [RFC 7946 section 3.1.9](https://www.rfc-editor.org/rfc/rfc7946#section-3.1.9).
|
||||
Classification rejects unsplit edges spanning more than 180 degrees rather than
|
||||
silently treating them as the complementary global area.
|
||||
|
||||
## Authentication
|
||||
|
||||
Public reads and the WebSocket need no credentials. Only the `/api/v1/admin`
|
||||
subtree is protected, by a bearer key (see [Admin authentication](#admin-authentication)).
|
||||
Serve Beacon behind an HTTPS reverse proxy.
|
||||
|
||||
---
|
||||
|
||||
## WebSocket API
|
||||
|
||||
Connect to `ws://host:8080/ws`.
|
||||
|
||||
On connect the server sends a `hello`:
|
||||
|
||||
```json
|
||||
{ "v": 1, "type": "hello", "serverTime": 1234567890000, "connectionId": "uuid" }
|
||||
```
|
||||
|
||||
The connection closes after 90 seconds of inactivity. Clients should send a
|
||||
`ping` every 30 seconds.
|
||||
|
||||
### Client → Server messages
|
||||
|
||||
**Subscribe** — add a filter to this connection. Multiple subscriptions are
|
||||
unioned (OR semantics): an event matches if it satisfies any active
|
||||
subscription. The server replies with a `subscriptionId` to use for
|
||||
unsubscribing.
|
||||
|
||||
```json
|
||||
{
|
||||
"v": 1,
|
||||
"type": "subscribe",
|
||||
"id": "sub-1",
|
||||
"scope": {
|
||||
"iatas": ["YOW", "YYZ"],
|
||||
"regionIds": ["1"],
|
||||
"regionSlugs": ["western-canada"],
|
||||
"payloadTypes": [4, 5],
|
||||
"routeTypes": [1, 2],
|
||||
"channelHashes": ["11"],
|
||||
"observerIds": ["<observer uuid>"],
|
||||
"events": ["packetObservation", "channelMessage"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
All scope fields are optional; omitted or empty means no filter on that
|
||||
dimension. `routeTypes` and `observerIds` filter `packetObservation` events
|
||||
(`observerIds` also `observerStatus`), and `channelHashes` only `channelMessage`. `regionIds` and
|
||||
`regionSlugs` are both expanded to their member IATAs server-side.
|
||||
|
||||
**Unsubscribe** — remove a specific subscription by ID.
|
||||
|
||||
```json
|
||||
{
|
||||
"v": 1,
|
||||
"type": "unsubscribe",
|
||||
"id": "unsub-1",
|
||||
"subscriptionId": "<uuid from subscribed reply>"
|
||||
}
|
||||
```
|
||||
|
||||
The server replies `{ "v": 1, "type": "unsubscribed", "id": "unsub-1", "subscriptionId": "..." }`.
|
||||
|
||||
**Configure** — connection-wide flags, all default `false`:
|
||||
|
||||
- `resolvePath` adds per-hop node resolution to `packetObservation` events (see below).
|
||||
- `includeObserverKey` adds `observation.observerPublicKey`.
|
||||
- `includeRepeats` also streams later hearings of an already-stored observation
|
||||
over a new path, as `packetObservation` events with `packet.isRepeat: true`
|
||||
and `observationCount: 0`.
|
||||
|
||||
Unlike `subscribe`, this is not additive: each `configure` sets all three flags
|
||||
to exactly the values sent, so an omitted flag turns off.
|
||||
|
||||
```json
|
||||
{ "v": 1, "type": "configure", "id": "cfg-1", "resolvePath": true, "includeRepeats": true }
|
||||
```
|
||||
|
||||
The server replies:
|
||||
|
||||
```json
|
||||
{ "v": 1, "type": "configured", "id": "cfg-1", "resolvePath": true, "includeObserverKey": false, "includeRepeats": true }
|
||||
```
|
||||
|
||||
**Ping**
|
||||
|
||||
```json
|
||||
{ "v": 1, "type": "ping", "id": "ping-1" }
|
||||
```
|
||||
|
||||
### Server → Client events
|
||||
|
||||
| Type | Description |
|
||||
| ------------------- | --------------------------------------------------- |
|
||||
| `packetObservation` | New observation written to DB |
|
||||
| `observerStatus` | Observer status update |
|
||||
| `nodeUpdate` | Node upserted from advert |
|
||||
| `channelMessage` | Decrypted channel message (scope must include hash) |
|
||||
|
||||
When `resolvePath` is enabled via `configure`, `packetObservation` events
|
||||
include a `resolvedPath` array: one entry per hop in the packet's path, each
|
||||
with a `confidence` (`"high"` for exactly one candidate node, `"ambiguous"`
|
||||
for multiple, `"none"` for zero) and the matching node(s)' id, name,
|
||||
lat/lng, and public key. Without `resolvePath` enabled, `resolvedPath` is
|
||||
`null` — this keeps the default event payload small; only opt in if you're
|
||||
actually rendering path connections (e.g. drawing hops on a map).
|
||||
|
||||
### Backpressure
|
||||
|
||||
The server write buffer per connection is bounded at 256 events. If a client
|
||||
falls behind, the server drops the oldest queued events and sends a `lagged`
|
||||
notice:
|
||||
|
||||
```json
|
||||
{ "v": 1, "type": "lagged", "droppedCount": 12, "since": 1234567890000 }
|
||||
```
|
||||
|
||||
Clients should respond by backfilling over REST, then resume streaming:
|
||||
`/api/v1/packets/backfill?afterObservationId=<last observation id>` and
|
||||
`/api/v1/messages/backfill?afterId=<last message id>`.
|
||||
|
||||
### Reconnection
|
||||
|
||||
Subscriptions are not persisted — they exist only for the lifetime of the
|
||||
connection. On any disconnect the client should reconnect with backoff, re-issue
|
||||
all subscriptions, and backfill via the same REST endpoints.
|
||||
|
||||
### Connection limits
|
||||
|
||||
- At most `websocket.max_connections_per_ip` (default 5) concurrent connections
|
||||
per IP. A connection over the cap is accepted and then closed with code
|
||||
`1013` (try again later) before `hello`.
|
||||
- At most `websocket.max_connects_per_minute` (default 10) upgrade attempts per
|
||||
IP; beyond that the handshake gets `429` with `Retry-After: 60`.
|
||||
- Browsers may only connect from the same host unless their origin is listed in
|
||||
`websocket.allowed_origins` (`https://*.example.com` covers every subdomain).
|
||||
|
||||
Behind a reverse proxy both limits need `server.trusted_proxies`, or every
|
||||
visitor counts as the proxy's IP.
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
Base path: `/api/v1`
|
||||
|
||||
All list endpoints support cursor-based pagination via `cursor` and `limit`
|
||||
query params. See the Swagger UI at `http://localhost:8080/swagger/index.html`
|
||||
for full parameter documentation.
|
||||
|
||||
### Authentication and limits
|
||||
|
||||
Only `/admin/*` needs `Authorization: Bearer <key>`. All `/api/v1` routes are
|
||||
rate limited per client IP; see [Reverse proxies and rate limits](#reverse-proxies-and-rate-limits).
|
||||
List `limit` values are clamped to 1–200.
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
| -------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
|
||||
| `GET` | `/admin/accounts` | List operator accounts (bearer key) |
|
||||
| `POST` | `/admin/accounts` | Create an operator account (bearer key) |
|
||||
| `GET` | `/admin/accounts/{id}` | Get an operator account (bearer key) |
|
||||
| `DELETE` | `/admin/accounts/{id}` | Deactivate an operator account (bearer key) |
|
||||
| `GET` | `/admin/backup` | Download a private database and saved-config backup (bearer key, opt-in) |
|
||||
| `GET` | `/admin/config` | Inspect selected running configuration (bearer key) |
|
||||
| `PUT` | `/admin/config` | Replace runtime CORS origins (bearer key) |
|
||||
| `GET` | `/brokers` | List MQTT brokers and connection status |
|
||||
| `GET` | `/channels` | List channels (`hash`, `iata`/`iatas`, `keyKnown`, `pageCursor`, `limit`) |
|
||||
| `GET` | `/channels/{channelID}` | Get channel detail by integer ID |
|
||||
| `GET` | `/channels/{channelID}/messages` | List messages for a channel (`since`, `iatas`, `region`/`regionId`, `scope`, `cursor`, `limit`) |
|
||||
| `GET` | `/iatas` | List all known IATA codes |
|
||||
| `GET` | `/iatas/{iata}` | Get a single IATA code |
|
||||
| `GET` | `/iatas/{iata}/border` | Get an IATA's GeoJSON border, if any (204 if not) |
|
||||
| `GET` | `/messages` | List channel messages (`channelID`, `channelHash`, `since`, `iatas`, `region`/`regionId`, `scope`, `cursor`, `limit`) |
|
||||
| `GET` | `/messages/backfill` | Messages after `afterId` |
|
||||
| `GET` | `/nodes` | List nodes (type, IATA/region, name, scope, `pubkey`/`pubkeyPrefix`, capability flags, `neighbors`) |
|
||||
| `GET` | `/nodes/{nodeId}` | Get node detail |
|
||||
| `GET` | `/nodes/{nodeId}/neighbors` | List neighboring nodes observed in the mesh |
|
||||
| `GET` | `/nodes/{nodeId}/observations` | List observations of packets from a node |
|
||||
| `GET` | `/observers` | List observers (IATA/region, `type`, `broker`, `status=online\|offline`, `name`, `scope`) |
|
||||
| `GET` | `/observers/{observerId}` | Get observer detail including broker last-seen timestamps |
|
||||
| `GET` | `/observers/{observerId}/activity` | Heard-activity history (`range`, `interval`, `until`) |
|
||||
| `GET` | `/observers/{observerId}/adverts` | Adverts heard by observer |
|
||||
| `GET` | `/observers/{observerId}/telemetry` | Telemetry history (`range`, `interval=1h\|6h\|24h`, `afterId`) |
|
||||
| `GET` | `/packets` | List packets (payload/route type, IATA/region, scope, `since`/`until`; plural params take CSV) |
|
||||
| `GET` | `/packets/backfill` | Packets after `afterObservationId` |
|
||||
| `GET` | `/packets/{packetHash}` | Get packet with all observations |
|
||||
| `GET` | `/regions` | List all regions (summary) |
|
||||
| `GET` | `/regions/{regionId}` | Get a single region with IATA list |
|
||||
| `GET` | `/routes` | List known routes (`iata`, `hopCount`, `cursor`+`cursorId`) |
|
||||
| `GET` | `/routes/search` | Search routes by source and destination hash |
|
||||
| `GET` | `/routes/cross` | Search for routes crossing IATA boundaries |
|
||||
| `GET` | `/routes/{iata}/{pathKey}/observations` | Retained observations matching a saved route (see below) |
|
||||
| `GET` | `/scopes` | List transport scope names; IATA/region filters use configured regions and MeshMapper catalogues |
|
||||
| `GET` | `/scopes/{name}` | Get scope detail |
|
||||
| `GET` | `/stats/clock-drift` | Repeaters and room servers whose clock drifted past the threshold, worst first |
|
||||
| `GET` | `/stats/node-types` | Node type breakdown |
|
||||
| `GET` | `/stats/observations` | Hourly observation counts per IATA (last 7 days by default) |
|
||||
| `GET` | `/stats/observer-comparison` | Compare flood packets reported by two observers |
|
||||
| `GET` | `/stats/overview` | Network overview over the last 24 rolled hours |
|
||||
| `GET` | `/stats/paths` | Path-entry and hash-width distributions |
|
||||
| `GET` | `/stats/payload-breakdown` | Observation counts by payload type (last 24h by default) |
|
||||
| `GET` | `/stats/radio-presets` | Radio preset usage by IATA |
|
||||
| `GET` | `/stats/scopes` | Region scopes with hourly packet, node and observer counts (last 7 days by default) |
|
||||
| `GET` | `/stats/series` | Hourly card metrics and window summary (`since`/`until` required) |
|
||||
| `GET` | `/stats/signal` | Reception signal distributions and hourly trends |
|
||||
| `GET` | `/stats/top-advertisers` | Top N nodes by distinct ADVERT packets, split flood/direct (last 24h by default) |
|
||||
| `GET` | `/stats/top-nodes` | Top N nodes by advert hearings (last 7 days by default) |
|
||||
| `GET` | `/stats/top-observers` | Top N observers by observation count (last 24h by default) |
|
||||
| `GET` | `/stats/top-talkers` | Top N companion names by decrypted channel message count (last 24h by default) |
|
||||
| `GET` | `/traces` | List trace tags (`type=TRACE\|PING`, IATA/region, scope, `since`/`until`, `cursor`+`cursorTag`) |
|
||||
| `GET` | `/traces/{tag}` | Get full trace detail with resolved routes |
|
||||
|
||||
---
|
||||
Fill in `POSTGRES_DSN` and your MQTT broker credentials in `.env`. Every variable is described
|
||||
in [Configuration](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/configuration.md),
|
||||
and the fully annotated `config.yaml` lives in beacon-docs as well; the copy here is a working
|
||||
starter. Migrations run on startup. The API listens on `LISTEN_ADDR` (default `:8080`) and
|
||||
Swagger is at `http://localhost:8080/swagger/index.html`.
|
||||
|
||||
To run the web frontend against it, see
|
||||
[Running the full stack locally](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/CONTRIBUTING.md#running-the-full-stack-locally).
|
||||
|
||||
The published image is `ghcr.io/meshcore-beacon/beacon-server`. `latest` tracks stable
|
||||
releases, `dev` follows the development branch, and each release is also tagged `X.Y.Z` and
|
||||
`X.Y`.
|
||||
|
||||
### What you see on an empty database
|
||||
|
||||
Path resolution, capability detection and known routes depend on nodes having advertised to a
|
||||
local observer. On a fresh database every hop shows `"confidence": "none"` and
|
||||
`supportsMultibytePaths` is `false` until adverts populate `node_short_ids`. That is expected
|
||||
and fills in as the mesh is observed.
|
||||
|
||||
GRP_TXT packets whose channel key is not known yet are stored hash-only. After adding the key to
|
||||
`config.yaml`, the next restart decrypts the matching history; look for
|
||||
`config: backfilled N previously-undecrypted channel message(s)` in the log.
|
||||
|
||||
## Documentation
|
||||
|
||||
In [beacon-docs](https://github.com/MeshCore-Beacon/beacon-docs):
|
||||
|
||||
- [Deploy with Docker](https://github.com/MeshCore-Beacon/beacon-docs#deploy-the-all-in-one-stack)
|
||||
- [Getting packets in](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/getting-packets-in.md): brokers, the subscriber account, topics
|
||||
- [Configuration](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/configuration.md): environment variables and `config.yaml`
|
||||
- [API contract](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/api-contract.md): REST, WebSocket, admin endpoints
|
||||
- [Reverse proxy and rate limits](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/reverse-proxy.md)
|
||||
- [Upgrading](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/upgrading.md)
|
||||
- [Operations](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/operations.md), [backup and export](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/backup-export.md) and [CPU profiling](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/profiling.md)
|
||||
- [High level design](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/high-level-design.md)
|
||||
- [Releases and versioning](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/releases.md)
|
||||
|
||||
In this repo: [Historical stats](docs/historical-stats.md) explains the hourly rollups behind
|
||||
the stats endpoints, for anyone changing them. `docs/swagger.yaml` is the generated OpenAPI
|
||||
description.
|
||||
|
||||
## Contributing
|
||||
|
||||
[CONTRIBUTING.md](CONTRIBUTING.md) covers code style, tests, and database and API changes.
|
||||
Branches, commits and the release flow are shared across the Beacon repos and live in
|
||||
[beacon-docs/CONTRIBUTING.md](https://github.com/MeshCore-Beacon/beacon-docs/blob/main/CONTRIBUTING.md).
|
||||
Please read the [Code of Conduct](CODE_OF_CONDUCT.md). Security reports go through
|
||||
[SECURITY.md](SECURITY.md).
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the people who have helped build
|
||||
Beacon.
|
||||
|
||||
Beacon stands on the shoulders of giants. See [SHOULDERS.md](SHOULDERS.md) for
|
||||
the full list of open source projects that make this possible.
|
||||
|
||||
## Application logging
|
||||
|
||||
Beacon writes application logs to stderr. Configure the minimum level and output format in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
log:
|
||||
level: info # debug, info, warn, error
|
||||
format: text # text or json
|
||||
```
|
||||
|
||||
`LOG_LEVEL` and `LOG_FORMAT` override file settings; empty settings use `info` and `text`. Invalid values prevent startup. Configuration-loading failures can use the bootstrap text logger before file settings are available; failures after initialization retain error severity at every supported level.
|
||||
|
||||
Records include a component field. Ingest workers also include their broker name, and HTTP completion records include the validated client address, route, status and duration. Query strings and protocol hello payloads are excluded. Expected ingest skips and routine WebSocket lifecycle details are debug-level. Changing the application's format does not change Caddy/Apache access logs or their fail2ban configuration. Collect/rotate stderr through Docker or systemd.
|
||||
|
||||
### Analytics retention
|
||||
|
||||
Historical stats (traffic, payload, observer activity, talker, advertiser, node,
|
||||
scope, Signal and Paths) are read from hourly rollup tables kept for
|
||||
`analytics.rollup_retention` (default 90 days), independently of `packets.retention`.
|
||||
One background task rolls each UTC hour about 95 minutes after it closes;
|
||||
packet cleanup waits for unrolled hours (up to 24h) so they can be rolled in full.
|
||||
An hour whose raw rows were deleted first is marked partial and reported without values.
|
||||
Packet drill-down, sub-hour observer activity and observer comparison still read
|
||||
retained raw data. See [docs/historical-stats.md](docs/historical-stats.md).
|
||||
|
||||
### Observer monitoring metrics
|
||||
|
||||
Observer activity returns buckets in `[windowStart, windowEnd)`, including the current partial bucket for live requests, plus
|
||||
`generatedAt`, `source` (`raw` or `hourly`) and `summary`. `recordedPackets` is the
|
||||
sum of stored observations in those buckets; repeated broker delivery of the
|
||||
same retained packet/observer pair counts once. Unknown payload types appear as
|
||||
`-1` rather than disappearing. Freshness fields are measured at `generatedAt` even for historical `until` requests; `recordedPackets` alone follows the selected window. `lastCompleteHour` uses the previous complete UTC
|
||||
hour and includes its own start/end; `latestRecordedAt` is the latest retained
|
||||
reception timestamp. Missing records do not prove downtime. The optional `until`
|
||||
(epoch milliseconds, within the last 30 days) aligns two observers' charts.
|
||||
|
||||
The existing observer `observationCount` remains a legacy cumulative presence
|
||||
counter for compatibility, including status/neighbour events. It is not a
|
||||
period packet total. Broker presence and packet-arrival timestamps are now
|
||||
updated separately; this cannot reconstruct previously overwritten timestamps.
|
||||
Hourly activity comes from the analytics rollups, plus raw rows for hours not yet rolled
|
||||
(at most 24h back), and includes unknown-type activity. `rolledUntil` is the end of the newest
|
||||
complete rollup hour and `rawFrom` the start of the raw tail; hours between them are uncovered.
|
||||
|
||||
### Saved-route evidence
|
||||
|
||||
`GET /api/v1/routes/{iata}/{pathKey}/observations` returns retained observations that match a saved
|
||||
route's full path bytes, hash size and hop count within its IATA. `range` defaults to `24h` (max `720h`),
|
||||
`limit` to 50 (max 200); page with `nextPageCursor` → `pageCursor`. Evidence expires with raw packets.
|
||||
See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the people who have helped build Beacon, and
|
||||
[SHOULDERS.md](SHOULDERS.md) for the open source projects it stands on.
|
||||
|
||||
+1
-1
@@ -5,7 +5,7 @@
|
||||
Please do not report security vulnerabilities through public GitHub issues.
|
||||
|
||||
Instead, contact the maintainers directly via the MeshCore Canada Discord
|
||||
server: [MeshCore Canada Discord](https://discord.gg/Gz3KvJx2hf) — reach out to
|
||||
server: [MeshCore Canada Discord](https://discord.gg/Gz3KvJx2hf); reach out to
|
||||
**dedskelly** directly. Include as much detail as possible: the nature of the
|
||||
issue, steps to reproduce, and any potential impact.
|
||||
|
||||
|
||||
+23
-23
@@ -5,48 +5,48 @@ projects that make this possible:
|
||||
|
||||
## Core dependencies
|
||||
|
||||
- [meshcore-go](https://github.com/meshcore-go/meshcore-go) — MeshCore packet
|
||||
- [meshcore-go](https://github.com/meshcore-go/meshcore-go): MeshCore packet
|
||||
decoding
|
||||
- [paho.mqtt.golang](https://github.com/eclipse/paho.mqtt.golang) — MQTT client
|
||||
- [pgx](https://github.com/jackc/pgx) — PostgreSQL driver and connection pool
|
||||
- [sqlc](https://sqlc.dev) — Type-safe SQL code generation
|
||||
- [chi](https://github.com/go-chi/chi) — HTTP router
|
||||
- [go-chi/cors](https://github.com/go-chi/cors) — CORS middleware for chi
|
||||
- [go-chi/httprate](https://github.com/go-chi/httprate) — Per-client HTTP
|
||||
- [paho.mqtt.golang](https://github.com/eclipse/paho.mqtt.golang): MQTT client
|
||||
- [pgx](https://github.com/jackc/pgx): PostgreSQL driver and connection pool
|
||||
- [sqlc](https://sqlc.dev): Type-safe SQL code generation
|
||||
- [chi](https://github.com/go-chi/chi): HTTP router
|
||||
- [go-chi/cors](https://github.com/go-chi/cors): CORS middleware for chi
|
||||
- [go-chi/httprate](https://github.com/go-chi/httprate): Per-client HTTP
|
||||
sliding-window rate limits and retry headers
|
||||
- [go-redis](https://github.com/redis/go-redis) — Redis client
|
||||
- [coder/websocket](https://github.com/coder/websocket) — WebSocket
|
||||
- [go-redis](https://github.com/redis/go-redis): Redis client
|
||||
- [coder/websocket](https://github.com/coder/websocket): WebSocket
|
||||
implementation
|
||||
- [swaggo/swag](https://github.com/swaggo/swag) — OpenAPI documentation
|
||||
- [swaggo/swag](https://github.com/swaggo/swag): OpenAPI documentation
|
||||
generation
|
||||
- [godotenv](https://github.com/joho/godotenv) — .env file loading
|
||||
- [yaml.v3](https://github.com/go-yaml/yaml) — YAML config parsing
|
||||
- [google/uuid](https://github.com/google/uuid) — UUID generation
|
||||
- [golang.org/x/sync](https://pkg.go.dev/golang.org/x/sync) — singleflight for
|
||||
- [godotenv](https://github.com/joho/godotenv): .env file loading
|
||||
- [yaml.v3](https://github.com/go-yaml/yaml): YAML config parsing
|
||||
- [google/uuid](https://github.com/google/uuid): UUID generation
|
||||
- [golang.org/x/sync](https://pkg.go.dev/golang.org/x/sync): singleflight for
|
||||
collapsing concurrent cache misses
|
||||
- [paulmach/orb](https://github.com/paulmach/orb) — GeoJSON border geometry and
|
||||
- [paulmach/orb](https://github.com/paulmach/orb): GeoJSON border geometry and
|
||||
point-in-polygon checks
|
||||
- [swaggo/http-swagger](https://github.com/swaggo/http-swagger) — Swagger UI
|
||||
- [swaggo/http-swagger](https://github.com/swaggo/http-swagger): Swagger UI
|
||||
handler
|
||||
|
||||
## Testing
|
||||
|
||||
- [go.uber.org/mock](https://github.com/uber-go/mock) — gomock mocks for the
|
||||
- [go.uber.org/mock](https://github.com/uber-go/mock): gomock mocks for the
|
||||
generated queries
|
||||
- [miniredis](https://github.com/alicebob/miniredis) — In-memory Redis for cache
|
||||
- [miniredis](https://github.com/alicebob/miniredis): In-memory Redis for cache
|
||||
tests
|
||||
|
||||
## Data
|
||||
|
||||
- [OurAirports](https://ourairports.com/data/) — IATA airport geographic data
|
||||
- [OurAirports](https://ourairports.com/data/): IATA airport geographic data
|
||||
(CC0 public domain)
|
||||
|
||||
## Infrastructure
|
||||
|
||||
- [PostgreSQL](https://www.postgresql.org) — Primary data store
|
||||
- [Redis](https://redis.io) — Optional caching layer
|
||||
- [Caddy](https://caddyserver.com) — Reverse proxy
|
||||
- [PostgreSQL](https://www.postgresql.org): Primary data store
|
||||
- [Redis](https://redis.io): Optional caching layer
|
||||
- [Caddy](https://caddyserver.com): Reverse proxy
|
||||
|
||||
## Language
|
||||
|
||||
- [Go](https://go.dev) — The language that makes this all possible
|
||||
- [Go](https://go.dev): The language that makes this all possible
|
||||
|
||||
+4
-3
@@ -1,5 +1,6 @@
|
||||
# Beacon configuration file
|
||||
# Copy to config.yaml and adjust as needed.
|
||||
# Beacon configuration file: local development starter. Copy to config.yaml and adjust.
|
||||
# The fully annotated reference, with every key and its default, is
|
||||
# https://github.com/MeshCore-Beacon/beacon-docs/blob/main/app_config/config.yaml.example
|
||||
# Durations are Go strings (30s, 1h, 168h); negative values prevent startup
|
||||
# (observers.delete_after excepted, where nonpositive disables it).
|
||||
|
||||
@@ -263,7 +264,7 @@ websocket:
|
||||
|
||||
# CORS configuration.
|
||||
# Controls which origins, methods and headers are allowed for cross-origin requests.
|
||||
# Defaults to allowing all origins with GET/HEAD/OPTIONS if omitted — appropriate
|
||||
# Defaults to allowing all origins with GET/HEAD/OPTIONS if omitted, appropriate
|
||||
# for a public read-only API. Restrict allowed_origins if you expose write endpoints.
|
||||
#cors:
|
||||
# allowed_origins:
|
||||
|
||||
@@ -1,178 +0,0 @@
|
||||
# Database and saved-config export and verification
|
||||
|
||||
`beacon-backup` is the export and archive-validation foundation for issue #72. It
|
||||
produces a private, versioned `.tar.gz` using PostgreSQL's `pg_dump` and can verify
|
||||
an existing bundle offline. It does not yet provide a web interface, account
|
||||
login, scheduled/remote storage or automatic import.
|
||||
|
||||
Build it with `go build ./cmd/beacon-backup`. For exports, install `pg_dump` in the
|
||||
same runtime as this command; an installation on the Docker host does not install
|
||||
it inside an app container. Use a client of the same major version as the source
|
||||
PostgreSQL server; an older client cannot dump a newer server.
|
||||
|
||||
Set libpq's standard `PGHOST`, `PGPORT`, `PGDATABASE`, `PGUSER` and TLS settings.
|
||||
`PGDATABASE` is required and must be a plain database name. Prefer a private
|
||||
`PGPASSFILE` (0600 on Unix) or an existing libpq service configuration for secrets.
|
||||
The command does not load `.env`, read `POSTGRES_DSN`, start Beacon, run migrations,
|
||||
subscribe to MQTT or connect to Redis. Connection settings are not command-line
|
||||
arguments and client stderr is not printed because it can contain private data.
|
||||
|
||||
For example, after configuring those connection settings:
|
||||
|
||||
```sh
|
||||
beacon-backup -config /private/config.yaml -output /private/beacon-20260913.tar.gz
|
||||
```
|
||||
|
||||
Use an output directory controlled by the operator. Staging directories are
|
||||
0700 and files are 0600 on Unix; Windows operators must use a directory with
|
||||
appropriately restricted ACLs. Existing destinations, including symlinks, are
|
||||
never replaced. A hard link publishes the finished archive atomically, so the
|
||||
destination filesystem must support hard links. Unsupported filesystems fail
|
||||
without publishing an output. Normal failure, timeout and handled interruption
|
||||
remove temporary files; a power loss or SIGKILL can leave a private
|
||||
`.beacon-backup-*` staging directory for the operator to inspect.
|
||||
|
||||
## Protected download API
|
||||
|
||||
Set `backup.enabled: true` in saved YAML and configure the existing admin bearer
|
||||
key to enable `GET /api/v1/admin/backup`. Supply the key only in the `Authorization:
|
||||
Bearer ...` header over HTTPS. No query parameters or request body are accepted.
|
||||
This is an operator API, not account login or a browser admin panel. Keep it off
|
||||
on public previews that do not need private exports.
|
||||
|
||||
The server requires `pg_dump` on its own PATH when this option is enabled. The
|
||||
Docker image includes PostgreSQL 16's client by default. Its Alpine 3.19 base does
|
||||
not provide PostgreSQL 17/18 packages: changing `POSTGRES_CLIENT_MAJOR` alone is
|
||||
not an upgrade path. For a newer database, supply a runtime with a compatible
|
||||
client. Startup compares `pg_dump --version` with `server_version_num` and leaves
|
||||
backup unavailable if the client is missing, unrecognized or older than the server.
|
||||
Match the source database major version. A client on the host or in a different container is not
|
||||
sufficient. No Docker socket or host-command bridge is used.
|
||||
|
||||
The download uses the startup `POSTGRES_DSN`, not the standalone export's ambient
|
||||
`PGDATABASE`. This first adapter supports a single-host `postgres://` or
|
||||
`postgresql://` URL with an explicit user and database. Passwords, IPv6 and URL
|
||||
escaping are supported. It copies TLS settings, `passfile`, `connect_timeout`,
|
||||
`application_name`, `target_session_attrs` and `options`; known `pool_*` controls
|
||||
are omitted from libpq. Supported PG environment defaults are captured at startup
|
||||
and URL values override them. Prefer explicit `sslmode=verify-full` and a trusted
|
||||
`sslrootcert` for remote databases; password-file/TLS paths must be readable in
|
||||
the same runtime. Their contents are not included in the archive.
|
||||
|
||||
Keyword DSNs, service indirection, multiple hosts, duplicate/unknown query options,
|
||||
newer protocol negotiation settings and service-file values with line breaks or
|
||||
edge whitespace are rejected rather than silently changing targets or TLS policy.
|
||||
Ambient `PGSERVICE`, `PGSERVICEFILE`, `PGSSLNEGOTIATION`, `PGMINPROTOCOLVERSION`,
|
||||
`PGMAXPROTOCOLVERSION` and `PGTZ` are also unsupported for this adapter. The
|
||||
standalone command retains native PG* support for those deployments. Errors do
|
||||
not echo the DSN. Any failed backup prerequisite logs its reason and leaves only
|
||||
this feature unavailable; Beacon still starts ingest, WebSocket and other APIs.
|
||||
Fix the prerequisite and restart to enable downloads. Native connection settings live only in a 0600 service file in
|
||||
private staging; process arguments contain no connection string or credential,
|
||||
and no request changes Beacon's environment.
|
||||
|
||||
Only one export or transfer is allowed at a time; another request receives 409.
|
||||
The existing ten-minute/1 GiB SQL/1 MiB YAML limits apply, and disconnecting
|
||||
cancels an in-progress export. The completed transfer has a ten-minute write
|
||||
deadline. Temporary files are removed after completion/failure/cancellation;
|
||||
abrupt process termination can leave private staging. Responses use `no-store`
|
||||
and a fixed attachment filename. A failed export returns an error without any
|
||||
partial archive. Authentication missing/invalid is 401, unconfigured access is
|
||||
503, invalid input is 400, export failure is 500, timeout is 504 and exceeding the
|
||||
configured export size limit is 507. Other export failures retain sanitized
|
||||
operator diagnostics in the server log; raw pg_dump stderr remains discarded.
|
||||
|
||||
Provision a private writable temporary directory (`TMPDIR` on Unix) with space
|
||||
for roughly twice the SQL limit plus overhead. A small read-only-container tmpfs
|
||||
is insufficient for a large backup. Exports include saved YAML verbatim and may
|
||||
include keys/message data; the exclusions below still apply. This API does not
|
||||
implement archive upload, import/overwrite, schedules or remote storage.
|
||||
|
||||
## Format 1
|
||||
|
||||
The tar contains exactly three regular files with fixed names:
|
||||
|
||||
- `manifest.json`: format version, creation time, exporter version, dump format,
|
||||
uncompressed payload sizes/SHA-256 hashes and explicit exclusions.
|
||||
- `database.sql`: one consistent `pg_dump` snapshot of schema and data, including
|
||||
Beacon's migration journal. Ownership, ACLs and tablespace placement are omitted
|
||||
so objects can be restored under the destination operator.
|
||||
- `config.yaml`: the supplied saved file, byte-for-byte, including comments and
|
||||
any keys. It is captured before the database snapshot; avoid configuration edits
|
||||
during export if the files must describe the same deployment state.
|
||||
|
||||
This is sensitive, unencrypted data. Store and transfer it privately. The bundle
|
||||
does not include deployment environment variables, `.env`, external files such as
|
||||
`borderFile` inputs or TLS keys, runtime-only changes, PostgreSQL roles/cluster
|
||||
settings, Redis or service/deployment files. Retain those separately. A config
|
||||
export is not the sanitized admin-config response and is not a complete server
|
||||
recovery package by itself.
|
||||
|
||||
Defaults are ten minutes and a 1 GiB uncompressed SQL limit. Saved YAML is capped
|
||||
at 1 MiB. `-timeout` and `-max-bytes` set finite positive limits (SQL maximum 1 TiB).
|
||||
Allow disk space for both the uncompressed SQL and compressed bundle, roughly
|
||||
twice the chosen SQL limit plus overhead. A five-second lock-wait limit prevents
|
||||
waiting indefinitely behind schema changes. Export failure publishes no backup;
|
||||
check the client version, privileges, connection settings and available capacity
|
||||
privately. The command deliberately does not expose raw client diagnostics.
|
||||
|
||||
## Verify an existing archive
|
||||
|
||||
```sh
|
||||
beacon-backup -verify /private/beacon-20260913.tar.gz
|
||||
```
|
||||
|
||||
Verification reads a regular file without extracting members, writing files,
|
||||
executing SQL or connecting to PostgreSQL, Redis or MQTT. It needs neither a
|
||||
configuration file nor `pg_dump` or connection credentials. Do not combine
|
||||
`-verify` with the export-only `-config` or `-output` flags.
|
||||
|
||||
The verifier accepts the native format-1 bundle: exactly the three fixed regular
|
||||
members with 0600 modes and USTAR headers (or the size-only PAX header needed for
|
||||
SQL over 8 GiB). It rejects duplicates, unexpected paths, links, devices, other
|
||||
extended metadata, incomplete tar/gzip endings and trailing or concatenated
|
||||
streams. Manifest fields must be known, with no duplicate keys or case aliases;
|
||||
the payload sizes and SHA-256 hashes must match the bytes actually read.
|
||||
|
||||
The same `-max-bytes` SQL limit applies, with a separate 1 MiB config limit and
|
||||
64 KiB manifest limit. Compressed input is capped at the SQL limit plus 1% and
|
||||
2 MiB, matching the exporter. Headers and decompressed data are bounded too;
|
||||
SQL and config are hashed as streams, without keeping their contents in memory.
|
||||
The ten-minute default `-timeout` and interrupt handling check cancellation
|
||||
between reads; they cannot interrupt a blocked filesystem read.
|
||||
|
||||
Exit status 0 means the archive structure and checksums passed. Failure returns
|
||||
a nonzero status with no payload, manifest values or input path in its diagnostic.
|
||||
The manifest is not signed: verification does **not** establish authenticity,
|
||||
safe SQL, configuration validity or restorability. It does not make an untrusted
|
||||
dump safe to restore, and does not provide import/overwrite or deployment recovery.
|
||||
|
||||
## Restore verification
|
||||
|
||||
Use trusted bundles only: PostgreSQL dumps can contain executable SQL. Run the
|
||||
archive verification above, then extract into private staging. For a
|
||||
**new, empty disposable database**, with its own explicit `PGDATABASE` and
|
||||
target-role connection settings:
|
||||
|
||||
```sh
|
||||
psql -X --set ON_ERROR_STOP=on --single-transaction --file database.sql
|
||||
```
|
||||
|
||||
Restore with a compatible PostgreSQL version and the required extensions already
|
||||
available. Reconcile the migration journal and representative records/relationships
|
||||
before relying on the bundle. Review saved configuration and restore external
|
||||
secrets/files separately before starting a new Beacon server. Never restore over
|
||||
live data simply to test an export; overwrite/import requires a separate workflow.
|
||||
|
||||
PostgreSQL references: [pg_dump](https://www.postgresql.org/docs/16/app-pgdump.html),
|
||||
[connection environment](https://www.postgresql.org/docs/16/libpq-envars.html),
|
||||
[password files](https://www.postgresql.org/docs/16/libpq-pgpass.html).
|
||||
|
||||
CI runs the compiled command and its PostgreSQL round-trip test with a dedicated
|
||||
PostgreSQL 16 service. The test also verifies the export offline with no client on
|
||||
PATH or connection settings, and rejects a truncated copy. To repeat it privately,
|
||||
build the command, set the `PG*` connection variables for an isolated test server and set
|
||||
`BEACON_BACKUP_TEST_POSTGRES=1` plus `BEACON_BACKUP_TEST_BINARY` to the command's
|
||||
absolute path. Run `go test ./internal/backup -run '^TestExportPostgres$' -v`.
|
||||
The test role needs permission to create/drop its two randomly named databases;
|
||||
the test migrates and restores only those databases and removes them afterward.
|
||||
@@ -1,28 +0,0 @@
|
||||
# Packet summaries
|
||||
|
||||
Packet lists, regional lists, reconnect backfill and live `packetObservation`
|
||||
events share an optional `summary` string. It describes metadata carried by that
|
||||
packet, rather than looking up a node's current name or decoding history again.
|
||||
|
||||
| Payload | Summary | Source |
|
||||
|---|---|---|
|
||||
| ADVERT | Advertised name | Saved `appData.name` / verified decoded advert |
|
||||
| ACK | `ACK 01020304` | Four-byte acknowledgement checksum |
|
||||
| TRACE | `TRACE efbeadde` | Trace tag |
|
||||
| TRACE classified as PING | `PING efbeadde` | Trace tag, with the existing single-destination classification |
|
||||
|
||||
References use eight lowercase hexadecimal characters in the same byte order
|
||||
as packet detail's `checksum` / `traceTag`. Extended ACK retry bytes do not change
|
||||
the displayed checksum. Zero is a valid reference. These short references are
|
||||
not globally unique packet identifiers or proof of delivery/authentication; no
|
||||
message body, trace auth code or inferred sender/recipient is added to the summary.
|
||||
|
||||
Historical reads require the expected stored type and a string containing
|
||||
exactly eight hex characters. Missing, malformed or unsupported references omit
|
||||
the field. Advert behavior is unchanged. List/backfill queries project the text
|
||||
from saved JSON in their existing query, without a schema change, backfill job
|
||||
or per-packet database lookup. The existing web summary display needs no new API
|
||||
field or client release to render these values.
|
||||
|
||||
Other packet types remain separate follow-ups under issue #99. A missing summary
|
||||
does not mean the packet is invalid or unreadable in packet detail.
|
||||
+4
-1
@@ -1,3 +1,6 @@
|
||||
# Local development starter. Every variable is documented at
|
||||
# https://github.com/MeshCore-Beacon/beacon-docs/blob/main/docs/configuration.md
|
||||
|
||||
LISTEN_ADDR=:8080
|
||||
|
||||
# Path to the YAML config (default: config.yaml).
|
||||
@@ -13,7 +16,7 @@ LISTEN_ADDR=:8080
|
||||
|
||||
POSTGRES_DSN=postgres://beacon:beacon@localhost:5432/beacon?sslmode=disable
|
||||
|
||||
# Redis (optional — leave REDIS_ADDR unset to disable caching)
|
||||
# Redis (optional; leave REDIS_ADDR unset to disable caching)
|
||||
REDIS_ADDR=localhost:6379
|
||||
REDIS_PASSWORD=
|
||||
REDIS_DB=0
|
||||
|
||||
@@ -29945,7 +29945,7 @@
|
||||
599256,"GB-1333","heliport","Clifton Dykes",54.644211,-2.69254,,"EU","GB","GB-U-A",,"no",,,,,,,
|
||||
599308,"GB-1334","closed","RAF Long Kesh",54.490669,-6.103835,115,"EU","GB","GB-NIR","Maze, County Down","no",,,,,,"https://en.wikipedia.org/wiki/RAF_Long_Kesh",
|
||||
599309,"GB-1335","heliport","Lisburn Heliport",54.49328,-6.097745,118,"EU","GB","GB-NIR","Lisburn, County Down","no",,,,,,,
|
||||
599921,"GB-1336","small_airport","(Duplicete)East Kirkby Airfield",53.13889,-0.00056,39,"EU","GB","GB-ENG","East Kirkby","no",,,,,"https://lincsaviation.co.uk","https://en.m.wikipedia.org/wiki/Lincolnshire_Aviation_Heritage_Centre?utm_source=chatgpt.com","RAF East Kirkby; Lincolnshire Aviation Heritage Centre; Bomber Command; Avro Lancaster NX611 “Just Jane”; Mosquito HJ711; East Lindsey; WW2 airfield; museum; heritage."
|
||||
599921,"GB-1336","small_airport","(Duplicete)East Kirkby Airfield",53.13889,-0.00056,39,"EU","GB","GB-ENG","East Kirkby","no",,,,,"https://lincsaviation.co.uk","https://en.m.wikipedia.org/wiki/Lincolnshire_Aviation_Heritage_Centre","RAF East Kirkby; Lincolnshire Aviation Heritage Centre; Bomber Command; Avro Lancaster NX611 “Just Jane”; Mosquito HJ711; East Lindsey; WW2 airfield; museum; heritage."
|
||||
599951,"GB-1337","closed","RAF Woodhall Spa",53.131667,-0.18444,33,"EU","GB","GB-ENG","East Linsdey, Lincolnshire","no",,,,,"https://www.lincstrust.org.uk/get-involved/top-reserves/woodhall-spa-airfield","https://en.wikipedia.org/wiki/RAF_Woodhall_Spa","Tattershall Thorpe, No. 617 Squadron, Dambusters, Bristol Bloodhound Missile site, Thorpe Camp Visitor Centre, Lincolnshire Wildlife Trust, RAF Coningsby satellite"
|
||||
600636,"GB-1338","heliport","Chequers Helicopter Landing Ground",51.742163,-0.781822,,"EU","GB","GB-ENG","Ellesborough, Bucks.","no",,,,,,,
|
||||
602208,"GB-1339","heliport","Uxbridge Business Park Heliport",51.554853,-0.483221,112,"EU","GB","GB-ENG","Uxbridge, Greater London","no",,,,,,,
|
||||
@@ -30972,7 +30972,7 @@
|
||||
30801,"HRYI","small_airport","Butare Airport",-2.59583,29.7367,5801,"AF","RW","RW-05","Butare","no","HRYI","BTQ","HRYI",,,"https://en.wikipedia.org/wiki/Butare_Airport",
|
||||
31612,"HRYN","small_airport","Nemba Airport",-2.336167,30.206658,4905,"AF","RW","RW-02","Nemba","no","HRYN",,"HRYN",,,"https://en.wikipedia.org/wiki/Nemba_Airport",
|
||||
31613,"HRYO","small_airport","Gabiro Airport",-1.5499999523162842,30.382999420166016,4806,"AF","RW","RW-02","Gabiro","no",,,"HRYO",,,,
|
||||
3237,"HRYR","large_airport","Kigali International Airport",-1.96863,30.1395,4859,"AF","RW","RW-01","Kigali","yes","HRYR","KGL","HRYR",,"https://www.rac.co.rw/?utm_source=chatgpt.com","https://en.wikipedia.org/wiki/Kigali_International_Airport","Gregoire Kayibanda International Airport, Kanombe International Airport"
|
||||
3237,"HRYR","large_airport","Kigali International Airport",-1.96863,30.1395,4859,"AF","RW","RW-01","Kigali","yes","HRYR","KGL","HRYR",,"https://www.rac.co.rw/","https://en.wikipedia.org/wiki/Kigali_International_Airport","Gregoire Kayibanda International Airport, Kanombe International Airport"
|
||||
32208,"HRYU","small_airport","Ruhengeri Airport",-1.496873,29.631343,6102,"AF","RW","RW-03","Ruhengeri","no","HRYU","RHG","HRYU",,,"https://en.wikipedia.org/wiki/Ruhengeri_Airport",
|
||||
3238,"HRZA","medium_airport","Kamembe Airport",-2.462239980697632,28.907899856567383,5192,"AF","RW","RW-04","Kamembe","yes","HRZA","KME","HRZA",,,"https://en.wikipedia.org/wiki/Kamembe_Airport",
|
||||
43976,"HSAK","small_airport","Akobo Airport",7.7779,33.0027,1352,"AF","SS","SS-JG","Akobo","no",,,"HJAK",,,"https://en.wikipedia.org/wiki/Akobo_Airport","HSAK, Ahobo"
|
||||
|
||||
|
Can't render this file because it is too large.
|
Reference in New Issue
Block a user