mirror of
https://github.com/MeshCore-Beacon/beacon-server.git
synced 2026-09-01 16:48:19 +00:00
docs: update readme and contribution
This commit is contained in:
Enot (ded) Skelly
+212
-26
@@ -1,51 +1,237 @@
|
||||
# Contributing to Beacon
|
||||
|
||||
Thank you for your interest in contributing. Beacon is a focused project and we
|
||||
want contributions to be high quality and sustainable. Please read this guide
|
||||
before opening a PR.
|
||||
|
||||
---
|
||||
|
||||
## Before you start
|
||||
|
||||
**Open or comment on an issue before starting work.** This avoids duplicate
|
||||
effort and lets maintainers flag if something is already in progress or out of
|
||||
scope. For small bug fixes a brief comment is fine; for larger features please
|
||||
discuss the approach first.
|
||||
|
||||
**One thing per PR.** Each pull request should cover one logical change — a bug
|
||||
fix, a new endpoint, a refactor, a new test. PRs that touch many unrelated parts
|
||||
of the codebase are hard to review and hard to revert if something goes wrong.
|
||||
|
||||
**No fully AI-generated contributions.** We welcome developers who use AI tools
|
||||
to assist their work, but PRs should reflect the author's own understanding and
|
||||
judgement. PRs that appear to be unreviewed AI output may be closed without
|
||||
further comment.
|
||||
|
||||
---
|
||||
|
||||
## Branches
|
||||
|
||||
- `main` — stable releases only, protected
|
||||
- `dev` — active development, all PRs target this branch
|
||||
- `main` — stable releases only, protected. Never target this directly.
|
||||
- `dev` — active development. All PRs target `dev`.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Fork or create a branch from `dev`
|
||||
2. Make your changes
|
||||
3. Open a pull request against `dev`
|
||||
4. One commit per PR (squash before opening or use squash merge)
|
||||
3. Run the checklist below
|
||||
4. Open a pull request against `dev` with a clear description of what changed
|
||||
and why, referencing any related issues
|
||||
|
||||
---
|
||||
|
||||
## Checklist before opening a PR
|
||||
|
||||
```
|
||||
go build ./... # must compile
|
||||
gofmt -l . # must be empty (no unformatted files)
|
||||
go vet ./... # no warnings
|
||||
go test ./... # all tests pass
|
||||
swag init # if you changed any handler or api type (see below)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Code style
|
||||
|
||||
- Run `gofmt -w .` before committing — CI will fail on unformatted files
|
||||
- Run `go vet ./...` — no warnings
|
||||
- Follow the existing patterns in each package before introducing new ones
|
||||
- Keep functions small and single-purpose
|
||||
- Prefer explicit error handling over panic
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
- **Add tests for any new pure functions.** Pure functions (no DB, no network,
|
||||
no side effects) should have unit tests. See `internal/hub/hub_test.go`,
|
||||
`internal/api/nodes_test.go`, and `internal/keystore/keystore_test.go` for
|
||||
examples of the style we use.
|
||||
- Integration tests (requiring a real DB) are not yet required but are welcome.
|
||||
They will be gated before release.
|
||||
- Run `go test ./...` before opening a PR. All tests must pass.
|
||||
- If you are fixing a bug, add a test that would have caught it.
|
||||
|
||||
---
|
||||
|
||||
## Database changes
|
||||
|
||||
All schema changes must include a proper migration path:
|
||||
|
||||
- Add SQL to `db/migrations/001_schema.sql` (we use a single migration file for
|
||||
now — append to the appropriate section with a comment)
|
||||
- Update `db/queries/queries.sql` with any new or modified queries
|
||||
- Re-run `sqlc generate` to regenerate `db/sqlc/`
|
||||
- Update the store layer in `db/` to expose the new functionality
|
||||
- Update `internal/api/reader.go` if the change needs to be exposed via the API
|
||||
|
||||
Never edit files under `db/sqlc/` by hand — they are generated by sqlc and will
|
||||
be overwritten. If you need to work around a sqlc limitation, document it
|
||||
clearly in the query comment.
|
||||
|
||||
To regenerate after modifying `db/queries/queries.sql`:
|
||||
|
||||
```bash
|
||||
sqlc generate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API changes
|
||||
|
||||
Any new or modified REST endpoint must have swagger annotations and regenerated
|
||||
docs:
|
||||
|
||||
- Add or update `// @Summary`, `// @Param`, `// @Success`, `// @Failure`, and
|
||||
`// @Router` comments on the handler function
|
||||
- Response types are defined in `internal/api/` — add new types there, not
|
||||
inline in handlers
|
||||
- After changing any handler or API type, regenerate the swagger docs and commit
|
||||
the updated `docs/` directory alongside your changes:
|
||||
|
||||
```bash
|
||||
swag init -g cmd/beacon/main.go -o docs --parseInternal --parseDependency
|
||||
```
|
||||
|
||||
Install swag if you don't have it:
|
||||
|
||||
```bash
|
||||
go install github.com/swaggo/swag/cmd/swag@latest
|
||||
```
|
||||
|
||||
Each handler function should have a godoc-style annotation block:
|
||||
|
||||
```go
|
||||
// listThings godoc
|
||||
//
|
||||
// @Summary Short description shown in the UI
|
||||
// @Tags TagName
|
||||
// @Produce json
|
||||
// @Param paramName query string false "Description"
|
||||
// @Param id path string true "Resource ID"
|
||||
// @Success 200 {object} api.MyResponseType
|
||||
// @Failure 400 {object} handlers.APIError
|
||||
// @Failure 500 {object} handlers.APIError
|
||||
// @Router /things [get]
|
||||
func listThings(reader api.Reader) http.HandlerFunc {
|
||||
```
|
||||
|
||||
For paginated responses use the generic page wrapper:
|
||||
|
||||
```go
|
||||
// @Success 200 {object} api.Page[api.MyType]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Updating the IATA database
|
||||
|
||||
Beacon includes a static IATA → country/continent mapping compiled into the
|
||||
binary, generated from the [OurAirports](https://ourairports.com/data/) public
|
||||
dataset.
|
||||
|
||||
To refresh it with the latest airport data:
|
||||
|
||||
```bash
|
||||
rm internal/iatadb/gen/airports.csv
|
||||
go generate ./internal/iatadb/
|
||||
```
|
||||
|
||||
This fetches a fresh `airports.csv` from OurAirports, saves it locally, and
|
||||
regenerates `internal/iatadb/db.go`. Commit both files.
|
||||
|
||||
To use a local CSV instead (e.g. in a restricted network environment):
|
||||
|
||||
```bash
|
||||
AIRPORTS_CSV=/path/to/airports.csv go run ./internal/iatadb/gen
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a new dependency
|
||||
|
||||
- Run `go get <module>` and `go mod tidy`
|
||||
- Add an entry to [SHOULDERS.md](SHOULDERS.md) with a brief description of what
|
||||
the dependency does and why it was added
|
||||
|
||||
---
|
||||
|
||||
## Commit messages
|
||||
|
||||
Use the conventional commits format:
|
||||
|
||||
- `feat: add route search endpoint`
|
||||
- `fix: correct lat/lon divisor for advert payloads`
|
||||
- `chore: update airports.csv`
|
||||
- `docs: update README project layout`
|
||||
```
|
||||
feat(routes): add observation_count to known routes response
|
||||
fix(ingest): correct lat/lon divisor for advert payloads
|
||||
refactor(hub): collapse RegionIATAs into IATAs in Scope
|
||||
test(api): add unit tests for NodeTypeName and NodeTypeFromString
|
||||
chore: update airports.csv
|
||||
docs: expand CONTRIBUTING.md
|
||||
```
|
||||
|
||||
## Code style
|
||||
Scopes are optional but helpful for larger codebases. Common scopes: `api`,
|
||||
`db`, `ingest`, `hub`, `ws`, `handlers`, `config`, `keystore`.
|
||||
|
||||
- Run `gofmt -w .` before committing
|
||||
- Run `go vet ./...` — no warnings
|
||||
- Run `go build ./...` — must compile
|
||||
---
|
||||
|
||||
## Tests
|
||||
## Project structure
|
||||
|
||||
- Add tests for any new pure functions
|
||||
- Run `go test ./...` before opening a PR
|
||||
- Integration tests are not yet required but welcome
|
||||
```
|
||||
cmd/beacon/ — main entry point, wiring, startup
|
||||
db/ — store layer: sqlc-generated code + thin mapping layer
|
||||
migrations/ — SQL schema (single file, append only)
|
||||
queries/ — SQL queries (input to sqlc)
|
||||
sqlc/ — generated Go code (do not edit by hand)
|
||||
internal/
|
||||
api/ — response types and Reader interface
|
||||
handlers/ — HTTP handlers (validation, routing, response)
|
||||
router/ — chi router wiring
|
||||
config/ — config loading and scope key derivation
|
||||
hub/ — WebSocket fan-out broker
|
||||
ingest/ — MQTT packet ingestion and side effects
|
||||
iatadb/ — in-memory IATA airport lookup
|
||||
keystore/ — channel key lookup
|
||||
scopestore/ — transport scope key lookup
|
||||
ws/ — WebSocket connection handling
|
||||
docs/ — generated swagger docs (do not edit by hand)
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
Key patterns to understand before contributing:
|
||||
|
||||
When adding a new dependency please add it to [SHOULDERS.md](SHOULDERS.md) with
|
||||
a brief description of what it does.
|
||||
- **Store layer** (`db/`): thin wrappers around sqlc-generated queries. Each
|
||||
method maps between the ingest/api param structs and sqlc param structs. Never
|
||||
put business logic here.
|
||||
- **Ingest layer** (`internal/ingest/`): processes raw MQTT packets, calls the
|
||||
store, and broadcasts hub events. The `DB` interface in `ingest.go` defines
|
||||
exactly what the ingest layer needs from the store — keep it minimal.
|
||||
- **Hub** (`internal/hub/`): pure fan-out broker. Events are pre-serialised JSON
|
||||
before entering the hub so the broadcast loop never touches encoding.
|
||||
- **Reader interface** (`internal/api/reader.go`): defines everything the API
|
||||
layer can read. The store implements it. All handler tests use a stub reader.
|
||||
|
||||
## Pull requests
|
||||
|
||||
- Target `dev` not `main`
|
||||
- One logical change per PR
|
||||
- Include a brief description of what changed and why
|
||||
- Reference any related issues
|
||||
---
|
||||
|
||||
## Releases
|
||||
|
||||
Merges from `dev` to `main` are done by maintainers and represent a versioned
|
||||
release.
|
||||
release. Do not open PRs directly against `main`.
|
||||
|
||||
@@ -4,15 +4,15 @@ MeshCore Beacon is a MeshCore network observation backend. It connects to one or
|
||||
more MeshCore MQTT brokers, ingests LoRa packet traffic in real time, stores it
|
||||
in PostgreSQL, and streams live events to WebSocket clients.
|
||||
|
||||
[](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/ci.yml) -
|
||||
[](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/ci.yml)
|
||||
[](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/docker-publish.yml)
|
||||
|
||||
## What it does
|
||||
|
||||
- Subscribes to MeshCore MQTT brokers and decodes incoming LoRa packets using
|
||||
[meshcore-go](https://github.com/meshcore-go/meshcore-go)
|
||||
- Stores packets, observations, nodes, observers, and channel messages in
|
||||
PostgreSQL
|
||||
- Stores packets, observations, nodes, observers, traces, routes and channel
|
||||
messages in PostgreSQL (more backends to come)
|
||||
- Deduplicates observations across multiple brokers (same packet heard by two
|
||||
brokers is one observation per observer)
|
||||
- Decrypts group text messages for known channel keys
|
||||
@@ -43,40 +43,12 @@ For deployment instructions including the frontend app, see the deployment docs.
|
||||
|
||||
---
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
beacon-server/
|
||||
├── cmd/beacon/ entry point
|
||||
├── db/ store implementations and sqlc generated code
|
||||
│ ├── migrations/ SQL schema
|
||||
│ ├── queries/ sqlc query definitions
|
||||
│ └── sqlc/ generated Go DB code (do not edit)
|
||||
├── internal/
|
||||
│ ├── api/ REST API types, Reader interface, route handlers
|
||||
│ │ └── handlers/ HTTP route handlers
|
||||
│ ├── config/ config file loading and DB seeding
|
||||
│ ├── hub/ WebSocket fan-out broker
|
||||
│ ├── iatadb/ static IATA → country/continent map (generated)
|
||||
│ ├── ingest/ MQTT ingest pipeline
|
||||
│ ├── keystore/ channel key store
|
||||
│ ├── scopestore/ transport scope key store
|
||||
│ └── ws/ WebSocket handler and IP limiter
|
||||
├── config.yaml.example
|
||||
├── env.example
|
||||
├── docker-compose.yml
|
||||
└── sqlc.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Go 1.26+
|
||||
- Docker and Docker Compose
|
||||
- [sqlc](https://sqlc.dev) (only needed if modifying queries)
|
||||
|
||||
### 1. Clone and configure
|
||||
|
||||
@@ -105,6 +77,12 @@ start via `docker-entrypoint-initdb.d`.
|
||||
go run ./cmd/beacon
|
||||
```
|
||||
|
||||
Or pull and run the Docker image:
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/meshcore-beacon/beacon-server:latest
|
||||
```
|
||||
|
||||
Beacon will:
|
||||
|
||||
- Load `.env` and `config.yaml`
|
||||
@@ -210,6 +188,14 @@ not auto-created.
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
API authentication is not yet implemented. Beacon is intended for trusted
|
||||
internal network or reverse-proxy deployments. Do not expose it directly to the
|
||||
public internet without an authentication layer in front of it.
|
||||
|
||||
---
|
||||
|
||||
## WebSocket API
|
||||
|
||||
Connect to `ws://host:8080/ws`.
|
||||
@@ -308,13 +294,15 @@ configurable via `websocket.max_connections_per_ip` in `config.yaml`.
|
||||
|
||||
Base path: `/api/v1`
|
||||
|
||||
All list endpoints support `afterId` for cursor-based pagination:
|
||||
All list endpoints support cursor-based pagination via `cursor` and `limit`
|
||||
query params. See the Swagger UI at `http://localhost:8080/swagger/index.html`
|
||||
for full parameter documentation.
|
||||
|
||||
```
|
||||
GET /api/v1/packets?iata=YOW&afterId=12345&limit=100
|
||||
```
|
||||
### Authentication
|
||||
|
||||
### Implemented
|
||||
Not yet implemented — see the Authentication section above.
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ----------------------------------- | -------------------------------------------------------------------------------------------------- |
|
||||
@@ -333,7 +321,7 @@ GET /api/v1/packets?iata=YOW&afterId=12345&limit=100
|
||||
| `GET` | `/observers` | List observers (optional: `?iata=<code>&type=<str>&broker=<name>&status=online\|offline`) |
|
||||
| `GET` | `/observers/{observerId}` | Get observer detail including broker last-seen timestamps |
|
||||
| `GET` | `/observers/{observerId}/adverts` | Adverts heard by observer |
|
||||
| `GET` | `/observers/{observerId}/telemetry` | Observer telemetry history |
|
||||
| `GET` | `/observers/{observerId}/telemetry` | Observer telemetry history (optional: `?range=24h&interval=1h\|6h\|24h`) |
|
||||
| `GET` | `/packets` | List packets with filters |
|
||||
| `GET` | `/packets/backfill` | Backfill packets after a given observation ID |
|
||||
| `GET` | `/packets/{packetHash}` | Get packet with all observations |
|
||||
@@ -341,6 +329,7 @@ GET /api/v1/packets?iata=YOW&afterId=12345&limit=100
|
||||
| `GET` | `/regions/{id}` | Get a single region with IATA list |
|
||||
| `GET` | `/routes` | List known routes (all hops high confidence) |
|
||||
| `GET` | `/routes/search` | Search routes by source and destination hash |
|
||||
| `GET` | `/routes/cross` | Search for routes crossing IATA boundaries |
|
||||
| `GET` | `/scopes` | List transport scopes |
|
||||
| `GET` | `/scopes/{name}` | Get scope detail |
|
||||
| `GET` | `/stats/observations` | Hourly observation time series (last 7 days by default) |
|
||||
@@ -354,140 +343,16 @@ GET /api/v1/packets?iata=YOW&afterId=12345&limit=100
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
### Modifying DB queries
|
||||
|
||||
Edit `db/queries/queries.sql`, then regenerate:
|
||||
|
||||
```bash
|
||||
sqlc generate
|
||||
```
|
||||
|
||||
### API documentation (Swagger)
|
||||
|
||||
Beacon uses [swaggo/swag](https://github.com/swaggo/swag) to generate OpenAPI
|
||||
documentation from annotations in the handler comments.
|
||||
|
||||
Start the server and open `http://localhost:8080/swagger/index.html`.
|
||||
|
||||
After adding or modifying any handler, regenerate the docs and commit the
|
||||
updated `docs/` directory alongside your handler changes:
|
||||
|
||||
```bash
|
||||
swag init -g cmd/beacon/main.go -o docs --parseDependecy
|
||||
```
|
||||
|
||||
Install swag:
|
||||
|
||||
```bash
|
||||
go install github.com/swaggo/swag/cmd/swag@latest
|
||||
```
|
||||
|
||||
Each handler closure should have a godoc-style annotation block immediately
|
||||
above the `r.Get()`/`r.Post()` call:
|
||||
|
||||
```go
|
||||
// listThings godoc
|
||||
//
|
||||
// @Summary Short description shown in the UI
|
||||
// @Tags TagName
|
||||
// @Produce json
|
||||
// @Param paramName query string false "Description"
|
||||
// @Param id path string true "Resource ID"
|
||||
// @Success 200 {object} api.MyResponseType
|
||||
// @Failure 400 {object} handlers.APIError
|
||||
// @Failure 500 {object} handlers.APIError
|
||||
// @Router /things [get]
|
||||
r.Get("/", func(w http.ResponseWriter, r *http.Request) {
|
||||
```
|
||||
|
||||
For paginated responses use the generic page wrapper:
|
||||
|
||||
```go
|
||||
// @Success 200 {object} api.Page[api.MyType]
|
||||
```
|
||||
|
||||
### Updating the IATA database
|
||||
|
||||
Beacon includes a static IATA → country/continent mapping compiled into the
|
||||
binary, generated from the [OurAirports](https://ourairports.com/data/) public
|
||||
dataset.
|
||||
|
||||
To refresh it with the latest airport data:
|
||||
|
||||
```bash
|
||||
rm internal/iatadb/gen/airports.csv
|
||||
go generate ./internal/iatadb/
|
||||
```
|
||||
|
||||
This fetches a fresh `airports.csv` from OurAirports, saves it locally, and
|
||||
regenerates `internal/iatadb/db.go`. Commit both files.
|
||||
|
||||
To use a local CSV instead (e.g. in a restricted network environment):
|
||||
|
||||
```bash
|
||||
AIRPORTS_CSV=/path/to/airports.csv go run ./internal/iatadb/gen
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Road Map
|
||||
|
||||
### Done
|
||||
|
||||
- [x] MQTT ingest pipeline (two brokers, cross-broker dedup)
|
||||
- [x] Packet decode via meshcore-go
|
||||
- [x] Observer upsert and status processing
|
||||
- [x] Node upsert from advert payloads
|
||||
- [x] Channel message storage with key-based decryption
|
||||
- [x] Firmware capability detection scaffolding
|
||||
- [x] Hub-based WebSocket fan-out with subscription filtering
|
||||
- [x] WebSocket server (hello, subscribe, unsubscribe, ping/pong, lagged,
|
||||
events)
|
||||
- [x] WebSocket regionId expansion via region_iatas DB lookup
|
||||
- [x] WebSocket per-IP connection limits
|
||||
- [x] Config file loading (regions, IATA overrides, channel keys)
|
||||
- [x] Observer radio settings on observations
|
||||
- [x] DB seeding on startup
|
||||
- [x] Observer telemetry storage with configurable resolution and retention
|
||||
- [x] Packet retention cleanup goroutine
|
||||
- [x] Hashtag channel PSK derivation (SHA256("#tag")[:16])
|
||||
- [x] Channel hash collision handling via key fingerprint
|
||||
- [x] REST API: IATAs, Regions
|
||||
- [x] REST API: Channels (list + detail + messages) with IATA filter
|
||||
- [x] REST API: Messages (cross-channel) with IATA filter
|
||||
- [x] REST API: Observers (heard adverts, telemetry, list + detail with broker
|
||||
last-seen)
|
||||
- [x] REST API: Brokers (list with connection status)
|
||||
- [x] REST API: Pagination
|
||||
- [x] REST API: Nodes (list + detail + observations)
|
||||
- [x] REST API: Packets (list + detail)
|
||||
- [x] REST API: Stats
|
||||
- [x] Materialized view refresh (mv_hourly_iata_stats, mv_top_nodes_by_iata)
|
||||
- [x] Swagger/OpenAPI documentation via swaggo/swag
|
||||
- [x] Path resolution (node short ID lookup)
|
||||
- [x] Parse payloads (that we can decrypt) into DB and return with packet
|
||||
details
|
||||
- [x] Propagation time calculation
|
||||
- [x] Trace route resolution via path hashes (resolvedRoute on packet detail)
|
||||
- [x] REST API: Trace packets: trace tag storage, list and detail endpoints with
|
||||
resolved routes
|
||||
- [x] REST API: Known routes: fully resolved paths stored at ingest, list and
|
||||
search endpoints
|
||||
- [x] Node neighbor detection and storage from advert path resolution
|
||||
- [x] REST API: Node neighbors endpoint
|
||||
|
||||
### Future
|
||||
|
||||
- [ ] Redis caching for stats endpoints
|
||||
- [ ] Caddy reverse proxy config for production
|
||||
|
||||
- [ ] Admin authentication middleware
|
||||
- [ ] Server management via API (currently config-file only)
|
||||
- [ ] Observer owner tracking (schema exists, API excluded by design)
|
||||
- [ ] Server management via API (currently config-file only)
|
||||
- [ ] Log levels, debug and info
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
Beacon stands on the shoulders of giants. See [SHOULDERS.md](SHOULDERS.md) for
|
||||
|
||||
Reference in New Issue
Block a user