diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6c625f5..535f364 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,51 +1,237 @@ # Contributing to Beacon +Thank you for your interest in contributing. Beacon is a focused project and we +want contributions to be high quality and sustainable. Please read this guide +before opening a PR. + +--- + +## Before you start + +**Open or comment on an issue before starting work.** This avoids duplicate +effort and lets maintainers flag if something is already in progress or out of +scope. For small bug fixes a brief comment is fine; for larger features please +discuss the approach first. + +**One thing per PR.** Each pull request should cover one logical change — a bug +fix, a new endpoint, a refactor, a new test. PRs that touch many unrelated parts +of the codebase are hard to review and hard to revert if something goes wrong. + +**No fully AI-generated contributions.** We welcome developers who use AI tools +to assist their work, but PRs should reflect the author's own understanding and +judgement. PRs that appear to be unreviewed AI output may be closed without +further comment. + +--- + ## Branches -- `main` — stable releases only, protected -- `dev` — active development, all PRs target this branch +- `main` — stable releases only, protected. Never target this directly. +- `dev` — active development. All PRs target `dev`. ## Workflow 1. Fork or create a branch from `dev` 2. Make your changes -3. Open a pull request against `dev` -4. One commit per PR (squash before opening or use squash merge) +3. Run the checklist below +4. Open a pull request against `dev` with a clear description of what changed + and why, referencing any related issues + +--- + +## Checklist before opening a PR + +``` +go build ./... # must compile +gofmt -l . # must be empty (no unformatted files) +go vet ./... # no warnings +go test ./... # all tests pass +swag init # if you changed any handler or api type (see below) +``` + +--- + +## Code style + +- Run `gofmt -w .` before committing — CI will fail on unformatted files +- Run `go vet ./...` — no warnings +- Follow the existing patterns in each package before introducing new ones +- Keep functions small and single-purpose +- Prefer explicit error handling over panic + +--- + +## Tests + +- **Add tests for any new pure functions.** Pure functions (no DB, no network, + no side effects) should have unit tests. See `internal/hub/hub_test.go`, + `internal/api/nodes_test.go`, and `internal/keystore/keystore_test.go` for + examples of the style we use. +- Integration tests (requiring a real DB) are not yet required but are welcome. + They will be gated before release. +- Run `go test ./...` before opening a PR. All tests must pass. +- If you are fixing a bug, add a test that would have caught it. + +--- + +## Database changes + +All schema changes must include a proper migration path: + +- Add SQL to `db/migrations/001_schema.sql` (we use a single migration file for + now — append to the appropriate section with a comment) +- Update `db/queries/queries.sql` with any new or modified queries +- Re-run `sqlc generate` to regenerate `db/sqlc/` +- Update the store layer in `db/` to expose the new functionality +- Update `internal/api/reader.go` if the change needs to be exposed via the API + +Never edit files under `db/sqlc/` by hand — they are generated by sqlc and will +be overwritten. If you need to work around a sqlc limitation, document it +clearly in the query comment. + +To regenerate after modifying `db/queries/queries.sql`: + +```bash +sqlc generate +``` + +--- + +## API changes + +Any new or modified REST endpoint must have swagger annotations and regenerated +docs: + +- Add or update `// @Summary`, `// @Param`, `// @Success`, `// @Failure`, and + `// @Router` comments on the handler function +- Response types are defined in `internal/api/` — add new types there, not + inline in handlers +- After changing any handler or API type, regenerate the swagger docs and commit + the updated `docs/` directory alongside your changes: + +```bash +swag init -g cmd/beacon/main.go -o docs --parseInternal --parseDependency +``` + +Install swag if you don't have it: + +```bash +go install github.com/swaggo/swag/cmd/swag@latest +``` + +Each handler function should have a godoc-style annotation block: + +```go +// listThings godoc +// +// @Summary Short description shown in the UI +// @Tags TagName +// @Produce json +// @Param paramName query string false "Description" +// @Param id path string true "Resource ID" +// @Success 200 {object} api.MyResponseType +// @Failure 400 {object} handlers.APIError +// @Failure 500 {object} handlers.APIError +// @Router /things [get] +func listThings(reader api.Reader) http.HandlerFunc { +``` + +For paginated responses use the generic page wrapper: + +```go +// @Success 200 {object} api.Page[api.MyType] +``` + +--- + +## Updating the IATA database + +Beacon includes a static IATA → country/continent mapping compiled into the +binary, generated from the [OurAirports](https://ourairports.com/data/) public +dataset. + +To refresh it with the latest airport data: + +```bash +rm internal/iatadb/gen/airports.csv +go generate ./internal/iatadb/ +``` + +This fetches a fresh `airports.csv` from OurAirports, saves it locally, and +regenerates `internal/iatadb/db.go`. Commit both files. + +To use a local CSV instead (e.g. in a restricted network environment): + +```bash +AIRPORTS_CSV=/path/to/airports.csv go run ./internal/iatadb/gen +``` + +--- + +## Adding a new dependency + +- Run `go get ` and `go mod tidy` +- Add an entry to [SHOULDERS.md](SHOULDERS.md) with a brief description of what + the dependency does and why it was added + +--- ## Commit messages Use the conventional commits format: -- `feat: add route search endpoint` -- `fix: correct lat/lon divisor for advert payloads` -- `chore: update airports.csv` -- `docs: update README project layout` +``` +feat(routes): add observation_count to known routes response +fix(ingest): correct lat/lon divisor for advert payloads +refactor(hub): collapse RegionIATAs into IATAs in Scope +test(api): add unit tests for NodeTypeName and NodeTypeFromString +chore: update airports.csv +docs: expand CONTRIBUTING.md +``` -## Code style +Scopes are optional but helpful for larger codebases. Common scopes: `api`, +`db`, `ingest`, `hub`, `ws`, `handlers`, `config`, `keystore`. -- Run `gofmt -w .` before committing -- Run `go vet ./...` — no warnings -- Run `go build ./...` — must compile +--- -## Tests +## Project structure -- Add tests for any new pure functions -- Run `go test ./...` before opening a PR -- Integration tests are not yet required but welcome +``` +cmd/beacon/ — main entry point, wiring, startup +db/ — store layer: sqlc-generated code + thin mapping layer + migrations/ — SQL schema (single file, append only) + queries/ — SQL queries (input to sqlc) + sqlc/ — generated Go code (do not edit by hand) +internal/ + api/ — response types and Reader interface + handlers/ — HTTP handlers (validation, routing, response) + router/ — chi router wiring + config/ — config loading and scope key derivation + hub/ — WebSocket fan-out broker + ingest/ — MQTT packet ingestion and side effects + iatadb/ — in-memory IATA airport lookup + keystore/ — channel key lookup + scopestore/ — transport scope key lookup + ws/ — WebSocket connection handling +docs/ — generated swagger docs (do not edit by hand) +``` -## Dependencies +Key patterns to understand before contributing: -When adding a new dependency please add it to [SHOULDERS.md](SHOULDERS.md) with -a brief description of what it does. +- **Store layer** (`db/`): thin wrappers around sqlc-generated queries. Each + method maps between the ingest/api param structs and sqlc param structs. Never + put business logic here. +- **Ingest layer** (`internal/ingest/`): processes raw MQTT packets, calls the + store, and broadcasts hub events. The `DB` interface in `ingest.go` defines + exactly what the ingest layer needs from the store — keep it minimal. +- **Hub** (`internal/hub/`): pure fan-out broker. Events are pre-serialised JSON + before entering the hub so the broadcast loop never touches encoding. +- **Reader interface** (`internal/api/reader.go`): defines everything the API + layer can read. The store implements it. All handler tests use a stub reader. -## Pull requests - -- Target `dev` not `main` -- One logical change per PR -- Include a brief description of what changed and why -- Reference any related issues +--- ## Releases Merges from `dev` to `main` are done by maintainers and represent a versioned -release. +release. Do not open PRs directly against `main`. diff --git a/README.md b/README.md index 06a0a47..b3079b0 100644 --- a/README.md +++ b/README.md @@ -4,15 +4,15 @@ MeshCore Beacon is a MeshCore network observation backend. It connects to one or more MeshCore MQTT brokers, ingests LoRa packet traffic in real time, stores it in PostgreSQL, and streams live events to WebSocket clients. -[![CI](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/ci.yml/badge.svg)](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/ci.yml) - +[![CI](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/ci.yml/badge.svg)](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/ci.yml) [![Docker](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/MeshCore-Beacon/beacon-server/actions/workflows/docker-publish.yml) ## What it does - Subscribes to MeshCore MQTT brokers and decodes incoming LoRa packets using [meshcore-go](https://github.com/meshcore-go/meshcore-go) -- Stores packets, observations, nodes, observers, and channel messages in - PostgreSQL +- Stores packets, observations, nodes, observers, traces, routes and channel + messages in PostgreSQL (more backends to come) - Deduplicates observations across multiple brokers (same packet heard by two brokers is one observation per observer) - Decrypts group text messages for known channel keys @@ -43,40 +43,12 @@ For deployment instructions including the frontend app, see the deployment docs. --- -## Project layout - -``` -beacon-server/ -├── cmd/beacon/ entry point -├── db/ store implementations and sqlc generated code -│ ├── migrations/ SQL schema -│ ├── queries/ sqlc query definitions -│ └── sqlc/ generated Go DB code (do not edit) -├── internal/ -│ ├── api/ REST API types, Reader interface, route handlers -│ │ └── handlers/ HTTP route handlers -│ ├── config/ config file loading and DB seeding -│ ├── hub/ WebSocket fan-out broker -│ ├── iatadb/ static IATA → country/continent map (generated) -│ ├── ingest/ MQTT ingest pipeline -│ ├── keystore/ channel key store -│ ├── scopestore/ transport scope key store -│ └── ws/ WebSocket handler and IP limiter -├── config.yaml.example -├── env.example -├── docker-compose.yml -└── sqlc.yaml -``` - ---- - ## Getting started ### Prerequisites - Go 1.26+ - Docker and Docker Compose -- [sqlc](https://sqlc.dev) (only needed if modifying queries) ### 1. Clone and configure @@ -105,6 +77,12 @@ start via `docker-entrypoint-initdb.d`. go run ./cmd/beacon ``` +Or pull and run the Docker image: + +```bash +docker pull ghcr.io/meshcore-beacon/beacon-server:latest +``` + Beacon will: - Load `.env` and `config.yaml` @@ -210,6 +188,14 @@ not auto-created. --- +## Authentication + +API authentication is not yet implemented. Beacon is intended for trusted +internal network or reverse-proxy deployments. Do not expose it directly to the +public internet without an authentication layer in front of it. + +--- + ## WebSocket API Connect to `ws://host:8080/ws`. @@ -308,13 +294,15 @@ configurable via `websocket.max_connections_per_ip` in `config.yaml`. Base path: `/api/v1` -All list endpoints support `afterId` for cursor-based pagination: +All list endpoints support cursor-based pagination via `cursor` and `limit` +query params. See the Swagger UI at `http://localhost:8080/swagger/index.html` +for full parameter documentation. -``` -GET /api/v1/packets?iata=YOW&afterId=12345&limit=100 -``` +### Authentication -### Implemented +Not yet implemented — see the Authentication section above. + +### Endpoints | Method | Path | Description | | ------ | ----------------------------------- | -------------------------------------------------------------------------------------------------- | @@ -333,7 +321,7 @@ GET /api/v1/packets?iata=YOW&afterId=12345&limit=100 | `GET` | `/observers` | List observers (optional: `?iata=&type=&broker=&status=online\|offline`) | | `GET` | `/observers/{observerId}` | Get observer detail including broker last-seen timestamps | | `GET` | `/observers/{observerId}/adverts` | Adverts heard by observer | -| `GET` | `/observers/{observerId}/telemetry` | Observer telemetry history | +| `GET` | `/observers/{observerId}/telemetry` | Observer telemetry history (optional: `?range=24h&interval=1h\|6h\|24h`) | | `GET` | `/packets` | List packets with filters | | `GET` | `/packets/backfill` | Backfill packets after a given observation ID | | `GET` | `/packets/{packetHash}` | Get packet with all observations | @@ -341,6 +329,7 @@ GET /api/v1/packets?iata=YOW&afterId=12345&limit=100 | `GET` | `/regions/{id}` | Get a single region with IATA list | | `GET` | `/routes` | List known routes (all hops high confidence) | | `GET` | `/routes/search` | Search routes by source and destination hash | +| `GET` | `/routes/cross` | Search for routes crossing IATA boundaries | | `GET` | `/scopes` | List transport scopes | | `GET` | `/scopes/{name}` | Get scope detail | | `GET` | `/stats/observations` | Hourly observation time series (last 7 days by default) | @@ -354,140 +343,16 @@ GET /api/v1/packets?iata=YOW&afterId=12345&limit=100 --- -## Development - -### Modifying DB queries - -Edit `db/queries/queries.sql`, then regenerate: - -```bash -sqlc generate -``` - -### API documentation (Swagger) - -Beacon uses [swaggo/swag](https://github.com/swaggo/swag) to generate OpenAPI -documentation from annotations in the handler comments. - -Start the server and open `http://localhost:8080/swagger/index.html`. - -After adding or modifying any handler, regenerate the docs and commit the -updated `docs/` directory alongside your handler changes: - -```bash -swag init -g cmd/beacon/main.go -o docs --parseDependecy -``` - -Install swag: - -```bash -go install github.com/swaggo/swag/cmd/swag@latest -``` - -Each handler closure should have a godoc-style annotation block immediately -above the `r.Get()`/`r.Post()` call: - -```go -// listThings godoc -// -// @Summary Short description shown in the UI -// @Tags TagName -// @Produce json -// @Param paramName query string false "Description" -// @Param id path string true "Resource ID" -// @Success 200 {object} api.MyResponseType -// @Failure 400 {object} handlers.APIError -// @Failure 500 {object} handlers.APIError -// @Router /things [get] -r.Get("/", func(w http.ResponseWriter, r *http.Request) { -``` - -For paginated responses use the generic page wrapper: - -```go -// @Success 200 {object} api.Page[api.MyType] -``` - -### Updating the IATA database - -Beacon includes a static IATA → country/continent mapping compiled into the -binary, generated from the [OurAirports](https://ourairports.com/data/) public -dataset. - -To refresh it with the latest airport data: - -```bash -rm internal/iatadb/gen/airports.csv -go generate ./internal/iatadb/ -``` - -This fetches a fresh `airports.csv` from OurAirports, saves it locally, and -regenerates `internal/iatadb/db.go`. Commit both files. - -To use a local CSV instead (e.g. in a restricted network environment): - -```bash -AIRPORTS_CSV=/path/to/airports.csv go run ./internal/iatadb/gen -``` - ---- - ## Road Map -### Done - -- [x] MQTT ingest pipeline (two brokers, cross-broker dedup) -- [x] Packet decode via meshcore-go -- [x] Observer upsert and status processing -- [x] Node upsert from advert payloads -- [x] Channel message storage with key-based decryption -- [x] Firmware capability detection scaffolding -- [x] Hub-based WebSocket fan-out with subscription filtering -- [x] WebSocket server (hello, subscribe, unsubscribe, ping/pong, lagged, - events) -- [x] WebSocket regionId expansion via region_iatas DB lookup -- [x] WebSocket per-IP connection limits -- [x] Config file loading (regions, IATA overrides, channel keys) -- [x] Observer radio settings on observations -- [x] DB seeding on startup -- [x] Observer telemetry storage with configurable resolution and retention -- [x] Packet retention cleanup goroutine -- [x] Hashtag channel PSK derivation (SHA256("#tag")[:16]) -- [x] Channel hash collision handling via key fingerprint -- [x] REST API: IATAs, Regions -- [x] REST API: Channels (list + detail + messages) with IATA filter -- [x] REST API: Messages (cross-channel) with IATA filter -- [x] REST API: Observers (heard adverts, telemetry, list + detail with broker - last-seen) -- [x] REST API: Brokers (list with connection status) -- [x] REST API: Pagination -- [x] REST API: Nodes (list + detail + observations) -- [x] REST API: Packets (list + detail) -- [x] REST API: Stats -- [x] Materialized view refresh (mv_hourly_iata_stats, mv_top_nodes_by_iata) -- [x] Swagger/OpenAPI documentation via swaggo/swag -- [x] Path resolution (node short ID lookup) -- [x] Parse payloads (that we can decrypt) into DB and return with packet - details -- [x] Propagation time calculation -- [x] Trace route resolution via path hashes (resolvedRoute on packet detail) -- [x] REST API: Trace packets: trace tag storage, list and detail endpoints with - resolved routes -- [x] REST API: Known routes: fully resolved paths stored at ingest, list and - search endpoints -- [x] Node neighbor detection and storage from advert path resolution -- [x] REST API: Node neighbors endpoint - -### Future - - [ ] Redis caching for stats endpoints - [ ] Caddy reverse proxy config for production - - [ ] Admin authentication middleware - - [ ] Server management via API (currently config-file only) -- [ ] Observer owner tracking (schema exists, API excluded by design) +- [ ] Server management via API (currently config-file only) - [ ] Log levels, debug and info +--- + ## Acknowledgements Beacon stands on the shoulders of giants. See [SHOULDERS.md](SHOULDERS.md) for