## What this does The frontend half of serving the UI apart from the server. The server half was #2362. It gives CoreScope three deployment modes. The server container is the same image in all three, and it still serves everything, the classic UI included. | Mode | The UI is served by | API | Server CORS setting | Logins | |---|---|---|---|---| | All-in-one container (today) | the server container | same origin | none | yes | | Split containers (new) | a web container that forwards the API | same origin | none | yes | | Static host or CDN (new) | Cloudflare Pages, a CDN or any web server | another origin | `CORS_ALLOWED_ORIGINS` | no, links to the classic UI | ### Commit 1: the UI as static files, for any host - **One place for API URLs:** `public/deployment.js` says where the API is, and `public/api-base.js` (`CSApi`) builds every API and WebSocket URL. A lint rule keeps raw `/api` fetches out of the other files. The embedded UI keeps `apiBase: ''`, so it behaves as before, except that its WebSocket now uses `/ws` instead of `/`. - **Build:** `make web` (`scripts/static/build.sh`) writes `dist/web`, which is `public/` byte for byte except for the cache buster in `index.html` and `deployment.js`. - It is `dist/web` rather than `dist/` because the Go binaries are built into `dist/` and `make clean` removes that folder. - `API_BASE=same-origin` is for a host that forwards `/api` and `/ws` itself, as the web container does. - **Anonymous build (`AUTH=false`):** for another origin. It has no account features: login, account, admin and other writes link to the same page on the classic UI. It sends no cookies, and its three JSON POSTs go as `text/plain`, so there is no preflight. - **Error notices:** a blocked API answer or an HTML answer each show a notice. - **Channel badge:** `beta`, `next` and `dev` builds show a badge that opens a pre-filled issue, plus an "Open in classic" link. - **Runbook:** `deploy/static/README.md` covers Cloudflare Pages, previews, server CORS and rollback. ### Commit 2: the web container and the three modes - **`Dockerfile.web`:** Caddy on `alpine:3.24`, like the main image. It packages the UI with `build.sh` when the image is built, serves it, and forwards `/api/*` and `/ws` to `BACKEND` (default `corescope:3000`). `SITE_ADDRESS` set to a domain name turns on automatic HTTPS. - **Cache rules:** `docker/web/Caddyfile` sends the same no-cache rules as `deploy/static/_headers`, and compresses responses with zstd or gzip. - **`docker-compose.split.yml`:** the stock server image (its own Caddy off) plus the web container: `CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge docker compose -f docker-compose.split.yml up -d --build`. - `CORESCOPE_IMAGE` is required and must match the checkout's version, so a release server is never paired with a newer UI by default. - The server's built-in broker, which accepts anonymous clients, is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - **Docs:** `docs/deployment.md` has a new "Deployment modes" section, and `DEPLOY.md` points to it. ### Keeping new code on `CSApi` - **Lint rule:** `.eslintrc.json` now rejects every direct network call in `public/*.js`. That covers `fetch`, `XMLHttpRequest`, `EventSource`, `sendBeacon` and `WebSocket` without `CSApi.wsUrl`, plus `/api` paths in `href`, `src` and `action` (assigned, set with `setAttribute` or written into markup), `window.open`, `location.assign`, `location.replace` and `location =`. `api-base.js` is the one file exempt. - **HTML pages:** `eslint public/*.js` never sees the inline scripts and markup of `public/*.html`, so `test-lint-network-calls.js` applies the same rule to them, at their page lines. `area-map.html`'s two fetches carry an `eslint-disable` with the reason: its Server field can name another server. - **Why it changed:** the earlier rule only caught a literal `/api…` passed straight to `fetch`. - **Check:** CI already fails on lint errors, and `tests/e2e/test-lint-network-calls.js` pins exactly what the rule reports: 25 bad statements reported, 15 good ones not, `api-base.js` exempt. - **AGENTS.md:** a new "API calls" section covers: - which helper to use for each need; - the lint rule; - how anonymous builds gate login-only features; - loading `api-base.js` in vm and jsdom tests (the region quick-picks suite missed that, and failed shard 1 on the first CI run). ### Review follow-ups - **Account tokens:** the channel badge's pre-filled issue drops the query on `#/account/…` routes, so mail-link tokens never reach a GitHub URL. "Open in classic" keeps the full route, since the classic UI needs the token. - **HTML answers:** on a packaged UI (static host, web container), `api()` rejects an HTML answer before status and retry handling. An HTML 404 or 503 (a wrong `API_BASE`, a proxy error page) shows "Server returned an unexpected response" after one request. It used to give a generic error, or six "warming up" retries for a 503. JSON errors and the server's JSON 503 warm-up behave as before. The embedded UI keeps its old handling (`CSApi.embedded`). - **Logins:** `docs/deployment.md`, `Dockerfile.web` and `docker-compose.split.yml` say `userManagement.publicBaseUrl` must be the web container's public origin. ### Audit fixes - **No broken logins build:** `build.sh` refuses `AUTH=true` with a cross-origin `API_BASE`. The server allows no credentialed CORS, so every API request of such a build was blocked; in Chromium all 12 were, and the pages stayed empty. - **No accidental deletes:** `build.sh` refuses an `OUT` that is neither empty nor a previous build. It used to wipe whatever folder or file `OUT` named. - **Lint:** the rule now also reports `/api/` paths in `href`, `src` and `action` attributes in single-quoted strings and template literals. `tests/e2e/test-lint-network-calls.js` pins exactly what it reports. - **Compose:** `CORESCOPE_IMAGE` has no default, and MQTT is published on localhost only. - **Runbook:** the deploy and rollback examples use `vX.Y.Z`, since v3.16.0 and v3.15.0 predate the static build. It also names the server version the static build needs: before #2362 the warm-up headers stay hidden and wildcard previews are refused. ### Audit notes, fixed - **Cache busting on every page:** the tool pages (`area-map.html`, `geofilter-builder.html`) load their scripts with `?v=__BUST__`. The server and `build.sh` replace it in every top-level page, and `_headers` and the Caddyfile make every page revalidate. Behind Caddy, a returning visitor could otherwise get a new tool page with an old `api-base.js`. - **HTTP/3:** the split compose publishes 443 over UDP too, for the HTTP/3 Caddy advertises once `SITE_ADDRESS` is a domain. Checked with Chromium forced onto QUIC: the page and `/api/healthz` loaded over `h3`. - **WebSocket on `/`:** a Go test pins that an upgrade on `/` still works, for UIs cached from before `/ws`. ### Second review - **Trusted proxies:** the docs said to trust the compose subnet, and also to publish port 3000 if wanted. Together they let a client spoof its address: the subnet holds the gateway, where a client reaching a published 3000 through Docker's proxy arrives from. The compose file now fixes the subnet (`SPLIT_NET`, default `172.30.83.0/24`) and the web container's address (`172.30.83.200`), and the docs say to trust that address alone. - **Classic UI unchanged:** the HTML-answer notice above applies to packaged UIs only. On the embedded UI a proxy's HTML 503 is retried as before, and an unknown `/api` path fails to parse as before, with no notice. - **Lint coverage:** the HTML pages and the extra sinks above. AGENTS.md says what the rule cannot see: a URL assembled elsewhere and passed in a variable. - **AGENTS.md:** `build.sh` is described as packaging (a copy, `deployment.js` and `__BUST__`, never a code transform), and rule 3 says `__BUST__` is filled in every top-level page. - **CI builds the web image:** the image-check job (no publish) builds `Dockerfile.web`, runs it, and checks the busted, no-cache `index.html`, the same-origin `deployment.js` and that `/api` reaches `BACKEND`. It also validates the split compose file. - **Cross-origin logins:** a hand-edited `deployment.js` with `auth: true` and another origin now logs a warning naming the setting, instead of failing with CORS errors alone. ## Upgrade notes - **WebSocket path:** the UI's live feed connects to `/ws` instead of `/`. The server has answered `/ws` since #1793 and still upgrades `/`, so browsers holding the previous UI keep working. - A reverse proxy that forwards WebSocket upgrades only for an exact `/` must forward `/ws` too. - The nginx example in `docs/deployment.md` (`location /`) already covers it. - **Static UI from another origin:** - The server needs `CORS_ALLOWED_ORIGINS`, and a version with #2362 for the warm-up banner and wildcard preview origins. - `build.sh` refuses `AUTH=true` there, since the server allows no credentialed CORS. - **Split containers:** - `CORESCOPE_IMAGE` is required. - With user accounts on, `userManagement.publicBaseUrl` must be the web container's public origin. - The built-in MQTT broker is published on `127.0.0.1` unless `MQTT_BIND` says otherwise. - The compose network is `172.30.83.0/24`, with the web container at `.200`. If Docker reports that the pool overlaps another network, set `SPLIT_NET`. ## Testing - `go test ./...`, `go vet` and `gofmt` are clean in `cmd/server`, with new tests for busting every page and the WebSocket upgrade on `/`. - `sh test-all.sh` passes. New unit suites: - `test-static-build.js`: the build and its input checks, `same-origin` included, cross-origin logins refused, and an unrelated `OUT` left intact. - `test-web-container.js`: the Dockerfile, Caddyfile and compose file agree on packaging, forwarded paths, cache rules and the server port; the server image has no default and MQTT stays on localhost. - The `api-base`, anonymous mode, badge and `dev-static` suites. - `eslint public/*.js` has 0 errors. `test-lint-network-calls.js` passes 47/47, the four pages included; it needs eslint, so CI runs it on E2E shard 1. - **Static host or CDN mode:** `test-split-anonymous-e2e.js` passes 11/11. That is the static build on one origin against an API-only server on another, which CI runs on E2E shard 1. - **Split containers mode:** - The image builds, and `caddy validate` passes (Caddy 2.11.4). The new CI step passes locally, and fails when the Caddyfile stops forwarding `/api`. - Trusted proxies, with the stack up: the web container gets `172.30.83.200`. With only that address trusted, a spoofed `X-Forwarded-For` sent straight to a published 3000 is ignored, and one sent through Caddy is replaced with the client's address. With the subnet trusted, as the docs used to say, the spoofed address was believed: a deny entry for it rejected the upgrade. - With `docker-compose.split.yml` on the E2E fixture, seeded the way CI seeds it, `test-e2e-playwright.js` passes 149 of 152 (3 skipped) both through the web container and against the server directly. - Headers, checked with curl: - `no-cache` on `/`, `/index.html` and `/deployment.js`; - zstd on static assets; - `/api/*` proxied (`Via: 1.1 Caddy`); - a `101` WebSocket upgrade on `/ws`. - In Chromium: - every API request and the WebSocket stay on the web container's origin; - there are no page errors and no horizontal overflow at 390 px; - no badge shows on the `static` channel. - Logins through the web container: with user management on and `userManagement.publicBaseUrl` set to the web container's origin, a Chromium run passed every step, all on the web container's origin with no page errors: - registration, and activation from the mail link (which uses `publicBaseUrl`); - the session surviving a reload; - a CSRF-protected `PATCH /api/account`; - logout, then login with the form. - The same login with the backend's origin as `Origin` is refused with 403. The server checks `Origin` against `publicBaseUrl`, not `Host`, which is why the docs ask for it. ## Performance There are no hot-path changes. API calls go through `CSApi.url()`, a string concatenation. In the web container, Caddy serves files from disk and compresses them; API responses pass through with one proxy hop, the same as the all-in-one image's built-in Caddy. ## Notes - **Per-IP limits:** behind the web container, as behind the all-in-one image's Caddy, requests reach the server from the proxy's address. The per-IP limits stay off until `trustedProxies` lists it: `172.30.83.200`, the web container's fixed address. The docs say so. - **`depends_on`:** the web container starts with the server, not after its healthcheck, like the all-in-one image's Caddy. Gating on health would make every `up` wait for the first check (30 s) and fail outright when a large database loads for longer than the healthcheck allows; until the server listens, Caddy answers `/api` with 502. - **Not in this PR:** - publishing a `corescope-web` image (CI builds and smokes it, without publishing); - Cross-origin login—including a static frontend and API on sibling custom domains—is outside this PR. In the deployments supported by this PR, account support requires the frontend and API to share one browser origin, either served together or through a reverse proxy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
32 KiB
CoreScope Deployment Guide
Comprehensive guide to deploying and operating CoreScope. For a quick start, see DEPLOY.md.
Table of Contents
- System Requirements
- Deployment modes
- Docker Deployment
- Configuration Reference
- MQTT Setup
- TLS / HTTPS
- Behind a CDN (Cloudflare, Fastly)
- Monitoring & Health Checks
- Backup & Restore
- Troubleshooting
System Requirements
| Resource | Minimum | Recommended |
|---|---|---|
| RAM | 256 MB | 512 MB+ |
| Disk | 500 MB (image + DB) | 2 GB+ for long-term data |
| CPU | 1 core | 2+ cores |
| Architecture | linux/amd64, linux/arm64 |
— |
| Docker (run the published image) | 20.10+ | Latest stable |
| Docker (build the image yourself) | 23.0+, or 20.10+ with DOCKER_BUILDKIT=1 |
Latest stable |
The Dockerfile uses BuildKit features (# syntax= and RUN --mount). BuildKit is the default builder from Docker 23.0; on 20.10 to 22.x the legacy builder is used unless DOCKER_BUILDKIT=1 is set, and it rejects RUN --mount.
CoreScope runs well on Raspberry Pi 4/5 (ARM64). The Go server uses ~300 MB RAM for 56K+ packets.
Deployment modes
The UI is static files (public/) that call the server's API. The three modes
differ only in what serves those files. The server container is the same image
in all three, and it still serves the classic UI itself.
| Mode | The UI is served by | API | Server CORS setting | Logins |
|---|---|---|---|---|
| All-in-one container | the server container | same origin | none | yes |
| Split containers | a web container (Dockerfile.web) that forwards the API |
same origin | none | yes |
| Static host or CDN | Cloudflare Pages, a CDN or any web server | another origin | CORS_ALLOWED_ORIGINS |
no, links to the classic UI |
All-in-one container
The default, and what the rest of this guide describes: one image runs the
server, the ingestor and, optionally, Mosquitto and Caddy. The server serves the
UI from /app/public and the API on one origin. See
Docker Deployment.
Split containers
docker-compose.split.yml runs the server image unchanged next to a web
container built from Dockerfile.web. The web container is Caddy: it serves the
UI, packaged by scripts/static/build.sh when the image is built, and forwards
/api/* and /ws to the server. The browser sees one origin, so the server
needs no CORS setting. Logins work as on the classic UI when account
configuration uses the public web origin, as described below.
CORESCOPE_IMAGE=ghcr.io/kpa-clawbot/corescope:edge \
docker compose -f docker-compose.split.yml up -d --build
| Variable | Default | |
|---|---|---|
HTTP_PORT, HTTPS_PORT |
80, 443 |
Published by the web container; 443 over TCP and UDP (HTTP/3) |
SITE_ADDRESS |
:80 |
A domain name turns on Caddy's automatic HTTPS and HTTP/3. Keep the default ports then: certificate issuance and the HTTP/3 advertisement both assume 80 and 443 |
CORESCOPE_IMAGE |
none, required | The server image of the same version as the checkout the web image is built from: :edge on master, :vX.Y.Z on a release tag |
DATA_DIR |
./data |
The server's data directory |
GIT_COMMIT |
the build time | The UI's cache buster |
CHANNEL |
static |
beta, next or dev shows a badge |
MQTT_BIND, MQTT_PORT |
127.0.0.1, 1883 |
Where the server's built-in broker is published. It accepts anonymous clients, so set MQTT_BIND=0.0.0.0 only to let gateways on a network you trust publish to it |
SPLIT_NET |
172.30.83 |
The compose network's /24, as its first three octets; the web container is .200 in it. Change it only if Docker reports that the pool overlaps another network, and update trustedProxies to match |
- When
userManagement.enabledis true, setuserManagement.publicBaseUrlin the server's config to the public web origin, including its scheme and port (for example,http://localhost:8080for a web container published on port 8080). Login and other state-changing requests compare the browser'sOriginorRefererwith this configured origin; forwarding the publicHostheader alone is not enough. This value also determines mail links. See account configuration. - The web container finds the server through
BACKEND(corescope:3000in the compose file). Port 3000 stays inside the compose network, and the server still serves everything there, the classic UI included; publish it if you want to reach that directly. - The web image is built from your checkout, so keep the checkout and
CORESCOPE_IMAGEon the same version: a newer UI may call endpoints an older server lacks. To update only the UI, rebuild the web container:docker compose -f docker-compose.split.yml up -d --build web. - Behind the web container, as behind the all-in-one image's Caddy, every
request reaches the server from the proxy's address, so per-IP limits are off
until
webSocket.trustedProxiesanduserManagement.trustedProxieslist it. The compose file fixes that address at172.30.83.200: list it alone. Never list the compose subnet. It also holds the gateway address, and a client that reaches a published port 3000 through Docker's proxy (from the host itself or over IPv6, for example) arrives from there, so its ownX-Forwarded-Forwould be believed.
Static host or CDN
make web (or scripts/static/build.sh) writes the anonymous build to
dist/web for Cloudflare Pages or any static host. It calls the server
cross-origin, so the server lists the UI's origin in CORS_ALLOWED_ORIGINS.
It sends no cookies; logins and admin pages link to the classic UI on the
server. Setup, previews, server settings and rollback:
deploy/static/README.md.
Docker Deployment
Quick Start (one command)
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
Open http://localhost — you'll see an empty dashboard ready to receive packets.
No config.json is required. The server starts with sensible defaults:
- HTTP on port 3000 (Caddy proxies port 80 → 3000 internally)
- Internal Mosquitto MQTT broker on port 1883
- Ingestor connects to
mqtt://localhost:1883automatically - SQLite database at
/app/data/meshcore.db
Full docker run Reference (recommended)
The bare docker run command is the primary deployment method. One image, documented parameters — run it however you want.
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 443:443 -p 1883:1883 \
-e DISABLE_MOSQUITTO=false \
-e DISABLE_CADDY=false \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:latest
Parameters
| Parameter | Required | Description |
|---|---|---|
-p 80:80 |
Yes | HTTP web UI |
-p 443:443 |
No | HTTPS (only if using built-in Caddy with a domain) |
-p 1883:1883 |
No | MQTT broker (expose if external gateways connect directly) |
-v /your/data:/app/data |
Yes | Persistent data: SQLite DB, config.json, theme.json |
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro |
No | Custom Caddyfile for HTTPS |
-v /your/caddy-data:/data/caddy |
No | Caddy TLS certificate storage |
-e DISABLE_MOSQUITTO=true |
No | Skip the internal Mosquitto broker (use your own) |
-e DISABLE_CADDY=true |
No | Skip the built-in Caddy reverse proxy |
-e MQTT_BROKER=mqtt://host:1883 |
No | Override MQTT broker URL |
-e RUN_AS_UID=1000 |
No | Run the Go services as this uid instead of root — see Running the Go services unprivileged |
-e RUN_AS_GID=1000 |
No | Group for RUN_AS_UID (defaults to it) |
/app/data/.env convenience file
Instead of passing -e flags, you can drop a .env file in your data volume:
# /your/data/.env
DISABLE_MOSQUITTO=true
DISABLE_CADDY=true
MQTT_BROKER=mqtt://my-broker:1883
The entrypoint sources this file before starting services. This works with any launch method (docker run, compose, or manage.sh).
Two exceptions, both because the entrypoint runs this file as root from a directory the Go services can write once they run unprivileged:
RUN_AS_UIDandRUN_AS_GIDare never taken from.env. Set them with-eor in compose'senvironment:.- With
RUN_AS_UIDset,.envis read only if it is owned by root and writable by no one else (sudo chown root:root .env && sudo chmod 644 .env). Otherwise the log says it was skipped, and why. "Owned by root" means uid 0 inside the container. With rootless Docker or a user-namespace remap (userns-remap), host root is not that uid: look at the owner the container reports (docker exec <container> stat -c '%u' /app/data/.env) and give the host file whichever owner id maps to container uid 0.
Docker Compose (legacy alternative)
Docker Compose files are maintained for backward compatibility but are no longer the recommended approach.
curl -sL https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/docker-compose.example.yml \
-o docker-compose.yml
docker compose up -d
Compose environment variables
| Variable | Default | Description |
|---|---|---|
HTTP_PORT |
80 |
Host port for the web UI |
DATA_DIR |
./data |
Host path for persistent data |
DISABLE_MOSQUITTO |
false |
Set true to use an external MQTT broker |
DISABLE_CADDY |
false |
Set true to skip the built-in Caddy proxy |
manage.sh (legacy alternative)
The manage.sh wrapper script provides a setup wizard and convenience commands. It uses Docker Compose internally. See DEPLOY.md for usage. New deployments should prefer bare docker run.
Image tags
| Tag | Use case |
|---|---|
v3.4.1 |
Pinned release — recommended for production |
v3.4 |
Latest patch in the v3.4.x series |
v3 |
Latest minor+patch in v3.x |
latest |
Latest release tag |
edge |
Built from master on every push — unstable |
Updating
docker compose pull
docker compose up -d
For docker run users:
docker pull ghcr.io/kpa-clawbot/corescope:latest
docker stop corescope && docker rm corescope
docker run -d --name corescope ... # same flags as before
Data is preserved in the volume — updates are non-destructive.
Running the Go services unprivileged
corescope-server and corescope-ingestor can run as a normal user instead of
root, so that a code-execution bug in either one does not land as root in a
container that has your data directory and config.json mounted. supervisord
itself stays root, because mosquitto and caddy still need to bind their ports.
It is opt-in, with RUN_AS_UID:
RUN_AS_UID |
Services run as |
|---|---|
| unset (the default) | root — as in earlier releases |
| set | RUN_AS_UID / RUN_AS_GID |
Nothing about an upgrade changes until you set it: the container does not pick a
uid for you, and does not change the owner of anything on your host. That is
deliberate rather than cautious. Every release before this one ran as root, so
the meshcore.db in your data directory — with its -wal and -shm, users.db
and the backups beside them — is root-owned. Switching user without also handing
those files over leaves the services unable to write the database, so the switch
has to be something you ask for.
To turn it on, pick the uid you want and restart. Setting RUN_AS_UID is
taken as permission to chown the data directory, and what an earlier root-run
container left in it, to that uid. This happens at every start, so a file
created there later as root (with docker exec, say) is handed over at the
next restart:
docker run -d --name corescope \
-e RUN_AS_UID=1000 -e RUN_AS_GID=1000 \
-v /your/data:/app/data \
... # same flags as before
ghcr.io/kpa-clawbot/corescope:latest
Check which user it picked:
docker logs corescope 2>&1 | grep '\[entrypoint\] Go services run as'
# [entrypoint] Go services run as corescope (uid=1000 gid=1000)
If you would rather the container never chown anything, do the hand-over
yourself first. Then there is nothing left for it to change:
docker stop corescope
sudo chown -R 1000:1000 /your/data # uid of your own account is usually 1000
docker start corescope # with RUN_AS_UID=1000 set as above
To go back, unset RUN_AS_UID and chown -R root:root /your/data.
Notes:
RUN_AS_GIDdefaults toRUN_AS_UIDwhen omitted.RUN_AS_UID=0is the same as leaving it unset.- If the uid already belongs to another account inside the image (
caddyis uid 100,mosquitto101), that account is used as it is, keeping its own group, and the log says so. Pick a different uid if you needRUN_AS_GIDhonoured. - With
RUN_AS_UIDset to a uid the container cannot use, startup fails rather than quietly falling back to root. - What the hand-over leaves alone: anything mounted into the data directory on
its own, such as
-v ./config.json:/app/data/config.json:ro, which keeps the owner it has on your host (make sure the uid can read it), and.env(see above). Each mount it skips is named in the log. - If the hand-over cannot change a file, for example on NFS with
root_squash, the log says how many and which. Fix their owner on the host. /appstays root's. The services write only to the data directory and/tmp.- A fresh install is root-owned and so starts as root, like any other. Set
RUN_AS_UIDfrom the first run to avoid a hand-over later.
The uid selection lives in docker/select-service-user.sh, with its branches
covered by docker/test-select-service-user.sh.
Configuration Reference
CoreScope uses a layered configuration system (highest priority wins):
- Environment variables —
MQTT_BROKER,DB_PATH, etc. /app/data/config.json— full config file (volume-mounted)- Built-in defaults — work out of the box with no config
Environment variable overrides
| Variable | Default | Description |
|---|---|---|
MQTT_BROKER |
mqtt://localhost:1883 |
MQTT broker URL (overrides config file) |
MQTT_TOPIC |
meshcore/# |
MQTT topic subscription pattern |
DB_PATH |
data/meshcore.db |
SQLite database path |
DISABLE_MOSQUITTO |
false |
Skip the internal Mosquitto broker |
DISABLE_CADDY |
false |
Skip the built-in Caddy reverse proxy |
config.json
For advanced configuration, create a config.json and mount it at /app/data/config.json:
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
-v ./config.json:/app/data/config.json:ro \
ghcr.io/kpa-clawbot/corescope:latest
See config.example.json in the repository for all available options including:
- MQTT sources (multiple brokers)
- Channel encryption keys
- Branding and theming
- Health thresholds
- Region filters
- Retention policies
- Geo-filtering
- Map tile providers (OSM, Stamen, Carto, etc.)
Map Tile Providers
Map tile providers are enabled and configured via the config.json file. You can provide your custom API credentials (e.g. osm_url, stamen_api_key, mapbox_api_key) to activate external tile services. Once configured on the server, users can select their preferred tile provider from the Customizer UI on the client, and their choice will be persisted automatically.
CARTO now requires an API key
CARTO's raster basemaps (carto-dark, carto-light, carto-voyager,
carto-voyager-dark, positron-dark) require an API key. Without one, tiles
are still returned with a 200 status but every one of them is stamped
API KEY REQUIRED — so nothing errors, no healthcheck fires, and the map just
quietly looks broken.
Get a free key at https://carto.com/basemaps/apikey — no CARTO account, no
credit card, emailed back immediately, 5 million tile requests per calendar
month across raster and vector. Then set it as key:
"map": {
"tiles": {
"providers": {
"carto": { "enabled": true, "key": "YOUR_CARTO_KEY" }
}
}
}
Notes:
- The query parameter is
key.api_keyis silently ignored and still serves the watermarked tile. - The key is sent to the browser, so treat it as public. CARTO asks for a domain when the key is issued, but whether that restriction is enforced on every request has not been verified here. Assume anyone can read the key from devtools and spend your quota. The 5M/month fair-use ceiling makes that low-impact for most deployments, but it is worth knowing.
- A wrong or expired key fails the same silent way as no key at all. Verify by looking at a tile, not by checking for a 200.
- CARTO has said it is considering freezing data updates to the raster
basemaps. If you would rather not depend on them, set
"carto": { "enabled": false }and use another provider —esrineeds no key at all, andosm/stamenaccept their own tokens.
MQTT Setup
CoreScope receives MeshCore packets via MQTT. The container ships with an internal Mosquitto broker — no setup needed for basic use.
Internal broker (default)
The built-in Mosquitto broker listens on port 1883 inside the container. Point your MeshCore gateways at it:
# Expose MQTT port for external gateways
docker run -d --name corescope \
-p 80:80 -p 1883:1883 \
-v corescope-data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
External broker
To use your own MQTT broker (Mosquitto, EMQX, HiveMQ, etc.):
-
Disable the internal broker:
-e DISABLE_MOSQUITTO=true -
Point the ingestor at your broker:
-e MQTT_BROKER=mqtt://your-broker:1883Or via
config.json:{ "mqttSources": [ { "name": "my-broker", "broker": "mqtt://your-broker:1883", "username": "user", "password": "pass", "topics": ["meshcore/#"] } ] }
Multiple brokers
CoreScope can connect to multiple MQTT brokers simultaneously:
{
"mqttSources": [
{
"name": "local",
"broker": "mqtt://localhost:1883",
"topics": ["meshcore/#"]
},
{
"name": "remote",
"broker": "mqtts://remote-broker:8883",
"username": "reader",
"password": "secret",
"topics": ["meshcore/+/+/packets"]
}
]
}
MQTT topic format
MeshCore gateways typically publish to meshcore/<gateway>/<region>/packets. The default subscription meshcore/# catches all of them.
TLS / HTTPS
Option 1: External reverse proxy (recommended)
Run CoreScope behind nginx, Traefik, or Cloudflare Tunnel for TLS termination:
# nginx example
server {
listen 443 ssl;
server_name corescope.example.com;
ssl_certificate /etc/ssl/certs/corescope.pem;
ssl_certificate_key /etc/ssl/private/corescope.key;
location / {
proxy_pass http://localhost:80;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
}
The Upgrade and Connection headers are required for WebSocket support.
Option 2: Built-in Caddy (auto-TLS)
The container includes Caddy for automatic Let's Encrypt certificates:
-
Create a Caddyfile:
corescope.example.com { reverse_proxy localhost:3000 } -
Mount it and expose TLS ports:
docker run -d --name corescope \ -p 80:80 -p 443:443 \ -v corescope-data:/app/data \ -v caddy-certs:/data/caddy \ -v ./Caddyfile:/etc/caddy/Caddyfile:ro \ ghcr.io/kpa-clawbot/corescope:latest
Caddy handles certificate issuance and renewal automatically.
API Documentation
CoreScope auto-generates an OpenAPI 3.0 specification from its route definitions. The spec is always in sync with the running server — no manual maintenance required.
Endpoints
| URL | Description |
|---|---|
/api/spec |
OpenAPI 3.0 JSON schema — machine-readable API definition |
/api/docs |
Interactive Swagger UI — browse and test all 40+ endpoints |
Usage
Browse the API interactively:
http://your-instance/api/docs
Fetch the spec programmatically:
curl http://your-instance/api/spec | jq .
For bot/integration developers: The spec includes all request parameters, response schemas, and example values. Import it into Postman, Insomnia, or any OpenAPI-compatible tool.
Public instance
The live instance at analyzer.00id.net has all API endpoints publicly accessible:
- Spec: analyzer.00id.net/api/spec
- Docs: analyzer.00id.net/api/docs
Behind a CDN (Cloudflare, Fastly)
If you front CoreScope with a CDN — Cloudflare, Fastly, Akamai, or
similar — you must configure the CDN to bypass cache for /api/*.
The server emits Cache-Control: no-store on every API response
(see #1551), but Cloudflare's zone-level Cache Rules and legacy Page
Rules can override origin headers. When that happens, observers, packets,
stats and other API responses get cached at the edge for minutes to hours,
producing observer-flap, stale dashboards and inconsistent state across
viewers.
1. Verify whether your CDN is caching /api/*
From outside the CDN (a different network than your origin), run:
curl -sI 'https://<your-domain>/api/observers' | grep -iE 'cf-cache|age|cache-control'
Healthy output (cache is bypassed):
cache-control: no-store
cf-cache-status: BYPASS
age: 0
Unhealthy output (CDN is caching despite no-store):
cache-control: no-store
cf-cache-status: HIT
age: 4732
HIT or age > 0 means an intermediary is serving cached JSON. Fix it
before relying on the dashboard.
You can also run the bundled helper, which exits non-zero with a precise diagnostic when caching is detected:
scripts/check-cdn-bypass.sh https://<your-domain>
2. Cloudflare: add a Cache Rule (recommended)
Cloudflare Dashboard → your zone → Caching → Cache Rules → Create rule:
- When incoming requests match: Field =
URI Path, Operator =starts with, Value =/api/ - Then: Cache eligibility → Bypass cache
Save and deploy. Re-run the curl above; you should now see
cf-cache-status: BYPASS.
Legacy Page Rules equivalent (if your zone has no Cache Rules):
- URL pattern:
*your-domain*/api/* - Setting: Cache Level → Bypass
3. Fastly / other CDNs
Apply the equivalent bypass-cache rule for the /api/ path prefix.
The key invariant is: any response from /api/* must reach the
browser uncached (no shared-cache HIT, no positive Age header).
4. Re-verify
After applying the rule, run step 1's curl from outside the CDN again
and confirm cf-cache-status: BYPASS (or absence of HIT) and age: 0.
5. Watch the server log
The server logs a one-shot warning at the first request bearing a
CDN-specific header (CF-Connecting-IP, CF-Ray, Fastly-Client-IP,
or True-Client-IP):
Generic reverse-proxy headers (X-Forwarded-For, X-Real-IP) are
deliberately NOT used as the signal — every nginx/Caddy/Traefik/k8s
deploy sets them, so they'd produce false positives on every
reverse-proxied install.
[security] WARNING: detected request via CDN (CF-Ray header present).
Ensure /api/* is bypassed in your CDN config — see docs/deployment-behind-cdn.md.
Cached API responses cause observer-flap and incorrect dashboards.
This is informational — the request is not blocked. Treat it as a prompt to run the verification curl above. The warning logs at most once per process boot, regardless of how many CDN-fronted requests arrive.
Why this can't be fixed server-side
CDN cache policy is operator-controlled. The application emits the
most conservative cache header it can (Cache-Control: no-store),
but Cloudflare Cache Rules / Page Rules have higher precedence than
origin headers in many zone configurations. The only durable fix is
the operator-side bypass rule.
Monitoring & Health Checks
Docker health check
The container includes a built-in health check that hits /api/stats:
docker inspect --format='{{.State.Health.Status}}' corescope
Docker reports healthy or unhealthy automatically. The check runs every 30 seconds.
Manual health check
curl -f http://localhost/api/stats
Returns JSON with packet counts, node counts, and version info:
{
"totalPackets": 56234,
"totalNodes": 142,
"totalObservers": 12,
"packetsLastHour": 830,
"packetsLast24h": 19644,
"engine": "go",
"version": "v3.4.1"
}
Log monitoring
# All logs
docker compose logs -f
# Server only
docker compose logs -f | grep '\[server\]'
# Ingestor only
docker compose logs -f | grep '\[ingestor\]'
Resource monitoring
docker stats corescope
Backup & Restore
Backup
Use GET /api/backup for the telemetry database. It uses SQLite's
VACUUM INTO to produce a consistent
snapshot while the ingestor is running, including committed data in the WAL.
Do not copy the live meshcore.db file: a checkpoint followed by a file copy
also races new writes. Allow room for a full snapshot in the server's temporary
directory and at the download destination; large snapshots cost CPU and disk I/O.
When accounts are enabled, sign in as an administrator and open /api/backup
on the same site to download a snapshot. For command-line or scheduled backups,
configure a randomly generated apiKey in config.json (at least 16 characters).
Known weak/default keys are rejected even when they meet the length requirement.
Create a private header file, ~/.config/corescope/backup.header, containing one line:
X-API-Key: <your configured key>. Set its permissions to 600; keep the key out
of shell history, URLs, and scripts checked into Git.
On the backup host, install curl and the sqlite3 CLI. Replace the example
HTTPS origin below with your instance. The download is published as a .db
only after it is complete, nonempty, and passes SQLite's consistency check:
(
set -eu
umask 077
backup_dir="$HOME/corescope-backups"
mkdir -p "$backup_dir"
tmp=$(mktemp "$backup_dir/corescope-$(date -u +%Y%m%dT%H%M%SZ).XXXXXXXX")
trap 'rm -f "$tmp"' EXIT
curl -q --fail --silent --show-error \
--connect-timeout 30 --max-time 1800 \
--header "@$HOME/.config/corescope/backup.header" \
--output "$tmp" https://corescope.example.org/api/backup
test -s "$tmp"
test "$(sqlite3 -readonly "$tmp" 'PRAGMA quick_check;')" = ok
mv "$tmp" "$tmp.db"
)
The temporary-file template ends in X characters for both GNU and BSD/macOS
mktemp. The 30-minute limit is curl's client-side ceiling for generating and
downloading the snapshot. CoreScope separately sets a 60-second HTTP write
timeout, which the backup handler does not override; reverse proxies can impose
their own deadlines. Raising --max-time does not extend those server or proxy limits.
Timeouts fail the script and remove the incomplete download without replacing a
previous completed backup.
This is a point-in-time telemetry snapshot, not continuous replication or a full instance backup. Keep protected copies off the CoreScope host, and also preserve:
config.json,theme.json, deployment settings, and any configured keys/secrets.- When accounts are enabled,
users.dbseparately: use the account backup and restore guide, which covers its own snapshot endpoint and scheduled backups. Do not copy that database live.
Restore
- Verify the downloaded file is nonempty and run
sqlite3 -readonly backup.db 'PRAGMA quick_check;'; the result must beok. Keep an untouched copy of this snapshot. - Stop every server and ingestor using the target database, including any
separate instances. For the single-container example, use
docker stop corescope; fordocker-compose.example.yml, usedocker compose -f docker-compose.example.yml stop corescope. - Locate the host directory or named volume mounted at
/app/data. For the example Compose file, this isDATA_DIR(default./data). While all database users remain stopped, move the oldmeshcore.dband any matching-wal,-shm, and-journalfiles together into a separate recovery directory. Keep that set intact; never leave an old WAL beside the replacement database. - Copy the checked snapshot into that data mount as
meshcore.db. Preserve the previous database's owner and permissions so the configured service user can read and write it. Leave configuration, keys, andusers.dbin place. - Start the stopped services (
docker start corescope, or the matching Composestart corescopecommand). Check the logs,/api/healthz, and expected historical packets in the dashboard. Keep the recovery directory until these checks pass.
To roll back, stop all database users again, move the replacement database and its sidecars aside, then restore the entire original file set before restarting.
Automated backups
Run the checked-download block above from a private script using cron or a
systemd timer. Use absolute paths and the backup user's protected header file;
schedule one run at a time during quieter hours. Alert on nonzero exit status.
Prune old completed .db snapshots only after a new backup succeeds, according
to your retention policy, and periodically test a restore in an isolated instance.
Troubleshooting
Container starts but dashboard is empty
This is normal on first start with no MQTT sources configured. The dashboard shows data once packets arrive via MQTT. Either:
- Point a MeshCore gateway at the container's MQTT broker (port 1883)
- Configure an external MQTT source in
config.json
"no MQTT connections established" in logs
The ingestor couldn't connect to any MQTT broker. Check:
- Is the internal Mosquitto running? (
DISABLE_MOSQUITTOshould befalse) - Is the external broker reachable? Test with
mosquitto_sub -h broker -t meshcore/# - Are credentials correct in
config.json?
WebSocket disconnects / real-time updates stop
If behind a reverse proxy, ensure WebSocket upgrade headers are forwarded:
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
Also check proxy timeouts — set them to at least 300s for long-lived WebSocket connections.
High memory usage
The in-memory packet store grows with retained packets. Configure retention limits in config.json:
{
"packetStore": {
"retentionHours": 24,
"maxMemoryMB": 512
},
"retention": {
"nodeDays": 7,
"packetDays": 30
}
}
packetStore.maxMemoryMB bounds the store and the caches that belong to it — the decoded-packet cache and per-packet index entries, not just the stored rows. It is enforced in two places: the startup load stops at the budget, and the store evicts oldest-first when it exceeds it. Leaving it unset means no limit. Actual usage is on /api/perf as packetStore.trackedMB, next to maxMB.
Database locked errors
SQLite doesn't support concurrent writers well. Ensure only one CoreScope instance accesses the database file. If running multiple containers, each needs its own database.
Container unhealthy
Check logs: docker compose logs --tail 50. Common causes:
- Port 3000 already in use inside the container
- Database file permissions (must be writable by the container user)
- Corrupted database — restore from backup
ARM / Raspberry Pi issues
- Use
linux/arm64images (Pi 4 and 5). Pi 3 (armv7) is not supported. - First pull may be slow — the multi-arch manifest selects the right image automatically.
- If memory is tight, set
packetStore.maxMemoryMBto limit RAM usage.