cb0cd3c8dd
Follow-up to #2125: the six non-blocking notes on its approval, with note 3 as the one that changes behaviour. **What changes** - `.env` is now read last. The hand-over of the data directory moves before it, and the only code that runs after the file is sourced reads `prepare_service_user`'s positional parameters: the account, the container's `RUN_AS_UID`/`RUN_AS_GID` (set or unset, and value) and the file's path are parked in `$1..$6` before the `.`. - The comment gives the real reason the account was already safe: the `.` happens inside `load_env_file`, so a `set -- root` in `.env` lands in that function's `$1`, not in `prepare_service_user`'s (note 1). It also says that a trusted `.env` is root's code and this does not defend against it (it can redefine functions or set `PATH`); what it cannot do by assigning variables is change the account, where the hand-over goes, or `RUN_AS_UID`/`RUN_AS_GID` as the services see them. - `svc_log` uses `printf` (note 5); the entrypoint check also pins that it does not assign `CORESCOPE_SERVICE_USER` (note 4); the test header is reflowed (note 6); `TEST_SHELL` picks the inner shell so the suite runs under dash, bash or busybox without editing. **Why** A sourced file shares the shell's variables: every global, the locals of `load_env_file`, and, by dynamic scoping, the locals of its caller too (checked under dash, bash 5.3 and busybox ash 1.37). So a trusted `.env` that assigned `_uid_before` or `_uid_was` skipped the restore and left `RUN_AS_UID=0` in the services' environment, as note 3 says. The same class reached the hand-over: with `SERVICE_DATA_DIR=/` in `.env`, the walk ran over `/`. Nothing but the entrypoint reads `RUN_AS_UID`/`RUN_AS_GID`, so the effect was a wrong value in the services' environment and a log line that said "ignoring" when it had not. **What I checked** - `docker/test-select-service-user.sh`: 116 assertions, 0 failed, under dash and bash. The new cases fail on master for the stated reasons and nothing else does. - New cases: `.env` running `set -- root` (note 1; this is the case that catches sourcing the file inline, the names case alone does not); `.env` assigning the restore's names, `_env_file`, `SERVICE_ENV_FILE` and `RUN_AS_*`, with uid 1000 and gid 1001 so a swapped parameter shows; `RUN_AS_GID` alone; `.env` pointing the three `SERVICE_*` paths elsewhere; a logged path containing `\012`. - Mutations, each applied to a copy and confirmed to make the suite fail: hand-over moved back after `.env`; restore from `load_env_file`'s locals (master); restore from `prepare_service_user`'s locals; file sourced inline in `prepare_service_user` with the refusal log kept; restore dropped; only the uid compared; gid restored from the uid's parameter; set/unset flipped; the log reading `SERVICE_ENV_FILE` after the `.`; the entrypoint reassigning the account. `svc_log` back to `echo` fails under dash only, which is where note 5 applies. - The image, built from this branch (linux/amd64), with a root-owned, root-only-writable `.env` containing all of the above plus `DISABLE_MOSQUITTO=true`, `DISABLE_CADDY=true` and `set -- root`: (A) `RUN_AS_UID` unset: root, nothing re-owned, `DISABLE_*` honoured, `RUN_AS_*` not in the services' environment. (B) `RUN_AS_UID=1000 RUN_AS_GID=1001` on A's data directory, root-owned `meshcore.db` inside: services run as 1000, their environment has `RUN_AS_UID=1000 RUN_AS_GID=1001`, `meshcore.db` and the directory are 1000:1001, `.env` stays root's, `/etc/passwd` stays root's. (C) `.env` owned by uid 1000: refused with the reason, services run as 1000. The same image with master's two scripts mounted over it, in B's shape: services' environment `RUN_AS_UID=0 RUN_AS_GID=0`, `/etc/passwd` owned by 1000, `.env` owned by 1000:1001. - `sh test-all.sh` passes; no JS or Go change. **Not checked** - Windows: the suite stops at the symlink fixture, as before. - busybox as `TEST_SHELL` on my machine: Ubuntu's busybox prefers its own applets over `PATH`, so the stubs are bypassed and 28 cases fail, on master and on this branch alike. Your WSL Debian run (103 passed) and the image itself, which runs Alpine's ash with real binaries, are what cover ash. - Rootless Docker and `userns-remap`: unchanged, docs gap already noted. Note 2: kept the names case as a second guard; the `set -- root` case is the one that pins the design. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: nullrouten <201262769+nullrouten0@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
CoreScope
High-performance mesh network analyzer powered by Go. Sub-millisecond packet queries, ~300 MB memory for 56K+ packets, real-time WebSocket broadcast, full channel decryption.
Self-hosted, open-source MeshCore packet analyzer. Collects MeshCore packets via MQTT, decodes them in real time, and presents a full web UI with live packet feed, interactive maps, channel chat, packet tracing, and per-node analytics.
⚡ Performance
The Go backend serves its packet and analytics API from an in-memory packet store with 7 indexes (hash, txID, obsID, observer, node, path hop, payload type). SQLite is for persistence only — reads never touch disk.
| Metric | Value |
|---|---|
| Packet queries | < 1 ms (in-memory) |
| All API endpoints | < 100 ms |
| Memory (56K packets) | ~300 MB (vs 1.3 GB on Node.js) |
| WebSocket broadcast | Real-time to all connected browsers |
| Channel decryption | AES-128-ECB with rainbow table |
| GOMEMLIMIT (memory-constrained hosts) | set to ≥1.5× working set (e.g. 1536 MiB on a 2 GB Pi for a ~1 GB store). Lower values trigger a GC death-spiral. Configure via the GOMEMLIMIT env var or runtime.maxMemoryMB in config.json; env wins. Applies to both server and ingestor. See #1010. |
See PERFORMANCE.md for full benchmarks.
✨ Features
📡 Live Trace Map
Real-time animated map with packet route visualization, VCR-style playback controls, and a retro LCD clock. Replay the last 24 hours of mesh activity, scrub through the timeline, or watch packets flow live at up to 8× speed.
📦 Packet Feed
Filterable real-time packet stream with byte-level breakdown, Excel-like resizable columns, and a detail pane. Toggle "My Nodes" to focus on your mesh.
🗺️ Network Overview
At-a-glance mesh stats — node counts, packet volume, observer coverage.
📊 Node Analytics
Per-node deep dive with interactive charts: activity timeline, packet type breakdown, SNR distribution, hop count analysis, peer network graph, and hourly heatmap.
💬 Channel Chat
Decoded group messages with sender names, @mentions, timestamps — like reading a Discord channel for your mesh.
📱 Mobile Ready
Full experience on your phone — proper touch controls, iOS safe area support, and a compact VCR bar.
And More
- 17 Analytics Tabs — RF, topology, channels, hash stats, distance, route patterns, scopes, and more
- Node Directory — searchable list with role tabs, detail panel, QR codes, advert timeline
- Packet Tracing — follow individual packets across observers with SNR/RSSI timeline
- Observer Status — health monitoring, packet counts, uptime, per-observer analytics
- Hash Collision Matrix — detect address collisions across the mesh
- Scope Audit: per repeater, which declared regions show no forwarding and which scopes it forwards without declaring them
- Channel Key Auto-Derivation — hashtag channels (
#channel) keys derived via SHA256 - Multi-Broker MQTT — connect to multiple brokers with per-source IATA filtering
- Dark / Light Mode — auto-detects system preference, map tiles swap too
- Theme Customizer — design your theme in-browser, export as
theme.json - Global Search — search packets, nodes, and channels (Ctrl+K)
- Shareable URLs — deep links to packets, channels, and observer detail pages
- Protobuf API Contract — typed API definitions in
proto/ - Accessible — ARIA patterns, keyboard navigation, screen reader support
Optional Features
Off by default, enabled in config.json:
- User Accounts: registration with email activation, settings sync, an admin area, hashtag channel proposals, mail notifications for watched nodes, and a personal data export. The dashboard stays public. See docs/user-guide/accounts.md
- Mobile RX Coverage: crowdsourced coverage from phones connected to a companion radio, drawn as hex bins on the RX Coverage page and on each node's Reach page. See docs/client-rx-coverage.md
Quick Start
Pre-built Image (Recommended)
No build step required — just run:
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 1883:1883 \
-v /your/data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
Open http://localhost — done. No config file needed; CoreScope starts with sensible defaults.
For HTTPS with a custom domain, add -p 443:443 and mount your Caddyfile:
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 443:443 -p 1883:1883 \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:latest
Disable built-in services with -e DISABLE_MOSQUITTO=true or -e DISABLE_CADDY=true, or drop a .env file in your data volume. See docs/deployment.md for the full reference.
Build from Source
git clone https://github.com/Kpa-clawbot/CoreScope.git
cd CoreScope
./manage.sh setup
The setup wizard walks you through config, domain, HTTPS, build, and run.
./manage.sh status # Health check + packet/node counts
./manage.sh logs # Follow logs
./manage.sh backup # Backup database
./manage.sh update # Pull latest + rebuild + restart
./manage.sh mqtt-test # Check if observer data is flowing
./manage.sh help # All commands
Configure
Copy config.example.json to config.json and edit:
{
"port": 3000,
"mqtt": {
"broker": "mqtt://localhost:1883",
"topic": "meshcore/+/+/packets"
},
"mqttSources": [
{
"name": "remote-feed",
"broker": "mqtts://remote-broker:8883",
"topics": ["meshcore/+/+/packets"],
"username": "user",
"password": "pass",
"iataFilter": ["SJC", "SFO", "OAK"]
}
],
"channelKeys": {
"public": "8b3387e9c5cdea6ac9e5edbaa115cd72"
},
"defaultRegion": "SJC"
}
| Field | Description |
|---|---|
port |
HTTP server port (default: 3000) |
mqtt.broker |
Local MQTT broker URL ("" to disable) |
mqttSources |
External MQTT broker connections (optional) |
channelKeys |
Channel decryption keys (hex). Hashtag channels auto-derived via SHA256 |
defaultRegion |
Default IATA region code for the UI |
dbPath |
SQLite database path (default: data/meshcore.db) |
Environment Variables
| Variable | Description |
|---|---|
DB_PATH |
Override SQLite database path |
GOMEMLIMIT |
Go soft memory limit, wins over runtime.maxMemoryMB |
The HTTP port is set with port in config.json or the server's -port flag.
Architecture
┌─────────────────────────────────────────────┐
│ Docker Container │
│ │
Observer → USB → │ Mosquitto ──→ Go Ingestor ──→ SQLite DB │
meshcoretomqtt → MQTT ──→│ │ │
│ Go HTTP Server ──→ WebSocket │
│ │ │ │
│ Caddy (HTTPS) ←───────┘ │
└────────────────────┼────────────────────────┘
│
Browser
Two-process model: The Go ingestor handles MQTT ingestion and packet decoding. The Go HTTP server loads all packets into an in-memory store on startup (5 indexes for fast lookups) and serves the REST API + WebSocket broadcast. Both are managed by supervisord inside a single container with Caddy for HTTPS and Mosquitto for local MQTT.
MQTT Setup
- Flash an observer node with
MESH_PACKET_LOGGING=1build flag - Connect via USB to a host running meshcoretomqtt
- Configure meshcoretomqtt with your IATA region code and MQTT broker address
- Packets appear on topic
meshcore/{IATA}/{PUBKEY}/packets
To decode a single raw hex packet without storing it, use POST /api/decode.
Project Structure
corescope/
├── cmd/
│ ├── server/ # Go HTTP server + WebSocket + REST API
│ │ ├── main.go # Entry point
│ │ ├── routes.go # 40+ API endpoint handlers
│ │ ├── store.go # In-memory packet store (5 indexes)
│ │ ├── db.go # SQLite persistence layer
│ │ ├── decoder.go # MeshCore packet decoder
│ │ ├── websocket.go # WebSocket broadcast
│ │ └── *_test.go # Server tests
│ └── ingestor/ # Go MQTT ingestor
│ ├── main.go # MQTT subscription + packet processing
│ ├── decoder.go # Packet decoder (shared logic)
│ ├── db.go # SQLite write path
│ └── *_test.go # Ingestor tests
├── proto/ # Protobuf API definitions
├── public/ # Vanilla JS frontend (no build step)
│ ├── index.html # SPA shell
│ ├── app.js # Router, WebSocket, utilities
│ ├── packets.js # Packet feed + hex breakdown
│ ├── map.js # Leaflet map + route visualization
│ ├── live.js # Live trace + VCR playback
│ ├── channels.js # Channel chat
│ ├── nodes.js # Node directory + detail views
│ ├── analytics.js # Analytics dashboard
│ └── style.css # CSS variable theming (light/dark)
├── docker/
│ ├── supervisord-go.conf # Process manager (server + ingestor)
│ ├── mosquitto.conf # MQTT broker config
│ ├── Caddyfile # Reverse proxy + HTTPS
│ └── entrypoint-go.sh # Container entrypoint
├── Dockerfile # Multi-stage Go build + Alpine runtime
├── config.example.json # Example configuration
├── tests/ # Node.js test suite: unit/ (test-all.sh) and e2e/ (Playwright)
├── test-all.sh # Runs every suite in tests/unit
└── tools/ # Generators, E2E tests, utilities
For Developers
Building
make build # all four binaries for your machine, into dist/
make build-server # just one
make crossbuild # static linux/amd64 + linux/arm64 binaries
The SQLite driver is mattn/go-sqlite3, which
is cgo, so a plain GOOS=linux go build from a Mac will not work: cross-compiling
needs a C compiler that can target the other platform. make crossbuild uses
zig as that compiler (install it and it just
works) and links statically against musl, so the result is one self-contained file
that runs on Alpine or scratch. The container build does the same thing — see
Dockerfile.
Test Suite
Go tests cover the backend, Node.js unit tests in tests/unit/ cover the frontend logic, and Playwright tests in tests/e2e/ validate the UI in a browser.
# Go backend tests
cd cmd/server && go test ./... -v
cd cmd/ingestor && go test ./... -v
# Or across all 16 modules at once
make test
# Node.js frontend + integration tests
npm test
# Playwright E2E (requires running server on localhost:3000)
node tests/e2e/test-e2e-playwright.js
Generate Test Data
node tools/generate-packets.js --api --count 200
Migrating from Node.js
If you're running an existing Node.js deployment, see docs/go-migration.md for a step-by-step guide. The Go engine reads the same SQLite database and config.json — no data migration needed.
Contributing
Contributions welcome. Please read AGENTS.md for coding conventions, testing requirements, and engineering principles before submitting a PR.
Live instance: analyzer.on8ar.eu: all read endpoints are public, no auth required.
API Documentation: CoreScope auto-generates an OpenAPI 3.0 spec. Browse the interactive Swagger UI at /api/docs or fetch the machine-readable spec at /api/spec.
License
GPL-3.0-or-later