mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-26 04:54:02 +00:00
914bd4cf0e
## Show each channel message's region scope Each message in the Channels view now shows the region scope it was sent with, as a small chip in the meta line: the region name (for example `#be`), `unknown scope`, or nothing. This is the channel-message part of #1852 by @dborup, extracted as a focused change. #1852 was closed unmerged because it had grown to the whole fork diff. The implementation follows dborup's commitsc686ae3f,350bf7eeanda94d57edon dborup/CoreScope, adapted to current master. dborup is co-author on the commit. ### What changed - `cmd/server/db.go:2099,2195`: `GetChannelMessages` selects `t.scope_name` when the column exists and returns it as `scope_name`. - `cmd/server/store.go:5654`: the in-memory `GetChannelMessages` returns `scope_name`, so the field does not depend on which path serves the endpoint. - `cmd/server/store.go:2966,3244`: both WebSocket broadcast builders carry `scope_name`, so a live message shows its region immediately. - `public/channels.js:342,2278`: `messageScopeChipHtml` renders the chip with the existing `.sa-chip-declared` / `.sa-chip-unmatched` styles from `scope-audit.css`. The name goes through `escapeHtml`. No new CSS. - `public/channels.js:678,695,1435,1487`: the decrypt path and the WebSocket path keep `scope_name` on the message. ### Differences from #1852 - The field is `scope_name`, the name `/api/packets` already uses. - No `routeType` field. `transmissions.scope_name` already tells the states apart: NULL means no transport code, an empty string means a transport code that no configured region key matched. The frontend uses `??`, not `||`, so the empty string is kept. - A chip instead of `scope: <name>` text. The area label from later #1852 commits is not included. ### Perf One extra column per observation row in the page query (at most `limit` transmissions), and one extra map entry per broadcast observation. No new queries, loops or API calls. ### Tests - `cmd/server/channel_message_scope_name_test.go`: the three states through the DB query, the store, `/api/channels/{hash}/messages` over both paths, a schema without the column, and both broadcast builders. 5 of its 6 tests fail without the change; the sixth guards the missing-column case and passes either way. - `test-issue-1851-channel-message-scope.js`: the REST, WebSocket and client-side decrypt paths, escaping, and the name / unknown / none render. 4/4 fail without the change. Registered in `test-all.sh` and the unit step of `deploy.yml`. - Mutation checks: returning `nil` for `scope_name` in the DB path fails the DB and endpoint tests; `||` instead of `??` in the WebSocket path fails the WebSocket test. - `go test ./...` in `cmd/server`: ok. gofmt and go vet clean. ### Browser validation On a staging instance with live traffic (build `e84d2da6`), in Chrome: - `/api/channels/{hash}/messages` carries the `scope_name` key on every message in the 19 channels whose results I read. `#hamradio`, latest 50: 34 named, 1 empty string, 15 NULL. - Opening `#hamradio` renders 104 chips: `#nl` 53, `#be` 32, `#de` 11, `#eu` 6, `#bx` 1 and `unknown scope` 1, and no chip on unscoped messages. Chip text `rgb(26, 26, 46)` on `rgb(238, 242, 255)` in the light theme. ### Not verified - Dark theme not checked. - The real-decrypt branch of `decryptCandidates` has no test and was not exercised in the browser; the already-decrypted branch is tested. - Messages already in the client decrypt cache show no chip until they are decrypted again. - `go test -race` and the Playwright E2E suite were not run locally. Fixes #1851 ## Review follow-up (commit `50346589`) An independent review found no correctness or XSS problem and confirmed DB, store and WebSocket agree on the value. Changed: - **Real decrypt branch tested.** A new test runs the real AES+HMAC decrypt branch in `decryptCandidates` with one packet per scope state; deleting `scope_name` there now fails 2 of 7 tests. - **Tooltip wording.** The unknown-scope tooltip now says the scope "could not be matched to a single region on this instance" (`public/channels.js:338-347`). The ingestor stores an empty name both when no key matches and when several match without exactly one operator-configured key (`cmd/ingestor/region_keys.go:364-393`), so "matches none of the configured keys" was wrong for the second case. - **Old decrypt cache.** Decrypted messages cached before this change had no `scope_name` key and stayed chipless as long as the candidate count did not change. A cached message missing the key now forces one full decrypt; a cache that has it still takes the delta path. A test covers each case. - **Docs.** `docs/api-spec.md` documents `scope_name` on the channel messages response, with the null / empty string / name semantics. Corrections to the description: - **Test counts.** With the `db.go` and `store.go` changes reverted, 4 of the 5 top-level Go tests fail (6 of 7 counting subtests); only the missing-column test passes. - **Broadcast payload.** `scope_name` is added to `pkt`, which is copied into `broadcastMap` and also nested as `packet` (`store.go` ~2974-2980, ~3252-3257), so the key appears twice per observation: 36 bytes for `null`, 48 bytes for `"#belgium"`. - **Side effect on the Packets page.** The live table reads `m.data.packet` (`packets.js` ~1316-1318), so flat rows and expanded group children now show Scope for live packets. In grouped mode a new group copies a fixed field list without `scope_name` (~1384-1395) and shows the empty placeholder until reload. Before this PR every live row showed that placeholder, so this is not a regression. The three copies of the three-state scope rendering (`app.js`, `packets.js`, `channels.js`) are left as they are. --------- Co-authored-by: dborup <3627142+dborup@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
CoreScope
High-performance mesh network analyzer powered by Go. Sub-millisecond packet queries, ~300 MB memory for 56K+ packets, real-time WebSocket broadcast, full channel decryption.
Self-hosted, open-source MeshCore packet analyzer. Collects MeshCore packets via MQTT, decodes them in real time, and presents a full web UI with live packet feed, interactive maps, channel chat, packet tracing, and per-node analytics.
⚡ Performance
The Go backend serves all 40+ API endpoints from an in-memory packet store with 5 indexes (hash, txID, obsID, observer, node). SQLite is for persistence only — reads never touch disk.
| Metric | Value |
|---|---|
| Packet queries | < 1 ms (in-memory) |
| All API endpoints | < 100 ms |
| Memory (56K packets) | ~300 MB (vs 1.3 GB on Node.js) |
| WebSocket broadcast | Real-time to all connected browsers |
| Channel decryption | AES-128-ECB with rainbow table |
| GOMEMLIMIT (memory-constrained hosts) | set to ≥1.5× working set (e.g. 1536 MiB on a 2 GB Pi for a ~1 GB store). Lower values trigger a GC death-spiral. Configure via the GOMEMLIMIT env var or runtime.maxMemoryMB in config.json; env wins. Applies to both server and ingestor. See #1010. |
See PERFORMANCE.md for full benchmarks.
✨ Features
📡 Live Trace Map
Real-time animated map with packet route visualization, VCR-style playback controls, and a retro LCD clock. Replay the last 24 hours of mesh activity, scrub through the timeline, or watch packets flow live at up to 4× speed.
📦 Packet Feed
Filterable real-time packet stream with byte-level breakdown, Excel-like resizable columns, and a detail pane. Toggle "My Nodes" to focus on your mesh.
🗺️ Network Overview
At-a-glance mesh stats — node counts, packet volume, observer coverage.
📊 Node Analytics
Per-node deep dive with interactive charts: activity timeline, packet type breakdown, SNR distribution, hop count analysis, peer network graph, and hourly heatmap.
💬 Channel Chat
Decoded group messages with sender names, @mentions, timestamps — like reading a Discord channel for your mesh.
📱 Mobile Ready
Full experience on your phone — proper touch controls, iOS safe area support, and a compact VCR bar.
And More
- 11 Analytics Tabs — RF, topology, channels, hash stats, distance, route patterns, and more
- Node Directory — searchable list with role tabs, detail panel, QR codes, advert timeline
- Packet Tracing — follow individual packets across observers with SNR/RSSI timeline
- Observer Status — health monitoring, packet counts, uptime, per-observer analytics
- Hash Collision Matrix — detect address collisions across the mesh
- Channel Key Auto-Derivation — hashtag channels (
#channel) keys derived via SHA256 - Multi-Broker MQTT — connect to multiple brokers with per-source IATA filtering
- Dark / Light Mode — auto-detects system preference, map tiles swap too
- Theme Customizer — design your theme in-browser, export as
theme.json - Global Search — search packets, nodes, and channels (Ctrl+K)
- Shareable URLs — deep links to packets, channels, and observer detail pages
- Protobuf API Contract — typed API definitions in
proto/ - Accessible — ARIA patterns, keyboard navigation, screen reader support
Quick Start
Pre-built Image (Recommended)
No build step required — just run:
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 1883:1883 \
-v /your/data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
Open http://localhost — done. No config file needed; CoreScope starts with sensible defaults.
For HTTPS with a custom domain, add -p 443:443 and mount your Caddyfile:
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 443:443 -p 1883:1883 \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:latest
Disable built-in services with -e DISABLE_MOSQUITTO=true or -e DISABLE_CADDY=true, or drop a .env file in your data volume. See docs/deployment.md for the full reference.
Build from Source
git clone https://github.com/Kpa-clawbot/CoreScope.git
cd CoreScope
./manage.sh setup
The setup wizard walks you through config, domain, HTTPS, build, and run.
./manage.sh status # Health check + packet/node counts
./manage.sh logs # Follow logs
./manage.sh backup # Backup database
./manage.sh update # Pull latest + rebuild + restart
./manage.sh mqtt-test # Check if observer data is flowing
./manage.sh help # All commands
Configure
Copy config.example.json to config.json and edit:
{
"port": 3000,
"mqtt": {
"broker": "mqtt://localhost:1883",
"topic": "meshcore/+/+/packets"
},
"mqttSources": [
{
"name": "remote-feed",
"broker": "mqtts://remote-broker:8883",
"topics": ["meshcore/+/+/packets"],
"username": "user",
"password": "pass",
"iataFilter": ["SJC", "SFO", "OAK"]
}
],
"channelKeys": {
"public": "8b3387e9c5cdea6ac9e5edbaa115cd72"
},
"defaultRegion": "SJC"
}
| Field | Description |
|---|---|
port |
HTTP server port (default: 3000) |
mqtt.broker |
Local MQTT broker URL ("" to disable) |
mqttSources |
External MQTT broker connections (optional) |
channelKeys |
Channel decryption keys (hex). Hashtag channels auto-derived via SHA256 |
defaultRegion |
Default IATA region code for the UI |
dbPath |
SQLite database path (default: data/meshcore.db) |
Environment Variables
| Variable | Description |
|---|---|
PORT |
Override config port |
DB_PATH |
Override SQLite database path |
Architecture
┌─────────────────────────────────────────────┐
│ Docker Container │
│ │
Observer → USB → │ Mosquitto ──→ Go Ingestor ──→ SQLite DB │
meshcoretomqtt → MQTT ──→│ │ │
│ Go HTTP Server ──→ WebSocket │
│ │ │ │
│ Caddy (HTTPS) ←───────┘ │
└────────────────────┼────────────────────────┘
│
Browser
Two-process model: The Go ingestor handles MQTT ingestion and packet decoding. The Go HTTP server loads all packets into an in-memory store on startup (5 indexes for fast lookups) and serves the REST API + WebSocket broadcast. Both are managed by supervisord inside a single container with Caddy for HTTPS and Mosquitto for local MQTT.
MQTT Setup
- Flash an observer node with
MESH_PACKET_LOGGING=1build flag - Connect via USB to a host running meshcoretomqtt
- Configure meshcoretomqtt with your IATA region code and MQTT broker address
- Packets appear on topic
meshcore/{IATA}/{PUBKEY}/packets
Or POST raw hex packets to POST /api/packets for manual injection.
Project Structure
corescope/
├── cmd/
│ ├── server/ # Go HTTP server + WebSocket + REST API
│ │ ├── main.go # Entry point
│ │ ├── routes.go # 40+ API endpoint handlers
│ │ ├── store.go # In-memory packet store (5 indexes)
│ │ ├── db.go # SQLite persistence layer
│ │ ├── decoder.go # MeshCore packet decoder
│ │ ├── websocket.go # WebSocket broadcast
│ │ └── *_test.go # 327 test functions
│ └── ingestor/ # Go MQTT ingestor
│ ├── main.go # MQTT subscription + packet processing
│ ├── decoder.go # Packet decoder (shared logic)
│ ├── db.go # SQLite write path
│ └── *_test.go # 53 test functions
├── proto/ # Protobuf API definitions
├── public/ # Vanilla JS frontend (no build step)
│ ├── index.html # SPA shell
│ ├── app.js # Router, WebSocket, utilities
│ ├── packets.js # Packet feed + hex breakdown
│ ├── map.js # Leaflet map + route visualization
│ ├── live.js # Live trace + VCR playback
│ ├── channels.js # Channel chat
│ ├── nodes.js # Node directory + detail views
│ ├── analytics.js # 11-tab analytics dashboard
│ └── style.css # CSS variable theming (light/dark)
├── docker/
│ ├── supervisord-go.conf # Process manager (server + ingestor)
│ ├── mosquitto.conf # MQTT broker config
│ ├── Caddyfile # Reverse proxy + HTTPS
│ └── entrypoint-go.sh # Container entrypoint
├── Dockerfile # Multi-stage Go build + Alpine runtime
├── config.example.json # Example configuration
├── test-*.js # Node.js test suite (frontend + legacy)
└── tools/ # Generators, E2E tests, utilities
For Developers
Test Suite
380 Go tests covering the backend, plus 150+ Node.js tests for the frontend and legacy logic, plus 49 Playwright E2E tests for browser validation.
# Go backend tests
cd cmd/server && go test ./... -v
cd cmd/ingestor && go test ./... -v
# Node.js frontend + integration tests
npm test
# Playwright E2E (requires running server on localhost:3000)
node test-e2e-playwright.js
Generate Test Data
node tools/generate-packets.js --api --count 200
Migrating from Node.js
If you're running an existing Node.js deployment, see docs/go-migration.md for a step-by-step guide. The Go engine reads the same SQLite database and config.json — no data migration needed.
Contributing
Contributions welcome. Please read AGENTS.md for coding conventions, testing requirements, and engineering principles before submitting a PR.
Live instance: analyzer.00id.net — all API endpoints are public, no auth required.
API Documentation: CoreScope auto-generates an OpenAPI 3.0 spec. Browse the interactive Swagger UI at /api/docs or fetch the machine-readable spec at /api/spec.
License
GPL-3.0-or-later