mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-10-23 06:52:21 +00:00
efitenandClaude Opus 5
0fea3f2a75
## Summary Adds a breakdown of flood adverts by sender role to `/api/scope-stats` and the Scopes tab, in the descriptive shape agreed in #1979: per node role, how many flood adverts were unscoped, scoped with an unnamed region, or scoped with a named region. It reports what was sent, not why. ## Changes - `cmd/server/db.go:3114-3144`: one grouped query in `GetScopeStats`. ADVERT packets on flood routes (TRANSPORT_FLOOD 0, FLOOD 1) in the window, `LEFT JOIN nodes` on `from_pubkey`, split by the three `scope_name` states (NULL, empty string, name). A missing or empty role becomes `"unknown"`. Ordered by total descending, then role. Zero-hop adverts are excluded because firmware sends them as DIRECT/TRANSPORT_DIRECT (`src/Mesh.cpp:717-730`, `Mesh::sendZeroHop`), so they would inflate "unscoped". - `cmd/server/types.go:116-133`: `ScopeAdvertRoleCount` and `ScopeStatsResponse.AdvertsByRole` (`advertsByRole`, always an array). - `public/analytics.js:4760`: `scopeAdvertsByRoleHtml` renders a table under the time-series chart with the count per state and its share of the row. Role text is escaped. It reuses the existing `/scope-stats` response, so there is no extra request. - `docs/api-spec.md:1763-1786`: documents the new field. `/api/scope-stats` is in `openapi_known_gaps.json`, so there is no `openapi.go` entry to update. API addition (existing fields unchanged): "advertsByRole": [ { "role": "repeater", "unscoped": 7741, "unknownScope": 7, "named": 1562 } ] ## Performance The query runs inside `GetScopeStats`, so it shares the existing 30s cache per window. The unary `+` on `payload_type` keeps SQLite on the `first_seen` range index. Without it the planner picked the `payload_type` index and walked every stored advert whatever the window. Read-only timing on a production DB (1,063,345 transmissions, 161,634 adverts, sqlite3 CLI 3.45.1): | Window | payload_type index | first_seen index (this PR) | |---|---|---| | 7d | 0.231s | 0.059s | | 24h | 0.214s | 0.008s | | 1h | 0.213s | 0.001s | ## Tests - `TestGetScopeStatsAdvertsByRole` (`cmd/server/db_test.go:2295`): the three states, flood-only routes, non-advert and out-of-window exclusion, `unknown` for a missing node row, an empty role and a NULL `from_pubkey`, and ordering. Mutation checked: widening to routes 0-3 and dropping the empty-role fallback both fail it. - `TestGetScopeStatsAdvertsByRoleEmpty` (`:2372`): empty result is `[]`, not null. - `test-issue-1979-scope-adverts-by-role.js`: renders the real `analytics.js` helper in a vm sandbox. Covers row order, totals and shares, columns, escaping (mutation checked), the empty state, and the non-causal caption. Registered in `test-all.sh` and the deploy.yml unit step. - `go test ./...` in `cmd/server` passes, gofmt and go vet are clean, `check-css-vars.js` OK, `check-xss-sinks.sh --diff` exits 0. ## Browser validation On a staging instance with live traffic (build `139e484e`, together with #1074's branch), in Chrome, no console errors: `/#/analytics?tab=scopes` shows "Flood adverts by node role" under the time-series chart with its caption, and a table of 6 roles for the default window, for example `repeater 1.361 | 1.109 (81.5%) | 2 (0.1%) | 250 (18.4%)`. Shares in each row add up to 100%. `/api/scope-stats?window=7d` returns `advertsByRole` with the same six roles. ## Not verified - Timings are from the sqlite3 CLI, not the modernc driver in the server process. - Role is the sender's current `nodes.role`. A node whose advert type changed within the window is counted under its latest role. The data also contains a raw `type-13` role, shown as is. - Switching the window in the browser was not exercised; the API was checked for 7d. - No Playwright E2E added. Fixes #1979 ## Review follow-up (commit `43af46b8`) An independent review found no correctness, security or performance problem: counts are per transmission, the window matches the rest of the Scopes tab, zero-hop adverts are DIRECT per firmware, and the `+t.payload_type` hint holds on modernc SQLite 3.46.0. It found two test gaps and a docs gap. Changed: - **Ordering.** The Go fixture gave companion, repeater and unknown 3 adverts each, so ordering by total was never checked (`ORDER BY COUNT(*) ASC` still passed). The fixture now has repeater 4, unknown 3, companion 2, sensor 2, so the expected order differs from alphabetical and the companion/sensor tie checks the role-name tie-break. Reversing the count order, dropping it, or reversing the tie-break now fails the test. - **Column positions.** The JS test only checked that each cell string appeared somewhere in the row, so swapping two columns passed. It now compares each row's cells and the header cells by position; both swaps fail. - **Docs.** `docs/api-spec.md` and the `ScopeAdvertRoleCount` comment now name every source of `unknown`: a NULL `from_pubkey` (legacy rows the #1143 backfill has not reached), a sender with no `nodes` row, including one moved to `inactive_nodes` by node retention (inside the 7d window only with `retention.nodeDays` below 7), and an empty role. Not added: a query-plan test pinning the `+t.payload_type` hint. The SQL is inline in `GetScopeStats`, so the test would have to copy it or the query would have to move into a constant; left for a follow-up if wanted. The raw `type-13` role in the table is the ingestor's placeholder for reserved advert types 5-15 (`cmd/ingestor/decoder.go:1229`, #1279), shown as is. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
CoreScope
High-performance mesh network analyzer powered by Go. Sub-millisecond packet queries, ~300 MB memory for 56K+ packets, real-time WebSocket broadcast, full channel decryption.
Self-hosted, open-source MeshCore packet analyzer. Collects MeshCore packets via MQTT, decodes them in real time, and presents a full web UI with live packet feed, interactive maps, channel chat, packet tracing, and per-node analytics.
⚡ Performance
The Go backend serves all 40+ API endpoints from an in-memory packet store with 5 indexes (hash, txID, obsID, observer, node). SQLite is for persistence only — reads never touch disk.
| Metric | Value |
|---|---|
| Packet queries | < 1 ms (in-memory) |
| All API endpoints | < 100 ms |
| Memory (56K packets) | ~300 MB (vs 1.3 GB on Node.js) |
| WebSocket broadcast | Real-time to all connected browsers |
| Channel decryption | AES-128-ECB with rainbow table |
| GOMEMLIMIT (memory-constrained hosts) | set to ≥1.5× working set (e.g. 1536 MiB on a 2 GB Pi for a ~1 GB store). Lower values trigger a GC death-spiral. Configure via the GOMEMLIMIT env var or runtime.maxMemoryMB in config.json; env wins. Applies to both server and ingestor. See #1010. |
See PERFORMANCE.md for full benchmarks.
✨ Features
📡 Live Trace Map
Real-time animated map with packet route visualization, VCR-style playback controls, and a retro LCD clock. Replay the last 24 hours of mesh activity, scrub through the timeline, or watch packets flow live at up to 4× speed.
📦 Packet Feed
Filterable real-time packet stream with byte-level breakdown, Excel-like resizable columns, and a detail pane. Toggle "My Nodes" to focus on your mesh.
🗺️ Network Overview
At-a-glance mesh stats — node counts, packet volume, observer coverage.
📊 Node Analytics
Per-node deep dive with interactive charts: activity timeline, packet type breakdown, SNR distribution, hop count analysis, peer network graph, and hourly heatmap.
💬 Channel Chat
Decoded group messages with sender names, @mentions, timestamps — like reading a Discord channel for your mesh.
📱 Mobile Ready
Full experience on your phone — proper touch controls, iOS safe area support, and a compact VCR bar.
And More
- 11 Analytics Tabs — RF, topology, channels, hash stats, distance, route patterns, and more
- Node Directory — searchable list with role tabs, detail panel, QR codes, advert timeline
- Packet Tracing — follow individual packets across observers with SNR/RSSI timeline
- Observer Status — health monitoring, packet counts, uptime, per-observer analytics
- Hash Collision Matrix — detect address collisions across the mesh
- Channel Key Auto-Derivation — hashtag channels (
#channel) keys derived via SHA256 - Multi-Broker MQTT — connect to multiple brokers with per-source IATA filtering
- Dark / Light Mode — auto-detects system preference, map tiles swap too
- Theme Customizer — design your theme in-browser, export as
theme.json - Global Search — search packets, nodes, and channels (Ctrl+K)
- Shareable URLs — deep links to packets, channels, and observer detail pages
- Protobuf API Contract — typed API definitions in
proto/ - Accessible — ARIA patterns, keyboard navigation, screen reader support
Quick Start
Pre-built Image (Recommended)
No build step required — just run:
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 1883:1883 \
-v /your/data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
Open http://localhost — done. No config file needed; CoreScope starts with sensible defaults.
For HTTPS with a custom domain, add -p 443:443 and mount your Caddyfile:
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 443:443 -p 1883:1883 \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:latest
Disable built-in services with -e DISABLE_MOSQUITTO=true or -e DISABLE_CADDY=true, or drop a .env file in your data volume. See docs/deployment.md for the full reference.
Build from Source
git clone https://github.com/Kpa-clawbot/CoreScope.git
cd CoreScope
./manage.sh setup
The setup wizard walks you through config, domain, HTTPS, build, and run.
./manage.sh status # Health check + packet/node counts
./manage.sh logs # Follow logs
./manage.sh backup # Backup database
./manage.sh update # Pull latest + rebuild + restart
./manage.sh mqtt-test # Check if observer data is flowing
./manage.sh help # All commands
Configure
Copy config.example.json to config.json and edit:
{
"port": 3000,
"mqtt": {
"broker": "mqtt://localhost:1883",
"topic": "meshcore/+/+/packets"
},
"mqttSources": [
{
"name": "remote-feed",
"broker": "mqtts://remote-broker:8883",
"topics": ["meshcore/+/+/packets"],
"username": "user",
"password": "pass",
"iataFilter": ["SJC", "SFO", "OAK"]
}
],
"channelKeys": {
"public": "8b3387e9c5cdea6ac9e5edbaa115cd72"
},
"defaultRegion": "SJC"
}
| Field | Description |
|---|---|
port |
HTTP server port (default: 3000) |
mqtt.broker |
Local MQTT broker URL ("" to disable) |
mqttSources |
External MQTT broker connections (optional) |
channelKeys |
Channel decryption keys (hex). Hashtag channels auto-derived via SHA256 |
defaultRegion |
Default IATA region code for the UI |
dbPath |
SQLite database path (default: data/meshcore.db) |
Environment Variables
| Variable | Description |
|---|---|
PORT |
Override config port |
DB_PATH |
Override SQLite database path |
Architecture
┌─────────────────────────────────────────────┐
│ Docker Container │
│ │
Observer → USB → │ Mosquitto ──→ Go Ingestor ──→ SQLite DB │
meshcoretomqtt → MQTT ──→│ │ │
│ Go HTTP Server ──→ WebSocket │
│ │ │ │
│ Caddy (HTTPS) ←───────┘ │
└────────────────────┼────────────────────────┘
│
Browser
Two-process model: The Go ingestor handles MQTT ingestion and packet decoding. The Go HTTP server loads all packets into an in-memory store on startup (5 indexes for fast lookups) and serves the REST API + WebSocket broadcast. Both are managed by supervisord inside a single container with Caddy for HTTPS and Mosquitto for local MQTT.
MQTT Setup
- Flash an observer node with
MESH_PACKET_LOGGING=1build flag - Connect via USB to a host running meshcoretomqtt
- Configure meshcoretomqtt with your IATA region code and MQTT broker address
- Packets appear on topic
meshcore/{IATA}/{PUBKEY}/packets
Or POST raw hex packets to POST /api/packets for manual injection.
Project Structure
corescope/
├── cmd/
│ ├── server/ # Go HTTP server + WebSocket + REST API
│ │ ├── main.go # Entry point
│ │ ├── routes.go # 40+ API endpoint handlers
│ │ ├── store.go # In-memory packet store (5 indexes)
│ │ ├── db.go # SQLite persistence layer
│ │ ├── decoder.go # MeshCore packet decoder
│ │ ├── websocket.go # WebSocket broadcast
│ │ └── *_test.go # 327 test functions
│ └── ingestor/ # Go MQTT ingestor
│ ├── main.go # MQTT subscription + packet processing
│ ├── decoder.go # Packet decoder (shared logic)
│ ├── db.go # SQLite write path
│ └── *_test.go # 53 test functions
├── proto/ # Protobuf API definitions
├── public/ # Vanilla JS frontend (no build step)
│ ├── index.html # SPA shell
│ ├── app.js # Router, WebSocket, utilities
│ ├── packets.js # Packet feed + hex breakdown
│ ├── map.js # Leaflet map + route visualization
│ ├── live.js # Live trace + VCR playback
│ ├── channels.js # Channel chat
│ ├── nodes.js # Node directory + detail views
│ ├── analytics.js # 11-tab analytics dashboard
│ └── style.css # CSS variable theming (light/dark)
├── docker/
│ ├── supervisord-go.conf # Process manager (server + ingestor)
│ ├── mosquitto.conf # MQTT broker config
│ ├── Caddyfile # Reverse proxy + HTTPS
│ └── entrypoint-go.sh # Container entrypoint
├── Dockerfile # Multi-stage Go build + Alpine runtime
├── config.example.json # Example configuration
├── test-*.js # Node.js test suite (frontend + legacy)
└── tools/ # Generators, E2E tests, utilities
For Developers
Test Suite
380 Go tests covering the backend, plus 150+ Node.js tests for the frontend and legacy logic, plus 49 Playwright E2E tests for browser validation.
# Go backend tests
cd cmd/server && go test ./... -v
cd cmd/ingestor && go test ./... -v
# Node.js frontend + integration tests
npm test
# Playwright E2E (requires running server on localhost:3000)
node test-e2e-playwright.js
Generate Test Data
node tools/generate-packets.js --api --count 200
Migrating from Node.js
If you're running an existing Node.js deployment, see docs/go-migration.md for a step-by-step guide. The Go engine reads the same SQLite database and config.json — no data migration needed.
Contributing
Contributions welcome. Please read AGENTS.md for coding conventions, testing requirements, and engineering principles before submitting a PR.
Live instance: analyzer.00id.net — all API endpoints are public, no auth required.
API Documentation: CoreScope auto-generates an OpenAPI 3.0 spec. Browse the interactive Swagger UI at /api/docs or fetch the machine-readable spec at /api/spec.
License
GPL-3.0-or-later