dandri/meshcore-analyzer
1
0 Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-10-23 12:53:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
fbeb9e4d71a7bfaebed198e008169a2b91774ca4
meshcore-analyzer/README.md
T
efitenandClaude Opus 5.5 59708ee72e ci: publish the README badges to a badges branch with GITHUB_TOKEN (#2202) 
Fixes #2136.

## Problem

The README badges are stale: `e2e-tests` shows 821 passed (1161 when
#2136 was filed), both Go coverage badges have not changed since
2026-03-30 and frontend coverage since June. There are two causes.

1. **The PAT.** The `publish` job wrote the badge JSON to `master`
through the Contents API with `BADGE_PUSH_TOKEN`, a PAT that could push
past the branch protection on `master`. That token returns 401: run
38360160932 on 2026-10-13 logs `gh: Bad credentials (HTTP 401)` for
every badge. The step is `continue-on-error` and only echoed `Failed to
update`, so every run stayed green.
2. **Old badges inside `e2e-badges`.** `.badges/` is tracked on
`master`, so every e2e shard checkout and the aggregate checkout already
hold the old files. The shard upload and the aggregate `cp` shipped them
in `e2e-badges`, and `publish` downloaded `e2e-badges` after
`go-badges`, so the old values overwrote the fresh ones. In run
38407143126, `go-badges` holds 85.8% and 77.1%, while `e2e-badges` holds
85.4%, 70.2% and frontend 36.64%, the same as `master`'s `.badges/`.
Fixing only the token would have left three of the four README badges
frozen.

## Change

1. `5f430966` `ci:` the `publish` job pushes the badges to an
unprotected `badges` branch with its own `GITHUB_TOKEN`.
- Job-level `permissions: contents: write`. No other job gains a
permission. The checkout uses `persist-credentials: false`, since the
push passes its own token.
- The branch is a single orphan commit, force-pushed, so it does not
grow. Master push runs are already serialised by the workflow
`concurrency` group.
- Each run starts from the branch as it is and overwrites only the
badges it downloaded, so a missing artifact keeps those badges at their
previous value. The first time, when `git ls-remote --exit-code` reports
the branch absent (exit 2), it starts from `master`'s `.badges/`.
- No downloaded artifacts, or any other `ls-remote`, fetch, extract,
commit or push failure, ends in a `::warning::` in the run summary and
leaves the branch alone. The step stays `continue-on-error`, so a badge
problem never turns the pipeline red.
   - `go-badges` is downloaded after `e2e-badges`, so its files win.
2. `2134c1e5` `docs:` the four README badge URLs read
`.../CoreScope/badges/<name>.json` instead of
`.../master/.badges/<name>.json`.
3. `47e57482` `ci:` the frontend-coverage step in the e2e shards and the
aggregate badge step start from an empty `.badges/`, so `e2e-badges`
carries only what the run produced. This also stops shard 3's old
`frontend-coverage.json` from replacing the one shard 2 generates.

No PAT is involved any more, so nothing expires.

## Testing

- The publish step script, taken from the YAML, run against a local bare
repository standing in for `origin`:

| Run | Downloaded | Result on `badges` |
|---|---|---|
| 1, branch absent | a, b | a, b new; c from `.badges/`; one commit |
| 2 | c only | a, b kept from run 1; c new; still one commit |
| 3 | none | `::warning::`, branch unchanged, step exit 0 |
| 4, `origin` unreachable for the read | a | `::warning::`, branch
unchanged, step exit 0 |
| 5, push to a missing remote | a | `::warning::`, branch unchanged,
step exit 0 |

- The artifacts of run 38407143126 land flat in the download path, so
`badges-new/*.json` matches them.
- `shellcheck` at style level on the step script: no findings.
- `actionlint` 1.7.7 (shellcheck off): one finding, the self-hosted
runner label `meshcore-runner-2`, which `master` reports as well.

**Not tested:** the e2e change and the push with a real `GITHUB_TOKEN`.
Both only run on a push to `master`, so the first master run after merge
is the real test. The `badges` branch already exists (`80fda6c0`, a copy
of `master`'s `.badges/`), so the README badges keep working from the
merge on, with the old values until that run. After it, I will check the
`badges` branch contents against that run's go and frontend numbers, the
step log and the rendered README.

## Not in this PR

- Removing the now unused `.badges/` from `master`: the first run seeds
the branch from it, so that goes in a follow-up once the branch exists.
- Deleting the `BADGE_PUSH_TOKEN` secret: that needs a repository admin.
- `frontend-tests.json`: no CI step produces it and no README badge uses
it. It is carried over to the branch unchanged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:25:42 +02:00

294 lines
13 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CoreScope
[![Go Server Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/badges/go-server-coverage.json)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml)
[![Go Ingestor Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/badges/go-ingestor-coverage.json)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml)
[![E2E Tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/badges/e2e-tests.json)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml)
[![Frontend Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/badges/frontend-coverage.json)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml)
[![Deploy](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml/badge.svg)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml)
> High-performance mesh network analyzer powered by Go. Sub-millisecond packet queries, ~300 MB memory for 56K+ packets, real-time WebSocket broadcast, full channel decryption.
Self-hosted, open-source MeshCore packet analyzer. Collects MeshCore packets via MQTT, decodes them in real time, and presents a full web UI with live packet feed, interactive maps, channel chat, packet tracing, and per-node analytics.
## ⚡ Performance
The Go backend serves all 40+ API endpoints from an in-memory packet store with 5 indexes (hash, txID, obsID, observer, node). SQLite is for persistence only — reads never touch disk.
| Metric | Value |
|--------|-------|
| Packet queries | **< 1 ms** (in-memory) |
| All API endpoints | **< 100 ms** |
| Memory (56K packets) | **~300 MB** (vs 1.3 GB on Node.js) |
| WebSocket broadcast | **Real-time** to all connected browsers |
| Channel decryption | **AES-128-ECB** with rainbow table |
| GOMEMLIMIT (memory-constrained hosts) | **set to ≥1.5× working set** (e.g. 1536 MiB on a 2 GB Pi for a ~1 GB store). Lower values trigger a GC death-spiral. Configure via the `GOMEMLIMIT` env var or `runtime.maxMemoryMB` in `config.json`; env wins. Applies to both server and ingestor. See [#1010](https://github.com/Kpa-clawbot/CoreScope/issues/1010). |
See [PERFORMANCE.md](PERFORMANCE.md) for full benchmarks.
## ✨ Features
### 📡 Live Trace Map
Real-time animated map with packet route visualization, VCR-style playback controls, and a retro LCD clock. Replay the last 24 hours of mesh activity, scrub through the timeline, or watch packets flow live at up to 4× speed.
![Live VCR playback — watch packets flow across the Bay Area mesh](docs/screenshots/MeshVCR.gif)
### 📦 Packet Feed
Filterable real-time packet stream with byte-level breakdown, Excel-like resizable columns, and a detail pane. Toggle "My Nodes" to focus on your mesh.
![Packets view](docs/screenshots/packets1.png)
### 🗺️ Network Overview
At-a-glance mesh stats — node counts, packet volume, observer coverage.
![Network overview](docs/screenshots/mesh-overview.png)
### 📊 Node Analytics
Per-node deep dive with interactive charts: activity timeline, packet type breakdown, SNR distribution, hop count analysis, peer network graph, and hourly heatmap.
![Node analytics](docs/screenshots/node-analytics.png)
### 💬 Channel Chat
Decoded group messages with sender names, @mentions, timestamps — like reading a Discord channel for your mesh.
![Channels](docs/screenshots/channels1.png)
### 📱 Mobile Ready
Full experience on your phone — proper touch controls, iOS safe area support, and a compact VCR bar.
<img src="docs/screenshots/Live-view-iOS.png" alt="Live view on iOS" width="300">
### And More
- **11 Analytics Tabs** — RF, topology, channels, hash stats, distance, route patterns, and more
- **Node Directory** — searchable list with role tabs, detail panel, QR codes, advert timeline
- **Packet Tracing** — follow individual packets across observers with SNR/RSSI timeline
- **Observer Status** — health monitoring, packet counts, uptime, per-observer analytics
- **Hash Collision Matrix** — detect address collisions across the mesh
- **Channel Key Auto-Derivation** — hashtag channels (`#channel`) keys derived via SHA256
- **Multi-Broker MQTT** — connect to multiple brokers with per-source IATA filtering
- **Dark / Light Mode** — auto-detects system preference, map tiles swap too
- **Theme Customizer** — design your theme in-browser, export as `theme.json`
- **Global Search** — search packets, nodes, and channels (Ctrl+K)
- **Shareable URLs** — deep links to packets, channels, and observer detail pages
- **Protobuf API Contract** — typed API definitions in `proto/`
- **Accessible** — ARIA patterns, keyboard navigation, screen reader support
## Quick Start
### Pre-built Image (Recommended)
No build step required — just run:
```bash
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 1883:1883 \
-v /your/data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
```
Open `http://localhost` — done. No config file needed; CoreScope starts with sensible defaults.
For HTTPS with a custom domain, add `-p 443:443` and mount your Caddyfile:
```bash
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 443:443 -p 1883:1883 \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:latest
```
Disable built-in services with `-e DISABLE_MOSQUITTO=true` or `-e DISABLE_CADDY=true`, or drop a `.env` file in your data volume. See [docs/deployment.md](docs/deployment.md) for the full reference.
### Build from Source
```bash
git clone https://github.com/Kpa-clawbot/CoreScope.git
cd CoreScope
./manage.sh setup
```
The setup wizard walks you through config, domain, HTTPS, build, and run.
```bash
./manage.sh status # Health check + packet/node counts
./manage.sh logs # Follow logs
./manage.sh backup # Backup database
./manage.sh update # Pull latest + rebuild + restart
./manage.sh mqtt-test # Check if observer data is flowing
./manage.sh help # All commands
```
### Configure
Copy `config.example.json` to `config.json` and edit:
```json
{
"port": 3000,
"mqtt": {
"broker": "mqtt://localhost:1883",
"topic": "meshcore/+/+/packets"
},
"mqttSources": [
{
"name": "remote-feed",
"broker": "mqtts://remote-broker:8883",
"topics": ["meshcore/+/+/packets"],
"username": "user",
"password": "pass",
"iataFilter": ["SJC", "SFO", "OAK"]
}
],
"channelKeys": {
"public": "8b3387e9c5cdea6ac9e5edbaa115cd72"
},
"defaultRegion": "SJC"
}
```
| Field | Description |
|-------|-------------|
| `port` | HTTP server port (default: 3000) |
| `mqtt.broker` | Local MQTT broker URL (`""` to disable) |
| `mqttSources` | External MQTT broker connections (optional) |
| `channelKeys` | Channel decryption keys (hex). Hashtag channels auto-derived via SHA256 |
| `defaultRegion` | Default IATA region code for the UI |
| `dbPath` | SQLite database path (default: `data/meshcore.db`) |
### Environment Variables
| Variable | Description |
|----------|-------------|
| `PORT` | Override config port |
| `DB_PATH` | Override SQLite database path |
## Architecture
```
┌─────────────────────────────────────────────┐
│ Docker Container │
│ │
Observer → USB → │ Mosquitto ──→ Go Ingestor ──→ SQLite DB │
meshcoretomqtt → MQTT ──→│ │ │
│ Go HTTP Server ──→ WebSocket │
│ │ │ │
│ Caddy (HTTPS) ←───────┘ │
└────────────────────┼────────────────────────┘
│
Browser
```
**Two-process model:** The Go ingestor handles MQTT ingestion and packet decoding. The Go HTTP server loads all packets into an in-memory store on startup (5 indexes for fast lookups) and serves the REST API + WebSocket broadcast. Both are managed by supervisord inside a single container with Caddy for HTTPS and Mosquitto for local MQTT.
## MQTT Setup
1. **Flash an observer node** with `MESH_PACKET_LOGGING=1` build flag
2. **Connect via USB** to a host running [meshcoretomqtt](https://github.com/Cisien/meshcoretomqtt)
3. **Configure meshcoretomqtt** with your IATA region code and MQTT broker address
4. **Packets appear** on topic `meshcore/{IATA}/{PUBKEY}/packets`
Or POST raw hex packets to `POST /api/packets` for manual injection.
## Project Structure
```
corescope/
├── cmd/
│ ├── server/ # Go HTTP server + WebSocket + REST API
│ │ ├── main.go # Entry point
│ │ ├── routes.go # 40+ API endpoint handlers
│ │ ├── store.go # In-memory packet store (5 indexes)
│ │ ├── db.go # SQLite persistence layer
│ │ ├── decoder.go # MeshCore packet decoder
│ │ ├── websocket.go # WebSocket broadcast
│ │ └── *_test.go # 327 test functions
│ └── ingestor/ # Go MQTT ingestor
│ ├── main.go # MQTT subscription + packet processing
│ ├── decoder.go # Packet decoder (shared logic)
│ ├── db.go # SQLite write path
│ └── *_test.go # 53 test functions
├── proto/ # Protobuf API definitions
├── public/ # Vanilla JS frontend (no build step)
│ ├── index.html # SPA shell
│ ├── app.js # Router, WebSocket, utilities
│ ├── packets.js # Packet feed + hex breakdown
│ ├── map.js # Leaflet map + route visualization
│ ├── live.js # Live trace + VCR playback
│ ├── channels.js # Channel chat
│ ├── nodes.js # Node directory + detail views
│ ├── analytics.js # 11-tab analytics dashboard
│ └── style.css # CSS variable theming (light/dark)
├── docker/
│ ├── supervisord-go.conf # Process manager (server + ingestor)
│ ├── mosquitto.conf # MQTT broker config
│ ├── Caddyfile # Reverse proxy + HTTPS
│ └── entrypoint-go.sh # Container entrypoint
├── Dockerfile # Multi-stage Go build + Alpine runtime
├── config.example.json # Example configuration
├── tests/ # Node.js test suite: unit/ (test-all.sh) and e2e/ (Playwright)
├── test-all.sh # Runs every suite in tests/unit
└── tools/ # Generators, E2E tests, utilities
```
## For Developers
### Building
```bash
make build # all four binaries for your machine, into dist/
make build-server # just one
make crossbuild # static linux/amd64 + linux/arm64 binaries
```
The SQLite driver is [`mattn/go-sqlite3`](https://github.com/mattn/go-sqlite3), which
is cgo, so a plain `GOOS=linux go build` from a Mac will not work: cross-compiling
needs a C compiler that can target the other platform. `make crossbuild` uses
[`zig`](https://ziglang.org/download/) as that compiler (install it and it just
works) and links statically against musl, so the result is one self-contained file
that runs on Alpine or scratch. The container build does the same thing — see
`Dockerfile`.
### Test Suite
**380 Go tests** covering the backend, plus **150+ Node.js tests** for the frontend and legacy logic, plus **49 Playwright E2E tests** for browser validation.
```bash
# Go backend tests
cd cmd/server && go test ./... -v
cd cmd/ingestor && go test ./... -v
# Or across all 14 modules at once
make test
# Node.js frontend + integration tests
npm test
# Playwright E2E (requires running server on localhost:3000)
node tests/e2e/test-e2e-playwright.js
```
### Generate Test Data
```bash
node tools/generate-packets.js --api --count 200
```
### Migrating from Node.js
If you're running an existing Node.js deployment, see [docs/go-migration.md](docs/go-migration.md) for a step-by-step guide. The Go engine reads the same SQLite database and `config.json` — no data migration needed.
## Contributing
Contributions welcome. Please read [AGENTS.md](AGENTS.md) for coding conventions, testing requirements, and engineering principles before submitting a PR.
**Live instance:** [analyzer.00id.net](https://analyzer.00id.net) — all API endpoints are public, no auth required.
**API Documentation:** CoreScope auto-generates an OpenAPI 3.0 spec. Browse the interactive Swagger UI at [`/api/docs`](https://analyzer.00id.net/api/docs) or fetch the machine-readable spec at [`/api/spec`](https://analyzer.00id.net/api/spec).
## License
GPL-3.0-or-later
Reference in new issue View Git Blame Copy Permalink