Compare commits

..

3 Commits

Author SHA1 Message Date
you 5190139746 fix: data race on backfillTotal, O(n*chunks) rescan, unchecked stmt.Exec
- backfillTotal: plain int64 → atomic.Int64 (written by goroutine, read by
  stats handler — data race under -race)
- Collect all pending obs refs in a single pass upfront, then slice into
  chunks. Eliminates repeated full-scan of store.packets per chunk.
- Check stmt.Exec error return instead of silently discarding it.
- Scope X-CoreScope-Status header to /api/ paths only (skip static files).
2026-04-05 07:23:07 +00:00
you 2c25017185 fix: use atomic.Int64 for backfillTotal to prevent data race 2026-04-05 07:22:09 +00:00
you 8d3ff0ada3 perf: async chunked backfill — HTTP serves within 2 minutes (#612)
Move backfillResolvedPaths from blocking startup to a background goroutine
that runs after ListenAndServe. The HTTP server now starts immediately
after Load() + index builds (~2 min), regardless of database size.

Changes:
- New backfillResolvedPathsAsync() processes observations in chunks of 5K
  with 100ms yields between batches to avoid starving HTTP handlers
- Re-picks best observation per-chunk for affected transmissions
- Adds backfillComplete atomic flag + progress counters to PacketStore
- X-CoreScope-Status response header: 'backfilling' or 'ready' on all
  API responses via middleware
- /api/stats includes backfilling (bool) and backfillProgress (0.0-1.0)
- Logs progress: [store] backfill progress: 50000/7100000 observations (0.7%)

The original synchronous backfillResolvedPaths is preserved for potential
use in tests or one-off scripts.
2026-04-05 07:17:52 +00:00
36 changed files with 265 additions and 2962 deletions
+2 -2
View File
@@ -236,7 +236,7 @@ jobs:
build:
name: "🏗️ Build Docker Image"
needs: [e2e-test]
runs-on: [self-hosted, meshcore-runner-2]
runs-on: [self-hosted, meshcore-vm]
steps:
- name: Checkout code
uses: actions/checkout@v5
@@ -271,7 +271,7 @@ jobs:
name: "🚀 Deploy Staging"
if: github.event_name == 'push'
needs: [build]
runs-on: [self-hosted, meshcore-runner-2]
runs-on: [self-hosted, meshcore-vm]
steps:
- name: Checkout code
uses: actions/checkout@v5
-67
View File
@@ -1,67 +0,0 @@
name: Publish Docker Image
on:
push:
tags: ['v*']
branches: [master]
workflow_dispatch:
jobs:
publish:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/kpa-clawbot/corescope
tags: |
# On tag push: v1.2.3, v1.2, v1, latest
type=semver,pattern=v{{version}}
type=semver,pattern=v{{major}}.{{minor}}
type=semver,pattern=v{{major}}
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
# On master push: edge
type=edge,branch=master
- name: Set build time
id: buildtime
run: echo "value=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> "$GITHUB_OUTPUT"
- name: Compute app version
id: appversion
run: |
if [[ "${{ github.ref }}" == refs/tags/v* ]]; then
echo "value=${{ github.ref_name }}" >> "$GITHUB_OUTPUT"
else
echo "value=${{ github.sha }}" >> "$GITHUB_OUTPUT"
fi
- uses: docker/build-push-action@v6
with:
context: .
push: true
platforms: linux/amd64,linux/arm64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
APP_VERSION=${{ steps.appversion.outputs.value }}
GIT_COMMIT=${{ github.sha }}
BUILD_TIME=${{ steps.buildtime.outputs.value }}
cache-from: type=gha
cache-to: type=gha,mode=max
-131
View File
@@ -1,131 +0,0 @@
# Deploy CoreScope
Pre-built images are published to GHCR for `linux/amd64` and `linux/arm64` (Raspberry Pi 4/5).
## Quick Start
### Docker run
```bash
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
-e DISABLE_CADDY=true \
ghcr.io/kpa-clawbot/corescope:latest
```
Open `http://localhost` — done.
### Docker Compose
```bash
curl -sL https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/docker-compose.example.yml \
-o docker-compose.yml
docker compose up -d
```
## Image Tags
| Tag | Description |
|-----|-------------|
| `v3.4.1` | Pinned release (recommended for production) |
| `v3.4` | Latest patch in v3.4.x |
| `v3` | Latest minor+patch in v3.x |
| `latest` | Latest release tag |
| `edge` | Built from master — unstable, for testing |
## Configuration
Settings can be overridden via environment variables:
| Variable | Default | Description |
|----------|---------|-------------|
| `DISABLE_CADDY` | `false` | Skip internal Caddy (set `true` behind a reverse proxy) |
| `DISABLE_MOSQUITTO` | `false` | Skip internal MQTT broker (use external) |
| `HTTP_PORT` | `80` | Host port mapping |
| `DATA_DIR` | `./data` | Host path for persistent data |
For advanced configuration, mount a `config.json` into `/app/data/config.json`. See `config.example.json` in the repo.
## Updating
```bash
docker compose pull
docker compose up -d
```
## Data
All persistent data lives in `/app/data`:
- `meshcore.db` — SQLite database (packets, nodes)
- `config.json` — custom config (optional)
- `theme.json` — custom theme (optional)
**Backup:** `cp data/meshcore.db ~/backup/`
## TLS
Option A — **External reverse proxy** (recommended): Run with `DISABLE_CADDY=true`, put nginx/traefik/Cloudflare in front.
Option B — **Built-in Caddy**: Mount a custom Caddyfile at `/etc/caddy/Caddyfile` and expose ports 80+443.
---
## Migrating from manage.sh (existing admins)
If you're currently deploying with `manage.sh` (git clone + local build), you have two options going forward:
### Option A: Keep using manage.sh (no changes needed)
`manage.sh update` continues to work exactly as before — it fetches the latest tag, builds locally, and restarts. Nothing breaks.
```bash
./manage.sh update # latest release
./manage.sh update v3.5.0 # specific version
```
### Option B: Switch to pre-built images (recommended)
Pre-built images skip the build step entirely — faster updates, no Go toolchain needed.
**One-time migration:**
1. Stop the current deployment:
```bash
./manage.sh stop
```
2. Your data is in `~/meshcore-data/` (or whatever `PROD_DATA_DIR` is set to). It's untouched — the database, config, and theme files persist.
3. Copy `docker-compose.example.yml` to where you want to run from:
```bash
cp docker-compose.example.yml ~/docker-compose.yml
```
4. Start with the pre-built image:
```bash
cd ~ && docker compose up -d
```
5. Verify it picked up your existing data:
```bash
curl http://localhost/api/stats
```
**Updates after migration:**
```bash
docker compose pull && docker compose up -d
```
### What about manage.sh features?
| manage.sh command | Pre-built equivalent |
|---|---|
| `./manage.sh update` | `docker compose pull && docker compose up -d` |
| `./manage.sh stop` | `docker compose down` |
| `./manage.sh start` | `docker compose up -d` |
| `./manage.sh logs` | `docker compose logs -f` |
| `./manage.sh status` | `docker compose ps` |
| `./manage.sh setup` | Copy `docker-compose.example.yml`, edit env vars |
`manage.sh` remains available for advanced use cases (building from source, custom patches, development). Pre-built images are recommended for most production deployments.
+4 -16
View File
@@ -74,23 +74,9 @@ Full experience on your phone — proper touch controls, iOS safe area support,
## Quick Start
### Pre-built Image (Recommended)
### Docker (Recommended)
No build step required — just run:
```bash
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
```
Open `http://localhost` — done. No config file needed; CoreScope starts with sensible defaults.
See [DEPLOY.md](DEPLOY.md) for image tags, Docker Compose, and migration from `manage.sh`.
See [docs/deployment.md](docs/deployment.md) for the full deployment guide — MQTT setup, HTTPS options, backups, monitoring, and troubleshooting.
### Build from Source
No Go installation needed — everything builds inside the container.
```bash
git clone https://github.com/Kpa-clawbot/CoreScope.git
@@ -109,6 +95,8 @@ The setup wizard walks you through config, domain, HTTPS, build, and run.
./manage.sh help # All commands
```
See [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) for the full deployment guide — HTTPS options (auto cert, bring your own, Cloudflare Tunnel), MQTT security, backups, and troubleshooting.
### Configure
Copy `config.example.json` to `config.json` and edit:
+6 -24
View File
@@ -2,9 +2,7 @@ package main
import (
"encoding/json"
"errors"
"fmt"
"log"
"os"
"strings"
@@ -81,21 +79,15 @@ func (c *Config) NodeDaysOrDefault() int {
}
// LoadConfig reads configuration from a JSON file, with env var overrides.
// If the config file does not exist, sensible defaults are used (zero-config startup).
func LoadConfig(path string) (*Config, error) {
var cfg Config
data, err := os.ReadFile(path)
if err != nil {
if !errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("reading config %s: %w", path, err)
}
// Config file doesn't exist — use defaults (zero-config mode)
log.Printf("config file %s not found, using sensible defaults", path)
} else {
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("parsing config %s: %w", path, err)
}
return nil, fmt.Errorf("reading config %s: %w", path, err)
}
var cfg Config
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("parsing config %s: %w", path, err)
}
// Env var overrides
@@ -129,16 +121,6 @@ func LoadConfig(path string) (*Config, error) {
}}
}
// Default MQTT source: connect to localhost broker when no sources configured
if len(cfg.MQTTSources) == 0 {
cfg.MQTTSources = []MQTTSource{{
Name: "local",
Broker: "mqtt://localhost:1883",
Topics: []string{"meshcore/#"},
}}
log.Printf("no MQTT sources configured, defaulting to mqtt://localhost:1883")
}
return &cfg, nil
}
+5 -21
View File
@@ -32,25 +32,9 @@ func TestLoadConfigValidJSON(t *testing.T) {
}
func TestLoadConfigMissingFile(t *testing.T) {
t.Setenv("DB_PATH", "")
t.Setenv("MQTT_BROKER", "")
cfg, err := LoadConfig("/nonexistent/path/config.json")
if err != nil {
t.Fatalf("missing config should not error (zero-config mode), got: %v", err)
}
if cfg.DBPath != "data/meshcore.db" {
t.Errorf("dbPath=%s, want data/meshcore.db", cfg.DBPath)
}
// Should default to localhost MQTT
if len(cfg.MQTTSources) != 1 {
t.Fatalf("mqttSources len=%d, want 1", len(cfg.MQTTSources))
}
if cfg.MQTTSources[0].Broker != "mqtt://localhost:1883" {
t.Errorf("default broker=%s, want mqtt://localhost:1883", cfg.MQTTSources[0].Broker)
}
if cfg.MQTTSources[0].Name != "local" {
t.Errorf("default source name=%s, want local", cfg.MQTTSources[0].Name)
_, err := LoadConfig("/nonexistent/path/config.json")
if err == nil {
t.Error("expected error for missing file")
}
}
@@ -212,8 +196,8 @@ func TestLoadConfigLegacyMQTTEmptyBroker(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if len(cfg.MQTTSources) != 1 || cfg.MQTTSources[0].Name != "local" {
t.Errorf("mqttSources should default to local broker when legacy broker is empty, got %v", cfg.MQTTSources)
if len(cfg.MQTTSources) != 0 {
t.Errorf("mqttSources should be empty when legacy broker is empty, got %d", len(cfg.MQTTSources))
}
}
+4 -1
View File
@@ -49,6 +49,9 @@ func main() {
}
sources := cfg.ResolvedSources()
if len(sources) == 0 {
log.Fatal("no MQTT sources configured — set mqttSources in config or MQTT_BROKER env var")
}
store, err := OpenStoreWithInterval(cfg.DBPath, cfg.MetricsSampleInterval())
if err != nil {
@@ -160,7 +163,7 @@ func main() {
}
if len(clients) == 0 {
log.Fatal("no MQTT connections established — check broker is running (default: mqtt://localhost:1883). Set MQTT_BROKER env var or configure mqttSources in config.json")
log.Fatal("no MQTT connections established")
}
log.Printf("Running — %d MQTT source(s) connected", len(clients))
-111
View File
@@ -1,111 +0,0 @@
package main
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestIsWeakAPIKey(t *testing.T) {
// Known defaults must be detected
for _, weak := range []string{
"your-secret-api-key-here", "change-me", "example", "test",
"password", "admin", "apikey", "api-key", "secret", "default",
} {
if !IsWeakAPIKey(weak) {
t.Errorf("expected %q to be weak", weak)
}
}
// Case-insensitive
if !IsWeakAPIKey("Password") {
t.Error("expected case-insensitive match for Password")
}
if !IsWeakAPIKey("YOUR-SECRET-API-KEY-HERE") {
t.Error("expected case-insensitive match")
}
// Short keys (<16 chars) are weak
if !IsWeakAPIKey("short") {
t.Error("expected short key to be weak")
}
if !IsWeakAPIKey("exactly15chars!") { // 15 chars
t.Error("expected 15-char key to be weak")
}
// Empty key is NOT weak (handled separately as "disabled")
if IsWeakAPIKey("") {
t.Error("empty key should not be flagged as weak")
}
// Strong keys pass
if IsWeakAPIKey("a-very-strong-key-1234") {
t.Error("expected strong key to pass")
}
if IsWeakAPIKey("xK9!mP2@nL5#qR8$") {
t.Error("expected 17-char random key to pass")
}
}
func TestRequireAPIKey_RejectsWeakKey(t *testing.T) {
s := &Server{cfg: &Config{APIKey: "test"}}
handler := s.requireAPIKey(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest("POST", "/api/packets", nil)
req.Header.Set("X-API-Key", "test")
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusForbidden {
t.Errorf("expected 403 for weak key, got %d", rr.Code)
}
}
func TestRequireAPIKey_AcceptsStrongKey(t *testing.T) {
strongKey := "a-very-strong-key-1234"
s := &Server{cfg: &Config{APIKey: strongKey}}
handler := s.requireAPIKey(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest("POST", "/api/packets", nil)
req.Header.Set("X-API-Key", strongKey)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Errorf("expected 200 for strong key, got %d", rr.Code)
}
}
func TestRequireAPIKey_EmptyKeyDisablesEndpoints(t *testing.T) {
s := &Server{cfg: &Config{APIKey: ""}}
handler := s.requireAPIKey(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest("POST", "/api/packets", nil)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusForbidden {
t.Errorf("expected 403 for empty key, got %d", rr.Code)
}
}
func TestRequireAPIKey_WrongKeyUnauthorized(t *testing.T) {
s := &Server{cfg: &Config{APIKey: "a-very-strong-key-1234"}}
handler := s.requireAPIKey(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest("POST", "/api/packets", nil)
req.Header.Set("X-API-Key", "wrong-key-entirely-here")
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusUnauthorized {
t.Errorf("expected 401 for wrong key, got %d", rr.Code)
}
}
+1 -1
View File
@@ -21,7 +21,7 @@ func TestBackfillAsyncChunked(t *testing.T) {
}
// No pending observations → should complete immediately.
backfillResolvedPathsAsync(store, "", 100, time.Millisecond, 24)
backfillResolvedPathsAsync(store, "", 100, time.Millisecond)
if !store.backfillComplete.Load() {
t.Fatal("expected backfillComplete to be true with empty store")
}
-56
View File
@@ -57,47 +57,6 @@ type Config struct {
Timestamps *TimestampConfig `json:"timestamps,omitempty"`
DebugAffinity bool `json:"debugAffinity,omitempty"`
ResolvedPath *ResolvedPathConfig `json:"resolvedPath,omitempty"`
NeighborGraph *NeighborGraphConfig `json:"neighborGraph,omitempty"`
}
// weakAPIKeys is the blocklist of known default/example API keys that must be rejected.
var weakAPIKeys = map[string]bool{
"your-secret-api-key-here": true,
"change-me": true,
"example": true,
"test": true,
"password": true,
"admin": true,
"apikey": true,
"api-key": true,
"secret": true,
"default": true,
}
// IsWeakAPIKey returns true if the key is in the blocklist or shorter than 16 characters.
func IsWeakAPIKey(key string) bool {
if key == "" {
return false // empty is handled separately (endpoints disabled)
}
if weakAPIKeys[strings.ToLower(key)] {
return true
}
if len(key) < 16 {
return true
}
return false
}
// ResolvedPathConfig controls async backfill behavior.
type ResolvedPathConfig struct {
BackfillHours int `json:"backfillHours"` // how far back (hours) to scan for NULL resolved_path (default 24)
}
// NeighborGraphConfig controls neighbor edge pruning.
type NeighborGraphConfig struct {
MaxAgeDays int `json:"maxAgeDays"` // edges older than this are pruned (default 5)
}
// PacketStoreConfig controls in-memory packet store limits.
@@ -123,21 +82,6 @@ func (c *Config) MetricsRetentionDays() int {
return 30
}
// BackfillHours returns configured backfill window or 24h default.
func (c *Config) BackfillHours() int {
if c.ResolvedPath != nil && c.ResolvedPath.BackfillHours > 0 {
return c.ResolvedPath.BackfillHours
}
return 24
}
// NeighborMaxAgeDays returns configured max edge age or 30 days default.
func (c *Config) NeighborMaxAgeDays() int {
if c.NeighborGraph != nil && c.NeighborGraph.MaxAgeDays > 0 {
return c.NeighborGraph.MaxAgeDays
}
return 5
}
type TimestampConfig struct {
DefaultMode string `json:"defaultMode"` // "ago" | "absolute"
-177
View File
@@ -1,177 +0,0 @@
package main
import (
"database/sql"
"path/filepath"
"testing"
"time"
_ "modernc.org/sqlite"
)
func TestBackfillHoursDefault(t *testing.T) {
cfg := &Config{}
if got := cfg.BackfillHours(); got != 24 {
t.Errorf("BackfillHours() = %d, want 24", got)
}
}
func TestBackfillHoursConfigured(t *testing.T) {
cfg := &Config{ResolvedPath: &ResolvedPathConfig{BackfillHours: 48}}
if got := cfg.BackfillHours(); got != 48 {
t.Errorf("BackfillHours() = %d, want 48", got)
}
}
func TestBackfillHoursZeroFallsBack(t *testing.T) {
cfg := &Config{ResolvedPath: &ResolvedPathConfig{BackfillHours: 0}}
if got := cfg.BackfillHours(); got != 24 {
t.Errorf("BackfillHours() = %d, want 24 (default for zero)", got)
}
}
func TestNeighborMaxAgeDaysDefault(t *testing.T) {
cfg := &Config{}
if got := cfg.NeighborMaxAgeDays(); got != 5 {
t.Errorf("NeighborMaxAgeDays() = %d, want 5", got)
}
}
func TestNeighborMaxAgeDaysConfigured(t *testing.T) {
cfg := &Config{NeighborGraph: &NeighborGraphConfig{MaxAgeDays: 7}}
if got := cfg.NeighborMaxAgeDays(); got != 7 {
t.Errorf("NeighborMaxAgeDays() = %d, want 7", got)
}
}
func TestGraphPruneOlderThan(t *testing.T) {
g := NewNeighborGraph()
now := time.Now().UTC()
// Add a recent edge
g.upsertEdge("aaa", "bbb", "bb", "obs1", nil, now)
// Add an old edge
g.upsertEdge("ccc", "ddd", "dd", "obs1", nil, now.Add(-60*24*time.Hour))
if len(g.AllEdges()) != 2 {
t.Fatalf("expected 2 edges, got %d", len(g.AllEdges()))
}
cutoff := now.Add(-30 * 24 * time.Hour)
pruned := g.PruneOlderThan(cutoff)
if pruned != 1 {
t.Errorf("PruneOlderThan pruned %d, want 1", pruned)
}
edges := g.AllEdges()
if len(edges) != 1 {
t.Fatalf("expected 1 edge after prune, got %d", len(edges))
}
if edges[0].NodeA != "aaa" && edges[0].NodeB != "aaa" {
t.Errorf("wrong edge survived prune: %+v", edges[0])
}
}
func TestPruneNeighborEdgesDB(t *testing.T) {
tmpDir := t.TempDir()
dbPath := filepath.Join(tmpDir, "test.db")
db, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
defer db.Close()
_, err = db.Exec(`CREATE TABLE neighbor_edges (
node_a TEXT NOT NULL,
node_b TEXT NOT NULL,
count INTEGER DEFAULT 1,
last_seen TEXT,
PRIMARY KEY (node_a, node_b)
)`)
if err != nil {
t.Fatal(err)
}
now := time.Now().UTC()
old := now.Add(-60 * 24 * time.Hour)
db.Exec("INSERT INTO neighbor_edges (node_a, node_b, count, last_seen) VALUES (?, ?, 5, ?)",
"aaa", "bbb", now.Format(time.RFC3339))
db.Exec("INSERT INTO neighbor_edges (node_a, node_b, count, last_seen) VALUES (?, ?, 3, ?)",
"ccc", "ddd", old.Format(time.RFC3339))
g := NewNeighborGraph()
g.upsertEdge("aaa", "bbb", "bb", "obs1", nil, now)
g.upsertEdge("ccc", "ddd", "dd", "obs1", nil, old)
pruned, err := PruneNeighborEdges(dbPath, g, 30)
if err != nil {
t.Fatal(err)
}
if pruned != 1 {
t.Errorf("PruneNeighborEdges pruned %d DB rows, want 1", pruned)
}
var count int
db.QueryRow("SELECT COUNT(*) FROM neighbor_edges").Scan(&count)
if count != 1 {
t.Errorf("expected 1 row in DB after prune, got %d", count)
}
if len(g.AllEdges()) != 1 {
t.Errorf("expected 1 in-memory edge after prune, got %d", len(g.AllEdges()))
}
}
func TestBackfillRespectsHourWindow(t *testing.T) {
store := &PacketStore{}
now := time.Now().UTC()
oldTime := now.Add(-48 * time.Hour).Format(time.RFC3339Nano)
newTime := now.Add(-30 * time.Minute).Format(time.RFC3339Nano)
store.packets = []*StoreTx{
{
ID: 1,
Hash: "old-hash",
FirstSeen: oldTime,
Observations: []*StoreObs{
{ID: 1, PathJSON: `["abc"]`},
},
},
{
ID: 2,
Hash: "new-hash",
FirstSeen: newTime,
Observations: []*StoreObs{
{ID: 2, PathJSON: `["def"]`},
},
},
}
// With a 1-hour window, only the new tx should be processed.
// backfillResolvedPathsAsync will find no prefix map and finish quickly,
// but we can verify the pending count reflects the window.
go backfillResolvedPathsAsync(store, "", 100, time.Millisecond, 1)
// Wait for completion
for i := 0; i < 100; i++ {
if store.backfillComplete.Load() {
break
}
time.Sleep(10 * time.Millisecond)
}
if !store.backfillComplete.Load() {
t.Fatal("backfill did not complete")
}
// With no prefix map, total should be 0 (early exit) or just the new one
// The function exits early when pm == nil, so backfillTotal stays at 0
// if there were pending items but no pm. Let's verify it didn't process
// the old one by checking total <= 1.
total := store.backfillTotal.Load()
if total > 1 {
t.Errorf("backfill total = %d, want <= 1 (old tx should be excluded by hour window)", total)
}
}
-26
View File
@@ -276,29 +276,3 @@ func TestNewPacketStoreNilConfig(t *testing.T) {
t.Fatalf("expected retentionHours=0, got %f", store.retentionHours)
}
}
func TestCacheTTLFromConfig(t *testing.T) {
// With config values: analyticsHashSizes and analyticsRF should override defaults.
cacheTTL := map[string]interface{}{
"analyticsHashSizes": float64(7200),
"analyticsRF": float64(300),
}
store := NewPacketStore(nil, nil, cacheTTL)
if store.collisionCacheTTL != 7200*time.Second {
t.Fatalf("expected collisionCacheTTL=7200s, got %v", store.collisionCacheTTL)
}
if store.rfCacheTTL != 300*time.Second {
t.Fatalf("expected rfCacheTTL=300s, got %v", store.rfCacheTTL)
}
}
func TestCacheTTLDefaults(t *testing.T) {
// Without config, defaults should apply.
store := NewPacketStore(nil, nil)
if store.collisionCacheTTL != 3600*time.Second {
t.Fatalf("expected default collisionCacheTTL=3600s, got %v", store.collisionCacheTTL)
}
if store.rfCacheTTL != 15*time.Second {
t.Fatalf("expected default rfCacheTTL=15s, got %v", store.rfCacheTTL)
}
}
+19 -105
View File
@@ -104,8 +104,6 @@ func main() {
}
if cfg.APIKey == "" {
log.Printf("[security] WARNING: no apiKey configured — write endpoints are BLOCKED (set apiKey in config.json to enable them)")
} else if IsWeakAPIKey(cfg.APIKey) {
log.Printf("[security] WARNING: API key is weak or a known default — write endpoints are vulnerable")
}
// Resolve DB path
@@ -141,7 +139,7 @@ func main() {
}
// In-memory packet store
store := NewPacketStore(database, cfg.PacketStore, cfg.CacheTTL)
store := NewPacketStore(database, cfg.PacketStore)
if err := store.Load(); err != nil {
log.Fatalf("[store] failed to load: %v", err)
}
@@ -155,7 +153,7 @@ func main() {
// NOTE on startup ordering (review item #10): ensureResolvedPathColumn runs AFTER
// OpenDB/detectSchema, so db.hasResolvedPath will be false on first run with a
// pre-existing DB. This means Load() won't SELECT resolved_path from SQLite.
// Async backfill runs after HTTP starts (see backfillResolvedPathsAsync below)
// That's OK: backfillResolvedPaths (below) computes and persists them in-memory
// AND to SQLite. On next restart, detectSchema finds the column and Load() reads it.
if err := ensureResolvedPathColumn(dbPath); err != nil {
log.Printf("[store] warning: could not add resolved_path column: %v", err)
@@ -168,59 +166,24 @@ func main() {
store.graph = loadNeighborEdgesFromDB(database.conn)
log.Printf("[neighbor] loaded persisted neighbor graph")
} else {
log.Printf("[neighbor] no persisted edges found, will build in background...")
store.graph = NewNeighborGraph() // empty graph — gets populated by background goroutine
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[neighbor] graph build panic recovered: %v", r)
}
}()
rw, rwErr := openRW(dbPath)
if rwErr == nil {
edgeCount := buildAndPersistEdges(store, rw)
rw.Close()
log.Printf("[neighbor] persisted %d edges", edgeCount)
}
built := BuildFromStore(store)
store.mu.Lock()
store.graph = built
store.mu.Unlock()
log.Printf("[neighbor] graph build complete")
}()
log.Printf("[neighbor] no persisted edges found, building from store...")
rw, rwErr := openRW(dbPath)
if rwErr == nil {
edgeCount := buildAndPersistEdges(store, rw)
rw.Close()
log.Printf("[neighbor] persisted %d edges", edgeCount)
}
store.graph = BuildFromStore(store)
}
// Initial pickBestObservation runs in background — doesn't need to block HTTP.
// API serves best-effort data until this completes (~10s for 100K txs).
// Processes in chunks of 5000, releasing the lock between chunks so API
// handlers remain responsive.
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[store] pickBestObservation panic recovered: %v", r)
}
}()
const chunkSize = 5000
store.mu.RLock()
totalPackets := len(store.packets)
store.mu.RUnlock()
for i := 0; i < totalPackets; i += chunkSize {
end := i + chunkSize
if end > totalPackets {
end = totalPackets
}
store.mu.Lock()
for j := i; j < end && j < len(store.packets); j++ {
pickBestObservation(store.packets[j])
}
store.mu.Unlock()
if end < totalPackets {
time.Sleep(10 * time.Millisecond) // yield to API handlers
}
}
log.Printf("[store] initial pickBestObservation complete (%d transmissions)", totalPackets)
}()
// Backfill resolved_path runs asynchronously after HTTP starts (see below).
// Initial pickBestObservation runs with whatever resolved_path data was
// loaded from SQLite; the async backfill will re-pick affected transmissions.
store.mu.Lock()
for _, tx := range store.packets {
pickBestObservation(tx)
}
store.mu.Unlock()
// WebSocket hub
hub := NewHub()
@@ -268,11 +231,6 @@ func main() {
close(pruneDone)
}
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[prune] panic recovered: %v", r)
}
}()
time.Sleep(1 * time.Minute)
if n, err := database.PruneOldPackets(days); err != nil {
log.Printf("[prune] error: %v", err)
@@ -306,11 +264,6 @@ func main() {
close(metricsPruneDone)
}
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[metrics-prune] panic recovered: %v", r)
}
}()
time.Sleep(2 * time.Minute) // stagger after packet prune
database.PruneOldMetrics(metricsDays)
for {
@@ -325,42 +278,6 @@ func main() {
log.Printf("[metrics-prune] auto-prune enabled: metrics older than %d days", metricsDays)
}
// Auto-prune old neighbor edges
var stopEdgePrune func()
{
maxAgeDays := cfg.NeighborMaxAgeDays()
edgePruneTicker := time.NewTicker(24 * time.Hour)
edgePruneDone := make(chan struct{})
stopEdgePrune = func() {
edgePruneTicker.Stop()
close(edgePruneDone)
}
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[neighbor-prune] panic recovered: %v", r)
}
}()
time.Sleep(4 * time.Minute) // stagger after metrics prune
store.mu.RLock()
g := store.graph
store.mu.RUnlock()
PruneNeighborEdges(dbPath, g, maxAgeDays)
for {
select {
case <-edgePruneTicker.C:
store.mu.RLock()
g := store.graph
store.mu.RUnlock()
PruneNeighborEdges(dbPath, g, maxAgeDays)
case <-edgePruneDone:
return
}
}
}()
log.Printf("[neighbor-prune] auto-prune enabled: edges older than %d days", maxAgeDays)
}
// Graceful shutdown
httpServer := &http.Server{
Addr: fmt.Sprintf(":%d", cfg.Port),
@@ -386,9 +303,6 @@ func main() {
if stopMetricsPrune != nil {
stopMetricsPrune()
}
if stopEdgePrune != nil {
stopEdgePrune()
}
// 2. Gracefully drain HTTP connections (up to 15s)
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
@@ -410,7 +324,7 @@ func main() {
log.Printf("[server] CoreScope (Go) listening on http://localhost:%d", cfg.Port)
// Start async backfill in background — HTTP is now available.
go backfillResolvedPathsAsync(store, dbPath, 5000, 100*time.Millisecond, cfg.BackfillHours())
go backfillResolvedPathsAsync(store, dbPath, 5000, 100*time.Millisecond)
if err := httpServer.ListenAndServe(); err != http.ErrServerClosed {
log.Fatalf("[server] %v", err)
+14 -25
View File
@@ -20,20 +20,19 @@ type NeighborResponse struct {
}
type NeighborEntry struct {
Pubkey *string `json:"pubkey"`
Prefix string `json:"prefix"`
Name *string `json:"name"`
Role *string `json:"role"`
Count int `json:"count"`
Score float64 `json:"score"`
FirstSeen string `json:"first_seen"`
LastSeen string `json:"last_seen"`
AvgSNR *float64 `json:"avg_snr"`
DistanceKm *float64 `json:"distance_km,omitempty"`
Observers []string `json:"observers"`
Ambiguous bool `json:"ambiguous"`
Unresolved bool `json:"unresolved,omitempty"`
Candidates []CandidateEntry `json:"candidates,omitempty"`
Pubkey *string `json:"pubkey"`
Prefix string `json:"prefix"`
Name *string `json:"name"`
Role *string `json:"role"`
Count int `json:"count"`
Score float64 `json:"score"`
FirstSeen string `json:"first_seen"`
LastSeen string `json:"last_seen"`
AvgSNR *float64 `json:"avg_snr"`
Observers []string `json:"observers"`
Ambiguous bool `json:"ambiguous"`
Unresolved bool `json:"unresolved,omitempty"`
Candidates []CandidateEntry `json:"candidates,omitempty"`
}
type CandidateEntry struct {
@@ -116,15 +115,9 @@ func (s *Server) handleNodeNeighbors(w http.ResponseWriter, r *http.Request) {
edges := graph.Neighbors(pubkey)
now := time.Now()
// Build node info lookup for names/roles/coordinates.
// Build node info lookup for names/roles.
nodeMap := s.buildNodeInfoMap()
// Look up the queried node's GPS coordinates for distance computation.
var srcInfo nodeInfo
if nodeMap != nil {
srcInfo = nodeMap[pubkey]
}
var entries []NeighborEntry
totalObs := 0
@@ -177,10 +170,6 @@ func (s *Server) handleNodeNeighbors(w http.ResponseWriter, r *http.Request) {
if info, ok := nodeMap[strings.ToLower(neighborPK)]; ok {
entry.Name = &info.Name
entry.Role = &info.Role
if srcInfo.HasGPS && info.HasGPS {
d := haversineKm(srcInfo.Lat, srcInfo.Lon, info.Lat, info.Lon)
entry.DistanceKm = &d
}
}
}
-63
View File
@@ -347,69 +347,6 @@ func TestNeighborGraphAPI_AmbiguousEdgesCount(t *testing.T) {
}
}
func TestNeighborAPI_DistanceKm_WithGPS(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen)
VALUES ('aaaa', 'NodeA', 'repeater', 51.5074, -0.1278, '2026-01-01T00:00:00Z', '2025-01-01T00:00:00Z')`)
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen)
VALUES ('bbbb', 'NodeB', 'repeater', 51.5200, -0.1200, '2026-01-01T00:00:00Z', '2025-01-01T00:00:00Z')`)
cfg := &Config{Port: 3000}
hub := NewHub()
srv := NewServer(db, cfg, hub)
srv.store = NewPacketStore(db, nil)
now := time.Now()
srv.neighborGraph = makeTestGraph(newEdge("aaaa", "bbbb", "bb", 50, now))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 neighbor, got %d", len(resp.Neighbors))
}
n := resp.Neighbors[0]
if n.DistanceKm == nil {
t.Fatal("expected distance_km to be set for GPS-enabled nodes")
}
if *n.DistanceKm <= 0 {
t.Errorf("expected positive distance, got %f", *n.DistanceKm)
}
}
func TestNeighborAPI_DistanceKm_NoGPS(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
// Nodes with 0,0 coords → HasGPS=false
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen)
VALUES ('aaaa', 'NodeA', 'repeater', 0, 0, '2026-01-01T00:00:00Z', '2025-01-01T00:00:00Z')`)
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen)
VALUES ('bbbb', 'NodeB', 'repeater', 0, 0, '2026-01-01T00:00:00Z', '2025-01-01T00:00:00Z')`)
cfg := &Config{Port: 3000}
hub := NewHub()
srv := NewServer(db, cfg, hub)
srv.store = NewPacketStore(db, nil)
now := time.Now()
srv.neighborGraph = makeTestGraph(newEdge("aaaa", "bbbb", "bb", 50, now))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 neighbor, got %d", len(resp.Neighbors))
}
if resp.Neighbors[0].DistanceKm != nil {
t.Errorf("expected nil distance_km for nodes without GPS, got %f", *resp.Neighbors[0].DistanceKm)
}
}
func TestNeighborGraphAPI_RegionFilter(t *testing.T) {
now := time.Now()
// Edge with observer "obs-sjc" — would match region SJC if we had region resolution.
-21
View File
@@ -542,24 +542,3 @@ func minLen(s string, n int) int {
}
return n
}
// PruneOlderThan removes all edges with LastSeen before cutoff.
// Returns the number of edges removed.
func (g *NeighborGraph) PruneOlderThan(cutoff time.Time) int {
g.mu.Lock()
defer g.mu.Unlock()
pruned := 0
for key, edge := range g.edges {
if edge.LastSeen.Before(cutoff) {
// Remove from byNode index
g.removeFromByNode(edge.NodeA, edge)
if edge.NodeB != "" {
g.removeFromByNode(edge.NodeB, edge)
}
delete(g.edges, key)
pruned++
}
}
return pruned
}
+158 -123
View File
@@ -343,46 +343,137 @@ func unmarshalResolvedPath(s string) []*string {
return result
}
// backfillResolvedPaths resolves paths for all observations that have NULL resolved_path.
func backfillResolvedPaths(store *PacketStore, dbPath string) int {
// Collect pending observations and snapshot immutable fields under read lock.
// graph is set in main.go before backfill is called; nil-safe throughout (review item #6).
type obsRef struct {
obsID int
pathJSON string
observerID string
txJSON string // snapshot of DecodedJSON for extractFromNode
payloadType *int
}
store.mu.RLock()
pm := store.nodePM
graph := store.graph
var pending []obsRef
for _, tx := range store.packets {
for _, obs := range tx.Observations {
if obs.ResolvedPath == nil && obs.PathJSON != "" && obs.PathJSON != "[]" {
pending = append(pending, obsRef{
obsID: obs.ID,
pathJSON: obs.PathJSON,
observerID: obs.ObserverID,
txJSON: tx.DecodedJSON,
payloadType: tx.PayloadType,
})
}
}
}
store.mu.RUnlock()
if len(pending) == 0 || pm == nil {
return 0
}
// Resolve paths outside the lock — resolvePathForObs only reads pm and graph.
type resolved struct {
obsID int
rp []*string
rpJSON string
}
var results []resolved
for _, ref := range pending {
// Build a minimal StoreTx for extractFromNode (only needs DecodedJSON + PayloadType).
fakeTx := &StoreTx{DecodedJSON: ref.txJSON, PayloadType: ref.payloadType}
rp := resolvePathForObs(ref.pathJSON, ref.observerID, fakeTx, pm, graph)
if len(rp) > 0 {
rpJSON := marshalResolvedPath(rp)
if rpJSON != "" {
results = append(results, resolved{ref.obsID, rp, rpJSON})
}
}
}
if len(results) == 0 {
return 0
}
// Persist to SQLite (no lock needed — separate RW connection).
rw, err := openRW(dbPath)
if err != nil {
log.Printf("[store] backfill: open rw error: %v", err)
return 0
}
defer rw.Close()
sqlTx, err := rw.Begin()
if err != nil {
log.Printf("[store] backfill: begin tx error: %v", err)
return 0
}
defer sqlTx.Rollback()
stmt, err := sqlTx.Prepare("UPDATE observations SET resolved_path = ? WHERE id = ?")
if err != nil {
log.Printf("[store] backfill: prepare error: %v", err)
return 0
}
defer stmt.Close()
var firstErr error
for _, r := range results {
if _, err := stmt.Exec(r.rpJSON, r.obsID); err != nil && firstErr == nil {
firstErr = err
}
}
if firstErr != nil {
log.Printf("[store] backfill resolved_path exec error (first): %v", firstErr)
}
if err := sqlTx.Commit(); err != nil {
log.Printf("[store] backfill: commit error: %v", err)
return 0
}
// Update in-memory state under write lock.
store.mu.Lock()
count := 0
for _, r := range results {
if obs, ok := store.byObsID[r.obsID]; ok {
obs.ResolvedPath = r.rp
count++
}
}
store.mu.Unlock()
return count
}
// backfillResolvedPathsAsync processes observations with NULL resolved_path in
// chunks, yielding between batches so HTTP handlers remain responsive. It sets
// store.backfillComplete when finished and re-picks best observations for any
// transmissions affected by newly resolved paths.
func backfillResolvedPathsAsync(store *PacketStore, dbPath string, chunkSize int, yieldDuration time.Duration, backfillHours int) {
defer func() {
if r := recover(); r != nil {
log.Printf("[store] backfillResolvedPathsAsync panic recovered: %v", r)
}
}()
// Collect ALL pending obs refs upfront in one pass under a single RLock (fix A).
func backfillResolvedPathsAsync(store *PacketStore, dbPath string, chunkSize int, yieldDuration time.Duration) {
// Snapshot of pending observation references — collected once under read lock.
type obsRef struct {
obsID int
pathJSON string
observerID string
txJSON string
payloadType *int
txHash string // to re-pick best obs
txHash string
}
cutoff := time.Now().UTC().Add(-time.Duration(backfillHours) * time.Hour)
store.mu.RLock()
pm := store.nodePM
var allPending []obsRef
graph := store.graph
var pending []obsRef
for _, tx := range store.packets {
// Skip transmissions older than the backfill window.
if tx.FirstSeen != "" {
if ts, err := time.Parse(time.RFC3339Nano, tx.FirstSeen); err == nil && ts.Before(cutoff) {
continue
}
// Also try the common SQLite format
if ts, err := time.Parse("2006-01-02 15:04:05", tx.FirstSeen); err == nil && ts.Before(cutoff) {
continue
}
}
for _, obs := range tx.Observations {
if obs.ResolvedPath == nil && obs.PathJSON != "" && obs.PathJSON != "[]" {
allPending = append(allPending, obsRef{
pending = append(pending, obsRef{
obsID: obs.ID,
pathJSON: obs.PathJSON,
observerID: obs.ObserverID,
@@ -395,54 +486,31 @@ func backfillResolvedPathsAsync(store *PacketStore, dbPath string, chunkSize int
}
store.mu.RUnlock()
totalPending := len(allPending)
if totalPending == 0 || pm == nil {
if len(pending) == 0 || pm == nil {
store.backfillComplete.Store(true)
log.Printf("[store] async resolved_path backfill: nothing to do")
return
}
store.backfillTotal.Store(int64(totalPending))
store.backfillTotal.Store(int64(len(pending)))
store.backfillProcessed.Store(0)
log.Printf("[store] async resolved_path backfill starting: %d observations", totalPending)
log.Printf("[store] async resolved_path backfill starting: %d observations", len(pending))
// Open RW connection once before the chunk loop (fix B).
var rw *sql.DB
if dbPath != "" {
var err error
rw, err = openRW(dbPath)
if err != nil {
log.Printf("[store] async backfill: open rw error: %v", err)
}
type resolved struct {
obsID int
rp []*string
rpJSON string
txHash string
}
defer func() {
if rw != nil {
rw.Close()
}
}()
totalProcessed := 0
for totalProcessed < totalPending {
end := totalProcessed + chunkSize
if end > totalPending {
end = totalPending
for i := 0; i < len(pending); i += chunkSize {
end := i + chunkSize
if end > len(pending) {
end = len(pending)
}
chunk := allPending[totalProcessed:end]
chunk := pending[i:end]
// Re-read graph under RLock at the start of each chunk so we pick up
// a freshly-built graph once the background build goroutine completes,
// instead of using the potentially-empty graph captured at cold start.
store.mu.RLock()
graph := store.graph
store.mu.RUnlock()
// Resolve paths outside any lock.
type resolved struct {
obsID int
rp []*string
rpJSON string
txHash string
}
// Resolve paths outside the lock.
var results []resolved
for _, ref := range chunk {
fakeTx := &StoreTx{DecodedJSON: ref.txJSON, PayloadType: ref.payloadType}
@@ -455,63 +523,61 @@ func backfillResolvedPathsAsync(store *PacketStore, dbPath string, chunkSize int
}
}
// Persist to SQLite using the shared connection.
if len(results) > 0 && rw != nil {
sqlTx, err := rw.Begin()
// Persist to SQLite in a single transaction per chunk.
if len(results) > 0 {
rw, err := openRW(dbPath)
if err != nil {
log.Printf("[store] async backfill: begin tx error: %v", err)
log.Printf("[store] async backfill: open rw error: %v", err)
} else {
stmt, err := sqlTx.Prepare("UPDATE observations SET resolved_path = ? WHERE id = ?")
sqlTx, err := rw.Begin()
if err != nil {
log.Printf("[store] async backfill: prepare error: %v", err)
sqlTx.Rollback()
log.Printf("[store] async backfill: begin tx error: %v", err)
} else {
var execErr error
for _, r := range results {
if _, e := stmt.Exec(r.rpJSON, r.obsID); e != nil && execErr == nil {
execErr = e
stmt, err := sqlTx.Prepare("UPDATE observations SET resolved_path = ? WHERE id = ?")
if err != nil {
log.Printf("[store] async backfill: prepare error: %v", err)
sqlTx.Rollback()
} else {
for _, r := range results {
if _, err := stmt.Exec(r.rpJSON, r.obsID); err != nil {
log.Printf("[store] async backfill: exec error obs %d: %v", r.obsID, err)
}
}
stmt.Close()
if err := sqlTx.Commit(); err != nil {
log.Printf("[store] async backfill: commit error: %v", err)
}
}
if execErr != nil {
log.Printf("[store] async backfill: exec error (first): %v", execErr)
}
stmt.Close()
if err := sqlTx.Commit(); err != nil {
log.Printf("[store] async backfill: commit error: %v", err)
}
}
rw.Close()
}
// Update in-memory state and re-pick best observation under a single
// write lock. The per-tx pickBestObservation is O(observations) which is
// typically <10 per tx — negligible cost vs. the race risk of splitting
// the lock (pollAndMerge can append to tx.Observations concurrently).
// Update in-memory state + re-pick best obs for affected txs.
affectedTxs := make(map[string]bool)
store.mu.Lock()
affectedSet := make(map[string]bool)
for _, r := range results {
if obs, ok := store.byObsID[r.obsID]; ok {
obs.ResolvedPath = r.rp
}
if !affectedSet[r.txHash] {
affectedSet[r.txHash] = true
if tx, ok := store.byHash[r.txHash]; ok {
pickBestObservation(tx)
}
affectedTxs[r.txHash] = true
}
for hash := range affectedTxs {
if tx, ok := store.byHash[hash]; ok {
pickBestObservation(tx)
}
}
store.mu.Unlock()
}
totalProcessed += len(chunk)
store.backfillProcessed.Store(int64(totalProcessed))
pct := float64(totalProcessed) / float64(totalPending) * 100
log.Printf("[store] backfill progress: %d/%d observations (%.1f%%)", totalProcessed, totalPending, pct)
store.backfillProcessed.Store(int64(end))
pct := float64(end) / float64(len(pending)) * 100
log.Printf("[store] backfill progress: %d/%d observations (%.1f%%)", end, len(pending), pct)
time.Sleep(yieldDuration)
}
store.backfillComplete.Store(true)
log.Printf("[store] async resolved_path backfill complete: %d observations processed", totalProcessed)
log.Printf("[store] async resolved_path backfill complete: %d observations processed", len(pending))
}
// ─── Shared helpers ────────────────────────────────────────────────────────────
@@ -592,34 +658,3 @@ func openRW(dbPath string) (*sql.DB, error) {
rw.SetMaxOpenConns(1)
return rw, nil
}
// PruneNeighborEdges removes edges older than maxAgeDays from both SQLite and
// the in-memory graph. Uses openRW internally because the shared database.conn
// is opened with mode=ro and DELETEs would silently fail.
func PruneNeighborEdges(dbPath string, graph *NeighborGraph, maxAgeDays int) (int, error) {
cutoff := time.Now().UTC().Add(-time.Duration(maxAgeDays) * 24 * time.Hour)
// 1. Prune from SQLite using a read-write connection
var dbPruned int64
rw, err := openRW(dbPath)
if err != nil {
return 0, fmt.Errorf("prune neighbor_edges: open rw: %w", err)
}
defer rw.Close()
res, err := rw.Exec("DELETE FROM neighbor_edges WHERE last_seen < ?", cutoff.Format(time.RFC3339))
if err != nil {
return 0, fmt.Errorf("prune neighbor_edges: %w", err)
}
dbPruned, _ = res.RowsAffected()
// 2. Prune from in-memory graph
memPruned := 0
if graph != nil {
memPruned = graph.PruneOlderThan(cutoff)
}
if dbPruned > 0 || memPruned > 0 {
log.Printf("[neighbor-prune] removed %d DB rows, %d in-memory edges older than %d days", dbPruned, memPruned, maxAgeDays)
}
return int(dbPruned), nil
}
-359
View File
@@ -1,359 +0,0 @@
package main
import (
"encoding/json"
"fmt"
"net/http"
"sort"
"strings"
"github.com/gorilla/mux"
)
// routeMeta holds metadata for a single API route.
type routeMeta struct {
Summary string `json:"summary"`
Description string `json:"description,omitempty"`
Tag string `json:"tag"`
Auth bool `json:"auth,omitempty"`
QueryParams []paramMeta `json:"queryParams,omitempty"`
}
type paramMeta struct {
Name string `json:"name"`
Description string `json:"description"`
Required bool `json:"required,omitempty"`
Type string `json:"type"` // "string", "integer", "boolean"
}
// routeDescriptions returns metadata for all known API routes.
// Key format: "METHOD /path/pattern"
func routeDescriptions() map[string]routeMeta {
return map[string]routeMeta{
// Config
"GET /api/config/cache": {Summary: "Get cache configuration", Tag: "config"},
"GET /api/config/client": {Summary: "Get client configuration", Tag: "config"},
"GET /api/config/regions": {Summary: "Get configured regions", Tag: "config"},
"GET /api/config/theme": {Summary: "Get theme configuration", Description: "Returns color maps, CSS variables, and theme defaults.", Tag: "config"},
"GET /api/config/map": {Summary: "Get map configuration", Tag: "config"},
"GET /api/config/geo-filter": {Summary: "Get geo-filter configuration", Tag: "config"},
// Admin / system
"GET /api/health": {Summary: "Health check", Description: "Returns server health, uptime, and memory stats.", Tag: "admin"},
"GET /api/stats": {Summary: "Network statistics", Description: "Returns aggregate stats (node counts, packet counts, observer counts). Cached for 10s.", Tag: "admin"},
"GET /api/perf": {Summary: "Performance statistics", Description: "Returns per-endpoint request timing and slow query log.", Tag: "admin"},
"POST /api/perf/reset": {Summary: "Reset performance stats", Tag: "admin", Auth: true},
"POST /api/admin/prune": {Summary: "Prune old data", Description: "Deletes packets and nodes older than the configured retention period.", Tag: "admin", Auth: true},
"GET /api/debug/affinity": {Summary: "Debug neighbor affinity scores", Tag: "admin", Auth: true},
// Packets
"GET /api/packets": {Summary: "List packets", Description: "Returns decoded packets with filtering, sorting, and pagination.", Tag: "packets",
QueryParams: []paramMeta{
{Name: "limit", Description: "Max packets to return", Type: "integer"},
{Name: "offset", Description: "Pagination offset", Type: "integer"},
{Name: "sort", Description: "Sort field", Type: "string"},
{Name: "order", Description: "Sort order (asc/desc)", Type: "string"},
{Name: "type", Description: "Filter by packet type", Type: "string"},
{Name: "observer", Description: "Filter by observer ID", Type: "string"},
{Name: "timeRange", Description: "Time range filter (e.g. 1h, 24h, 7d)", Type: "string"},
{Name: "search", Description: "Full-text search", Type: "string"},
{Name: "groupByHash", Description: "Group duplicate packets by hash", Type: "boolean"},
}},
"POST /api/packets": {Summary: "Ingest a packet", Description: "Submit a raw packet for decoding and storage.", Tag: "packets", Auth: true},
"GET /api/packets/{id}": {Summary: "Get packet detail", Tag: "packets"},
"GET /api/packets/timestamps": {Summary: "Get packet timestamp ranges", Tag: "packets"},
"POST /api/packets/observations": {Summary: "Batch submit observations", Description: "Submit multiple observer sightings for existing packets.", Tag: "packets"},
// Decode
"POST /api/decode": {Summary: "Decode a raw packet", Description: "Decodes a hex-encoded packet without storing it.", Tag: "packets"},
// Nodes
"GET /api/nodes": {Summary: "List nodes", Description: "Returns all known mesh nodes with status and metadata.", Tag: "nodes",
QueryParams: []paramMeta{
{Name: "role", Description: "Filter by node role", Type: "string"},
{Name: "status", Description: "Filter by status (active/stale/offline)", Type: "string"},
}},
"GET /api/nodes/search": {Summary: "Search nodes", Description: "Search nodes by name or public key prefix.", Tag: "nodes", QueryParams: []paramMeta{{Name: "q", Description: "Search query", Type: "string", Required: true}}},
"GET /api/nodes/bulk-health": {Summary: "Bulk node health", Description: "Returns health status for all nodes in one call.", Tag: "nodes"},
"GET /api/nodes/network-status": {Summary: "Network status summary", Description: "Returns counts of active, stale, and offline nodes.", Tag: "nodes"},
"GET /api/nodes/{pubkey}": {Summary: "Get node detail", Description: "Returns full detail for a single node by public key.", Tag: "nodes"},
"GET /api/nodes/{pubkey}/health": {Summary: "Get node health", Tag: "nodes"},
"GET /api/nodes/{pubkey}/paths": {Summary: "Get node routing paths", Tag: "nodes"},
"GET /api/nodes/{pubkey}/analytics": {Summary: "Get node analytics", Description: "Per-node packet counts, timing, and RF stats.", Tag: "nodes"},
"GET /api/nodes/{pubkey}/neighbors": {Summary: "Get node neighbors", Description: "Returns neighbor nodes with affinity scores.", Tag: "nodes"},
// Analytics
"GET /api/analytics/rf": {Summary: "RF analytics", Description: "SNR/RSSI distributions and statistics.", Tag: "analytics"},
"GET /api/analytics/topology": {Summary: "Network topology", Description: "Hop-count distribution and route analysis.", Tag: "analytics"},
"GET /api/analytics/channels": {Summary: "Channel analytics", Description: "Message counts and activity per channel.", Tag: "analytics"},
"GET /api/analytics/distance": {Summary: "Distance analytics", Description: "Geographic distance calculations between nodes.", Tag: "analytics"},
"GET /api/analytics/hash-sizes": {Summary: "Hash size analysis", Description: "Distribution of hash prefix sizes across the network.", Tag: "analytics"},
"GET /api/analytics/hash-collisions": {Summary: "Hash collision detection", Description: "Identifies nodes sharing hash prefixes.", Tag: "analytics"},
"GET /api/analytics/subpaths": {Summary: "Subpath analysis", Description: "Common routing subpaths through the mesh.", Tag: "analytics"},
"GET /api/analytics/subpaths-bulk": {Summary: "Bulk subpath analysis", Tag: "analytics"},
"GET /api/analytics/subpath-detail": {Summary: "Subpath detail", Tag: "analytics"},
"GET /api/analytics/neighbor-graph": {Summary: "Neighbor graph", Description: "Full neighbor affinity graph for visualization.", Tag: "analytics"},
// Channels
"GET /api/channels": {Summary: "List channels", Description: "Returns known mesh channels with message counts.", Tag: "channels"},
"GET /api/channels/{hash}/messages": {Summary: "Get channel messages", Description: "Returns messages for a specific channel.", Tag: "channels"},
// Observers
"GET /api/observers": {Summary: "List observers", Description: "Returns all known packet observers/gateways.", Tag: "observers"},
"GET /api/observers/{id}": {Summary: "Get observer detail", Tag: "observers"},
"GET /api/observers/{id}/metrics": {Summary: "Get observer metrics", Description: "Packet rates, uptime, and performance metrics.", Tag: "observers"},
"GET /api/observers/{id}/analytics": {Summary: "Get observer analytics", Tag: "observers"},
"GET /api/observers/metrics/summary": {Summary: "Observer metrics summary", Description: "Aggregate metrics across all observers.", Tag: "observers"},
// Misc
"GET /api/resolve-hops": {Summary: "Resolve hop path", Description: "Resolves hash prefixes in a hop path to node names. Returns affinity scores and best candidates.", Tag: "nodes", QueryParams: []paramMeta{{Name: "hops", Description: "Comma-separated hop hash prefixes", Type: "string", Required: true}}},
"GET /api/traces/{hash}": {Summary: "Get packet traces", Description: "Returns all observer sightings for a packet hash.", Tag: "packets"},
"GET /api/iata-coords": {Summary: "Get IATA airport coordinates", Description: "Returns lat/lon for known airport codes (used for observer positioning).", Tag: "config"},
"GET /api/audio-lab/buckets": {Summary: "Audio lab frequency buckets", Description: "Returns frequency bucket data for audio analysis.", Tag: "analytics"},
}
}
// buildOpenAPISpec constructs an OpenAPI 3.0 spec by walking the mux router.
func buildOpenAPISpec(router *mux.Router, version string) map[string]interface{} {
descriptions := routeDescriptions()
// Collect routes from the router
type routeInfo struct {
path string
method string
authReq bool
}
var routes []routeInfo
router.Walk(func(route *mux.Route, router *mux.Router, ancestors []*mux.Route) error {
path, err := route.GetPathTemplate()
if err != nil {
return nil
}
if !strings.HasPrefix(path, "/api/") {
return nil
}
// Skip the spec/docs endpoints themselves
if path == "/api/spec" || path == "/api/docs" {
return nil
}
methods, err := route.GetMethods()
if err != nil {
return nil
}
for _, m := range methods {
routes = append(routes, routeInfo{path: path, method: m})
}
return nil
})
// Sort routes for deterministic output
sort.Slice(routes, func(i, j int) bool {
if routes[i].path != routes[j].path {
return routes[i].path < routes[j].path
}
return routes[i].method < routes[j].method
})
// Build paths object
paths := make(map[string]interface{})
tagSet := make(map[string]bool)
for _, ri := range routes {
key := ri.method + " " + ri.path
meta, hasMeta := descriptions[key]
// Convert mux path params {name} to OpenAPI {name} (same format, convenient)
openAPIPath := ri.path
// Build operation
op := map[string]interface{}{
"summary": func() string {
if hasMeta {
return meta.Summary
}
return ri.path
}(),
"responses": map[string]interface{}{
"200": map[string]interface{}{
"description": "Success",
"content": map[string]interface{}{
"application/json": map[string]interface{}{
"schema": map[string]interface{}{"type": "object"},
},
},
},
},
}
if hasMeta {
if meta.Description != "" {
op["description"] = meta.Description
}
if meta.Tag != "" {
op["tags"] = []string{meta.Tag}
tagSet[meta.Tag] = true
}
if meta.Auth {
op["security"] = []map[string]interface{}{
{"ApiKeyAuth": []string{}},
}
}
// Add query parameters
if len(meta.QueryParams) > 0 {
params := make([]interface{}, 0, len(meta.QueryParams))
for _, qp := range meta.QueryParams {
p := map[string]interface{}{
"name": qp.Name,
"in": "query",
"required": qp.Required,
"schema": map[string]interface{}{"type": qp.Type},
}
if qp.Description != "" {
p["description"] = qp.Description
}
params = append(params, p)
}
op["parameters"] = params
}
}
// Extract path parameters from {name} patterns
pathParams := extractPathParams(openAPIPath)
if len(pathParams) > 0 {
existing, _ := op["parameters"].([]interface{})
for _, pp := range pathParams {
existing = append(existing, map[string]interface{}{
"name": pp,
"in": "path",
"required": true,
"schema": map[string]interface{}{"type": "string"},
})
}
op["parameters"] = existing
}
// Add to paths
methodLower := strings.ToLower(ri.method)
if _, ok := paths[openAPIPath]; !ok {
paths[openAPIPath] = make(map[string]interface{})
}
paths[openAPIPath].(map[string]interface{})[methodLower] = op
}
// Build tags array (sorted)
tagOrder := []string{"admin", "analytics", "channels", "config", "nodes", "observers", "packets"}
tagDescriptions := map[string]string{
"admin": "Server administration and diagnostics",
"analytics": "Network analytics and statistics",
"channels": "Mesh channel operations",
"config": "Server configuration",
"nodes": "Mesh node operations",
"observers": "Packet observer/gateway operations",
"packets": "Packet capture and decoding",
}
var tags []interface{}
for _, t := range tagOrder {
if tagSet[t] {
tags = append(tags, map[string]interface{}{
"name": t,
"description": tagDescriptions[t],
})
}
}
spec := map[string]interface{}{
"openapi": "3.0.3",
"info": map[string]interface{}{
"title": "CoreScope API",
"description": "MeshCore network analyzer — packet capture, node tracking, and mesh analytics.",
"version": version,
"license": map[string]interface{}{
"name": "MIT",
},
},
"paths": paths,
"tags": tags,
"components": map[string]interface{}{
"securitySchemes": map[string]interface{}{
"ApiKeyAuth": map[string]interface{}{
"type": "apiKey",
"in": "header",
"name": "X-API-Key",
},
},
},
}
return spec
}
// extractPathParams returns parameter names from a mux-style path like /api/nodes/{pubkey}.
func extractPathParams(path string) []string {
var params []string
for {
start := strings.Index(path, "{")
if start == -1 {
break
}
end := strings.Index(path[start:], "}")
if end == -1 {
break
}
params = append(params, path[start+1:start+end])
path = path[start+end+1:]
}
return params
}
// handleOpenAPISpec serves the OpenAPI 3.0 spec as JSON.
// The router is injected via RegisterRoutes storing it on the Server.
func (s *Server) handleOpenAPISpec(w http.ResponseWriter, r *http.Request) {
spec := buildOpenAPISpec(s.router, s.version)
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Access-Control-Allow-Origin", "*")
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
if err := enc.Encode(spec); err != nil {
http.Error(w, fmt.Sprintf("failed to encode spec: %v", err), http.StatusInternalServerError)
}
}
// handleSwaggerUI serves a minimal Swagger UI page.
func (s *Server) handleSwaggerUI(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
fmt.Fprint(w, swaggerUIHTML)
}
const swaggerUIHTML = `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>CoreScope API Swagger UI</title>
<link rel="stylesheet" href="https://unpkg.com/swagger-ui-dist@5/swagger-ui.css">
<style>
html { box-sizing: border-box; overflow-y: scroll; }
*, *:before, *:after { box-sizing: inherit; }
body { margin: 0; background: #fafafa; }
.topbar { display: none; }
</style>
</head>
<body>
<div id="swagger-ui"></div>
<script src="https://unpkg.com/swagger-ui-dist@5/swagger-ui-bundle.js"></script>
<script>
SwaggerUIBundle({
url: '/api/spec',
dom_id: '#swagger-ui',
deepLinking: true,
presets: [
SwaggerUIBundle.presets.apis,
SwaggerUIBundle.SwaggerUIStandalonePreset
],
layout: 'BaseLayout'
});
</script>
</body>
</html>`
-142
View File
@@ -1,142 +0,0 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestOpenAPISpecEndpoint(t *testing.T) {
_, r := setupTestServer(t)
req := httptest.NewRequest("GET", "/api/spec", nil)
w := httptest.NewRecorder()
r.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
ct := w.Header().Get("Content-Type")
if ct != "application/json; charset=utf-8" {
t.Errorf("unexpected content-type: %s", ct)
}
var spec map[string]interface{}
if err := json.Unmarshal(w.Body.Bytes(), &spec); err != nil {
t.Fatalf("invalid JSON: %v", err)
}
// Check required OpenAPI fields
if spec["openapi"] != "3.0.3" {
t.Errorf("expected openapi 3.0.3, got %v", spec["openapi"])
}
info, ok := spec["info"].(map[string]interface{})
if !ok {
t.Fatal("missing info object")
}
if info["title"] != "CoreScope API" {
t.Errorf("unexpected title: %v", info["title"])
}
paths, ok := spec["paths"].(map[string]interface{})
if !ok {
t.Fatal("missing paths object")
}
// Should have at least 20 paths
if len(paths) < 20 {
t.Errorf("expected at least 20 paths, got %d", len(paths))
}
// Check a known path exists
if _, ok := paths["/api/nodes"]; !ok {
t.Error("missing /api/nodes path")
}
if _, ok := paths["/api/packets"]; !ok {
t.Error("missing /api/packets path")
}
// Check tags exist
tags, ok := spec["tags"].([]interface{})
if !ok || len(tags) == 0 {
t.Error("missing or empty tags")
}
// Check security schemes
components, ok := spec["components"].(map[string]interface{})
if !ok {
t.Fatal("missing components")
}
schemes, ok := components["securitySchemes"].(map[string]interface{})
if !ok {
t.Fatal("missing securitySchemes")
}
if _, ok := schemes["ApiKeyAuth"]; !ok {
t.Error("missing ApiKeyAuth security scheme")
}
// Spec should NOT contain /api/spec or /api/docs (self-referencing)
if _, ok := paths["/api/spec"]; ok {
t.Error("/api/spec should not appear in the spec")
}
if _, ok := paths["/api/docs"]; ok {
t.Error("/api/docs should not appear in the spec")
}
}
func TestSwaggerUIEndpoint(t *testing.T) {
_, r := setupTestServer(t)
req := httptest.NewRequest("GET", "/api/docs", nil)
w := httptest.NewRecorder()
r.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
ct := w.Header().Get("Content-Type")
if ct != "text/html; charset=utf-8" {
t.Errorf("unexpected content-type: %s", ct)
}
body := w.Body.String()
if len(body) < 100 {
t.Error("response too short for Swagger UI HTML")
}
if !strings.Contains(body, "swagger-ui") {
t.Error("response doesn't contain swagger-ui reference")
}
if !strings.Contains(body, "/api/spec") {
t.Error("response doesn't point to /api/spec")
}
}
func TestExtractPathParams(t *testing.T) {
tests := []struct {
path string
expect []string
}{
{"/api/nodes", nil},
{"/api/nodes/{pubkey}", []string{"pubkey"}},
{"/api/channels/{hash}/messages", []string{"hash"}},
}
for _, tt := range tests {
got := extractPathParams(tt.path)
if len(got) != len(tt.expect) {
t.Errorf("extractPathParams(%q) = %v, want %v", tt.path, got, tt.expect)
continue
}
for i := range got {
if got[i] != tt.expect[i] {
t.Errorf("extractPathParams(%q)[%d] = %q, want %q", tt.path, i, got[i], tt.expect[i])
}
}
}
}
+7 -24
View File
@@ -1,7 +1,6 @@
package main
import (
"crypto/subtle"
"database/sql"
"encoding/json"
"fmt"
@@ -43,9 +42,6 @@ type Server struct {
// Neighbor affinity graph (lazy-built, cached with TTL)
neighborMu sync.Mutex
neighborGraph *NeighborGraph
// Router reference for OpenAPI spec generation
router *mux.Router
}
// PerfStats tracks request performance.
@@ -102,7 +98,6 @@ func (s *Server) getMemStats() runtime.MemStats {
// RegisterRoutes sets up all HTTP routes on the given router.
func (s *Server) RegisterRoutes(r *mux.Router) {
s.router = r
// Performance instrumentation middleware
r.Use(s.perfMiddleware)
@@ -170,18 +165,16 @@ func (s *Server) RegisterRoutes(r *mux.Router) {
r.HandleFunc("/api/traces/{hash}", s.handleTraces).Methods("GET")
r.HandleFunc("/api/iata-coords", s.handleIATACoords).Methods("GET")
r.HandleFunc("/api/audio-lab/buckets", s.handleAudioLabBuckets).Methods("GET")
// OpenAPI spec + Swagger UI
r.HandleFunc("/api/spec", s.handleOpenAPISpec).Methods("GET")
r.HandleFunc("/api/docs", s.handleSwaggerUI).Methods("GET")
}
func (s *Server) backfillStatusMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if s.store != nil && s.store.backfillComplete.Load() {
w.Header().Set("X-CoreScope-Status", "ready")
} else {
w.Header().Set("X-CoreScope-Status", "backfilling")
if strings.HasPrefix(r.URL.Path, "/api/") {
if s.store != nil && s.store.backfillComplete.Load() {
w.Header().Set("X-CoreScope-Status", "ready")
} else {
w.Header().Set("X-CoreScope-Status", "backfilling")
}
}
next.ServeHTTP(w, r)
})
@@ -247,15 +240,10 @@ func (s *Server) requireAPIKey(next http.Handler) http.Handler {
writeError(w, http.StatusForbidden, "write endpoints disabled — set apiKey in config.json")
return
}
key := r.Header.Get("X-API-Key")
if !constantTimeEqual(key, s.cfg.APIKey) {
if r.Header.Get("X-API-Key") != s.cfg.APIKey {
writeError(w, http.StatusUnauthorized, "unauthorized")
return
}
if IsWeakAPIKey(key) {
writeError(w, http.StatusForbidden, "forbidden")
return
}
next.ServeHTTP(w, r)
})
}
@@ -2324,8 +2312,3 @@ func (s *Server) handleAdminPrune(w http.ResponseWriter, r *http.Request) {
log.Printf("[prune] deleted %d transmissions older than %d days", n, days)
writeJSON(w, map[string]interface{}{"deleted": n, "days": days})
}
// constantTimeEqual compares two strings in constant time to prevent timing attacks.
func constantTimeEqual(a, b string) bool {
return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
}
+6 -6
View File
@@ -47,7 +47,7 @@ func setupTestServerWithAPIKey(t *testing.T, apiKey string) (*Server, *mux.Route
}
func TestWriteEndpointsRequireAPIKey(t *testing.T) {
_, router := setupTestServerWithAPIKey(t, "test-secret-key-strong-enough")
_, router := setupTestServerWithAPIKey(t, "test-secret")
t.Run("missing key returns 401", func(t *testing.T) {
req := httptest.NewRequest("POST", "/api/perf/reset", nil)
@@ -65,7 +65,7 @@ func TestWriteEndpointsRequireAPIKey(t *testing.T) {
t.Run("wrong key returns 401", func(t *testing.T) {
req := httptest.NewRequest("POST", "/api/perf/reset", nil)
req.Header.Set("X-API-Key", "wrong-secret-key-strong-enough")
req.Header.Set("X-API-Key", "wrong-secret")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusUnauthorized {
@@ -75,7 +75,7 @@ func TestWriteEndpointsRequireAPIKey(t *testing.T) {
t.Run("correct key passes", func(t *testing.T) {
req := httptest.NewRequest("POST", "/api/perf/reset", nil)
req.Header.Set("X-API-Key", "test-secret-key-strong-enough")
req.Header.Set("X-API-Key", "test-secret")
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
@@ -3186,7 +3186,7 @@ func TestHashCollisionsClassification(t *testing.T) {
}
func TestHashCollisionsCacheTTL(t *testing.T) {
// Issue #420: collision cache should use dedicated TTL, default 3600s (1 hour)
// Issue #420: collision cache should use dedicated TTL (60s), not rfCacheTTL (15s)
db := setupTestDB(t)
seedTestData(t, db)
store := NewPacketStore(db, nil)
@@ -3194,8 +3194,8 @@ func TestHashCollisionsCacheTTL(t *testing.T) {
t.Fatalf("store.Load failed: %v", err)
}
if store.collisionCacheTTL != 3600*time.Second {
t.Errorf("expected collisionCacheTTL=3600s, got %v", store.collisionCacheTTL)
if store.collisionCacheTTL != 60*time.Second {
t.Errorf("expected collisionCacheTTL=60s, got %v", store.collisionCacheTTL)
}
if store.rfCacheTTL != 15*time.Second {
t.Errorf("expected rfCacheTTL=15s, got %v", store.rfCacheTTL)
+3 -77
View File
@@ -80,45 +80,6 @@ func (tx *StoreTx) ParsedDecoded() map[string]interface{} {
}
// PacketStore holds all transmissions in memory with indexes for fast queries.
//
// Lock ordering
// =============
// PacketStore uses several mutexes. To prevent deadlocks, locks MUST be
// acquired in the order listed below. Never acquire a higher-numbered lock
// while holding a lower-numbered one.
//
// 1. mu (sync.RWMutex) — guards the core packet data: packets,
// indexes (byHash, byTxID, byObsID, byObserver, byNode,
// byPathHop, byPayloadType), counters, and loaded flag.
//
// 2. cacheMu (sync.Mutex) — guards analytics response caches:
// rfCache, topoCache, hashCache, collisionCache, chanCache,
// distCache, subpathCache, and their TTLs/hit counters.
// Also guards rate-limited invalidation state
// (lastInvalidated, pendingInv).
//
// 3. channelsCacheMu (sync.Mutex) — guards the short-lived GetChannels
// cache (channelsCacheKey/Exp/Res).
//
// 4. groupedCacheMu (sync.Mutex) — guards the short-lived
// QueryGroupedPackets cache.
//
// 5. regionObsMu (sync.Mutex) — guards the region→observer mapping
// cache (regionObsCache, regionObsCacheTime).
//
// 6. hashSizeInfoMu (sync.Mutex) — guards the cached hash-size-info
// result (hashSizeInfoCache). Acquired independently or
// under mu (in EvictStale).
//
// Nesting that occurs today:
// - IngestNew: mu → cacheMu → channelsCacheMu (1 → 2 → 3, OK)
// - IngestObservations: mu → cacheMu (1 → 2, OK)
// - RunEviction/EvictStale: mu → cacheMu → channelsCacheMu (1 → 2 → 3, OK)
// - RunEviction/EvictStale: mu → hashSizeInfoMu (1 → 6, OK)
// - invalidateCachesFor: cacheMu → channelsCacheMu (2 → 3, OK)
//
// All other locks are acquired independently (no nesting).
// When adding new lock acquisitions, respect this ordering.
type PacketStore struct {
mu sync.RWMutex
db *DB
@@ -199,7 +160,7 @@ type PacketStore struct {
// Async backfill state: set after backfillResolvedPathsAsync completes.
backfillComplete atomic.Bool
// Progress tracking for async backfill (total pending and processed so far).
backfillTotal atomic.Int64 // set once at start of async backfill
backfillTotal atomic.Int64
backfillProcessed atomic.Int64
// Eviction config and stats
@@ -246,33 +207,8 @@ type cachedResult struct {
expiresAt time.Time
}
// cacheTTLSec extracts a duration from the cacheTTL config map.
// Values may be float64 (from JSON) or int. Returns false if key is missing or non-positive.
func cacheTTLSec(m map[string]interface{}, key string) (time.Duration, bool) {
v, ok := m[key]
if !ok {
return 0, false
}
var sec float64
switch n := v.(type) {
case float64:
sec = n
case int:
sec = float64(n)
case int64:
sec = float64(n)
default:
return 0, false
}
if sec <= 0 {
return 0, false
}
return time.Duration(sec * float64(time.Second)), true
}
// NewPacketStore creates a new empty packet store backed by db.
// cacheTTLs is the optional cacheTTL map from config.json; keys are strings, values are seconds.
func NewPacketStore(db *DB, cfg *PacketStoreConfig, cacheTTLs ...map[string]interface{}) *PacketStore {
func NewPacketStore(db *DB, cfg *PacketStoreConfig) *PacketStore {
ps := &PacketStore{
db: db,
packets: make([]*StoreTx, 0, 65536),
@@ -293,7 +229,7 @@ func NewPacketStore(db *DB, cfg *PacketStoreConfig, cacheTTLs ...map[string]inte
distCache: make(map[string]*cachedResult),
subpathCache: make(map[string]*cachedResult),
rfCacheTTL: 15 * time.Second,
collisionCacheTTL: 3600 * time.Second,
collisionCacheTTL: 60 * time.Second,
invCooldown: 10 * time.Second,
spIndex: make(map[string]int, 4096),
spTxIndex: make(map[string][]*StoreTx, 4096),
@@ -303,16 +239,6 @@ func NewPacketStore(db *DB, cfg *PacketStoreConfig, cacheTTLs ...map[string]inte
ps.retentionHours = cfg.RetentionHours
ps.maxMemoryMB = cfg.MaxMemoryMB
}
// Wire cacheTTL config values to server-side cache durations.
if len(cacheTTLs) > 0 && cacheTTLs[0] != nil {
ct := cacheTTLs[0]
if v, ok := cacheTTLSec(ct, "analyticsHashSizes"); ok {
ps.collisionCacheTTL = v
}
if v, ok := cacheTTLSec(ct, "analyticsRF"); ok {
ps.rfCacheTTL = v
}
}
return ps
}
+19 -73
View File
@@ -8,15 +8,13 @@
},
"https": {
"cert": "/path/to/cert.pem",
"key": "/path/to/key.pem",
"_comment": "TLS cert/key paths for direct HTTPS. Most deployments use Caddy (included in Docker) for auto-TLS instead."
"key": "/path/to/key.pem"
},
"branding": {
"siteName": "CoreScope",
"tagline": "Real-time MeshCore LoRa mesh network analyzer",
"logoUrl": null,
"faviconUrl": null,
"_comment": "Customize site name, tagline, logo, and favicon. logoUrl/faviconUrl can be absolute URLs or relative paths."
"faviconUrl": null
},
"theme": {
"accent": "#4a9eff",
@@ -25,75 +23,38 @@
"navBg2": "#1a1a2e",
"statusGreen": "#45644c",
"statusYellow": "#b08b2d",
"statusRed": "#b54a4a",
"_comment": "CSS color overrides. Use the in-app Theme Customizer for live preview, then export values here."
"statusRed": "#b54a4a"
},
"nodeColors": {
"repeater": "#dc2626",
"companion": "#2563eb",
"room": "#16a34a",
"sensor": "#d97706",
"observer": "#8b5cf6",
"_comment": "Marker/badge colors per node role. Used on map, nodes list, and live feed."
"observer": "#8b5cf6"
},
"home": {
"heroTitle": "CoreScope",
"heroSubtitle": "Find your nodes to start monitoring them.",
"steps": [
{
"emoji": "\ud83d\udce1",
"title": "Connect",
"description": "Link your node to the mesh"
},
{
"emoji": "\ud83d\udd0d",
"title": "Monitor",
"description": "Watch packets flow in real-time"
},
{
"emoji": "\ud83d\udcca",
"title": "Analyze",
"description": "Understand your network's health"
}
{ "emoji": "📡", "title": "Connect", "description": "Link your node to the mesh" },
{ "emoji": "🔍", "title": "Monitor", "description": "Watch packets flow in real-time" },
{ "emoji": "📊", "title": "Analyze", "description": "Understand your network's health" }
],
"checklist": [
{
"question": "How do I add my node?",
"answer": "Search for your node name or paste your public key."
},
{
"question": "What regions are covered?",
"answer": "Check the map page to see active observers and nodes."
}
{ "question": "How do I add my node?", "answer": "Search for your node name or paste your public key." },
{ "question": "What regions are covered?", "answer": "Check the map page to see active observers and nodes." }
],
"footerLinks": [
{
"label": "\ud83d\udce6 Packets",
"url": "#/packets"
},
{
"label": "\ud83d\uddfa\ufe0f Network Map",
"url": "#/map"
},
{
"label": "\ud83d\udd34 Live",
"url": "#/live"
},
{
"label": "\ud83d\udce1 All Nodes",
"url": "#/nodes"
},
{
"label": "\ud83d\udcac Channels",
"url": "#/channels"
}
],
"_comment": "Customize the landing page hero, onboarding steps, FAQ, and footer links."
{ "label": "📦 Packets", "url": "#/packets" },
{ "label": "🗺️ Network Map", "url": "#/map" },
{ "label": "🔴 Live", "url": "#/live" },
{ "label": "📡 All Nodes", "url": "#/nodes" },
{ "label": "💬 Channels", "url": "#/channels" }
]
},
"mqtt": {
"broker": "mqtt://localhost:1883",
"topic": "meshcore/+/+/packets",
"_comment": "Legacy single-broker config. Prefer mqttSources[] for multiple brokers."
"topic": "meshcore/+/+/packets"
},
"mqttSources": [
{
@@ -189,26 +150,11 @@
"timezone": "local",
"formatPreset": "iso",
"customFormat": "",
"allowCustomFormat": false,
"_comment": "defaultMode: ago|local|iso. timezone: local|utc. formatPreset: iso|us|eu. customFormat: strftime-style (requires allowCustomFormat: true)."
"allowCustomFormat": false
},
"packetStore": {
"maxMemoryMB": 1024,
"estimatedPacketBytes": 450,
"_comment": "In-memory packet store. maxMemoryMB caps RAM usage. All packets loaded on startup, served from RAM."
},
"resolvedPath": {
"backfillHours": 24,
"_comment": "How far back (hours) the async backfill scans for observations with NULL resolved_path. Default: 24. Set higher to backfill older data, lower to speed up startup."
},
"neighborGraph": {
"maxAgeDays": 5,
"_comment": "Neighbor edges older than this many days are pruned on startup and daily. Default: 5."
},
"_comment_mqttSources": "Each source connects to an MQTT broker. topics: what to subscribe to. iataFilter: only ingest packets from these regions (optional).",
"_comment_channelKeys": "Hex keys for decrypting channel messages. Key name = channel display name. public channel key is well-known.",
"_comment_hashChannels": "Channel names whose keys are derived via SHA256. Key = SHA256(name)[:16]. Listed here so the ingestor can auto-derive keys.",
"_comment_defaultRegion": "IATA code shown by default in region filters.",
"_comment_mapDefaults": "Initial map center [lat, lon] and zoom level.",
"_comment_regions": "IATA code to display name mapping. Packets are tagged with region codes by MQTT topic structure."
}
}
}
-20
View File
@@ -1,20 +0,0 @@
# CoreScope — simple deployment using pre-built image from GHCR
# Usage: docker compose -f docker-compose.example.yml up -d
# Docs: https://github.com/Kpa-clawbot/CoreScope/blob/master/DEPLOY.md
services:
corescope:
image: ghcr.io/kpa-clawbot/corescope:latest
ports:
- "${HTTP_PORT:-80}:80"
volumes:
- ${DATA_DIR:-./data}:/app/data
environment:
- DISABLE_CADDY=${DISABLE_CADDY:-true}
- DISABLE_MOSQUITTO=${DISABLE_MOSQUITTO:-false}
restart: unless-stopped
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:3000/api/stats"]
interval: 30s
timeout: 5s
retries: 3
-415
View File
@@ -1,415 +0,0 @@
# CoreScope Deployment Guide
Comprehensive guide to deploying and operating CoreScope. For a quick start, see [DEPLOY.md](../DEPLOY.md).
## Table of Contents
- [System Requirements](#system-requirements)
- [Docker Deployment](#docker-deployment)
- [Configuration Reference](#configuration-reference)
- [MQTT Setup](#mqtt-setup)
- [TLS / HTTPS](#tls--https)
- [Monitoring & Health Checks](#monitoring--health-checks)
- [Backup & Restore](#backup--restore)
- [Troubleshooting](#troubleshooting)
---
## System Requirements
| Resource | Minimum | Recommended |
|----------|---------|-------------|
| RAM | 256 MB | 512 MB+ |
| Disk | 500 MB (image + DB) | 2 GB+ for long-term data |
| CPU | 1 core | 2+ cores |
| Architecture | `linux/amd64`, `linux/arm64` | — |
| Docker | 20.10+ | Latest stable |
CoreScope runs well on Raspberry Pi 4/5 (ARM64). The Go server uses ~300 MB RAM for 56K+ packets.
---
## Docker Deployment
### Quick Start (one command)
```bash
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
```
Open `http://localhost` — you'll see an empty dashboard ready to receive packets.
No `config.json` is required. The server starts with sensible defaults:
- HTTP on port 3000 (Caddy proxies port 80 → 3000 internally)
- Internal Mosquitto MQTT broker on port 1883
- Ingestor connects to `mqtt://localhost:1883` automatically
- SQLite database at `/app/data/meshcore.db`
### Docker Compose (recommended for production)
Download the example compose file:
```bash
curl -sL https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/docker-compose.example.yml \
-o docker-compose.yml
docker compose up -d
```
#### Compose environment variables
| Variable | Default | Description |
|----------|---------|-------------|
| `HTTP_PORT` | `80` | Host port for the web UI |
| `DATA_DIR` | `./data` | Host path for persistent data |
| `DISABLE_MOSQUITTO` | `false` | Set `true` to use an external MQTT broker |
### Image tags
| Tag | Use case |
|-----|----------|
| `v3.4.1` | Pinned release — recommended for production |
| `v3.4` | Latest patch in the v3.4.x series |
| `v3` | Latest minor+patch in v3.x |
| `latest` | Latest release tag |
| `edge` | Built from master on every push — unstable |
### Updating
```bash
docker compose pull
docker compose up -d
```
For `docker run` users:
```bash
docker pull ghcr.io/kpa-clawbot/corescope:latest
docker stop corescope && docker rm corescope
docker run -d --name corescope ... # same flags as before
```
Data is preserved in the volume — updates are non-destructive.
---
## Configuration Reference
CoreScope uses a layered configuration system (highest priority wins):
1. **Environment variables**`MQTT_BROKER`, `DB_PATH`, etc.
2. **`/app/data/config.json`** — full config file (volume-mounted)
3. **Built-in defaults** — work out of the box with no config
### Environment variable overrides
| Variable | Default | Description |
|----------|---------|-------------|
| `MQTT_BROKER` | `mqtt://localhost:1883` | MQTT broker URL (overrides config file) |
| `MQTT_TOPIC` | `meshcore/#` | MQTT topic subscription pattern |
| `DB_PATH` | `data/meshcore.db` | SQLite database path |
| `DISABLE_MOSQUITTO` | `false` | Skip the internal Mosquitto broker |
### config.json
For advanced configuration, create a `config.json` and mount it at `/app/data/config.json`:
```bash
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
-v ./config.json:/app/data/config.json:ro \
ghcr.io/kpa-clawbot/corescope:latest
```
See `config.example.json` in the repository for all available options including:
- MQTT sources (multiple brokers)
- Channel encryption keys
- Branding and theming
- Health thresholds
- Region filters
- Retention policies
- Geo-filtering
---
## MQTT Setup
CoreScope receives MeshCore packets via MQTT. The container ships with an internal Mosquitto broker — no setup needed for basic use.
### Internal broker (default)
The built-in Mosquitto broker listens on port 1883 inside the container. Point your MeshCore gateways at it:
```bash
# Expose MQTT port for external gateways
docker run -d --name corescope \
-p 80:80 -p 1883:1883 \
-v corescope-data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
```
### External broker
To use your own MQTT broker (Mosquitto, EMQX, HiveMQ, etc.):
1. Disable the internal broker:
```bash
-e DISABLE_MOSQUITTO=true
```
2. Point the ingestor at your broker:
```bash
-e MQTT_BROKER=mqtt://your-broker:1883
```
Or via `config.json`:
```json
{
"mqttSources": [
{
"name": "my-broker",
"broker": "mqtt://your-broker:1883",
"username": "user",
"password": "pass",
"topics": ["meshcore/#"]
}
]
}
```
### Multiple brokers
CoreScope can connect to multiple MQTT brokers simultaneously:
```json
{
"mqttSources": [
{
"name": "local",
"broker": "mqtt://localhost:1883",
"topics": ["meshcore/#"]
},
{
"name": "remote",
"broker": "mqtts://remote-broker:8883",
"username": "reader",
"password": "secret",
"topics": ["meshcore/+/+/packets"]
}
]
}
```
### MQTT topic format
MeshCore gateways typically publish to `meshcore/<gateway>/<region>/packets`. The default subscription `meshcore/#` catches all of them.
---
## TLS / HTTPS
### Option 1: External reverse proxy (recommended)
Run CoreScope behind nginx, Traefik, or Cloudflare Tunnel for TLS termination:
```nginx
# nginx example
server {
listen 443 ssl;
server_name corescope.example.com;
ssl_certificate /etc/ssl/certs/corescope.pem;
ssl_certificate_key /etc/ssl/private/corescope.key;
location / {
proxy_pass http://localhost:80;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
}
```
The `Upgrade` and `Connection` headers are required for WebSocket support.
### Option 2: Built-in Caddy (auto-TLS)
The container includes Caddy for automatic Let's Encrypt certificates:
1. Create a Caddyfile:
```
corescope.example.com {
reverse_proxy localhost:3000
}
```
2. Mount it and expose TLS ports:
```bash
docker run -d --name corescope \
-p 80:80 -p 443:443 \
-v corescope-data:/app/data \
-v caddy-certs:/data/caddy \
-v ./Caddyfile:/etc/caddy/Caddyfile:ro \
ghcr.io/kpa-clawbot/corescope:latest
```
Caddy handles certificate issuance and renewal automatically.
---
## Monitoring & Health Checks
### Docker health check
The container includes a built-in health check that hits `/api/stats`:
```bash
docker inspect --format='{{.State.Health.Status}}' corescope
```
Docker reports `healthy` or `unhealthy` automatically. The check runs every 30 seconds.
### Manual health check
```bash
curl -f http://localhost/api/stats
```
Returns JSON with packet counts, node counts, and version info:
```json
{
"totalPackets": 56234,
"totalNodes": 142,
"totalObservers": 12,
"packetsLastHour": 830,
"packetsLast24h": 19644,
"engine": "go",
"version": "v3.4.1"
}
```
### Log monitoring
```bash
# All logs
docker compose logs -f
# Server only
docker compose logs -f | grep '\[server\]'
# Ingestor only
docker compose logs -f | grep '\[ingestor\]'
```
### Resource monitoring
```bash
docker stats corescope
```
---
## Backup & Restore
### Backup
All persistent data lives in `/app/data`. The critical file is the SQLite database:
```bash
# Copy from the Docker volume
docker cp corescope:/app/data/meshcore.db ./backup-$(date +%Y%m%d).db
# Or if using a bind mount
cp ./data/meshcore.db ./backup-$(date +%Y%m%d).db
```
Optional files to back up:
- `config.json` — custom configuration
- `theme.json` — custom theme/branding
### Restore
```bash
# Stop the container
docker stop corescope
# Replace the database
docker cp ./backup.db corescope:/app/data/meshcore.db
# Restart
docker start corescope
```
### Automated backups
```bash
# cron: daily backup at 3 AM, keep 7 days
0 3 * * * docker cp corescope:/app/data/meshcore.db /backups/corescope-$(date +\%Y\%m\%d).db && find /backups -name "corescope-*.db" -mtime +7 -delete
```
---
## Troubleshooting
### Container starts but dashboard is empty
This is normal on first start with no MQTT sources configured. The dashboard shows data once packets arrive via MQTT. Either:
- Point a MeshCore gateway at the container's MQTT broker (port 1883)
- Configure an external MQTT source in `config.json`
### "no MQTT connections established" in logs
The ingestor couldn't connect to any MQTT broker. Check:
1. Is the internal Mosquitto running? (`DISABLE_MOSQUITTO` should be `false`)
2. Is the external broker reachable? Test with `mosquitto_sub -h broker -t meshcore/#`
3. Are credentials correct in `config.json`?
### WebSocket disconnects / real-time updates stop
If behind a reverse proxy, ensure WebSocket upgrade headers are forwarded:
```nginx
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
```
Also check proxy timeouts — set them to at least 300s for long-lived WebSocket connections.
### High memory usage
The in-memory packet store grows with retained packets. Configure retention limits in `config.json`:
```json
{
"packetStore": {
"retentionHours": 24,
"maxMemoryMB": 512
},
"retention": {
"nodeDays": 7,
"packetDays": 30
}
}
```
### Database locked errors
SQLite doesn't support concurrent writers well. Ensure only one CoreScope instance accesses the database file. If running multiple containers, each needs its own database.
### Container unhealthy
Check logs: `docker compose logs --tail 50`. Common causes:
- Port 3000 already in use inside the container
- Database file permissions (must be writable by the container user)
- Corrupted database — restore from backup
### ARM / Raspberry Pi issues
- Use `linux/arm64` images (Pi 4 and 5). Pi 3 (armv7) is not supported.
- First pull may be slow — the multi-arch manifest selects the right image automatically.
- If memory is tight, set `packetStore.maxMemoryMB` to limit RAM usage.
-4
View File
@@ -431,10 +431,6 @@ Note: No hardcoded duty cycle limit line on charts. Duty cycle regulations vary
- All charts time-aligned, sharing X-axis, reboot markers spanning all charts
- Tests: delta computation, reboot handling, counter reset, gap insertion, downsampling, error rate calculation
#### M2 feedback improvements (post-M2)
- **Auto-scale airtime Y-axis**: clamp to min/max of actual data values (20% headroom, min 1%) instead of fixed 0-100%, matching noise floor chart behavior. Increases data-ink ratio for low-activity nodes.
- **Hover tooltips on all chart data points**: invisible SVG circles with `<title>` elements on every data point across all 4 charts (noise floor, airtime, error rate, battery). Shows exact value + UTC timestamp on hover. Detail-on-demand without cluttering the chart.
### M3: Pattern detection
- Implement after operators have used raw charts (M1M2) and provided feedback
- Jammer detection (NF spike + RX drop)
-144
View File
@@ -1,144 +0,0 @@
# Table Sorting Consistency Spec (#620)
## Problem
CoreScope has 20+ data tables. Only 2 are sortable (nodes list, channel activity). Those 2 use incompatible implementations — different property names (`column`/`direction` vs `col`/`dir`), different data attributes (`data-sort` vs `data-sort-col`), different function signatures. The remaining 18+ tables, including the packets table (30K+ rows), have zero sorting.
This violates AGENTS.md DRY rules and frustrates users who can see data but can't reorder it.
## Solution
One shared `TableSort` module. Every data table uses it. Same UX everywhere.
## Shared Utility Design
### Module: `public/table-sort.js`
IIFE pattern (like `channel-colors.js`). No dependencies. No build step.
```js
window.TableSort = (function() {
return { init, sort, destroy };
})();
```
### API
```js
TableSort.init(tableEl, {
defaultColumn: 'last_seen', // initial sort column
defaultDirection: 'desc', // 'asc' or 'desc'
storageKey: 'nodes-sort', // localStorage key (optional)
comparators: { // custom comparators for non-string columns
time: (a, b) => ...,
snr: (a, b) => ...,
},
onSort: (column, direction) => {} // callback after sort completes
});
```
### How It Works
1. Scans `<th>` elements for `data-sort="columnName"` attribute
2. Attaches click handlers — click toggles asc/desc
3. On sort: reads `<td data-value="...">` (raw sortable value) from each row
4. Sorts rows in-place via DOM reorder (no innerHTML rebuild — important for 30K rows)
5. Updates visual indicator and `aria-sort` on active `<th>`
### Visual Indicator
Active column header gets `▲` (ascending) or `▼` (descending) appended as a `<span class="sort-arrow">`. Inactive columns show no arrow. CSS class `.sort-active` on the active `<th>`.
### Built-in Comparators
| Type | Detected From | Behavior |
|------|--------------|----------|
| `numeric` | `data-type="number"` on `<th>` | `Number(a) - Number(b)`, NaN sorts last |
| `text` | default | `localeCompare` |
| `date` | `data-type="date"` | Parse as timestamp, numeric compare |
| `dbm` | `data-type="dbm"` | Strip " dBm" suffix, numeric compare |
Custom comparators in `options.comparators` override built-in types.
### Accessibility
- `aria-sort="ascending"`, `"descending"`, or `"none"` on every sortable `<th>`
- `role="columnheader"` (already implicit for `<th>`)
- `cursor: pointer` and `:hover` style on sortable headers
- Keyboard: sortable headers are focusable, Enter/Space triggers sort
### Performance (Critical for Packets Table)
- Sort via DOM node reorder (`appendChild` loop), not `innerHTML`. Browser batches reflows.
- `data-value` attributes hold raw values — no parsing during sort.
- For 30K rows: expected sort time ~100-200ms (single `Array.sort` + DOM reorder). If >500ms, add a virtual scroll layer in a follow-up — but don't pre-optimize.
- No re-render of row content. Sort only changes order.
## Milestones
### M1: Shared utility + packets table
- Create `public/table-sort.js`
- Unit tests: `test-table-sort.js` (Node.js, jsdom or vm.createContext)
- Integrate with packets table (highest impact — 30K rows, currently unsortable)
- Default sort: time descending
- Columns: all current packets columns (Region, Time, Hash, Size, HB, Type, Observer, Path, Rpt, Details)
- Browser validation: sort 30K rows, verify <500ms
### M2: Nodes list + node detail tables
- Migrate nodes list from custom sort to `TableSort.init()`
- Add sorting to neighbor table (side pane + detail page)
- Add sorting to observer stats table (detail page)
- Remove old `sortState`/`sortArrow` code from `nodes.js`
### M3: Analytics tables
- Hash collisions tables (node table, sizes table, collision prefixes)
- RF statistics table
- Route frequency, co-appearance, topology tables
- Node health tables (top by packets/SNR/observers, recently active)
- Distance tables (by link type, top 20 longest)
- Per-node analytics: peer contacts
### M4: Channels list + observers list + comparison table
- Channel activity table: migrate from custom sort to `TableSort.init()`
- Remove old `_channelSortState` code from `analytics.js`
- Observers list table
- Comparison table (`compare.js`)
### M5: Cleanup
- Remove all old sorting code (both implementations)
- Verify no dead CSS/JS from old sort code
- Final consistency audit: every data table uses `TableSort.init()`
### Out of Scope
- `packets.js` hex breakdown (structural decode, fixed order)
- `audio-lab.js` debug tables (not user-facing)
- Virtual scroll / pagination (separate issue if perf requires it)
## Testing
### Unit Tests (`test-table-sort.js`)
- Numeric sort ascending/descending
- Text sort with localeCompare
- Date sort
- dBm sort (strip suffix)
- Custom comparator override
- NaN/null/undefined sort to end
- Toggle direction on repeated click
- `aria-sort` attribute updates
- localStorage persistence (read + write)
- `data-value` attribute used over text content
### Integration (per milestone)
- Playwright test: click column header, verify row order changes
- Playwright test: click again, verify direction toggles
- Playwright test: visual indicator present on active column
### Performance
- Unit test: sort 30K mock rows in <500ms (assert timing)
- Required per AGENTS.md: perf claims need proof
## Migration Path
Existing sort code in `nodes.js` and `analytics.js` will be replaced, not wrapped. Both current implementations are <100 lines each — replacing is simpler than adapting. The shared utility subsumes all their functionality.
Old localStorage keys (`nodes-sort-*`, channel sort state) should be migrated or cleared on first use of the new utility.
+7 -42
View File
@@ -75,7 +75,7 @@
<h2>📊 Mesh Analytics</h2>
<p class="text-muted">Deep dive into your mesh network data</p>
<div id="analyticsRegionFilter" class="region-filter-container"></div>
<div class="analytics-tabs" id="analyticsTabs" role="tablist" aria-label="Analytics tabs">
<div class="analytics-tabs" id="analyticsTabs">
<button class="tab-btn active" data-tab="overview">Overview</button>
<button class="tab-btn" data-tab="rf">RF / Signal</button>
<button class="tab-btn" data-tab="topology">Topology</button>
@@ -90,7 +90,7 @@
<button class="tab-btn" data-tab="prefix-tool">Prefix Tool</button>
</div>
</div>
<div id="analyticsContent" class="analytics-content" aria-live="polite">
<div id="analyticsContent" class="analytics-content">
<div class="text-center text-muted" style="padding:40px">Loading analytics</div>
</div>
</div>`;
@@ -2945,20 +2945,6 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
}
// Shared helper: render X-axis time labels
function rfTooltipCircles(data, sx, sy, label, unit, formatV) {
let svg = '';
formatV = formatV || (v => v.toFixed(1));
data.forEach(d => {
const t = new Date(d.t);
const x = sx(t.getTime());
const y = sy(d.v);
const ts = t.toISOString().replace('T', ' ').replace(/\.\d+Z/, ' UTC');
const tip = `${label}: ${formatV(d.v)}${unit}\n${ts}`;
svg += `<circle cx="${x.toFixed(1)}" cy="${y.toFixed(1)}" r="8" fill="transparent" stroke="none" pointer-events="all"><title>${tip}</title></circle>`;
});
return svg;
}
function rfXAxisLabels(data, sx, h, pad) {
let svg = '';
const xTicks = Math.min(6, data.length);
@@ -2981,26 +2967,18 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
const minT = sharedMinT, maxT = sharedMaxT;
const rangeT = maxT - minT || 1;
// Auto-scale Y-axis to data range (20% headroom, min 1%)
let dataMax = 0;
for (let i = 0; i < txData.length; i++) { if (txData[i].v > dataMax) dataMax = txData[i].v; }
for (let i = 0; i < rxData.length; i++) { if (rxData[i].v > dataMax) dataMax = rxData[i].v; }
const yMax = Math.max(dataMax * 1.2, 1);
const sx = t => pad.left + ((t - minT) / rangeT) * cw;
const sy = v => pad.top + ch - (v / yMax) * ch;
const sy = v => pad.top + ch - (v / 100) * ch; // 0-100%
let svg = `<svg viewBox="0 0 ${w} ${h}" style="width:100%;max-height:${h}px" role="img" aria-label="Airtime chart"><title>Airtime %</title>`;
// Chart title
svg += `<text x="${pad.left}" y="12" font-size="10" fill="var(--text-muted)" font-weight="600">Airtime %</text>`;
// Y-axis: 5 ticks from 0 to yMax
const yTicks = 4;
for (let i = 0; i <= yTicks; i++) {
const v = yMax * i / yTicks;
const y = sy(v);
svg += `<text x="${pad.left - 4}" y="${(y + 3).toFixed(1)}" text-anchor="end" font-size="9" fill="var(--text-muted)">${v.toFixed(1)}</text>`;
// Y-axis: 0, 25, 50, 75, 100
for (let pct = 0; pct <= 100; pct += 25) {
const y = sy(pct);
svg += `<text x="${pad.left - 4}" y="${(y + 3).toFixed(1)}" text-anchor="end" font-size="9" fill="var(--text-muted)">${pct}</text>`;
svg += `<line x1="${pad.left}" y1="${y.toFixed(1)}" x2="${w - pad.right}" y2="${y.toFixed(1)}" stroke="var(--border)" stroke-width="0.3"/>`;
}
@@ -3041,10 +3019,6 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
const allData = txData.length >= rxData.length ? txData : rxData;
svg += rfXAxisLabels(allData, sx, h, pad);
// Hover tooltips
svg += rfTooltipCircles(txData, sx, sy, 'TX', '%');
svg += rfTooltipCircles(rxData, sx, sy, 'RX', '%');
svg += '</svg>';
return svg;
}
@@ -3094,9 +3068,6 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
// X-axis labels
svg += rfXAxisLabels(errData, sx, h, pad);
// Hover tooltips
svg += rfTooltipCircles(errData, sx, sy, 'Err', '%', v => v.toFixed(2));
svg += '</svg>';
return svg;
}
@@ -3155,9 +3126,6 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
// X-axis labels
svg += rfXAxisLabels(battData, sx, h, pad);
// Hover tooltips
svg += rfTooltipCircles(battData, sx, sy, 'Batt', 'V', v => (v/1000).toFixed(2));
svg += '</svg>';
return svg;
}
@@ -3215,9 +3183,6 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
// Data polyline
svg += `<polyline points="${pts}" fill="none" stroke="var(--accent)" stroke-width="1.5"/>`;
// Hover tooltips
svg += rfTooltipCircles(data, sx, sy, 'NF', ' dBm');
// Direct labels: min and max points
const times = data.map(d => new Date(d.t).getTime());
const maxIdx = values.indexOf(maxV);
-6
View File
@@ -472,12 +472,6 @@ function navigate() {
const ms = performance.now() - t0;
if (ms > 100) console.warn(`[SLOW PAGE] ${basePage} init took ${Math.round(ms)}ms`);
app.classList.remove('page-enter'); void app.offsetWidth; app.classList.add('page-enter');
// #630-7: SPA focus management — move focus to first heading or main content
requestAnimationFrame(function() {
var heading = app.querySelector('h1, h2, h3, [role="heading"]');
if (heading) { heading.setAttribute('tabindex', '-1'); heading.focus({ preventScroll: true }); }
else { app.setAttribute('tabindex', '-1'); app.focus({ preventScroll: true }); }
});
} else {
app.innerHTML = `<div style="padding:40px;text-align:center;color:#6b7280"><h2>${route}</h2><p>Page not yet implemented.</p></div>`;
}
-344
View File
@@ -1,344 +0,0 @@
/**
* Channel Color Quick-Assign Popover (M2, #271)
*
* Right-click (or long-press on mobile) a channel name in the live feed
* or packets table to open a color picker popover.
*
* Uses ChannelColors.set/get/remove from channel-colors.js (M1).
*/
(function() {
'use strict';
// Curated maximally-distinct palette (10 swatches, ColorBrewer-inspired)
var PRESET_COLORS = [
'#ef4444', // red
'#f97316', // orange
'#eab308', // yellow
'#22c55e', // green
'#06b6d4', // cyan
'#3b82f6', // blue
'#8b5cf6', // violet
'#ec4899', // pink
'#14b8a6', // teal
'#f43f5e' // rose
];
var popoverEl = null;
var currentChannel = null;
var longPressTimer = null;
function createPopover() {
if (popoverEl) return popoverEl;
var el = document.createElement('div');
el.className = 'cc-picker-popover';
el.setAttribute('role', 'dialog');
el.setAttribute('aria-label', 'Channel color picker');
el.style.display = 'none';
el.innerHTML =
'<div class="cc-picker-header">' +
'<span class="cc-picker-title" id="cc-picker-title"></span>' +
'<button class="cc-picker-close" title="Close" aria-label="Close">✕</button>' +
'</div>' +
'<div class="cc-picker-swatches" role="group" aria-label="Color swatches"></div>' +
'<div class="cc-picker-custom">' +
'<label>Custom: <input type="color" class="cc-picker-input" value="#3b82f6" aria-label="Custom color"></label>' +
'<button class="cc-picker-apply">Apply</button>' +
'</div>' +
'<button class="cc-picker-clear">Clear color</button>';
el.setAttribute('aria-labelledby', 'cc-picker-title');
// Build swatches
var swatchContainer = el.querySelector('.cc-picker-swatches');
for (var i = 0; i < PRESET_COLORS.length; i++) {
var sw = document.createElement('button');
sw.className = 'cc-swatch';
sw.style.background = PRESET_COLORS[i];
sw.setAttribute('data-color', PRESET_COLORS[i]);
sw.setAttribute('aria-label', PRESET_COLORS[i]);
sw.title = PRESET_COLORS[i];
swatchContainer.appendChild(sw);
}
// Event: swatch click
swatchContainer.addEventListener('click', function(e) {
var btn = e.target.closest('.cc-swatch');
if (!btn) return;
assignColor(btn.getAttribute('data-color'));
});
// Keyboard navigation for swatches (arrow keys)
swatchContainer.addEventListener('keydown', function(e) {
var btn = e.target.closest('.cc-swatch');
if (!btn) return;
var swatches = swatchContainer.querySelectorAll('.cc-swatch');
var idx = Array.prototype.indexOf.call(swatches, btn);
if (idx < 0) return;
var next = -1;
if (e.key === 'ArrowRight' || e.key === 'ArrowDown') next = (idx + 1) % swatches.length;
else if (e.key === 'ArrowLeft' || e.key === 'ArrowUp') next = (idx - 1 + swatches.length) % swatches.length;
else if (e.key === 'Enter' || e.key === ' ') { assignColor(btn.getAttribute('data-color')); e.preventDefault(); return; }
if (next >= 0) { swatches[next].focus(); e.preventDefault(); }
});
// Event: custom apply
el.querySelector('.cc-picker-apply').addEventListener('click', function() {
var input = el.querySelector('.cc-picker-input');
assignColor(input.value);
});
// Event: clear
el.querySelector('.cc-picker-clear').addEventListener('click', function() {
if (currentChannel && window.ChannelColors) {
window.ChannelColors.remove(currentChannel);
refreshVisibleRows();
}
hidePopover();
});
// Event: close button
el.querySelector('.cc-picker-close').addEventListener('click', function() {
hidePopover();
});
// Prevent right-click on the popover itself
el.addEventListener('contextmenu', function(e) { e.preventDefault(); });
document.body.appendChild(el);
popoverEl = el;
return el;
}
function assignColor(color) {
if (currentChannel && window.ChannelColors) {
window.ChannelColors.set(currentChannel, color);
refreshVisibleRows();
}
hidePopover();
}
function showPopover(channel, x, y) {
var el = createPopover();
currentChannel = channel;
// Update title
el.querySelector('.cc-picker-title').textContent = channel;
// Highlight current color
var current = window.ChannelColors ? window.ChannelColors.get(channel) : null;
var swatches = el.querySelectorAll('.cc-swatch');
for (var i = 0; i < swatches.length; i++) {
swatches[i].classList.toggle('cc-swatch-active', swatches[i].getAttribute('data-color') === current);
}
if (current) {
el.querySelector('.cc-picker-input').value = current;
}
// Show/hide clear button
el.querySelector('.cc-picker-clear').style.display = current ? '' : 'none';
// Position — on touch devices, CSS handles bottom-sheet via @media(pointer:coarse)
el.style.display = '';
var isTouch = window.matchMedia('(pointer: coarse)').matches;
if (!isTouch) {
el.style.left = '0';
el.style.top = '0';
var rect = el.getBoundingClientRect();
var pw = rect.width;
var ph = rect.height;
var vw = window.innerWidth;
var vh = window.innerHeight;
var finalX = x + pw > vw ? Math.max(0, vw - pw - 8) : x;
var finalY = y + ph > vh ? Math.max(0, vh - ph - 8) : y;
el.style.left = finalX + 'px';
el.style.top = finalY + 'px';
}
// Lock background scroll while popover is open
document.body.style.overflow = 'hidden';
// Focus first swatch for keyboard accessibility
var firstSwatch = el.querySelector('.cc-swatch');
if (firstSwatch) setTimeout(function() { firstSwatch.focus(); }, 0);
// Listen for outside click / Escape
setTimeout(function() {
document.addEventListener('click', onOutsideClick, true);
document.addEventListener('keydown', onEscape, true);
}, 0);
}
function hidePopover() {
if (popoverEl) popoverEl.style.display = 'none';
currentChannel = null;
document.body.style.overflow = '';
document.removeEventListener('click', onOutsideClick, true);
document.removeEventListener('keydown', onEscape, true);
}
function onOutsideClick(e) {
if (popoverEl && !popoverEl.contains(e.target)) {
hidePopover();
}
}
function onEscape(e) {
if (e.key === 'Escape') {
hidePopover();
e.stopPropagation();
}
// Trap Tab within the popover
if (e.key === 'Tab' && popoverEl && popoverEl.style.display !== 'none') {
var focusable = popoverEl.querySelectorAll('button, input, [tabindex]');
if (focusable.length === 0) return;
var first = focusable[0];
var last = focusable[focusable.length - 1];
if (e.shiftKey && document.activeElement === first) {
last.focus(); e.preventDefault();
} else if (!e.shiftKey && document.activeElement === last) {
first.focus(); e.preventDefault();
}
}
}
/** Refresh channel color styles on all visible feed items and packet rows. */
function refreshVisibleRows() {
if (!window.ChannelColors) return;
// Live feed items
var feedItems = document.querySelectorAll('.live-feed-item');
for (var i = 0; i < feedItems.length; i++) {
var item = feedItems[i];
var ch = item._ccChannel;
if (!ch) continue;
var style = window.ChannelColors.getRowStyle('GRP_TXT', ch);
// Remove old channel color styles, reapply
item.style.borderLeft = '';
item.style.background = '';
if (style) item.style.cssText += style;
}
// Packets table — trigger re-render via custom event
document.dispatchEvent(new CustomEvent('channel-colors-changed'));
}
/**
* Extract channel name from a packet object.
* Returns null if no channel found or not a GRP_TXT/CHAN type.
*/
function extractChannel(pkt) {
if (!pkt) return null;
var d = pkt.decoded || {};
var h = d.header || {};
var p = d.payload || {};
var type = h.payloadTypeName || '';
if (type !== 'GRP_TXT' && type !== 'CHAN') return null;
return p.channelName || null;
}
/**
* Extract channel from a packets-table decoded_json.
*/
function extractChannelFromDecoded(decoded) {
if (!decoded) return null;
var type = decoded.type || '';
if (type !== 'GRP_TXT' && type !== 'CHAN') return null;
return decoded.channel || null;
}
/**
* Install context-menu (right-click) and long-press handlers on the live feed.
*/
function installLiveFeedHandlers() {
var feed = document.getElementById('liveFeed');
if (!feed) return;
feed.addEventListener('contextmenu', function(e) {
var item = e.target.closest('.live-feed-item');
if (!item || !item._ccChannel) return;
var ch = item._ccChannel;
e.preventDefault();
showPopover(ch, e.clientX, e.clientY);
});
// Long-press for mobile
var longPressTriggered = false;
feed.addEventListener('touchstart', function(e) {
var item = e.target.closest('.live-feed-item');
if (!item || !item._ccChannel) return;
var ch = item._ccChannel;
if (!ch) return;
var touch = e.touches[0];
var tx = touch.clientX;
var ty = touch.clientY;
longPressTriggered = false;
// Don't preventDefault here — it blocks scroll initiation on feed items.
// CSS -webkit-touch-callout:none + user-select:none (on .live-feed-item)
// already suppress native context menu and text selection.
longPressTimer = setTimeout(function() {
longPressTimer = null;
longPressTriggered = true;
showPopover(ch, tx, ty);
}, 500);
}, { passive: true });
feed.addEventListener('touchend', function(e) {
if (longPressTimer) { clearTimeout(longPressTimer); longPressTimer = null; }
if (longPressTriggered) { e.preventDefault(); longPressTriggered = false; }
});
feed.addEventListener('touchmove', function() {
if (longPressTimer) { clearTimeout(longPressTimer); longPressTimer = null; }
});
// Prevent context menu on long-press (some browsers fire contextmenu after touch)
feed.addEventListener('contextmenu', function(e) {
if (longPressTriggered) e.preventDefault();
});
}
/**
* Install context-menu handler on the packets table.
*/
function installPacketsTableHandlers() {
var table = document.getElementById('packetsTableBody');
if (!table) return;
table.addEventListener('contextmenu', function(e) {
var row = e.target.closest('tr');
if (!row) return;
// Try to get decoded data from the row's data attribute
var decodedStr = row.getAttribute('data-decoded');
var decoded = null;
if (decodedStr) {
try { decoded = JSON.parse(decodedStr); } catch(ex) {}
}
// Fallback: check if the row has a chan-tag
if (!decoded) {
var chanTag = row.querySelector('.chan-tag');
if (chanTag) {
var ch = chanTag.textContent.trim();
if (ch) {
e.preventDefault();
showPopover(ch, e.clientX, e.clientY);
return;
}
}
return;
}
var ch = extractChannelFromDecoded(decoded);
if (!ch) return;
e.preventDefault();
showPopover(ch, e.clientX, e.clientY);
});
}
// Export for use by live.js feed item creation
window.ChannelColorPicker = {
install: function() {
installLiveFeedHandlers();
installPacketsTableHandlers();
},
installLiveFeed: installLiveFeedHandlers,
installPacketsTable: installPacketsTableHandlers,
show: showPopover,
hide: hidePopover
};
})();
-1
View File
@@ -95,7 +95,6 @@
<script src="packet-filter.js?v=__BUST__"></script>
<script src="packet-helpers.js?v=__BUST__"></script>
<script src="channel-colors.js?v=__BUST__"></script>
<script src="channel-color-picker.js?v=__BUST__"></script>
<script src="packets.js?v=__BUST__"></script>
<script src="geo-filter-overlay.js?v=__BUST__"></script>
<script src="map.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
+3 -9
View File
@@ -754,7 +754,7 @@
<label class="audio-slider-label">Vol <input type="range" id="audioVolSlider" min="0" max="100" value="30" class="audio-slider"><span id="audioVolVal">30</span></label>
</div>
</div>
<div class="live-overlay live-feed" id="liveFeed" aria-live="polite" aria-relevant="additions" role="log">
<div class="live-overlay live-feed" id="liveFeed">
<button class="feed-hide-btn" id="feedHideBtn" title="Hide feed"></button>
</div>
<button class="feed-show-btn hidden" id="feedShowBtn" title="Show feed">📋</button>
@@ -1323,7 +1323,7 @@
let html = `
<div style="padding:16px;">
<div style="display:flex;align-items:center;gap:8px;margin-bottom:12px;">
<span class="${statusDot}" style="font-size:18px" aria-hidden="true"></span>
<span class="${statusDot}" style="font-size:18px"></span>
<h3 style="margin:0;font-size:16px;font-weight:700;">${escapeHtml(n.name || 'Unknown')}</h3>
</div>
<div style="margin-bottom:12px;">
@@ -1563,7 +1563,6 @@
<span class="feed-text">${escapeHtml(preview)}</span>
<span class="feed-time">${formatLiveTimestampHtml(group.latestTs || Date.now())}</span>
`;
var _ccD = (pkt.decoded || {}), _ccH = (_ccD.header || {}), _ccP = (_ccD.payload || {}); if (_ccH.payloadTypeName === 'GRP_TXT' || _ccH.payloadTypeName === 'CHAN') item._ccChannel = _ccP.channelName || null; // channel color picker (#271 M2)
item.addEventListener('click', () => showFeedCard(item, pkt, color));
feed.appendChild(item);
@@ -2525,7 +2524,6 @@
<span class="feed-text">${escapeHtml(preview)}</span>
<span class="feed-time">${formatLiveTimestampHtml(pkt._ts || Date.now())}</span>
`;
var _ccD = (pkt.decoded || {}), _ccH = (_ccD.header || {}), _ccP = (_ccD.payload || {}); if (_ccH.payloadTypeName === 'GRP_TXT' || _ccH.payloadTypeName === 'CHAN') item._ccChannel = _ccP.channelName || null; // channel color picker (#271 M2)
item.addEventListener('click', () => showFeedCard(item, pkt, color));
feed.appendChild(item);
}
@@ -2597,7 +2595,6 @@
<span class="feed-text">${escapeHtml(preview)}</span>
<span class="feed-time">${formatLiveTimestampHtml(pkt._ts || Date.now())}</span>
`;
var _ccD = (pkt.decoded || {}), _ccH = (_ccD.header || {}), _ccP = (_ccD.payload || {}); if (_ccH.payloadTypeName === 'GRP_TXT' || _ccH.payloadTypeName === 'CHAN') item._ccChannel = _ccP.channelName || null; // channel color picker (#271 M2)
item.addEventListener('click', () => showFeedCard(item, pkt, color));
feed.prepend(item);
requestAnimationFrame(() => requestAnimationFrame(() => item.classList.remove('live-feed-enter')));
@@ -2717,10 +2714,7 @@
if (activeNodeDetailKey) showNodeDetail(activeNodeDetailKey);
};
window.addEventListener('theme-refresh', _themeRefreshHandler);
var result = init(app, routeParam);
// Install channel color picker (M2, #271)
if (window.ChannelColorPicker) window.ChannelColorPicker.installLiveFeed();
return result;
return init(app, routeParam);
},
destroy: function() {
if (_themeRefreshHandler) { window.removeEventListener('theme-refresh', _themeRefreshHandler); _themeRefreshHandler = null; }
+2 -18
View File
@@ -204,9 +204,6 @@
: '<span class="text-muted">—</span>';
var scoreTitle = 'Observations: ' + nb.count;
if (nb.avg_snr != null) scoreTitle += ' · Avg SNR: ' + Number(nb.avg_snr).toFixed(1) + ' dB';
var distanceCell = nb.distance_km != null
? Number(nb.distance_km).toFixed(1) + ' km'
: '<span class="text-muted">—</span>';
var showOnMap = nb.pubkey
? ' <button class="btn-link neighbor-show-map" data-pubkey="' + escapeHtml(nb.pubkey) + '" style="font-size:11px;padding:1px 6px;white-space:nowrap">📍 Map</button>'
: '';
@@ -216,7 +213,6 @@
'<td title="' + escapeHtml(scoreTitle) + '">' + Number(nb.score).toFixed(2) + '</td>' +
'<td>' + nb.count + '</td>' +
'<td>' + renderNodeTimestampHtml(nb.last_seen) + '</td>' +
'<td>' + distanceCell + '</td>' +
'<td><span title="' + conf.label + '">' + conf.icon + '</span></td>' +
'<td style="text-align:right">' + showOnMap + '</td>' +
'</tr>';
@@ -225,7 +221,7 @@
function renderNeighborTable(neighbors, limit) {
return '<table class="data-table" style="font-size:12px">' +
'<thead><tr><th>Neighbor</th><th>Role</th><th>Score</th><th>Obs</th><th>Last Seen</th><th>Distance</th><th>Conf</th><th></th></tr></thead>' +
'<thead><tr><th>Neighbor</th><th>Role</th><th>Score</th><th>Obs</th><th>Last Seen</th><th>Conf</th><th></th></tr></thead>' +
'<tbody>' + renderNeighborRows(neighbors, limit) + '</tbody></table>';
}
@@ -324,7 +320,7 @@
</div>
<div id="nodesRegionFilter" class="region-filter-container"></div>
<div class="split-layout">
<div class="panel-left" id="nodesLeft" aria-live="polite" aria-relevant="additions removals"></div>
<div class="panel-left" id="nodesLeft"></div>
<div class="panel-right empty" id="nodesRight"><span>Select a node to view details</span></div>
</div>
</div>`;
@@ -920,17 +916,6 @@
}
});
// #630: Close button for node detail panel (important for mobile full-screen overlay)
document.getElementById('nodesRight').addEventListener('click', function(e) {
if (e.target.closest('.panel-close-btn')) {
const panel = document.getElementById('nodesRight');
panel.classList.add('empty');
panel.innerHTML = '<span>Select a node to view details</span>';
selectedKey = null;
renderRows();
}
});
renderRows();
}
@@ -1018,7 +1003,6 @@
const dupBadge = dupNameBadge(n.name, n.public_key, dupMap);
panel.innerHTML = `
<button class="panel-close-btn" title="Close detail pane (Esc)"></button>
<div class="node-detail">
<div class="node-detail-name">${escapeHtml(n.name || '(unnamed)')}${dupBadge}</div>
<div class="node-detail-role">${renderNodeBadges(n, roleColor)}
+5 -21
View File
@@ -63,9 +63,7 @@
const getParsedDecoded = window.getParsedDecoded;
// --- Virtual scroll state ---
let VSCROLL_ROW_HEIGHT = 36; // measured dynamically on first render; fallback 36px
let _vscrollRowHeightMeasured = false;
let _vscrollTheadHeight = 40; // measured dynamically on first render; fallback 40px
const VSCROLL_ROW_HEIGHT = 36; // estimated row height in px
const VSCROLL_BUFFER = 30; // extra rows above/below viewport
let _displayPackets = []; // filtered packets for current view
let _displayGrouped = false; // whether _displayPackets is in grouped mode
@@ -283,7 +281,7 @@
}
}
app.innerHTML = `<div class="split-layout detail-collapsed">
<div class="panel-left" id="pktLeft" aria-live="polite" aria-relevant="additions removals"></div>
<div class="panel-left" id="pktLeft"></div>
<div class="panel-right empty" id="pktRight" aria-live="polite">
<div class="panel-resize-handle" id="pktResizeHandle"></div>
${PANEL_CLOSE_HTML}
@@ -1279,10 +1277,8 @@
// Calculate visible range based on scroll position
const scrollTop = scrollContainer.scrollTop;
const viewportHeight = scrollContainer.clientHeight;
// Account for thead height (measured dynamically)
const theadEl = scrollContainer.querySelector('thead');
if (theadEl) _vscrollTheadHeight = theadEl.offsetHeight || _vscrollTheadHeight;
const theadHeight = _vscrollTheadHeight;
// Account for thead height (~40px)
const theadHeight = 40;
const adjustedScrollTop = Math.max(0, scrollTop - theadHeight);
// Find the first entry whose cumulative row offset covers the scroll position
@@ -1340,14 +1336,6 @@
tbody.appendChild(topSpacer);
tbody.insertAdjacentHTML('beforeend', visibleHtml);
tbody.appendChild(bottomSpacer);
// Measure actual row height from first rendered data row (#407)
if (!_vscrollRowHeightMeasured) {
const firstRow = topSpacer.nextElementSibling;
if (firstRow && firstRow !== bottomSpacer) {
const h = firstRow.offsetHeight;
if (h > 0) { VSCROLL_ROW_HEIGHT = h; _vscrollRowHeightMeasured = true; }
}
}
if (window.__PERF_LOG_RENDER) console.log('[perf] renderVisibleRows: full rebuild %d entries, %.2fms', endIdx - startIdx, performance.now() - _rvr_t0);
return;
}
@@ -2179,10 +2167,7 @@
init: function(app, routeParam) {
_themeRefreshHandler = () => { if (typeof renderTableRows === 'function') renderTableRows(); };
window.addEventListener('theme-refresh', _themeRefreshHandler);
var result = init(app, routeParam);
// Install channel color picker on packets table (M2, #271)
if (window.ChannelColorPicker) window.ChannelColorPicker.installPacketsTable();
return result;
return init(app, routeParam);
},
destroy: function() {
if (_themeRefreshHandler) { window.removeEventListener('theme-refresh', _themeRefreshHandler); _themeRefreshHandler = null; }
@@ -2193,7 +2178,6 @@
// Standalone packet detail page: #/packet/123 or #/packet/HASH
// Expose pure functions for unit testing (vm.createContext pattern)
if (typeof window !== 'undefined') {
document.addEventListener('channel-colors-changed', function() { renderVisibleRows(); });
window._packetsTestAPI = {
typeName,
obsName,
-202
View File
@@ -2037,205 +2037,3 @@ tr[data-hops]:hover { background: rgba(59,130,246,0.1); }
.rf-time-selector { gap: 3px; }
.rf-custom-inputs { margin-left: 0; margin-top: 4px; flex-wrap: wrap; }
}
/* Channel Color Picker Popover (M2, #271) */
.cc-picker-popover {
position: fixed;
z-index: 10000;
background: var(--surface-1, #1e1e2e);
border: 1px solid var(--border, #444);
border-radius: 8px;
padding: 10px;
min-width: 200px;
max-width: 260px;
box-shadow: 0 4px 16px rgba(0,0,0,0.4);
font-size: 13px;
}
.cc-picker-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 8px;
}
.cc-picker-title {
font-weight: 600;
color: var(--text-primary, #e0e0e0);
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.cc-picker-close {
background: none;
border: none;
color: var(--muted, #888);
cursor: pointer;
font-size: 14px;
padding: 2px 4px;
}
.cc-picker-close:hover { color: var(--text-primary, #e0e0e0); }
.cc-picker-swatches {
display: flex;
flex-wrap: wrap;
gap: 6px;
margin-bottom: 8px;
}
.cc-swatch {
width: 24px;
height: 24px;
border-radius: 4px;
border: 2px solid transparent;
cursor: pointer;
padding: 0;
transition: border-color 0.15s;
}
.cc-swatch:hover { border-color: var(--text-primary, #e0e0e0); }
.cc-swatch:focus-visible { border-color: #fff; outline: 2px solid var(--accent, #3b82f6); outline-offset: 1px; }
.cc-swatch-active { border-color: #fff; box-shadow: 0 0 0 1px rgba(255,255,255,0.5); }
/* Mobile: larger touch targets, hide native color picker, safe areas */
@media (pointer: coarse) {
.cc-swatch {
width: 40px;
height: 40px;
border-radius: 6px;
}
.cc-picker-swatches {
gap: 8px;
}
.cc-picker-custom {
display: none !important;
}
.cc-picker-popover {
position: fixed !important;
left: 0 !important;
right: 0 !important;
bottom: 0 !important;
top: auto !important;
width: 100% !important;
max-width: 100% !important;
border-radius: 16px 16px 0 0;
padding: 16px;
padding-bottom: calc(16px + env(safe-area-inset-bottom));
box-sizing: border-box;
}
.live-feed-item {
-webkit-touch-callout: none;
user-select: none;
}
}
.cc-picker-custom {
display: flex;
align-items: center;
gap: 6px;
margin-bottom: 8px;
}
.cc-picker-custom label {
color: var(--muted, #888);
display: flex;
align-items: center;
gap: 4px;
font-size: 12px;
}
.cc-picker-input {
width: 32px;
height: 24px;
border: none;
padding: 0;
cursor: pointer;
background: none;
}
.cc-picker-apply {
background: var(--accent, #3b82f6);
color: #fff;
border: none;
border-radius: 4px;
padding: 3px 8px;
cursor: pointer;
font-size: 12px;
}
.cc-picker-apply:hover { opacity: 0.85; }
.cc-picker-clear {
background: none;
border: 1px solid var(--border, #444);
color: var(--muted, #888);
border-radius: 4px;
padding: 4px 8px;
cursor: pointer;
font-size: 12px;
width: 100%;
}
.cc-picker-clear:hover { color: var(--text-primary, #e0e0e0); border-color: var(--text-primary, #e0e0e0); }
/* === #630 — Mobile Accessibility Fixes === */
/* #630-1: Touch targets — minimum 44px on touch devices */
@media (pointer: coarse) {
.filter-bar .btn,
.filter-group .btn,
.tab-btn,
.filter-bar input,
.filter-bar select,
.nav-btn,
.region-pill,
.region-dropdown-trigger,
.multi-select-trigger,
.node-count-pill,
.analytics-time-range button,
.detail-back-btn,
.filter-toggle-btn {
min-height: 44px;
min-width: 44px;
}
.filter-bar input,
.filter-bar select {
height: 44px;
}
.region-dropdown-trigger,
.multi-select-trigger {
height: 44px;
}
}
/* #630-3: Status text labels — visually hidden text for screen readers */
.sr-status-label { font-size: 11px; margin-left: 4px; }
/* #630-4: Detail panel as full-width overlay on mobile */
@media (max-width: 640px) {
.split-layout .panel-right:not(.empty) {
display: block;
position: fixed;
top: 52px;
left: 0;
right: 0;
bottom: 0;
width: 100%;
min-width: 0;
z-index: 150;
overflow-y: auto;
-webkit-overflow-scrolling: touch;
}
}
/* #630-5: Analytics tabs — horizontal scroll on small screens */
@media (max-width: 640px) {
.analytics-tabs {
flex-wrap: nowrap;
overflow-x: auto;
-webkit-overflow-scrolling: touch;
scrollbar-width: thin;
padding-bottom: 4px;
}
.analytics-tabs .tab-btn {
flex-shrink: 0;
white-space: nowrap;
}
}
/* #630-6: Tables — horizontal scroll wrapper */
.table-scroll-wrap {
overflow-x: auto;
-webkit-overflow-scrolling: touch;
}
@media (max-width: 640px) {
.data-table { min-width: 480px; }
}
-85
View File
@@ -168,91 +168,6 @@ test('getRowStyle returns empty when channel has no assigned color', function()
assert.strictEqual(ctx.window.ChannelColors.getRowStyle('GRP_TXT', '#nope'), '');
});
// ── M2: Channel Color Picker tests ──
test('channel-color-picker.js loads without error in sandbox', function() {
const ctx = makeSandbox();
// Provide minimal DOM stubs for the picker
const elements = {};
const createdEls = [];
ctx.document = {
createElement: function(tag) {
var el = {
tagName: tag.toUpperCase(),
className: '', style: { cssText: '', display: '' },
innerHTML: '', textContent: '', title: '',
children: [],
_attrs: {},
_listeners: {},
setAttribute: function(k, v) { this._attrs[k] = v; },
getAttribute: function(k) { return this._attrs[k] || null; },
addEventListener: function(ev, fn, opts) { this._listeners[ev] = fn; },
removeEventListener: function() {},
appendChild: function(c) { this.children.push(c); return c; },
querySelector: function(sel) {
// Very basic selector matching for test
if (sel === '.cc-picker-swatches') return { addEventListener: function(){}, appendChild: function(c){} };
if (sel === '.cc-picker-apply') return { addEventListener: function(){} };
if (sel === '.cc-picker-clear') return { addEventListener: function(){}, style: {} };
if (sel === '.cc-picker-close') return { addEventListener: function(){} };
if (sel === '.cc-picker-title') return { textContent: '' };
if (sel === '.cc-picker-input') return { value: '#000000' };
return null;
},
querySelectorAll: function() { return []; },
classList: { toggle: function(){}, remove: function(){}, add: function(){} },
contains: function() { return false; },
closest: function() { return null; },
getBoundingClientRect: function() { return { width: 200, height: 200 }; }
};
createdEls.push(el);
return el;
},
getElementById: function() { return null; },
addEventListener: function() {},
removeEventListener: function() {},
body: { appendChild: function(c) {} },
querySelectorAll: function() { return []; }
};
ctx.setTimeout = function(fn) { fn(); };
ctx.window.innerWidth = 1024;
ctx.window.innerHeight = 768;
const pickerSrc = fs.readFileSync(__dirname + '/public/channel-color-picker.js', 'utf8');
vm.runInContext(pickerSrc, ctx);
assert.ok(ctx.window.ChannelColorPicker, 'ChannelColorPicker should be exported');
assert.strictEqual(typeof ctx.window.ChannelColorPicker.install, 'function');
assert.strictEqual(typeof ctx.window.ChannelColorPicker.show, 'function');
assert.strictEqual(typeof ctx.window.ChannelColorPicker.hide, 'function');
});
test('ChannelColorPicker.install does not throw when elements missing', function() {
const ctx = makeSandbox();
ctx.document = {
createElement: function() {
return { className: '', style: {}, innerHTML: '', _attrs: {}, children: [],
setAttribute: function(){}, getAttribute: function(){ return null; },
addEventListener: function(){}, appendChild: function(c){ this.children.push(c); return c; },
querySelector: function(){ return { addEventListener: function(){}, style: {}, textContent: '' }; },
querySelectorAll: function(){ return []; },
getBoundingClientRect: function(){ return {width:0,height:0}; },
contains: function(){ return false; }
};
},
getElementById: function() { return null; },
addEventListener: function() {},
removeEventListener: function() {},
body: { appendChild: function(){} },
querySelectorAll: function() { return []; }
};
ctx.setTimeout = function(fn) { fn(); };
ctx.window.innerWidth = 1024;
ctx.window.innerHeight = 768;
const pickerSrc = fs.readFileSync(__dirname + '/public/channel-color-picker.js', 'utf8');
vm.runInContext(pickerSrc, ctx);
// Should not throw when feed/table elements don't exist
ctx.window.ChannelColorPicker.install();
});
// Summary
console.log(`\n${passed} passed, ${failed} failed\n`);
process.exit(failed ? 1 : 0);