diff --git a/README.md b/README.md index a237671c..1b949400 100644 --- a/README.md +++ b/README.md @@ -254,6 +254,8 @@ Contributions welcome. Please read [AGENTS.md](AGENTS.md) for coding conventions **Live instance:** [analyzer.00id.net](https://analyzer.00id.net) — all API endpoints are public, no auth required. +**API Documentation:** CoreScope auto-generates an OpenAPI 3.0 spec. Browse the interactive Swagger UI at [`/api/docs`](https://analyzer.00id.net/api/docs) or fetch the machine-readable spec at [`/api/spec`](https://analyzer.00id.net/api/spec). + ## License MIT diff --git a/docs/deployment.md b/docs/deployment.md index ff72b507..924bbd89 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -261,6 +261,38 @@ Caddy handles certificate issuance and renewal automatically. --- +## API Documentation + +CoreScope auto-generates an OpenAPI 3.0 specification from its route definitions. The spec is always in sync with the running server — no manual maintenance required. + +### Endpoints + +| URL | Description | +|-----|-------------| +| `/api/spec` | OpenAPI 3.0 JSON schema — machine-readable API definition | +| `/api/docs` | Interactive Swagger UI — browse and test all 40+ endpoints | + +### Usage + +**Browse the API interactively:** +``` +http://your-instance/api/docs +``` + +**Fetch the spec programmatically:** +```bash +curl http://your-instance/api/spec | jq . +``` + +**For bot/integration developers:** The spec includes all request parameters, response schemas, and example values. Import it into Postman, Insomnia, or any OpenAPI-compatible tool. + +### Public instance +The live instance at [analyzer.00id.net](https://analyzer.00id.net) has all API endpoints publicly accessible: +- Spec: [analyzer.00id.net/api/spec](https://analyzer.00id.net/api/spec) +- Docs: [analyzer.00id.net/api/docs](https://analyzer.00id.net/api/docs) + +--- + ## Monitoring & Health Checks ### Docker health check diff --git a/docs/release-notes/v3.5.0.md b/docs/release-notes/v3.5.0.md new file mode 100644 index 00000000..71903481 --- /dev/null +++ b/docs/release-notes/v3.5.0.md @@ -0,0 +1,248 @@ +# CoreScope v3.5.0 Release Notes + +**95 commits** since v3.4.1 — the biggest release yet. + +--- + +## 🐳 Pre-built Docker Images — One-Line Deploy + +CoreScope now publishes pre-built Docker images to GitHub Container Registry. **No more building from source.** + +### Quick start +```bash +docker run -d --name corescope \ + -p 80:80 -p 1883:1883 \ + -v corescope-data:/app/data \ + ghcr.io/kpa-clawbot/corescope:latest +``` + +### Transitioning from source builds + +If you're currently building your own image: + +1. **Stop your existing container** — `docker stop corescope && docker rm corescope` +2. **Keep your data volume** — your SQLite DB and config.json are preserved in the bind mount +3. **Pull the pre-built image** — `docker pull ghcr.io/kpa-clawbot/corescope:latest` +4. **Run with your existing data** — mount the same volume: + ```bash + docker run -d --name corescope \ + -p 80:80 -p 1883:1883 \ + -v /path/to/your/data:/app/data \ + ghcr.io/kpa-clawbot/corescope:latest + ``` + +### Image tags +- `ghcr.io/kpa-clawbot/corescope:latest` — latest stable release +- `ghcr.io/kpa-clawbot/corescope:vX.Y.Z` — pinned version (recommended for production) +- `ghcr.io/kpa-clawbot/corescope:edge` — tip of master (unstable, for testing) + +### Zero-config defaults +No `config.json` required. The server starts with sensible defaults: HTTP on port 80 (via Caddy), internal MQTT broker on 1883, SQLite in `/app/data/`. Drop a `config.json` into your data volume to customize. + +### New environment variables +- `DISABLE_CADDY=true` — skip the Caddy reverse proxy (if you bring your own) +- `DISABLE_MOSQUITTO=true` — skip the internal MQTT broker (if using an external one) + +--- + +## ⚡ Performance + +**35 performance optimizations** — CoreScope now serves HTTP within 2 minutes on any database size. + +### Headlines +- **Packets page**: p50 83% faster, p95 97% faster +- **Topology analytics**: p50 83% faster +- **Async backfill**: resolved_path backfill runs in the background — server is responsive during startup even with millions of observations +- **Memory**: prefix map capped at 8 chars (10x memory reduction) + +### Backend +- Incremental max ID tracking — no more full table scans on ingest +- Batched eviction index cleanup +- Combined filter passes (single scan instead of chained) +- Single sort for RF analytics arrays +- Early time filter in node analytics +- In-memory hop resolution (eliminates N+1 DB queries) +- Incremental distance index updates +- Region observer caching (30s TTL) +- Store stats: 5→2 concurrent queries +- Subpath detail index instead of full packet scan +- Grouped packets: cached observer count, deferred map construction + +### Frontend +- Coalesced WebSocket renders with requestAnimationFrame +- Replaced N+1 API calls with bulk `expand=observations` query +- Parallelized `replayRecent()` fetches +- Map markers repositioned on zoom instead of full rebuild +- Network status computed client-side +- Combined 4 subpath API calls into single bulk endpoint +- Batched observation fetching eliminates N+1 on sort +- Timeline updates skipped when tab is hidden +- Incremental DOM diff in virtual scroll +- Lazy observation maps in `txToMap` +- og-image compressed from 1.1MB to 235KB + +--- + +## 🔬 RF Health Dashboard + +New **RF Health** tab on the Analytics page with comprehensive observer radio metrics: + +- **Small multiples grid** — one card per observer showing key metrics at a glance +- **Noise floor column chart** — color-coded bars (green/yellow/red by threshold) replacing the old line chart. Hover tooltips show exact dBm + timestamp +- **Airtime charts** — TX/RX airtime with delta computation +- **Error rate tracking** — receive errors per observer over time +- **Battery charts** — observer battery level monitoring +- **Region filter** — filter RF metrics by geographic region +- **Observer detail** — click any observer to see detailed metrics in a side panel + +--- + +## 🗺️ Live Map Enhancements + +### TRACE packet real path visualization +Trace packets now show where the trace **actually reached** vs the full intended route: +- **Solid animated line** for completed hops +- **Dashed/ghosted line** (25% opacity) for unreached hops +- Helps operators see exactly where traces fail + +### Movable panels +- Overlay panels (feed, legend, node detail) have corner-position toggle buttons +- Non-scrolling panel headers keep controls accessible +- Panels remember their position in localStorage + +### Collapsible panels +- All live map panels can be collapsed to just their header bar +- Medium breakpoint responsive layout + +--- + +## 📊 Analytics & Data + +### Hash Stats accuracy +- "By Repeaters" section now correctly filters to repeater-role nodes only (was including companions/rooms) +- Hash size display correctly shows "Unknown" for zero-hop direct adverts instead of misleading "1 byte" + +### Prefix Tool +- New tab on the Analytics page showing prefix collision data +- Text consistency improvements + +### Paths through node accuracy +- Fixed prefix collision bug where packets through a different node with the same 2-char prefix were incorrectly shown +- Now uses `resolved_path` (neighbor affinity graph) for accurate disambiguation + +### Table sorting +- Sortable column headers on nodes list, neighbor tables, observer tables, and packets table +- Sort indicators (▲/▼) show active sort column and direction +- Stable sort with timestamp tiebreaker + +--- + +## 🔗 Deep Linking + +- **Nodes page**: URL updates with node pubkey, tab, and search state +- **Packets page**: URL includes time window and region filter +- **Channels page**: URL reflects selected channel and node +- **Analytics page**: URL includes active tab + +*Note: Packet filter expressions and desktop node selection deep links are coming in a follow-up.* + +--- + +## 🎨 UI/UX + +### Distance unit preference +- Choose between km, mi, or auto (locale-based) in the Customizer → Display tab +- Small imperial distances show in feet (not "0.0 mi") +- Analytics, node neighbors, and all distance displays respect the setting + +### Channel color highlighting +- Assign colors to channels for visual distinction in live feed and packets table +- 8-color palette, left-border encoding +- Color dots on channels page and live feed items for easy access +- Persists in localStorage + +### Mobile accessibility +- Touch targets minimum 44×44px +- ARIA labels on interactive elements +- Small viewport support improvements + +### Map byte-size filter +- Filter map markers by packet size + +### OpenAPI / Swagger +- Auto-generated OpenAPI 3.0 spec at **`/api/spec`** — returns the full API schema as JSON +- Interactive Swagger UI at **`/api/docs`** — browse and test all endpoints from your browser +- Spec is generated from route definitions, always in sync with the running server +- Useful for bot developers, integrators, and anyone building on top of CoreScope +- Example: `curl https://your-instance/api/spec | jq .info` + +--- + +## 🔒 Security + +### API key enforcement +- Weak/default API keys (e.g., "test", "admin", "password") are rejected at startup +- Write endpoints blocked when no API key is configured +- Startup warning when API key is missing + +### License +- Changed to GPL v3 + +--- + +## 🐛 Bug Fixes + +- Fixed `originLat` out-of-scope ReferenceError killing live map animations +- Fixed zero-hop direct adverts showing incorrect hash size +- Fixed live map animations stuck on any advert (block-scoping bug) +- Fixed hash stats including non-repeater nodes +- Fixed paths-through-node showing unrelated packets (prefix collision) +- Fixed channel color picker completely non-functional (data shape mismatch) +- Fixed panel corner toggle buttons invisible/scrolling away +- Fixed table sorting broken on all node tables (attribute mismatch) +- Fixed `time.Tick` leak in prune goroutine +- Fixed View Route on Map button on packet detail page +- Fixed virtual scroll row height measurement +- Fixed collision cache not respecting TTL config +- Fixed prefix map unbounded memory growth (capped at 8 chars) +- Fixed memory eviction using inaccurate memory estimates (now uses runtime heap stats) + +--- + +## 🏗️ Infrastructure + +- CI pipeline consolidated: build → test → publish to GHCR → deploy staging +- Staging compose: standard ports (80/1883), no memory limit +- CI build time reduced from ~14 min to ~3-4 min (removed arm64 QEMU cross-compilation) + +--- + +## 📖 Documentation + +- Deployment guide with Docker quick start, compose examples, and configuration reference +- User guide: getting started, page descriptions, config options, FAQ +- OpenAPI 3.0 spec (auto-generated from route definitions) +- Security analysis: AES-128-ECB block-level vulnerability assessment +- Lock ordering documentation for concurrent access patterns + +--- + +## Upgrading + +### From v3.4.1 + +1. Pull the new image: `docker pull ghcr.io/kpa-clawbot/corescope:v3.5.0` +2. Stop and remove old container: `docker stop corescope && docker rm corescope` +3. Start with the new image (keep your data volume): + ```bash + docker run -d --name corescope \ + -p 80:80 -p 1883:1883 \ + -v /your/data:/app/data \ + ghcr.io/kpa-clawbot/corescope:v3.5.0 + ``` + +**First startup note:** The server will backfill `resolved_path` data for existing observations in the background. The HTTP server is available within ~2 minutes. Full backfill may take longer on large databases but does not block serving. + +### Breaking changes +- None. All API endpoints are backward compatible. +- `resolved_path` is added to API responses (new field, `omitempty` — won't break existing clients). diff --git a/docs/test-plans/v3.4.2-manual-validation.md b/docs/test-plans/v3.4.2-manual-validation.md new file mode 100644 index 00000000..74dacb15 --- /dev/null +++ b/docs/test-plans/v3.4.2-manual-validation.md @@ -0,0 +1,162 @@ +# v3.4.2 Manual Validation Checklist + +**Tester:** _______________ +**Staging:** http://20.109.157.39 +**Prod:** https://analyzer.00id.net (READ ONLY — do not deploy until staging passes) +**Browser:** Chrome + Firefox + Safari (mobile for responsive items) +**Time estimate:** ~45 minutes + +--- + +## 🔴 HIGH RISK — Test First + +### 1. Zero-hop hash size display (#649, #653) +- [ ] Go to Packets page, find a DIRECT advert (route_type=2, 0 hops) +- [ ] Open packet detail — hash size should say "Unknown (zero-hop)" or be hidden, NOT "1 byte" +- [ ] Check "Path Length" field shows `hash_count=0 (direct advert)` +- [ ] Find a FLOOD advert with 0 hops — it SHOULD show hash size (this is different from DIRECT) + +### 2. TRACE packet real path (#651, #656) +- [ ] Send a trace from your companion +- [ ] Watch Live map — the animated dot should only travel along completed hops (solid line) +- [ ] Unreached hops should show as dashed/ghosted line at reduced opacity +- [ ] If trace completes fully, entire path should be solid +- [ ] Ghost line should auto-clean after ~10 seconds + +### 3. "Paths through this node" accuracy (#655, #658) +- [ ] Go to: http://20.109.157.39/#/nodes/c0dedad4208acb6cbe44b848943fc6d3c5d43cf38a21e48b43826a70862980e4 +- [ ] Check "Packets through this node" — packets should actually have this node in their path +- [ ] Compare with a node that shares a 2-char prefix (e.g. C0ffee SF) — they should show DIFFERENT packets +- [ ] Spot-check 3-4 packets: click through, verify path contains the node + +### 4. Hash Stats "By Repeaters" (#652, #654) +- [ ] Go to Analytics → Hash Stats +- [ ] "By Repeaters" section should only show repeater-role nodes +- [ ] Compare count in "Multi-Byte Hash Adopters" vs "By Repeaters" — adopters may include companions, repeaters section should not +- [ ] Check that companions/rooms/sensors are excluded from the repeater distribution + +### 5. Noise floor column chart (#600, #659) +- [ ] Go to Analytics → RF Health +- [ ] Noise floor chart should show vertical color-coded bars, NOT a line +- [ ] Green bars (< -100 dBm), yellow (-100 to -85), red (≥ -85) +- [ ] Hover over a bar — tooltip should show exact dBm + timestamp +- [ ] Check with only 1 observer selected — chart should still render (division by zero edge case) +- [ ] Reboot markers (if any) should show as vertical dashed lines + +### 6. Async backfill on startup +- [ ] SSH to staging: `ssh -i ~/.ssh/id_ed25519 runner@20.109.157.39` +- [ ] `docker restart corescope-staging-go` +- [ ] Within 30 seconds, hit `curl http://localhost:82/api/stats` — should return data (not hang) +- [ ] Check `backfilling` and `backfillProgress` fields in stats response +- [ ] Server should be serving HTTP while backfill runs in background + +--- + +## 🟡 MEDIUM RISK — Features + +### 7. Distance unit preference (#621, #646) +- [ ] Go to Customizer → Display tab +- [ ] Change distance unit to "mi" — all distances should show in miles +- [ ] Change to "km" — all distances should show in km +- [ ] Change to "auto" — should use locale (US = miles, EU = km) +- [ ] Check Analytics page distances update after customizer change (no page reload needed) +- [ ] Check Node detail → Neighbors table distances +- [ ] Very small distances (<0.1 mi) should show in feet, not "0.0 mi" + +### 8. Panel corner toggle (#608, #657) +- [ ] Go to Live map page +- [ ] Each panel (feed, legend, node detail) should have a small corner-toggle button +- [ ] Click the button — panel should snap to next corner (TL → TR → BR → BL) +- [ ] Refresh page — panel positions should persist (localStorage) +- [ ] Move two panels to same corner — collision avoidance should skip to next free corner +- [ ] On mobile viewport (<768px?) — toggle buttons should be hidden + +### 9. Deep linking (#536, #618) +- [ ] Navigate to Nodes page, click a node → URL should update with pubkey hash +- [ ] Copy URL, open in new tab → should land on same node +- [ ] Apply packet filters → URL hash should include filter params +- [ ] Channels page: select a node → URL should reflect selection +- [ ] Analytics tabs: switch tabs → URL should include tab name +- [ ] Share a deep link with someone — they should see the same view + +### 10. Sortable tables (#620, #638, #639) +- [ ] Nodes list: click column headers — should sort ascending/descending +- [ ] Sort indicator (arrow) should be visible on active column +- [ ] Node detail → Neighbors table: sortable +- [ ] Node detail → Observers table: sortable +- [ ] Packets table: sortable by column headers + +### 11. Channel color highlighting (#271, #607, #611) +- [ ] Go to Channels page +- [ ] Assign a color to a channel using the color picker +- [ ] Feed rows should highlight with that color +- [ ] Change color — should update immediately +- [ ] Refresh — color assignment should persist + +### 12. Collapsible panels (#606) +- [ ] Live map: panels should have collapse/expand toggle +- [ ] Collapsed panel should show just the header +- [ ] State should persist across page navigations + +### 13. Mobile accessibility (#630, #633) +- [ ] Open staging on phone (or Chrome DevTools mobile emulation) +- [ ] Touch targets should be at least 44×44px +- [ ] Channel color picker should work on mobile +- [ ] No horizontal scroll on any page +- [ ] ARIA labels present on interactive elements (inspect with accessibility tools) + +### 14. Map byte-size filter (#565, #568) +- [ ] Go to Map page +- [ ] Find the byte-size filter control +- [ ] Filter by packet size — map should update to show only matching packets +- [ ] Clear filter — all packets should return + +### 15. API key security (#532, #628) +- [ ] Try accessing a write endpoint without API key — should be blocked +- [ ] Try with a weak key (e.g., "test", "admin") — should be rejected at startup +- [ ] Check staging logs for API key warning: `docker logs corescope-staging-go 2>&1 | grep -i "apiKey\|api_key\|security"` + +### 16. OpenAPI/Swagger (#530, #632) +- [ ] Hit http://20.109.157.39/api/spec — should return valid OpenAPI 3.0 spec +- [ ] Hit http://20.109.157.39/api/docs — should show Swagger UI +- [ ] Try an endpoint from Swagger UI — should work + +--- + +## 🟢 LOW RISK — Verify Quickly + +### 17. View Route on Map button +- [ ] Go to any packet detail page +- [ ] Click "View Route on Map" — should navigate to map with route highlighted + +### 18. og-image compression +- [ ] Check page source or network tab — og-image.png should be < 300KB (was 1.1MB) + +### 19. Prefix Tool +- [ ] Analytics → Prefix Tool tab should load +- [ ] Should show collision data + +### 20. License +- [ ] Check repo footer/LICENSE — should be GPL v3 + +### 21. Docker DISABLE_CADDY +- [ ] (If testable) Set DISABLE_CADDY=true — Caddy should not start + +### 22. Region filter on RF Health +- [ ] RF Health tab: change region filter — charts should update + +--- + +## 🏁 Sign-off + +| Section | Status | Notes | +|---------|--------|-------| +| High risk (1-6) | ☐ | | +| Medium risk (7-16) | ☐ | | +| Low risk (17-22) | ☐ | | +| **Overall** | ☐ | | + +**Tested by:** _______________ +**Date:** _______________ +**Staging version:** `curl -s http://20.109.157.39/api/stats | jq .version` +**Ready for release:** ☐ Yes / ☐ No — blockers: _______________ diff --git a/docs/test-plans/v3.4.2-release-test-plan.md b/docs/test-plans/v3.4.2-release-test-plan.md new file mode 100644 index 00000000..a60013dd --- /dev/null +++ b/docs/test-plans/v3.4.2-release-test-plan.md @@ -0,0 +1,309 @@ +# v3.4.2 Release Test Plan + +**Scope:** 90 commits since v3.4.1 (84 files, +14,931 / -1,005) +**Categories:** 19 perf, 19 feat, 18 fix, 15 docs, 3 chore, 1 test, 1 refactor, 1 ci +**Date:** 2026-04-08 + +--- + +## A. Automated Tests — Verify All Pass + +### Go Backend +```bash +cd cmd/server && go test -race -count=1 ./... +cd cmd/ingestor && go test -race -count=1 ./... +``` + +**Test files (27 total):** + +| File | Tests For | +|------|-----------| +| `cmd/server/decoder_test.go` | Hash size zero-hop, TRACE hopsCompleted, transport direct | +| `cmd/server/backfill_async_test.go` | **NEW** — Async chunked backfill | +| `cmd/server/eviction_test.go` | Memory eviction with runtime heap stats | +| `cmd/server/apikey_security_test.go` | **NEW** — Weak/default API key rejection | +| `cmd/server/openapi_test.go` | **NEW** — OpenAPI spec generation | +| `cmd/server/routes_test.go` | Batch observations endpoint, subpaths-bulk, expand=observations | +| `cmd/server/cache_invalidation_test.go` | cacheTTL config wiring | +| `cmd/server/config_knobs_test.go` | cacheTTLSec helper | +| `cmd/server/helpers_test.go` | constantTimeEqual, IsWeakAPIKey | +| `cmd/server/obs_dedup_test.go` | UniqueObserverCount tracking | +| `cmd/server/neighbor_*.go` (4 files) | Neighbor graph, affinity, persistence | +| `cmd/server/perfstats_race_test.go` | Perf stats concurrency | +| `cmd/server/resolve_context_test.go` | Resolved path filtering | +| `cmd/server/advert_pubkey_test.go` | Advert pubkey tracking | +| `cmd/server/db_test.go` | SQLite operations | +| `cmd/server/config_test.go` | Config loading | +| `cmd/server/coverage_test.go` | Coverage helpers | +| `cmd/server/parity_test.go` | Go/JS decoder parity | +| `cmd/server/websocket_test.go` | WebSocket broadcast | +| `cmd/ingestor/decoder_test.go` | Ingestor decoder (hash size zero-hop) | +| `cmd/ingestor/db_test.go` | Ingestor DB writes | +| `cmd/ingestor/config_test.go` | Ingestor config | +| `cmd/ingestor/main_test.go` | Ingestor entry | +| `cmd/ingestor/coverage_boost_test.go` | Coverage helpers | + +### Frontend Unit Tests +```bash +node test-packet-filter.js +node test-aging.js +node test-frontend-helpers.js +node test-table-sort.js # NEW — shared table sort utility +node test-channel-colors.js # NEW — channel color model +node test-panel-corner.js # NEW — panel corner toggle +node test-packets.js # NEW — packets page logic +node test-hop-resolver-affinity.js +node test-customizer-v2.js +node test-live.js +node test-live-dedup.js +``` + +### E2E / Playwright +```bash +BASE_URL=http://localhost:13581 node test-e2e-playwright.js +``` + +**Expected:** All existing tests pass + new tests added for sortable tables, deep linking, collapsible panels. + +--- + +## B. Manual Browser Verification + +### B1. HIGH RISK — Data Correctness + +| # | Feature | Page | What to Check | +|---|---------|------|---------------| +| 1 | Hash size zero-hop | Packets detail | Find a direct (route_type=0) packet → hash_size should show 0, not a bogus computed value | +| 2 | TRACE hopsCompleted | Packets detail / Live map | Find a TRACE packet → verify `hopsCompleted` shows in decoded JSON, live map shows real path length vs intended | +| 3 | Transport direct hash size | Packets detail | Find route_type=RouteTransportDirect packet → hash_size=0 | +| 4 | resolved_path filtering | Node detail → Paths tab | Verify path-hop candidates use resolved_path, no prefix collision false positives | +| 5 | Hash stats repeater filter | Analytics → Hash Issues | "By Repeaters" should only show nodes with repeater role, not companions/sensors | +| 6 | Async chunked backfill | Server startup | Start server with large DB → verify HTTP serves within 2 minutes, `X-CoreScope-Status: backfilling` header present, then transitions to `ready` | +| 7 | Memory eviction (heap stats) | Admin/stats | Verify `/api/stats` shows realistic memory numbers from runtime heap, not the old estimation | +| 8 | Distance/subpath/path-hop indexes | Analytics → Distances, Subpaths | Verify analytics data matches v3.4.1 output (no missing or extra entries) | +| 9 | cacheTTL config wiring | Config | Set `cacheTTL.analyticsHashSizes: 300` in config → verify collision cache respects it | + +### B2. MEDIUM RISK — User-Facing Features + +| # | Feature | Page | What to Check | +|---|---------|------|---------------| +| 10 | Distance unit preference | Nodes detail, Map | Toggle km/mi/auto in settings → distances update throughout UI | +| 11 | Panel corner toggle | Live page | Click corner toggle → panel moves to opposite corner, persists on reload | +| 12 | Noise floor column chart | Analytics → RF | Verify column chart renders with color-coded thresholds, hover shows values | +| 13 | Deep linking UI states | All pages | Navigate to `#/nodes?tab=neighbors`, `#/packets?observer=X`, `#/channels?node=Y` → correct state loads. Copy URL, open in new tab → same state | +| 14 | Sortable tables | Nodes list, Neighbors, Observers | Click column headers → sort asc/desc, indicator arrow shows, persists correctly | +| 15 | Channel color highlighting | Channels, Live feed | Assign color to channel → feed rows show that color, persists on reload | +| 16 | Mobile accessibility | All pages (phone viewport) | Touch targets ≥44px, ARIA labels present, small viewport doesn't overflow | +| 17 | Collapsible panels | Live map | Collapse/expand panels, medium breakpoint auto-collapses, state persists | +| 18 | Byte-size map filter | Map page | Filter by byte size → markers update correctly | +| 19 | OpenAPI/Swagger | `/api/spec`, `/api/docs` | Spec loads valid JSON, Swagger UI renders and all endpoints are documented | +| 20 | API key rejection | Protected endpoints | Send weak key (e.g. "changeme", "test123") → 403 forbidden | +| 21 | Channel color picker mobile | Channels (phone viewport) | Color picker usable on touch, doesn't overflow | +| 22 | RF Health dashboard | Analytics → RF Health | Observer metrics grid, airtime charts, battery charts, error rate, region filter | +| 23 | Prefix Tool tab | Analytics → Prefix Tool | Renders correctly, collision data consistent with Hash Issues | +| 24 | View Route on Map | Packet detail page | Button works and shows route on map | + +### B3. LOWER RISK — Performance (Verify No Regressions) + +| # | Feature | Page | What to Check | +|---|---------|------|---------------| +| 25 | Incremental DOM diff | Packets (30K+) | Virtual scroll renders smoothly, no visible flicker | +| 26 | Coalesced WS renders | Live page | Rapid packets don't cause frame drops (rAF coalescing) | +| 27 | Marker reposition on zoom | Map | Zoom/resize → markers move smoothly, no full rebuild flash | +| 28 | Parallel replay fetches | Live → VCR | Replay loads quickly (parallel observation fetches) | +| 29 | Batch observations API | Packets page (sort change) | Changing sort fetches observations in batch (network tab: 1 POST not N GETs) | +| 30 | Client-side network status | Analytics | No separate API call for network status | +| 31 | og-image compression | `/og-image.png` | Verify loads, ~235KB not ~1.1MB | + +--- + +## C. API Regression Tests + +Run against a local server with test-fixture DB: + +```bash +BASE=http://localhost:13581 + +# Core endpoints — verify response shape +curl -s "$BASE/api/stats" | jq '.totalPackets, .backfilling, .backfillProgress' +curl -s "$BASE/api/packets?limit=5" | jq '.packets[0] | keys' +curl -s "$BASE/api/packets?limit=5&expand=observations" | jq '.packets[0].observations | length' +curl -s "$BASE/api/nodes?limit=5" | jq '.[0] | keys' + +# New endpoints +curl -s -X POST "$BASE/api/packets/observations" \ + -H 'Content-Type: application/json' \ + -d '{"hashes":["test123"]}' | jq '.results | keys' + +curl -s "$BASE/api/analytics/subpaths-bulk?hops=A,B&hops=B,C" | jq 'keys' + +curl -s "$BASE/api/observers/metrics/summary" | jq 'type' +curl -s "$BASE/api/spec" | jq '.openapi' +curl -s "$BASE/api/docs" | head -5 # Should return HTML + +# Backfill status header +curl -sI "$BASE/api/stats" | grep X-CoreScope-Status + +# API key rejection +curl -s -H 'X-API-Key: changeme' "$BASE/api/debug/vars" | jq '.error' +curl -s -H 'X-API-Key: test' "$BASE/api/debug/vars" | jq '.error' + +# Existing endpoints — verify not broken +curl -s "$BASE/api/analytics/rf?timeRange=24h" | jq 'keys' +curl -s "$BASE/api/analytics/hash-sizes" | jq 'type' +curl -s "$BASE/api/analytics/distances" | jq 'type' +curl -s "$BASE/api/analytics/subpaths" | jq 'type' +curl -s "$BASE/api/channels" | jq 'type' +curl -s "$BASE/api/config/client" | jq 'keys' +``` + +### Expected response shape changes from v3.4.1: +- `/api/stats` now includes `backfilling` (bool) and `backfillProgress` (float 0-1) +- `/api/packets` no longer strips observations by default (lazy via `ExpandObservations` flag) — verify `observations` key absent without `expand=observations` +- Decoded packets with route_type=direct now have `hashSize: 0` +- TRACE packets now have `path.hopsCompleted` field + +--- + +## D. Performance Regression Tests + +### D1. Server Startup Time +```bash +# Start server with production-size DB (~30K packets) +# Measure time from process start to first successful HTTP response +time curl -s http://localhost:13581/api/stats > /dev/null +# Target: < 2 minutes (async backfill requirement) +``` + +### D2. Go Benchmarks +```bash +cd cmd/server && go test -bench=. -benchmem -count=3 +``` +Key benchmarks to compare with v3.4.1 baseline: +- `BenchmarkQueryPackets` — should not regress with new indexes +- `BenchmarkEvictStale` — batch removal from secondary indexes +- `BenchmarkGetStoreStats` — 2 concurrent queries vs 5 sequential +- `BenchmarkIngestNew` — additional index maintenance overhead + +### D3. Frontend Performance +- Open Packets page with 30K+ packets → measure initial render time (DevTools Performance tab) +- Scroll rapidly through virtual scroll → should maintain 60fps +- Switch sort column on packets → single batch POST, not N+1 GETs +- Open Analytics page → no redundant API calls in network tab + +### D4. Memory Usage +- After loading 30K packets, check `/api/stats` memory figure +- Compare with v3.4.1 baseline (prefix map cap at 8 chars should reduce ~10x) +- Verify eviction triggers at correct memory threshold using runtime heap stats + +--- + +## E. Infrastructure / Deployment Tests + +### E1. Docker Build +```bash +docker build -t corescope:test . +docker run --rm -p 13581:13581 corescope:test +# Verify: container starts, HTTP responds, WebSocket connects +``` + +### E2. GHCR Publish (CI) +- Verify CI publishes to `ghcr.io/kpa-clawbot/corescope` +- Verify tags: `edge` (master), `vX.Y.Z` (release) + +### E3. Staging Deploy +```bash +# Verify staging compose works with standard ports +docker compose -f docker-compose.staging.yml up -d +# Check: no 3GB memory limit, standard port binding +``` + +### E4. DISABLE_CADDY +```bash +docker run --rm -e DISABLE_CADDY=true corescope:test +# Verify: Caddy not started, Go server serves directly +``` + +### E5. CI Pipeline +- Verify consolidated pipeline: build → publish GHCR → deploy staging +- Verify runs on `meshcore-runner-2` + +--- + +## F. Edge Cases & Integration Tests + +### F1. Cross-Feature Interactions +| Scenario | Risk | +|----------|------| +| Deep link to sorted table → sort state matches URL params | Medium | +| Channel color + deep link → color persists in linked URL | Medium | +| Panel corner toggle + collapsible panels → both states persist independently | Low | +| Distance unit pref + neighbor table sort by distance → sort uses correct unit | Medium | +| Noise floor chart + region filter → chart respects filter | Medium | +| Byte-size map filter + channel color highlighting → both active simultaneously | Low | + +### F2. Data Correctness Edge Cases +| Scenario | Risk | +|----------|------| +| Zero-hop TRACE packet (should NOT reset hashSize — TRACE exemption) | **High** | +| Packet with all hops having same 2-char prefix → resolved_path filtering prevents false match | **High** | +| Node that switches role (repeater → companion) → hash stats updates | Medium | +| Backfill interrupted mid-chunk (server restart) → resumes or completes on next start | Medium | +| Empty DB startup → no errors, backfill completes instantly | Low | +| DB with 100K+ packets → async backfill doesn't OOM, progress reported | **High** | + +### F3. Concurrency / Race Conditions +| Scenario | Risk | +|----------|------| +| Concurrent API requests during backfill → no deadlock (lock ordering documented) | **High** | +| Eviction running while analytics query in progress → no stale pointer panic | **High** | +| Multiple WebSocket clients during high ingest rate → coalesced broadcasts don't drop | Medium | +| `time.NewTicker` cleanup on graceful shutdown (replaced `time.Tick`) | Low | + +### F4. API Key Security +| Scenario | Expected | +|----------|----------| +| No API key configured → write endpoints disabled | 403 "write endpoints disabled" | +| Weak key "changeme" → rejected even if configured | 403 "forbidden" | +| Timing-safe comparison → no timing oracle | Constant-time via `crypto/subtle` | +| Empty string key → rejected | 401 "unauthorized" | + +### F5. Browser Compatibility +- Test on Chrome, Firefox, Safari (latest) +- Test on iOS Safari, Android Chrome +- Verify touch targets on mobile (44px minimum) +- Verify ARIA labels with screen reader + +--- + +## G. Test Coverage Gaps — Action Items + +| Gap | Priority | Action | +|-----|----------|--------| +| No automated test for distance unit preference rendering | Medium | Add Playwright test | +| No automated test for noise floor column chart | Medium | Add Playwright test | +| No automated test for deep link state restoration | **High** | Add Playwright tests for each deep-linkable state | +| No automated test for channel color persistence | Medium | `test-channel-colors.js` covers model; need Playwright for UI | +| No automated test for mobile viewport behavior | Medium | Add Playwright test with mobile viewport | +| No automated test for backfill progress header | Low | Add to `routes_test.go` | +| No automated test for `time.NewTicker` cleanup | Low | Add to graceful shutdown test | +| Observer metrics endpoints not covered in route tests | Medium | Add to `routes_test.go` | +| Subpaths-bulk endpoint needs test | Medium | Add to `routes_test.go` | +| No load test for batch observations endpoint (200 hash limit) | Low | Add boundary test | + +--- + +## H. Release Checklist + +- [ ] All Go tests pass with `-race` flag +- [ ] All frontend unit tests pass +- [ ] Playwright E2E tests pass +- [ ] Manual browser verification (Section B) complete +- [ ] API regression tests (Section C) pass +- [ ] Docker build succeeds +- [ ] Staging deploy verified +- [ ] No console errors on any page +- [ ] Performance spot-checks (Section D) — no regressions +- [ ] Coverage badges updated (backend ≥85%, frontend ≥42%) +- [ ] CHANGELOG updated +- [ ] Tag `v3.4.2` created diff --git a/docs/user-guide/faq.md b/docs/user-guide/faq.md index 47c22170..34d84efa 100644 --- a/docs/user-guide/faq.md +++ b/docs/user-guide/faq.md @@ -52,3 +52,14 @@ CoreScope uses URL hashes for deep linking. Copy the URL from your browser — i - `#/packets/abc123` — a specific packet - `#/analytics?tab=collisions` — the hash issues tab - `#/nodes/pubkey123` — a specific node's detail page + +### Where is the API documentation? + +CoreScope auto-generates an OpenAPI 3.0 specification from its route definitions: + +- **Interactive docs (Swagger UI):** `/api/docs` — browse and test all 40+ endpoints from your browser +- **Machine-readable spec:** `/api/spec` — import into Postman, Insomnia, or any OpenAPI tool + +The spec is always in sync with the running server. No manual maintenance needed. + +On the public instance: [analyzer.00id.net/api/docs](https://analyzer.00id.net/api/docs)