mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-10-22 19:45:07 +00:00
96d0bbe48730b18d3ef965804fee70c6ccfacdd9
1163
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Kpa-clawbotandyou
|
96d0bbe487 |
fix: replace Euclidean distance with haversine in analytics hop distances (#478)
## Summary Fixes #433 — Replace the inaccurate Euclidean distance approximation in `analytics.js` hop distances with proper haversine calculation, matching the server-side computation introduced in PR #415. ## Problem PR #415 moved collision analysis server-side and switched from the frontend's Euclidean approximation (`dLat×111, dLon×85`) to proper haversine. However, the **hop distance** calculation in `analytics.js` (subpath detail panel) still used the old Euclidean formula. This caused: - **Inconsistent distances** between hop distances and collision distances - **Significant errors at high latitudes** — e.g., Oslo→Stockholm: Euclidean gives ~627km, haversine gives ~415km (51% error) - The `dLon×85` constant assumes ~40° latitude; at 60° latitude the real scale factor is ~55.5km/degree, not 85 ## Changes | File | Change | |------|--------| | `public/analytics.js` | Replace `dLat*111, dLon*85` Euclidean with `HopResolver.haversineKm()` (with inline fallback) | | `public/hop-resolver.js` | Export `haversineKm` in the public API for reuse | | `test-frontend-helpers.js` | Add 4 tests: export check, zero distance, SF→LA accuracy, Euclidean vs haversine divergence | | `cmd/server/helpers_test.go` | Add `TestHaversineKm`: zero, SF→LA, symmetry, Oslo→Stockholm accuracy | | `public/index.html` | Cache buster bump | ## Performance No performance impact — `haversineKm` replaces an inline arithmetic expression with another inline arithmetic expression of identical O(1) complexity. Only called per hop pair in the subpath detail panel (typically <10 hops). ## Testing - `node test-frontend-helpers.js` — 248 passed, 0 failed - `go test -run TestHaversineKm` — PASS Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
6712da7d7c |
fix: add region filtering to hash-collisions endpoint (#477)
## Summary The `/api/analytics/hash-collisions` endpoint always returned global results, ignoring the active region filter. Every other analytics endpoint (RF, topology, hash-sizes, channels, distance, subpaths) respected the `?region=` query parameter — this was the only one that didn't. Fixes #438 ## Changes ### Backend (`cmd/server/`) - **routes.go**: Extract `region` query param and pass to `GetAnalyticsHashCollisions(region)` - **store.go**: - `collisionCache` changed from `*cachedResult` → `map[string]*cachedResult` (keyed by region, `""` = global) — consistent with `rfCache`, `topoCache`, etc. - `GetAnalyticsHashCollisions(region)` and `computeHashCollisions(region)` now accept a region parameter - When region is specified, resolves regional observers, scans packets for nodes seen by those observers, and filters the node list before computing collisions - Cache invalidation updated to clear the map (not set to nil) ### Frontend (`public/`) - **analytics.js**: The hash-collisions fetch was missing `+ sep` (the region query string). All other fetches in the same `Promise.all` block had it — this was simply overlooked in PR #415. - **index.html**: Cache busters bumped ### Tests (`cmd/server/routes_test.go`) - `TestHashCollisionsRegionParamIgnored` → renamed to `TestHashCollisionsRegionParam` with updated comments reflecting that region is now accepted (with no configured regional observers, results match global — which the test verifies) ## Performance No new hot-path work. Region filtering adds one scan of `s.packets` (same as every other region-filtered analytics endpoint) only when `?region=` is provided. Results are cached per-region with the existing 60s TTL. Without `?region=`, behavior is unchanged. Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
6aef83c82a |
fix: remove duplicate return statement in _cumulativeRowOffsets() (#476)
## Summary Removes an unreachable duplicate `return offsets;` statement in the `_cumulativeRowOffsets()` function in `packets.js`. The second return was dead code found during review of PR #402. ## Changes - **`public/packets.js`**: Removed the duplicate `return offsets;` on what was line 1137 (the line immediately after the first, reachable `return offsets;`) - **`public/index.html`**: Cache buster bump ## Testing This is a dead code removal — the duplicate return was unreachable. No behavior change. No new tests needed as existing tests already cover `_cumulativeRowOffsets()` behavior. Fixes #447 Co-authored-by: you <you@example.com> |
||
|
|
9f14c74b3e |
ci: add Docker cleanup before build to prevent disk space exhaustion (#473)
## Summary Fixes #472 The Docker build job on the self-hosted runner fails with `no space left on device` because Docker build cache and Go module downloads accumulate between runs. The existing cleanup (line ~330) runs in the **deploy** step *after* the build — too late to help. ## Changes - Added a "Free disk space" step at the start of the build job, **before** "Build Go Docker image": - `docker system prune -af` — removes all unused images, containers, networks - `docker builder prune -af` — clears the build cache - `df -h /` — logs available disk space for visibility - Kept the existing post-deploy cleanup as belt-and-suspenders --------- Co-authored-by: you <you@example.com> Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com> |
||
|
|
0b8b1e91a6 |
perf(live): replace animation setIntervals with requestAnimationFrame and cap concurrency (#470)
## Summary Replace all `setInterval`-based animations in `live.js` with `requestAnimationFrame` loops and add a concurrency cap to prevent unbounded animation accumulation under high packet throughput. Fixes #384 ## Problem Under high throughput (≥5 packets/sec), the live map accumulated unbounded `setInterval` timers: - `pulseNode()`: 26ms interval per pulse ring - `drawAnimatedLine()`: 33ms interval per hop line + 52ms nested interval for fade-out - Ghost hop pulse: 600ms interval per ghost marker At 5 pkts/sec × 3 hops = **15+ concurrent intervals**, climbing without limit. This caused UI jank, rising CPU usage, and potential memory leaks from leaked Leaflet markers. ## Changes ### `public/live.js` | Function | Before | After | |----------|--------|-------| | `pulseNode()` | `setInterval` (26ms) + `setTimeout` safety | `requestAnimationFrame` loop, self-terminates at 2s or opacity ≤ 0 | | `drawAnimatedLine()` | `setInterval` (33ms) for line + nested `setInterval` (52ms) for fade | Two `requestAnimationFrame` loops (line advance + fade-out) | | Ghost hop pulse | `setInterval` (600ms) + `setTimeout` (3s) | `requestAnimationFrame` loop with 3s expiry | | `animatePath()` | No concurrency limit | Returns early when `activeAnims >= MAX_CONCURRENT_ANIMS` (20) | ### `public/index.html` - Cache buster version bump ### `test-live-anims.js` (new) - 7 tests verifying: - No `setInterval` in `pulseNode`, `drawAnimatedLine`, or `animatePath` - `MAX_CONCURRENT_ANIMS` defined and set to 20 - Concurrency check present in `animatePath` - No stale `setInterval` in animation hot paths ## Complexity & Scale - **Time complexity**: O(1) per animation frame (no change in per-frame work) - **Concurrency**: Hard-capped at 20 simultaneous animations (previously unbounded) - **At 5 pkts/sec, 3 hops**: Excess animations silently dropped instead of accumulating timers - **rAF benefit**: Browser coalesces all animations into single paint cycle; paused tabs stop animating automatically ## Test Results ``` === Animation interval elimination === ✅ pulseNode does not use setInterval ✅ drawAnimatedLine does not use setInterval ✅ ghost hop pulse does not use setInterval === Concurrency cap === ✅ MAX_CONCURRENT_ANIMS is defined ✅ MAX_CONCURRENT_ANIMS is set to 20 ✅ animatePath checks MAX_CONCURRENT_ANIMS before proceeding === Safety: no stale setInterval in animation functions === ✅ no setInterval remains in animation hot path 7 passed, 0 failed ``` All existing tests pass (packet-filter: 62, aging: 29, frontend-helpers: 241). ## Performance Proof (Rule 0 compliance) Benchmark: `node test-anim-perf.js` — simulates timer/animation accumulation under realistic throughput. ### Timer count: old (setInterval) vs new (rAF + cap) | Scenario | Old model (peak concurrent timers) | New model (peak concurrent animations) | |----------|-----------------------------------:|---------------------------------------:| | 5 pkt/s × 3 hops, 30s sustained | **123** | **20** | | 5 pkt/s × 3 hops, 5min sustained | **123** | **20** | | 20 pkt/s × 3 hops, 10s burst | **246** | **20** | **Before:** Each hop spawns 3 `setInterval` timers (pulse 26ms, line 33ms, fade 52ms) that live 0.6–2s each. At 5 pkt/s × 3 hops = 15 timers/sec, peak concurrent timers reach **123** (limited only by timer lifetime, not by any cap). Under burst traffic (20 pkt/s), this climbs to **246+**. **After:** `MAX_CONCURRENT_ANIMS = 20` hard-caps active animations. Excess packets are silently dropped. rAF loops replace all `setInterval` calls, coalescing into single paint cycles. Peak concurrent animations: **always ≤ 20**, regardless of throughput or duration. --------- Co-authored-by: you <you@example.com> Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com> |
||
|
Kpa-clawbotandyou
|
c678555e75 |
fix: display channel hash as hex instead of decimal (#471)
## Summary Fixes #465 — Channel hash was displaying in decimal instead of hexadecimal in `channels.js`. ## Changes - Added `formatHashHex()` helper to `channels.js` that formats numeric hashes as `0x` hex (e.g. `0x0A`) and passes string hashes through unchanged - Applied to both display sites: `renderChannelList` fallback name and `selectChannel` header text - Consistent with `packets.js` and `analytics.js` which already use `.toString(16).padStart(2, '0').toUpperCase()` ## Tests - 3 new tests in `test-frontend-helpers.js` verifying the helper exists, is used at display sites, and produces correct output for numeric and string inputs - All 244 frontend tests pass, plus packet-filter (62) and aging (29) tests Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
623ebc879b |
fix: add mutex synchronization to PerfStats to eliminate data races (#469)
## Summary Fixes #361 — `perfMiddleware()` wrote to shared `PerfStats` fields (`Requests`, `TotalMs`, `Endpoints` map, `SlowQueries` slice) without any synchronization, causing data races under concurrent HTTP requests. ## Changes ### `cmd/server/routes.go` - **Added `sync.Mutex` to `PerfStats` struct** — single mutex protects all fields - **`perfMiddleware`** — all shared state mutations (counter increments, endpoint map access, slice appends) now happen under lock. Key normalization (regex, mux route lookup) moved outside the lock since it uses no shared state - **`handleHealth`** — snapshots `Requests`, `TotalMs`, `SlowQueries` under lock before building response - **`handlePerf`** — copies all endpoint data and slow queries under lock into local snapshots, then does expensive work (sorting, percentile calculation) outside the lock - **`handlePerfReset`** — resets fields in-place instead of replacing the pointer (avoids unlocking a different mutex) ### `cmd/server/perfstats_race_test.go` (new) - Regression test: 50 concurrent writer goroutines + 10 concurrent reader goroutines hammering `PerfStats` simultaneously - Verifies no race conditions (via `-race` flag) and counter consistency ## Design Decisions - **Single mutex over atomics**: The issue suggested `atomic.Int64` for counters, but since slices/maps need a mutex anyway, a single mutex is simpler and the critical section is small (microseconds). No measurable contention at CoreScope's scale. - **Copy-under-lock pattern**: Expensive operations (sorting, percentile computation) happen outside the lock to minimize hold time. - **In-place reset**: `handlePerfReset` clears fields rather than replacing the `PerfStats` pointer, ensuring the mutex remains valid for concurrent goroutines. ## Testing - `go test -race -count=1 ./cmd/server/...` — **PASS** (all existing tests + new race test) - New `TestPerfStatsConcurrentAccess` specifically validates concurrent access patterns Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
0b1924d401 |
perf(packets): replace observers.find() linear scans with Map lookups (#468)
## Summary Replace all `observers.find()` linear scans in `packets.js` with O(1) `Map.get()` lookups, eliminating ~300K comparisons per render cycle at 30K+ rows. ## Changes - Added `observerMap` (`Map<id, observer>`) built once when observers load - Replaced all 6 `observers.find()` call sites with `observerMap.get()`: - `obsName()` — called per row for observer name display - Region filter check in packet filtering - Observer dropdown label in filter UI - Group header region lookup - Child row region lookup - Flat row region lookup - Map is cleared on reset and rebuilt on each `loadObservers()` call ## Complexity - **Before:** O(k) per row × 30K rows = O(30K × k) where k = observer count (~10) - **After:** O(1) per row × 30K rows = O(30K) - Map construction: O(k) once, negligible ## Testing - All Go tests pass (`cmd/server`, `cmd/ingestor`) - All frontend tests pass (`test-packet-filter.js`: 62 passed, `test-aging.js`: 29 passed, `test-frontend-helpers.js`: 241 passed) Fixes #383 Co-authored-by: you <you@example.com> |
||
|
efiten
|
0f502370c5 |
fix: VCR timeline and clock respect UTC/local timezone setting (#459)
## Problem Fixes #324. The VCR LCD clock and timeline hover/touch tooltip always showed local time, ignoring the UTC/local timezone setting in the customizer Display tab. ## Root cause Three sites in `live.js` bypassed the shared `getTimestampTimezone()` utility: - `updateVCRClock()` — used `d.getHours()` / `d.getMinutes()` / `d.getSeconds()` (always local) - Timeline mousemove tooltip — used `d.toLocaleTimeString()` (always local) - Timeline touchmove tooltip — same ## Fix Added `vcrFormatTime(tsMs)` helper that checks `getTimestampTimezone()` and uses `getUTC*` methods when set to `'utc'`, otherwise local `get*`. Applied to all three sites. Exposed as `window._vcrFormatTime` for testing. ## Tests 4 new unit tests in `test-frontend-helpers.js` covering UTC mode, local mode, and zero-padding. ## Checklist - [x] Branches from `upstream/master` - [x] No Matomo or local-only commits - [x] Cache busters bumped (`v=1775073838`) - [x] 233 tests pass, 0 fail 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
efitenandClaude Sonnet 4.6
|
e47c39ffda |
fix: null-guard animLayer and liveAnimCount in nextHop after destroy (#462)
## Summary - `nextHop()` schedules `setInterval`/`setTimeout` callbacks that can fire after `destroy()` has set `animLayer = null` and removed DOM elements - This caused three console errors on the Live page when navigating away mid-animation: `Cannot read properties of null (reading 'hasLayer')` and `Cannot set properties of null (setting 'textContent')` - Added null guards at each async callback site; no behavioral change when the page is active ## Changes - `public/live.js`: early return if `animLayer` is null at start of `nextHop()`; null-safe `animLayer.hasLayer` checks in `setInterval`/`setTimeout`; null-safe `liveAnimCount` element access - `public/index.html`: cache buster bumped - `test-frontend-helpers.js`: 4 source-inspection tests verifying the null guards are present ## Test plan - [ ] Open Live page, trigger some packet animations, navigate away quickly — no console errors - [ ] `node test-frontend-helpers.js` passes (233 tests, 0 failures) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
efiten
|
1499a55ba7 |
perf: upsert known nodes in-place on ADVERT, skip full reload (#461)
## Problem Fixes #399. On every ADVERT WebSocket batch the nodes page invalidated the entire `_allNodes` cache and triggered a full `/nodes?limit=5000` fetch — even when every advertising node was already cached. The 90s API TTL was actively bypassed. ## Root cause ```js wsHandler = debouncedOnWS(function (msgs) { if (msgs.some(isAdvertMessage)) { _allNodes = null; // wipe cache unconditionally invalidateApiCache('/nodes'); // bust API TTL loadNodes(true); // full 5k fetch } }, 5000); ``` ## Fix ADVERT decoded payloads include `pubKey`, `name`, `lat`, `lon` — enough to update known nodes in place: - **Known node** (pubKey found in `_allNodes`): upsert `name`, `lat`, `lon`, `last_seen` directly — no fetch, no cache bust, just re-render. - **New node** (pubKey not in cache) or **no pubKey** in payload: fall back to full reload as before. This covers the common case on an active mesh: all advertising nodes are already cached. The full reload path is preserved for node discovery. ## Tests 2 new unit tests: known-node upsert (asserts 0 API calls, fields updated) and unknown-node fallback (asserts full reload triggered). All 231 tests pass. ## Checklist - [x] Branches from `upstream/master` - [x] No Matomo or local-only commits - [x] Cache busters bumped - [x] 231 tests pass, 0 fail 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
efiten
|
f71e117cdd |
fix: reset restores home steps after SITE_CONFIG contamination (#460)
## Problem Fixes #325. Removing all home steps and clicking "Reset my theme" did not restore them. ## Root cause Two-part bug: **1. `SITE_CONFIG.home` permanently mutated at page load** `app.js` calls `mergeUserHomeConfig(SITE_CONFIG, userTheme)` which does `SITE_CONFIG.home = Object.assign({}, serverHome, userTheme.home)`. If the user had `steps: []` saved in localStorage, this sets `SITE_CONFIG.home.steps = []` globally — permanently for the lifetime of the page. **2. `initState()` reads the contaminated config** When the customizer opens (or Reset is clicked), `initState()` reads `cfg = window.SITE_CONFIG`. Since `SITE_CONFIG.home.steps` is already `[]`, `state.home.steps` stays `[]` even after `localStorage.removeItem`. `autoSave()` then re-saves `steps: []` straight back. **Secondary issue:** `data-rm-step` / add / move handlers didn't call `autoSave()`, making step persistence non-deterministic (only saved if a text field edit happened to be pending). ## Fix - **`app.js`**: snapshot `SITE_CONFIG.home` before `mergeUserHomeConfig` → `window._SITE_CONFIG_ORIGINAL_HOME` - **`customize.js`**: `initState()` uses `_SITE_CONFIG_ORIGINAL_HOME` instead of the contaminated `cfg.home` - **`customize.js`**: add `autoSave()` to rm/move/add handlers for steps, checklist, and footer links ## Tests 2 new unit tests covering the snapshot bypass and DEFAULTS fallback. 231 tests pass. ## Checklist - [x] Branches from `upstream/master` - [x] No Matomo or local-only commits - [x] Cache busters bumped - [x] 231 tests pass, 0 fail 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Kpa-clawbotandyou
|
75f1295a06 |
fix: always refresh staging config from prod (#467)
## Summary Fixes #466 — staging config was not refreshed from prod due to a stale `-nt` timestamp guard. ## Root Cause `prepare_staging_config()` only copied prod config when staging was missing or prod was newer by mtime. However, the `sed -i` that applies the STAGING siteName updated staging's mtime, making it appear newer than prod. Subsequent runs skipped the copy entirely. ## Changes - **`manage.sh`**: Removed the `-nt` timestamp conditional in `prepare_staging_config()`. Staging config is now always copied fresh from prod with the STAGING siteName applied. Note: `prepare_staging_db()` already copies unconditionally — no change needed there. Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
b1b76acb77 |
feat: manage.sh update supports pinning to release tags (#456)
## Summary `manage.sh update` now supports pinning to specific release tags instead of always pulling tip of master. Fixes #455 ## Changes ### `cmd_update` — accepts optional version argument - **No argument**: fetches tags, checks out latest release tag (`git tag -l 'v*' --sort=-v:refname | head -1`) - **`latest`**: explicit opt-in to tip of master (bleeding edge) - **Specific tag** (e.g. `v3.1.0`): checks out that exact tag, with error message + available tags if not found ### `cmd_setup` — defaults to latest tag - After Docker check, fetches tags and pins to latest release tag - Skips if already on the latest tag - Uses state tracking (`version_pin`) so re-runs don't repeat ### `cmd_status` — shows version - Displays current version (exact tag name or short commit hash) at the top of status output ### Help text - Updated to reflect new `update [version]` syntax ## Usage ```bash ./manage.sh update # checkout latest release tag (e.g. v3.2.0) ./manage.sh update v3.1.0 # pin to specific version ./manage.sh update latest # explicit tip of master (bleeding edge) ./manage.sh status # now shows "Version: v3.2.0" ``` ## Testing - `bash -n manage.sh` passes (syntax valid) - Logic follows existing patterns (git fetch, checkout, rebuild, restart) --------- Co-authored-by: you <you@example.com> |
||
|
|
f87eb3601c |
fix: graceful container shutdown for reliable deployments (#453)
## Summary Fixes #450 — staging deployment flaky due to container not shutting down cleanly. ## Root Causes 1. **Server never closed DB on shutdown** — SQLite WAL lock held indefinitely, blocking new container startup 2. **`httpServer.Close()` instead of `Shutdown()`** — abruptly kills connections instead of draining them 3. **No `stop_grace_period` in compose configs** — Docker sends SIGTERM then immediately SIGKILL (default 10s is often not enough for WAL checkpoint) 4. **Supervisor didn't forward SIGTERM** — missing `stopsignal`/`stopwaitsecs` meant Go processes got SIGKILL instead of graceful shutdown 5. **Deploy scripts used default `docker stop` timeout** — only 10s grace period ## Changes ### Go Server (`cmd/server/`) - **Graceful HTTP shutdown**: `httpServer.Shutdown(ctx)` with 15s context timeout — drains in-flight requests before closing - **WebSocket cleanup**: New `Hub.Close()` method sends `CloseGoingAway` frames to all connected clients - **DB close on shutdown**: Explicitly closes DB after HTTP server stops (was never closed before) - **WAL checkpoint**: `PRAGMA wal_checkpoint(TRUNCATE)` before DB close — flushes WAL to main DB file and removes WAL/SHM lock files ### Go Ingestor (`cmd/ingestor/`) - **WAL checkpoint on shutdown**: New `Store.Checkpoint()` method, called before `Close()` - **Longer MQTT disconnect timeout**: 5s (was 1s) to allow in-flight messages to drain ### Docker Compose (all 4 variants) - Added `stop_grace_period: 30s` and `stop_signal: SIGTERM` ### Supervisor Configs (both variants) - Added `stopsignal=TERM` and `stopwaitsecs=20` to server and ingestor programs ### Deploy Scripts - `deploy-staging.sh`: `docker stop -t 30` with explicit grace period - `deploy-live.sh`: `docker stop -t 30` with explicit grace period ## Shutdown Sequence (after fix) 1. Docker sends SIGTERM to supervisord (PID 1) 2. Supervisord forwards SIGTERM to server + ingestor (waits up to 20s each) 3. Server: stops poller → drains HTTP (15s) → closes WS clients → checkpoints WAL → closes DB 4. Ingestor: stops tickers → disconnects MQTT (5s) → checkpoints WAL → closes DB 5. Docker waits up to 30s total before SIGKILL ## Tests All existing tests pass: - `cd cmd/server && go test ./...` ✅ - `cd cmd/ingestor && go test ./...` ✅ --------- Co-authored-by: you <you@example.com> Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com> |
||
|
Kpa-clawbotandyou
|
ec4dd58cb6 |
fix: null-guard pathHops to prevent detail pane crash (#451) (#454)
## Summary Fixes #451 — packet detail pane crash on direct routed packets where `pathHops` is `null`. ## Root Cause `JSON.parse(pkt.path_json)` can return literal `null` when the DB stores `"null"` for direct routed packets. The existing code only had a catch block for parse errors, but `null` is valid JSON — so the parse succeeds and `pathHops` ends up `null` instead of `[]`. ## Changes - **`public/packets.js`**: Added `|| []` after `JSON.parse(...)` in both `buildFlatRowHtml` (table rows) and the detail pane (`selectPacket`), ensuring `pathHops` is always an array. - **`test-frontend-helpers.js`**: Added 2 regression tests verifying the null guards exist in both code paths. - **`public/index.html`**: Cache buster bump. ## Testing - All 229 frontend helper tests pass - All 62 packet filter tests pass - All 29 aging tests pass Co-authored-by: you <you@example.com> |
||
|
|
044a5387af |
perf(packets): virtual scroll + debounced WS renders for packets table (#402)
## Summary Fixes the critical performance issue where `renderTableRows()` rebuilt the **entire** table innerHTML (up to 50K rows) on every update — WebSocket arrivals, filter changes, group expand/collapse, and theme refreshes. ## Changes ### Lazy Row Generation (`renderVisibleRows`) — fixes #422 - Row HTML strings are **only generated for the visible slice + 30-row buffer** on each render - `_displayPackets` stores the filtered data array; `renderVisibleRows()` calls `buildGroupRowHtml`/`buildFlatRowHtml` lazily for ~60-90 visible entries - Previously, `displayPackets.map(buildGroupRowHtml)` built HTML for ALL 30K+ packets on every render — the expensive work (JSON.parse, observer lookups, template literals) ran for every packet regardless of visibility ### Unified Row Count via `_getRowCount()` — fixes #424 - Single function `_getRowCount(p)` computes DOM row count for any entry (1 for flat/collapsed, 1+children for expanded groups) - Used by BOTH `_rowCounts` computation AND `renderVisibleRows` — eliminates divergence risk between row counting and row building ### Hoisted Observer Filter Set — fixes #427 - `_observerFilterSet` created once in `renderTableRows()`, reused across `buildGroupRowHtml`, `_getRowCount`, and child filtering - Previously, `new Set(filters.observer.split(','))` was created inside `buildGroupRowHtml` for every packet AND again in the row count callback ### Dynamic Colspan — fixes #426 - `_getColCount()` reads column count from the thead instead of hardcoded `colspan="11"` - Spacers and empty-state messages use the actual column count ### Null-Safety in `buildFlatRowHtml` — fixes #430 - `p.decoded_json || '{}'` fallback added, matching `buildGroupRowHtml`'s existing null-safety - Prevents TypeError on null/undefined `decoded_json` in flat (ungrouped) mode ### Behavioral Tests — fixes #428 - Replaced 5 source-grep tests with behavioral unit tests for `_getRowCount`: - Flat mode always returns 1 - Collapsed group returns 1 - Expanded group returns 1 + child count - Observer filter correctly reduces child count - Null `_children` handled gracefully - Retained source-level assertions only where behavioral testing isn't practical (e.g., verifying lazy generation pattern exists) ### Other Improvements - Cumulative row offsets cached in `_cumulativeOffsetsCache`, invalidated on row count changes - Debounced WebSocket renders (200ms) coalesce rapid packet arrivals - `destroy()` properly cleans up all virtual scroll state ## Performance Benchmarks — fixes #423 **Methodology:** Row building cost measured by counting `buildGroupRowHtml` calls per render cycle on 30K grouped packets. | Scenario | Before (eager) | After (lazy) | Improvement | |----------|----------------|--------------|-------------| | Initial render (30K packets) | 30,000 `buildGroupRowHtml` calls | ~90 calls (60 visible + 30 buffer) | **333× fewer calls** | | Scroll event | 0 calls (pre-built) | ~90 calls (rebuild visible slice) | Trades O(1) scroll for O(n) initial savings | | WS packet arrival | 30,000 calls (full rebuild) | ~90 calls (debounced + lazy) | **333× fewer calls** | | Filter change | 30,000 calls | ~90 calls | **333× fewer calls** | | Memory (row HTML cache) | ~2MB string array for 30K packets | 0 (no cache, build on demand) | **~2MB saved** | **Per-call cost of `buildGroupRowHtml`:** Each call performs JSON.parse of `decoded_json`, `path_json`, `observers.find()` lookup, and template literal construction. At 30K packets, the eager approach spent ~400-500ms on row building alone (measured via `performance.now()` on staging data). The lazy approach builds ~90 rows in ~1-2ms. **Net effect:** `renderTableRows()` goes from O(n) string building + O(1) DOM insertion to O(1) data assignment + O(visible) string building + O(visible) DOM insertion. For n=30K and visible≈60, this is ~333× less work per render cycle. **Trade-off:** Scrolling now rebuilds ~90 rows per RAF frame instead of slicing pre-built strings. This costs ~1-2ms per scroll event, well within the 16ms frame budget. The trade-off is overwhelmingly positive since renders happen far more frequently than full-table scrolls. ## Tests - 247 frontend helper tests pass (including 18 virtual scroll tests) - 62 packet filter tests pass - 29 aging tests pass - Go backend tests pass ## Remaining Debt (tracked in issues) - #425: Hardcoded `VSCROLL_ROW_HEIGHT=36` and `theadHeight=40` — should be measured from DOM - #429: 200ms WS debounce delay — value works well in practice but lacks formal justification - #431: No scroll position preservation on filter change or group expand/collapse Fixes #380 --------- Co-authored-by: you <you@example.com> Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com> |
||
|
|
01ca843309 |
perf: move collision analysis to server-side endpoint (fixes #386) (#415)
## Summary Moves the hash collision analysis from the frontend to a new server-side endpoint, eliminating a major performance bottleneck on the analytics collision tab. Fixes #386 ## Problem The collision tab was: 1. **Downloading all nodes** (`/nodes?limit=2000`) — ~500KB+ of data 2. **Running O(n²) pairwise distance calculations** on the browser main thread (~2M comparisons with 2000 nodes) 3. **Building prefix maps client-side** (`buildOneBytePrefixMap`, `buildTwoBytePrefixInfo`, `buildCollisionHops`) iterating all nodes multiple times ## Solution ### New endpoint: `GET /api/analytics/hash-collisions` Returns pre-computed collision analysis with: - `inconsistent_nodes` — nodes with varying hash sizes - `by_size` — per-byte-size (1, 2, 3) collision data: - `stats` — node counts, space usage, collision counts - `collisions` — pre-computed collisions with pairwise distances and classifications (local/regional/distant/incomplete) - `one_byte_cells` — 256-cell prefix map for 1-byte matrix rendering - `two_byte_cells` — first-byte-grouped data for 2-byte matrix rendering ### Caching Uses the existing `cachedResult` pattern with a new `collisionCache` map. Invalidated on `hasNewTransmissions` (same trigger as the hash-sizes cache) and on eviction. ### Frontend changes - `renderCollisionTab` now accepts pre-fetched `collisionData` from the parallel API load - New `renderHashMatrixFromServer` and `renderCollisionsFromServer` functions consume server-computed data directly - No more `/nodes?limit=2000` fetch from the collision tab - Old client-side functions (`buildOneBytePrefixMap`, etc.) preserved for test helper exports ## Test results - `go test ./...` (server): ✅ pass - `go test ./...` (ingestor): ✅ pass - `test-packet-filter.js`: ✅ 62 passed - `test-aging.js`: ✅ 29 passed - `test-frontend-helpers.js`: ✅ 227 passed ## Performance impact | Metric | Before | After | |--------|--------|-------| | Data transferred | ~500KB (all nodes) | ~50KB (collision data only) | | Client computation | O(n²) distance calc | None (server-cached) | | Main thread blocking | Yes (2000 nodes × pairwise) | No | | Server caching | N/A | 15s TTL, invalidated on new transmissions | --------- Co-authored-by: you <you@example.com> Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com> |
||
|
Kpa-clawbotandyou
|
5f50e80931 |
perf: replace server round-trip with client-side filter for My Nodes toggle (#401)
## Summary Fixes #381 — The "My Nodes" filter in `packets.js` was making a **server API call inside `renderTableRows()`** on every render cycle. With WebSocket updates arriving every few seconds while the toggle was active, this created continuous unnecessary server load. ## What Changed **`public/packets.js`** — Replaced the `api('/packets?nodes=...')` server call with a pure client-side filter: ```js // Before: server round-trip on every render const myData = await api('/packets?nodes=' + allKeys.join(',') + '&limit=500'); displayPackets = myData.packets || []; // After: filter already-loaded packets client-side displayPackets = displayPackets.filter(p => { const dj = p.decoded_json || ''; return allKeys.some(k => dj.includes(k)); }); ``` This uses the exact same matching logic as the server's `QueryMultiNodePackets()` — a string contains check on `decoded_json` for each pubkey — but without the network round-trip. **`test-frontend-helpers.js`** — Added 5 unit tests for the filter logic: - Single and multiple pubkey matching - No matches / empty keys edge case - Null/empty `decoded_json` handled gracefully **`public/index.html`** — Cache busters bumped. ## Test Results - Frontend helpers: **232 passed, 0 failed** (including 5 new tests) - Packet filter: **62 passed, 0 failed** - Aging: **29 passed, 0 failed** Co-authored-by: you <you@example.com> |
||
|
you
|
8f3d12eca5 | docs: perf claims require proof — benchmarks, timings, or test assertions | ||
|
you
|
357f7952f7 | docs: add Rule 0 — performance-first mindset in AGENTS.md | ||
|
Kpa-clawbotandyou
|
47d081c705 |
perf: targeted analytics cache invalidation (fixes #375) (#379)
## Problem Every time new data is ingested (`IngestNewFromDB`, `IngestNewObservations`, `EvictStale`), **all 6 analytics caches** are wiped by creating new empty maps — regardless of what kind of data actually changed. With the poller running every 1 second, this means the 15s cache TTL is effectively bypassed because caches are cleared far more frequently than they expire. ## Fix Introduces a `cacheInvalidation` flags struct and `invalidateCachesFor()` method that selectively clears only the caches affected by the ingested data: | Flag | Caches Cleared | |------|----------------| | `hasNewObservations` | RF (SNR/RSSI data changed) | | `hasNewPaths` | Topology, Distance, Subpaths | | `hasNewTransmissions` | Hash sizes | | `hasChannelData` | Channels (GRP_TXT payload_type 5) + channels list cache | | `eviction` | All (data removed, everything potentially stale) | ### Impact For a typical ingest cycle with ADVERT/ACK/TXT_MSG packets (no GRP_TXT): - **Before:** All 6 caches cleared every cycle - **After:** Channel cache preserved (most common case), hash cache preserved on observation-only ingestion For observation-only ingestion (`IngestNewObservations`): - **Before:** All 6 caches cleared - **After:** Only RF cache cleared (+ topo/dist/subpath if paths actually changed) ## Tests 7 new unit tests in `cache_invalidation_test.go` covering: - Eviction clears all caches - Observation-only ingest preserves non-RF caches - Transmission-only ingest clears only hash cache - Channel data clears only channel cache - Path changes clear topo/dist/subpath - Combined flags work correctly - No flags = no invalidation All existing tests pass. ### Post-rebase fix Restored `channelsCacheRes` invalidation that was accidentally dropped during the refactor. The old code cleared this separate channels list cache on every ingest, but `invalidateCachesFor()` didn't include it. Now cleared on `hasChannelData` and `eviction`. Fixes #375 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
be313f60cb |
fix: extract score/direction from MQTT, strip units, fix type safety issues (#371)
## Summary Fixes #353 — addresses all 5 findings from the CoreScope code analysis. ## Changes ### Finding 1 (Major): `score` field never extracted from MQTT - Added `Score *float64` field to `PacketData` and `MQTTPacketMessage` structs - Extract `msg["score"]` with `msg["Score"]` case fallback via `toFloat64` in all three MQTT handlers (raw packet, channel message, direct message) - Pass through to DB observation insert instead of hardcoded `nil` ### Finding 2 (Major): `direction` field never extracted from MQTT - Added `Direction *string` field to `PacketData` and `MQTTPacketMessage` structs - Extract `msg["direction"]` with `msg["Direction"]` case fallback as string in all three MQTT handlers - Pass through to DB observation insert instead of hardcoded `nil` ### Finding 3 (Minor): `toFloat64` doesn't strip units - Added `stripUnitSuffix()` that removes common RF/signal unit suffixes (dBm, dB, mW, km, mi, m) case-insensitively before `ParseFloat` - Values like `"-110dBm"` or `"5.5dB"` now parse correctly ### Finding 4 (Minor): Bare type assertions in store.go - Changed `firstSeen` and `lastSeen` from `interface{}` to typed `string` variables at `store.go:5020` - Removed unsafe `.(string)` type assertions in comparisons ### Finding 5 (Minor): `distHopRecord.SNR` typed as `interface{}` - Changed `distHopRecord.SNR` from `interface{}` to `*float64` - Updated assignment (removed intermediate `snrVal` variable, pass `tx.SNR` directly) - Updated output serialization to use `floatPtrOrNil(h.SNR)` for consistent JSON output ## Tests Added - `TestBuildPacketDataScoreAndDirection` — verifies Score/Direction flow through BuildPacketData - `TestBuildPacketDataNilScoreDirection` — verifies nil handling when fields absent - `TestInsertTransmissionWithScoreAndDirection` — end-to-end: inserts with score/direction, verifies DB values - `TestStripUnitSuffix` — covers all supported suffixes, case insensitivity, and passthrough - `TestToFloat64WithUnits` — verifies unit-bearing strings parse correctly All existing tests pass. Co-authored-by: you <you@example.com> |
||
|
efitenandClaude Sonnet 4.6
|
8a0862523d |
fix: add migration for missing observations.timestamp index (#332)
## Problem On installations where the database predates the `idx_observations_timestamp` index, `/api/stats` takes 30s+ because `GetStoreStats()` runs two full table scans: ```sql SELECT COUNT(*) FROM observations WHERE timestamp > ? -- last hour SELECT COUNT(*) FROM observations WHERE timestamp > ? -- last 24h ``` The index is only created in the `if !obsExists` block, so any database where the `observations` table already existed before that code was added never gets it. ## Fix Adds a one-time migration (`obs_timestamp_index_v1`) that runs at ingestor startup: ```sql CREATE INDEX IF NOT EXISTS idx_observations_timestamp ON observations(timestamp) ``` On large installations this index creation may take a few seconds on first startup after the upgrade, but subsequent stats queries become instant. ## Test plan - [ ] Restart ingestor on an older database and confirm `[migration] observations timestamp index created` appears in logs - [ ] Confirm `/api/stats` response time drops from 30s+ to <100ms 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
efitenandClaude Sonnet 4.6
|
7e8b30aa1f |
perf: fix slow /api/packets and /api/channels on large stores (#328)
## Problem Two endpoints were slow on larger installations: **`/packets?limit=50000&groupByHash=true` — 16s+** `QueryGroupedPackets` did two expensive things on every request: 1. O(n × observations) scan per packet to find `latest` timestamp 2. Held `s.mu.RLock()` during the O(n log n) sort, blocking all concurrent reads **`/channels` — 13s+** `GetChannels` iterated all payload-type-5 packets and JSON-unmarshaled each one while holding `s.mu.RLock()`, blocking all concurrent reads for the full duration. ## Fix **Packets (`QueryGroupedPackets`):** - Add `LatestSeen string` to `StoreTx`, maintained incrementally in all three observation write paths. Eliminates the per-packet observation scan at query time. - Build output maps under the read lock, sort the local copy after releasing it. - Cache the full sorted result for 3 seconds keyed by filter params. **Channels (`GetChannels`):** - Copy only the fields needed (firstSeen, decodedJSON, region match) under the read lock, then release before JSON unmarshaling. - Cache the result for 15 seconds keyed by region param. - Invalidate cache on new packet ingestion. ## Test plan - [ ] Open packets page on a large store — load time should drop from 16s to <1s - [ ] Open channels page — should load in <100ms instead of 13s+ - [ ] `[SLOW API]` warnings gone for both endpoints - [ ] Packet/channel data is correct (hashes, counts, observer counts) - [ ] Filters (region, type, since/until) still work correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Kpa-clawbotandyou
|
b2279b230b |
fix: handle string, uint, and uint64 types in toFloat64 (#352)
## Summary Fixes #350 — `toFloat64()` silently drops SNR/RSSI values when bridges send strings instead of numbers. ## Problem Some MQTT bridges serialize numeric fields (SNR, RSSI, battery_mv, etc.) as JSON strings like `"-7.5"` instead of numbers. The existing `toFloat64()` switch only handled `float64`, `float32`, `int`, `int64`, and `json.Number`, so string values fell through to the default case returning `(0, false)` — silently dropping the data. ## Changes - **`cmd/ingestor/main.go`**: Added `string`, `uint`, and `uint64` cases to `toFloat64()` - `string`: uses `strconv.ParseFloat(strings.TrimSpace(n), 64)` to handle whitespace-padded numeric strings - `uint` / `uint64`: straightforward numeric conversion - Added `strconv` import - **`cmd/ingestor/main_test.go`**: Updated `TestToFloat64` with new cases: - Valid string (`"3.14"`), string with spaces (`" -7.5 "`), string integer (`"42"`) - Invalid string (`"hello"`), empty string - `uint(10)`, `uint64(999)` ## Testing All ingestor tests pass (`go test ./...`). Co-authored-by: you <you@example.com> |
||
|
|
d1cb84b596 |
feat: Priority+ nav pattern for tablet viewports (768-1023px) (#345)
## Priority+ Navigation Pattern for Tablet Viewports Phase 2 of responsive nav improvements for #322. ### What this does On **tablet viewports (768-1023px)**, implements the [Priority+ navigation pattern](https://css-tricks.com/the-priority-plus-navigation-pattern/): - **5 high-priority tabs** shown inline: Home, Nodes, Packets, Map, Live - **6 low-priority tabs** collapse into a "More ▾" dropdown: Channels, Traces, Observers, Analytics, Perf, Lab - The "More" button highlights when a low-priority page is active **Desktop (>=1024px)** and **mobile (<768px)** behavior is unchanged. ### Changes | File | Change | |------|--------| | `public/index.html` | Added `data-priority="high"` to 5 primary nav links; added More button + dropdown menu | | `public/style.css` | Split ≤1023px hamburger query into tablet Priority+ (768-1023px) and mobile hamburger (<768px); added More dropdown styles | | `public/app.js` | Added `closeMoreMenu()`, More button toggle, outside-click/Escape close, active state on More button | | Cache busters | Bumped in same commit | ### Accessibility - `aria-haspopup="true"` and `aria-expanded` on More button - `role="menu"` / `role="menuitem"` on dropdown - Focus moves to first item on open - Escape key closes dropdown ### Testing - All 308 existing tests pass (217 frontend-helpers + 62 packet-filter + 29 aging) - No new dependencies added - No build step changes ### Breakpoint summary | Viewport | Behavior | |----------|----------| | >= 1024px | Full horizontal nav (unchanged) | | 768-1023px | Priority+ pattern: 5 tabs + More dropdown **← NEW** | | < 768px | Hamburger drawer with all items (unchanged) | --------- Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>v3.2.0 |
||
|
Kpa-clawbotandCopilot
|
711889c823 |
chore: bump version to 3.2.0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Kpa-clawbotandCopilot
|
738d5fef39 |
fix: poller uses store max IDs to prevent replaying entire DB
When GetMaxTransmissionID() fails silently (e.g., corrupted DB returns 0 from COALESCE), the poller starts from ID 0 and replays the entire database over WebSocket — broadcasting thousands of old packets per second. Fix: after querying the DB, use the in-memory store's MaxTransmissionID and MaxObservationID as a floor. Since Load() already read the full DB successfully, the store has the correct max IDs. Root cause discovered on staging: DB corruption caused MAX(id) query to fail, returning 0. Poller log showed 'starting from transmission ID 0' followed by 1000-2000 broadcasts per tick walking through 76K rows. Also adds MaxObservationID() to PacketStore for observation cursor safety. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
8e6fc9602f |
fix: stabilize Playwright packets test with explicit time window (#348)
## Summary
Fixes the Playwright CI regression on master where the "Packets page
loads with filter" test times out after 15 seconds waiting for able
tbody tr to appear.
## Root Cause
Three packets tests used an bout:blank round-trip pattern to force a
full page reload:
`
page.goto(BASE) → set localStorage → page.goto('about:blank') →
page.goto(BASE/#/packets)
`
This cross-origin round-trip through bout:blank causes the SPA's config
fetch and router to not fire reliably in CI's headless Chromium, leaving
the page uninitialized past the 15-second timeout.
## Fix
Replace the bout:blank pattern with page.reload() in all three affected
tests:
`
page.goto(BASE/#/packets) → set localStorage → page.reload()
`
This stays on the same origin throughout. Playwright handles same-origin
reloads predictably — the page fully re-initializes, the IIFE re-reads
localStorage, and loadPackets() uses the correct time window.
## Tests affected
| Test | Change |
|------|--------|
| Packets page loads with filter | bout:blank → page.reload() |
| Packets initial fetch honors persisted time window | bout:blank →
page.reload() |
| Packets groupByHash toggle works | bout:blank → page.reload() |
## Validation
- All 318 unit tests pass (packet-filter: 62, aging: 29, frontend: 227)
- No public/ files changed — no cache buster needed
- Single file changed: est-e2e-playwright.js (9 insertions, 15
deletions)
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Kpa-clawbotandCopilot
|
e2556eaaff |
fix: filter WebSocket packets by time window on packets page
WS broadcast pushes all packets regardless of the selected time window filter. This caused old packets to appear in the table even when the API correctly returned zero results for the time range. Add time window check to the WS packet filter — drops packets with timestamps older than the selected window cutoff. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Kpa-clawbotandCopilot
|
e7232c0d29 |
fix: rename shadowed isMobile in renderLeft to avoid TDZ crash
The outer IIFE declares const isMobile (line 27, from #340) and renderLeft() declares its own const isMobile (line 821, pre-existing). JavaScript hoists const declarations within the function scope, so referencing isMobile at line 574 (inside renderLeft but before line 821) throws 'Cannot access isMobile before initialization'. Rename the inner declaration to isNarrow since it uses a different breakpoint (640px for column hiding vs 1024px for packet limit). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
f7c182c5f7 |
fix: packets page crash on mobile — time filter default and limit cap (#340)
## Summary Fixes #326 — the packets page crashes mobile browsers (iOS Safari, Edge) by loading 50K+ packets when no time filter is persisted in localStorage. ## Root Cause Two problems in public/packets.js: ### Bug 1: savedTimeWindowMin defaults to 0 instead of 15 localStorage.getItem('meshcore-time-window') returns ull when never set. Number(null) = 0. The guard checked < 0 but not <= 0, so savedTimeWindowMin = 0 meant "All time" — fetching all 50K+ packets. **Fix:** Changed < 0 to <= 0 in both the initialization guard (line 30) and the change handler (line 758). ### Bug 2: No mobile protection against large packet loads Even with valid large time windows, mobile browsers crash under the weight of thousands of DOM rows and packet data (~1.4 GB WebKit memory limit). **Fix:** - Detect mobile viewport: window.innerWidth <= 768 - Cap limit at 1000 on mobile (vs 50000 on desktop) - Disable 6h/12h/24h options and hide "All time" on mobile - Reset persisted windows >3h to 15 min on mobile ## Testing Added 9 unit tests in est-frontend-helpers.js covering: - savedTimeWindowMin defaults to 15 when localStorage returns null - savedTimeWindowMin defaults to 15 when localStorage returns "0" - Valid values (60) are preserved - Negative and NaN values default to 15 - PACKET_LIMIT is 1000 on mobile, 50000 on desktop - Mobile caps large time windows (1440 → 15) but allows 180 All 218 frontend helper tests pass. Packet filter (62) and aging (29) tests also pass. ## Changes | File | Change | |------|--------| | public/packets.js | Fix <= 0 guard, add mobile detection, cap limit, restrict time options | | public/index.html | Cache buster bump | | est-frontend-helpers.js | 9 new regression tests for time window defaults and mobile caps | --------- Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
2d8203ae17 |
fix: extend responsive nav hamburger breakpoint to 1024px (#343)
## Summary Extends the hamburger menu activation breakpoint from max-width: 640px to max-width: 1023px, making all 11 nav items accessible on tablets and small laptops where they were previously clipped/invisible. Fixes #322 ## Changes ### public/style.css - New @media (max-width: 1023px) block activates the hamburger menu and vertical drawer - Drawer has max-height: calc(100dvh - 52px) with overflow-y: auto for scrollability - z-index set to 1100 (consistent with nav layer) - ody.nav-open locks background scroll when drawer is open - Mobile-only rules (brand-text hidden, tighter nav-right gap) remain at 640px ### public/app.js - Extracted closeNav() helper for consistent drawer close behavior - Hamburger toggle now adds/removes ody.nav-open class - Drawer closes on: nav link click, Escape key, and route change (SPA navigation) ### public/index.html - Cache busters bumped for all CSS/JS assets ## What's NOT changed - Desktop layout (>=1024px) is completely untouched - No Priority+ pattern (Phase 2) - No map layout changes (Phase 3) - No new dependencies ## Testing - All 308 frontend tests pass ( est-frontend-helpers.js, est-packet-filter.js, est-aging.js) - Visual verification: hamburger activates at <=1023px, full bar at >=1024px --------- Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
4cdc554b40 |
fix: use latest advert for node hash size instead of historical mode (#341)
## Summary Fixes #303 — Repeater hash stats now reflect the **latest advert** instead of the historical mode (most frequent). When a node is reconfigured (e.g. from 1-byte to 2-byte hash size), the analytics and node detail pages now show the updated value immediately after the next advert is received. ## Changes ### cmd/server/store.go 1. **computeNodeHashSizeInfo** — Changed hash size determination from statistical mode to latest advert. The most recent advert in chronological order now determines hash_size. The hash_sizes_seen and hash_size_inconsistent tracking is preserved for multi-byte analytics. 2. **computeAnalyticsHashSizes** — Two fixes: - **yNode keyed by pubKey** instead of name, so same-name nodes with different public keys are counted separately in distributionByRepeaters. - **Zero-hop adverts included** — advert originator tracking now happens before the hops check, so zero-hop adverts contribute to per-node stats. ### cmd/server/routes_test.go Added 4 new tests: - TestGetNodeHashSizeInfoLatestWins — 4 historical 1-byte adverts + 1 recent 2-byte advert → hash size should be 2 (not 1 from mode) - TestGetNodeHashSizeInfoNoAdverts — node with no ADVERT packets → graceful nil, no crash - TestAnalyticsHashSizeSameNameDifferentPubkey — two nodes named "SameName" with different pubkeys → counted as 2 separate entries - Updated TestGetNodeHashSizeInfoDominant comment to reflect new behavior ## Context Community report from contributor @kizniche: after reconfiguring a repeater from 1-byte to 2-byte hash and sending a flood advert, the analytics page still showed 1-byte. Root cause was the mode-based computation which required many new adverts to shift the majority. The upstream firmware bug causing stale path bytes (meshcore-dev/MeshCore#2154) has been fixed, making the latest advert reliable. ## Testing - `go vet ./...` — clean - `go test ./... -count=1` — all tests pass (including 4 new ones) - `cmd/ingestor` tests — pass --------- Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
81bf3b4b12 |
fix: improve side pane contrast in dark mode (#334) (#342)
## Summary Fixes the poor contrast in the node side pane's "Paths through this node" section in dark mode. ## Root Cause .node-detail-section (side pane) had no background or border — it inherited the lighter --detail-bg (#232340) from .panel-right. The same content on the full detail page sits inside .node-full-card which uses the darker --card-bg (#1a1a2e) + a visible border, giving it proper contrast. | Context | Container | Background | Contrast | |---------|-----------|------------|----------| | Full detail page | .node-full-card | --card-bg (darker) | ✅ Good | | Side pane | .node-detail-section | inherited --detail-bg (lighter) | ❌ Poor | ## Fix Give .node-detail-section the same card treatment as .node-full-card: `css .node-detail-section { background: var(--card-bg); border: 1px solid var(--border); border-radius: 8px; padding: 12px; margin-bottom: 8px; } ` - All colors use CSS variables — no hardcoded hex values - Both light and dark themes benefit from the card treatment - No JS changes needed — CSS-only fix - Cache busters bumped in the same commit Fixes #334 Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
ce6e8d5237 |
feat: show transport code (T_FLOOD) in packets view (#337)
## Summary Surfaces transport route types in the packets view by adding a **"T" badge** next to the payload type badge for packets with `TRANSPORT_FLOOD` (route type 0) or `TRANSPORT_DIRECT` (route type 3) routes. This helps mesh analysis — communities can quickly identify transported packets and gain insights into scope usage adoption. Closes #241 ## What Changed ### Frontend (`public/`) - **app.js**: Added `isTransportRoute(rt)` and `transportBadge(rt)` helper functions that render a `<span class="badge badge-transport">T</span>` badge with the full route type name as a tooltip - **packets.js**: Applied `transportBadge()` in all three packet row render paths: - Flat (ungrouped) packet rows - Grouped packet header rows - Grouped packet child rows - **style.css**: Added `.badge-transport` class with amber styling and CSS variable support (`--transport-badge-bg`, `--transport-badge-fg`) for theme customization ### Backend (`cmd/server/`) - **decoder_test.go**: Added 6 new tests covering: - `TestDecodeHeader_TransportFlood` — verifies route type 0 decodes as TRANSPORT_FLOOD - `TestDecodeHeader_TransportDirect` — verifies route type 3 decodes as TRANSPORT_DIRECT - `TestDecodeHeader_Flood` — verifies route type 1 (non-transport) decodes correctly - `TestIsTransportRoute` — verifies the helper identifies transport vs non-transport routes - `TestDecodePacket_TransportFloodHasCodes` — verifies transport codes are extracted from T_FLOOD packets - `TestDecodePacket_FloodHasNoCodes` — verifies FLOOD packets have no transport codes ## Visual In the packets table Type column, transport packets now show: ``` [Channel Msg] [T] ← transport packet [Channel Msg] ← normal flood packet ``` The "T" badge has an amber color scheme and shows the full route type name on hover. ## Tests - All Go tests pass (`cmd/server` and `cmd/ingestor`) - All frontend tests pass (`test-packet-filter.js`, `test-aging.js`, `test-frontend-helpers.js`) - Cache busters bumped in `index.html` --------- Co-authored-by: you <you@example.com> Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
4898541bce |
fix(ingestor): observer metadata nested stats + SNR/RSSI case fallback (#336)
## Problem Two data integrity bugs in the Go ingestor cause observer metadata and signal quality data to be missing for all Go-backend users. ### #320 — Observer metadata never populated `extractObserverMeta()` reads `battery_mv`, `uptime_secs`, and `noise_floor` from the **top level** of the MQTT status message. However, the actual MQTT payload nests these under a `stats` object: ```json { "status": "online", "origin": "ObserverName", "model": "Heltec V3", "firmware_version": "v1.14.0-9f1a3ea", "stats": { "battery_mv": 4174, "uptime_secs": 80277, "noise_floor": -110 } } ``` Result: battery, uptime, and noise floor are always NULL in the database. ### #321 — SNR and RSSI always missing on raw packets The raw packet handler reads `msg["SNR"]` and `msg["RSSI"]` (uppercase only). Some MQTT bridges send these as lowercase `snr`/`rssi`. The companion BLE handler already has a case-insensitive fallback — the raw packet path did not. Result: SNR/RSSI are NULL for all raw packet observations from bridges that use lowercase keys. ## Fix ### #320 — Nested stats with top-level fallback - Added `nestedOrTopLevel()` helper that checks `msg["stats"][key]` first, then `msg[key]` - `extractObserverMeta` now uses this helper for `battery_mv`, `uptime_secs`, `noise_floor` - Top-level fallback preserved for backward compatibility with bridges that flatten the structure - Safe type assertion: `stats, _ := msg["stats"].(map[string]interface{})` — no crash if stats is missing or wrong type ### #321 — Lowercase SNR/RSSI fallback - Raw packet handler now uses `else if` to check lowercase `snr`/`rssi` when uppercase keys are absent - Matches the pattern already used in the companion channel and direct message handlers ## Tests 10 new test cases added: | Test | What it verifies | |------|-----------------| | `TestExtractObserverMetaNestedStats` | All 5 fields populated from nested stats object | | `TestExtractObserverMetaNestedStatsPrecedence` | Nested stats wins over top-level when both present | | `TestExtractObserverMetaFlatFallback` | Flat structure still works (backward compat) | | `TestExtractObserverMetaEmptyStats` | Empty stats object — no crash, model still works | | `TestExtractObserverMetaStatsNotAMap` | stats is a string — no crash, falls back to top-level | | `TestExtractObserverMetaNoiseFloorFloat` | Float precision preserved (noise_floor REAL migration) | | `TestHandleMessageWithLowercaseSNRRSSI` | Lowercase snr/rssi both stored correctly | | `TestHandleMessageSNRRSSIUppercaseWins` | When both cases present, uppercase takes precedence | | `TestHandleMessageNoSNRRSSI` | Neither key present — nil, no crash | | Existing `TestExtractObserverMeta` | Still passes (flat structure backward compat) | All tests pass: `go test ./... -count=1` and `go vet ./...` clean. Closes #320 Closes #321 --------- Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
38e5f02a00 |
ci: add Docker image cleanup to prevent runner disk exhaustion (#333)
## Problem The self-hosted runner (`meshcore-runner-2`) filled its 29GB disk to 100%, blocking all CI runs: ``` Filesystem Size Used Avail Use% /dev/root 29G 29G 2.3M 100% Docker Images: 67 total, 2 active, 18.83GB reclaimable (99%) ``` Root cause: no Docker image cleanup after builds. Each CI run builds a new image but never prunes old ones. ## Fix ### 1. Docker image cleanup after deploy (`deploy` job) - Runs with `if: always()` so it executes even if deploy fails - `docker image prune -af --filter "until=24h"` — removes images older than 24h (safe: current build is minutes old) - `docker builder prune -f --keep-storage=1GB` — caps build cache - Logs before/after `docker system df` for visibility ### 2. Runner log cleanup at start of E2E job - Prunes runner diagnostic logs older than 3 days (was 53MB and growing) - Reports `df -h` for disk visibility in CI output ## Impact After manual cleanup today, disk went from 100% → 35% (19GB free). This PR prevents recurrence. ## Test plan - [x] Manual cleanup verified on runner via `az vm run-command` - [ ] Next CI run should show cleanup step output in deploy job logs Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
TC²
|
dc9d6ba8df | Fix directory name in README for cloning (#327) | ||
|
efitenandClaude Sonnet 4.6
|
fe314be3a8 |
feat: geo_filter enforcement, DB pruning, geofilter-builder tool, HB column (#215)
## Summary
Several features and fixes from a live deployment of the Go v3.0.0
backend.
### geo_filter — full enforcement
- **Go backend config** (`cmd/server/config.go`,
`cmd/ingestor/config.go`): added `GeoFilterConfig` struct so
`geo_filter.polygon` and `bufferKm` from `config.json` are parsed by
both the server and ingestor
- **Ingestor** (`cmd/ingestor/geo_filter.go`, `cmd/ingestor/main.go`):
ADVERT packets from nodes outside the configured polygon + buffer are
dropped *before* any DB write — no transmission, node, or observation
data is stored
- **Server API** (`cmd/server/geo_filter.go`, `cmd/server/routes.go`):
`GET /api/config/geo-filter` endpoint returns the polygon + bufferKm to
the frontend; `/api/nodes` responses filter out any out-of-area nodes
already in the DB
- **Frontend** (`public/map.js`, `public/live.js`): blue polygon overlay
(solid inner + dashed buffer zone) on Map and Live pages, toggled via
"Mesh live area" checkbox, state shared via localStorage
### Automatic DB pruning
- Add `retention.packetDays` to `config.json` to delete transmissions +
observations older than N days on a daily schedule (1 min after startup,
then every 24h). Nodes and observers are never pruned.
- `POST /api/admin/prune?days=N` for manual runs (requires `X-API-Key`
header if `apiKey` is set)
```json
"retention": {
"nodeDays": 7,
"packetDays": 30
}
```
### tools/geofilter-builder.html
Standalone HTML tool (no server needed) — open in browser, click to
place polygon points on a Leaflet map, set `bufferKm`, copy the
generated `geo_filter` JSON block into `config.json`.
### scripts/prune-nodes-outside-geo-filter.py
Utility script to clean existing out-of-area nodes from the database
(dry-run + confirm). Useful after first enabling geo_filter on a
populated DB.
### HB column in packets table
Shows the hop hash size in bytes (1–4) decoded from the path byte of
each packet's raw hex. Displayed as **HB** between Size and Type
columns, hidden on small screens.
## Test plan
- [x] ADVERT from node outside polygon is not stored (no new row in
nodes or transmissions)
- [x] `GET /api/config/geo-filter` returns polygon + bufferKm when
configured, `{polygon: null, bufferKm: 0}` when not
- [x] `/api/nodes` excludes nodes outside polygon even if present in DB
- [x] Map and Live pages show blue polygon overlay when configured;
checkbox toggles it
- [x] `retention.packetDays: 30` deletes old transmissions/observations
on startup and daily
- [x] `POST /api/admin/prune?days=30` returns `{deleted: N, days: 30}`
- [x] `tools/geofilter-builder.html` opens standalone, draws polygon,
copies valid JSON
- [x] HB column shows 1–4 for all packets in grouped and flat view
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Kpa-clawbotandCopilot
|
3c97a424de |
chore: bump version to 3.1.0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>v3.1.0 |
||
|
|
424b054039 |
Fix packets Last X time filter persistence on reload (#312)
## Summary
- fix packets initial load to honor persisted `meshcore-time-window`
before the filter UI is rendered
- keep the dropdown and effective query window in sync via a shared
`savedTimeWindowMin` value
- add a frontend regression test to ensure `loadPackets()` falls back to
persisted time window when `#fTimeWindow` is not yet present
- bump cache busters in `public/index.html`
## Root cause
`loadPackets()` could run before the filter bar existed, so
`document.getElementById('fTimeWindow')` was null and it fell back to
`15` minutes even though localStorage had a different saved value.
## Testing
- `node test-frontend-helpers.js`
- `node test-packet-filter.js`
- `node test-aging.js`
---------
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
|
1624d6e244 |
fix: unprotect /api/decode from API key auth (#323)
## Fix: unprotect /api/decode from API key auth Fixes #304 ### Problem PR #283 applied `requireAPIKey` to all POST endpoints including `/api/decode`. But BYOP decode is a stateless read-only decoder — it never writes to the database. Users see "write endpoints disabled" when trying to decode packets. ### Fix - Removed `requireAPIKey` wrapper from `/api/decode` in `cmd/server/routes.go` - Updated auth tests to use `/api/perf/reset` (actual write endpoint) instead of `/api/decode` - Added tests proving `/api/decode` works without API key, even when apiKey is configured or empty ### Note Decoder consolidation (`internal/decoder/` shared package) is tracked separately and not included here to keep the PR clean. ### Tests - `cd cmd/server && go test ./...` ✅ Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
8534cfdcc7 |
Fix reopened #284 customizer home regression (#317)
## Summary - prevent customizer panel open from auto-saving before initialization completes - stop `autoSave()` from mutating `window.SITE_CONFIG.home` - rehydrate `userTheme.home` from localStorage into `window.SITE_CONFIG` during app boot - add frontend regression tests for auto-save guard and home rehydration merge - bump `public/index.html` cache busters for updated frontend assets ## Validation - `npm run test:unit` Fixes #284 --------- Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
1211d2fcbf |
fix(manage): migrate legacy config path during start/update (#316)
## Summary
- simplified `migrate_config` in `manage.sh` to only migrate legacy
`./config.json` into `${PROD_DATA}/config.json`
- removed the custom destination-path prompt to avoid suggesting
unsupported runtime paths
- added mode-aware behavior:
- `cmd_start` calls `migrate_config "$PROD_DATA"` (interactive prompt)
- `cmd_update` calls `migrate_config "$PROD_DATA" auto` (non-interactive
auto-migration)
- improved interactive migration copy with explicit source/destination
and runtime note
- when migration is declined from `cmd_start`, script aborts with clear
instructions
## Behavior details
- If `${PROD_DATA}/config.json` exists: no-op
- If only `./config.json` exists:
- `cmd_start` shows:
- `Source: ./config.json (repo root)`
- `Destination: ${PROD_DATA}/config.json`
- `Note: CoreScope reads config from the data directory at runtime.`
- prompt: `Move to ${PROD_DATA}/config.json? [Y/n]`
- yes/default: copies to `${PROD_DATA}/config.json`
- no: aborts with guidance to move the file and rerun
- `cmd_update`: auto-copies with:
- `→ Migrating config.json from repo root to
${PROD_DATA}/config.json...`
- `✓ Config migrated.`
- If neither exists: existing `ensure_config` fallback remains in place
## Validation performed
- ✅ Target exists (`${PROD_DATA}/config.json`) → no-op
- ✅ Legacy exists (`./config.json`), `cmd_start` → prompts, copies on
yes
- ✅ Legacy exists (`./config.json`), `cmd_update` → auto-copies without
prompt
- ✅ Both missing → `ensure_config` fallback flow still handles config
creation/abort path
- ✅ `bash -n manage.sh` passes
## Tests run
- `bash -n manage.sh`
- `node test-packet-filter.js`
- `node test-aging.js`
- `node test-frontend-helpers.js`
- `go test ./...` in `cmd/server`
- `go test ./...` in `cmd/ingestor`
---------
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
|
114b6eea1f |
Show build age next to commit hash in UI (#311)
## Summary - show relative build age next to the commit hash in the nav stats version badge (e.g. `abc1234 (3h ago)`) - use `stats.buildTime` from `/api/stats` and existing `timeAgo()` formatting in `public/app.js` - keep behavior unchanged when `buildTime` is missing/unknown ## What changed - updated `formatVersionBadge()` signature to accept `buildTime` - appended a `build-age` span after the commit link when `buildTime` is valid - passed `stats.buildTime` from `updateNavStats()` - updated frontend helper tests for the new function signature - added regression tests for build-age rendering/skip behavior - bumped cache busters in `public/index.html` ## API check - verified Go server already exposes `buildTime` on `/api/stats` and `/api/health` via `cmd/server/routes.go` - no backend API changes required ## Tests - `node test-frontend-helpers.js` - `node test-packet-filter.js` - `node test-aging.js` All passed locally. ## Browser validation - Not run in this environment (no browser session available). Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Kpa-clawbot
|
90db37e07f |
Backend: channel region filtering for messages API (#310)
## Backend: Channel region filtering Refs #280 — implements backend scope from the [Final Spec & Decision Record](https://github.com/Kpa-clawbot/CoreScope/issues/280#issuecomment-4159480529). ### What changed - `cmd/server/routes.go` — `handleChannelMessages` reads `?region=` param, passes to store - `cmd/server/store.go` — `GetChannelMessages` filters by observer IDs before JSON unmarshal - `cmd/server/db.go` — SQL region prefilter with EXISTS/JOIN on observers.iata, IATA normalization (`TrimSpace + ToUpper`) - `cmd/ingestor/db.go` — IATA normalization on observer upsert write path - Multi-region support: `?region=` accepts comma-separated lists ### Tests - `/api/channels/{hash}/messages?region=X` returns only matching messages - Cross-region same-channel text excluded - IATA case normalization verified - `cd cmd/server && go test ./...` ✅ - `cd cmd/ingestor && go test ./...` ✅ |
||
|
Kpa-clawbotandCopilot
|
b51ced8655 |
Wire channel region filtering end-to-end
Pass region through channel message routes, apply DB/store filtering, normalize IATA at read and write boundaries, and add regression coverage for routes/server/ingestor. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Kpa-clawbot
|
be217981c2 |
docs: add AGENTS.md rule 13 — git worktree isolation (#315)
Adds rule 13 to AGENTS.md: mandatory git worktree isolation for parallel agents. - Implementation agents must work in dedicated worktrees, never the main checkout - Review agents must read remote branches via `git show origin/<branch>:`, never the working tree - Prevents the stale-worktree bug that caused multiple incorrect reviews tonight |