Commit Graph

1154 Commits

Author SHA1 Message Date
efiten e47c39ffda fix: null-guard animLayer and liveAnimCount in nextHop after destroy (#462)
## Summary
- `nextHop()` schedules `setInterval`/`setTimeout` callbacks that can
fire after `destroy()` has set `animLayer = null` and removed DOM
elements
- This caused three console errors on the Live page when navigating away
mid-animation: `Cannot read properties of null (reading 'hasLayer')` and
`Cannot set properties of null (setting 'textContent')`
- Added null guards at each async callback site; no behavioral change
when the page is active

## Changes
- `public/live.js`: early return if `animLayer` is null at start of
`nextHop()`; null-safe `animLayer.hasLayer` checks in
`setInterval`/`setTimeout`; null-safe `liveAnimCount` element access
- `public/index.html`: cache buster bumped
- `test-frontend-helpers.js`: 4 source-inspection tests verifying the
null guards are present

## Test plan
- [ ] Open Live page, trigger some packet animations, navigate away
quickly — no console errors
- [ ] `node test-frontend-helpers.js` passes (233 tests, 0 failures)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-02 01:52:34 +00:00
efiten 1499a55ba7 perf: upsert known nodes in-place on ADVERT, skip full reload (#461)
## Problem

Fixes #399. On every ADVERT WebSocket batch the nodes page invalidated
the entire `_allNodes` cache and triggered a full `/nodes?limit=5000`
fetch — even when every advertising node was already cached. The 90s API
TTL was actively bypassed.

## Root cause

```js
wsHandler = debouncedOnWS(function (msgs) {
  if (msgs.some(isAdvertMessage)) {
    _allNodes = null;               // wipe cache unconditionally
    invalidateApiCache('/nodes');   // bust API TTL
    loadNodes(true);                // full 5k fetch
  }
}, 5000);
```

## Fix

ADVERT decoded payloads include `pubKey`, `name`, `lat`, `lon` — enough
to update known nodes in place:

- **Known node** (pubKey found in `_allNodes`): upsert `name`, `lat`,
`lon`, `last_seen` directly — no fetch, no cache bust, just re-render.
- **New node** (pubKey not in cache) or **no pubKey** in payload: fall
back to full reload as before.

This covers the common case on an active mesh: all advertising nodes are
already cached. The full reload path is preserved for node discovery.

## Tests

2 new unit tests: known-node upsert (asserts 0 API calls, fields
updated) and unknown-node fallback (asserts full reload triggered). All
231 tests pass.

## Checklist
- [x] Branches from `upstream/master`
- [x] No Matomo or local-only commits
- [x] Cache busters bumped
- [x] 231 tests pass, 0 fail

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-04-02 01:47:56 +00:00
efiten f71e117cdd fix: reset restores home steps after SITE_CONFIG contamination (#460)
## Problem

Fixes #325. Removing all home steps and clicking "Reset my theme" did
not restore them.

## Root cause

Two-part bug:

**1. `SITE_CONFIG.home` permanently mutated at page load**
`app.js` calls `mergeUserHomeConfig(SITE_CONFIG, userTheme)` which does
`SITE_CONFIG.home = Object.assign({}, serverHome, userTheme.home)`. If
the user had `steps: []` saved in localStorage, this sets
`SITE_CONFIG.home.steps = []` globally — permanently for the lifetime of
the page.

**2. `initState()` reads the contaminated config**
When the customizer opens (or Reset is clicked), `initState()` reads
`cfg = window.SITE_CONFIG`. Since `SITE_CONFIG.home.steps` is already
`[]`, `state.home.steps` stays `[]` even after
`localStorage.removeItem`. `autoSave()` then re-saves `steps: []`
straight back.

**Secondary issue:** `data-rm-step` / add / move handlers didn't call
`autoSave()`, making step persistence non-deterministic (only saved if a
text field edit happened to be pending).

## Fix

- **`app.js`**: snapshot `SITE_CONFIG.home` before `mergeUserHomeConfig`
→ `window._SITE_CONFIG_ORIGINAL_HOME`
- **`customize.js`**: `initState()` uses `_SITE_CONFIG_ORIGINAL_HOME`
instead of the contaminated `cfg.home`
- **`customize.js`**: add `autoSave()` to rm/move/add handlers for
steps, checklist, and footer links

## Tests

2 new unit tests covering the snapshot bypass and DEFAULTS fallback. 231
tests pass.

## Checklist
- [x] Branches from `upstream/master`
- [x] No Matomo or local-only commits
- [x] Cache busters bumped
- [x] 231 tests pass, 0 fail

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-04-01 18:45:15 -07:00
Kpa-clawbot 75f1295a06 fix: always refresh staging config from prod (#467)
## Summary

Fixes #466 — staging config was not refreshed from prod due to a stale
`-nt` timestamp guard.

## Root Cause

`prepare_staging_config()` only copied prod config when staging was
missing or prod was newer by mtime. However, the `sed -i` that applies
the STAGING siteName updated staging's mtime, making it appear newer
than prod. Subsequent runs skipped the copy entirely.

## Changes

- **`manage.sh`**: Removed the `-nt` timestamp conditional in
`prepare_staging_config()`. Staging config is now always copied fresh
from prod with the STAGING siteName applied.

Note: `prepare_staging_db()` already copies unconditionally — no change
needed there.

Co-authored-by: you <you@example.com>
2026-04-01 18:19:10 -07:00
Kpa-clawbot b1b76acb77 feat: manage.sh update supports pinning to release tags (#456)
## Summary

`manage.sh update` now supports pinning to specific release tags instead
of always pulling tip of master.

Fixes #455

## Changes

### `cmd_update` — accepts optional version argument
- **No argument**: fetches tags, checks out latest release tag (`git tag
-l 'v*' --sort=-v:refname | head -1`)
- **`latest`**: explicit opt-in to tip of master (bleeding edge)
- **Specific tag** (e.g. `v3.1.0`): checks out that exact tag, with
error message + available tags if not found

### `cmd_setup` — defaults to latest tag
- After Docker check, fetches tags and pins to latest release tag
- Skips if already on the latest tag
- Uses state tracking (`version_pin`) so re-runs don't repeat

### `cmd_status` — shows version
- Displays current version (exact tag name or short commit hash) at the
top of status output

### Help text
- Updated to reflect new `update [version]` syntax

## Usage

```bash
./manage.sh update          # checkout latest release tag (e.g. v3.2.0)
./manage.sh update v3.1.0   # pin to specific version
./manage.sh update latest   # explicit tip of master (bleeding edge)
./manage.sh status          # now shows "Version: v3.2.0"
```

## Testing

- `bash -n manage.sh` passes (syntax valid)
- Logic follows existing patterns (git fetch, checkout, rebuild,
restart)

---------

Co-authored-by: you <you@example.com>
2026-04-01 12:20:28 -07:00
Kpa-clawbot f87eb3601c fix: graceful container shutdown for reliable deployments (#453)
## Summary

Fixes #450 — staging deployment flaky due to container not shutting down
cleanly.

## Root Causes

1. **Server never closed DB on shutdown** — SQLite WAL lock held
indefinitely, blocking new container startup
2. **`httpServer.Close()` instead of `Shutdown()`** — abruptly kills
connections instead of draining them
3. **No `stop_grace_period` in compose configs** — Docker sends SIGTERM
then immediately SIGKILL (default 10s is often not enough for WAL
checkpoint)
4. **Supervisor didn't forward SIGTERM** — missing
`stopsignal`/`stopwaitsecs` meant Go processes got SIGKILL instead of
graceful shutdown
5. **Deploy scripts used default `docker stop` timeout** — only 10s
grace period

## Changes

### Go Server (`cmd/server/`)
- **Graceful HTTP shutdown**: `httpServer.Shutdown(ctx)` with 15s
context timeout — drains in-flight requests before closing
- **WebSocket cleanup**: New `Hub.Close()` method sends `CloseGoingAway`
frames to all connected clients
- **DB close on shutdown**: Explicitly closes DB after HTTP server stops
(was never closed before)
- **WAL checkpoint**: `PRAGMA wal_checkpoint(TRUNCATE)` before DB close
— flushes WAL to main DB file and removes WAL/SHM lock files

### Go Ingestor (`cmd/ingestor/`)
- **WAL checkpoint on shutdown**: New `Store.Checkpoint()` method,
called before `Close()`
- **Longer MQTT disconnect timeout**: 5s (was 1s) to allow in-flight
messages to drain

### Docker Compose (all 4 variants)
- Added `stop_grace_period: 30s` and `stop_signal: SIGTERM`

### Supervisor Configs (both variants)
- Added `stopsignal=TERM` and `stopwaitsecs=20` to server and ingestor
programs

### Deploy Scripts
- `deploy-staging.sh`: `docker stop -t 30` with explicit grace period
- `deploy-live.sh`: `docker stop -t 30` with explicit grace period

## Shutdown Sequence (after fix)

1. Docker sends SIGTERM to supervisord (PID 1)
2. Supervisord forwards SIGTERM to server + ingestor (waits up to 20s
each)
3. Server: stops poller → drains HTTP (15s) → closes WS clients →
checkpoints WAL → closes DB
4. Ingestor: stops tickers → disconnects MQTT (5s) → checkpoints WAL →
closes DB
5. Docker waits up to 30s total before SIGKILL

## Tests

All existing tests pass:
- `cd cmd/server && go test ./...` 
- `cd cmd/ingestor && go test ./...` 

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
2026-04-01 12:19:20 -07:00
Kpa-clawbot ec4dd58cb6 fix: null-guard pathHops to prevent detail pane crash (#451) (#454)
## Summary

Fixes #451 — packet detail pane crash on direct routed packets where
`pathHops` is `null`.

## Root Cause

`JSON.parse(pkt.path_json)` can return literal `null` when the DB stores
`"null"` for direct routed packets. The existing code only had a catch
block for parse errors, but `null` is valid JSON — so the parse succeeds
and `pathHops` ends up `null` instead of `[]`.

## Changes

- **`public/packets.js`**: Added `|| []` after `JSON.parse(...)` in both
`buildFlatRowHtml` (table rows) and the detail pane (`selectPacket`),
ensuring `pathHops` is always an array.
- **`test-frontend-helpers.js`**: Added 2 regression tests verifying the
null guards exist in both code paths.
- **`public/index.html`**: Cache buster bump.

## Testing

- All 229 frontend helper tests pass
- All 62 packet filter tests pass
- All 29 aging tests pass

Co-authored-by: you <you@example.com>
2026-04-01 10:48:08 -07:00
Kpa-clawbot 044a5387af perf(packets): virtual scroll + debounced WS renders for packets table (#402)
## Summary

Fixes the critical performance issue where `renderTableRows()` rebuilt
the **entire** table innerHTML (up to 50K rows) on every update —
WebSocket arrivals, filter changes, group expand/collapse, and theme
refreshes.

## Changes

### Lazy Row Generation (`renderVisibleRows`) — fixes #422
- Row HTML strings are **only generated for the visible slice + 30-row
buffer** on each render
- `_displayPackets` stores the filtered data array;
`renderVisibleRows()` calls `buildGroupRowHtml`/`buildFlatRowHtml`
lazily for ~60-90 visible entries
- Previously, `displayPackets.map(buildGroupRowHtml)` built HTML for ALL
30K+ packets on every render — the expensive work (JSON.parse, observer
lookups, template literals) ran for every packet regardless of
visibility

### Unified Row Count via `_getRowCount()` — fixes #424
- Single function `_getRowCount(p)` computes DOM row count for any entry
(1 for flat/collapsed, 1+children for expanded groups)
- Used by BOTH `_rowCounts` computation AND `renderVisibleRows` —
eliminates divergence risk between row counting and row building

### Hoisted Observer Filter Set — fixes #427
- `_observerFilterSet` created once in `renderTableRows()`, reused
across `buildGroupRowHtml`, `_getRowCount`, and child filtering
- Previously, `new Set(filters.observer.split(','))` was created inside
`buildGroupRowHtml` for every packet AND again in the row count callback

### Dynamic Colspan — fixes #426
- `_getColCount()` reads column count from the thead instead of
hardcoded `colspan="11"`
- Spacers and empty-state messages use the actual column count

### Null-Safety in `buildFlatRowHtml` — fixes #430
- `p.decoded_json || '{}'` fallback added, matching
`buildGroupRowHtml`'s existing null-safety
- Prevents TypeError on null/undefined `decoded_json` in flat
(ungrouped) mode

### Behavioral Tests — fixes #428
- Replaced 5 source-grep tests with behavioral unit tests for
`_getRowCount`:
  - Flat mode always returns 1
  - Collapsed group returns 1
  - Expanded group returns 1 + child count
  - Observer filter correctly reduces child count
  - Null `_children` handled gracefully
- Retained source-level assertions only where behavioral testing isn't
practical (e.g., verifying lazy generation pattern exists)

### Other Improvements
- Cumulative row offsets cached in `_cumulativeOffsetsCache`,
invalidated on row count changes
- Debounced WebSocket renders (200ms) coalesce rapid packet arrivals
- `destroy()` properly cleans up all virtual scroll state

## Performance Benchmarks — fixes #423

**Methodology:** Row building cost measured by counting
`buildGroupRowHtml` calls per render cycle on 30K grouped packets.

| Scenario | Before (eager) | After (lazy) | Improvement |
|----------|----------------|--------------|-------------|
| Initial render (30K packets) | 30,000 `buildGroupRowHtml` calls | ~90
calls (60 visible + 30 buffer) | **333× fewer calls** |
| Scroll event | 0 calls (pre-built) | ~90 calls (rebuild visible slice)
| Trades O(1) scroll for O(n) initial savings |
| WS packet arrival | 30,000 calls (full rebuild) | ~90 calls (debounced
+ lazy) | **333× fewer calls** |
| Filter change | 30,000 calls | ~90 calls | **333× fewer calls** |
| Memory (row HTML cache) | ~2MB string array for 30K packets | 0 (no
cache, build on demand) | **~2MB saved** |

**Per-call cost of `buildGroupRowHtml`:** Each call performs JSON.parse
of `decoded_json`, `path_json`, `observers.find()` lookup, and template
literal construction. At 30K packets, the eager approach spent
~400-500ms on row building alone (measured via `performance.now()` on
staging data). The lazy approach builds ~90 rows in ~1-2ms.

**Net effect:** `renderTableRows()` goes from O(n) string building +
O(1) DOM insertion to O(1) data assignment + O(visible) string building
+ O(visible) DOM insertion. For n=30K and visible≈60, this is ~333× less
work per render cycle.

**Trade-off:** Scrolling now rebuilds ~90 rows per RAF frame instead of
slicing pre-built strings. This costs ~1-2ms per scroll event, well
within the 16ms frame budget. The trade-off is overwhelmingly positive
since renders happen far more frequently than full-table scrolls.

## Tests

- 247 frontend helper tests pass (including 18 virtual scroll tests)
- 62 packet filter tests pass
- 29 aging tests pass
- Go backend tests pass

## Remaining Debt (tracked in issues)

- #425: Hardcoded `VSCROLL_ROW_HEIGHT=36` and `theadHeight=40` — should
be measured from DOM
- #429: 200ms WS debounce delay — value works well in practice but lacks
formal justification
- #431: No scroll position preservation on filter change or group
expand/collapse

Fixes #380

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
2026-04-01 09:34:04 -07:00
Kpa-clawbot 01ca843309 perf: move collision analysis to server-side endpoint (fixes #386) (#415)
## Summary

Moves the hash collision analysis from the frontend to a new server-side
endpoint, eliminating a major performance bottleneck on the analytics
collision tab.

Fixes #386

## Problem

The collision tab was:
1. **Downloading all nodes** (`/nodes?limit=2000`) — ~500KB+ of data
2. **Running O(n²) pairwise distance calculations** on the browser main
thread (~2M comparisons with 2000 nodes)
3. **Building prefix maps client-side** (`buildOneBytePrefixMap`,
`buildTwoBytePrefixInfo`, `buildCollisionHops`) iterating all nodes
multiple times

## Solution

### New endpoint: `GET /api/analytics/hash-collisions`

Returns pre-computed collision analysis with:
- `inconsistent_nodes` — nodes with varying hash sizes
- `by_size` — per-byte-size (1, 2, 3) collision data:
  - `stats` — node counts, space usage, collision counts
- `collisions` — pre-computed collisions with pairwise distances and
classifications (local/regional/distant/incomplete)
  - `one_byte_cells` — 256-cell prefix map for 1-byte matrix rendering
- `two_byte_cells` — first-byte-grouped data for 2-byte matrix rendering

### Caching

Uses the existing `cachedResult` pattern with a new `collisionCache`
map. Invalidated on `hasNewTransmissions` (same trigger as the
hash-sizes cache) and on eviction.

### Frontend changes

- `renderCollisionTab` now accepts pre-fetched `collisionData` from the
parallel API load
- New `renderHashMatrixFromServer` and `renderCollisionsFromServer`
functions consume server-computed data directly
- No more `/nodes?limit=2000` fetch from the collision tab
- Old client-side functions (`buildOneBytePrefixMap`, etc.) preserved
for test helper exports

## Test results

- `go test ./...` (server):  pass
- `go test ./...` (ingestor):  pass
- `test-packet-filter.js`:  62 passed
- `test-aging.js`:  29 passed
- `test-frontend-helpers.js`:  227 passed

## Performance impact

| Metric | Before | After |
|--------|--------|-------|
| Data transferred | ~500KB (all nodes) | ~50KB (collision data only) |
| Client computation | O(n²) distance calc | None (server-cached) |
| Main thread blocking | Yes (2000 nodes × pairwise) | No |
| Server caching | N/A | 15s TTL, invalidated on new transmissions |

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
2026-04-01 09:21:23 -07:00
Kpa-clawbot 5f50e80931 perf: replace server round-trip with client-side filter for My Nodes toggle (#401)
## Summary

Fixes #381 — The "My Nodes" filter in `packets.js` was making a **server
API call inside `renderTableRows()`** on every render cycle. With
WebSocket updates arriving every few seconds while the toggle was
active, this created continuous unnecessary server load.

## What Changed

**`public/packets.js`** — Replaced the `api('/packets?nodes=...')`
server call with a pure client-side filter:

```js
// Before: server round-trip on every render
const myData = await api('/packets?nodes=' + allKeys.join(',') + '&limit=500');
displayPackets = myData.packets || [];

// After: filter already-loaded packets client-side
displayPackets = displayPackets.filter(p => {
  const dj = p.decoded_json || '';
  return allKeys.some(k => dj.includes(k));
});
```

This uses the exact same matching logic as the server's
`QueryMultiNodePackets()` — a string contains check on `decoded_json`
for each pubkey — but without the network round-trip.

**`test-frontend-helpers.js`** — Added 5 unit tests for the filter
logic:
- Single and multiple pubkey matching
- No matches / empty keys edge case
- Null/empty `decoded_json` handled gracefully

**`public/index.html`** — Cache busters bumped.

## Test Results

- Frontend helpers: **232 passed, 0 failed** (including 5 new tests)
- Packet filter: **62 passed, 0 failed**
- Aging: **29 passed, 0 failed**

Co-authored-by: you <you@example.com>
2026-04-01 08:27:06 -07:00
you 8f3d12eca5 docs: perf claims require proof — benchmarks, timings, or test assertions 2026-04-01 15:08:06 +00:00
you 357f7952f7 docs: add Rule 0 — performance-first mindset in AGENTS.md 2026-04-01 14:55:35 +00:00
Kpa-clawbot 47d081c705 perf: targeted analytics cache invalidation (fixes #375) (#379)
## Problem

Every time new data is ingested (`IngestNewFromDB`,
`IngestNewObservations`, `EvictStale`), **all 6 analytics caches** are
wiped by creating new empty maps — regardless of what kind of data
actually changed. With the poller running every 1 second, this means the
15s cache TTL is effectively bypassed because caches are cleared far
more frequently than they expire.

## Fix

Introduces a `cacheInvalidation` flags struct and
`invalidateCachesFor()` method that selectively clears only the caches
affected by the ingested data:

| Flag | Caches Cleared |
|------|----------------|
| `hasNewObservations` | RF (SNR/RSSI data changed) |
| `hasNewPaths` | Topology, Distance, Subpaths |
| `hasNewTransmissions` | Hash sizes |
| `hasChannelData` | Channels (GRP_TXT payload_type 5) + channels list
cache |
| `eviction` | All (data removed, everything potentially stale) |

### Impact

For a typical ingest cycle with ADVERT/ACK/TXT_MSG packets (no GRP_TXT):
- **Before:** All 6 caches cleared every cycle
- **After:** Channel cache preserved (most common case), hash cache
preserved on observation-only ingestion

For observation-only ingestion (`IngestNewObservations`):
- **Before:** All 6 caches cleared
- **After:** Only RF cache cleared (+ topo/dist/subpath if paths
actually changed)

## Tests

7 new unit tests in `cache_invalidation_test.go` covering:
- Eviction clears all caches
- Observation-only ingest preserves non-RF caches
- Transmission-only ingest clears only hash cache
- Channel data clears only channel cache
- Path changes clear topo/dist/subpath
- Combined flags work correctly
- No flags = no invalidation

All existing tests pass.

### Post-rebase fix

Restored `channelsCacheRes` invalidation that was accidentally dropped
during the refactor. The old code cleared this separate channels list
cache on every ingest, but `invalidateCachesFor()` didn't include it.
Now cleared on `hasChannelData` and `eviction`.

Fixes #375

---------

Co-authored-by: you <you@example.com>
2026-04-01 07:37:39 -07:00
Kpa-clawbot be313f60cb fix: extract score/direction from MQTT, strip units, fix type safety issues (#371)
## Summary

Fixes #353 — addresses all 5 findings from the CoreScope code analysis.

## Changes

### Finding 1 (Major): `score` field never extracted from MQTT
- Added `Score *float64` field to `PacketData` and `MQTTPacketMessage`
structs
- Extract `msg["score"]` with `msg["Score"]` case fallback via
`toFloat64` in all three MQTT handlers (raw packet, channel message,
direct message)
- Pass through to DB observation insert instead of hardcoded `nil`

### Finding 2 (Major): `direction` field never extracted from MQTT
- Added `Direction *string` field to `PacketData` and
`MQTTPacketMessage` structs
- Extract `msg["direction"]` with `msg["Direction"]` case fallback as
string in all three MQTT handlers
- Pass through to DB observation insert instead of hardcoded `nil`

### Finding 3 (Minor): `toFloat64` doesn't strip units
- Added `stripUnitSuffix()` that removes common RF/signal unit suffixes
(dBm, dB, mW, km, mi, m) case-insensitively before `ParseFloat`
- Values like `"-110dBm"` or `"5.5dB"` now parse correctly

### Finding 4 (Minor): Bare type assertions in store.go
- Changed `firstSeen` and `lastSeen` from `interface{}` to typed
`string` variables at `store.go:5020`
- Removed unsafe `.(string)` type assertions in comparisons

### Finding 5 (Minor): `distHopRecord.SNR` typed as `interface{}`
- Changed `distHopRecord.SNR` from `interface{}` to `*float64`
- Updated assignment (removed intermediate `snrVal` variable, pass
`tx.SNR` directly)
- Updated output serialization to use `floatPtrOrNil(h.SNR)` for
consistent JSON output

## Tests Added

- `TestBuildPacketDataScoreAndDirection` — verifies Score/Direction flow
through BuildPacketData
- `TestBuildPacketDataNilScoreDirection` — verifies nil handling when
fields absent
- `TestInsertTransmissionWithScoreAndDirection` — end-to-end: inserts
with score/direction, verifies DB values
- `TestStripUnitSuffix` — covers all supported suffixes, case
insensitivity, and passthrough
- `TestToFloat64WithUnits` — verifies unit-bearing strings parse
correctly

All existing tests pass.

Co-authored-by: you <you@example.com>
2026-04-01 07:26:23 -07:00
efiten 8a0862523d fix: add migration for missing observations.timestamp index (#332)
## Problem

On installations where the database predates the
`idx_observations_timestamp` index, `/api/stats` takes 30s+ because
`GetStoreStats()` runs two full table scans:

```sql
SELECT COUNT(*) FROM observations WHERE timestamp > ?  -- last hour
SELECT COUNT(*) FROM observations WHERE timestamp > ?  -- last 24h
```

The index is only created in the `if !obsExists` block, so any database
where the `observations` table already existed before that code was
added never gets it.

## Fix

Adds a one-time migration (`obs_timestamp_index_v1`) that runs at
ingestor startup:

```sql
CREATE INDEX IF NOT EXISTS idx_observations_timestamp ON observations(timestamp)
```

On large installations this index creation may take a few seconds on
first startup after the upgrade, but subsequent stats queries become
instant.

## Test plan
- [ ] Restart ingestor on an older database and confirm `[migration]
observations timestamp index created` appears in logs
- [ ] Confirm `/api/stats` response time drops from 30s+ to <100ms

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 07:06:54 -07:00
efiten 7e8b30aa1f perf: fix slow /api/packets and /api/channels on large stores (#328)
## Problem

Two endpoints were slow on larger installations:

**`/packets?limit=50000&groupByHash=true` — 16s+**
`QueryGroupedPackets` did two expensive things on every request:
1. O(n × observations) scan per packet to find `latest` timestamp
2. Held `s.mu.RLock()` during the O(n log n) sort, blocking all
concurrent reads

**`/channels` — 13s+**
`GetChannels` iterated all payload-type-5 packets and JSON-unmarshaled
each one while holding `s.mu.RLock()`, blocking all concurrent reads for
the full duration.

## Fix

**Packets (`QueryGroupedPackets`):**
- Add `LatestSeen string` to `StoreTx`, maintained incrementally in all
three observation write paths. Eliminates the per-packet observation
scan at query time.
- Build output maps under the read lock, sort the local copy after
releasing it.
- Cache the full sorted result for 3 seconds keyed by filter params.

**Channels (`GetChannels`):**
- Copy only the fields needed (firstSeen, decodedJSON, region match)
under the read lock, then release before JSON unmarshaling.
- Cache the result for 15 seconds keyed by region param.
- Invalidate cache on new packet ingestion.

## Test plan
- [ ] Open packets page on a large store — load time should drop from
16s to <1s
- [ ] Open channels page — should load in <100ms instead of 13s+
- [ ] `[SLOW API]` warnings gone for both endpoints
- [ ] Packet/channel data is correct (hashes, counts, observer counts)
- [ ] Filters (region, type, since/until) still work correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 07:06:21 -07:00
Kpa-clawbot b2279b230b fix: handle string, uint, and uint64 types in toFloat64 (#352)
## Summary

Fixes #350 — `toFloat64()` silently drops SNR/RSSI values when bridges
send strings instead of numbers.

## Problem

Some MQTT bridges serialize numeric fields (SNR, RSSI, battery_mv, etc.)
as JSON strings like `"-7.5"` instead of numbers. The existing
`toFloat64()` switch only handled `float64`, `float32`, `int`, `int64`,
and `json.Number`, so string values fell through to the default case
returning `(0, false)` — silently dropping the data.

## Changes

- **`cmd/ingestor/main.go`**: Added `string`, `uint`, and `uint64` cases
to `toFloat64()`
- `string`: uses `strconv.ParseFloat(strings.TrimSpace(n), 64)` to
handle whitespace-padded numeric strings
  - `uint` / `uint64`: straightforward numeric conversion
  - Added `strconv` import

- **`cmd/ingestor/main_test.go`**: Updated `TestToFloat64` with new
cases:
- Valid string (`"3.14"`), string with spaces (`" -7.5 "`), string
integer (`"42"`)
  - Invalid string (`"hello"`), empty string
  - `uint(10)`, `uint64(999)`

## Testing

All ingestor tests pass (`go test ./...`).

Co-authored-by: you <you@example.com>
2026-04-01 06:58:27 -07:00
Kpa-clawbot d1cb84b596 feat: Priority+ nav pattern for tablet viewports (768-1023px) (#345)
## Priority+ Navigation Pattern for Tablet Viewports

Phase 2 of responsive nav improvements for #322.

### What this does

On **tablet viewports (768-1023px)**, implements the [Priority+
navigation
pattern](https://css-tricks.com/the-priority-plus-navigation-pattern/):

- **5 high-priority tabs** shown inline: Home, Nodes, Packets, Map, Live
- **6 low-priority tabs** collapse into a "More ▾" dropdown: Channels,
Traces, Observers, Analytics, Perf, Lab
- The "More" button highlights when a low-priority page is active

**Desktop (>=1024px)** and **mobile (<768px)** behavior is unchanged.

### Changes

| File | Change |
|------|--------|
| `public/index.html` | Added `data-priority="high"` to 5 primary nav
links; added More button + dropdown menu |
| `public/style.css` | Split ≤1023px hamburger query into tablet
Priority+ (768-1023px) and mobile hamburger (<768px); added More
dropdown styles |
| `public/app.js` | Added `closeMoreMenu()`, More button toggle,
outside-click/Escape close, active state on More button |
| Cache busters | Bumped in same commit |

### Accessibility

- `aria-haspopup="true"` and `aria-expanded` on More button
- `role="menu"` / `role="menuitem"` on dropdown
- Focus moves to first item on open
- Escape key closes dropdown

### Testing

- All 308 existing tests pass (217 frontend-helpers + 62 packet-filter +
29 aging)
- No new dependencies added
- No build step changes

### Breakpoint summary

| Viewport | Behavior |
|----------|----------|
| >= 1024px | Full horizontal nav (unchanged) |
| 768-1023px | Priority+ pattern: 5 tabs + More dropdown **← NEW** |
| < 768px | Hamburger drawer with all items (unchanged) |

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
v3.2.0
2026-03-31 23:38:54 -07:00
Kpa-clawbot 711889c823 chore: bump version to 3.2.0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 23:36:25 -07:00
Kpa-clawbot 738d5fef39 fix: poller uses store max IDs to prevent replaying entire DB
When GetMaxTransmissionID() fails silently (e.g., corrupted DB returns 0
from COALESCE), the poller starts from ID 0 and replays the entire
database over WebSocket — broadcasting thousands of old packets per second.

Fix: after querying the DB, use the in-memory store's MaxTransmissionID
and MaxObservationID as a floor. Since Load() already read the full DB
successfully, the store has the correct max IDs.

Root cause discovered on staging: DB corruption caused MAX(id) query to
fail, returning 0. Poller log showed 'starting from transmission ID 0'
followed by 1000-2000 broadcasts per tick walking through 76K rows.

Also adds MaxObservationID() to PacketStore for observation cursor safety.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 23:28:56 -07:00
Kpa-clawbot 8e6fc9602f fix: stabilize Playwright packets test with explicit time window (#348)
## Summary

Fixes the Playwright CI regression on master where the "Packets page
loads with filter" test times out after 15 seconds waiting for able
tbody tr to appear.

## Root Cause

Three packets tests used an bout:blank round-trip pattern to force a
full page reload:

`
page.goto(BASE) → set localStorage → page.goto('about:blank') →
page.goto(BASE/#/packets)
`

This cross-origin round-trip through bout:blank causes the SPA's config
fetch and router to not fire reliably in CI's headless Chromium, leaving
the page uninitialized past the 15-second timeout.

## Fix

Replace the bout:blank pattern with page.reload() in all three affected
tests:

`
page.goto(BASE/#/packets)  →  set localStorage  →  page.reload()
`

This stays on the same origin throughout. Playwright handles same-origin
reloads predictably — the page fully re-initializes, the IIFE re-reads
localStorage, and loadPackets() uses the correct time window.

## Tests affected

| Test | Change |
|------|--------|
| Packets page loads with filter | bout:blank → page.reload() |
| Packets initial fetch honors persisted time window | bout:blank →
page.reload() |
| Packets groupByHash toggle works | bout:blank → page.reload() |

## Validation

- All 318 unit tests pass (packet-filter: 62, aging: 29, frontend: 227)
- No public/ files changed — no cache buster needed
- Single file changed: est-e2e-playwright.js (9 insertions, 15
deletions)

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 23:27:51 -07:00
Kpa-clawbot e2556eaaff fix: filter WebSocket packets by time window on packets page
WS broadcast pushes all packets regardless of the selected time
window filter. This caused old packets to appear in the table even
when the API correctly returned zero results for the time range.

Add time window check to the WS packet filter — drops packets
with timestamps older than the selected window cutoff.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 23:09:42 -07:00
Kpa-clawbot e7232c0d29 fix: rename shadowed isMobile in renderLeft to avoid TDZ crash
The outer IIFE declares const isMobile (line 27, from #340) and
renderLeft() declares its own const isMobile (line 821, pre-existing).
JavaScript hoists const declarations within the function scope, so
referencing isMobile at line 574 (inside renderLeft but before line 821)
throws 'Cannot access isMobile before initialization'.

Rename the inner declaration to isNarrow since it uses a different
breakpoint (640px for column hiding vs 1024px for packet limit).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 22:58:34 -07:00
Kpa-clawbot f7c182c5f7 fix: packets page crash on mobile — time filter default and limit cap (#340)
## Summary

Fixes #326 — the packets page crashes mobile browsers (iOS Safari, Edge)
by loading 50K+ packets when no time filter is persisted in
localStorage.

## Root Cause

Two problems in public/packets.js:

### Bug 1: savedTimeWindowMin defaults to 0 instead of 15

localStorage.getItem('meshcore-time-window') returns 
ull when never set. Number(null) = 0. The guard checked < 0 but not <=
0, so savedTimeWindowMin = 0 meant "All time" — fetching all 50K+
packets.

**Fix:** Changed < 0 to <= 0 in both the initialization guard (line 30)
and the change handler (line 758).

### Bug 2: No mobile protection against large packet loads

Even with valid large time windows, mobile browsers crash under the
weight of thousands of DOM rows and packet data (~1.4 GB WebKit memory
limit).

**Fix:**
- Detect mobile viewport: window.innerWidth <= 768
- Cap limit at 1000 on mobile (vs 50000 on desktop)
- Disable 6h/12h/24h options and hide "All time" on mobile
- Reset persisted windows >3h to 15 min on mobile

## Testing

Added 9 unit tests in 	est-frontend-helpers.js covering:
- savedTimeWindowMin defaults to 15 when localStorage returns null
- savedTimeWindowMin defaults to 15 when localStorage returns "0"
- Valid values (60) are preserved
- Negative and NaN values default to 15
- PACKET_LIMIT is 1000 on mobile, 50000 on desktop
- Mobile caps large time windows (1440 → 15) but allows 180

All 218 frontend helper tests pass. Packet filter (62) and aging (29)
tests also pass.

## Changes

| File | Change |
|------|--------|
| public/packets.js | Fix <= 0 guard, add mobile detection, cap limit,
restrict time options |
| public/index.html | Cache buster bump |
| est-frontend-helpers.js | 9 new regression tests for time window
defaults and mobile caps |

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-01 05:39:16 +00:00
Kpa-clawbot 2d8203ae17 fix: extend responsive nav hamburger breakpoint to 1024px (#343)
## Summary

Extends the hamburger menu activation breakpoint from max-width: 640px
to max-width: 1023px, making all 11 nav items accessible on tablets and
small laptops where they were previously clipped/invisible.

Fixes #322

## Changes

### public/style.css
- New @media (max-width: 1023px) block activates the hamburger menu and
vertical drawer
- Drawer has max-height: calc(100dvh - 52px) with overflow-y: auto for
scrollability
- z-index set to 1100 (consistent with nav layer)
- ody.nav-open locks background scroll when drawer is open
- Mobile-only rules (brand-text hidden, tighter nav-right gap) remain at
640px

### public/app.js
- Extracted closeNav() helper for consistent drawer close behavior
- Hamburger toggle now adds/removes ody.nav-open class
- Drawer closes on: nav link click, Escape key, and route change (SPA
navigation)

### public/index.html
- Cache busters bumped for all CSS/JS assets

## What's NOT changed
- Desktop layout (>=1024px) is completely untouched
- No Priority+ pattern (Phase 2)
- No map layout changes (Phase 3)
- No new dependencies

## Testing
- All 308 frontend tests pass ( est-frontend-helpers.js,
est-packet-filter.js, est-aging.js)
- Visual verification: hamburger activates at <=1023px, full bar at
>=1024px

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 22:34:32 -07:00
Kpa-clawbot 4cdc554b40 fix: use latest advert for node hash size instead of historical mode (#341)
## Summary

Fixes #303 — Repeater hash stats now reflect the **latest advert**
instead of the historical mode (most frequent).

When a node is reconfigured (e.g. from 1-byte to 2-byte hash size), the
analytics and node detail pages now show the updated value immediately
after the next advert is received.

## Changes

### cmd/server/store.go

1. **computeNodeHashSizeInfo** — Changed hash size determination from
statistical mode to latest advert. The most recent advert in
chronological order now determines hash_size. The hash_sizes_seen and
hash_size_inconsistent tracking is preserved for multi-byte analytics.

2. **computeAnalyticsHashSizes** — Two fixes:
- **yNode keyed by pubKey** instead of name, so same-name nodes with
different public keys are counted separately in distributionByRepeaters.
- **Zero-hop adverts included** — advert originator tracking now happens
before the hops check, so zero-hop adverts contribute to per-node stats.

### cmd/server/routes_test.go

Added 4 new tests:
- TestGetNodeHashSizeInfoLatestWins — 4 historical 1-byte adverts + 1
recent 2-byte advert → hash size should be 2 (not 1 from mode)
- TestGetNodeHashSizeInfoNoAdverts — node with no ADVERT packets →
graceful nil, no crash
- TestAnalyticsHashSizeSameNameDifferentPubkey — two nodes named
"SameName" with different pubkeys → counted as 2 separate entries
- Updated TestGetNodeHashSizeInfoDominant comment to reflect new
behavior

## Context

Community report from contributor @kizniche: after reconfiguring a
repeater from 1-byte to 2-byte hash and sending a flood advert, the
analytics page still showed 1-byte. Root cause was the mode-based
computation which required many new adverts to shift the majority. The
upstream firmware bug causing stale path bytes
(meshcore-dev/MeshCore#2154) has been fixed, making the latest advert
reliable.

## Testing

- `go vet ./...` — clean
- `go test ./... -count=1` — all tests pass (including 4 new ones)
- `cmd/ingestor` tests — pass

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 22:26:32 -07:00
Kpa-clawbot 81bf3b4b12 fix: improve side pane contrast in dark mode (#334) (#342)
## Summary

Fixes the poor contrast in the node side pane's "Paths through this
node" section in dark mode.

## Root Cause

.node-detail-section (side pane) had no background or border — it
inherited the lighter --detail-bg (#232340) from .panel-right. The same
content on the full detail page sits inside .node-full-card which uses
the darker --card-bg (#1a1a2e) + a visible border, giving it proper
contrast.

| Context | Container | Background | Contrast |
|---------|-----------|------------|----------|
| Full detail page | .node-full-card | --card-bg (darker) |  Good |
| Side pane | .node-detail-section | inherited --detail-bg (lighter) | 
Poor |

## Fix

Give .node-detail-section the same card treatment as .node-full-card:

`css
.node-detail-section {
  background: var(--card-bg);
  border: 1px solid var(--border);
  border-radius: 8px;
  padding: 12px;
  margin-bottom: 8px;
}
`

- All colors use CSS variables — no hardcoded hex values
- Both light and dark themes benefit from the card treatment
- No JS changes needed — CSS-only fix
- Cache busters bumped in the same commit

Fixes #334

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 22:16:20 -07:00
Kpa-clawbot ce6e8d5237 feat: show transport code (T_FLOOD) in packets view (#337)
## Summary

Surfaces transport route types in the packets view by adding a **"T"
badge** next to the payload type badge for packets with
`TRANSPORT_FLOOD` (route type 0) or `TRANSPORT_DIRECT` (route type 3)
routes.

This helps mesh analysis — communities can quickly identify transported
packets and gain insights into scope usage adoption.

Closes #241

## What Changed

### Frontend (`public/`)
- **app.js**: Added `isTransportRoute(rt)` and `transportBadge(rt)`
helper functions that render a `<span class="badge
badge-transport">T</span>` badge with the full route type name as a
tooltip
- **packets.js**: Applied `transportBadge()` in all three packet row
render paths:
  - Flat (ungrouped) packet rows
  - Grouped packet header rows
  - Grouped packet child rows
- **style.css**: Added `.badge-transport` class with amber styling and
CSS variable support (`--transport-badge-bg`, `--transport-badge-fg`)
for theme customization

### Backend (`cmd/server/`)
- **decoder_test.go**: Added 6 new tests covering:
- `TestDecodeHeader_TransportFlood` — verifies route type 0 decodes as
TRANSPORT_FLOOD
- `TestDecodeHeader_TransportDirect` — verifies route type 3 decodes as
TRANSPORT_DIRECT
- `TestDecodeHeader_Flood` — verifies route type 1 (non-transport)
decodes correctly
- `TestIsTransportRoute` — verifies the helper identifies transport vs
non-transport routes
- `TestDecodePacket_TransportFloodHasCodes` — verifies transport codes
are extracted from T_FLOOD packets
- `TestDecodePacket_FloodHasNoCodes` — verifies FLOOD packets have no
transport codes

## Visual

In the packets table Type column, transport packets now show:
```
[Channel Msg] [T]    ← transport packet
[Channel Msg]        ← normal flood packet
```

The "T" badge has an amber color scheme and shows the full route type
name on hover.

## Tests

- All Go tests pass (`cmd/server` and `cmd/ingestor`)
- All frontend tests pass (`test-packet-filter.js`, `test-aging.js`,
`test-frontend-helpers.js`)
- Cache busters bumped in `index.html`

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 18:39:38 -07:00
Kpa-clawbot 4898541bce fix(ingestor): observer metadata nested stats + SNR/RSSI case fallback (#336)
## Problem

Two data integrity bugs in the Go ingestor cause observer metadata and
signal quality data to be missing for all Go-backend users.

### #320 — Observer metadata never populated

`extractObserverMeta()` reads `battery_mv`, `uptime_secs`, and
`noise_floor` from the **top level** of the MQTT status message.
However, the actual MQTT payload nests these under a `stats` object:

```json
{
  "status": "online",
  "origin": "ObserverName",
  "model": "Heltec V3",
  "firmware_version": "v1.14.0-9f1a3ea",
  "stats": {
    "battery_mv": 4174,
    "uptime_secs": 80277,
    "noise_floor": -110
  }
}
```

Result: battery, uptime, and noise floor are always NULL in the
database.

### #321 — SNR and RSSI always missing on raw packets

The raw packet handler reads `msg["SNR"]` and `msg["RSSI"]` (uppercase
only). Some MQTT bridges send these as lowercase `snr`/`rssi`. The
companion BLE handler already has a case-insensitive fallback — the raw
packet path did not.

Result: SNR/RSSI are NULL for all raw packet observations from bridges
that use lowercase keys.

## Fix

### #320 — Nested stats with top-level fallback

- Added `nestedOrTopLevel()` helper that checks `msg["stats"][key]`
first, then `msg[key]`
- `extractObserverMeta` now uses this helper for `battery_mv`,
`uptime_secs`, `noise_floor`
- Top-level fallback preserved for backward compatibility with bridges
that flatten the structure
- Safe type assertion: `stats, _ :=
msg["stats"].(map[string]interface{})` — no crash if stats is missing or
wrong type

### #321 — Lowercase SNR/RSSI fallback

- Raw packet handler now uses `else if` to check lowercase `snr`/`rssi`
when uppercase keys are absent
- Matches the pattern already used in the companion channel and direct
message handlers

## Tests

10 new test cases added:

| Test | What it verifies |
|------|-----------------|
| `TestExtractObserverMetaNestedStats` | All 5 fields populated from
nested stats object |
| `TestExtractObserverMetaNestedStatsPrecedence` | Nested stats wins
over top-level when both present |
| `TestExtractObserverMetaFlatFallback` | Flat structure still works
(backward compat) |
| `TestExtractObserverMetaEmptyStats` | Empty stats object — no crash,
model still works |
| `TestExtractObserverMetaStatsNotAMap` | stats is a string — no crash,
falls back to top-level |
| `TestExtractObserverMetaNoiseFloorFloat` | Float precision preserved
(noise_floor REAL migration) |
| `TestHandleMessageWithLowercaseSNRRSSI` | Lowercase snr/rssi both
stored correctly |
| `TestHandleMessageSNRRSSIUppercaseWins` | When both cases present,
uppercase takes precedence |
| `TestHandleMessageNoSNRRSSI` | Neither key present — nil, no crash |
| Existing `TestExtractObserverMeta` | Still passes (flat structure
backward compat) |

All tests pass: `go test ./... -count=1` and `go vet ./...` clean.

Closes #320
Closes #321

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 17:53:04 -07:00
Kpa-clawbot 38e5f02a00 ci: add Docker image cleanup to prevent runner disk exhaustion (#333)
## Problem

The self-hosted runner (`meshcore-runner-2`) filled its 29GB disk to
100%, blocking all CI runs:

```
Filesystem  Size  Used Avail Use%
/dev/root    29G   29G  2.3M 100%

Docker Images: 67 total, 2 active, 18.83GB reclaimable (99%)
```

Root cause: no Docker image cleanup after builds. Each CI run builds a
new image but never prunes old ones.

## Fix

### 1. Docker image cleanup after deploy (`deploy` job)
- Runs with `if: always()` so it executes even if deploy fails
- `docker image prune -af --filter "until=24h"` — removes images older
than 24h (safe: current build is minutes old)
- `docker builder prune -f --keep-storage=1GB` — caps build cache
- Logs before/after `docker system df` for visibility

### 2. Runner log cleanup at start of E2E job
- Prunes runner diagnostic logs older than 3 days (was 53MB and growing)
- Reports `df -h` for disk visibility in CI output

## Impact

After manual cleanup today, disk went from 100% → 35% (19GB free). This
PR prevents recurrence.

## Test plan
- [x] Manual cleanup verified on runner via `az vm run-command`
- [ ] Next CI run should show cleanup step output in deploy job logs

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 16:00:31 -07:00
TC² dc9d6ba8df Fix directory name in README for cloning (#327) 2026-03-31 08:37:26 -07:00
efiten fe314be3a8 feat: geo_filter enforcement, DB pruning, geofilter-builder tool, HB column (#215)
## Summary

Several features and fixes from a live deployment of the Go v3.0.0
backend.

### geo_filter — full enforcement

- **Go backend config** (`cmd/server/config.go`,
`cmd/ingestor/config.go`): added `GeoFilterConfig` struct so
`geo_filter.polygon` and `bufferKm` from `config.json` are parsed by
both the server and ingestor
- **Ingestor** (`cmd/ingestor/geo_filter.go`, `cmd/ingestor/main.go`):
ADVERT packets from nodes outside the configured polygon + buffer are
dropped *before* any DB write — no transmission, node, or observation
data is stored
- **Server API** (`cmd/server/geo_filter.go`, `cmd/server/routes.go`):
`GET /api/config/geo-filter` endpoint returns the polygon + bufferKm to
the frontend; `/api/nodes` responses filter out any out-of-area nodes
already in the DB
- **Frontend** (`public/map.js`, `public/live.js`): blue polygon overlay
(solid inner + dashed buffer zone) on Map and Live pages, toggled via
"Mesh live area" checkbox, state shared via localStorage

### Automatic DB pruning

- Add `retention.packetDays` to `config.json` to delete transmissions +
observations older than N days on a daily schedule (1 min after startup,
then every 24h). Nodes and observers are never pruned.
- `POST /api/admin/prune?days=N` for manual runs (requires `X-API-Key`
header if `apiKey` is set)

```json
"retention": {
  "nodeDays": 7,
  "packetDays": 30
}
```

### tools/geofilter-builder.html

Standalone HTML tool (no server needed) — open in browser, click to
place polygon points on a Leaflet map, set `bufferKm`, copy the
generated `geo_filter` JSON block into `config.json`.

### scripts/prune-nodes-outside-geo-filter.py

Utility script to clean existing out-of-area nodes from the database
(dry-run + confirm). Useful after first enabling geo_filter on a
populated DB.

### HB column in packets table

Shows the hop hash size in bytes (1–4) decoded from the path byte of
each packet's raw hex. Displayed as **HB** between Size and Type
columns, hidden on small screens.

## Test plan

- [x] ADVERT from node outside polygon is not stored (no new row in
nodes or transmissions)
- [x] `GET /api/config/geo-filter` returns polygon + bufferKm when
configured, `{polygon: null, bufferKm: 0}` when not
- [x] `/api/nodes` excludes nodes outside polygon even if present in DB
- [x] Map and Live pages show blue polygon overlay when configured;
checkbox toggles it
- [x] `retention.packetDays: 30` deletes old transmissions/observations
on startup and daily
- [x] `POST /api/admin/prune?days=30` returns `{deleted: N, days: 30}`
- [x] `tools/geofilter-builder.html` opens standalone, draws polygon,
copies valid JSON
- [x] HB column shows 1–4 for all packets in grouped and flat view

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 01:10:56 -07:00
Kpa-clawbot 3c97a424de chore: bump version to 3.1.0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
v3.1.0
2026-03-31 01:00:41 -07:00
Kpa-clawbot 424b054039 Fix packets Last X time filter persistence on reload (#312)
## Summary
- fix packets initial load to honor persisted `meshcore-time-window`
before the filter UI is rendered
- keep the dropdown and effective query window in sync via a shared
`savedTimeWindowMin` value
- add a frontend regression test to ensure `loadPackets()` falls back to
persisted time window when `#fTimeWindow` is not yet present
- bump cache busters in `public/index.html`

## Root cause
`loadPackets()` could run before the filter bar existed, so
`document.getElementById('fTimeWindow')` was null and it fell back to
`15` minutes even though localStorage had a different saved value.

## Testing
- `node test-frontend-helpers.js`
- `node test-packet-filter.js`
- `node test-aging.js`

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 00:29:36 -07:00
Kpa-clawbot 1624d6e244 fix: unprotect /api/decode from API key auth (#323)
## Fix: unprotect /api/decode from API key auth

Fixes #304

### Problem
PR #283 applied `requireAPIKey` to all POST endpoints including
`/api/decode`. But BYOP decode is a stateless read-only decoder — it
never writes to the database. Users see "write endpoints disabled" when
trying to decode packets.

### Fix
- Removed `requireAPIKey` wrapper from `/api/decode` in
`cmd/server/routes.go`
- Updated auth tests to use `/api/perf/reset` (actual write endpoint)
instead of `/api/decode`
- Added tests proving `/api/decode` works without API key, even when
apiKey is configured or empty

### Note
Decoder consolidation (`internal/decoder/` shared package) is tracked
separately and not included here to keep the PR clean.

### Tests
- `cd cmd/server && go test ./...` 

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 00:13:31 -07:00
Kpa-clawbot 8534cfdcc7 Fix reopened #284 customizer home regression (#317)
## Summary
- prevent customizer panel open from auto-saving before initialization
completes
- stop `autoSave()` from mutating `window.SITE_CONFIG.home`
- rehydrate `userTheme.home` from localStorage into `window.SITE_CONFIG`
during app boot
- add frontend regression tests for auto-save guard and home rehydration
merge
- bump `public/index.html` cache busters for updated frontend assets

## Validation
- `npm run test:unit`

Fixes #284

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 00:01:31 -07:00
Kpa-clawbot 1211d2fcbf fix(manage): migrate legacy config path during start/update (#316)
## Summary
- simplified `migrate_config` in `manage.sh` to only migrate legacy
`./config.json` into `${PROD_DATA}/config.json`
- removed the custom destination-path prompt to avoid suggesting
unsupported runtime paths
- added mode-aware behavior:
  - `cmd_start` calls `migrate_config "$PROD_DATA"` (interactive prompt)
- `cmd_update` calls `migrate_config "$PROD_DATA" auto` (non-interactive
auto-migration)
- improved interactive migration copy with explicit source/destination
and runtime note
- when migration is declined from `cmd_start`, script aborts with clear
instructions

## Behavior details
- If `${PROD_DATA}/config.json` exists: no-op
- If only `./config.json` exists:
  - `cmd_start` shows:
    - `Source: ./config.json (repo root)`
    - `Destination: ${PROD_DATA}/config.json`
    - `Note: CoreScope reads config from the data directory at runtime.`
  - prompt: `Move to ${PROD_DATA}/config.json? [Y/n]`
  - yes/default: copies to `${PROD_DATA}/config.json`
  - no: aborts with guidance to move the file and rerun
  - `cmd_update`: auto-copies with:
- `→ Migrating config.json from repo root to
${PROD_DATA}/config.json...`
    - `✓ Config migrated.`
- If neither exists: existing `ensure_config` fallback remains in place

## Validation performed
-  Target exists (`${PROD_DATA}/config.json`) → no-op
-  Legacy exists (`./config.json`), `cmd_start` → prompts, copies on
yes
-  Legacy exists (`./config.json`), `cmd_update` → auto-copies without
prompt
-  Both missing → `ensure_config` fallback flow still handles config
creation/abort path
-  `bash -n manage.sh` passes

## Tests run
- `bash -n manage.sh`
- `node test-packet-filter.js`
- `node test-aging.js`
- `node test-frontend-helpers.js`
- `go test ./...` in `cmd/server`
- `go test ./...` in `cmd/ingestor`

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 23:32:31 -07:00
Kpa-clawbot 114b6eea1f Show build age next to commit hash in UI (#311)
## Summary
- show relative build age next to the commit hash in the nav stats
version badge (e.g. `abc1234 (3h ago)`)
- use `stats.buildTime` from `/api/stats` and existing `timeAgo()`
formatting in `public/app.js`
- keep behavior unchanged when `buildTime` is missing/unknown

## What changed
- updated `formatVersionBadge()` signature to accept `buildTime`
- appended a `build-age` span after the commit link when `buildTime` is
valid
- passed `stats.buildTime` from `updateNavStats()`
- updated frontend helper tests for the new function signature
- added regression tests for build-age rendering/skip behavior
- bumped cache busters in `public/index.html`

## API check
- verified Go server already exposes `buildTime` on `/api/stats` and
`/api/health` via `cmd/server/routes.go`
- no backend API changes required

## Tests
- `node test-frontend-helpers.js`
- `node test-packet-filter.js`
- `node test-aging.js`

All passed locally.

## Browser validation
- Not run in this environment (no browser session available).

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 23:20:35 -07:00
Kpa-clawbot 90db37e07f Backend: channel region filtering for messages API (#310)
## Backend: Channel region filtering

Refs #280 — implements backend scope from the [Final Spec & Decision
Record](https://github.com/Kpa-clawbot/CoreScope/issues/280#issuecomment-4159480529).

### What changed

- `cmd/server/routes.go` — `handleChannelMessages` reads `?region=`
param, passes to store
- `cmd/server/store.go` — `GetChannelMessages` filters by observer IDs
before JSON unmarshal
- `cmd/server/db.go` — SQL region prefilter with EXISTS/JOIN on
observers.iata, IATA normalization (`TrimSpace + ToUpper`)
- `cmd/ingestor/db.go` — IATA normalization on observer upsert write
path
- Multi-region support: `?region=` accepts comma-separated lists

### Tests
- `/api/channels/{hash}/messages?region=X` returns only matching
messages
- Cross-region same-channel text excluded
- IATA case normalization verified
- `cd cmd/server && go test ./...` 
- `cd cmd/ingestor && go test ./...` 
2026-03-30 23:07:45 -07:00
Kpa-clawbot b51ced8655 Wire channel region filtering end-to-end
Pass region through channel message routes, apply DB/store filtering, normalize IATA at read and write boundaries, and add regression coverage for routes/server/ingestor.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 23:03:56 -07:00
Kpa-clawbot be217981c2 docs: add AGENTS.md rule 13 — git worktree isolation (#315)
Adds rule 13 to AGENTS.md: mandatory git worktree isolation for parallel
agents.

- Implementation agents must work in dedicated worktrees, never the main
checkout
- Review agents must read remote branches via `git show
origin/<branch>:`, never the working tree
- Prevents the stale-worktree bug that caused multiple incorrect reviews
tonight
2026-03-30 23:02:55 -07:00
Kpa-clawbot 1679c0aee1 Merge branch 'master' into docs/agents-rule-13-worktrees 2026-03-30 23:02:41 -07:00
Kpa-clawbot 4ea4c9c0d5 fix: update blame-ignore SHA for LF normalization (#319)
# Update .git-blame-ignore-revs SHA to match the actual squash merge
commit (b6e4ebf) from PR #314.
2026-03-30 22:59:40 -07:00
Kpa-clawbot b39a7e7601 fix: update blame-ignore-revs SHA to match squash merge commit
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 22:59:06 -07:00
Kpa-clawbot b6e4ebf12e chore: normalize all files to LF line endings (#314)
## Summary
This PR performs a **one-time line-ending normalization** with **no
functional changes**.

- Normalized previously CRLF-indexed files to LF using `git add
--renormalize .`
- Added `.git-blame-ignore-revs` to preserve usable blame history for
this bulk formatting-only commit
- Added explicit LF enforcement for shell scripts in `.gitattributes`:
  - `manage.sh text eol=lf`
  - `*.sh text eol=lf`

## Why this is safe
- This is a text normalization pass only (CRLF → LF)
- No logic, behavior, APIs, or runtime paths were changed
- Review diff noise is expected due to line-ending-only rewrites

## File-count context
- There were **148 known CRLF-indexed files** targeted for normalization
- The renormalization pass touched **151 files total** in this
repository snapshot

## Blame preservation
GitHub reads `.git-blame-ignore-revs` natively, so blame views can skip
the normalization commit.

For local git blame setup:

```bash
git config blame.ignoreRevsFile .git-blame-ignore-revs
```

## Validation
Executed required no-regression checks:

```bash
node test-frontend-helpers.js && node test-packet-filter.js && node test-aging.js
```

All passed.
2026-03-30 22:56:56 -07:00
Kpa-clawbot 23a018a16f chore: add blame-ignore-revs for line ending normalization 2026-03-30 22:52:46 -07:00
Kpa-clawbot 5aa4fbb600 chore: normalize all files to LF line endings 2026-03-30 22:52:46 -07:00
Kpa-clawbot a2c9211dac Fix tilde expansion for .env paths in manage.sh (#318)
## Summary
- fix safe .env parser in manage.sh to expand a leading ~ before export
- ensure setup-time PROD_DATA_DIR read from .env also expands ~
- keep behavior unchanged for non-tilde values

## Why
xport "=" does not perform tilde expansion, so values like
PROD_DATA_DIR=~/meshcore-data stayed literal and broke path-based
operations in manage.sh.

## Validation
- ash -n manage.sh
- manual reasoning: PROD_DATA_DIR=~/meshcore-data now resolves to
$HOME/meshcore-data

## Notes
Docker Compose handling is unchanged (compose already expands ~); this
PR only fixes manage.sh runtime parsing.

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 22:05:10 -07:00
Kpa-clawbot bd98a19662 docs: add rule 13 — git worktree isolation for parallel agents
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 21:46:42 -07:00
Kpa-clawbot 4fe6b656c4 chore: add blame-ignore-revs for line ending normalization
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-30 21:46:41 -07:00