dandri/meshcore-analyzer
1
0
Fork 0
mirror of https://github.com/Kpa-clawbot/meshcore-analyzer.git synced 2026-05-28 07:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,188 Commits 81 Branches 28 Tags
d8c76ca47a39dccfb6918ccba3b4bb3dfab04475
Commit Graph

1188 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
you d8c76ca47a fix: customizer v2 type colors default to black (#514) 
- Server: add default typeColors in handleConfigTheme matching roles.js
- Client: fall back to window.TYPE_COLORS before #000000 in _renderNodes
- Tests: verify typeColors defaults in theme API and frontend fallback
 2026-04-03 05:25:57 +00:00
Kpa-clawbot 813b424ca1 fix: Show Neighbors uses affinity API for collision disambiguation (#484) — milestone 3 (#512) 
## Summary

Replace broken client-side path walking in `selectReferenceNode()` with
server-side `/api/nodes/{pubkey}/neighbors` API call, fixing #484 where
Show Neighbors returned zero results due to hash collision
disambiguation failures.

**Fixes #484** | Part of #482

## What changed

### `public/map.js` — `selectReferenceNode()` function

**Before:** Client-side path walking — fetched
`/api/nodes/{pubkey}/paths`, walked each path to find hops adjacent to
the selected node by comparing full pubkeys. This fails on hash
collisions because path hops only contain short prefixes (1-2 bytes),
and the hop resolver can pick the wrong collision candidate.

**After:** Server-side affinity resolution — fetches
`/api/nodes/{pubkey}/neighbors?min_count=3` which uses the neighbor
affinity graph (built in M1/M2) to return disambiguated neighbors. For
ambiguous edges, all candidates are included in the neighbor set (better
to show extra markers than miss real neighbors).

**Fallback:** When the affinity API returns zero neighbors (cold start,
insufficient data), the function falls back to the original path-walking
approach. This ensures the feature works even before the affinity graph
has accumulated enough observations.

## Tests

4 new Playwright E2E tests (in both `test-show-neighbors.js` and
`test-e2e-playwright.js`):

1. **Happy path** — Verifies the `/neighbors` API is called and the
reference node UI activates
2. **Hash collision disambiguation** — Two nodes sharing prefix "C0" get
different neighbor sets via the affinity API (THE critical test for
#484)
3. **Fallback to path walking** — Empty affinity response triggers
fallback to `/paths` API
4. **Ambiguous candidates** — Ambiguous edge candidates are included in
the neighbor set

All tests use Playwright route interception to mock API responses,
testing the frontend logic independently of server state.

## Spec reference

See [neighbor-affinity-graph.md](docs/specs/neighbor-affinity-graph.md),
sections:
- "Replacing Show Neighbors on the map" (lines ~461-504)
- "Milestone 3: Show Neighbors Fix (#484)" (lines ~1136-1152)
- Test specs a & b (lines ~754-800)

---------

Co-authored-by: you <you@example.com>
 2026-04-02 22:04:03 -07:00
Kpa-clawbot e66085092e feat: neighbor affinity API endpoints (#482) — milestone 2 (#508) 
## Summary

Milestone 2 of the neighbor affinity graph (#482). Adds two API
endpoints that expose the neighbor graph built in M1 (PR #507).

### Endpoints

#### `GET /api/nodes/{pubkey}/neighbors`
Returns neighbors for a specific node with affinity scores.

**Query params:** `min_count` (default 1), `min_score` (default 0.0),
`include_ambiguous` (default true)

**Response shape:**
```json
{
  "node": "pubkey",
  "neighbors": [
    { "pubkey": "...", "prefix": "BB", "name": "...", "role": "repeater",
      "count": 847, "score": 0.95, "first_seen": "...", "last_seen": "...",
      "avg_snr": -8.2, "observers": ["obs1"], "ambiguous": false }
  ],
  "total_observations": 847
}
```

Ambiguous entries have `candidates` array; unresolved prefixes have
`unresolved: true`.

#### `GET /api/analytics/neighbor-graph`
Returns full graph summary for analytics/visualization.

**Query params:** `min_count` (default 5), `min_score` (default 0.1),
`region` (IATA code filter)

**Response shape:**
```json
{
  "nodes": [{ "pubkey": "...", "name": "...", "role": "...", "neighbor_count": 5 }],
  "edges": [{ "source": "...", "target": "...", "weight": 847, "score": 0.95, "ambiguous": false }],
  "stats": { "total_nodes": 42, "total_edges": 87, "ambiguous_edges": 3, "avg_cluster_size": 4.2 }
}
```

### Wiring
- `NeighborGraph` + `neighborMu` added to `Server` struct
- Lazy initialization: graph built on first API call, cached with 60s
TTL
- Node name/role lookups via existing `getCachedNodesAndPM()`
- Region filtering via existing `resolveRegionObservers()`

### Tests (15 tests)
- Empty graph, single neighbor, multiple neighbors (sorted by score)
- Ambiguous candidates with candidate list
- Unresolved prefix (orphan) with `unresolved: true`
- `min_count` filter, `min_score` filter, `include_ambiguous=false`
filter
- Unknown node returns 200 with empty neighbors
- Graph endpoint: empty, with edges, default min_count, ambiguous count
- Region filter (graceful when no store)
- Response shape validation (all required keys present)

All existing tests continue to pass.

Part of #482

---------

Co-authored-by: you <you@example.com>
 2026-04-02 21:30:23 -07:00
Kpa-clawbot 4a56be0b48 feat: neighbor affinity graph builder (#482) — milestone 1 (#507) 
## Summary

Milestone 1 of 7 for the neighbor affinity graph feature (#482).
Implements the core `NeighborGraph` data structure and
`BuildFromStore()` algorithm.

**Spec:** `docs/specs/neighbor-affinity-graph.md` on
`spec/482-neighbor-affinity` branch.

## What's Built

### `cmd/server/neighbor_graph.go`
- **`NeighborGraph` struct** — thread-safe (sync.RWMutex) in-memory
graph with edge map and per-node index
- **`BuildFromStore(*PacketStore)`** — iterates all packets/observations
to extract first-hop edges:
- `originator ↔ path[0]` for ADVERT packets only (originator identity
known)
  - `observer ↔ path[last]` for ALL packet types
  - Zero-hop ADVERTs: `originator ↔ observer` direct edge
- **Affinity scoring** — `score = min(1.0, count/100) × exp(-λ × hours)`
with 7-day half-life
- **Jaccard disambiguation** — resolves ambiguous hash prefixes using
mutual-neighbor overlap
- **Confidence threshold** — auto-resolve only when best ≥ 3×
second-best AND ≥ 3 observations
- **Transitivity poisoning guard** — only fully-resolved edges used as
evidence
- **Orphan prefix handling** — unknown prefixes stored as unresolved
markers
- **Cache management** — 60s TTL, `IsStale()` check for rebuild
triggering

### `cmd/server/neighbor_graph_test.go`
22 unit tests covering all spec requirements:

| Test | What it validates |
|------|-------------------|
| EmptyStore | Empty graph from empty store |
| AdvertSingleHopPath | Both edge types from single-hop ADVERT |
| AdvertMultiHopPath | originator↔path[0] + observer↔path[last] |
| AdvertZeroHop | Direct originator↔observer edge |
| NonAdvertEmptyPath | No edges from non-ADVERT empty path |
| NonAdvertOnlyObserverEdge | Only observer↔last_hop for non-ADVERTs |
| NonAdvertSingleHop | observer↔path[0] only |
| HashCollision | Ambiguous edge with candidates |
| JaccardScoring | Jaccard coefficient computation |
| ConfidenceAutoResolve | Auto-resolve when ratio ≥ 3× |
| EqualScoresAmbiguous | Remains ambiguous with equal scores |
| ObserverSelfEdgeGuard | No self-edges |
| OrphanPrefix | Unresolved prefix handling |
| AffinityScore_Fresh | Score ≈ 1.0 for fresh high-count |
| AffinityScore_Decayed | Score ≈ 0.5 at 7-day half-life |
| AffinityScore_LowCount | Score ≈ 0.05 for count=5 |
| AffinityScore_StaleAndLow | Score ≈ 0 for old low-count |
| CountAccumulation | 5 observations → count=5 |
| MultipleObservers | Observer set tracks all witnesses |
| TimeDecayOldObservations | Month-old edge scores very low |
| ADVERTOnlyConstraint | Non-ADVERTs don't create originator edges |
| CacheTTL | Stale detection works correctly |

## Not in scope (future milestones)
- API endpoints (M2)
- Frontend integration (M3-M5)
- Debug tools (M6)
- Analytics visualization (M7)

Part of #482

---------

Co-authored-by: you <you@example.com>
 2026-04-02 21:14:58 -07:00
Kpa-clawbot 64745f89b1 feat: customizer v2 — event-driven state management (#502) (#503) 
## Summary

Implements the customizer v2 per the [approved
spec](docs/specs/customizer-rework.md), replacing the v1 customizer's
scattered state management with a clean event-driven architecture.
Resolves #502.

## What Changed

### New: `public/customize-v2.js`
Complete rewrite of the customizer as a self-contained IIFE with:

- **Single localStorage key** (`cs-theme-overrides`) replacing 7
scattered keys
- **Three state layers:** server defaults (immutable) → user overrides
(delta) → effective config (computed)
- **Full data flow pipeline:** `write → read-back → merge → atomic
SITE_CONFIG assign → apply CSS → dispatch theme-changed`
- **Color picker optimistic CSS** (Decision #12): `input` events update
CSS directly for responsiveness; `change` events trigger the full
pipeline
- **Override indicator dots** (●) on each field — click to reset
individual values
- **Section-level override count badges** on tabs
- **Browser-local banner** in panel header: "These settings are saved in
your browser only"
- **Auto-save status indicator** in footer: "All changes saved" /
"Saving..." / "⚠️ Storage full"
- **Export/Import** with full shape validation (`validateShape()`)
- **Presets** flow through the standard pipeline
(`writeOverrides(presetData) → pipeline`)
- **One-time migration** from 7 legacy localStorage keys (exact field
mapping per spec)
- **Validation** on all writes: color format, opacity range, timestamp
enum values
- **QuotaExceededError handling** with visible user warning

### Modified: `public/app.js`
Replaced ~80 lines of inline theme application code with a 15-line
`_customizerV2.init(cfg)` call. The customizer v2 handles all merging,
CSS application, and global state updates.

### Modified: `public/index.html`
Swapped `customize.js` → `customize-v2.js` script tag.

### Added: `docs/specs/customizer-rework.md`
The full approved spec, included in the repo for reference.

## Migration

On first page load:
1. Checks if `cs-theme-overrides` already exists → skip if yes
2. Reads all 7 legacy keys (`meshcore-user-theme`,
`meshcore-timestamp-*`, `meshcore-heatmap-opacity`,
`meshcore-live-heatmap-opacity`)
3. Maps them to the new delta format per the spec's field-by-field
mapping
4. Writes to `cs-theme-overrides`, removes all legacy keys
5. Continues with normal init

Users with existing customizations will see them preserved
automatically.

## Dark/Light Mode

- `theme` section stores light mode overrides, `themeDark` stores dark
mode overrides
- `meshcore-theme` localStorage key remains **separate** (view
preference, not customization)
- Switching modes re-runs the full pipeline with the correct section

## Testing

- All existing tests pass (`test-packet-filter.js`, `test-aging.js`,
`test-frontend-helpers.js`)
- Old `customize.js` is NOT modified — left in place for reference but
no longer loaded

## Not in Scope (per spec)

- Undo/redo stack
- Cross-tab synchronization
- Server-side admin import endpoint
- Map config / geo-filter overrides

---------

Co-authored-by: you <you@example.com>
 2026-04-02 21:14:38 -07:00
Kpa-clawbot c9c473279e fix: add null-guards to rAF callbacks in live page animations (#506) 
## Summary

Fixes #483 — navigating away from the live page while matrix/hop
animations are running throws `TypeError: Cannot read properties of null
(reading 'addLayer')`.

## Root Cause

`destroy()` sets `animLayer = null` and `pathsLayer = null`, but
in-flight `requestAnimationFrame` callbacks continue executing and
attempt to call `.addTo(animLayer)` or `.removeLayer()` on the now-null
references.

The entry guards at the top of `drawMatrixLine()` and
`drawAnimatedLine()` only protect the initial call — not the rAF
continuation loops inside `tick()`, `fadeOut()`, `animateLine()`, and
`animateFade()`.

## Fix

Added null-guards (`if (!animLayer || !pathsLayer) return`) at the top
of all four rAF callback functions in `live.js`:

1. **`tick()`** (line ~2203) — matrix animation main loop
2. **`fadeOut()`** (line ~2253) — matrix animation fade-out
3. **`animateLine()`** (line ~2302) — standard line animation main loop
4. **`animateFade()`** (line ~2337) — standard line fade-out

This pattern is already used elsewhere in the file (e.g., line 1873,
1886) for the same purpose.

## Testing

- All unit tests pass (`npm test` — 0 failures)
- Go server tests pass (`cmd/server` + `cmd/ingestor`)
- Change is defensive only (early return on null) — no behavioral change
when layers exist

---------

Co-authored-by: you <you@example.com>
 2026-04-02 20:14:52 -07:00
Kpa-clawbot ad97c0fdd1 fix: clear stale parsed cache on observation packets (#505) 
## Summary

Fixes #504 — Expanding a packet in the packets UI showed the same path
on every observation instead of each observation's unique path.

## Root Cause

PR #400 (fixing #387) added caching of `JSON.parse` results as
`_parsedPath` and `_parsedDecoded` properties on packet objects. When
observation packets are created via object spread (`{...parentPacket,
...obs}`), these cache properties are copied from the parent. Subsequent
calls to `getParsedPath(obsPacket)` hit the stale cache and return the
parent's path, ignoring the observation's own `path_json`.

## Fix

After every object spread that creates an observation packet from a
parent packet, delete the cache properties so they get re-parsed from
the observation's own data:

```js
delete obsPacket._parsedPath;
delete obsPacket._parsedDecoded;
```

Applied to all 5 spread sites in `public/packets.js`:
- Line 271: detail pane observation selection
- Line 504: flat view observation expansion
- Line 840: grouped view observation expansion
- Line 1012: child observation selection in grouped view
- Line 1982: WebSocket live update observation expansion

## Tests

Added 2 new tests in `test-frontend-helpers.js`:
1. Verifies observation packets get their own path after cache
invalidation (not the parent's)
2. Verifies observation path differs from parent path after cache
invalidation

All 431 frontend helper tests pass. All 62 packet filter tests pass.

---------

Co-authored-by: you <you@example.com>
 2026-04-02 19:47:17 -07:00
P. Clawmogorov c7f655e419 perf(frontend): cache JSON.parse results for packet data (#400) 
## Problem
As described in #387, `JSON.parse()` is called repeatedly on the same
packet data across render cycles. With 30K packets, each render cycle
parses 60K+ JSON strings unnecessarily.

## Analysis
The server sends `decoded_json` and `path_json` as JSON strings. The
frontend parses them on-demand in multiple locations:
- `renderTableRows()` — for every row, every render
- WebSocket handling — when processing filtered packets
- `loadPackets()` — during packet loading
- Detail view rendering — when showing packet details

This creates O(n×m) parsing overhead where n = packet count and m =
render cycles.

## Solution
Add cached parse helpers that store parsed results on the packet object:

```javascript
function getParsedPath(p) {
  if (p._parsedPath === undefined) {
    try { p._parsedPath = JSON.parse(p.path_json || '[]'); } catch { p._parsedPath = []; }
  }
  return p._parsedPath;
}
```

Same pattern for `getParsedDecoded()`.

## Changes
- `public/packets.js`: Add helpers + replace 15+ JSON.parse calls
- `public/live.js`: Add helpers + replace 5 JSON.parse calls

## Benchmarks
Before: 60K+ JSON.parse calls per render cycle (30K packets)
After: ~30K parse calls (one per packet, cached thereafter)

Memory impact: Negligible (stores parsed objects that were already
created temporarily)

## Notes
- Cache uses `undefined` check to distinguish "not cached" from "cached
empty result"
- Property names `_parsedPath` and `_parsedDecoded` prefixed to avoid
collision with server fields
- No breaking changes to existing code paths

Fixes #387

---------

Co-authored-by: P. Clawmogorov <262173731+Alm0stSurely@users.noreply.github.com>
Co-authored-by: you <you@example.com>
 2026-04-03 01:11:02 +00:00
efiten b1d89d7d9f fix: apply region filter in GetNodes — was silently ignored (#496) (#497) 
## Summary
- `db.GetNodes` accepted a `region` param from the HTTP handler but
never used it — every region-filter selection was silently ignored and
all nodes were always returned
- Added a subquery filtering `nodes.public_key` against ADVERT
transmissions (payload_type=4) observed by observers with matching IATA
codes
- Handles both v2 (`observer_id TEXT`) and v3 (`observer_idx INT`)
schemas

## Test plan
- [x] 4 new subtests added to `TestGetNodesFiltering`: SJC (1 node), SFO
(1 node), SJC,SFO multi (1 node deduped), AMS unknown (0 nodes)
- [x] All existing Go tests still pass
- [x] Deploy to staging, open `/nodes`, select a region in the filter
bar — only nodes observed by observers in that region should appear

Closes #496

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
 2026-04-02 17:49:57 -07:00
efiten c173ab7e80 perf: skip JSON parse in indexByNode when no pubkey fields present (#376) (#499) 
## Summary
- `indexByNode` was calling `json.Unmarshal` for every packet during
`Load()` and `IngestNewFromDB()`, even channel messages and other
payloads that can never contain node pubkey fields
- All three target fields (`"pubKey"`, `"destPubKey"`, `"srcPubKey"`)
share the common substring `"ubKey"` — added a `strings.Contains`
pre-check that skips the JSON parse entirely for packets that don't
match
- At 30K+ packets on startup, this eliminates the majority of
`json.Unmarshal` calls in `indexByNode` (channel messages, status
packets, etc. all bypass it)

## Test plan
- [x] 5 new subtests in `TestIndexByNodePreCheck`: ADVERT with pubKey
indexed, destPubKey indexed, channel message skipped, empty JSON
skipped, duplicate hash deduped
- [x] All existing Go tests pass
- [x] Deploy to staging and verify node-filtered packet queries still
work correctly

Closes #376

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
 2026-04-02 17:44:02 -07:00
Jukka Väisänen 4664c90db4 fix: skip zero-hop adverts when checking node hash size (#493) 
Fixes issue router IDs flapping between 1byte and multi-byte as
described in https://github.com/Kpa-clawbot/CoreScope/issues/303 with a
minimal patch + test coverage.

This fix is critical for regions using multi-byte IDs.

Closes https://github.com/Kpa-clawbot/CoreScope/issues/303

---------

Co-authored-by: you <you@example.com>
 2026-04-03 00:33:20 +00:00
Kpa-clawbot 2755dc3875 test: push ingestor coverage from 70% to 84% (#344) (#492) 
## Summary

Push Go ingestor test coverage from **70.2% → 84.0%** (92.8% excluding
the untestable `main()` and `init()` functions).

Part of #344 — ingestor coverage

## What Changed

Added `coverage_boost_test.go` with 60+ new test functions covering
previously untested code paths:

### Coverage Before → After by Function

| Function | Before | After |
|----------|--------|-------|
| `NodeDaysOrDefault` | 0% | 100% |
| `MoveStaleNodes` | 0% | 76.5% |
| `NodePassesGeoFilter` | 40% | 100% |
| `handleMessage` | 41.4% | 92.1% |
| `ResolvedSources` | 71.4% | 100% |
| `extractObserverMeta` | 100% | 100% |
| `decodeAdvert` | 88.2% | 94.1% |
| `decryptChannelMessage` | 88.4% | 93.0% |
| **Total** | **70.2%** | **84.0%** |

### Test Categories Added

- **Config**: `NodeDaysOrDefault` all branches, broker scheme
normalization (`mqtt://` → `tcp://`, `mqtts://` → `ssl://`)
- **Database**: `MoveStaleNodes` (stale/fresh/replace), duplicate
transmission handling, default timestamps, telemetry updates, schema
migration verification
- **Decoder**: Sensor telemetry parsing, location + features with
truncated data, `countNonPrintable` with invalid UTF-8,
`decryptChannelMessage` error paths (invalid
key/MAC/ciphertext/alignment), short payload handling
- **Geo Filter**: All branches (nil filter, nil coords, inside/outside)
- **Message Handler**: Channel messages (with/without sender, empty
text), direct messages, geo-filtered adverts, corrupted adverts
(all-zero pubkey), non-advert packets, `Score`/`Direction`
case-insensitive fallbacks, status messages with full hardware metadata

### Why Not 90%+

The remaining ~16% uncovered statements are:
- `main()` function (68 blocks) — program entry point with MQTT client
setup, signal handling, goroutines — not unit-testable without major
refactoring
- `init()` function — `--version` flag + `os.Exit(0)` — kills the test
process
- `prepareStatements()` error returns — only trigger on
corrupted/incompatible SQLite databases
- `applySchema()` migration error paths — only trigger on
filesystem/SQLite failures

Excluding `main()` and `init()`, effective coverage is **92.8%**.

## Test Results

All 100+ tests pass (existing + new):
```
ok  github.com/corescope/ingestor  25.945s  coverage: 84.0% of statements
```

---------

Co-authored-by: you <you@example.com>
 2026-04-02 17:31:47 -07:00
efiten 5228e67604 fix: use packet timestamp in bufferPacket instead of arrival time (#475) (#491) 
## Summary
- `bufferPacket()` was overwriting `_ts` with `Date.now()` (receive
time) for every live WS packet
- Packets arriving in the same batch all got identical timestamps,
making the message history show the same "Xs ago" for every entry (e.g.,
all show "5s ago")
- Fix: use `pkt.timestamp || pkt.created_at` (mirroring
`dbPacketToLive`) so each packet reflects its actual origination time,
falling back to `Date.now()` only when the packet has no timestamp

## Root cause
```js
// before
pkt._ts = Date.now();

// after
pkt._ts = new Date(pkt.timestamp || pkt.created_at || Date.now()).getTime();
```

The WS broadcast includes `timestamp` (= `tx.FirstSeen`) in the packet
map (store.go:1182), so the field is always present for real packets.

## Test plan
- [x] Open Live page, observe packets arriving — each should show its
own relative time, not all the same value
- [x] `node test-frontend-helpers.js` passes (235 tests, 0 failures)

Closes #475

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
 2026-04-02 17:30:55 -07:00
Kpa-clawbot 698514e5e6 test: comprehensive live.js coverage (71 tests) (#489) 
## Summary

Add comprehensive test coverage for `live.js` — the largest and most
complex frontend file (2500+ lines) covering animation modes, VCR
playback, WebSocket handling, audio integration, and the live map.

Part of #344 — live.js coverage.

## What's Tested (71 tests)

### Pure function tests via `vm.createContext`
- **`dbPacketToLive`** — DB packet → live format conversion, null
`decoded_json`, `payload_type_name` fallback, `created_at` timestamp
fallback
- **`expandToBufferEntries`** — observation expansion (1→N entries),
empty observations, multi-packet batches
- **`SEG_MAP`** — 7-segment LCD digit mapping completeness (all digits,
colon, space, VCR mode letters)
- **VCR state machine** — mode transitions (`LIVE`→`PAUSED`→`REPLAY`),
`frozenNow` lifecycle, speed cycling (1→2→4→8→1), pause idempotency
- **`getFavoritePubkeys`** — localStorage merging from
`meshcore-favorites` + `meshcore-my-nodes`, corrupt data handling, falsy
filtering
- **`packetInvolvesFavorite`** — sender pubKey matching, hop prefix
matching, missing decoded fields
- **`isNodeFavorited`** — basic favorite lookup, empty state
- **`formatLiveTimestampHtml`** — timestamp formatting with tooltip,
null input, numeric input, future warning icon
- **`resolveHopPositions`** — HopResolver integration, ghost hop
interpolation between known nodes
- **`bufferPacket`** — VCR buffer management, 2000-entry cap with
playhead adjustment, missed count in PAUSED mode

### Source-level safety checks (20 tests)
- Null guards: `renderPacketTree`, `animatePath`, `pulseNode`, `nextHop`
(all verified via source-level checks)
- Animation limit enforcement (`MAX_CONCURRENT_ANIMS`)
- Tab visibility optimization (skip animations when hidden, clear
propagation buffer on restore)
- WebSocket auto-reconnect
- `addNodeMarker` deduplication
- All toggle state persistence to localStorage (matrix, rain, realistic,
favorites, ghost hops)
- `clearNodeMarkers` resets HopResolver
- `startReplay` pre-aggregates by hash
- Orientation change retry delays
- `vcrRewind` deduplicates buffer entries by ID

## Changes
- `public/live.js` — expose 14 additional functions via `window._live*`
for testing (following existing pattern)
- `test-live.js` — new test file, 841 lines, 71 tests

## Constraints
- No new dependencies
- Tests run via `vm.createContext` against real code (not copies)
- No build step — vanilla JS

---------

Co-authored-by: you <you@example.com>
 2026-04-02 17:16:03 -07:00
Kpa-clawbot cf3a383bb2 test: comprehensive app.js coverage — 100+ new tests (#490) 
## Summary

Adds 100+ new tests for previously untested `app.js` functions,
significantly improving frontend coverage toward the 90%+ target.

## What's Tested

All pure/testable functions from `app.js` that lacked coverage:

| Function Group | Tests Added | Description |
|---|---|---|
| `payloadTypeColor` | 13 | All PAYLOAD_COLORS mappings +
unknown/null/undefined fallback |
| `pad2` / `pad3` | 10 | Zero-padding for 1-3 digit values, no
truncation |
| `formatIsoLike` | 5 | UTC/local timezone, with/without milliseconds,
zero-padding |
| `formatTimestampCustom` | 5 | Token replacement
(YYYY/MM/DD/HH/mm/ss/SSS/Z), partial formats, invalid format rejection |
| `formatAbsoluteTimestamp` | 3 | Custom format integration, locale+UTC,
null/invalid date handling |
| `getTimestamp*` getters | 11 | localStorage priority, server config
fallback, invalid value rejection for
Mode/Timezone/FormatPreset/CustomFormat |
| `invalidateApiCache` | 3 | Prefix-based selective invalidation, full
clear, cache→invalidate→re-fetch lifecycle |
| `formatHex` | 5 | Byte spacing, single byte, null/empty, odd-length
hex |
| `createColoredHexDump` | 6 | Range-based coloring, override
precedence, null/empty hex+ranges |
| `buildHexLegend` | 5 | Label deduplication, correct swatch colors per
label class, null/empty |
| Favorites (`getFavorites`/`isFavorite`/`toggleFavorite`/`favStar`) | 9
| CRUD operations, corrupt JSON resilience, star HTML rendering with
custom classes |
| `debounce` | 3 | Delay behavior, timer reset on rapid calls, argument
forwarding |
| `mergeUserHomeConfig` | 5 | Null/missing siteConfig/userTheme,
non-object home, missing home creation |
| Constants | 2 | Exhaustive ROUTE_TYPES (4) and PAYLOAD_TYPES (13)
mapping verification |

## Approach

- Tests use the existing `vm.createContext` sandbox pattern from
`test-frontend-helpers.js`
- Tests the **real code** loaded from `public/app.js` — no copies
- No new dependencies
- Each `invalidateApiCache` test uses an isolated sandbox to avoid async
race conditions

## Test Results

```
Frontend helpers: 343 passed, 0 failed
```

Part of #344 — app.js coverage

---------

Co-authored-by: you <you@example.com>
 2026-04-02 17:03:35 -07:00
efiten a45ac71508 fix: restore color-coded hex breakdown in packet detail (#329) (#500) 
## Summary
- `BuildBreakdown` was never ported from the deleted Node.js
`decoder.js` to Go — the server has returned `breakdown: {}` since the
Go migration (commit `742ed865`), so `createColoredHexDump()` and
`buildHexLegend()` in the frontend always received an empty `ranges`
array and rendered everything as monochrome
- Implemented `BuildBreakdown()` in `decoder.go` — computes labeled byte
ranges matching the frontend's `LABEL_CLASS` map: `Header`, `Transport
Codes`, `Path Length`, `Path`, `Payload`; ADVERT packets get sub-ranges:
`PubKey`, `Timestamp`, `Signature`, `Flags`, `Latitude`, `Longitude`,
`Name`
- Wired into `handlePacketDetail` (was `struct{}{}`)
- Also adds per-section color classes to the field breakdown table
(`section-header`, `section-transport`, `section-path`,
`section-payload`) so the table rows get matching background tints

## Test plan
- [x] Open any packet detail pane — hex dump should show color-coded
sections (red header, orange path length, blue transport codes, green
path hops, yellow/colored payload)
- [x] Legend below action buttons should appear with color swatches
- [x] ADVERT packets: PubKey/Timestamp/Signature/Flags each get their
own distinct color
- [x] Field breakdown table section header rows should be tinted per
section
- [x] 8 new Go tests: all pass

Closes #329

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-02 16:54:59 -07:00
Kpa-clawbot 016b87b33c test: add 64 unit tests for packets.js (Part of #344) (#488) 
## Summary

Adds 64 unit tests for `packets.js` — the largest untested frontend file
(2000+ lines) covering filter engine integration, time window logic,
groupByHash rendering, and packet detail display.

Part of #344 — packets.js coverage.

## Approach

Follows the existing `test-frontend-helpers.js` pattern: loads real
source files into a `vm.createContext` sandbox and tests actual code (no
copies).

Added a `window._packetsTestAPI` export at the end of the packets.js
IIFE to expose pure functions for testing without changing any runtime
behavior.

## What's Tested

| Function | Tests | What it covers |
|----------|-------|----------------|
| `typeName` | 2 | Type code → name mapping, unknown fallback |
| `obsName` | 2 | Observer name lookup, falsy/missing handling |
| `kv` | 1 | Key-value HTML helper |
| `sectionRow` / `fieldRow` | 3 | Table section/field HTML builders |
| `getDetailPreview` | 17 | All packet types: CHAN, ADVERT
(repeater/room/sensor/companion), GRP_TXT
(no_key/decryption_failed/channelHashHex), TXT_MSG, PATH, REQ, RESPONSE,
ANON_REQ, text fallback, public_key fallback, empty |
| `getPathHopCount` | 4 | Valid path, empty, null, invalid JSON |
| `sortGroupChildren` | 3 | Default observer sort, header update, null
safety |
| `renderTimestampCell` | 2 | Timestamp HTML output, null handling |
| `renderPath` | 3 | Empty/null, multi-hop with arrows, single hop |
| `renderDecodedPacket` | 6 | Header/path/payload/nested objects/null
skip/raw hex |
| `buildFieldTable` | 11 | All payload types (ADVERT with
flags/location/name, GRP_TXT, CHAN, ACK, destHash, raw fallback),
transport codes, path hops, hash_size calculation, empty hex |
| `_getRowCount` | 1 | Virtual scroll row counting |
| `buildFlatRowHtml` | 3 | Row rendering, size calculation, missing hex
|
| `buildGroupRowHtml` | 3 | Single/multi group, observation badge |
| Test API exposure | 1 | Verifies window._packetsTestAPI |

## Constraints Met

- No new test dependencies
- Tests real code via `vm.createContext`, not copies
- No build step — vanilla JS
- All existing tests still pass (254 frontend-helpers, 62 packet-filter,
29 aging)

Co-authored-by: you <you@example.com>
 2026-04-02 16:42:25 -07:00
Kpa-clawbot 889107a5e1 fix: address PR #487 review feedback (#501) 
## Summary

Addresses review feedback from PR #487 (nodes.js coverage).

### Changes

1. **Replace fragile `exportInternals` regex source patching with stable
test hooks** — `getStatusInfo` and `getStatusTooltip` are now exposed
via `window._nodesGetStatusInfo` and `window._nodesGetStatusTooltip`,
matching the existing pattern used by all other test-accessible
functions. The brittle regex `.replace()` approach that modified source
code at runtime has been removed entirely.

2. **Strengthen weak null assertion** — The `renderNodeTimestampHtml
handles null` test previously asserted `html.includes('—') ||
html.length > 0`, which is a near-tautology (any non-empty string
passes). Now strictly asserts `html.includes('—')`.

### Files changed
- `public/nodes.js` — 2 new test hook lines
- `test-frontend-helpers.js` — removed 21-line `exportInternals` branch,
updated tests to use hooks

### Testing
- All 309 frontend helper tests pass
- All 62 packet filter tests pass
- All 29 aging tests pass

Closes review items from #487.

Co-authored-by: you <you@example.com>
 2026-04-02 16:40:11 -07:00
Kpa-clawbot 50f94603c1 test: P0 coverage for nodes.js — sort, status, timestamps, sync (#487) 
## Summary

Add 67 new unit tests for `nodes.js`, raising frontend helper test count
from 233 to 300.

Part of #344 — nodes.js coverage.

## What's Tested

### Sort System (`toggleSort`, `sortNodes`, `sortArrow`)
- Direction toggling on same column (asc↔desc)
- Default sort directions per column type (name→asc, last_seen→desc,
advert_count→desc)
- localStorage persistence of sort state
- All 5 sort columns: `name`, `public_key`, `role`, `last_seen`,
`advert_count`
- Both ascending and descending for each column
- Case-insensitive name sorting
- Unnamed nodes sort last
- Timestamp fallback chain: `last_heard` → `last_seen` → 0
- Missing timestamp handling
- Empty array edge case
- Unknown column graceful handling
- `sortArrow` rendering for active (▲/▼) and inactive columns

### Status Calculation (`getStatusInfo`, `getStatusTooltip`)
- `_lastHeard` takes priority over `last_heard`
- `last_seen` used as fallback when `last_heard` missing
- No-timestamp nodes return stale with `lastHeardMs: 0`
- Infrastructure threshold (72h) for rooms
- Standard threshold (24h) for sensors and companions
- Explanation text varies by role and status
- Unknown role defaults to gray color `#6b7280`
- All role/status tooltip combinations

### Timestamp Rendering (`renderNodeTimestampHtml`,
`renderNodeTimestampText`)
- HTML output includes tooltip and `timestamp-text` class
- Future timestamps show ⚠️ warning icon
- Null input produces dash
- Text output is plain (no HTML tags)

### Favorites Sync (`syncClaimedToFavorites`)
- Claimed pubkeys added to favorites
- No-op when all already synced
- Empty my-nodes handled
- Missing localStorage keys don't crash

## Implementation

- Added test hooks on `window` for closure-scoped functions
(non-invasive, follows existing pattern)
- Tests use `vm.createContext` to load real `nodes.js` code — no copies
- No new dependencies

## Test Results

```
Frontend helpers: 300 passed, 0 failed
```

---------

Co-authored-by: you <you@example.com>
 2026-04-02 23:32:41 +00:00
efiten b799f54700 perf: bound memory growth and reduce render CPU on packets page (#421) 
## Problem

On a long-running session the packets page consumed 8 GB of browser
memory and 20%+ CPU on an 8-core machine. Root causes:

1. **Unbounded `packets` array growth via WebSocket** —
`packets.unshift()` was called for every new unique hash, but nothing
ever trimmed the array. After hours of live traffic the array grew well
past the initial 50 k load limit.
2. **Unbounded `pauseBuffer`** — all WS messages queued while paused, no
cap.
3. **Unbounded `_children` growth** — expanded groups received a
`unshift(p)` on every matching WS message with no size limit.
4. **O(n) `observers.find()` inside the O(n) render loop** — with 50 k
rows, each render triggered up to 50 k linear scans through the
observers list.
5. **Full DOM rebuild on every WS message** — `renderTableRows()` was
called synchronously on every WebSocket batch, reconstructing the entire
table on each incoming packet.

## Changes

- `packets[]` is now trimmed to `PACKET_LIMIT` after each WS batch;
evicted entries are also removed from `hashIndex` to prevent stale
references.
- `pauseBuffer` capped at 2 000 entries (oldest dropped).
- `_children` capped at 200 entries on WS prepend.
- `renderTableRows()` on the WS path is debounced to 200 ms, batching
rapid updates into a single redraw.
- `observersById = new Map()` pre-built from the observers array; all
`observers.find()` calls in the render loop and WS filter replaced with
O(1) `Map.get()`.

## Test plan

- [x] Load the packets page and leave it running for several minutes
with live WebSocket traffic — memory in DevTools should remain stable
rather than growing continuously
- [x] Pause live updates, wait for several messages, then resume —
buffer replays correctly and display updates
- [x] Expand a packet group and leave it open during live traffic —
children update but don't grow past 200
- [x] Region filter still works correctly (relies on the observer Map
lookup)
- [x] Observer name / IATA badge renders correctly in grouped and flat
mode

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-02 16:04:01 -07:00
Kpa-clawbot d5b300a8ba fix: derive version from git tags instead of package.json (#486) 
## Summary

Fixes #485 — the app version was derived from `package.json` via
Node.js, which is a meaningless artifact for this Go project. This
caused version mismatches (e.g., v3.3.0 release showing "3.2.0") when
someone forgot to bump `package.json`.

## Changes

### `manage.sh`
- **Line 43**: Replace `node -p "require('./package.json').version"`
with `git describe --tags --match "v*"` — version is now derived
automatically from git tags
- **Line 515**: Add `--force` to `git fetch origin --tags` in setup
command
- **Line 1320**: Add `--force` to `git fetch origin --tags` in update
command — prevents "would clobber existing tag" errors when tags are
moved

### `package.json`
- Version field set to `0.0.0-use-git-tags` to make it clear this is not
the source of truth. File kept because npm scripts and devDependencies
are still used for testing.

## How it works

`git describe --tags --match "v*"` produces:
- `v3.3.0` — when on an exact tag
- `v3.3.0-3-gabcdef1` — when 3 commits after a tag (useful for
debugging)
- Falls back to `unknown` if no tags exist

## Testing

- All Go tests pass (`cmd/server`, `cmd/ingestor`)
- All frontend unit tests pass (254/254)
- No changes to application logic — only build-time version derivation

Co-authored-by: you <you@example.com>
 2026-04-02 00:53:38 -07:00
you 2af4259eca chore: bump version to 3.3.0 v3.3.0 2026-04-02 07:27:32 +00:00
Kpa-clawbot bf2e721dd7 feat: auto-inject cache busters at server startup — eliminates merge conflicts (#481) 
## Problem

Every PR that touches `public/` files requires manually bumping cache
buster timestamps in `index.html` (e.g. `?v=1775111407`). Since all PRs
change the same lines in the same file, this causes **constant merge
conflicts** — it's been the #1 source of unnecessary PR friction.

## Solution

Replace all hardcoded `?v=TIMESTAMP` values in `index.html` with a
`?v=__BUST__` placeholder. The Go server replaces `__BUST__` with the
current Unix timestamp **once at startup** when it reads `index.html`,
then serves the pre-processed HTML from memory.

Every server restart automatically picks up fresh cache busters — no
manual intervention needed.

## What changed

| File | Change |
|------|--------|
| `public/index.html` | All `v=1775111407` → `v=__BUST__` (28
occurrences) |
| `cmd/server/main.go` | `spaHandler` reads index.html at init, replaces
`__BUST__` with Unix timestamp, serves from memory for `/`,
`/index.html`, and SPA fallback |
| `cmd/server/helpers_test.go` | New `TestSpaHandlerCacheBust` —
verifies placeholder replacement works for root, SPA fallback, and
direct `/index.html` requests. Also added tests for root `/` and
`/index.html` routes |
| `AGENTS.md` | Rule 3 updated: cache busters are now automatic, agents
should not manually edit them |

## Testing

- `go build ./...` — compiles cleanly
- `go test ./...` — all tests pass (including new cache-bust tests)
- `node test-frontend-helpers.js && node test-packet-filter.js && node
test-aging.js` — all frontend tests pass
- No hardcoded timestamps remain in `index.html`

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: you <you@example.com>
 2026-04-01 23:59:59 -07:00
Kpa-clawbot f20431d816 fix: implement 'Show direct neighbors' map filter (#480) 
## Summary

Fixes #457 — The "Show direct neighbors" checkbox on the map was a UI
stub that did nothing. This PR implements the full feature.

## What Changed

### `public/map.js`
- **New state**: `selectedReferenceNode` (pubkey) and `neighborPubkeys`
(Set) track which node is the reference and who its direct neighbors are
- **`selectReferenceNode(pubkey, name)`**: Fetches
`/api/nodes/{pubkey}/paths`, parses path hops to find all nodes directly
adjacent to the reference node in any observed path, then auto-enables
the neighbor filter
- **Neighbor filter in `_renderMarkersInner()`**: When
`filters.neighbors` is on and a reference node is selected, only the
reference node and its direct (1-hop) neighbors are shown on the map
- **Popup "Show Neighbors" link**: Each node popup now has a "Show
Neighbors" action that sets it as the reference node
- **Sidebar UI hints**: Shows the reference node name when selected, or
a hint to click a node when the filter is enabled without a reference
- **Cleanup on `destroy()`**: Clears reference state and global handler

### `test-frontend-helpers.js`
- 6 new unit tests covering:
  - Filter off shows all nodes
  - Filter on without reference shows all nodes (graceful no-op)
  - Filter on with reference + neighbors filters correctly
  - Filter on with empty neighbor set shows only reference
  - Neighbor filter respects role filters
  - Neighbor extraction from path data

### `public/index.html`
- Cache buster bump

## How It Works

1. User clicks a node marker on the map → popup shows "Show Neighbors"
link
2. Clicking "Show Neighbors" fetches that node's paths from
`/api/nodes/{pubkey}/paths`
3. Adjacent hops in each path are identified as direct neighbors
4. The map filters to show only the reference node + its neighbors
5. The sidebar shows which node is the reference
6. Unchecking the checkbox restores the full node view

## Test Results

```
Frontend helpers: 250 passed, 0 failed
Packet filter:     62 passed, 0 failed
```

---------

Co-authored-by: you <you@example.com>
 2026-04-01 23:49:10 -07:00
Kpa-clawbot f9cfad9cd4 fix: update observer last_seen on packet ingestion (#479) 
## Summary

Related to #463 (partial fix — addresses packet path, status message
path still needs investigation) — Observers incorrectly showing as
offline despite actively forwarding packets.

## Root Cause

Observer `last_seen` was only updated when status topic messages
(`meshcore/<region>/<observer_id>/status`) were received via
`UpsertObserver`. When packets were ingested from an observer, the
observer's `last_seen` was **not** updated — only the `observer_idx` was
resolved for the observation record.

This meant observers with low traffic that published status messages
less frequently than the 10-minute online threshold would appear offline
on the observers page, even though they were clearly alive and
forwarding packets.

## Changes

**`cmd/ingestor/db.go`:**
- Added `stmtUpdateObserverLastSeen` prepared statement: `UPDATE
observers SET last_seen = ? WHERE rowid = ?`
- In `InsertTransmission`, after resolving `observer_idx`, update the
observer's `last_seen` to the packet timestamp
- This ensures any observer actively forwarding traffic stays marked as
online

**`cmd/ingestor/db_test.go`:**
- Added `TestInsertTransmissionUpdatesObserverLastSeen` — verifies that
inserting a packet from an observer updates its `last_seen` from a
backdated value to the packet timestamp

## Performance

The added `UPDATE` is a single-row update by `rowid` (primary key) —
O(1) with no index overhead. It runs once per packet insertion when an
observer is resolved, which was already doing a `SELECT` by `rowid`
anyway. No measurable impact on ingestion throughput.

## Test Results

All existing tests pass:
- `cmd/ingestor`: 26.6s ✅
- `cmd/server`: 3.7s ✅

---------

Co-authored-by: you <you@example.com>
 2026-04-01 23:43:47 -07:00
Kpa-clawbot 96d0bbe487 fix: replace Euclidean distance with haversine in analytics hop distances (#478) 
## Summary

Fixes #433 — Replace the inaccurate Euclidean distance approximation in
`analytics.js` hop distances with proper haversine calculation, matching
the server-side computation introduced in PR #415.

## Problem

PR #415 moved collision analysis server-side and switched from the
frontend's Euclidean approximation (`dLat×111, dLon×85`) to proper
haversine. However, the **hop distance** calculation in `analytics.js`
(subpath detail panel) still used the old Euclidean formula. This
caused:

- **Inconsistent distances** between hop distances and collision
distances
- **Significant errors at high latitudes** — e.g., Oslo→Stockholm:
Euclidean gives ~627km, haversine gives ~415km (51% error)
- The `dLon×85` constant assumes ~40° latitude; at 60° latitude the real
scale factor is ~55.5km/degree, not 85

## Changes

| File | Change |
|------|--------|
| `public/analytics.js` | Replace `dLat*111, dLon*85` Euclidean with
`HopResolver.haversineKm()` (with inline fallback) |
| `public/hop-resolver.js` | Export `haversineKm` in the public API for
reuse |
| `test-frontend-helpers.js` | Add 4 tests: export check, zero distance,
SF→LA accuracy, Euclidean vs haversine divergence |
| `cmd/server/helpers_test.go` | Add `TestHaversineKm`: zero, SF→LA,
symmetry, Oslo→Stockholm accuracy |
| `public/index.html` | Cache buster bump |

## Performance

No performance impact — `haversineKm` replaces an inline arithmetic
expression with another inline arithmetic expression of identical O(1)
complexity. Only called per hop pair in the subpath detail panel
(typically <10 hops).

## Testing

- `node test-frontend-helpers.js` — 248 passed, 0 failed
- `go test -run TestHaversineKm` — PASS

Co-authored-by: you <you@example.com>
 2026-04-01 23:37:01 -07:00
Kpa-clawbot 6712da7d7c fix: add region filtering to hash-collisions endpoint (#477) 
## Summary

The `/api/analytics/hash-collisions` endpoint always returned global
results, ignoring the active region filter. Every other analytics
endpoint (RF, topology, hash-sizes, channels, distance, subpaths)
respected the `?region=` query parameter — this was the only one that
didn't.

Fixes #438

## Changes

### Backend (`cmd/server/`)

- **routes.go**: Extract `region` query param and pass to
`GetAnalyticsHashCollisions(region)`
- **store.go**:
- `collisionCache` changed from `*cachedResult` →
`map[string]*cachedResult` (keyed by region, `""` = global) — consistent
with `rfCache`, `topoCache`, etc.
- `GetAnalyticsHashCollisions(region)` and
`computeHashCollisions(region)` now accept a region parameter
- When region is specified, resolves regional observers, scans packets
for nodes seen by those observers, and filters the node list before
computing collisions
  - Cache invalidation updated to clear the map (not set to nil)

### Frontend (`public/`)

- **analytics.js**: The hash-collisions fetch was missing `+ sep` (the
region query string). All other fetches in the same `Promise.all` block
had it — this was simply overlooked in PR #415.
- **index.html**: Cache busters bumped

### Tests (`cmd/server/routes_test.go`)

- `TestHashCollisionsRegionParamIgnored` → renamed to
`TestHashCollisionsRegionParam` with updated comments reflecting that
region is now accepted (with no configured regional observers, results
match global — which the test verifies)

## Performance

No new hot-path work. Region filtering adds one scan of `s.packets`
(same as every other region-filtered analytics endpoint) only when
`?region=` is provided. Results are cached per-region with the existing
60s TTL. Without `?region=`, behavior is unchanged.

Co-authored-by: you <you@example.com>
 2026-04-01 23:27:34 -07:00
Kpa-clawbot 6aef83c82a fix: remove duplicate return statement in _cumulativeRowOffsets() (#476) 
## Summary

Removes an unreachable duplicate `return offsets;` statement in the
`_cumulativeRowOffsets()` function in `packets.js`. The second return
was dead code found during review of PR #402.

## Changes

- **`public/packets.js`**: Removed the duplicate `return offsets;` on
what was line 1137 (the line immediately after the first, reachable
`return offsets;`)
- **`public/index.html`**: Cache buster bump

## Testing

This is a dead code removal — the duplicate return was unreachable. No
behavior change. No new tests needed as existing tests already cover
`_cumulativeRowOffsets()` behavior.

Fixes #447

Co-authored-by: you <you@example.com>
 2026-04-01 23:25:07 -07:00
Kpa-clawbot 9f14c74b3e ci: add Docker cleanup before build to prevent disk space exhaustion (#473) 
## Summary

Fixes #472

The Docker build job on the self-hosted runner fails with `no space left
on device` because Docker build cache and Go module downloads accumulate
between runs. The existing cleanup (line ~330) runs in the **deploy**
step *after* the build — too late to help.

## Changes

- Added a "Free disk space" step at the start of the build job,
**before** "Build Go Docker image":
- `docker system prune -af` — removes all unused images, containers,
networks
  - `docker builder prune -af` — clears the build cache
  - `df -h /` — logs available disk space for visibility
- Kept the existing post-deploy cleanup as belt-and-suspenders

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
 2026-04-01 22:27:12 -07:00
Kpa-clawbot 0b8b1e91a6 perf(live): replace animation setIntervals with requestAnimationFrame and cap concurrency (#470) 
## Summary

Replace all `setInterval`-based animations in `live.js` with
`requestAnimationFrame` loops and add a concurrency cap to prevent
unbounded animation accumulation under high packet throughput.

Fixes #384

## Problem

Under high throughput (≥5 packets/sec), the live map accumulated
unbounded `setInterval` timers:

- `pulseNode()`: 26ms interval per pulse ring
- `drawAnimatedLine()`: 33ms interval per hop line + 52ms nested
interval for fade-out
- Ghost hop pulse: 600ms interval per ghost marker

At 5 pkts/sec × 3 hops = **15+ concurrent intervals**, climbing without
limit. This caused UI jank, rising CPU usage, and potential memory leaks
from leaked Leaflet markers.

## Changes

### `public/live.js`

| Function | Before | After |
|----------|--------|-------|
| `pulseNode()` | `setInterval` (26ms) + `setTimeout` safety |
`requestAnimationFrame` loop, self-terminates at 2s or opacity ≤ 0 |
| `drawAnimatedLine()` | `setInterval` (33ms) for line + nested
`setInterval` (52ms) for fade | Two `requestAnimationFrame` loops (line
advance + fade-out) |
| Ghost hop pulse | `setInterval` (600ms) + `setTimeout` (3s) |
`requestAnimationFrame` loop with 3s expiry |
| `animatePath()` | No concurrency limit | Returns early when
`activeAnims >= MAX_CONCURRENT_ANIMS` (20) |

### `public/index.html`
- Cache buster version bump

### `test-live-anims.js` (new)
- 7 tests verifying:
- No `setInterval` in `pulseNode`, `drawAnimatedLine`, or `animatePath`
  - `MAX_CONCURRENT_ANIMS` defined and set to 20
  - Concurrency check present in `animatePath`
  - No stale `setInterval` in animation hot paths

## Complexity & Scale

- **Time complexity**: O(1) per animation frame (no change in per-frame
work)
- **Concurrency**: Hard-capped at 20 simultaneous animations (previously
unbounded)
- **At 5 pkts/sec, 3 hops**: Excess animations silently dropped instead
of accumulating timers
- **rAF benefit**: Browser coalesces all animations into single paint
cycle; paused tabs stop animating automatically

## Test Results

```
=== Animation interval elimination ===
  ✅ pulseNode does not use setInterval
  ✅ drawAnimatedLine does not use setInterval
  ✅ ghost hop pulse does not use setInterval
=== Concurrency cap ===
  ✅ MAX_CONCURRENT_ANIMS is defined
  ✅ MAX_CONCURRENT_ANIMS is set to 20
  ✅ animatePath checks MAX_CONCURRENT_ANIMS before proceeding
=== Safety: no stale setInterval in animation functions ===
  ✅ no setInterval remains in animation hot path
7 passed, 0 failed
```

All existing tests pass (packet-filter: 62, aging: 29, frontend-helpers:
241).
## Performance Proof (Rule 0 compliance)

Benchmark: `node test-anim-perf.js` — simulates timer/animation
accumulation under realistic throughput.

### Timer count: old (setInterval) vs new (rAF + cap)

| Scenario | Old model (peak concurrent timers) | New model (peak
concurrent animations) |

|----------|-----------------------------------:|---------------------------------------:|
| 5 pkt/s × 3 hops, 30s sustained | **123** | **20** |
| 5 pkt/s × 3 hops, 5min sustained | **123** | **20** |
| 20 pkt/s × 3 hops, 10s burst | **246** | **20** |

**Before:** Each hop spawns 3 `setInterval` timers (pulse 26ms, line
33ms, fade 52ms) that live 0.6–2s each. At 5 pkt/s × 3 hops = 15
timers/sec, peak concurrent timers reach **123** (limited only by timer
lifetime, not by any cap). Under burst traffic (20 pkt/s), this climbs
to **246+**.

**After:** `MAX_CONCURRENT_ANIMS = 20` hard-caps active animations.
Excess packets are silently dropped. rAF loops replace all `setInterval`
calls, coalescing into single paint cycles. Peak concurrent animations:
**always ≤ 20**, regardless of throughput or duration.

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
 2026-04-01 20:13:16 -07:00
Kpa-clawbot c678555e75 fix: display channel hash as hex instead of decimal (#471) 
## Summary

Fixes #465 — Channel hash was displaying in decimal instead of
hexadecimal in `channels.js`.

## Changes

- Added `formatHashHex()` helper to `channels.js` that formats numeric
hashes as `0x` hex (e.g. `0x0A`) and passes string hashes through
unchanged
- Applied to both display sites: `renderChannelList` fallback name and
`selectChannel` header text
- Consistent with `packets.js` and `analytics.js` which already use
`.toString(16).padStart(2, '0').toUpperCase()`

## Tests

- 3 new tests in `test-frontend-helpers.js` verifying the helper exists,
is used at display sites, and produces correct output for numeric and
string inputs
- All 244 frontend tests pass, plus packet-filter (62) and aging (29)
tests

Co-authored-by: you <you@example.com>
 2026-04-01 19:45:16 -07:00
Kpa-clawbot 623ebc879b fix: add mutex synchronization to PerfStats to eliminate data races (#469) 
## Summary

Fixes #361 — `perfMiddleware()` wrote to shared `PerfStats` fields
(`Requests`, `TotalMs`, `Endpoints` map, `SlowQueries` slice) without
any synchronization, causing data races under concurrent HTTP requests.

## Changes

### `cmd/server/routes.go`
- **Added `sync.Mutex` to `PerfStats` struct** — single mutex protects
all fields
- **`perfMiddleware`** — all shared state mutations (counter increments,
endpoint map access, slice appends) now happen under lock. Key
normalization (regex, mux route lookup) moved outside the lock since it
uses no shared state
- **`handleHealth`** — snapshots `Requests`, `TotalMs`, `SlowQueries`
under lock before building response
- **`handlePerf`** — copies all endpoint data and slow queries under
lock into local snapshots, then does expensive work (sorting, percentile
calculation) outside the lock
- **`handlePerfReset`** — resets fields in-place instead of replacing
the pointer (avoids unlocking a different mutex)

### `cmd/server/perfstats_race_test.go` (new)
- Regression test: 50 concurrent writer goroutines + 10 concurrent
reader goroutines hammering `PerfStats` simultaneously
- Verifies no race conditions (via `-race` flag) and counter consistency

## Design Decisions

- **Single mutex over atomics**: The issue suggested `atomic.Int64` for
counters, but since slices/maps need a mutex anyway, a single mutex is
simpler and the critical section is small (microseconds). No measurable
contention at CoreScope's scale.
- **Copy-under-lock pattern**: Expensive operations (sorting, percentile
computation) happen outside the lock to minimize hold time.
- **In-place reset**: `handlePerfReset` clears fields rather than
replacing the `PerfStats` pointer, ensuring the mutex remains valid for
concurrent goroutines.

## Testing

- `go test -race -count=1 ./cmd/server/...` — **PASS** (all existing
tests + new race test)
- New `TestPerfStatsConcurrentAccess` specifically validates concurrent
access patterns

Co-authored-by: you <you@example.com>
 2026-04-01 19:26:11 -07:00
Kpa-clawbot 0b1924d401 perf(packets): replace observers.find() linear scans with Map lookups (#468) 
## Summary

Replace all `observers.find()` linear scans in `packets.js` with O(1)
`Map.get()` lookups, eliminating ~300K comparisons per render cycle at
30K+ rows.

## Changes

- Added `observerMap` (`Map<id, observer>`) built once when observers
load
- Replaced all 6 `observers.find()` call sites with `observerMap.get()`:
  - `obsName()` — called per row for observer name display
  - Region filter check in packet filtering
  - Observer dropdown label in filter UI
  - Group header region lookup
  - Child row region lookup  
  - Flat row region lookup
- Map is cleared on reset and rebuilt on each `loadObservers()` call

## Complexity

- **Before:** O(k) per row × 30K rows = O(30K × k) where k = observer
count (~10)
- **After:** O(1) per row × 30K rows = O(30K)
- Map construction: O(k) once, negligible

## Testing

- All Go tests pass (`cmd/server`, `cmd/ingestor`)
- All frontend tests pass (`test-packet-filter.js`: 62 passed,
`test-aging.js`: 29 passed, `test-frontend-helpers.js`: 241 passed)

Fixes #383

Co-authored-by: you <you@example.com>
 2026-04-01 19:20:37 -07:00
efiten 0f502370c5 fix: VCR timeline and clock respect UTC/local timezone setting (#459) 
## Problem

Fixes #324. The VCR LCD clock and timeline hover/touch tooltip always
showed local time, ignoring the UTC/local timezone setting in the
customizer Display tab.

## Root cause

Three sites in `live.js` bypassed the shared `getTimestampTimezone()`
utility:

- `updateVCRClock()` — used `d.getHours()` / `d.getMinutes()` /
`d.getSeconds()` (always local)
- Timeline mousemove tooltip — used `d.toLocaleTimeString()` (always
local)
- Timeline touchmove tooltip — same

## Fix

Added `vcrFormatTime(tsMs)` helper that checks `getTimestampTimezone()`
and uses `getUTC*` methods when set to `'utc'`, otherwise local `get*`.
Applied to all three sites. Exposed as `window._vcrFormatTime` for
testing.

## Tests

4 new unit tests in `test-frontend-helpers.js` covering UTC mode, local
mode, and zero-padding.

## Checklist
- [x] Branches from `upstream/master`
- [x] No Matomo or local-only commits
- [x] Cache busters bumped (`v=1775073838`)
- [x] 233 tests pass, 0 fail

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-02 01:54:09 +00:00
efiten e47c39ffda fix: null-guard animLayer and liveAnimCount in nextHop after destroy (#462) 
## Summary
- `nextHop()` schedules `setInterval`/`setTimeout` callbacks that can
fire after `destroy()` has set `animLayer = null` and removed DOM
elements
- This caused three console errors on the Live page when navigating away
mid-animation: `Cannot read properties of null (reading 'hasLayer')` and
`Cannot set properties of null (setting 'textContent')`
- Added null guards at each async callback site; no behavioral change
when the page is active

## Changes
- `public/live.js`: early return if `animLayer` is null at start of
`nextHop()`; null-safe `animLayer.hasLayer` checks in
`setInterval`/`setTimeout`; null-safe `liveAnimCount` element access
- `public/index.html`: cache buster bumped
- `test-frontend-helpers.js`: 4 source-inspection tests verifying the
null guards are present

## Test plan
- [ ] Open Live page, trigger some packet animations, navigate away
quickly — no console errors
- [ ] `node test-frontend-helpers.js` passes (233 tests, 0 failures)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-02 01:52:34 +00:00
efiten 1499a55ba7 perf: upsert known nodes in-place on ADVERT, skip full reload (#461) 
## Problem

Fixes #399. On every ADVERT WebSocket batch the nodes page invalidated
the entire `_allNodes` cache and triggered a full `/nodes?limit=5000`
fetch — even when every advertising node was already cached. The 90s API
TTL was actively bypassed.

## Root cause

```js
wsHandler = debouncedOnWS(function (msgs) {
  if (msgs.some(isAdvertMessage)) {
    _allNodes = null;               // wipe cache unconditionally
    invalidateApiCache('/nodes');   // bust API TTL
    loadNodes(true);                // full 5k fetch
  }
}, 5000);
```

## Fix

ADVERT decoded payloads include `pubKey`, `name`, `lat`, `lon` — enough
to update known nodes in place:

- **Known node** (pubKey found in `_allNodes`): upsert `name`, `lat`,
`lon`, `last_seen` directly — no fetch, no cache bust, just re-render.
- **New node** (pubKey not in cache) or **no pubKey** in payload: fall
back to full reload as before.

This covers the common case on an active mesh: all advertising nodes are
already cached. The full reload path is preserved for node discovery.

## Tests

2 new unit tests: known-node upsert (asserts 0 API calls, fields
updated) and unknown-node fallback (asserts full reload triggered). All
231 tests pass.

## Checklist
- [x] Branches from `upstream/master`
- [x] No Matomo or local-only commits
- [x] Cache busters bumped
- [x] 231 tests pass, 0 fail

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-02 01:47:56 +00:00
efiten f71e117cdd fix: reset restores home steps after SITE_CONFIG contamination (#460) 
## Problem

Fixes #325. Removing all home steps and clicking "Reset my theme" did
not restore them.

## Root cause

Two-part bug:

**1. `SITE_CONFIG.home` permanently mutated at page load**
`app.js` calls `mergeUserHomeConfig(SITE_CONFIG, userTheme)` which does
`SITE_CONFIG.home = Object.assign({}, serverHome, userTheme.home)`. If
the user had `steps: []` saved in localStorage, this sets
`SITE_CONFIG.home.steps = []` globally — permanently for the lifetime of
the page.

**2. `initState()` reads the contaminated config**
When the customizer opens (or Reset is clicked), `initState()` reads
`cfg = window.SITE_CONFIG`. Since `SITE_CONFIG.home.steps` is already
`[]`, `state.home.steps` stays `[]` even after
`localStorage.removeItem`. `autoSave()` then re-saves `steps: []`
straight back.

**Secondary issue:** `data-rm-step` / add / move handlers didn't call
`autoSave()`, making step persistence non-deterministic (only saved if a
text field edit happened to be pending).

## Fix

- **`app.js`**: snapshot `SITE_CONFIG.home` before `mergeUserHomeConfig`
→ `window._SITE_CONFIG_ORIGINAL_HOME`
- **`customize.js`**: `initState()` uses `_SITE_CONFIG_ORIGINAL_HOME`
instead of the contaminated `cfg.home`
- **`customize.js`**: add `autoSave()` to rm/move/add handlers for
steps, checklist, and footer links

## Tests

2 new unit tests covering the snapshot bypass and DEFAULTS fallback. 231
tests pass.

## Checklist
- [x] Branches from `upstream/master`
- [x] No Matomo or local-only commits
- [x] Cache busters bumped
- [x] 231 tests pass, 0 fail

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-04-01 18:45:15 -07:00
Kpa-clawbot 75f1295a06 fix: always refresh staging config from prod (#467) 
## Summary

Fixes #466 — staging config was not refreshed from prod due to a stale
`-nt` timestamp guard.

## Root Cause

`prepare_staging_config()` only copied prod config when staging was
missing or prod was newer by mtime. However, the `sed -i` that applies
the STAGING siteName updated staging's mtime, making it appear newer
than prod. Subsequent runs skipped the copy entirely.

## Changes

- **`manage.sh`**: Removed the `-nt` timestamp conditional in
`prepare_staging_config()`. Staging config is now always copied fresh
from prod with the STAGING siteName applied.

Note: `prepare_staging_db()` already copies unconditionally — no change
needed there.

Co-authored-by: you <you@example.com>
 2026-04-01 18:19:10 -07:00
Kpa-clawbot b1b76acb77 feat: manage.sh update supports pinning to release tags (#456) 
## Summary

`manage.sh update` now supports pinning to specific release tags instead
of always pulling tip of master.

Fixes #455

## Changes

### `cmd_update` — accepts optional version argument
- **No argument**: fetches tags, checks out latest release tag (`git tag
-l 'v*' --sort=-v:refname | head -1`)
- **`latest`**: explicit opt-in to tip of master (bleeding edge)
- **Specific tag** (e.g. `v3.1.0`): checks out that exact tag, with
error message + available tags if not found

### `cmd_setup` — defaults to latest tag
- After Docker check, fetches tags and pins to latest release tag
- Skips if already on the latest tag
- Uses state tracking (`version_pin`) so re-runs don't repeat

### `cmd_status` — shows version
- Displays current version (exact tag name or short commit hash) at the
top of status output

### Help text
- Updated to reflect new `update [version]` syntax

## Usage

```bash
./manage.sh update          # checkout latest release tag (e.g. v3.2.0)
./manage.sh update v3.1.0   # pin to specific version
./manage.sh update latest   # explicit tip of master (bleeding edge)
./manage.sh status          # now shows "Version: v3.2.0"
```

## Testing

- `bash -n manage.sh` passes (syntax valid)
- Logic follows existing patterns (git fetch, checkout, rebuild,
restart)

---------

Co-authored-by: you <you@example.com>
 2026-04-01 12:20:28 -07:00
Kpa-clawbot f87eb3601c fix: graceful container shutdown for reliable deployments (#453) 
## Summary

Fixes #450 — staging deployment flaky due to container not shutting down
cleanly.

## Root Causes

1. **Server never closed DB on shutdown** — SQLite WAL lock held
indefinitely, blocking new container startup
2. **`httpServer.Close()` instead of `Shutdown()`** — abruptly kills
connections instead of draining them
3. **No `stop_grace_period` in compose configs** — Docker sends SIGTERM
then immediately SIGKILL (default 10s is often not enough for WAL
checkpoint)
4. **Supervisor didn't forward SIGTERM** — missing
`stopsignal`/`stopwaitsecs` meant Go processes got SIGKILL instead of
graceful shutdown
5. **Deploy scripts used default `docker stop` timeout** — only 10s
grace period

## Changes

### Go Server (`cmd/server/`)
- **Graceful HTTP shutdown**: `httpServer.Shutdown(ctx)` with 15s
context timeout — drains in-flight requests before closing
- **WebSocket cleanup**: New `Hub.Close()` method sends `CloseGoingAway`
frames to all connected clients
- **DB close on shutdown**: Explicitly closes DB after HTTP server stops
(was never closed before)
- **WAL checkpoint**: `PRAGMA wal_checkpoint(TRUNCATE)` before DB close
— flushes WAL to main DB file and removes WAL/SHM lock files

### Go Ingestor (`cmd/ingestor/`)
- **WAL checkpoint on shutdown**: New `Store.Checkpoint()` method,
called before `Close()`
- **Longer MQTT disconnect timeout**: 5s (was 1s) to allow in-flight
messages to drain

### Docker Compose (all 4 variants)
- Added `stop_grace_period: 30s` and `stop_signal: SIGTERM`

### Supervisor Configs (both variants)
- Added `stopsignal=TERM` and `stopwaitsecs=20` to server and ingestor
programs

### Deploy Scripts
- `deploy-staging.sh`: `docker stop -t 30` with explicit grace period
- `deploy-live.sh`: `docker stop -t 30` with explicit grace period

## Shutdown Sequence (after fix)

1. Docker sends SIGTERM to supervisord (PID 1)
2. Supervisord forwards SIGTERM to server + ingestor (waits up to 20s
each)
3. Server: stops poller → drains HTTP (15s) → closes WS clients →
checkpoints WAL → closes DB
4. Ingestor: stops tickers → disconnects MQTT (5s) → checkpoints WAL →
closes DB
5. Docker waits up to 30s total before SIGKILL

## Tests

All existing tests pass:
- `cd cmd/server && go test ./...` ✅
- `cd cmd/ingestor && go test ./...` ✅

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
 2026-04-01 12:19:20 -07:00
Kpa-clawbot ec4dd58cb6 fix: null-guard pathHops to prevent detail pane crash (#451) (#454) 
## Summary

Fixes #451 — packet detail pane crash on direct routed packets where
`pathHops` is `null`.

## Root Cause

`JSON.parse(pkt.path_json)` can return literal `null` when the DB stores
`"null"` for direct routed packets. The existing code only had a catch
block for parse errors, but `null` is valid JSON — so the parse succeeds
and `pathHops` ends up `null` instead of `[]`.

## Changes

- **`public/packets.js`**: Added `|| []` after `JSON.parse(...)` in both
`buildFlatRowHtml` (table rows) and the detail pane (`selectPacket`),
ensuring `pathHops` is always an array.
- **`test-frontend-helpers.js`**: Added 2 regression tests verifying the
null guards exist in both code paths.
- **`public/index.html`**: Cache buster bump.

## Testing

- All 229 frontend helper tests pass
- All 62 packet filter tests pass
- All 29 aging tests pass

Co-authored-by: you <you@example.com>
 2026-04-01 10:48:08 -07:00
Kpa-clawbot 044a5387af perf(packets): virtual scroll + debounced WS renders for packets table (#402) 
## Summary

Fixes the critical performance issue where `renderTableRows()` rebuilt
the **entire** table innerHTML (up to 50K rows) on every update —
WebSocket arrivals, filter changes, group expand/collapse, and theme
refreshes.

## Changes

### Lazy Row Generation (`renderVisibleRows`) — fixes #422
- Row HTML strings are **only generated for the visible slice + 30-row
buffer** on each render
- `_displayPackets` stores the filtered data array;
`renderVisibleRows()` calls `buildGroupRowHtml`/`buildFlatRowHtml`
lazily for ~60-90 visible entries
- Previously, `displayPackets.map(buildGroupRowHtml)` built HTML for ALL
30K+ packets on every render — the expensive work (JSON.parse, observer
lookups, template literals) ran for every packet regardless of
visibility

### Unified Row Count via `_getRowCount()` — fixes #424
- Single function `_getRowCount(p)` computes DOM row count for any entry
(1 for flat/collapsed, 1+children for expanded groups)
- Used by BOTH `_rowCounts` computation AND `renderVisibleRows` —
eliminates divergence risk between row counting and row building

### Hoisted Observer Filter Set — fixes #427
- `_observerFilterSet` created once in `renderTableRows()`, reused
across `buildGroupRowHtml`, `_getRowCount`, and child filtering
- Previously, `new Set(filters.observer.split(','))` was created inside
`buildGroupRowHtml` for every packet AND again in the row count callback

### Dynamic Colspan — fixes #426
- `_getColCount()` reads column count from the thead instead of
hardcoded `colspan="11"`
- Spacers and empty-state messages use the actual column count

### Null-Safety in `buildFlatRowHtml` — fixes #430
- `p.decoded_json || '{}'` fallback added, matching
`buildGroupRowHtml`'s existing null-safety
- Prevents TypeError on null/undefined `decoded_json` in flat
(ungrouped) mode

### Behavioral Tests — fixes #428
- Replaced 5 source-grep tests with behavioral unit tests for
`_getRowCount`:
  - Flat mode always returns 1
  - Collapsed group returns 1
  - Expanded group returns 1 + child count
  - Observer filter correctly reduces child count
  - Null `_children` handled gracefully
- Retained source-level assertions only where behavioral testing isn't
practical (e.g., verifying lazy generation pattern exists)

### Other Improvements
- Cumulative row offsets cached in `_cumulativeOffsetsCache`,
invalidated on row count changes
- Debounced WebSocket renders (200ms) coalesce rapid packet arrivals
- `destroy()` properly cleans up all virtual scroll state

## Performance Benchmarks — fixes #423

**Methodology:** Row building cost measured by counting
`buildGroupRowHtml` calls per render cycle on 30K grouped packets.

| Scenario | Before (eager) | After (lazy) | Improvement |
|----------|----------------|--------------|-------------|
| Initial render (30K packets) | 30,000 `buildGroupRowHtml` calls | ~90
calls (60 visible + 30 buffer) | **333× fewer calls** |
| Scroll event | 0 calls (pre-built) | ~90 calls (rebuild visible slice)
| Trades O(1) scroll for O(n) initial savings |
| WS packet arrival | 30,000 calls (full rebuild) | ~90 calls (debounced
+ lazy) | **333× fewer calls** |
| Filter change | 30,000 calls | ~90 calls | **333× fewer calls** |
| Memory (row HTML cache) | ~2MB string array for 30K packets | 0 (no
cache, build on demand) | **~2MB saved** |

**Per-call cost of `buildGroupRowHtml`:** Each call performs JSON.parse
of `decoded_json`, `path_json`, `observers.find()` lookup, and template
literal construction. At 30K packets, the eager approach spent
~400-500ms on row building alone (measured via `performance.now()` on
staging data). The lazy approach builds ~90 rows in ~1-2ms.

**Net effect:** `renderTableRows()` goes from O(n) string building +
O(1) DOM insertion to O(1) data assignment + O(visible) string building
+ O(visible) DOM insertion. For n=30K and visible≈60, this is ~333× less
work per render cycle.

**Trade-off:** Scrolling now rebuilds ~90 rows per RAF frame instead of
slicing pre-built strings. This costs ~1-2ms per scroll event, well
within the 16ms frame budget. The trade-off is overwhelmingly positive
since renders happen far more frequently than full-table scrolls.

## Tests

- 247 frontend helper tests pass (including 18 virtual scroll tests)
- 62 packet filter tests pass
- 29 aging tests pass
- Go backend tests pass

## Remaining Debt (tracked in issues)

- #425: Hardcoded `VSCROLL_ROW_HEIGHT=36` and `theadHeight=40` — should
be measured from DOM
- #429: 200ms WS debounce delay — value works well in practice but lacks
formal justification
- #431: No scroll position preservation on filter change or group
expand/collapse

Fixes #380

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
 2026-04-01 09:34:04 -07:00
Kpa-clawbot 01ca843309 perf: move collision analysis to server-side endpoint (fixes #386) (#415) 
## Summary

Moves the hash collision analysis from the frontend to a new server-side
endpoint, eliminating a major performance bottleneck on the analytics
collision tab.

Fixes #386

## Problem

The collision tab was:
1. **Downloading all nodes** (`/nodes?limit=2000`) — ~500KB+ of data
2. **Running O(n²) pairwise distance calculations** on the browser main
thread (~2M comparisons with 2000 nodes)
3. **Building prefix maps client-side** (`buildOneBytePrefixMap`,
`buildTwoBytePrefixInfo`, `buildCollisionHops`) iterating all nodes
multiple times

## Solution

### New endpoint: `GET /api/analytics/hash-collisions`

Returns pre-computed collision analysis with:
- `inconsistent_nodes` — nodes with varying hash sizes
- `by_size` — per-byte-size (1, 2, 3) collision data:
  - `stats` — node counts, space usage, collision counts
- `collisions` — pre-computed collisions with pairwise distances and
classifications (local/regional/distant/incomplete)
  - `one_byte_cells` — 256-cell prefix map for 1-byte matrix rendering
- `two_byte_cells` — first-byte-grouped data for 2-byte matrix rendering

### Caching

Uses the existing `cachedResult` pattern with a new `collisionCache`
map. Invalidated on `hasNewTransmissions` (same trigger as the
hash-sizes cache) and on eviction.

### Frontend changes

- `renderCollisionTab` now accepts pre-fetched `collisionData` from the
parallel API load
- New `renderHashMatrixFromServer` and `renderCollisionsFromServer`
functions consume server-computed data directly
- No more `/nodes?limit=2000` fetch from the collision tab
- Old client-side functions (`buildOneBytePrefixMap`, etc.) preserved
for test helper exports

## Test results

- `go test ./...` (server): ✅ pass
- `go test ./...` (ingestor): ✅ pass
- `test-packet-filter.js`: ✅ 62 passed
- `test-aging.js`: ✅ 29 passed
- `test-frontend-helpers.js`: ✅ 227 passed

## Performance impact

| Metric | Before | After |
|--------|--------|-------|
| Data transferred | ~500KB (all nodes) | ~50KB (collision data only) |
| Client computation | O(n²) distance calc | None (server-cached) |
| Main thread blocking | Yes (2000 nodes × pairwise) | No |
| Server caching | N/A | 15s TTL, invalidated on new transmissions |

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
 2026-04-01 09:21:23 -07:00
Kpa-clawbot 5f50e80931 perf: replace server round-trip with client-side filter for My Nodes toggle (#401) 
## Summary

Fixes #381 — The "My Nodes" filter in `packets.js` was making a **server
API call inside `renderTableRows()`** on every render cycle. With
WebSocket updates arriving every few seconds while the toggle was
active, this created continuous unnecessary server load.

## What Changed

**`public/packets.js`** — Replaced the `api('/packets?nodes=...')`
server call with a pure client-side filter:

```js
// Before: server round-trip on every render
const myData = await api('/packets?nodes=' + allKeys.join(',') + '&limit=500');
displayPackets = myData.packets || [];

// After: filter already-loaded packets client-side
displayPackets = displayPackets.filter(p => {
  const dj = p.decoded_json || '';
  return allKeys.some(k => dj.includes(k));
});
```

This uses the exact same matching logic as the server's
`QueryMultiNodePackets()` — a string contains check on `decoded_json`
for each pubkey — but without the network round-trip.

**`test-frontend-helpers.js`** — Added 5 unit tests for the filter
logic:
- Single and multiple pubkey matching
- No matches / empty keys edge case
- Null/empty `decoded_json` handled gracefully

**`public/index.html`** — Cache busters bumped.

## Test Results

- Frontend helpers: **232 passed, 0 failed** (including 5 new tests)
- Packet filter: **62 passed, 0 failed**
- Aging: **29 passed, 0 failed**

Co-authored-by: you <you@example.com>
 2026-04-01 08:27:06 -07:00
you 8f3d12eca5 docs: perf claims require proof — benchmarks, timings, or test assertions 2026-04-01 15:08:06 +00:00
you 357f7952f7 docs: add Rule 0 — performance-first mindset in AGENTS.md 2026-04-01 14:55:35 +00:00
Kpa-clawbot 47d081c705 perf: targeted analytics cache invalidation (fixes #375) (#379) 
## Problem

Every time new data is ingested (`IngestNewFromDB`,
`IngestNewObservations`, `EvictStale`), **all 6 analytics caches** are
wiped by creating new empty maps — regardless of what kind of data
actually changed. With the poller running every 1 second, this means the
15s cache TTL is effectively bypassed because caches are cleared far
more frequently than they expire.

## Fix

Introduces a `cacheInvalidation` flags struct and
`invalidateCachesFor()` method that selectively clears only the caches
affected by the ingested data:

| Flag | Caches Cleared |
|------|----------------|
| `hasNewObservations` | RF (SNR/RSSI data changed) |
| `hasNewPaths` | Topology, Distance, Subpaths |
| `hasNewTransmissions` | Hash sizes |
| `hasChannelData` | Channels (GRP_TXT payload_type 5) + channels list
cache |
| `eviction` | All (data removed, everything potentially stale) |

### Impact

For a typical ingest cycle with ADVERT/ACK/TXT_MSG packets (no GRP_TXT):
- **Before:** All 6 caches cleared every cycle
- **After:** Channel cache preserved (most common case), hash cache
preserved on observation-only ingestion

For observation-only ingestion (`IngestNewObservations`):
- **Before:** All 6 caches cleared
- **After:** Only RF cache cleared (+ topo/dist/subpath if paths
actually changed)

## Tests

7 new unit tests in `cache_invalidation_test.go` covering:
- Eviction clears all caches
- Observation-only ingest preserves non-RF caches
- Transmission-only ingest clears only hash cache
- Channel data clears only channel cache
- Path changes clear topo/dist/subpath
- Combined flags work correctly
- No flags = no invalidation

All existing tests pass.

### Post-rebase fix

Restored `channelsCacheRes` invalidation that was accidentally dropped
during the refactor. The old code cleared this separate channels list
cache on every ingest, but `invalidateCachesFor()` didn't include it.
Now cleared on `hasChannelData` and `eviction`.

Fixes #375

---------

Co-authored-by: you <you@example.com>
 2026-04-01 07:37:39 -07:00
Kpa-clawbot be313f60cb fix: extract score/direction from MQTT, strip units, fix type safety issues (#371) 
## Summary

Fixes #353 — addresses all 5 findings from the CoreScope code analysis.

## Changes

### Finding 1 (Major): `score` field never extracted from MQTT
- Added `Score *float64` field to `PacketData` and `MQTTPacketMessage`
structs
- Extract `msg["score"]` with `msg["Score"]` case fallback via
`toFloat64` in all three MQTT handlers (raw packet, channel message,
direct message)
- Pass through to DB observation insert instead of hardcoded `nil`

### Finding 2 (Major): `direction` field never extracted from MQTT
- Added `Direction *string` field to `PacketData` and
`MQTTPacketMessage` structs
- Extract `msg["direction"]` with `msg["Direction"]` case fallback as
string in all three MQTT handlers
- Pass through to DB observation insert instead of hardcoded `nil`

### Finding 3 (Minor): `toFloat64` doesn't strip units
- Added `stripUnitSuffix()` that removes common RF/signal unit suffixes
(dBm, dB, mW, km, mi, m) case-insensitively before `ParseFloat`
- Values like `"-110dBm"` or `"5.5dB"` now parse correctly

### Finding 4 (Minor): Bare type assertions in store.go
- Changed `firstSeen` and `lastSeen` from `interface{}` to typed
`string` variables at `store.go:5020`
- Removed unsafe `.(string)` type assertions in comparisons

### Finding 5 (Minor): `distHopRecord.SNR` typed as `interface{}`
- Changed `distHopRecord.SNR` from `interface{}` to `*float64`
- Updated assignment (removed intermediate `snrVal` variable, pass
`tx.SNR` directly)
- Updated output serialization to use `floatPtrOrNil(h.SNR)` for
consistent JSON output

## Tests Added

- `TestBuildPacketDataScoreAndDirection` — verifies Score/Direction flow
through BuildPacketData
- `TestBuildPacketDataNilScoreDirection` — verifies nil handling when
fields absent
- `TestInsertTransmissionWithScoreAndDirection` — end-to-end: inserts
with score/direction, verifies DB values
- `TestStripUnitSuffix` — covers all supported suffixes, case
insensitivity, and passthrough
- `TestToFloat64WithUnits` — verifies unit-bearing strings parse
correctly

All existing tests pass.

Co-authored-by: you <you@example.com>
 2026-04-01 07:26:23 -07:00
efiten 8a0862523d fix: add migration for missing observations.timestamp index (#332) 
## Problem

On installations where the database predates the
`idx_observations_timestamp` index, `/api/stats` takes 30s+ because
`GetStoreStats()` runs two full table scans:

```sql
SELECT COUNT(*) FROM observations WHERE timestamp > ?  -- last hour
SELECT COUNT(*) FROM observations WHERE timestamp > ?  -- last 24h
```

The index is only created in the `if !obsExists` block, so any database
where the `observations` table already existed before that code was
added never gets it.

## Fix

Adds a one-time migration (`obs_timestamp_index_v1`) that runs at
ingestor startup:

```sql
CREATE INDEX IF NOT EXISTS idx_observations_timestamp ON observations(timestamp)
```

On large installations this index creation may take a few seconds on
first startup after the upgrade, but subsequent stats queries become
instant.

## Test plan
- [ ] Restart ingestor on an older database and confirm `[migration]
observations timestamp index created` appears in logs
- [ ] Confirm `/api/stats` response time drops from 30s+ to <100ms

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 07:06:54 -07:00
efiten 7e8b30aa1f perf: fix slow /api/packets and /api/channels on large stores (#328) 
## Problem

Two endpoints were slow on larger installations:

**`/packets?limit=50000&groupByHash=true` — 16s+**
`QueryGroupedPackets` did two expensive things on every request:
1. O(n × observations) scan per packet to find `latest` timestamp
2. Held `s.mu.RLock()` during the O(n log n) sort, blocking all
concurrent reads

**`/channels` — 13s+**
`GetChannels` iterated all payload-type-5 packets and JSON-unmarshaled
each one while holding `s.mu.RLock()`, blocking all concurrent reads for
the full duration.

## Fix

**Packets (`QueryGroupedPackets`):**
- Add `LatestSeen string` to `StoreTx`, maintained incrementally in all
three observation write paths. Eliminates the per-packet observation
scan at query time.
- Build output maps under the read lock, sort the local copy after
releasing it.
- Cache the full sorted result for 3 seconds keyed by filter params.

**Channels (`GetChannels`):**
- Copy only the fields needed (firstSeen, decodedJSON, region match)
under the read lock, then release before JSON unmarshaling.
- Cache the result for 15 seconds keyed by region param.
- Invalidate cache on new packet ingestion.

## Test plan
- [ ] Open packets page on a large store — load time should drop from
16s to <1s
- [ ] Open channels page — should load in <100ms instead of 13s+
- [ ] `[SLOW API]` warnings gone for both endpoints
- [ ] Packet/channel data is correct (hashes, counts, observer counts)
- [ ] Filters (region, type, since/until) still work correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 07:06:21 -07:00