## Summary
Several features and fixes from a live deployment of the Go v3.0.0
backend.
### geo_filter — full enforcement
- **Go backend config** (`cmd/server/config.go`,
`cmd/ingestor/config.go`): added `GeoFilterConfig` struct so
`geo_filter.polygon` and `bufferKm` from `config.json` are parsed by
both the server and ingestor
- **Ingestor** (`cmd/ingestor/geo_filter.go`, `cmd/ingestor/main.go`):
ADVERT packets from nodes outside the configured polygon + buffer are
dropped *before* any DB write — no transmission, node, or observation
data is stored
- **Server API** (`cmd/server/geo_filter.go`, `cmd/server/routes.go`):
`GET /api/config/geo-filter` endpoint returns the polygon + bufferKm to
the frontend; `/api/nodes` responses filter out any out-of-area nodes
already in the DB
- **Frontend** (`public/map.js`, `public/live.js`): blue polygon overlay
(solid inner + dashed buffer zone) on Map and Live pages, toggled via
"Mesh live area" checkbox, state shared via localStorage
### Automatic DB pruning
- Add `retention.packetDays` to `config.json` to delete transmissions +
observations older than N days on a daily schedule (1 min after startup,
then every 24h). Nodes and observers are never pruned.
- `POST /api/admin/prune?days=N` for manual runs (requires `X-API-Key`
header if `apiKey` is set)
```json
"retention": {
"nodeDays": 7,
"packetDays": 30
}
```
### tools/geofilter-builder.html
Standalone HTML tool (no server needed) — open in browser, click to
place polygon points on a Leaflet map, set `bufferKm`, copy the
generated `geo_filter` JSON block into `config.json`.
### scripts/prune-nodes-outside-geo-filter.py
Utility script to clean existing out-of-area nodes from the database
(dry-run + confirm). Useful after first enabling geo_filter on a
populated DB.
### HB column in packets table
Shows the hop hash size in bytes (1–4) decoded from the path byte of
each packet's raw hex. Displayed as **HB** between Size and Type
columns, hidden on small screens.
## Test plan
- [x] ADVERT from node outside polygon is not stored (no new row in
nodes or transmissions)
- [x] `GET /api/config/geo-filter` returns polygon + bufferKm when
configured, `{polygon: null, bufferKm: 0}` when not
- [x] `/api/nodes` excludes nodes outside polygon even if present in DB
- [x] Map and Live pages show blue polygon overlay when configured;
checkbox toggles it
- [x] `retention.packetDays: 30` deletes old transmissions/observations
on startup and daily
- [x] `POST /api/admin/prune?days=30` returns `{deleted: N, days: 30}`
- [x] `tools/geofilter-builder.html` opens standalone, draws polygon,
copies valid JSON
- [x] HB column shows 1–4 for all packets in grouped and flat view
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Complete CI pipeline restructure. Sequential fail-fast chain, E2E tests
against Go server with real staging data, all deprecated Node.js server
tests removed.
### Pipeline (PR):
1. **Go unit tests** — fail-fast, coverage + badges
2. **Playwright E2E** — against Go server with fixture DB, frontend
coverage, fail-fast on first failure
3. **Docker build** — verify containers build
### Pipeline (master merge):
Same chain + deploy to staging + badge publishing
### Removed:
- All Node.js server-side unit tests (deprecated JS server)
- `npm ci` / `npm run test` steps
- JS server coverage collection (`COVERAGE=1 node server.js`)
- Changed-files detection logic
- Docs-only CI skip logic
- Cancel-workflow API hacks
### Added:
- `test-fixtures/e2e-fixture.db` — real data from staging (200 nodes, 31
observers, 500 packets)
- `scripts/capture-fixture.sh` — refresh fixture from staging API
- Go server launches with `-port 13581 -db test-fixtures/e2e-fixture.db
-public public-instrumented`
---------
Co-authored-by: Kpa-clawbot <kpabap+clawdbot@gmail.com>
Co-authored-by: you <you@example.com>
Three optimizations to reduce wall-clock time:
1. Reduce safeClick timeout from 3000ms to 500ms
- Elements either exist immediately after navigation or don't exist at all
- ~75 safeClick calls; if ~30 miss, saves ~75s of dead wait time
2. Replace 18 page.goto() calls with SPA hash navigation
- After initial page load, the SPA shell is already in the DOM
- page.goto() reloads the entire page (network round-trip + parse)
- Hash navigation via location.hash triggers the SPA router instantly
- Only 3 page.goto() remain: initial load + 2 home page loads after localStorage.clear()
3. Remove redundant final route sweep
- All 10 routes were already visited during the page-specific sections
- The sweep just re-navigated to pages that had already been exercised
- Saves ~2s of redundant navigation
Also:
- Reduce inter-route wait from 200ms to 50ms (SPA router is synchronous)
- Merge utility function + packet filter exercises into single evaluate() call
- Use navHash() helper for consistent hash navigation with 150ms settle time
Three optimizations to the CI frontend test pipeline:
1. Run E2E tests and coverage collection concurrently
- Previously sequential (E2E ~1.5min, then coverage ~5.75min)
- Now both run in parallel against the same instrumented server
- Expected savings: ~5 min (coverage runs alongside E2E instead of after)
2. Replace networkidle with domcontentloaded in coverage collector
- SPA uses hash routing — networkidle waits 500ms for network silence
on every navigation, adding ~10-15s of dead time across 23 navigations
- domcontentloaded fires immediately once HTML is parsed; JS initializes
the route handler synchronously
- For in-page hash changes, use 200ms setTimeout instead of
waitForLoadState (which would never re-fire for same-document nav)
3. Extract coverage from E2E tests too
- E2E tests already exercise the app against the instrumented server
- Now writes window.__coverage__ to .nyc_output/e2e-coverage.json
- nyc merges both coverage files for higher total coverage
Also:
- Split Playwright install into browser + deps steps (deps skip if present)
- Replace sleep 5 with health-check poll in quick E2E path
Remove all 169 waitForTimeout() calls (totaling 104.1s of blind sleeping)
from scripts/collect-frontend-coverage.js:
- Helper functions (safeClick, safeFill, safeSelect, clickAll, cycleSelect):
removed 300-400ms waits after every interaction — Playwright's built-in
actionability checks handle waiting for elements automatically
- Post-navigation waits: removed redundant sleeps after page.goto() calls
that already use waitUntil: 'networkidle'
- Hash-change navigations: replaced waitForTimeout with
waitForLoadState('networkidle') for proper SPA route settling
- Toggle/button waits: removed — event handlers execute synchronously
before click() resolves
- Post-evaluate waits: removed — evaluate() is synchronous
Local benchmark (Windows, sparse test data):
Before: 744.8s
After: 484.8s (35% faster, 260s saved)
On CI runner (ARM Linux with real mesh data), savings will be
proportionally better since most elements exist and the 104s
of blind sleeping was the dominant bottleneck.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The page.evaluate() calls corrupting localStorage and firing fake events
caused page error-reloads, losing accumulated coverage. Reverting to
the 42% version which was the actual high water mark.
Exercise every major code path across all frontend files:
app.js: all routes, bad routes, hashchange, theme toggle x4,
hamburger menu, favorites dropdown, global search, Ctrl+K,
apiPerf(), timeAgo/truncate/routeTypeName utils
nodes.js: sort every column (both directions), every role tab,
every status filter, cycle all Last Heard options, click rows
for side pane, navigate to detail page, copy URL, show all
paths, node analytics day buttons (1/7/30/365), scroll target
packets.js: 12 filter expressions including bad ones, cycle all
time windows, group by hash toggle, My Nodes toggle, observer
menu, type filter menu, hash input, node filter, observer sort,
column toggle menu, hex hash toggle, pause button, resize handle,
deep-link to packet hash
map.js: all role checkboxes toggle, clusters/heatmap/neighbors/
hash labels toggles, cycle Last Heard, status filter buttons,
jump buttons, markers, zoom controls, dark mode tile swap
analytics.js: all 9 tabs clicked, deep-link to each tab via URL,
observer selector on topology, navigate rows on collisions/
subpaths, sortable headers on nodes tab, region filter
customize.js: all 5 tabs, all preset themes, branding text inputs,
theme color inputs, node color inputs, type color inputs, reset
buttons, home tab fields (hero, journey steps, checklist, links),
export tab, reset preview/user theme
live.js: VCR pause/speed/missed/prompt buttons, all visualization
toggles (heat/ghost/realistic/favorites/matrix/rain), audio
toggle + BPM slider, timeline click, resize event
channels.js: click rows, navigate to specific channel
observers.js: click rows, navigate to detail, cycle days select
traces.js: click rows
perf.js: refresh + reset buttons
home.js: both chooser paths, search + suggest, my-node cards,
health/packets buttons, remove buttons, toggle level, timeline
Also exercises packet-filter parser and region-filter directly.
- Install nyc for Istanbul instrumentation
- Add scripts/instrument-frontend.sh to instrument public/*.js
- Add scripts/collect-frontend-coverage.js to extract window.__coverage__
- Add scripts/combined-coverage.sh for combined server+frontend coverage
- Make server.js serve public-instrumented/ when COVERAGE=1 is set
- Add test:full-coverage npm script
- Add public-instrumented/ and .nyc_output/ to .gitignore
Creates transmissions and observations tables from existing packets table.
- Groups packets by hash → 1 transmission per unique hash
- Creates 1 observation per original packet row with FK to transmission
- Idempotent: drops and recreates new tables on each run
- Does NOT modify the original packets table
- Prints stats and verifies counts match
Tested on test DB: 33813 packets → 11530 transmissions (2.93x dedup ratio)
