Compare commits

...

262 Commits

Author SHA1 Message Date
you 5bec14222a fix: address PR #905 review — migration error handling, backfill heuristic, test comment
1. Migration ALTER error no longer swallowed: check error from ALTER TABLE
   and return if it fails (unless column already exists). Migration is not
   marked complete on failure.

2. Backfill heuristic fixed: use observations table JOIN instead of
   packet_count > 0, since UpsertObserver sets packet_count = 1 on INSERT
   even for status-only observers.

3. Test clarifying comment: document that InsertTransmission uses
   data.Timestamp (not time.Now()) as source-of-truth for last_packet_at,
   so the hardcoded assertion is correct.
2026-04-24 15:38:05 +00:00
you e9f977cd70 fix: bump obs-table min-width to 720px for new Last Packet column
The addition of the Last Packet column brings the table to 8 columns.
The previous min-width of 640px was tight for 7 columns; 720px prevents
cramped rendering and ensures the horizontal scroll trigger is appropriate
on narrow viewports.
2026-04-24 15:31:23 +00:00
you 8a15ea903b test: add last_packet_at tests for ingestor and server
- Ingestor: verify last_packet_at is NULL after UpsertObserver (status path),
  set after InsertTransmission, and unchanged by subsequent UpsertObserver calls
- Server: verify last_packet_at reads back through GetObservers and GetObserverByID
2026-04-24 15:26:59 +00:00
you 330970cce9 feat(ui): show separate Last Status and Last Packet columns for observers
- observers.js: rename 'Last Seen' column to 'Last Status', add 'Last Packet'
  column with a warning badge when no packets observed or packets lag behind
  status by >10min
- observer-detail.js: add 'Last Status Update' and 'Last Packet Observation'
  stat cards with relative + absolute timestamps
2026-04-24 15:26:53 +00:00
you d3a40919f2 feat: add last_packet_at column to observers
Add a new 'last_packet_at' column to the observers table that is only
bumped when an actual packet observation lands (InsertTransmission path),
while 'last_seen' continues to be bumped on both status updates and packets.

This allows the UI to distinguish between an observer that is alive
(sending status pings) vs one that is actively forwarding packets.

Schema migration backfills last_packet_at = last_seen for observers
with packet_count > 0. Server API now returns last_packet_at in the
Observer JSON response.
2026-04-24 15:26:47 +00:00
Kpa-clawbot a47fe26085 fix(channels): allow removing user-added keys for server-known channels (#898)
## Problem
Adding a channel key in the Channels UI for a channel the server already
knows about (e.g. `#public` from rainbow / config) leaves the
localStorage entry **unremovable**:

- `mergeUserChannels` sees the name already exists in the channel list
and skips the user entry.
- The existing channel row is never marked `userAdded:true`.
- The ✕ button (`[data-remove-channel]`) is only rendered for
`userAdded` rows.
- Result: stuck localStorage key, no UI to delete it.

There was also a latent bug in the remove handler — for non-`user:`
rows, it used the raw hash (e.g. `enc_11`) as the
`ChannelDecrypt.removeKey()` argument, but the storage key is the
channel **name**.

## Fix
1. **`mergeUserChannels`**: when a stored key matches an existing
channel by name/hash, mark the existing channel `userAdded=true` so the
✕ renders on it. (No magical/auto deletion of stored keys — the user
explicitly chooses to remove.)
2. **Remove handler**:
- Look up the channel object to get the correct display name for the
localStorage key.
- Keep server-known channels in the list when their ✕ is clicked (only
the user's localStorage entry + cache are cleared, `userAdded` is
unset). The channel still exists upstream.
   - Pure `user:`-prefixed channels are removed from the list as before.

## Repro
1. Open Channels.
2. Add a key for `#public` (or any rainbow-known channel).
3. Reload. Before this PR: row has no ✕, key is stuck. After this PR: ✕
appears, click clears the local key and cache.

## Files
- `public/channels.js` only.

## Notes
- No backend changes.
- No new APIs.
- Behaviour for purely user-added channels (e.g. `user:#somechannel` not
known to the server) is unchanged.

---------

Co-authored-by: you <you@example.com>
2026-04-22 21:41:43 -07:00
Kpa-clawbot abd9c46aa7 fix: side-panel Details button opens full-screen on desktop (#892)
## Symptom
🔍 Details button in the nodes side panel does nothing on click.

## Root cause (4th regression of the same shape)
- Row click → `selectNode()` → `history.replaceState(null, '',
'#/nodes/' + pk)`
- Details button click → `location.hash = '#/nodes/' + pk`
- Hash is already that value → assignment is a no-op → no `hashchange`
event → no router → panel stays open.

## Fix
Mirror the analytics-link branch already inside the panel click handler:
`destroy()` then `init(appEl, pubkey)` directly (which hits the
`directNode` full-screen branch unconditionally). Also `replaceState` to
keep the URL in sync.

## Test
New Playwright E2E: open side panel via row click, click Details, assert
`.node-fullscreen` appears.

## Why this keeps regressing
Every time we tighten the row-click handler to use `replaceState`
(correct — avoids hashchange flicker), the button-click handler that
uses `location.hash` becomes a no-op for the same pubkey. Need to
remember they're coupled. Worth a follow-up to extract a
`navigateToNode(pk)` helper that always works regardless of current hash
state — filing as #890-followup if not already there.

Co-authored-by: you <you@example.com>
2026-04-21 22:37:15 -07:00
Kpa-clawbot 6ca5e86df6 fix: compute hex-dump byte ranges client-side from per-obs raw_hex (#891)
## Symptom
The colored byte strip in the packet detail pane is offset from the
labeled byte breakdown below it. Off by N bytes where N is the
difference between the top-level packet's path length and the displayed
observation's path length.

## Root cause
Server computes `breakdown.ranges` once from the top-level packet's
raw_hex (in `BuildBreakdown`) and ships it in the API response. After
#882 we render each observation's own raw_hex, but we keep using the
top-level breakdown — so a 7-hop top-level packet shipped "Path: bytes
2-8", and when we rendered an 8-hop observation we coloured 7 of the 8
path bytes and bled into the payload.

The labeled rows below (which use `buildFieldTable`) parse the displayed
raw_hex on the client, so they were correct — they just didn't match the
strip above.

## Fix
Port `BuildBreakdown()` to JS as `computeBreakdownRanges()` in `app.js`.
Use it in `renderDetail()` from the actually-rendered (per-obs) raw_hex.

## Test
Manually verified the JS function output matches the Go implementation
for FLOOD/non-transport, transport, ADVERT, and direct-advert (zero
hops) cases.

Closes nothing (caught in post-tag bug bash).

---------

Co-authored-by: you <you@example.com>
2026-04-21 22:17:14 -07:00
Kpa-clawbot 56ec590bc4 fix(#886): derive path_json from raw_hex at ingest (#887)
## Problem

Per-observation `path_json` disagrees with `raw_hex` path section for
TRACE packets.

**Reproducer:** packet `af081a2c41281b1e`, observer `lutin🏡`
- `path_json`: `["67","33","D6","33","67"]` (5 hops — from TRACE
payload)
- `raw_hex` path section: `30 2D 0D 23` (4 bytes — SNR values in header)

## Root Cause

`DecodePacket` correctly parses TRACE packets by replacing `path.Hops`
with hop IDs from the payload's `pathData` field (the actual route).
However, the header path bytes for TRACE packets contain **SNR values**
(one per completed hop), not hop IDs.

`BuildPacketData` used `decoded.Path.Hops` to build `path_json`, which
for TRACE packets contained the payload-derived hops — not the header
path bytes that `raw_hex` stores. This caused `path_json` and `raw_hex`
to describe completely different paths.

## Fix

- Added `DecodePathFromRawHex(rawHex)` — extracts header path hops
directly from raw hex bytes, independent of any TRACE payload
overwriting.
- `BuildPacketData` now calls `DecodePathFromRawHex(msg.Raw)` instead of
using `decoded.Path.Hops`, guaranteeing `path_json` always matches the
`raw_hex` path section.

## Tests (8 new)

**`DecodePathFromRawHex` unit tests:**
- hash_size 1, 2, 3, 4
- zero-hop direct packets
- transport route (4-byte transport codes before path)

**`BuildPacketData` integration tests:**
- TRACE packet: asserts path_json matches raw_hex header path (not
payload hops)
- Non-TRACE packet: asserts path_json matches raw_hex header path

All existing tests continue to pass (`go test ./...` for both ingestor
and server).

Fixes #886

---------

Co-authored-by: you <you@example.com>
2026-04-21 21:13:58 -07:00
Kpa-clawbot 67aa47175f fix: path pill and byte breakdown agree on hop count (#885)
## Problem
On the packet detail pane, the **path pill** (top) and the **byte
breakdown** (bottom) showed different numbers of hops for the same
packet. Example: `46cf35504a21ef0d` rendered as `1 hop` badge followed
by 8 node names in the path pill, while the byte breakdown listed only 1
hop row.

## Root cause
Mixed data sources:
- Path-pill badge used `(raw_hex path_len) & 0x3F` (= firmware truth for
one observer = 1)
- Path-pill names used `path_json.length` (= server-aggregated longest
path across observers = 8)
- Byte breakdown section header used `(raw_hex path_len) & 0x3F` (= 1)
- Byte breakdown rows were sliced from `raw_hex` (= 1 row)
- `renderPath(pathHops, ...)` iterated all `path_json` entries

For group-header view, `packet.path_json` is aggregated across observers
and therefore longer than the raw_hex of any single observer's packet.

## Fix
Both surfaces now render from `pathHops` (= effective observation's
`path_json`). The raw_hex vs path_json mismatch is still logged as a
console.warn for diagnostics, but does not drive the UI.

With per-observation `raw_hex` (#882) shipped, clicking an observation
row already swaps the effective packet so both surfaces stay consistent.

## Testing
- Adds E2E regression `Packet detail path pill and byte breakdown agree
on hop count` that asserts:
  1. `pill badge count == byte breakdown section count`
  2. `rendered hop names ≈ badge count` (within 1 for separators)
  3. `byte breakdown rendered rows == section count`
- Manually reproduced on staging with `46cf35504a21ef0d` (8-name path +
`1 hop` badge before fix).

Related: #881 #882 #866

---------

Co-authored-by: you <you@example.com>
2026-04-21 17:57:06 -07:00
Kpa-clawbot 2b9f305698 fix(#874): hop-resolver affinity picker — score candidates by neighbor-graph edges + geographic centroid (#876)
## Problem

`pickByAffinity` in `hop-resolver.js` picks wrong regional candidates
when 1-byte pubkey prefixes collide. The old implementation only
considers one adjacent hop (forward OR backward pass), leading to
suboptimal picks when both neighbors provide useful context.

Measured on staging: **61.6% of hops have ≥2 same-prefix candidates**,
making collision resolution critical.

## Fix

Replaced the separate forward/backward pass disambiguation with a
**combined iterative resolver** that scores candidates against BOTH prev
and next resolved hops:

1. **Neighbor-graph edge weight** (priority 1): Sum edge scores to prev
+ next pubkeys. Pick max sum.
2. **Geographic centroid** (priority 2): Average lat/lon of prev + next
positions. Pick closest candidate by haversine distance.
3. **Single-anchor geo** (priority 3): When only one neighbor is
resolved, use it directly.
4. **Fallback** (priority 4): First candidate when no context exists.

The iterative approach resolves cascading dependencies — resolving one
ambiguous hop may unlock context for its neighbors.

### Dev-mode trace

Multi-candidate picks now emit: `[hop-resolver] hash=46 candidates=N
scored=[...] chose=<pubkey> method=graph|centroid|fallback`

## Before/After (staging, 1539 packets, 12928 hops)

| Metric | Before | After |
|--------|--------|-------|
| Unreliable hops | 39 (0.3%) | 23 (0.2%) |
| Packets with unreliable | 33 (2.14%) | 17 (1.10%) |

~41% reduction in unreliable hops, ~48% reduction in affected packets.

## Tests

5 new tests in `test-frontend-helpers.js`:
- Graph edge scoring picks correct regional candidate
- Next hop breaks tie when prev has no edges
- Centroid fallback when no graph edges exist
- Centroid uses average of prev+next positions
- Fallback when no context at all

All 595 tests pass. No regressions in `test-packet-filter.js` (62 pass)
or `test-aging.js` (29 pass).

Closes #874

---------

Co-authored-by: you <you@example.com>
2026-04-21 14:03:40 -07:00
Kpa-clawbot a605518d6d fix(#881): per-observation raw_hex — each observer sees different bytes on air (#882)
## Problem

Each MeshCore observer receives a physically distinct over-the-air byte
sequence for the same transmission (different path bytes, flags/hops
remaining). The `observations` table stored only `path_json` per
observer — all observations pointed at one `transmissions.raw_hex`. This
prevented the hex pane from updating when switching observations in the
packet detail view.

## Changes

| Layer | Change |
|-------|--------|
| **Schema** | `ALTER TABLE observations ADD COLUMN raw_hex TEXT`
(nullable). Migration: `observations_raw_hex_v1` |
| **Ingestor** | `stmtInsertObservation` now stores per-observer
`raw_hex` from MQTT payload |
| **View** | `packets_v` uses `COALESCE(o.raw_hex, t.raw_hex)` —
backward compatible with NULL historical rows |
| **Server** | `enrichObs` prefers `obs.RawHex` when non-empty, falls
back to `tx.RawHex` |
| **Frontend** | No changes — `effectivePkt.raw_hex` already flows
through `renderDetail` |

## Tests

- **Ingestor**: `TestPerObservationRawHex` — two MQTT packets for same
hash from different observers → both stored with distinct raw_hex
- **Server**: `TestPerObservationRawHexEnrich` — enrichObs returns
per-obs raw_hex when present, tx fallback when NULL
- **E2E**: Playwright assertion in `test-e2e-playwright.js` for hex pane
update on observation switch

E2E assertion added: `test-e2e-playwright.js:1794`

## Scope

- Historical observations: raw_hex stays NULL, UI falls back to
transmission raw_hex silently
- No backfill, no path_json reconstruction, no frontend changes

Closes #881

---------

Co-authored-by: you <you@example.com>
2026-04-21 13:45:29 -07:00
Kpa-clawbot 0ca559e348 fix(#866): per-observation children in expanded packet groups (#880)
## Problem
When a packet group is expanded in the Packets table, clicking any child
row pointed the side pane at the same aggregate packet — not the clicked
observation. URL would flip between `?obs=<packet_id>` values instead of
real observation ids.

## Root cause
The expand fetch used `/api/packets?hash=X&limit=20`, which returns ONE
aggregate row keyed by packet.id. Every child therefore carried
`data-value=<packet.id>`.

## Fix
Switch the expand fetch to `/api/packets/<hash>`, which includes the
full `observations[]` array. Build `_children` as `{...pkt, ...obs}` so
each child row gets a unique observation id and observation-level fields
(observer, path, timestamp, snr/rssi) override the aggregate.

## Verified live on staging
Tested on multiple packets:
- Click group-header → side pane shows observation 1 of N (first
observer)
- Click child row → pane updates to show THAT observer's details:
observer name, path, timestamp, obs counter (K of N), URL
`?obs=<observation_id>`

## Tests
592 frontend tests pass (no new ones — this is a wiring fix, live
E2E-verified instead).

Closes #866

---------

Co-authored-by: Kpa-clawbot <agent@corescope.local>
Co-authored-by: you <you@example.com>
2026-04-21 13:36:45 -07:00
Kpa-clawbot 1d449eabc7 fix(#872): replace strikethrough with warning badge on unreliable hops (#875)
## Problem

The `hop-unreliable` CSS class applied `text-decoration: line-through`
and `opacity: 0.5`, making hop names look "dead" to operators. This
caused confusion — the repeater itself is fine, only the name→hash
assignment is uncertain.

## Fix

- **CSS**: Removed `line-through` and heavy opacity from
`.hop-unreliable`. Kept subtle `opacity: 0.85` for scanability. Added
`.hop-unreliable-btn` style for the new badge.
- **JS**: Added a `⚠️` warning badge button next to unreliable hops
(similar pattern to existing conflict badges). The badge is always
visible, keyboard-focusable, and has both `title` and `aria-label` with
an informative tooltip explaining geographic inconsistency.
- **Tests**: Added 2 tests in `test-frontend-helpers.js` asserting the
badge renders for unreliable hops and does NOT render for reliable ones,
and that no `line-through` is present.

### Before → After

| Before | After |
|--------|-------|
| ~~NodeName~~ (struck through, 50% opacity) | NodeName ⚠️ (normal text,
small warning badge with tooltip) |

## Scope

Resolver logic untouched — #873 covers threshold tuning, #874 covers
picker correctness. No candidate-dropdown UX (follow-up per issue
discussion).

Closes #872

Co-authored-by: you <you@example.com>
2026-04-21 10:54:32 -07:00
Kpa-clawbot 42ff5a291b fix(#866): full-page obs-switch — update hex + path + direction per observation (#870)
## Problem

On `/#/packets/<hash>?obs=<id>`, clicking a different observation
updated summary fields (Observer, SNR/RSSI, Timestamp) but **not** hex
payload or path details. Sister bug to #849 (fixed in #851 for the
detail dialog).

## Root Causes

| Cause | Impact |
|-------|--------|
| `selectPacket` called `renderDetail` without `selectedObservationId` |
Initial render missed observation context on some code paths |
| `ObservationResp` missing `direction`, `resolved_path`, `raw_hex` |
Frontend obs-switch lost direction and resolved_path context |
| `obsPacket` construction omitted `direction` field | Direction not
preserved when switching observations |

## Fix

- `selectPacket` explicitly passes `selectedObservationId` to
`renderDetail`
- `ObservationResp` gains `Direction`, `ResolvedPath`, `RawHex` fields
- `mapSliceToObservations` copies the three new fields
- `obsPacket` spreads include `direction` from the observation

## Tests

7 new tests in `test-frontend-helpers.js`:
- Observation switch updates `effectivePkt` path
- `raw_hex` preserved from packet when obs has none
- `raw_hex` from obs overrides when API provides it
- `direction` carried through observation spread
- `resolved_path` carried through observation spread
- `getPathLenOffset` cross-check for transport routes
- URL hash `?obs=` round-trip encoding

All 584 frontend + 62 filter + 29 aging tests pass. Go server tests
pass.

Fixes #866

Co-authored-by: you <you@example.com>
2026-04-21 10:40:52 -07:00
Kpa-clawbot 99029e41aa ci(#768): publish multi-arch (amd64+arm64) Docker image (#869)
## Problem

`docker pull` on ARM devices fails because the published image is
amd64-only.

## Fix

Enable multi-arch Docker builds via `docker buildx`. **Builder stage
uses native Go cross-compilation; only the runtime-stage `RUN` steps use
QEMU emulation.**

### Changes

| File | Change |
|------|--------|
| `Dockerfile` | Pin builder stage to `--platform=$BUILDPLATFORM`
(always native), accept `ARG TARGETOS`/`ARG TARGETARCH` from buildx, set
`GOOS=$TARGETOS GOARCH=$TARGETARCH CGO_ENABLED=0` on every `go build` |
| `.github/workflows/deploy.yml` | Add `docker/setup-buildx-action@v3` +
`docker/setup-qemu-action@v3` (latter needed only for runtime-stage
RUNs), set `platforms: linux/amd64,linux/arm64` |

### Build architecture

- **Builder stage** (`FROM --platform=$BUILDPLATFORM
golang:1.22-alpine`) — runs natively on amd64. Go toolchain
cross-compiles the binaries to `$TARGETARCH` via `GOOS/GOARCH`. No
emulation, ~10× faster than emulated builds. Works because
`modernc.org/sqlite` is pure Go (no CGO).
- **Runtime stage** (`FROM alpine:3.20`) — buildx pulls the per-arch
base. RUN steps (`apk add`, `mkdir/chown`, `chmod`) execute inside the
target-arch image, so QEMU is required to interpret arm64 binaries on
the amd64 host. Only a handful of short shell commands run under
emulation, so the QEMU cost is small.

### Verify

After merge, on an ARM device:
```bash
docker pull ghcr.io/kpa-clawbot/corescope:edge
docker inspect ghcr.io/kpa-clawbot/corescope:edge --format '{{.Architecture}}'
# → arm64
```

> First arm64 image appears on the next push to master after this
merges.

Closes #768

---------

Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <agent@corescope.local>
2026-04-21 10:32:02 -07:00
Kpa-clawbot c99aa1dadf fix(#855, #856, #857) + feat(#862): /nodes detail panel + search improvements (#868)
## Summary

Four related `/nodes` page fixes batched to avoid merge conflicts (all
touch `public/nodes.js`).

---

### #855 — "Show all neighbors" link doesn't expand

**Problem:** The "View all N neighbors →" link in the side panel
navigated to the full detail page instead of expanding the truncated
list inline.

**Fix:** Replaced navigation link with an inline "Show all N neighbors
▼" button that re-renders the neighbor table without the limit.

**Acceptance:** Click the button → all neighbors appear in the same
panel without page navigation.

Closes #855

---

### #856 — "Details" button is a no-op

**Problem:** The "🔍 Details" link in the side panel was an `<a>` tag
whose `href` matched the current hash (set by `replaceState`), making
clicks a same-hash no-op.

**Fix:** Changed from `<a>` link to a `<button>` with a direct click
handler that sets `location.hash`, ensuring the router always fires.

**Acceptance:** Click "🔍 Details" → navigates to full-screen node detail
view.

Closes #856

---

### #857 — Recent Packets shows bullets but no content

**Problem:** The "Recent Packets (N)" section could render entries with
missing `hash` or `timestamp`, producing colored dots with no meaningful
content beside them.

**Fix:** Added `.filter(a => a.hash && a.timestamp)` before rendering,
and updated the count header to reflect filtered entries only.

**Acceptance:** Recent Packets section only shows entries with valid
data; count matches visible items.

Closes #857

---

### #862 — Pubkey prefix search on /#/nodes

**Problem:** Search box only matched node names. Operators couldn't
search by pubkey prefix.

**Fix:** Extended search to detect hex-only queries (`/^[0-9a-f]+$/i`)
and match them against pubkey prefix (`startsWith`). Non-hex queries
continue matching name as before. Both are composable in the same input.

**Acceptance:**
- Typing `3f` filters to nodes whose pubkey starts with `3f`
- Typing `foo` still filters by name
- Search placeholder updated to indicate pubkey support

5 new unit tests added for the search matching logic.

Closes #862

---------

Co-authored-by: you <you@example.com>
2026-04-21 10:24:27 -07:00
Kpa-clawbot 20843979a7 fix(#861): restore sticky table headers on mobile packets page (#867)
## What

Remove a single line in `makeColumnsResizable()` that set
`th.style.position = 'relative'` on every `<th>` except the last,
overriding the CSS `position: sticky` rule from `.data-table th`.

## Why

The column-resize feature added inline `position: relative` to each
header (except the last) to serve as a containing block for the
absolute-positioned resize handles. This inadvertently broke `position:
sticky` on all headers except "Details" (the last column) — visible on
mobile when scrolling the packets table.

`position: sticky` is itself a positioned value and serves as a
containing block for absolute children, so the resize handles work
identically without the override.

## Test

- Open `/#/packets` on mobile (or narrow viewport)
- Scroll down — ALL column headers now remain sticky at the top
- Column resize handles still function correctly on desktop

Fixes #861

Co-authored-by: you <you@example.com>
2026-04-21 09:53:31 -07:00
Kpa-clawbot ea78581eea fix(#858): packets/hour chart — bars rendering + x-axis label decimation (#865)
Two bugs in the Overview tab Packets/Hour chart:

1. **Bars not rendering**: `barW` went negative when `data.length` was
large (e.g. 720 hours for 30-day range), producing zero-width invisible
bars. Fix: `Math.max(1, ...)` floor on bar width.

2. **X-axis labels overlapping**: Every single hour label was emitted
(`02h03h04h...`). Fix: decimate labels based on time range — every 6h
for ≤24h, every 12h for ≤72h, every 24h beyond. Shows `MM-DD` on
midnight boundaries for multi-day ranges.

**Scope**: Only touches the Overview tab `Packets / Hour` section and
the shared `barChart` floor (one-line change). No modifications to
Topology, Channels, Distance, or other tabs.

Fixes #858

Co-authored-by: you <you@example.com>
2026-04-21 09:53:01 -07:00
Kpa-clawbot b5372d6f73 fix(#859): remove opacity gradient from Per-Observer Reachability rows (#863)
Fixes #859

## What

The "Per-Observer Reachability" and "Best Path to Each Node" sections in
the Topology tab had inline `opacity` styles on each `.reach-ring` row
that decreased with hop count (`1 - hops * 0.06`, floored at 0.3). This
made text progressively darker/unreadable toward the bottom.

## Fix

Removed the inline `opacity:${opacity}` style from both
`renderPerObserverReach()` and `renderBestPath()`. The rows now render
at full opacity with text colors governed by CSS variables as intended.

## Changed
- `public/analytics.js`: removed opacity computation and inline style in
two functions (4 lines removed, 2 added)

## Scope
Only touches Per-Observer Reachability and Best Path rendering. No
changes to Overview, Channels, or shared helpers.

Co-authored-by: you <you@example.com>
2026-04-21 09:52:18 -07:00
Kpa-clawbot 5afed0951b fix(#860): cap channel timeline chart to top 8 by volume (#864)
## What & Why

The "Messages / Hour by Channel" chart on `/#/analytics` Channels tab
rendered all channels in both the SVG and legend, causing legend
overflow when 20+ channels are present.

## Fix

- Sort channels by total message volume (descending)
- Render only the top 8 in the chart and legend
- Show "+N more" in the legend when channels are truncated
- `maxCount` for Y-axis scaling is computed from visible channels only,
so the chart uses its full vertical range

Single-file change: `public/analytics.js` — only
`renderChannelTimeline()` modified. No shared helpers touched.

Fixes #860

Co-authored-by: you <you@example.com>
2026-04-21 09:51:52 -07:00
Kpa-clawbot 3630a32310 fix(#852): transport-route path_len offset + var(--muted) → var(--text-muted) (#853)
## Problem

Two pre-existing bugs found during expert review of #851:

### 1. `hashSize` derivation ignores transport route types

`public/packets.js` hardcoded path-length byte at offset 1:
```js
const rawPathByte = pkt.raw_hex ? parseInt(pkt.raw_hex.slice(2, 4), 16) : NaN;
```

For transport routes (`route_type` 0 DIRECT or 3 TRANSPORT_ROUTE_FLOOD),
bytes 1–4 are `next_hop` + `last_hop` and path-length is at offset 5.
Same bug #846 fixed inside the byte-breakdown function.

### 2. `var(--muted)` CSS variable is undefined

Used in 6 places in `public/packets.js`. No `--muted` variable is
defined anywhere in `public/*.css` — only `--text-muted` exists. Text
styled with `var(--muted)` silently falls through to inherited color,
making badges/hints invisible.

## Fix

### Fix 1: transport-route path_len offset
```js
const plOff = (pkt.route_type === 0 || pkt.route_type === 3) ? 5 : 1;
const rawPathByte = pkt.raw_hex ? parseInt(pkt.raw_hex.slice(plOff * 2, plOff * 2 + 2), 16) : NaN;
```

### Fix 2: `var(--muted)` → `var(--text-muted)`
All 6 occurrences replaced.

## Tests (5 new, 572 total)

- `hashSize` extraction for flood route (route_type=1, offset 1)
- `hashSize` extraction for direct transport route (route_type=0, offset
5)
- `hashSize` extraction for transport route flood (route_type=3, offset
5)
- `hashSize` returns null for missing raw_hex
- Regression guard: no `var(--muted)` in any `public/` JS/CSS file

## Changes

- `public/packets.js`: 7 lines changed (1 offset fix + 6 CSS var fixes)
- `test-frontend-helpers.js`: 46 lines added (5 tests)

Closes #852

---------

Co-authored-by: you <you@example.com>
2026-04-21 09:27:16 -07:00
Kpa-clawbot ff05db7367 ci: fix staging smoke test port — read STAGING_GO_HTTP_PORT, not hardcoded 82 (#854)
## Problem
The "Deploy Staging" job's Smoke Test always fails with `Staging
/api/stats did not return engine field`.

Root cause: the step hardcodes `http://localhost:82/api/stats`, but
`docker-compose.staging.yml:21` publishes the container on
`${STAGING_GO_HTTP_PORT:-80}:80`. Default is port 80, not 82. curl gets
ECONNREFUSED, `-sf` swallows the error, `grep -q engine` sees empty
input → failure.

Verified on staging VM: `ss -lntp` shows only `:80` listening; `docker
ps` confirms `0.0.0.0:80->80/tcp`. A `curl http://localhost:82` returns
connection-refused.

## Fix
Read `STAGING_GO_HTTP_PORT` (same default as compose) so the smoke test
tracks the port the container was actually launched on. Failure message
now includes the resolved port to make future port mismatches
self-diagnosing.

## Tested
Logic only — the curl + grep pattern is unchanged. If any CI env
override sets `STAGING_GO_HTTP_PORT`, the smoke test now follows it.

Co-authored-by: Kpa-clawbot <agent@corescope.local>
2026-04-21 16:23:50 +00:00
Kpa-clawbot 441409203e feat(#845): bimodal_clock severity — surface flaky-RTC nodes instead of hiding as 'No Clock' (#850)
## Problem

Nodes with flaky RTC (firmware emitting interleaved good and nonsense
timestamps) were classified as `no_clock` because the broken samples
poisoned the recent median. Operators lost visibility into these nodes —
they showed "No Clock" even though ~60% of their adverts had valid
timestamps.

Observed on staging: a node with 31K samples where recent adverts
interleave good skew (-6.8s, -13.6s) with firmware nonsense (-56M, -60M
seconds). Under the old logic, median of the mixed window → `no_clock`.

## Solution

New `bimodal_clock` severity tier that surfaces flaky-RTC nodes with
their real (good-sample) skew value.

### Classification order (first match wins)

| Severity | Good Fraction | Description |
|----------|--------------|-------------|
| `no_clock` | < 10% | Essentially no real clock |
| `bimodal_clock` | 10–80% (and bad > 0) | Mixed good/bad — flaky RTC |
| `ok`/`warn`/`critical`/`absurd` | ≥ 80% | Normal classification |

"Good" = `|skew| <= 1 hour`; "bad" = likely uninitialized RTC nonsense.

When `bimodal_clock`, `recentMedianSkewSec` is computed from **good
samples only**, so the dashboard shows the real working-clock value
(e.g. -7s) instead of the broken median.

### Backend changes
- New constant `BimodalSkewThresholdSec = 3600`
- New severity `bimodal_clock` in classification logic
- New API fields: `goodFraction`, `recentBadSampleCount`,
`recentSampleCount`

### Frontend changes
- Amber `Bimodal` badge with tooltip showing bad-sample percentage
- Bimodal nodes render skew value like ok/warn/severe (not the "No
Clock" path)
- Warning line below sparkline: "⚠️ X of last Y adverts had nonsense
timestamps (likely RTC reset)"

### Tests
- 3 new Go unit tests: bimodal (60% good → bimodal_clock), all-bad (→
no_clock), 90%-good (→ ok)
- 1 new frontend test: bimodal badge rendering with tooltip
- Existing `TestReporterScenario_789` passes unchanged

Builds on #789 (recent-window severity).

Closes #845

---------

Co-authored-by: you <you@example.com>
2026-04-21 09:11:14 -07:00
Kpa-clawbot a371d35bfd feat(#847): dedupe Top Longest Hops by pair + add obs count and SNR cues (#848)
## Problem

The "Top 20 Longest Hops" RF analytics card shows the same repeater pair
filling most slots because the query sorts raw hop records by distance
with no pair deduplication. A single long link observed 12+ times
dominates the leaderboard.

## Fix

Dedupe by unordered `(pk1, pk2)` pair. Per pair, keep the max-distance
record and compute reliability metrics:

| Column | Description |
|--------|-------------|
| **Obs** | Total observations of this link |
| **Best SNR** | Maximum SNR seen (dB) |
| **Median SNR** | Median SNR across all observations (dB) |

Tooltip on each row shows the timestamp of the best observation.

### Before
| # | From | To | Distance | Type | SNR | Packet |
|---|------|----|----------|------|-----|--------|
| 1 | NodeX | NodeY | 200 mi | R↔R | 5 dB | abc… |
| 2 | NodeX | NodeY | 199 mi | R↔R | 6 dB | def… |
| 3 | NodeX | NodeY | 198 mi | R↔R | 4 dB | ghi… |

### After
| # | From | To | Distance | Type | Obs | Best SNR | Median SNR | Packet
|

|---|------|----|----------|------|-----|----------|------------|--------|
| 1 | NodeX | NodeY | 200 mi | R↔R | 12 | 8.0 dB | 5.2 dB | abc… |
| 2 | NodeA | NodeB | 150 mi | C↔R | 3 | 6.5 dB | 6.5 dB | jkl… |

## Changes

- **`cmd/server/store.go`**: Group `filteredHops` by unordered pair key,
accumulate obs count / best SNR / median SNR per group, sort by max
distance, take top 20
- **`cmd/server/types.go`**: Update `DistanceHop` struct — replace `SNR`
with `BestSnr`, `MedianSnr`, add `ObsCount`
- **`public/analytics.js`**: Replace single SNR column with Obs, Best
SNR, Median SNR; add row tooltip with best observation timestamp
- **`cmd/server/store_tophops_test.go`**: 3 unit tests — basic dedupe,
reverse-pair merge, nil SNR edge case

## Test Coverage

- `TestDedupeTopHopsByPair`: 5 records on pair (A,B) + 1 on (C,D) → 2
results, correct obsCount/dist/bestSnr/medianSnr
- `TestDedupeTopHopsReversePairMerges`: (B,A) and (A,B) merge into one
entry
- `TestDedupeTopHopsNilSNR`: all-nil SNR records → bestSnr and medianSnr
both nil
- Existing `TestAnalyticsRFEndpoint` and `TestAnalyticsRFWithRegion`
still pass

Closes #847

---------

Co-authored-by: you <you@example.com>
2026-04-21 09:09:39 -07:00
Kpa-clawbot 7c01a97178 fix(#849): Packet Detail dialog — show exact clicked observation, not cross-observer aggregate (#851)
## Problem

The Packet Detail dialog summary (Observer, Path, Hops, SNR/RSSI,
Timestamp) used the **aggregated cross-observer view** (`_parsedPath` /
`getParsedPath(pkt)`), which contradicted the byte breakdown after #844.
A packet observed with 2 hops by one observer would show "Path: 7 hops"
in the summary because it merged all observers' paths.

## Fix

The dialog is now **per-observation**:

- `renderDetail` resolves a `currentObservation` from
`selectedObservationId` (set when clicking an observation child row) or
defaults to `observations[0]`
- All summary fields read from the current observation: Observer,
SNR/RSSI, Timestamp, Path, Direction
- Hop count badge comes from `path_len & 0x3F` of the observation's
`raw_hex` (firmware truth, same source as byte breakdown). Cross-checked
against `path_json` length — logs a console warning on mismatch
- **Observations table** rendered inside the detail panel when multiple
observations exist. Clicking a row updates `currentObservation` and
re-renders the summary in-place (no dialog close/reopen)
- `.observation-current` CSS class highlights the selected observation
row

### Cross-observer aggregate (Option B)

A read-only "Cross-observer aggregate" section below the observations
table shows the longest observed path across all observers. This is
**not** the default view — it's always visible as secondary context.

## Tests

8 new tests in `test-frontend-helpers.js`:
- Hop count extraction from raw_hex (normal, direct, transport route
types)
- Inconsistency detection between path_json and raw_hex
- Per-observation field override of aggregated packet fields
- First observation used when no specific observation selected
- Observation row click selects that observation
- Null/missing raw_hex handling

All 572 tests pass (564 frontend + 62 filter + 29 aging).

## Acceptance

- Summary shows per-observation path/hops/SNR/RSSI/timestamp
- Switching observations in the detail updates everything
- Cross-observer aggregate available as secondary section
- Byte breakdown untouched (owned by #846)

## Related

- Closes #849
- Related: #844 (#846) — byte breakdown fix (separate PR, different code
region)

---------

Co-authored-by: you <you@example.com>
2026-04-21 09:08:58 -07:00
Kpa-clawbot f1eea9ee3c fix(#844): Packet Byte Breakdown — derive hop count from path_len, not aggregated _parsedPath (#846)
## Problem

The Packet Detail dialog's "Packet Byte Breakdown" section was using the
aggregated `_parsedPath` (longest path observed across all observers) to
render hop entries, instead of deriving the hop count from the
`path_len` byte in `raw_hex`. This caused:

- Wrong hop count (e.g., "Path (7 hops)" when `raw_hex` only contains 2)
- Hop values from the aggregated path displayed at incorrect byte
offsets
- Subsequent fields (pubkey, timestamp, signature) rendered at wrong
offsets because `off` was advanced by the wrong amount

## Fix

In `buildFieldTable()` (packets.js), the Path section now:

1. Derives `hashCountVal` from `path_len & 0x3F` (firmware truth per
`Packet.h:79-83`)
2. Derives `hashSize` from `(path_len >> 6) + 1`
3. Reads each hop's hex value directly from `raw_hex` at the correct
byte offset
4. Advances `off` by `hashSize * hashCountVal`
5. Skips the Path section entirely when `hashCountVal === 0` (direct
advert)

The "Path" summary section above the breakdown (which uses the
aggregated path for route visualization) is unchanged — only the byte
breakdown is fixed.

## Tests

3 new tests in `test-frontend-helpers.js`:
- Verifies 2 hops rendered (not 7) when `path_len=0x42` despite 7-hop
aggregated path
- Verifies pubkey offset is 6 (not 16) after a 2-hop path
- Verifies direct advert (`hashCount=0`) skips Path section

Also fixed pre-existing `HopDisplay is not defined` failures in the
`#765` transport offset test sandbox (added mock).

All 559 tests pass.

Closes #844

---------

Co-authored-by: you <you@example.com>
2026-04-21 08:26:12 -07:00
Kpa-clawbot f30e6bef28 qa(plan): reconcile §8.2/§5.3/§6.2 + add §8.7 (Recent Packets readability) (#838)
Doc-only reconciliation of v3.6.0-rc plan with what actually shipped.

## Changes
- **§8.2** — desktop deep link now opens full-screen view
(post-#823/#824), not split panel as the plan still asserted.
- **§5.3** — pin that severity now derives from `recentMedianSkewSec`
(#789), not the all-time `medianSkewSec` — a re-tester needs to know
which field drives the badge.
- **§6.2** — pin the existing observer-graph element location
(`public/analytics.js:2048-2051`).
- **New §8.7** — side-panel "Recent Packets" entries must navigate to a
valid packet detail (DB-fallback per #827) AND text must be readable in
the current theme (explicit color per #829). Both bugs surfaced this
session.

No CI gates.

Co-authored-by: Kpa-clawbot <agent@corescope.local>
2026-04-21 08:01:17 -07:00
Kpa-clawbot 20f456da58 fix(#840): map popup 'Show Neighbors' link does nothing on iOS Safari (#841)
Closes #840

## What
Switch the map-popup "Show Neighbors" link from `<a href="#">` to `<a
href="javascript:void(0)" role="button">` so iOS Safari doesn't navigate
when the document-level click delegation fails to fire.

## Why
On iOS Safari, when a user taps the link inside a Leaflet popup:
- The document-level click delegation at `public/map.js:927` calls
`e.preventDefault()` and triggers `selectReferenceNode`.
- BUT inside a Leaflet popup, `L.DomEvent.disableClickPropagation()` is
internally applied to popup content — on iOS Safari the click sometimes
doesn't bubble to `document`.
- When that happens, the browser's default `<a href="#">` action runs:
  - hash becomes empty (`#`)
- `navigate()` in `app.js:458` sees empty hash → defaults to `'packets'`
- map page is destroyed mid-tap → user perceives "nothing happened" (or
a brief flash if they back-button)

`href="javascript:void(0)"` removes the navigation fall-through
entirely. The `role="button"` keeps a11y semantics, `cursor:pointer`
keeps the visual cue.

## Tested
- Headless Chromium desktop + iPhone 13 emulation: tap fires
`/api/nodes/{pk}/neighbors?min_count=3`, marker count drops from 441 →
44, `#mcNeighbors` checkbox toggles on, URL stays on `/#/map`. Same as
before.
- Frontend helpers: 556/0
- Real iOS Safari fix verification needs a physical-device test
post-deploy

## Out of scope (follow-up)
- Same `<a href="#">` pattern exists for the topright "Close route"
control at `public/map.js:389` — uses `L.DomEvent.preventDefault` so
should work, but worth auditing if the symptom recurs.

Co-authored-by: Kpa-clawbot <agent@corescope.local>
2026-04-21 07:58:55 -07:00
Kpa-clawbot e31e14cae9 qa(plan): apply v3.6.0-rc QA findings (#832/#833/#836) (#837)
Apply v3.6.0-rc QA learnings to the plan.

## Changes
- **§1.1** — 1 GB cap is unrealistic on real DBs without `GOMEMLIMIT` +
bounded cold-load. Raised target to 3 GB and pointed to follow-up
**#836**. (Investigation showed cold-load transient blows past any
sub-2GB cap regardless of `maxMemoryMB` setting because
`runtime.MemStats.NextGC` ignores cgroup ceilings.)
- **§1.4** — `trackedBytes`/`trackedMB` is in-store packet bytes only
and under-reports RSS by ~3–5× (no indexes, caches, runtime overhead,
cgo). Switched the assertion to use `processRSSMB` exposed by **#832**
(PR **#835**).
- **§11.1** — noted the Playwright deep-link E2E assertion was updated
by **#833** (PR **#834**) to match the post-#823 full-screen behavior.

## Why
Three real findings from the QA ops sweep ([§1.4 fail
comment](https://github.com/Kpa-clawbot/CoreScope/issues/809#issuecomment-4286113141)).
Updating the plan so the next run doesn't replay the same
false-fail/false-pass conditions.

Co-authored-by: Kpa-clawbot <agent@corescope.local>
2026-04-20 23:29:18 -07:00
Kpa-clawbot bb0f816a6b fix(channels): only show lock for confirmed-encrypted #channel deep links (#825) (#826)
Closes #825

## Root cause
PR #815 added a `#`-prefix branch in `selectChannel` that
unconditionally rendered the lock affordance whenever the channel object
wasn't in the loaded `channels` list. With the encrypted toggle off,
unencrypted channels like `#test` are also absent from the list, so the
new branch wrongly locked them instead of falling through to the REST
fetch.

## Fix
When no stored key matches, refetch `/channels?includeEncrypted=true`
and check `ch.encrypted` before locking. Only render the lock when we
positively know the channel is encrypted; otherwise fall through to the
existing REST messages fetch.

This regresses #815's behavior **only for the unencrypted case** (which
is the bug). The encrypted-no-key (#811) and encrypted-with-stored-key
(#815) paths are preserved.

## Tests
3 new regression tests in `test-frontend-helpers.js`:
- `#test` (unencrypted) deep link → REST fetched, no lock
- `#private` (encrypted, no key) deep link → lock, no REST (#811
preserved)
- `#private` (encrypted, with stored key) deep link → decrypt path (#815
preserved)

`node test-frontend-helpers.js` → 556 passed, 0 failed.

## Perf
One extra REST call per cold deep link to a `#`-named channel that's not
in the toggle-off list — same endpoint already cached via
`CLIENT_TTL.channels`, so subsequent navigations are free.

---------

Co-authored-by: you <you@example.com>
2026-04-20 23:11:20 -07:00
Kpa-clawbot 3f26dc7190 obs: surface real RSS alongside tracked store bytes in /api/stats (#832) (#835)
Closes #832.

## Root cause confirmed
\`trackedMB\` (\`s.trackedBytes\` in \`store.go\`) only sums per-packet
struct + payload sizes recorded at insertion. It excludes the index maps
(\`byHash\`, \`byTxID\`, \`byNode\`, \`byObserver\`, \`byPathHop\`,
\`byPayloadType\`, hash-prefix maps, name lookups), the analytics LRUs
(rfCache/topoCache/hashCache/distCache/subpathCache/chanCache/collisionCache),
WS broadcast queues, and Go runtime overhead. It's \"useful packet
bytes,\" not RSS — typically 3–5× off on staging.

## Fix (Option C from the issue)
Expose four memory fields on \`/api/stats\` from a single cached
snapshot:

| Field | Source | Semantics |
|---|---|---|
| \`storeDataMB\` | \`s.trackedBytes\` | in-store packet bytes; eviction
watermark input |
| \`goHeapInuseMB\` | \`runtime.MemStats.HeapInuse\` | live Go heap |
| \`goSysMB\` | \`runtime.MemStats.Sys\` | total Go-managed memory |
| \`processRSSMB\` | \`/proc/self/status VmRSS\` (Linux), falls back to
\`goSysMB\` | what the kernel sees |

\`trackedMB\` is retained as a deprecated alias for \`storeDataMB\` so
existing dashboards/QA scripts keep working.

Field invariants are documented on \`MemorySnapshot\`: \`processRSSMB ≥
goSysMB ≥ goHeapInuseMB ≥ storeDataMB\` (typical).

## Performance
Single \`getMemorySnapshot\` call cached for 1s —
\`runtime.ReadMemStats\` (stop-the-world) and the \`/proc/self/status\`
read are amortized across burst polling. \`/proc\` read is bounded to 8
KiB, parsed with \`strconv\` only — no shell-out, no untrusted input.

\`cgoBytesMB\` is omitted: the build uses pure-Go
\`modernc.org/sqlite\`, so there is no cgo allocator to measure.
Documented in code comment.

## Tests
\`cmd/server/stats_memory_test.go\` asserts presence, types, sign, and
ordering invariants. Avoids the flaky \"matches RSS to ±X%\" pattern.

\`\`\`
$ go test ./... -count=1 -timeout 180s
ok  	github.com/corescope/server	19.410s
\`\`\`

## QA plan
§1.4 now compares \`processRSSMB\` against procfs RSS (the right
invariant); threshold stays at 0.20.

---------

Co-authored-by: MeshCore Agent <meshcore-agent@openclaw.local>
2026-04-20 23:10:33 -07:00
Kpa-clawbot 886aabf0ae fix(#827): /api/packets/{hash} falls back to DB when in-memory store misses (#831)
Closes #827.

## Problem
`/api/packets/{hash}` only consulted the in-memory `PacketStore`. When a
packet aged out of memory, the handler 404'd — even though SQLite still
had it and `/api/nodes/{pubkey}` `recentAdverts` (which reads from the
DB) was actively surfacing the hash. Net effect: the **Analyze →** link
on older adverts in the node detail page led to a dead "Not found".

Two-store inconsistency: DB has the packet, in-memory doesn't, node
detail surfaces it from DB → packet detail can't serve it.

## Fix
In `handlePacketDetail`:
- After in-memory miss, fall back to `db.GetPacketByHash` (already
existed) for hash lookups, and `db.GetTransmissionByID` for numeric IDs.
- Track when the result came from the DB; if so and the store has no
observations, populate from DB via a new `db.GetObservationsForHash` so
the response shows real observations instead of the misleading
`observation_count = 1` fallback.

## Tests
- `TestPacketDetailFallsBackToDBWhenStoreMisses` — insert a packet
directly into the DB after `store.Load()`, confirm store doesn't have
it, assert 200 + populated observations.
- `TestPacketDetail404WhenAbsentFromBoth` — neither store nor DB → 404
(no false positives).
- `TestPacketDetailPrefersStoreOverDB` — both have it; store result wins
(no double-fetch).
- `TestHandlePacketDetailNoStore` updated: it previously asserted the
old buggy 404 behavior; now asserts the correct DB-fallback 200.

All `go test ./... -run "PacketDetail|Packet|GetPacket"` and the full
`cmd/server` suite pass.

## Out of scope
The `/api/packets?hash=` filter is the live in-memory list endpoint and
intentionally store-only for performance. Not touched here — happy to
file a follow-up if you'd rather harmonise.

## Repro context
Verified against prod with a recently-adverting repeater whose recent
advert hash lives in `recentAdverts` (DB) but had been evicted from the
in-memory store; pre-fix 404, post-fix 200 with full observations.

Co-authored-by: you <you@example.com>
2026-04-20 22:50:01 -07:00
Kpa-clawbot a0fddb50aa fix(#789): severity from recent samples; Theil-Sen drift with outlier rejection (#828)
Closes #789.

## The two bugs

1. **Severity from stale median.** `classifySkew(absMedian)` used the
all-time `MedianSkewSec` over every advert ever recorded for the node. A
repeater that was off for hours and then GPS-corrected stayed pinned to
`absurd` because hundreds of historical bad samples poisoned the median.
Reporter's case: `medianSkewSec: -59,063,561.8` while `lastSkewSec:
-0.8` — current health was perfect, dashboard said catastrophic.

2. **Drift from a single correction jump.** Drift used OLS over every
`(ts, skew)` pair, with no outlier rejection. A single GPS-correction
event (skew jumps millions of seconds in ~30s) dominated the regression
and produced `+1,793,549.9 s/day` — physically nonsense; the existing
`maxReasonableDriftPerDay` cap then zeroed it (better than absurd, but
still useless).

## The two fixes

1. **Recent-window severity.** New field `recentMedianSkewSec` = median
over the last `N=5` samples or last `1h`, whichever is narrower (more
current view). Severity now derives from `abs(recentMedianSkewSec)`.
`MeanSkewSec`, `MedianSkewSec`, `LastSkewSec` are preserved unchanged so
the frontend, fleet view, and any external consumers continue to work.

2. **Theil-Sen drift with outlier filter.** Drift now uses the Theil-Sen
estimator (median of all pairwise slopes — textbook robust regression,
~29% breakdown point) on a series pre-filtered to drop samples whose
skew jumps more than `maxPlausibleSkewJumpSec = 60s` from the previous
accepted point. Real µC drift is fractions of a second per advert; clock
corrections fall well outside. Capped at `theilSenMaxPoints = 200`
(most-recent) so O(n²) stays bounded for chatty nodes.

## What stays the same

- Epoch-0 / out-of-range advert filter (PR #769).
- `minDriftSamples = 5` floor.
- `maxReasonableDriftPerDay = 86400` hard backstop.
- API shape: only additions (`recentMedianSkewSec`); no fields removed
or renamed.

## Tests

All in `cmd/server/clock_skew_test.go`:

- `TestSeverityUsesRecentNotMedian` — 100 bad samples (-60s) + 5 good
(-1s) → severity = `ok`, historical median still huge.
- `TestDriftRejectsCorrectionJump` — 30 min of clean linear drift + one
1000s jump → drift small (~12 s/day).
- `TestTheilSenMatchesOLSWhenClean` — clean linear data, Theil-Sen
within ~1% of OLS.
- `TestReporterScenario_789` — exact reproducer: 1662 samples, 1657 @
-683 days then 5 @ -1s → severity `ok`, `recentMedianSkewSec ≈ 0`, drift
bounded; legacy `medianSkewSec` preserved as historical context.

`go test ./... -count=1` (cmd/server) and `node
test-frontend-helpers.js` both pass.

---------

Co-authored-by: clawbot <bot@corescope.local>
Co-authored-by: you <you@example.com>
2026-04-20 22:47:10 -07:00
Kpa-clawbot bb09123f34 test(#833): update deep-link Playwright assertion for full-screen desktop view (#834)
Closes #833

## What
Update Playwright E2E assertion for desktop deep link to
`/#/nodes/{pubkey}`. Now expects `.node-fullscreen` to be present
(matches the spec set by PR #824 / issue #823).

## Why
The previous assertion encoded the old pre-#823 behavior — "split panel
on desktop deep link." PR #824 intentionally removed the
`window.innerWidth <= 640` gate so desktop deep links open the
full-screen view (matching the Details link path that #779/#785/#824
ultimately made work). The test failed on every PR that rebased onto
master, blocking `Deploy Staging`.

## Verified
- 1-test diff, no other behavior change
- Mobile-viewport assertions elsewhere already exercise the same
`.node-fullscreen` selector

Co-authored-by: Kpa-clawbot <agent@corescope.local>
2026-04-21 05:37:05 +00:00
Kpa-clawbot 31a0a944f9 fix(#829): node-detail side panel Recent Packets text invisible (#830)
Closes #829

## What
Add explicit `color: var(--text)` to `.advert-info` (and `var(--accent)`
to its links) so the side-panel "Recent Packets" entries stay readable
in all themes.

## Why
`.advert-info` had only `font-size` + `line-height` rules — text
inherited from ancestors. In default light/dark themes the inherited
color happens to differ enough from `--card-bg`. Under custom themes
where they collide, text becomes invisible — only the colored
`.advert-dot` shows. Operator screenshot confirmed the symptom.

Same class of bug as the existing fix at `style.css:660` ("Bug 7 fix:
neighbor table text inherits accent color — force readable text") which
forced `color: var(--text)` on `.node-detail-section .data-table td`.
The advert timeline doesn't use a data-table, so it fell through.

## Verified
- DOM contains correct text — only the rendered color was wrong
- `getComputedStyle(.advert-info).color` previously matched `--card-bg`
under affected themes
- After fix: `.advert-info` resolves to `var(--text)` regardless of
inherited chain
- Frontend helpers: 553/0
- Full-screen `node-full-card` view (separate `.node-activity-item`
markup) unaffected

Co-authored-by: Kpa-clawbot <agent@corescope.local>
2026-04-21 05:34:08 +00:00
efiten cad1f11073 fix: bypass IATA filter for status messages, fill SNR on duplicate obs (#694) (#802)
## Problems

Two independent ingestor bugs identified in #694:

### 1. IATA filter drops status messages from out-of-region observers

The IATA filter ran at the top of `handleMessage()` before any
message-type discrimination. Status messages carrying observer metadata
(`noise_floor`, battery, airtime) from observers outside the configured
IATA regions were silently discarded before `UpsertObserver()` and
`InsertMetrics()` ran.

**Impact:** Observers running `meshcoretomqtt/1.0.8.0` in BFL and LAX —
the only client versions that include `noise_floor` in status messages —
had their health data dropped entirely on prod instances filtering to
SJC.

**Fix:** Moved the IATA filter to the packet path only (after the
`parts[3] == "status"` branch). Status messages now always populate
observer health data regardless of configured region filter.

### 2. `INSERT OR IGNORE` discards SNR/RSSI on late arrival

When the same `(transmission_id, observer_idx, path_json)` observation
arrived twice — first without RF fields, then with — `INSERT OR IGNORE`
silently discarded the SNR/RSSI from the second arrival.

**Fix:** Changed to `ON CONFLICT(...) DO UPDATE SET snr =
COALESCE(excluded.snr, snr), rssi = ..., score = ...`. A later arrival
with SNR fills in a `NULL`; a later arrival without SNR does not
overwrite an existing value.

## Tests

- `TestIATAFilterDoesNotDropStatusMessages` — verifies BFL status
message is processed when IATA filter includes only SJC, and that BFL
packet is still filtered
- `TestInsertObservationSNRFillIn` — verifies SNR fills in on second
arrival, and is not overwritten by a subsequent null arrival

## Related

Partially addresses #694 (upstream client issue of missing SNR in packet
messages is out of scope)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-20 22:16:01 -07:00
efiten 7f024b7aa7 fix(#673): replace raw JSON text search with byNode index for node packet queries (#803)
## Summary

Fixes #673

- GRP_TXT packets whose message text contains a node's pubkey were
incorrectly counted as packets for that node, inflating packet counts
and type breakdowns
- Two code paths in `store.go` used `strings.Contains` on the full
`DecodedJSON` blob — this matched pubkeys appearing anywhere in the
JSON, including inside chat message text
- `filterPackets` slow path (combined node + other filters): replaced
substring search with a hash-set membership check against
`byNode[nodePK]`
- `GetNodeAnalytics`: removed the full-packet-scan + text search branch
entirely; always uses the `byNode` index (which already covers
`pubKey`/`destPubKey`/`srcPubKey` via structured field indexing)

## Test Plan

- [x] `TestGetNodeAnalytics_ExcludesGRPTXTWithPubkeyInText` — verifies a
GRP_TXT packet with the node's pubkey in its text field is not counted
in that node's analytics
- [x] `TestFilterPackets_NodeQueryDoesNotMatchChatText` — verifies the
combined-filter slow path of `filterPackets` returns only the indexed
ADVERT, not the chat packet

Both tests were written as failing tests against the buggy code and pass
after the fix.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-20 22:15:02 -07:00
Kpa-clawbot ddd18cb12f fix(nodes): Details link opens full-screen on desktop (#823) (#824)
Closes #823

## What
Remove the `window.innerWidth <= 640` gate on the `directNode`
full-screen branch in `init()` so the 🔍 Details link works on desktop.

## Why
- #739 (`e6ace95`) gated full-screen to mobile so desktop **deep links**
would land on the split panel.
- But the same gate broke the **Details link** flow (#779/#785): the
click handler calls `init(app, pubkey)` directly. On desktop the gated
branch was skipped, the list re-rendered with `selectedKey = pubkey`,
and the side panel was already open → no visible change.
- Dropping the gate makes the directNode branch the single, unambiguous
path to full-screen for both the Details link and any deep link.

## Why the desktop split-panel UX is still preserved
Row clicks call `selectNode()`, which uses `history.replaceState` — no
`hashchange` event, no router re-init, no `directNode` set. Only the
Details link handler (which calls `init()` explicitly) and a fresh
deep-link load reach this branch.

## Repro / verify
1. Desktop, viewport > 640px, open `/#/nodes`.
2. Click a node row → split panel opens (unchanged).
3. Click 🔍 Details inside the panel → full-screen single-node view (was
broken; now works).
4. Back button / Escape → back to list view.
5. Paste `/#/nodes/{pubkey}` directly → full-screen on both desktop and
mobile.

## Tests
`node test-frontend-helpers.js` → 553 passed, 0 failed.

Co-authored-by: you <you@example.com>
2026-04-21 05:13:52 +00:00
efiten 997bf190ce fix(mobile): close button accessible + toolbar scrollable (#797) (#805)
## Summary

- **Node detail `top: 60px` → `64px`**: aligns with other overlay
panels, gives proper clearance from the 52px fixed nav bar
- **Mobile bottom sheet `z-index: 1050`**: node detail now renders above
the VCR bar (`z-index: 1000`), close button never obscured
- **Mobile `max-height: 60vh` → `60dvh`**: respects iOS Safari browser
chrome correctly
- **`.live-toggles` horizontal scroll**: `overflow-x: auto; flex-wrap:
nowrap` — all 8 checkboxes reachable via horizontal swipe

Fixes #797

## Test plan

- [x] Mobile portrait (<640px): tap a map node → bottom sheet slides up,
close button (✕) visible and tappable above VCR bar
- [x] Mobile portrait: scroll the live-header toggles horizontally → all
checkboxes reachable
- [x] Desktop/tablet (>640px): node detail panel top-right corner fully
below the nav bar
- [x] Desktop: close button functional, panel hides correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-20 22:10:18 -07:00
Kpa-clawbot 5ff4b75a07 qa: automate §10.1/§10.2 nodeBlacklist test (#822)
Automates QA plan §10.1 (nodeBlacklist hide) and §10.2 (DB retain),
flipping both rows from `human` to `auto`. Stacks on top of #808.

**What**
- New `qa/scripts/blacklist-test.sh` — env-driven harness:
  - Args: `BASELINE_URL TARGET_URL TEST_PUBKEY`
- Env: `TARGET_SSH_HOST`, `TARGET_SSH_KEY` (default
`/root/.ssh/id_ed25519`), `TARGET_CONFIG_PATH`, `TARGET_CONTAINER`,
optional `TARGET_DB_PATH` / `ADMIN_API_TOKEN`.
- Edits `nodeBlacklist` on target via remote `jq` (python3 fallback),
atomic move with preserved perms.
  - Restarts container, waits up to 120 s for `/api/stats == 200`.
- §10.1 asserts `/api/nodes/{pk}` is 404 **or** absent from `/api/nodes`
listing, and `/api/topology` does not reference the pubkey.
- §10.2 prefers `/api/admin/transmissions` if `ADMIN_API_TOKEN` set,
else falls back to `sqlite3` inside the container (and host as last
resort).
- **Teardown is mandatory** (`trap … EXIT INT TERM`): removes pubkey,
restarts, verifies the node is visible again. Teardown failures count
toward exit code.
- Exit code = number of failures; per-step / with classified failure
modes (`ssh-failed`, `restart-stuck`, `hide-failed`, `retain-failed`,
`teardown-failed`).
- `qa/plans/v3.6.0-rc.md` §10.1 / §10.2 mode → `auto
(qa/scripts/blacklist-test.sh)`.

**Why**
Manual blacklist verification was the slowest item in the §10 block and
the easiest to get wrong (forgetting teardown leaks state into the next
QA pass). Now it's a single command, public-repo-safe (zero PII /
hardcoded hosts), and the trap guarantees the target is restored.

`bash -n` passes locally. Live run requires staging credentials.

---------

Co-authored-by: meshcore-agent <agent@meshcore>
Co-authored-by: meshcore-agent <meshcore@openclaw.local>
2026-04-21 04:53:55 +00:00
Kpa-clawbot 2460e33f94 fix(#810): /health.recentPackets resolved_path falls back to longest sibling obs (#821)
## What + why

`fetchResolvedPathForTxBest` (used by every API path that fills the
top-level `resolved_path`, including
`/api/nodes/{pk}/health.recentPackets`) picked the observation with the
longest `path_json` and queried SQL for that single obs ID. When the
longest-path obs had `resolved_path` NULL but a shorter sibling had one,
the helper returned nil and the top-level field was dropped — even
though the data exists. QA #809 §2.1 caught it on the health endpoint
because that page surfaces it per-tx.

Fix: keep the LRU-friendly fast path (try the longest-path obs), then
fall back to scanning all observations of the tx and picking the longest
`path_json` that actually has a stored `resolved_path`.

## Changes
- `cmd/server/resolved_index.go`: extend `fetchResolvedPathForTxBest`
with a fallback through `fetchResolvedPathsForTx`.
- `cmd/server/issue810_repro_test.go`: regression test — seeds a tx
whose longest-path obs lacks `resolved_path` and a shorter sibling has
it, then asserts `/api/packets` and
`/api/nodes/{pk}/health.recentPackets` agree.

## Tests
`go test ./... -count=1` from `cmd/server` — PASS (full suite, ~19s).

## Perf
Fast path unchanged (single LRU/SQL lookup, dominant case). Fallback
only runs when the longest-path obs has NULL `resolved_path` — one
indexed query per affected tx, bounded by observations-per-tx (small).

Closes #810

---------

Co-authored-by: you <you@example.com>
2026-04-21 04:51:24 +00:00
Kpa-clawbot f701121672 Add qa/ — project-specific QA artifacts for the qa-suite skill (#808)
Adds the CoreScope-side artifacts that pair with the generic [`qa-suite`
skill](https://github.com/Kpa-clawbot/ai-sdlc/pull/1).

## Layout

```
qa/
├── README.md
├── plans/
│   └── v3.6.0-rc.md       # 34-commit test plan since v3.5.1
└── scripts/
    └── api-contract-diff.sh  # CoreScope-tuned API contract diff
```

The skill ships the reusable engine + qa-engineer persona + an example
plan. This PR adds the CoreScope-tuned plan and the CoreScope-tuned
script (correct seed lookups for our `{packets, total}` response shape,
our endpoint list, our `resolved_path` requirement). Read by the parent
agent at runtime.

## How to use

From chat:

- `qa staging` — runs the latest `qa/plans/v*-rc.md` against staging,
files a fresh GH issue with the report
- `qa pr <N>` — uses `qa/plans/pr-<N>.md` if present, else latest RC
plan; comments on the PR
- `qa v3.6.0-rc` — runs that specific plan

The qa-engineer subagent walks every step, classifying each as `auto`
(script) / `browser` (UI assertion) / `human` (manual) / `browser+auto`.
Quantified pass criteria are mandatory — banned phrases: 'visually
aligned' / 'fast' / 'no regression'.

## Plan v3.6.0-rc contents

Covers the 34 commits since v3.5.1:
- §1 Memory & Load (#806, #790, #807) — heap thresholds, sawtooth
pattern
- §2 API contract (#806) — every endpoint that should carry
`resolved_path`, auto-checked by `api-contract-diff.sh`
- §3 Decoder & hashing (#787, #732, #747, #766, #794, #761)
- §4 Channels (#725 series M1–M5)
- §5 Clock skew (#690 series M1–M3)
- §6 Observers (#764, #774)
- §7 Multi-byte hash adopters (#758, #767)
- §8 Frontend nav & deep linking (#739, #740, #779, #785, #776, #745)
- §9 Geofilter (#735, #734)
- §10 Node blacklist (#742)
- §11 Deploy/ops

Release blockers: §1.2, §2, §3. §4 is the headline-feature gate.

## Adding new plans

Per release: copy `plans/v<last>-rc.md` to `plans/v<new>-rc.md` and
update commit-range header, new sections, GO criteria.

Per PR: create `plans/pr-<N>.md` with the bare minimum for that PR's
surface area.

Co-authored-by: you <you@example.com>
2026-04-20 21:46:57 -07:00
Kpa-clawbot d7fe24e2db Fix channel filter on Packets page (UI + API) — #812 (#816)
Closes #812

## Root causes

**Server (`/api/packets?channel=…` returned identical totals):**
The handler in `cmd/server/routes.go` never read the `channel` query
parameter into `PacketQuery`, so it was silently ignored by both the
SQLite path (`db.go::buildTransmissionWhere`) and the in-memory path
(`store.go::filterPackets`). The codebase already had everything else in
place — the `channel_hash` column with an index from #762, decoded
`channel` / `channelHashHex` fields on each packet — it just wasn't
wired up.

**UI (`/#/packets` had no channel filter):**
`public/packets.js` rendered observer / type / time-window / region
filters but no channel control, and didn't read `?channel=` from the
URL.

## Fix

### Server
- New `Channel` field on `PacketQuery`; `handlePackets` reads
`r.URL.Query().Get("channel")`.
- DB path filters by the indexed `channel_hash` column (exact match).
- In-memory path: helper `packetMatchesChannel` matches
`decoded.channel` (plaintext, e.g. `#test`, `public`) or `enc_<HEX>`
against `channelHashHex` for undecryptable GRP_TXT. Uses cached
`ParsedDecoded()` so it's O(1) after first parse. Fast-path index guards
and the grouped-cache key updated to include channel.
- Regression test (`channel_filter_test.go`): `channel=#test` returns ≥1
GRP_TXT packet and fewer than baseline; `channel=nonexistentchannel`
returns `total=0`.

### UI
- New `<select id="fChannel">` populated from `/api/channels`.
- Round-trips via `?channel=…` on the URL hash (read on init, written on
change).
- Pre-seeds the current value as an option so encrypted hashes not in
`/api/channels` still display as selected on reload.
- On change, calls `loadPackets()` so the server-side filter applies
before pagination.

## Perf

Filter adds at most one cached map lookup per packet (DB path uses
indexed column, store path uses `ParsedDecoded()` cache). Staging
baseline 149–190 ms for `?channel=#test&limit=50`; the new comparison is
negligible. Target ≤ 500 ms preserved.

## Tests
`cd cmd/server && go test ./... -count=1 -timeout 120s` → PASS.

---------

Co-authored-by: you <you@example.com>
2026-04-20 21:46:34 -07:00
Kpa-clawbot a9732e64ae fix(nodes): render clock-skew section in side panel (#813) (#814)
Closes #813

## Root cause
The Node detail **side panel** (`renderDetail()`,
`public/nodes.js:1145`) was missing both the `#node-clock-skew`
placeholder div and the `loadClockSkew()` IIFE loader. Those exist only
in the **full-screen** detail page (`loadFullNode`, lines 498 / 632), so
any node opened via deep link or click in the listing — which uses the
side panel — showed no clock-skew UI even when
`/api/nodes/{pk}/clock-skew` returned rich data.

## Fix
Mirror the full-screen template branch and IIFE in `renderDetail`:
- Add `<div class="node-detail-section skew-detail-section"
id="node-clock-skew" style="display:none">` to the side-panel template
(right above Observers).
- Add an async `loadClockSkewPanel()` IIFE after the panel `innerHTML`
is set, using the same severity/badge/drift/sparkline rendering and the
`severity === 'no_clock'` branch the full-screen view uses.

No new helpers — reuses existing window globals (`formatSkew`,
`formatDrift`, `renderSkewBadge`, `renderSkewSparkline`).

## Verification
- Syntax check: `node -c public/nodes.js` ✓
- `node test-frontend-helpers.js` → 553/553 ✓
- Browser: staging runs master so I couldn't validate the deployed UI
yet. Manual repro after deploy:
1. Open `https://analyzer.00id.net/#/nodes`, click any node with a known
skew (e.g. Puppy Solar `a8dde6d7…` shows ` -23d 8h` in listing).
2. Side panel should show a ** Clock Skew** section with median skew,
severity badge, drift line, and sparkline.
3. For `severity === 'no_clock'` (e.g. SKCE_RS `14531bd2…`), section
shows "No Clock" instead of skew value.

---------

Co-authored-by: you <you@example.com>
2026-04-20 21:45:42 -07:00
Kpa-clawbot 60be48dc5e fix(channels): lock affordance on deep link to encrypted channel without key (#815)
Closes #811

## What
Deep linking to `/#/channels/%23private` (encrypted channel, no key
configured) now shows the existing 🔒 lock affordance instead of an empty
"No messages in this channel yet" pane.

## Why
`selectChannel` only rendered the lock message inside the `if (ch &&
ch.encrypted)` branch. On a cold deep link:

- `loadChannels` omits encrypted channels unless the toggle is on, so
`ch` is `undefined`.
- The hash isn't `user:`-prefixed, so that branch is skipped too.
- Code falls through to the REST fetch, returns 0 messages, and
`renderMessages` shows the generic empty state.

## Fix
Add a `#`-prefixed-hash branch immediately before the REST fetch:

- If a stored key matches the channel name → decrypt and render.
- Otherwise → reuse the existing 🔒 "encrypted and no decryption key is
configured" message.

## Trace (URL → render)
1. `#/channels/%23private` → `init(routeParam='#private')`
2. `loadChannels()` → `channels` has no `#private` entry (toggle off)
3. `selectChannel('#private')` → `ch` undefined → skips encrypted
branches → **new check fires** → lock message
4. With key stored: same check → `decryptAndRender`

## Validation
- `node test-frontend-helpers.js` → 553 passed, 0 failed
- Manual trace above; change is a 15-line localized guard before the
REST fetch, no hot-path or perf impact.

Co-authored-by: meshcore-agent <agent@corescope.local>
2026-04-20 21:38:59 -07:00
Kpa-clawbot 9e90548637 perf(#800): remove per-StoreTx ResolvedPath, replace with membership index + on-demand decode (#806)
## Summary

Remove `ResolvedPath []*string` field from `StoreTx` and `StoreObs`
structs, replacing it with a compact membership index + on-demand SQL
decode. This eliminates the dominant heap cost identified in profiling
(#791, #799).

**Spec:** #800 (consolidated from two rounds of expert + implementer
review on #799)

Closes #800
Closes #791

## Design

### Removed
- `StoreTx.ResolvedPath []*string`
- `StoreObs.ResolvedPath []*string`
- `TransmissionResp.ResolvedPath`, `ObservationResp.ResolvedPath` struct
fields

### Added
| Structure | Purpose | Est. cost at 1M obs |
|---|---|---:|
| `resolvedPubkeyIndex map[uint64][]int` | FNV-1a(pubkey) → []txID
forward index | 50–120 MB |
| `resolvedPubkeyReverse map[int][]uint64` | txID → []hashes for clean
removal | ~40 MB |
| `apiResolvedPathLRU` (10K entries) | FIFO cache for on-demand API
decode | ~2 MB |

### Decode-window discipline
`resolved_path` JSON decoded once per packet. Consumers fed in order,
temp slice dropped — never stored on struct:
1. `addToByNode` — relay node indexing
2. `touchRelayLastSeen` — relay liveness DB updates
3. `byPathHop` resolved-key entries
4. `resolvedPubkeyIndex` + reverse insert
5. WebSocket broadcast map (raw JSON bytes)
6. Persist batch (raw JSON bytes for SQL UPDATE)

### Collision safety
When the forward index returns candidates, a batched SQL query confirms
exact pubkey presence using `LIKE '%"pubkey"%'` on the `resolved_path`
column.

### Feature flag
`useResolvedPathIndex` (default `true`). Off-path is conservative: all
candidates kept, index not consulted. For one-release rollback safety.

## Files changed

| File | Changes |
|---|---|
| `resolved_index.go` | **New** — index structures, LRU cache, on-demand
SQL helpers, collision safety |
| `store.go` | Remove RP fields, decode-window discipline in
Load/Ingest, on-demand txToMap/obsToMap/enrichObs, eviction cleanup via
SQL, memory accounting update |
| `types.go` | Remove RP fields from TransmissionResp/ObservationResp |
| `routes.go` | Replace `nodeInResolvedPath` with
`nodeInResolvedPathViaIndex`, remove RP from mapSlice helpers |
| `neighbor_persist.go` | Refactor backfill: reverse-map removal →
forward+reverse insert → LRU invalidation |

## Tests added (27 new)

**Unit:**
- `TestStoreTx_ResolvedPathFieldAbsent` — reflection guard
- `TestResolvedPubkeyIndex_BuildFromLoad` — forward+reverse consistency
- `TestResolvedPubkeyIndex_HashCollision` — SQL collision safety
- `TestResolvedPubkeyIndex_IngestUpdate` — maps reflect new ingests
- `TestResolvedPubkeyIndex_RemoveOnEvict` — clean removal via reverse
map
- `TestResolvedPubkeyIndex_PerObsCoverage` — non-best obs pubkeys
indexed
- `TestAddToByNode_WithoutResolvedPathField`
- `TestTouchRelayLastSeen_WithoutResolvedPathField`
- `TestWebSocketBroadcast_IncludesResolvedPath`
- `TestBackfill_InvalidatesLRU`
- `TestEviction_ByNodeCleanup_OnDemandSQL`
- `TestExtractResolvedPubkeys`, `TestMergeResolvedPubkeys`
- `TestResolvedPubkeyHash_Deterministic`
- `TestLRU_EvictionOnFull`

**Endpoint:**
- `TestPathsThroughNode_NilResolvedPathFallback`
- `TestPacketsAPI_OnDemandResolvedPath`
- `TestPacketsAPI_OnDemandResolvedPath_LRUHit`
- `TestPacketsAPI_OnDemandResolvedPath_Empty`

**Feature flag:**
- `TestFeatureFlag_OffPath_PreservesOldBehavior`
- `TestFeatureFlag_Toggle_NoStateLeak`

**Concurrency:**
- `TestReverseMap_NoLeakOnPartialFailure`
- `TestDecodeWindow_LockHoldTimeBounded`
- `TestLivePolling_LRUUnderConcurrentIngest`

**Regression:**
- `TestRepeaterLiveness_StillAccurate`

**Benchmarks:**
- `BenchmarkLoad_BeforeAfter`
- `BenchmarkResolvedPubkeyIndex_Memory`
- `BenchmarkPathsThroughNode_Latency`
- `BenchmarkLivePolling_UnderIngest`

## Benchmark results

```
BenchmarkResolvedPubkeyIndex_Memory/pubkeys=50K     429ms  103MB   777K allocs
BenchmarkResolvedPubkeyIndex_Memory/pubkeys=500K   4205ms  896MB  7.67M allocs
BenchmarkLoad_BeforeAfter                            65ms   20MB   202K allocs
BenchmarkPathsThroughNode_Latency                   3.9µs    0B      0 allocs
BenchmarkLivePolling_UnderIngest                    5.4µs  545B      7 allocs
```

Key: per-obs `[]*string` overhead completely eliminated. At 1M obs with
3 hops average, this saves ~72 bytes/obs × 1M = ~68 MB just from the
slice headers + pointers, plus the JSON-decoded string data (~900 MB at
scale per profiling).

## Design choices

- **FNV-1a instead of xxhash**: stdlib availability, no external
dependency. Performance is equivalent for this use case (pubkey strings
are short).
- **FIFO LRU instead of true LRU**: simpler implementation, adequate for
the access pattern (mostly sequential obs IDs from live polling).
- **Grouped packets view omits resolved_path**: cold path, not worth SQL
round-trip per page render.
- **Backfill pending check uses reverse-map presence** instead of
per-obs field: if a tx has any indexed pubkeys, its observations are
considered resolved.


Closes #807

---------

Co-authored-by: you <you@example.com>
2026-04-20 19:55:00 -07:00
Kpa-clawbot a8e1cea683 fix: use payload type bits only in content hash (not full header byte) (#787)
## Problem

The firmware computes packet content hash as:

```
SHA256(payload_type_byte + [path_len for TRACE] + payload)
```

Where `payload_type_byte = (header >> 2) & 0x0F` — just the payload type
bits (2-5).

CoreScope was using the **full header byte** in its hash computation,
which includes route type bits (0-1) and version bits (6-7). This meant
the same logical packet produced different content hashes depending on
route type — breaking dedup and packet lookup.

**Firmware reference:** `Packet.cpp::calculatePacketHash()` uses
`getPayloadType()` which returns `(header >> PH_TYPE_SHIFT) &
PH_TYPE_MASK`.

## Fix

- Extract only payload type bits: `payloadType := (headerByte >> 2) &
0x0F`
- Include `path_len` byte in hash for TRACE packets (matching firmware
behavior)
- Applied to both `cmd/server/decoder.go` and `cmd/ingestor/decoder.go`

## Tests Added

- **Route type independence:** Same payload with FLOOD vs DIRECT route
types produces identical hash
- **TRACE path_len inclusion:** TRACE packets with different `path_len`
produce different hashes
- **Firmware compatibility:** Hash output matches manual computation of
firmware algorithm

## Migration Impact

Existing packets in the DB have content hashes computed with the old
(incorrect) formula. Options:

1. **Recompute hashes** via migration (recommended for clean state)
2. **Dual lookup** — check both old and new hash on queries (backward
compat)
3. **Accept the break** — old hashes become stale, new packets get
correct hashes

Recommend option 1 (migration) as a follow-up. The volume of affected
packets depends on how many distinct route types were seen for the same
logical packet.

Fixes #786

---------

Co-authored-by: you <you@example.com>
2026-04-18 11:52:22 -07:00
Kpa-clawbot bf674ebfa2 feat: validate advert signatures on ingest, reject corrupt packets (#794)
## Summary

Validates ed25519 signatures on ADVERT packets during MQTT ingest.
Packets with invalid signatures are rejected before storage, preventing
corrupt/truncated adverts from polluting the database.

## Changes

### Ingestor (`cmd/ingestor/`)

- **Signature validation on ingest**: After decoding an ADVERT, checks
`SignatureValid` from the decoder. Invalid signatures → packet dropped,
never stored.
- **Config flag**: `validateSignatures` (default `true`). Set to `false`
to disable validation for backward compatibility with existing installs.
- **`dropped_packets` table**: New SQLite table recording every rejected
packet with full attribution:
- `hash`, `raw_hex`, `reason`, `observer_id`, `observer_name`,
`node_pubkey`, `node_name`, `dropped_at`
  - Indexed on `observer_id` and `node_pubkey` for investigation queries
- **`SignatureDrops` counter**: New atomic counter in `DBStats`, logged
in periodic stats output as `sig_drops=N`
- **Retention**: `dropped_packets` pruned alongside metrics on the same
`retention.metricsDays` schedule

### Server (`cmd/server/`)

- **`GET /api/dropped-packets`** (API key required): Returns recent
drops with optional `?observer=` and `?pubkey=` filters, `?limit=`
(default 100, max 500)
- **`signatureDrops`** field added to `/api/stats` response (count from
`dropped_packets` table)

### Tests (8 new)

| Test | What it verifies |
|------|-----------------|
| `TestSigValidation_ValidAdvertStored` | Valid advert passes validation
and is stored |
| `TestSigValidation_TamperedSignatureDropped` | Tampered signature →
dropped, recorded in `dropped_packets` with correct fields |
| `TestSigValidation_TruncatedAppdataDropped` | Truncated appdata
invalidates signature → dropped |
| `TestSigValidation_DisabledByConfig` | `validateSignatures: false`
skips validation, stores tampered packet |
| `TestSigValidation_DropCounterIncrements` | Counter increments
correctly across multiple drops |
| `TestSigValidation_LogContainsFields` | `dropped_packets` row contains
hash, reason, observer, pubkey, name |
| `TestPruneDroppedPackets` | Old entries pruned, recent entries
retained |
| `TestShouldValidateSignatures_Default` | Config helper returns correct
defaults |

### Config example

```json
{
  "validateSignatures": true
}
```

Fixes #793

---------

Co-authored-by: you <you@example.com>
2026-04-18 11:39:13 -07:00
Kpa-clawbot d596becca3 feat: bounded cold load — limit Load() by memory budget (#790)
## Implements #748 M1 — Bounded Cold Load

### Problem
`Load()` pulls the ENTIRE database into RAM before eviction runs. On a
1GB database, this means 3+ GB peak memory at startup, regardless of
`maxMemoryMB`. This is the root cause of #743 (OOM on 2GB VMs).

### Solution
Calculate the maximum number of transmissions that fit within the
`maxMemoryMB` budget and use a SQL subquery LIMIT to load only the
newest packets.

**Two-phase approach** (avoids the JOIN-LIMIT row count problem):
```sql
SELECT ... FROM transmissions t
LEFT JOIN observations o ON ...
WHERE t.id IN (SELECT id FROM transmissions ORDER BY first_seen DESC LIMIT ?)
ORDER BY t.first_seen ASC, o.timestamp DESC
```

### Changes
- **`estimateStoreTxBytesTypical(numObs)`** — estimates memory cost of a
typical transmission without needing an actual `StoreTx` instance. Used
for budget calculation.
- **Budget calculation in `Load()`** — `maxPackets = (maxMemoryMB *
1048576) / avgBytesPerPacket` with a floor of 1000 packets.
- **Subquery LIMIT** — loads only the newest N transmissions when
bounded.
- **`oldestLoaded` tracking** — records the oldest packet timestamp in
memory so future SQL fallback queries (M2+) know where in-memory data
ends.
- **Perf stats** — `oldestLoaded` exposed in `/api/perf/store-stats`.
- **Logging** — bounded loads show `Loaded X/Y transmissions (limited by
ZMB budget)`.

### When `maxMemoryMB=0` (unlimited)
Behavior is completely unchanged — no LIMIT clause, all packets loaded.

### Tests (6 new)
| Test | Validates |
|------|-----------|
| `TestBoundedLoad_LimitedMemory` | With 1MB budget, loads fewer than
total (hits 1000 minimum) |
| `TestBoundedLoad_NewestFirst` | Loaded packets are the newest, not
oldest |
| `TestBoundedLoad_OldestLoadedSet` | `oldestLoaded` matches first
packet's `FirstSeen` |
| `TestBoundedLoad_UnlimitedWithZero` | `maxMemoryMB=0` loads all
packets |
| `TestBoundedLoad_AscendingOrder` | Packets remain in ascending
`first_seen` order after bounded load |
| `TestEstimateStoreTxBytesTypical` | Estimate grows with observation
count, exceeds floor |

Plus benchmarks: `BenchmarkLoad_Bounded` vs `BenchmarkLoad_Unlimited`.

### Perf justification
On a 5000-transmission test DB with 1MB budget:
- Bounded: loads 1000 packets (the minimum) in ~1.3s
- The subquery uses SQLite's index on `first_seen` — O(N log N) for the
LIMIT, then indexed JOIN for observations
- No full table scan needed when bounded

### Next milestones
- **M2**: Packet list/search SQL fallback (uses `oldestLoaded` boundary)
- **M3**: Node analytics SQL fallback
- **M4-M5**: Remaining endpoint fallbacks + live-only memory store

---------

Co-authored-by: you <you@example.com>
2026-04-17 18:35:44 -07:00
Joel Claw b9ba447046 feat: add nodeBlacklist config to hide abusive/troll nodes (#742)
## Problem

Some mesh participants set offensive names, report deliberately false
GPS positions, or otherwise troll the network. Instance operators
currently have no way to hide these nodes from public-facing APIs
without deleting the underlying data.

## Solution

Add a `nodeBlacklist` array to `config.json` containing public keys of
nodes to exclude from all API responses.

### Blacklisted nodes are filtered from:

- `GET /api/nodes` — list endpoint
- `GET /api/nodes/search` — search results
- `GET /api/nodes/{pubkey}` — detail (returns 404)
- `GET /api/nodes/{pubkey}/health` — returns 404
- `GET /api/nodes/{pubkey}/paths` — returns 404
- `GET /api/nodes/{pubkey}/analytics` — returns 404
- `GET /api/nodes/{pubkey}/neighbors` — returns 404
- `GET /api/nodes/bulk-health` — filtered from results

### Config example

```json
{
  "nodeBlacklist": [
    "aabbccdd...",
    "11223344..."
  ]
}
```

### Design decisions

- **Case-insensitive** — public keys normalized to lowercase
- **Whitespace trimming** — leading/trailing whitespace handled
- **Empty entries ignored** — `""` or `" "` do not cause false positives
- **Nil-safe** — `IsBlacklisted()` on nil Config returns false
- **Backward-compatible** — empty/missing `nodeBlacklist` has zero
effect
- **Lazy-cached set** — blacklist converted to `map[string]bool` on
first lookup

### What this does NOT do (intentionally)

- Does **not** delete or modify database data — only filters API
responses
- Does **not** block packet ingestion — data still flows for analytics
- Does **not** filter `/api/packets` — only node-facing endpoints are
affected

## Testing

- Unit tests for `Config.IsBlacklisted()` (case sensitivity, whitespace,
empty entries, nil config)
- Integration tests for `/api/nodes`, `/api/nodes/{pubkey}`,
`/api/nodes/search`
- Full test suite passes with no regressions
2026-04-17 23:43:05 +00:00
Kpa-clawbot b8846c2db2 fix: show lock message for encrypted channels without key on deep link (#783)
## Problem

Deep-linking to an encrypted channel (e.g. `#/channels/42`) when the
user has no client-side decryption key falls through to the plaintext
API fetch, displaying gibberish base64/binary content instead of a
meaningful message.

## Root Cause

In `selectChannel()`, the encrypted channel key-matching loop iterates
all stored keys. If none match, execution falls through to the normal
plaintext message fetch — which returns raw encrypted data rendered as
gibberish.

## Fix

After the key-matching loop for encrypted channels, return early with
the lock message instead of falling through.

**3 lines added** in `public/channels.js`, **108 lines** regression test
in `test-frontend-helpers.js`.

## Investigation: Sidebar Display

The sidebar filtering is already correct:
- DB path: SQL filters out `enc_` prefix channel hashes
- In-memory path: Only returns `type: CHAN` (server-decrypted) channels,
with `hasGarbageChars` validation
- Server-side decryption: MAC verification (2-byte HMAC) + UTF-8 +
non-printable character validation prevents false-positive decryptions
- Encrypted channels only appear when the toggle is explicitly enabled

## Testing

- All existing tests pass
- New regression test verifies: lock message shown, messages API NOT
called for encrypted channels without key

Fixes #781

---------

Co-authored-by: you <you@example.com>
2026-04-17 16:40:18 -07:00
Kpa-clawbot 34b8dc8961 fix: improve #778 detail link — call init() directly instead of router teardown (#785)
Improves the fix for #778 (replaces #779's approach).

## Problem

When clicking "Details" in the node side panel, the hash is already
`#/nodes/{pubkey}` (set by `replaceState` in `selectNode`). The link
targets the same hash → no `hashchange` event → router never fires →
detail view never renders.

## What was wrong with #779

PR #779 used `replaceState('#/')` + `location.hash = target`
synchronously, which forces a full SPA router teardown/rebuild cycle
just to re-render the same page. This is wasteful and can cause visual
flicker.

## This fix

**Detail link** (`#/nodes/{pubkey}`): Calls `init(app, pubkey)` directly
— no router teardown, no page flash. The `init()` function already
handles rendering the detail view when `routeParam` is set.

**Analytics link** (`#/nodes/{pubkey}/analytics`): Uses `setTimeout` to
ensure reliable `hashchange` firing, since this routes to a different
page (`node-analytics`) that requires the full SPA router.

## Testing

- Frontend helper tests: 552/552 
- Packet filter tests: 62/62 
- Aging tests: 29/29 
- Go server tests: pass 
- Go ingestor tests: pass 

---------

Co-authored-by: you <you@example.com>
2026-04-17 16:39:57 -07:00
Joel Claw fa3f623bd6 feat: add observer retention — remove stale observers after configurable days (#764)
## Summary

Observers that stop actively sending data now get removed after a
configurable retention period (default 14 days).

Previously, observers remained in the `observers` table forever. This
meant nodes that were once observers for an instance but are no longer
connected (even if still active in the mesh elsewhere) would continue
appearing in the observer list indefinitely.

## Key Design Decisions

- **Active data requirement**: `last_seen` is only updated when the
observer itself sends packets (via `stmtUpdateObserverLastSeen`). Being
seen by another node does NOT update this field. So an observer must
actively send data to stay listed.
- **Default: 14 days** — observers not seen in 14 days are removed
- **`-1` = keep forever** — for users who want observers to never be
removed
- **`0` = use default (14 days)** — same as not setting the field
- **Runs on startup + daily ticker** — staggered 3 minutes after metrics
prune to avoid DB contention

## Changes

| File | Change |
|------|--------|
| `cmd/ingestor/config.go` | Add `ObserverDays` to `RetentionConfig`,
add `ObserverDaysOrDefault()` |
| `cmd/ingestor/db.go` | Add `RemoveStaleObservers()` — deletes
observers with `last_seen` before cutoff |
| `cmd/ingestor/main.go` | Wire up startup + daily ticker for observer
retention |
| `cmd/server/config.go` | Add `ObserverDays` to `RetentionConfig`, add
`ObserverDaysOrDefault()` |
| `cmd/server/db.go` | Add `RemoveStaleObservers()` (server-side, uses
read-write connection) |
| `cmd/server/main.go` | Wire up startup + daily ticker, shutdown
cleanup |
| `cmd/server/routes.go` | Admin prune API now also removes stale
observers |
| `config.example.json` | Add `observerDays: 14` with documentation |
| `cmd/ingestor/coverage_boost_test.go` | 4 tests: basic removal, empty
store, keep forever (-1), default (0→14) |
| `cmd/server/config_test.go` | 4 tests: `ObserverDaysOrDefault` edge
cases |

## Config Example

```json
{
  "retention": {
    "nodeDays": 7,
    "observerDays": 14,
    "packetDays": 30,
    "_comment": "observerDays: -1 = keep forever, 0 = use default (14)"
  }
}
```

## Admin API

The `/api/admin/prune` endpoint now also removes stale observers (using
`observerDays` from config) and reports `observers_removed` in the
response alongside `packets_deleted`.

## Test Plan

- [x] `TestRemoveStaleObservers` — old observer removed, recent observer
kept
- [x] `TestRemoveStaleObserversNone` — empty store, no errors
- [x] `TestRemoveStaleObserversKeepForever` — `-1` keeps even year-old
observers
- [x] `TestRemoveStaleObserversDefault` — `0` defaults to 14 days
- [x] `TestObserverDaysOrDefault` (ingestor) —
nil/zero/positive/keep-forever
- [x] `TestObserverDaysOrDefault` (server) —
nil/zero/positive/keep-forever
- [x] Both binaries compile cleanly (`go build`)
- [ ] Manual: verify observer count decreases after retention period on
a live instance
2026-04-17 09:24:40 -07:00
Kpa-clawbot dfe383cc51 fix: node detail panel Details/Analytics links don't navigate (#779)
Fixes #778

## Problem

The Details and Analytics links in the node side panel don't navigate
when clicked. This is a regression from #739 (desktop node deep
linking).

**Root cause:** When a node is selected, `selectNode()` uses
`history.replaceState()` to set the URL to `#/nodes/{pubkey}`. The
Details link has `href="#/nodes/{pubkey}"` — the same hash. Clicking an
anchor with the same hash as the current URL doesn't fire the
`hashchange` event, so the SPA router never triggers navigation.

## Fix

Added a click handler on the `nodesRight` panel that intercepts clicks
on `.btn-primary` navigation links:

1. `e.preventDefault()` to stop the default anchor behavior
2. If the current hash already matches the target, temporarily clear it
via `replaceState`
3. Set `location.hash` to the target, which fires `hashchange` and
triggers the SPA router

This handles both the Details link (`#/nodes/{pubkey}`) and the
Analytics link (`#/nodes/{pubkey}/analytics`).

## Testing

- All frontend helper tests pass (552/552)
- All packet filter tests pass (62/62)
- All aging tests pass (29/29)
- Go server tests pass

---------

Co-authored-by: you <you@example.com>
2026-04-16 23:21:05 -07:00
you fa348efe2a fix: force-remove staging container before deploy — handles both compose and docker-run containers
The deploy step used only 'docker compose down' which can't remove
containers created via 'docker run'. Now explicitly stops+removes
the named container first, then runs compose down as cleanup.

Permanent fix for the recurring CI deploy failure.
2026-04-17 05:08:32 +00:00
Kpa-clawbot a9a18ff051 fix: neighbor graph slider persists to localStorage, default 0.7 (#776)
## Summary

The neighbor graph min score slider didn't persist its value to
localStorage, resetting to 0.10 on every page load. This was a poor
default for most use cases.

## Changes

- **Default changed from 0.10 to 0.70** — more useful starting point
that filters out low-confidence edges
- **localStorage persistence** — slider value saved on change, restored
on page load
- **3 new tests** in `test-frontend-helpers.js` verifying default value,
load behavior, and save behavior

## Testing

- `node test-frontend-helpers.js` — 547 passed, 0 failed
- `node test-packet-filter.js` — 62 passed, 0 failed
- `node test-aging.js` — 29 passed, 0 failed

---------

Co-authored-by: you <you@example.com>
2026-04-16 22:04:51 -07:00
Kpa-clawbot ceea136e97 feat: observer graph representation (M1+M2) (#774)
## Summary

Fixes #753 — Milestones M1 and M2: Observer nodes in the neighbor graph
are now correctly labeled, colored, and filterable.

### M1: Label + color observers

**Backend** (`cmd/server/neighbor_api.go`):
- `buildNodeInfoMap()` now queries the `observers` table after building
from `nodes`
- Observer-only pubkeys (not already in the map as repeaters etc.) get
`role: "observer"` and their name from the observers table
- Observer-repeaters keep their repeater role (not overwritten)

**Frontend**:
- CSS variable `--role-observer: #8b5cf6` added to `:root`
- `ROLE_COLORS.observer` was already defined in `roles.js`

### M2: Observer filter checkbox (default unchecked)

**Frontend** (`public/analytics.js`):
- Observer checkbox added to the role filter section, **unchecked by
default**
- Observers create hub-and-spoke patterns (one observer can have 100+
edges) that drown out the actual repeater topology — hiding them by
default keeps the graph clean
- Fixed `applyNGFilters()` which previously always showed observers
regardless of checkbox state

### Tests

- Backend: `TestBuildNodeInfoMap_ObserverEnrichment` — verifies
observer-only pubkeys get name+role from observers table, and
observer-repeaters keep their repeater role
- All existing Go tests pass
- All frontend helper tests pass (544/544)

---------

Co-authored-by: you <you@example.com>
2026-04-16 21:35:14 -07:00
you 99dc4f805a fix: E2E neighbor test — use hash evaluation instead of page.goto for reliable SPA navigation
page.goto with hash-only change may not reliably trigger hashchange in
Playwright, causing the mobile full-screen node view to never render.
Use page.evaluate to set location.hash directly, which guarantees the
SPA router fires. Also increase timeout from 10s to 15s for CI margin.
2026-04-17 03:45:16 +00:00
Kpa-clawbot ba7cd0fba7 fix: clock skew sanity checks — filter epoch-0, cap drift, min samples (#769)
Nodes with dead RTCs show -690d skew and -3 billion s/day drift. Fix:

1. **No Clock severity**: |skew| > 365d → `no_clock`, skip drift
2. **Drift cap**: |drift| > 86400 s/day → nil (physically impossible)
3. **Min samples**: < 5 samples → no drift regression
4. **Frontend**: 'No Clock' badge, '–' for unreliable drift

Fixes the crazy stats on the Clock Health fleet view.

---------

Co-authored-by: you <you@example.com>
2026-04-16 08:10:47 -07:00
Kpa-clawbot 6a648dea11 fix: multi-byte adopters — all node types, role column, advert precedence (#754) (#767)
## Fix: Multi-Byte Adopters Table — Three Bugs (#754)

### Bug 1: Companions in "Unknown"
`computeMultiByteCapability()` was repeater-only. Extended to classify
**all node types** (companions, rooms, sensors). A companion advertising
with 2-byte hash is now correctly "Confirmed".

### Bug 2: No Role Column
Added a **Role** column to the merged Multi-Byte Hash Adopters table,
color-coded using `ROLE_COLORS` from `roles.js`. Users can now
distinguish repeaters from companions without clicking through to node
detail.

### Bug 3: Data Source Disagreement
When adopter data (from `computeAnalyticsHashSizes`) shows `hashSize >=
2` but capability only found path evidence ("Suspected"), the
advert-based adopter data now takes precedence → "Confirmed". The
adopter hash sizes are passed into `computeMultiByteCapability()` as an
additional confirmed evidence source.

### Changes
- `cmd/server/store.go`: Extended capability to all node types, accept
adopter hash sizes, prioritize advert evidence
- `public/analytics.js`: Added Role column with color-coded badges
- `cmd/server/multibyte_capability_test.go`: 3 new tests (companion
confirmed, role populated, adopter precedence)

### Tests
- All 10 multi-byte capability tests pass
- All 544 frontend helper tests pass
- All 62 packet filter tests pass
- All 29 aging tests pass

---------

Co-authored-by: you <you@example.com>
2026-04-16 00:51:38 -07:00
Kpa-clawbot 29157742eb feat: show collision details in Hash Usage Matrix for all hash sizes (#758)
## Summary

Shows which prefixes are colliding in the Hash Usage Matrix, making the
"PREFIX COLLISIONS: N" count actionable.

Fixes #757

## Changes

### Frontend (`public/analytics.js`)
- **Clickable collision count**: When collisions > 0, the stat card is
clickable and scrolls to the collision details section. Shows a `▼`
indicator.
- **3-byte collision table**: The collision risk section and
`renderCollisionsFromServer` now render for all hash sizes including
3-byte (was previously hidden/skipped for 3-byte).
- **Helpful hint**: 3-byte panel now says "See collision details below"
when collisions exist.

### Backend (`cmd/server/collision_details_test.go`)
- Test that collision details include correct prefix and node
name/pubkey pairs
- Test that collision details are empty when no collisions exist

### Frontend Tests (`test-frontend-helpers.js`)
- Test clickable stat card renders `onclick` and `cursor:pointer` when
collisions > 0
- Test non-clickable card when collisions = 0
- Test collision table renders correct node links (`#/nodes/{pubkey}`)
- Test no-collision message renders correctly

## What was already there

The backend already returned full collision details (prefix, nodes with
pubkeys/names/coords, distance classification) in the `hash-collisions`
API. The frontend already had `renderCollisionsFromServer` rendering a
rich table with node links. The gap was:
1. The 3-byte tab hid the collision risk section entirely
2. No visual affordance to navigate from the stat count to the details

## Perf justification

No new computation — collision data was already computed and returned by
the API. The only change is rendering it for 3-byte (same as
1-byte/2-byte). The collision list is already limited by the backend
sort+slice pattern.

---------

Co-authored-by: you <you@example.com>
2026-04-16 00:18:25 -07:00
Kpa-clawbot ed19a19473 fix: correct field table offsets for transport routes (#766)
## Summary

Fixes #765 — packet detail field table showed wrong byte offsets for
transport routes.

## Problem

`buildFieldTable()` hardcoded `path_length` at byte 1 for ALL packet
types. For `TRANSPORT_FLOOD` (route_type=0) and `TRANSPORT_DIRECT`
(route_type=3), transport codes occupy bytes 1-4, pushing `path_length`
to byte 5.

This caused:
- Wrong offset numbers in the field table for transport packets
- Transport codes displayed AFTER path length (wrong byte order)
- `Advertised Hash Size` row referenced wrong byte

## Fix

- Use dynamic `offset` tracking that accounts for transport codes
- Render transport code rows before path length (matching actual wire
format)
- Store `pathLenOffset` for correct reference in ADVERT payload section
- Reuse already-parsed `pathByte0` for hash size calculation in path
section

## Tests

Added 4 regression tests in `test-frontend-helpers.js`:
- FLOOD (route_type=1): path_length at byte 1, no transport codes
- TRANSPORT_FLOOD (route_type=0): transport codes at bytes 1-4,
path_length at byte 5
- TRANSPORT_DIRECT (route_type=3): same offsets as TRANSPORT_FLOOD
- Field table row order matches byte layout for transport routes

All existing tests pass (538 frontend helpers, 62 packet filter, 29
aging).

Co-authored-by: you <you@example.com>
2026-04-16 00:15:56 -07:00
copelaje d27a7a653e fix case on channel key so Public decode/display works right (#761)
Simple change. Before this change Public wasn't showing up in the
channels display due to the case issue.
2026-04-16 00:14:47 -07:00
Kpa-clawbot 0e286d85fd fix: channel query performance — add channel_hash column, SQL-level filtering (#762) (#763)
## Problem
Channel API endpoints scan entire DB — 2.4s for channel list, 30s for
messages.

## Fix
- Added `channel_hash` column to transmissions (populated on ingest,
backfilled on startup)
- `GetChannels()` rewrites to GROUP BY channel_hash (one row per channel
vs scanning every packet)
- `GetChannelMessages()` filters by channel_hash at SQL level with
proper LIMIT/OFFSET
- 60s cache for channel list
- Index: `idx_tx_channel_hash` for fast lookups

Expected: 2.4s → <100ms for list, 30s → <500ms for messages.

Fixes #762

---------

Co-authored-by: you <you@example.com>
2026-04-16 00:09:36 -07:00
Kpa-clawbot bffcbdaa0b feat: add channel UX — visible button, hint, status feedback (#760)
## Fixes #759

The "Add Channel" input was a bare text field with no visible submit
button and no feedback — users didn't know how to submit or whether it
worked.

### Changes

**`public/channels.js`**
- Replaced bare `<input>` with structured form: label, input + button
row, hint text, status div
- Added `showAddStatus()` helper for visual feedback during/after
channel add
- Status messages: loading → success (with decrypted message count) /
warning (no messages) / error
- Auto-hide status after 5 seconds
- Fallback click handler on the `+` button for browsers that don't fire
form submit

**`public/style.css`**
- `.ch-add-form` — form container
- `.ch-add-label` — bold 13px label
- `.ch-add-row` — flex row for input + button
- `.ch-add-btn` — 32×32 accent-colored submit button
- `.ch-add-hint` — muted helper text
- `.ch-add-status` — feedback with success/warn/error/loading variants

**`test-channel-add-ux.js`** — 20 tests validating HTML structure, CSS
classes, and feedback logic

### Before / After

**Before:** Bare input field, no button, no hint, no feedback  
**After:** Labeled section with visible `+` button, format hint, and
status messages showing decryption results

---------

Co-authored-by: you <you@example.com>
2026-04-15 18:54:35 -07:00
Kpa-clawbot 3bdf72b4cf feat: clock skew UI — node badges, detail sparkline, fleet analytics (#690 M2+M3) (#752)
## Summary

Frontend visualizations for clock skew detection.

Implements #690 M2 and M3. Does NOT close #690 — M4+M5 remain.

### M2: Node badges + detail sparkline
- Severity badges ( green/yellow/orange/red) on node list next to each
node
- Node detail: Clock Skew section with current value, severity, drift
rate
- Inline SVG sparkline showing skew history, color-coded by severity
zones

### M3: Fleet analytics view
- 'Clock Health' section on Analytics page
- Sortable table: Name | Skew | Severity | Drift | Last Advert
- Filter buttons by severity (OK/Warning/Critical/Absurd)
- Summary stats: X nodes OK, Y warning, Z critical
- Color-coded rows

### Changes
- `public/nodes.js` — badge rendering + detail section
- `public/analytics.js` — fleet clock health view
- `public/roles.js` — severity color helpers
- `public/style.css` — badge + sparkline + fleet table styles
- `cmd/server/clock_skew.go` — added fleet summary endpoint
- `cmd/server/routes.go` — wired fleet endpoint
- `test-frontend-helpers.js` — 11 new tests

---------

Co-authored-by: you <you@example.com>
2026-04-15 15:25:50 -07:00
Kpa-clawbot 401fd070f8 fix: improve trackedBytes accuracy for memory estimation (#751)
## Problem

Fixes #743 — High memory usage / OOM with relatively small dataset.

`trackedBytes` severely undercounted actual per-packet memory because it
only tracked base struct sizes and string field lengths, missing major
allocations:

| Structure | Untracked Cost | Scale Impact |
|-----------|---------------|--------------|
| `spTxIndex` (O(path²) subpath entries) | 40 bytes × path combos |
50-150MB |
| `ResolvedPath` on observations | 24 bytes × elements | ~25MB |
| Per-tx maps (`obsKeys`, `observerSet`) | 200 bytes/tx flat | ~11MB |
| `byPathHop` index entries | 50 bytes/hop | 20-40MB |

This caused eviction to trigger too late (or not at all), leading to
OOM.

## Fix

Expanded `estimateStoreTxBytes` and `estimateStoreObsBytes` to account
for:

- **Per-tx maps**: +200 bytes flat for `obsKeys` + `observerSet` map
headers
- **Path hop index**: +50 bytes per hop in `byPathHop`
- **Subpath index**: +40 bytes × `hops*(hops-1)/2` combinations for
`spTxIndex`
- **Resolved paths**: +24 bytes per `ResolvedPath` element on
observations

Updated the existing `TestEstimateStoreTxBytes` to match new formula.
All existing eviction tests continue to pass — the eviction logic itself
is unchanged.

Also exposed `avgBytesPerPacket` in the perf API (`/api/perf`) so
operators can monitor per-packet memory costs.

## Performance

Benchmark confirms negligible overhead (called on every insert):

```
BenchmarkEstimateStoreTxBytes    159M ops    7.5 ns/op    0 B/op    0 allocs
BenchmarkEstimateStoreObsBytes   1B ops      1.0 ns/op    0 B/op    0 allocs
```

## Tests

- 6 new tests in `tracked_bytes_test.go`:
  - Reasonable value ranges for different packet sizes
  - 10-hop packets estimate significantly more than 2-hop (subpath cost)
  - Observations with `ResolvedPath` estimate more than without
  - 15 observations estimate >10x a single observation
- `trackedBytes` matches sum of individual estimates after batch insert
  - Eviction triggers correctly with improved estimates
- 2 benchmarks confirming sub-10ns estimate cost
- Updated existing `TestEstimateStoreTxBytes` for new formula
- Full test suite passes

---------

Co-authored-by: you <you@example.com>
2026-04-15 07:53:32 -07:00
Kpa-clawbot 1b315bf6d0 feat: PSK channels, channel removal, message caching (#725 M3+M4+M5) (#750)
## Summary

Implements milestones M3, M4, and M5 from #725 — all client-side, zero
server changes.

### M3: PSK channel support

The channel input field now accepts both `#channelname` (hashtag
derivation) and raw 32-char hex keys (PSK). Auto-detection: if input
starts with `#`, derive key via SHA-256; otherwise validate as hex and
store directly. Same decrypt pipeline — `ChannelDecrypt.decrypt()` takes
key bytes regardless of source.

Input placeholder updated to: `#LongFast or paste hex key`

### M4: Channel removal

User-added channels now show a ✕ button on hover. Click → confirm dialog
→ removes:
- Key from localStorage (`ChannelDecrypt.removeKey()`)
- Cached messages from localStorage
(`ChannelDecrypt.clearChannelCache()`)
- Channel entry from sidebar

If the removed channel was selected, the view resets to the empty state.

### M5: localStorage message caching with delta fetch

After client-side decryption, results are cached in localStorage keyed
by channel name:

```
{ messages: [...], lastTimestamp: "...", count: N, ts: Date.now() }
```

On subsequent visits:
1. **Instant render** — cached messages displayed immediately via
`onCacheHit` callback
2. **Delta fetch** — only packets newer than `lastTimestamp` are fetched
and decrypted
3. **Merge** — new messages merged with cache, deduplicated by
`packetHash`
4. **Cache invalidation** — if total candidate count changes, full
re-decrypt triggered
5. **Size limit** — max 1000 messages cached per channel (most recent
kept)

### Performance

- Delta fetch avoids re-decrypting the full history on every page load
- Cache-first rendering provides instant UI response
- `deduplicateAndMerge()` uses a hash set for O(n) dedup
- 1000-message cap prevents localStorage quota issues

### Tests (24 new)

- M3: hex key detection (valid/invalid patterns)
- M3: key derivation round-trip, channel hash computation
- M3: PSK key storage and retrieval
- M4: channel removal clears both key and cache
- M5: cache size limit enforcement (1200 → 1000 stored)
- M5: cache stores count and lastTimestamp
- M5: clearChannelCache works independently
- All existing tests pass (523 frontend helpers, 62 packet filter)

### Files changed

| File | Change |
|------|--------|
| `public/channel-decrypt.js` | `removeKey()` now clears cache;
`clearChannelCache()`; `setCache()` with count + size limit |
| `public/channels.js` | Extracted `decryptCandidates()`,
`deduplicateAndMerge()`; delta fetch logic; remove button handler;
cache-first rendering |
| `public/style.css` | `.ch-remove-btn` styles (hover-reveal ✕) |
| `test-channel-decrypt-m345.js` | 24 new tests |

Implements #725

Co-authored-by: you <you@example.com>
2026-04-14 23:23:02 -07:00
Kpa-clawbot a815e70975 feat: Clock skew detection — backend computation (M1) (#746)
## Summary

Implements **Milestone 1** of #690 — backend clock skew computation for
nodes and observers.

## What's New

### Clock Skew Engine (`clock_skew.go`)

**Phase 1 — Raw Skew Calculation:**
For every ADVERT observation: `raw_skew = advert_timestamp -
observation_timestamp`

**Phase 2 — Observer Calibration:**
Same packet seen by multiple observers → compute each observer's clock
offset as the median deviation from the per-packet median observation
timestamp. This identifies observers with their own clock drift.

**Phase 3 — Corrected Node Skew:**
`corrected_skew = raw_skew + observer_offset` — compensates for observer
clock error.

**Phase 4 — Trend Analysis:**
Linear regression over time-ordered skew samples estimates drift rate in
seconds/day. Detects crystal drift vs stable offset vs sudden jumps.

### Severity Classification

| Level | Threshold | Meaning |
|-------|-----------|---------|
|  OK | < 5 min | Normal |
| ⚠️ Warning | 5 min – 1 hour | Clock drifting |
| 🔴 Critical | 1 hour – 30 days | Likely no time source |
| 🟣 Absurd | > 30 days | Firmware default or epoch 0 |

### New API Endpoints

- `GET /api/nodes/{pubkey}/clock-skew` — per-node skew data (mean,
median, last, drift, severity)
- `GET /api/observers/clock-skew` — observer calibration offsets
- Clock skew also included in `GET /api/nodes/{pubkey}/analytics`
response as `clockSkew` field

### Performance

- 30-second compute cache avoids reprocessing on every request
- Operates on in-memory `byPayloadType[ADVERT]` index — no DB queries
- O(n) in total ADVERT observations, O(m log m) for median calculations

## Tests

15 unit tests covering:
- Severity classification at all thresholds
- Median/mean math helpers
- ISO timestamp parsing
- Timestamp extraction from decoded JSON (nested and top-level)
- Observer calibration with single and multi-observer scenarios
- Observer offset correction direction (verified the sign is
`+obsOffset`)
- Drift estimation: stable, linear, insufficient data, short time span
- JSON number extraction edge cases

## What's NOT in This PR

- No UI changes (M2–M4)
- No customizer integration (M5)
- Thresholds are hardcoded constants (will be configurable in M5)

Implements #690 M1.

---------

Co-authored-by: you <you@example.com>
2026-04-14 23:22:35 -07:00
Kpa-clawbot aa84ce1e6a fix: correct hash_size detection for transport routes and zero-hop adverts (#747)
## Summary

Fixes #744
Fixes #722

Three bugs in hash_size computation caused zero-hop adverts to
incorrectly report `hash_size=1`, masking nodes that actually use
multi-byte hashes.

## Bugs Fixed

### 1. Wrong path byte offset for transport routes
(`computeNodeHashSizeInfo`)

Transport routes (types 0 and 3) have 4 transport code bytes before the
path byte. The code read the path byte from offset 1 (byte index
`RawHex[2:4]`) for all route types. For transport routes, the correct
offset is 5 (`RawHex[10:12]`).

### 2. Missing RouteTransportDirect skip (`computeNodeHashSizeInfo`)

Zero-hop adverts from `RouteDirect` (type 2) were correctly skipped, but
`RouteTransportDirect` (type 3) zero-hop adverts were not. Both have
locally-generated path bytes with unreliable hash_size bits.

### 3. Zero-hop adverts not skipped in analytics
(`computeAnalyticsHashSizes`)

`computeAnalyticsHashSizes()` unconditionally overwrote a node's
`hashSize` with whatever the latest advert reported. A zero-hop direct
advert with `hash_size=1` could overwrite a previously-correct
`hash_size=2` from a multi-hop flood advert.

Fix: skip hash_size update for zero-hop direct/transport-direct adverts
while still counting the packet and updating `lastSeen`.

## Tests Added

- `TestHashSizeTransportRoutePathByteOffset` — verifies transport routes
read path byte at offset 5, regular flood reads at offset 1
- `TestHashSizeTransportDirectZeroHopSkipped` — verifies both
RouteDirect and RouteTransportDirect zero-hop adverts are skipped
- `TestAnalyticsHashSizesZeroHopSkip` — verifies analytics hash_size is
not overwritten by zero-hop adverts
- Fixed 3 existing tests (`FlipFlop`, `Dominant`, `LatestWins`) that
used route_type 0 (TransportFlood) header bytes without proper transport
code padding

## Complexity

All changes are O(1) per packet — no new loops or data structures. The
additional offset computation and zero-hop check are constant-time
operations within the existing packet scan loop.

Co-authored-by: you <you@example.com>
2026-04-14 23:04:26 -07:00
Kpa-clawbot 2aea01f10c fix: merge repeater+observer into single map marker (#745)
## Problem

When a node acts as both a repeater and an observer (same public key —
common with powered repeaters running MQTT clients), the map shows two
separate markers at the same location: a repeater rectangle and an
observer star. This causes visual clutter and both markers get pushed
out from the real location by the deconfliction algorithm.

## Solution

Detect combined repeater+observer nodes by matching node `public_key`
against observer `id`. When matched:

- **Label mode (hash labels on):** The repeater label gets a gold ★
appended inside the rectangle
- **Icon mode (hash labels off):** The repeater diamond gets a small
star overlay in the top-right corner of the SVG
- **Popup:** Shows both REPEATER and OBSERVER badges
- **Observer markers:** Skipped when the observer is already represented
as a combined node marker
- **Legend count:** Observer count excludes combined nodes (shows
standalone observers only)

## Performance

- Observer lookup uses a `Map` keyed by lowercase pubkey — O(1) per node
check
- Legend count uses a `Set` of node pubkeys — O(n+m) instead of O(n×m)
- No additional API calls; uses existing `observers` array already
fetched

## Testing

- All 523 frontend helper tests pass
- All 62 packet filter tests pass
- Visual: combined nodes show as single marker with star indicator

Fixes #719

---------

Co-authored-by: you <you@example.com>
2026-04-14 22:47:28 -07:00
efiten b7c2cb070c docs: geofilter manual + config.example.json entry (#734)
## Summary

- Add missing `geo_filter` block to `config.example.json` with polygon
example, `bufferKm`, and inline `_comment`
- Add `docs/user-guide/geofilter.md`: full operator guide covering
config schema, GeoFilter Builder workflow, and prune script as one-time
migration tool
- Add Geographic filtering section to `docs/user-guide/configuration.md`
with link to the full guide

Closes #669 (M1: documentation)

## Test plan

- [x] `config.example.json` parses cleanly (no JSON errors)
- [x] `docs/user-guide/geofilter.md` renders correctly in GitHub preview
- [x] Link from `configuration.md` to `geofilter.md` resolves

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:43:19 -07:00
efiten 1de80a9eaf feat: serve geofilter builder from app, link from customizer (#735)
## Summary

Part of #669 — M2: Link the builder from the app.

- **`public/geofilter-builder.html`** — the existing
`tools/geofilter-builder.html` is now served by the static file server
at `/geofilter-builder.html`. Additions vs the original: a `← CoreScope`
back-link in the header, inline code comments explaining the output
format, and a help bar below the output panel with paste instructions
and a link to the documentation.
- **`public/customize-v2.js`** — adds a "Tools" section at the bottom of
the Export tab with a `🗺️ GeoFilter Builder →` link and a one-line
description.
- **`docs/user-guide/customization.md`** — documents the new GeoFilter
Builder entry in the Export tab.

> **Note:** `tools/geofilter-builder.html` is kept as-is for
local/offline use. The `public/` copy is what the server serves.

> **Depends on:** #734 (M1 docs) for `docs/user-guide/geofilter.md` —
the link in the help bar references that file. Can be merged
independently; the link still works once M1 lands.

## Test plan

- [x] Open the app, go to Customizer → Export tab — "Tools" section
appears with GeoFilter Builder link
- [x] Click the link — opens `/geofilter-builder.html` in a new tab
- [x] Builder loads the Leaflet map, draw 3+ points — JSON output
appears
- [x] Copy button works, output is valid `{ "geo_filter": { ... } }`
JSON
- [x] `← CoreScope` back-link navigates to `/`
- [x] Help bar shows paste instructions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:42:27 -07:00
efiten e6ace95059 fix: desktop node click updates URL hash, deep link opens split panel (#676) (#739)
## Problem

Clicking a node on desktop opened the side panel but never updated the
URL hash, making nodes non-shareable/bookmarkable on desktop. Loading
`#/nodes/{pubkey}` directly on desktop also incorrectly showed the
full-screen mobile view.

## Changes

- `selectNode()` on desktop: adds `history.replaceState(null, '',
'#/nodes/' + pubkey)` so the URL updates on every click
- `init()`: full-screen path is now gated to `window.innerWidth <= 640`
(mobile only); desktop with a `routeParam` falls through to the split
panel and calls `selectNode()` to pre-select the node
- Deselect (Escape / close button): also calls `history.replaceState`
back to `#/nodes`

## Test plan

- [x] Desktop: click a node → URL updates to `#/nodes/{pubkey}`, split
panel opens
- [x] Desktop: copy URL, open in new tab → split panel opens with that
node selected (not full-screen)
- [x] Desktop: press Escape → URL reverts to `#/nodes`
- [x] Mobile (≤640px): clicking a node still navigates to full-screen
view

Closes #676

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:38:06 -07:00
efiten f605d4ce7e fix: serialize filter params in URL hash for deep linking (#682) (#740)
## Problem

Applying packet filters (hash, node, observer, Wireshark expression) did
not update the URL hash, so filtered views could not be shared or
bookmarked.

## Changes

**`buildPacketsQuery()`** — extended to include:
- `hash=` from `filters.hash`
- `node=` from `filters.node`  
- `observer=` from `filters.observer`
- `filter=` from `filters._filterExpr` (Wireshark expression string)

**`updatePacketsUrl()`** — now called on every filter change:
- hash input (debounced)
- observer multi-select change
- node autocomplete select and clear
- Wireshark filter input (on valid expression or clear)

**URL restore on load** — `getHashParams()` now reads `hash`, `node`,
`observer`, `filter` and restores them into `filters` before the DOM is
built. Input fields pick up values from `filters` as before. Wireshark
expression is also recompiled and `filter-active` class applied.

## Test plan

- [ ] Type in hash filter → URL updates with `&hash=...`
- [ ] Copy URL, open in new tab → hash filter is pre-filled
- [ ] Select an observer → URL updates with `&observer=...`
- [ ] Select a node filter → URL updates with `&node=...`
- [ ] Type `type=ADVERT` in Wireshark filter → URL updates with
`&filter=type%3DADVERT`
- [ ] Load that URL → filter expression restored and active

Closes #682

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:37:33 -07:00
Kpa-clawbot 84f03f4f41 fix: hide undecryptable channel messages by default (#727) (#728)
## Problem

Channels page shows 53K 'Unknown' messages — undecryptable GRP_TXT
packets with no content. Pure noise.

## Fix

- Backend: channels API filters out undecrypted messages by default
- `?includeEncrypted=true` param to include them
- Frontend: 'Show encrypted' toggle in channels sidebar
- Unknown channels grayed out with '(no key)' label
- Toggle persists in localStorage

Fixes #727

---------

Co-authored-by: you <you@example.com>
2026-04-13 19:40:20 +00:00
Kpa-clawbot 8158631d02 feat: client-side channel decryption — add custom channels in browser (#725 M2) (#733)
## Summary

Pure client-side channel decryption. Users can add custom hashtag
channels or PSK channels directly in the browser. **The server never
sees the keys.**

Implements #725 M2 (revised). Does NOT close #725.

## How it works

1. User types `#channelname` or pastes a hex PSK in the channels sidebar
2. Browser derives key (`SHA256("#name")[:16]`) using Web Crypto API
3. Key stored in **localStorage** — never sent to the server
4. Browser fetches encrypted GRP_TXT packets via existing API
5. Browser decrypts client-side: AES-128-ECB + HMAC-SHA256 MAC
verification
6. Decrypted messages cached in localStorage
7. Progressive rendering — newest messages first, chunk-based

## Security

- Keys never leave the browser
- No new API endpoints
- No server-side changes whatsoever
- Channel interest partially observable via hash-based API requests
(documented, acceptable tradeoff)

## Changes

- `public/channels.js` — client-side decrypt module + UI integration
(+307 lines)
- `public/index.html` — no new script (inline in channels.js IIFE)
- `public/style.css` — add-channel input styling

---------

Co-authored-by: you <you@example.com>
2026-04-13 12:28:41 -07:00
Kpa-clawbot 14367488e2 fix: TRACE path_json uses path_sz from flags byte, not header hash_size (#732)
## Summary

TRACE packets encode their route hash size in the flags byte (`flags &
0x03`), not the header path byte. The decoder was using `path.HashSize`
from the header, which could be wrong or zero for direct-route TRACEs,
producing incorrect hop counts in `path_json`.

## Protocol Note

Per firmware, TRACE packets are **always direct-routed** (route_type 2 =
DIRECT, or 3 = TRANSPORT_DIRECT). FLOOD-routed TRACEs (route_type 1) are
anomalous — firmware explicitly rejects TRACE via flood. The decoder
handles these gracefully without crashing.

## Changes

**`cmd/server/decoder.go` and `cmd/ingestor/decoder.go`:**
- Read `pathSz` from TRACE flags byte: `(traceFlags & 0x03) + 1`
(0→1byte, 1→2byte, 2→3byte)
- Use `pathSz` instead of `path.HashSize` for splitting TRACE payload
path data into hops
- Update `path.HashSize` to reflect the actual TRACE path size
- Added `HopsCompleted` field to ingestor `Path` struct for parity with
server
- Updated comments to clarify TRACE is always direct-routed per firmware

**`cmd/server/decoder_test.go` — 5 new tests:**
- `TraceFlags1_TwoBytePathSz`: flags=1 → 2-byte hashes via DIRECT route
- `TraceFlags2_ThreeBytePathSz`: flags=2 → 3-byte hashes via DIRECT
route
- `TracePathSzUnevenPayload`: payload not evenly divisible by path_sz
- `TraceTransportDirect`: route_type=3 with transport codes + TRACE path
parsing
- `TraceFloodRouteGraceful`: anomalous FLOOD+TRACE handled without crash

All existing TRACE tests (flags=0, 1-byte hashes) continue to pass.

Fixes #731

---------

Co-authored-by: you <you@example.com>
2026-04-13 08:20:09 -07:00
Kpa-clawbot 71be54f085 feat: DB-backed channel messages for full history (#725 M1) (#726)
## Summary

Switches channel API endpoints to query SQLite instead of the in-memory
packet store, giving users access to the full message history.

Implements #725 (M1 only — DB-backed channel messages). Does NOT close
#725 — M2-M5 (custom channels, PSK, persistence, retroactive decryption)
remain.

## Problem

Channel endpoints (`/api/channels`, `/api/channels/{hash}/messages`)
preferred the in-memory packet store when available. The store is
bounded by `packetStore.maxMemoryMB` — typically showing only recent
messages. The SQLite database has the complete history (weeks/months of
channel messages) but was only used as a fallback when the store was nil
(never in production).

## Fix

Reversed the preference order: DB first, in-memory store fallback.
Region filtering added to the DB path.

Co-authored-by: you <you@example.com>
2026-04-12 23:22:52 -07:00
Kpa-clawbot c233c14156 feat: CLI tool to decrypt and export hashtag channel messages (#724)
## Summary

Adds `corescope-decrypt` — a standalone CLI tool that decrypts and
exports MeshCore hashtag channel messages from a CoreScope SQLite
database.

### What it does

MeshCore hashtag channels use symmetric encryption with keys derived
from the channel name. The CoreScope ingestor stores **all** GRP_TXT
packets, even those it can't decrypt. This tool enables retroactive
decryption — decrypt historical messages for any channel whose name you
learn after the fact.

### Architecture

- **`internal/channel/`** — Shared crypto package extracted from
ingestor logic:
  - `DeriveKey()` — `SHA-256("#name")[:16]`
  - `ChannelHash()` — 1-byte packet filter (`SHA-256(key)[0]`)
  - `Decrypt()` — HMAC-SHA256 MAC verify + AES-128-ECB
  - `ParsePlaintext()` — timestamp + flags + "sender: message" parsing

- **`cmd/decrypt/`** — CLI binary with three output formats:
  - `--format json` — Full metadata (observers, path, raw hex)
  - `--format html` — Self-contained interactive viewer with search/sort
  - `--format irc` (or `log`) — Plain-text IRC-style log, greppable

### Usage

```bash
# JSON export
corescope-decrypt --channel "#wardriving" --db meshcore.db

# Interactive HTML viewer
corescope-decrypt --channel wardriving --db meshcore.db --format html --output wardriving.html

# Greppable log
corescope-decrypt --channel "#wardriving" --db meshcore.db --format irc | grep "KE6QR"

# From Docker
docker exec corescope-prod /app/corescope-decrypt --channel "#wardriving" --db /app/data/meshcore.db
```

### Build & deployment

- Statically linked (`CGO_ENABLED=0`) — zero dependencies
- Added to Dockerfile (available at `/app/corescope-decrypt` in
container)
- CI: builds and tests in go-test job
- CI: attaches linux/amd64 and linux/arm64 binaries to GitHub Releases
on tags

### Testing

- `internal/channel/` — 9 tests: key derivation, encrypt/decrypt
round-trip, MAC rejection, wrong-channel rejection, plaintext parsing
- `cmd/decrypt/` — 7 tests: payload extraction, channel hash
consistency, all 3 output formats, JSON parseability, fixture DB
integration
- Verified against real fixture DB: successfully decrypts 17
`#wardriving` messages

### Limitations

- Hashtag channels only (name-derived keys). Custom PSK channels not
supported.
- No DM decryption (asymmetric, per-peer keys).
- Read-only database access.

Fixes #723

---------

Co-authored-by: you <you@example.com>
2026-04-12 22:07:41 -07:00
Kpa-clawbot 65482ff6f6 fix: cache invalidation tuning — 7% → 50-80% hit rate (#721)
## Cache Invalidation Tuning — 7% → 50-80% Hit Rate

Fixes #720

### Problem

Server-side cache hit rate was 7% (48 hits / 631 misses over 4.7 days).
Root causes from the [cache audit
report](https://github.com/Kpa-clawbot/CoreScope/issues/720):

1. **`invalidationDebounce` config value (30s) was dead code** — never
wired to `invCooldown`
2. **`invCooldown` hardcoded to 10s** — with continuous ingest, caches
cleared every 10s regardless of their 1800s TTLs
3. **`collisionCache` cleared on every `hasNewTransmissions`** — hash
collisions are structural (depend on node count), not per-packet

### Changes

| Change | File | Impact |
|--------|------|--------|
| Wire `invalidationDebounce` from config → `invCooldown` | `store.go` |
Config actually works now |
| Default `invCooldown` 10s → 300s (5 min) | `store.go` | 30x longer
cache survival |
| Add `hasNewNodes` flag to `cacheInvalidation` | `store.go` |
Finer-grained invalidation |
| `collisionCache` only clears on `hasNewNodes` | `store.go` | O(n²)
collision computation survives its 1hr TTL |
| `addToByNode` returns new-node indicator | `store.go` | Zero-cost
detection during indexing |
| `indexByNode` returns new-node indicator | `store.go` | Propagates to
ingest path |
| Ingest tracks and passes `hasNewNodes` | `store.go` | End-to-end
wiring |

### Tests Added

| Test | What it verifies |
|------|-----------------|
| `TestInvCooldownFromConfig` | Config value wired to `invCooldown`;
default is 300s |
| `TestCollisionCacheNotClearedByTransmissions` | `hasNewTransmissions`
alone does NOT clear `collisionCache` |
| `TestCollisionCacheClearedByNewNodes` | `hasNewNodes` DOES clear
`collisionCache` |
| `TestCacheSurvivesMultipleIngestCyclesWithinCooldown` | 5 rapid ingest
cycles don't clear any caches during cooldown |
| `TestNewNodesAccumulatedDuringCooldown` | `hasNewNodes` accumulated in
`pendingInv` and applied after cooldown |
| `BenchmarkAnalyticsLatencyCacheHitVsMiss` | 100% hit rate with
rate-limited invalidation |

All 200+ existing tests pass. Both benchmarks show 100% hit rate.

### Performance Justification

- **Before:** Effective cache lifetime = `min(TTL, invCooldown)` = 10s.
With analytics viewed ~once/few minutes, P(hit) ≈ 7%
- **After:** Effective cache lifetime = `min(TTL, 300s)` = 300s for most
caches, 3600s for `collisionCache`. Expected hit rate 50-80%
- **Complexity:** All changes are O(1) — `addToByNode` already checked
`nodeHashes[pubkey] == nil`, we just return the result
- **Benchmark proof:** `BenchmarkAnalyticsLatencyCacheHitVsMiss` → 100%
hit rate, 269ns/op

Co-authored-by: you <you@example.com>
2026-04-12 18:09:23 -07:00
Kpa-clawbot 7af91f7ef6 fix: perf page shows tracked memory instead of heap allocation (#718)
## Summary

The perf page "Memory Used" tile displayed `estimatedMB` (Go
`runtime.HeapAlloc`), which includes all Go runtime allocations — not
just packet store data. This made the displayed value misleading: it
showed ~2.4GB heap when only ~833MB was actual tracked packet data.

## Changes

### Frontend (`public/perf.js`)
- Primary tile now shows `trackedMB` as **"Tracked Memory"** — the
self-accounted packet store memory
- Added separate **"Heap (debug)"** tile showing `estimatedMB` for
runtime visibility

### Backend
- **`types.go`**: Added `TrackedMB` field to `HealthPacketStoreStats`
struct
- **`routes.go`**: Populate `TrackedMB` in `/health` endpoint response
from `GetPerfStoreStatsTyped()`
- **`routes_test.go`**: Assert `trackedMB` exists in health endpoint's
`packetStore`
- **`testdata/golden/shapes.json`**: Updated shape fixture with new
field

### What was already correct
- `/api/perf/stats` already exposed both `estimatedMB` and `trackedMB`
- `trackedMemoryMB()` method already existed in store.go
- Eviction logic already used `trackedBytes` (not HeapAlloc)

## Testing
- All Go tests pass (`go test ./... -count=1`)
- No frontend logic changes beyond template string field swap

Fixes #717

Co-authored-by: you <you@example.com>
2026-04-12 12:40:17 -07:00
Kpa-clawbot f95aa49804 fix: exclude TRACE packets from multi-byte capability suspected detection (#715)
## Summary

Exclude TRACE packets (payload_type 8) from the "suspected" multi-byte
capability inference logic. TRACE packets carry hash size in their own
flags — forwarding repeaters read it from the TRACE header, not their
compile-time `PATH_HASH_SIZE`. Pre-1.14 repeaters can forward multi-byte
TRACEs without actually supporting multi-byte hashes, creating false
positives.

Fixes #714

## Changes

### `cmd/server/store.go`
- In `computeMultiByteCapability()`, skip packets with `payload_type ==
8` (TRACE) when scanning `byPathHop` for suspected multi-byte nodes
- "Confirmed" detection (from adverts) is unaffected

### `cmd/server/multibyte_capability_test.go`
- `TestMultiByteCapability_TraceExcluded`: TRACE packet with 2-byte path
does NOT mark repeater as suspected
- `TestMultiByteCapability_NonTraceStillSuspected`: Non-TRACE packet
with 2-byte path still marks as suspected
- `TestMultiByteCapability_ConfirmedUnaffectedByTraceExclusion`:
Confirmed status from advert unaffected by TRACE exclusion

## Testing

All 7 multi-byte capability tests pass. Full `cmd/server` and
`cmd/ingestor` test suites pass.

Co-authored-by: you <you@example.com>
2026-04-12 00:11:20 -07:00
Kpa-clawbot 45623672d9 fix: integrate multi-byte capability into adopters table, fix filter buttons (#712) (#713)
## Summary

Fixes #712 — Multi-byte capability filter buttons broken + needs
integration with Hash Adopters.

### Changes

**M1: Fix filter buttons breaking after first click**
- Root cause: `section.replaceWith(newSection)` replaced the entire DOM
node, but the event listener was attached to the old node. After
replacement, clicks went unhandled.
- Fix: Instead of replacing the whole section, only swap the table
content inside a stable `#mbAdoptersTableWrap` div. The event listener
on `#mbAdoptersSection` persists across filter changes.
- Button active state is now toggled via `classList.toggle` instead of
full DOM rebuild.

**M2: Better button labels**
- Changed from icon-only (` 76`) to descriptive labels: ` Confirmed
(76)`, `⚠️ Suspected (81)`, ` Unknown (223)`

**M3: Integrate with Multi-Byte Hash Adopters**
- Merged capability status into the existing adopters table as a new
"Status" column
- Removed the separate "Repeater Multi-Byte Capability" section
- Filter buttons now apply to the integrated table
- Nodes without capability data default to  Unknown
- Capability data is looked up by pubkey from the existing
`multiByteCapability` API response (no backend changes needed)

### Performance

- No new API calls — capability data already exists in the hash sizes
response
- Filter toggle is O(n) where n = number of adopter nodes (typically
<500)
- Event delegation on stable parent — no listener re-attachment needed

### Tests

- Updated existing `renderMultiByteCapability` tests for new label
format
- Added 5 new tests for `renderMultiByteAdopters`: empty state, status
integration, text labels with counts, unknown default, Status column
presence
- All 507 frontend tests pass, all Go tests pass

Co-authored-by: you <you@example.com>
2026-04-11 23:07:44 -07:00
Kpa-clawbot 4a7e20a8cb fix: redesign memory eviction — self-accounting trackedBytes, watermarks, safety cap (#711)
## Problem

`HeapAlloc`-based eviction cascades on large databases — evicts down to
near-zero packets because Go runtime overhead exceeds `maxMemoryMB` even
with an empty packet store.

## Fix (per Carmack spec on #710)

1. **Self-accounting `trackedBytes`** — running counter maintained on
insert/evict, computed from actual struct sizes. No
`runtime.ReadMemStats`.
2. **High/low watermark hysteresis** (100%/85%) — evict to 85% of
budget, don't re-trigger until 100% crossed again.
3. **25% per-pass safety cap** — never evict more than a quarter of
packets in one cycle.
4. **Oldest-first** — evict from sorted head, O(1) candidate selection.

`maxMemoryMB` now means packet store budget, not total process heap.

Fixes #710

Co-authored-by: you <you@example.com>
2026-04-11 23:06:48 -07:00
Kpa-clawbot 7e0b904d09 fix: refresh live feed relative timestamps every 10s (#709)
## Summary

Fixes #701 — Live feed timestamps showed stale relative times (e.g. "2s
ago" never updated to "5m ago").

## Root Cause

`formatLiveTimestampHtml()` was called once when each feed item was
created and never refreshed. The dedup path (when a duplicate hash moves
an item to the top) also didn't update the timestamp.

## Changes

### `public/live.js`
- **`data-ts` attribute on `.feed-time` spans**: All three feed item
creation paths (VCR replay, `addFeedItemDOM`, `addFeedItem`) now store
the packet timestamp as `data-ts` on the `.feed-time` span element
- **10-second refresh interval**: A `setInterval` queries all
`.feed-time[data-ts]` elements and re-renders their content via
`formatLiveTimestampHtml()`, keeping relative times accurate
- **Dedup path timestamp update**: When a duplicate hash observation
moves an existing feed item to the top, the `.feed-time` span is updated
with the new observation's timestamp
- **Cleanup**: The interval is cleared on page teardown alongside other
intervals

### `test-live.js`
- 3 new tests: formatting idempotency, numeric timestamp acceptance,
`data-ts` round-trip correctness

## Performance

- The refresh interval runs every 10s, iterating over at most 25
`.feed-time` DOM elements (feed is capped at 25 items via `while
(feed.children.length > 25)`). Negligible overhead.
- Uses `querySelectorAll` with attribute selector — O(n) where n ≤ 25.

## Testing

- All 3 new tests pass
- All pre-existing test suites pass (70 live.js tests, 62 packet-filter,
501 frontend-helpers)
- 8 pre-existing failures in `test-live.js` are unrelated
(`getParsedDecoded` missing from sandbox)

Co-authored-by: you <you@example.com>
2026-04-11 21:30:38 -07:00
Kpa-clawbot e893a1b3c4 fix: index relay hops in byNode for liveness tracking (#708)
## Problem

Nodes that only appear as relay hops in packet paths (via
`resolved_path`) were never indexed in `byNode`, so `last_heard` was
never computed for them. This made relay-only nodes show as dead/stale
even when actively forwarding traffic.

Fixes #660

## Root Cause

`indexByNode()` only indexed pubkeys from decoded JSON fields (`pubKey`,
`destPubKey`, `srcPubKey`). Relay nodes appearing in `resolved_path`
were ignored entirely.

## Fix

`indexByNode()` now also iterates:
1. `ResolvedPath` entries from each observation
2. `tx.ResolvedPath` (best observation's resolved path, used for
DB-loaded packets)

A per-call `indexed` set prevents double-indexing when the same pubkey
appears in both decoded JSON and resolved path.

Extracted `addToByNode()` helper to deduplicate the nodeHashes/byNode
append logic.

## Scope

**Phase 1 only** — server-side in-memory indexing. No DB changes, no
ingestor changes. This makes `last_heard` reflect relay activity with
zero risk to persistence.

## Tests

5 new test cases in `TestIndexByNodeResolvedPath`:
- Resolved path pubkeys from observations get indexed
- Null entries in resolved path are skipped
- Relay-only nodes (no decoded JSON match) appear in `byNode`
- Dedup between decoded JSON and resolved path
- `tx.ResolvedPath` indexed when observations are empty

All existing tests pass unchanged.

## Complexity

O(observations × path_length) per packet — typically 1-3 observations ×
1-3 hops. No hot-path regression.

---------

Co-authored-by: you <you@example.com>
2026-04-11 21:25:42 -07:00
Kpa-clawbot fcba2a9f3d fix: set PRAGMA busy_timeout on all RW SQLite connections (#707)
## Problem

`SQLITE_BUSY` contention between the ingestor and server's async
persistence goroutine drops `resolved_path` and `neighbor_edges`
updates. The DSN parameter `_busy_timeout=10000` may not be honored by
the modernc/sqlite driver.

## Fix

- **`openRW()` now sets `PRAGMA busy_timeout = 5000`** after opening the
connection, guaranteeing SQLite retries for up to 5 seconds before
returning `SQLITE_BUSY`
- **Refactored `PruneOldPackets` and `PruneOldMetrics`** to use
`openRW()` instead of duplicating connection setup — all RW connections
now get consistent busy_timeout handling
- Added test verifying the pragma is set correctly

## Changes

| File | Change |
|------|--------|
| `cmd/server/neighbor_persist.go` | `openRW()` sets `PRAGMA
busy_timeout = 5000` after open |
| `cmd/server/db.go` | `PruneOldPackets` and `PruneOldMetrics` use
`openRW()` instead of inline `sql.Open` |
| `cmd/server/neighbor_persist_test.go` | `TestOpenRW_BusyTimeout`
verifies pragma is set |

## Performance

No performance impact — `PRAGMA busy_timeout` is a connection-level
setting with zero overhead on uncontended writes. Under contention, it
converts immediate `SQLITE_BUSY` failures into brief retries (up to 5s),
which is strictly better than dropping data.

Fixes #705

---------

Co-authored-by: you <you@example.com>
2026-04-11 21:25:23 -07:00
you c6a0f91b07 fix: add internal/sigvalidate to Dockerfile for both server and ingestor builds
PR #686 added internal/sigvalidate/ with replace directives in both
go.mod files but didn't update the Dockerfile to COPY it into the
Docker build context. go mod download fails with 'no such file'.
2026-04-12 04:14:56 +00:00
Kpa-clawbot ef8bce5002 feat: repeater multi-byte capability inference table (#706)
## Summary

Adds a new "Repeater Multi-Byte Capability" section to the Hash Stats
analytics tab that classifies each repeater's ability to handle
multi-byte hash prefixes (firmware >= v1.14).

Fixes #689

## What Changed

### Backend (`cmd/server/store.go`)
- New `computeMultiByteCapability()` method that infers capability for
each repeater using two evidence sources:
- **Confirmed** (100% reliable): node has advertised with `hash_size >=
2`, leveraging existing `computeNodeHashSizeInfo()` data
- **Suspected** (<100%): node's prefix appears as a hop in packets with
multi-byte path headers, using the `byPathHop` index. Prefix collisions
mean this isn't definitive.
- **Unknown**: no multi-byte evidence — could be pre-1.14 or 1.14+ with
default settings
- Extended `/api/analytics/hash-sizes` response with
`multiByteCapability` array

### Frontend (`public/analytics.js`)
- New `renderMultiByteCapability()` function on the Hash Stats tab
- Color-coded table: green confirmed, yellow suspected, gray unknown
- Filter buttons to show all/confirmed/suspected/unknown
- Column sorting by name, role, status, evidence, max hash size, last
seen
- Clickable rows link to node detail pages

### Tests (`cmd/server/multibyte_capability_test.go`)
- `TestMultiByteCapability_Confirmed`: advert with hash_size=2 →
confirmed
- `TestMultiByteCapability_Suspected`: path appearance only → suspected
- `TestMultiByteCapability_Unknown`: 1-byte advert only → unknown
- `TestMultiByteCapability_PrefixCollision`: two nodes sharing prefix,
one confirmed via advert, other correctly marked suspected (not
confirmed)

## Performance

- `computeMultiByteCapability()` runs once per cache cycle (15s TTL via
hash-sizes cache)
- Leverages existing `GetNodeHashSizeInfo()` cache (also 15s TTL) — no
redundant advert scanning
- Path hop scan is O(repeaters × prefix lengths) lookups in the
`byPathHop` map, with early break on first match per prefix
- Only computed for global (non-regional) requests to avoid unnecessary
work

---------

Co-authored-by: you <you@example.com>
2026-04-11 21:02:54 -07:00
copelaje 922ebe54e7 BYOP Advert signature validation (#686)
For BYOP mode in the packet analyzer, perform signature validation on
advert packets and display whether successful or not. This is added as
we observed many corrupted advert packets that would be easily
detectable as such if signature validation checks were performed.

At present this MR is just to add this status in BYOP mode so there is
minimal impact to the application and no performance penalty for having
to perform these checks on all packets. Moving forward it probably makes
sense to do these checks on all advert packets so that corrupt packets
can be ignored in several contexts (like node lists for example).

Let me know what you think and I can adjust as needed.

---------

Co-authored-by: you <you@example.com>
2026-04-12 04:02:17 +00:00
Kpa-clawbot 26c47df814 fix: entrypoint .env support + deployment docs for bare docker run (#704)
## Summary

Fixes #702 — `.env` file `DISABLE_MOSQUITTO`/`DISABLE_CADDY` ignored
when using `docker run`.

## Changes

### Entrypoint sources `/app/data/.env`
The entrypoint now sources `/app/data/.env` (if present) before the
`DISABLE_*` checks. This works regardless of how the container is
started — `docker run`, compose, or `manage.sh`.

```bash
if [ -f /app/data/.env ]; then
  set -a
  . /app/data/.env
  set +a
fi
```

### `DISABLE_CADDY` added to compose files
Both `docker-compose.yml` and `docker-compose.staging.yml` now forward
`DISABLE_CADDY` to the container environment (was missing — only
`DISABLE_MOSQUITTO` was wired).

### Deployment docs updated
- `docs/deployment.md`: bare `docker run` is now the primary/recommended
approach with a full parameter reference table
- Documents the `/app/data/.env` convenience feature
- Compose and `manage.sh` marked as legacy alternatives
- `DISABLE_CADDY` added to the environment variable reference

### README quick start updated
Shows the full `docker run` command with `--restart`, ports, and
volumes. Includes HTTPS variant. Documents `-e` flags and `.env` file.

### v3.5.0 release notes
Updated the env var documentation to mention the `.env` file support.

## Testing
- All Go server tests pass
- All Go ingestor tests pass
- No logic changes to Go code — entrypoint shell script + docs only

---------

Co-authored-by: you <you@example.com>
2026-04-11 20:43:16 -07:00
Kpa-clawbot bc22dbdb14 feat: DragManager — core drag mechanics (#608 M1) (#697)
## Summary

Implements M1 of the draggable panels spec from #608: the `DragManager`
class with core drag mechanics.

Fixes #608 (M1: DragManager core drag mechanics)

## What's New

### `public/drag-manager.js` (~215 lines)
- **State machine:** `IDLE → PENDING → DRAGGING → IDLE`
- **5px dead zone** on `.panel-header` to disambiguate click vs drag —
prevents hijacking corner toggle and close button clicks
- **Pointer events** with `setPointerCapture` for reliable tracking
- **`transform: translate()`** during drag — zero layout reflow
- **Snap-to-edge** on release: 20px threshold snaps to 12px margin
- **Z-index management** — dragged panel comes to front (counter from
1000)
- **`_detachFromCorner()`** — transitions panel from M0 corner CSS to
fixed positioning
- **Escape key** cancels drag and reverts to pre-drag position
- **`restorePositions()`** — applies saved viewport percentages on init
- **`handleResize()`** — clamps dragged panels inside viewport on window
resize
- **`enable()`/`disable()`** — responsive gate control

### `public/live.js` integration
- Instantiates `DragManager` after `initPanelPositions()`
- Registers `liveFeed`, `liveLegend`, `liveNodeDetail` panels
- **Responsive gate:** `matchMedia('(pointer: fine) and (min-width:
768px)')` — disables drag on touch/small screens, reverts to M0 corner
toggle
- **Resize clamping** debounced at 200ms

### `public/live.css` additions
- `cursor: grab/grabbing` on `.panel-header` (desktop only via `@media
(pointer: fine)`)
- `.is-dragging` class: opacity 0.92, elevated box-shadow, `will-change:
transform`, transitions disabled
- `[data-dragged="true"]` disables corner transition animations
- `prefers-reduced-motion` support

### Persistence
- **Format:** `panel-drag-{id}` → `{ xPct, yPct }` (viewport
percentages)
- **Survives resize:** positions recalculated from percentages
- **Corner toggle still works:** clicking corner button after drag
clears drag state (handled by existing M0 code)

## Tests

14 new unit tests in `test-drag-manager.js`:
- State machine transitions (IDLE → PENDING → DRAGGING → IDLE)
- Dead zone enforcement
- Button click guard (no drag on button pointerdown)
- Snap-to-edge behavior
- Position persistence as viewport percentages
- Restore from localStorage
- Resize clamping
- Disable/enable

## Performance

- `transform: translate()` during drag — compositor-only, no layout
reflow
- `will-change: transform` only during active drag (`.is-dragging`),
removed on drop
- `localStorage` write only on `pointerup`, never during `pointermove`
- Resize handler debounced at 200ms
- Single `style.transform` assignment per pointermove frame — negligible
cost

---------

Co-authored-by: you <you@example.com>
2026-04-11 20:41:35 -07:00
Kpa-clawbot 9917d50622 fix: resolve neighbor graph duplicate entries from different prefix lengths (#699)
## Problem

The neighbor graph creates separate entries for the same physical node
when observed with different prefix lengths. For example, a 1-byte
prefix `B0` (ambiguous, unresolved) and a 2-byte prefix `B05B` (resolved
to Busbee) appear as two separate neighbors of the same node.

Fixes #698

## Solution

### Part 1: Post-build resolution pass (Phase 1.5)

New function `resolveAmbiguousEdges(pm, graph)` in `neighbor_graph.go`:
- Called after `BuildFromStore()` completes the full graph, before any
API use
- Iterates all ambiguous edges and attempts resolution via
`resolveWithContext` with full graph context
- Only accepts high-confidence resolutions (`neighbor_affinity`,
`geo_proximity`, `unique_prefix`) — rejects
`first_match`/`gps_preference` fallbacks to avoid false positives
- Merges with existing resolved edges (count accumulation, max LastSeen)
or updates in-place
- Phase 1 edge collection loop is **unchanged**

### Part 2: API-layer dedup (defense-in-depth)

New function `dedupPrefixEntries()` in `neighbor_api.go`:
- Scans neighbor response for unresolved prefix entries matching
resolved pubkey entries
- Merges counts, timestamps, and observers; removes the unresolved entry
- O(n²) on ~50 neighbors per node — negligible cost

### Performance

Phase 1.5 runs O(ambiguous_edges × candidates). Per Carmack's analysis:
~50ms at 2K nodes on the 5-min rebuild cycle. Hot ingest path untouched.

## Tests

9 new tests in `neighbor_dedup_test.go`:

1. **Geo proximity resolution** — ambiguous edge resolved when candidate
has GPS near context node
2. **Merge with existing** — ambiguous edge merged into existing
resolved edge (count accumulation)
3. **No-match preservation** — ambiguous edge left as-is when prefix has
no candidates
4. **API dedup** — unresolved prefix merged with resolved pubkey in
response
5. **Integration** — node with both 1-byte and 2-byte prefix
observations shows single neighbor entry
6. **Phase 1 regression** — non-ambiguous edge collection unchanged
7. **LastSeen preservation** — merge keeps higher LastSeen timestamp
8. **No-match dedup** — API dedup doesn't merge non-matching prefixes
9. **Benchmark** — Phase 1.5 with 500+ edges

All existing tests pass (server + ingestor).

---------

Co-authored-by: you <you@example.com>
2026-04-10 11:19:54 -07:00
Kpa-clawbot 2e1a4a2e0d fix: handle companion nodes without adverts in My Mesh health cards (#696)
## Summary

Fixes #665 — companion nodes claimed in "My Mesh" showed "Could not load
data" because they never sent an advert, so they had no `nodes` table
entry, causing the health API to return 404.

## Three-Layer Fix

### 1. API Resilience (`cmd/server/store.go`)
`GetNodeHealth()` now falls back to building a partial response from the
in-memory packet store when `GetNodeByPubkey()` returns nil. Returns a
synthetic node stub (`role: "unknown"`, `name: "Unknown"`) with whatever
stats exist from packets, instead of returning nil → 404.

### 2. Ingestor Cleanup (`cmd/ingestor/main.go`)
Removed phantom sender node creation that used `"sender-" + name` as the
pubkey. Channel messages don't carry the sender's real pubkey, so these
synthetic entries were unreachable from the claiming/health flow — they
just polluted the nodes table with unmatchable keys.

### 3. Frontend UX (`public/home.js`)
The catch block in `loadMyNodes()` now distinguishes 404 (node not in DB
yet) from other errors:
- **404**: Shows 📡 "Waiting for first advert — this node has been seen
in channel messages but hasn't advertised yet"
- **Other errors**: Shows  "Could not load data" (unchanged)

## Tests
- Added `TestNodeHealthPartialFromPackets` — verifies a node with
packets but no DB entry returns 200 with synthetic node stub and stats
- Updated `TestHandleMessageChannelMessage` — verifies channel messages
no longer create phantom sender nodes
- All existing tests pass (`cmd/server`, `cmd/ingestor`)

Co-authored-by: you <you@example.com>
2026-04-09 20:03:52 -07:00
Kpa-clawbot fcad49594b fix: include path.hopsCompleted in TRACE WebSocket broadcasts (#695)
## Summary

Fixes #683 — TRACE packets on the live map were showing the full path
instead of distinguishing completed vs remaining hops.

## Root Cause

Both WebSocket broadcast builders in `store.go` constructed the
`decoded` map with only `header` and `payload` keys — `path` was never
included. The frontend reads `decoded.path.hopsCompleted` to split trace
routes into solid (completed) and dashed (remaining) segments, but that
field was always `undefined`.

## Fix

For TRACE packets (payload type 9), call `DecodePacket()` on the raw hex
during broadcast and include the resulting `Path` struct in
`decoded["path"]`. This populates `hopsCompleted` which the frontend
already knows how to consume.

Both broadcast builders are patched:
- `IngestNewFromDB()` — new transmissions path (~line 1419)
- `IngestNewObservations()` — new observations path (~line 1680)

TRACE packets are infrequent, so the per-packet decode overhead is
negligible.

## Testing

- Added `TestIngestTraceBroadcastIncludesPath` — verifies that TRACE
broadcast maps include `decoded.path` with correct `hopsCompleted` value
- All existing tests pass (`cmd/server` + `cmd/ingestor`)

Co-authored-by: you <you@example.com>
2026-04-09 20:02:46 -07:00
Kpa-clawbot a1e1e0bd2f fix: bottom-positioned panels overlap VCR bar (#693)
Fixes #685

## Problem

Corner positioning CSS (from PR #608) sets `bottom: 12px` for
bottom-positioned panels (`bl`, `br`), but the VCR bar at the bottom of
the live page is ~50px tall. This causes the legend (and any
bottom-positioned panel) to overlap the VCR controls.

## Fix

Changed `bottom: 12px` → `bottom: 58px` for both
`.live-overlay[data-position="bl"]` and
`.live-overlay[data-position="br"]`, matching the legend's original
`bottom: 58px` value that properly clears the VCR bar.

The VCR bar height is fixed (`.vcr-bar` class with consistent padding),
so a hardcoded value is appropriate here.

## Testing

- All existing tests pass (`npm test` — 13/13)
- CSS-only change, no logic affected

Co-authored-by: you <you@example.com>
2026-04-09 20:02:18 -07:00
efiten 34e7366d7c test: add RouteTransportDirect zero-hop cases to ingestor decoder tests (#684)
## Summary

Closes the symmetry gap flagged as a nit in PR #653 review:

> The ingestor decoder tests omit `RouteTransportDirect` zero-hop tests
— only the server decoder has those. Since the logic is identical, this
is not a blocker, but adding them would make the test suites symmetric.

- Adds `TestZeroHopTransportDirectHashSize` — `pathByte=0x00`, expects
`HashSize=0`
- Adds `TestZeroHopTransportDirectHashSizeWithNonZeroUpperBits` —
`pathByte=0xC0` (hash_size bits set, hash_count=0), expects `HashSize=0`

Both mirror the equivalent tests already present in
`cmd/server/decoder_test.go`.

## Test plan

- [ ] `cd cmd/ingestor && go test -run TestZeroHopTransportDirect -v` →
both new tests pass
- [ ] `cd cmd/ingestor && go test ./...` → no regressions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 17:36:34 -07:00
you 111b03cea1 docs: lead with pre-built Docker image as the headline 2026-04-08 07:22:07 +00:00
you 34c56d203e docs: promote API docs to own section with live analyzer.00id.net links, fix transition section 2026-04-08 07:21:11 +00:00
you cc9f25e5c8 docs: fix release notes — bind mount for caddy-data, no personal paths, add Caddyfile example 2026-04-08 07:20:02 +00:00
you 2e33eb7050 docs: add HTTPS/Caddyfile mount to release notes and upgrade steps 2026-04-08 07:14:15 +00:00
you 6dd0957507 docs: use v3.5.0 tag in release notes, :latest requires git tag 2026-04-08 07:05:58 +00:00
you e22ee3f0ad docs: docker run based upgrade, no compose 2026-04-08 07:03:05 +00:00
you f7f1bb08d0 docs: add cd to compose dir in upgrade steps 2026-04-08 07:01:43 +00:00
you 84da4d962d docs: release notes with juice 2026-04-08 07:00:59 +00:00
you ad0a10c009 docs: fix transition steps — compose-based, not docker run 2026-04-08 06:59:54 +00:00
you c1f268d3b9 docs: add concrete transition steps to release notes 2026-04-08 06:58:39 +00:00
you f5d25f75c6 docs: trim release notes — less book, more changelog 2026-04-08 06:56:50 +00:00
you cde62166cb docs: v3.5.0 release notes + API documentation across README, deployment guide, FAQ
- Release notes for 95 commits since v3.4.1
- OpenAPI/Swagger docs: /api/spec and /api/docs called out everywhere
- Deployment guide: new API Documentation section
- README: API docs link added
- FAQ: 'Where is the API documentation?' entry
- Test plans for v3.4.2 validation
2026-04-08 06:55:25 +00:00
Kpa-clawbot 5606bc639e fix: table sorting broken on all node tables — wrong data attribute (#679) (#680)
## Problem

All table sorting on the Nodes page was broken — clicking column headers
did nothing. Affected:
- Nodes list table
- Node detail → Neighbors table
- Node detail → Observers table

## Root Cause

**Not a race condition** — the actual bug was a **data attribute
mismatch**.

`TableSort.init()` (in `table-sort.js`) queries for `th[data-sort-key]`
to find sortable columns. But all table headers in `nodes.js` used
`data-sort="..."` instead of `data-sort-key="..."`. The selector never
matched any headers, so no click handlers were attached and sorting
silently failed.

Additionally, `data-type="number"` was used but TableSort's built-in
comparator is named `numeric`, causing numeric columns to fall back to
text comparison.

The packets table (`packets.js`) was unaffected because it already used
the correct `data-sort-key` and `data-type="numeric"` attributes.

## Fix

1. **`public/nodes.js`**: Changed all `data-sort="..."` to
`data-sort-key="..."` on `<th>` elements (nodes list, neighbors table,
observers table)
2. **`public/nodes.js`**: Changed `data-type="number"` to
`data-type="numeric"` to match TableSort's comparator names
3. **`public/packets.js`**: Added timestamp tiebreaker to packet sort
for stable ordering when primary column values are equal

## Testing

- All existing tests pass (`npm test`)
- No changes to test infrastructure needed — this was a pure HTML
attribute fix

Fixes #679

---------

Co-authored-by: you <you@example.com>
2026-04-07 23:30:31 -07:00
Kpa-clawbot 1373106b50 Fix panel corner toggle buttons invisible and scrolling away (#678)
## Summary

Panel corner toggle buttons (◫) were invisible due to small size, low
opacity, and `position: absolute` causing them to scroll away with panel
content.

## Changes

### Panel structure — non-scrolling header
All 3 live overlay panels (feed, node detail, legend) now use a flex
layout:
- **`.panel-header`** — non-scrolling row with corner toggle + close
button
- **`.panel-content`** — scrollable content area

### CSS updates
- `.live-overlay`: `display: flex; flex-direction: column`
- `.panel-header`: flex row, `flex-shrink: 0`
- `.panel-content`: `flex: 1; overflow-y: auto`
- `.panel-corner-btn`: removed `position: absolute`, increased to
28×28px, opacity 0.6, hover background

### JS updates
- Feed items now appended to `.panel-content` child instead of panel
root
- `rebuildFeedList` and `addFeedItem` updated to target `.panel-content`
- Resize handle still attaches to panel root (correct behavior)

## Testing
- All 490+ frontend helper tests pass
- All panel-corner tests pass (14/14)
- No test changes needed — tests exercise logic, not DOM structure

Fixes #677

---------

Co-authored-by: you <you@example.com>
2026-04-07 23:17:19 -07:00
Kpa-clawbot 68a4628edf fix: channel color picker — data shape mismatch + redesign for discoverability (#675)
## Fix: Channel Color Picker — Data Shape Mismatch + Redesign (#674)

### Problem

The channel color picker was completely non-functional — dead code.
Three locations in `live.js` attempted to read
`decoded.header.payloadTypeName` and `decoded.payload.channelName`, but:

1. The decoded payload structure is flat
(`decoded.payload.channelHash`), not nested with separate
`header`/`payload` objects within the payload
2. The field is `channelHash` (an integer), not `channelName`
3. `_ccChannel` was **never set** on any DOM element, so all picker
handlers exited early

Additionally, the picker had zero discoverability — hidden behind
right-click/long-press with no visual affordance.

### Changes

**M1 — Fix the data shape bug:**
- Fixed `_ccChannel` assignment in 3 locations in `live.js` to use
`decoded.payload.channelHash` (converted to string)
- Fixed `_getChannelStyle()` to use the same flat structure
- Channel colors now key on the hash string (e.g. `"5"`) matching the
channels API

**M2 — Redesign for discoverability:**
- Reduced palette from 10 to **8 maximally-distinct colors** (removed
teal/rose — too close to cyan/red)
- Removed `<input type="color">` custom picker, "Apply" button, title
bar, close button
- Popover is now just 8 circle swatches + "Clear color" — click outside
to dismiss
- Added **12px clickable color dots** next to channel names on the
channels page (primary configuration surface)
- Unassigned channels show a dashed-border empty circle; assigned show
filled
- Channel list items get `border-left: 3px solid` when colored
- **Removed long-press handler entirely** — dots handle mobile
interaction
- Mobile: bottom-sheet with 36px touch targets via `@media (pointer:
coarse)`

**M3 — Visual encoding:**
- Left border only (3px) — no background tint (per Tufte spec: minimum
effective dose)
- Consistent encoding across live feed items, channel list, packets
table

### Tests

17 new tests in `test-channel-color-picker.js`:
- `_ccChannel` correctly set for GRP_TXT with various `channelHash`
values (including 0)
- `_ccChannel` not set for non-GRP_TXT packets
- `getRowStyle` returns `border-left:3px` only (no background)
- Palette is exactly 8 colors, no teal/rose
- All existing tests pass (62 + 29 + 490)

Fixes #674

---------

Co-authored-by: you <you@example.com>
2026-04-07 23:03:57 -07:00
you 00953207fb ci: remove arm64 build + QEMU — amd64 only
Removes linux/arm64 from multi-platform build and drops QEMU setup.
All infra (prod + staging) is x86. QEMU emulation was adding ~12min
to every CI run for an unused architecture.
2026-04-08 05:23:41 +00:00
you 16a72b66a9 test: fix hash_size test for zero-hop behavior change (#653)
The buildFieldTable test expected hash_size=4 for path byte 0xC0 with
hash_count=0. After #653, zero hash_count shows 'hash_count=0 (direct
advert)' instead. Updated test and added new test verifying hash_size
IS shown when hash_count > 0.
2026-04-08 04:53:10 +00:00
Kpa-clawbot e0e9aaa324 feat: noise floor column chart with color-coded thresholds (#659)
## Noise Floor: Line Chart → Color-Coded Column Chart

Implements M3a from the [RF Health Dashboard
spec](https://github.com/Kpa-clawbot/CoreScope/issues/600#issuecomment-2784399622)
— replacing the noise floor line chart with discrete color-coded
columns.

### What changed

**`public/analytics.js`** — replaced `rfNFLineChart()` with
`rfNFColumnChart()`:

- **Color-coded bars by threshold**: green (`< -100 dBm`), yellow (`-100
to -85 dBm`), red (`≥ -85 dBm`)
- **Instant hover tooltips**: exact dBm value + UTC timestamp via native
SVG `<title>` — no delay
- **Column highlighting on hover**: CSS `:hover` with opacity change +
border stroke
- **Inline legend**: green/yellow/red threshold key in chart header
- **Removed reference lines**: the `-100 warning` and `-85 critical`
dashed lines are eliminated — threshold info is now encoded directly in
bar color (data-ink ratio improvement)
- **No gap detection**: column charts render discrete bars — each data
point is an independent observation, so line-chart-style gap detection
doesn't apply. Every sample gets a bar.
- **Reboot markers**: vertical dashed lines with "reboot" labels at
reboot timestamps (shared `rfRebootMarkers` helper, same as other RF
charts)
- **Division-by-zero guard**: constant values or single data points use
a ±5 dBm window so bars render with visible height
- **Sparklines unchanged**: fleet overview sparklines remain as
polylines (correct at 140×24px scale)

### Why columns instead of lines

A polyline connecting discrete 5-minute noise floor samples creates
false visual continuity — it implies interpolation between measurements
that doesn't exist. When readings jump between -115 and -95 irregularly,
the line becomes a jagged mess. Column bars encode each sample as a
discrete, independent observation: one bar = one measurement.

### Testing

- 12 unit tests in `test-frontend-helpers.js` covering: SVG output,
threshold color coding, tooltips, empty/single/constant data, legend
rendering, reboot markers, shared time axis
- All existing tests pass (packet-filter: 62, aging: 29,
frontend-helpers: 490)

### No backend changes

Pure frontend change — ~150 lines in `analytics.js`.

Fixes #600

---------

Co-authored-by: you <you@example.com>
2026-04-07 21:40:14 -07:00
Kpa-clawbot 22bf33700e Fix: filter path-hop candidates by resolved_path to prevent prefix collisions (#658)
## Problem

The "Paths Through This Node" API endpoint (`/api/nodes/{pubkey}/paths`)
returns unrelated packets when two nodes share a hex prefix. For
example, querying paths for "Kpa Roof Solar" (`c0dedad4...`) returns 316
packets that actually belong to "C0ffee SF" (`C0FFEEC7...`) because both
share the `c0` prefix in the `byPathHop` index.

Fixes #655

## Root Cause

`handleNodePaths()` in `routes.go` collects candidates from the
`byPathHop` index using 2-char and 4-char hex prefixes for speed, but
never verifies that the target node actually appears in each candidate's
resolved path. The broad index lookup is intentional, but the
**post-filter was missing**.

## Fix

Added `nodeInResolvedPath()` helper in `store.go` that checks whether a
transmission's `resolved_path` (from the neighbor affinity graph via
`resolveWithContext`) contains the target node's full pubkey. The
filter:

- **Includes** packets where `resolved_path` contains the target node's
full pubkey
- **Excludes** packets where `resolved_path` resolved to a different
node (prefix collision)
- **Excludes** packets where `resolved_path` is nil/empty (ambiguous —
avoids false positives)

The check examines both the best observation's resolved_path
(`tx.ResolvedPath`) and all individual observations, so packets are
included if *any* observation resolved the target.

## Tests

- `TestNodeInResolvedPath` — unit test for the helper with 5 cases
(match, different node, nil, all-nil elements, match in observation
only)
- `TestNodePathsPrefixCollisionFilter` — integration test: two nodes
sharing `aa` prefix, verifies the collision packet is excluded from one
and included for the other
- Updated test DB schema to include `resolved_path` column and seed data
with resolved pubkeys
- All existing tests pass (165 additions, 8 modifications)

## Performance

No impact on hot paths. The filter runs once per API call on the
already-collected candidate set (typically small). `nodeInResolvedPath`
is O(observations × hops) per candidate — negligible since observations
per transmission are typically 1–5.

---------

Co-authored-by: you <you@example.com>
2026-04-07 21:24:00 -07:00
Kpa-clawbot b8e9b04a97 feat: panel corner-position toggle (M0) (#657)
## Panel Corner-Position Toggle (M0)

Fixes #608

### What

Each overlay panel on the live map page (feed, legend, node detail) gets
a small corner-toggle button that cycles through **TL → TR → BR → BL**
placement. This solves the panel-blocking-map-data problem with minimal
complexity.

### Changes

**`public/live.css`** (~60 lines)
- CSS classes for 4 corner positions via `data-position` attribute
- Smooth transitions with `cubic-bezier` easing
- `prefers-reduced-motion` support
- Direction-aware hide animations for positioned panels
- `.panel-corner-btn` styling (subtle, hover-to-reveal)
- Mobile: corner buttons hidden (`<640px` — panels are hidden or
bottom-sheet)
- `.sr-only` class for screen reader announcements

**`public/live.js`** (~90 lines)
- `PANEL_DEFAULTS`, `CORNER_CYCLE`, `CORNER_ARROWS` constants
- `getPanelPositions()` — reads from localStorage with defaults
- `nextAvailableCorner()` — collision avoidance (skips occupied corners)
- `applyPanelPosition()` — sets `data-position` + updates button
- `onCornerClick()` — cycle logic + persistence + SR announcement
- `resetPanelPositions()` — clears saved positions
- Corner toggle buttons added to feed, legend, and node detail panel
HTML
- `initPanelPositions()` called during page init

**`test-panel-corner.js`** (14 tests)
- `nextAvailableCorner`: available, skip occupied, skip multiple,
self-exclusion
- `getPanelPositions`: defaults, saved values
- `applyPanelPosition`: attribute setting, button update, missing
element
- `onCornerClick`: cycling, collision avoidance
- `resetPanelPositions`: clear + restore defaults
- Cycle order and default position validation

### What this does NOT include

- Drag-and-drop (M1–M4)
- Snap-to-edge
- Z-index management
- Keyboard repositioning
- Any of the full drag system

### Design decisions

- **`data-position` + CSS classes** over inline transforms — avoids
conflict with existing show/hide `transform` animations
- **Cycle (TL→TR→BR→BL)** over toggle-to-opposite — predictable,
learnable
- **3 panels, 4 corners** — collision avoidance is trivial, always a
free corner
- **Header/stats panel excluded** — it's contextual chrome, not
repositionable

---------

Co-authored-by: you <you@example.com>
2026-04-07 21:20:29 -07:00
Kpa-clawbot 7d71dc857b feat: expose hopsCompleted for TRACE packets, show real path on live map (#656)
## Summary

TRACE packets on the live map previously animated the **full intended
route** regardless of how far the trace actually reached. This made it
impossible to distinguish a completed route from a failed one —
undermining the primary diagnostic purpose of trace packets.

## Changes

### Backend — `cmd/server/decoder.go`

- Added `HopsCompleted *int` field to the `Path` struct
- For TRACE packets, the header path contains SNR bytes (one per hop
that actually forwarded). Before overwriting `path.Hops` with the full
intended route from the payload, we now capture the header path's
`HashCount` as `hopsCompleted`
- This field is included in API responses and WebSocket broadcasts via
the existing JSON serialization

### Frontend — `public/live.js`

- For TRACE packets with `hopsCompleted < totalHops`:
  - Animate only the **completed** portion (solid line + pulse)
- Draw the **unreached** remainder as a dashed/ghosted line (25%
opacity, `6,8` dash pattern) with ghost markers
  - Dashed lines and ghost markers auto-remove after 10 seconds
- When `hopsCompleted` is absent or equals total hops, behavior is
unchanged

### Tests — `cmd/server/decoder_test.go`

- `TestDecodePacket_TraceHopsCompleted` — partial completion (2 of 4
hops)
- `TestDecodePacket_TraceNoSNR` — zero completion (trace not forwarded
yet)
- `TestDecodePacket_TraceFullyCompleted` — all hops completed

## How it works

The MeshCore firmware appends an SNR byte to `pkt->path[]` at each hop
that forwards a TRACE packet. The count of these SNR bytes (`path_len`)
indicates how far the trace actually got. CoreScope's decoder already
parsed the header path, but the TRACE-specific code overwrote it with
the payload hops (full intended route) without preserving the progress
information. Now we save that count first.

Fixes #651

---------

Co-authored-by: you <you@example.com>
2026-04-07 21:19:45 -07:00
Kpa-clawbot 088b4381c3 Fix: Hash Stats 'By Repeaters' includes non-repeater nodes (#654)
## Summary

The "By Repeaters" section on the Hash Stats analytics page was counting
**all** node types (companions, room servers, sensors, etc.) instead of
only repeaters. This made the "By Repeaters" distribution identical to
"Multi-Byte Hash Adopters", defeating the purpose of the breakdown.

Fixes #652

## Root Cause

`computeAnalyticsHashSizes()` in `cmd/server/store.go` built its
`byNode` map from advert packet data without cross-referencing node
roles from the node store. Both `distributionByRepeaters` and
`multiByteNodes` consumed this unfiltered map.

## Changes

### `cmd/server/store.go`
- Build a `nodeRoleByPK` lookup map from `getCachedNodesAndPM()` at the
start of the function
- Store `role` in each `byNode` entry when processing advert packets
- **`distributionByRepeaters`**: filter to only count nodes whose role
contains "repeater"
- **`multiByteNodes`**: include `role` field in output so the frontend
can filter/group by node type

### `cmd/server/coverage_test.go`
- Add `TestHashSizesDistributionByRepeatersFiltersRole`: verifies that
companion nodes are excluded from `distributionByRepeaters` but included
in `multiByteNodes` with correct role

### `cmd/server/routes_test.go`
- Fix `TestHashAnalyticsZeroHopAdvert`: invalidate node cache after DB
insert so role lookup works
- Fix `TestAnalyticsHashSizeSameNameDifferentPubkey`: insert node
records as repeaters + invalidate cache

## Testing

All `cmd/server` tests pass (68 insertions, 3 deletions across 3 files).

Co-authored-by: you <you@example.com>
2026-04-07 21:00:03 -07:00
you 1ff094b852 fix: staging compose — standard ports, remove 3GB memory limit
- HTTP: 82→80 (standard)
- MQTT: 1885→1883 (standard)
- Remove 3GB memory limit that was causing OOM on 1.5M observation DB
2026-04-08 03:50:07 +00:00
efiten 144e98bcdf fix: hide hash size for zero-hop direct adverts (#649) (#653)
## Fix: Zero-hop DIRECT packets report bogus hash_size

Closes #649

### Problem
When a DIRECT packet has zero hops (pathByte lower 6 bits = 0), the
generic `hash_size = (pathByte >> 6) + 1` formula produces a bogus value
(1-4) instead of 0/unknown. This causes incorrect hash size displays and
analytics for zero-hop direct adverts.

### Solution

**Frontend (JS):**
- `packets.js` and `nodes.js` now check `(pathByte & 0x3F) === 0` to
detect zero-hop packets and suppress bogus hash_size display.

**Backend (Go):**
- Both `cmd/server/decoder.go` and `cmd/ingestor/decoder.go` reset
`HashSize=0` for DIRECT packets where `pathByte & 0x3F == 0` (hash_count
is zero).
- TRACE packets are excluded since they use hashSize to parse hop data
from the payload.
- The condition uses `pathByte & 0x3F == 0` (not `pathByte == 0x00`) to
correctly handle the case where hash_size bits are non-zero but
hash_count is zero — matching the JS frontend approach.

### Testing

**Backend:**
- Added 4 tests each in `cmd/server/decoder_test.go` and
`cmd/ingestor/decoder_test.go`:
  - DIRECT + pathByte 0x00 → HashSize=0 
- DIRECT + pathByte 0x40 (hash_size bits set, hash_count=0) → HashSize=0

  - Non-DIRECT + pathByte 0x00 → HashSize=1 (unchanged) 
  - DIRECT + pathByte 0x01 (1 hop) → HashSize=1 (unchanged) 
- All existing tests pass (`go test ./...` in both cmd/server and
cmd/ingestor)

**Frontend:**
- Verified hash size display is suppressed for zero-hop direct adverts

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
2026-04-07 19:39:15 -07:00
efiten bd54707987 feat: distance unit preference — km, mi, or auto (#621) (#646)
## Summary

- **`app.js`**: `getDistanceUnit()`, `formatDistance(km)`,
`formatDistanceRound(km)` helpers. Auto mode uses `navigator.language` —
miles for `en-US`, `en-GB`, `my`, `lr`; km everywhere else.
- **`customize-v2.js`**: Distance Unit preference (km / mi / auto) in
Display Settings panel. Stored in
`localStorage['meshcore-distance-unit']` via the existing apply
pipeline. Override dot and reset work. Display tab badge counts it.
- **`nodes.js`**: Neighbor table distance cell uses `formatDistance()`.
- **`analytics.js`**: All rendered km values use `formatDistance()` or
`formatDistanceRound()`. Column headers (`km`/`mi`) respond to the
active unit. Collision classification thresholds (Local < 50 km /
Regional 50–200 km / Distant > 200 km) also adapt.

Default is `auto` — no change for existing users unless their locale
maps to miles.

## Test plan

- [x] `node test-frontend-helpers.js` — 456 passed, 0 failed (10 new
formatDistance tests)
- [ ] Set unit to **mi** in customize → Neighbors table shows `7.6 mi`
instead of `12.3 km`
- [ ] Analytics → Distance tab → stat cards, leaderboard, and column
headers all show miles
- [ ] Collision tool → Local/Regional/Distant thresholds show `31 mi` /
`124 mi`
- [ ] Route patterns popup shows miles per hop and total
- [ ] Reset override dot → unit returns to auto

Closes #621

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
2026-04-07 19:36:25 -07:00
efiten 1033555d00 fix: resolve originLat out-of-scope ReferenceError in resolveHopPositions (#647) (#648)
## Summary

- `originLat` was declared with `const` inside two block-scoped
`if`/`else` branches in `resolveHopPositions` (lines 1914 and 1921) but
referenced at line 1945 outside both blocks → `ReferenceError: originLat
is not defined` thrown on every packet render on the live page.
- Fix: introduce `senderLat` derived directly from
`payload.lat`/`payload.lon` at the point of use, using the same
null/zero guard as the existing declarations.

## Test plan

- [x] Live page no longer shows `ReferenceError: originLat is not
defined` in the console
- [x] Packet path animations still render correctly for packets with GPS
coords
- [x] Packets without GPS coords still handled (senderLat === null,
anchor not added)

Closes #647

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
2026-04-07 19:31:43 -07:00
Kpa-clawbot 37be3dcd1f fix: Prefix Tool text consistency — use 'repeaters' everywhere (#642) (#645)
## Summary

Fixes remaining text inconsistencies in the Prefix Tool after #643 added
the repeater filter.

The Torvalds review on #643 flagged:
1. **Must-fix (already addressed in #643):** "About these numbers" text
— fixed
2. **Out-of-scope:** Empty state says "No nodes" should say "No
repeaters"

This PR fixes ALL remaining "nodes" references in the Prefix Tool to say
"repeaters":

- Empty state: "No nodes in the network yet" → "No repeaters in the
network yet"
- Stat card label: "Total nodes" → "Total repeaters"
- Region note link: "Check all nodes →" → "Check all repeaters →"
- Recommendation text: "With N nodes" → "With N repeaters"

Verified: zero occurrences of stale "all nodes", "Total nodes", or "No
nodes" remain in the Prefix Tool section.

Closes #642

Co-authored-by: you <you@example.com>
2026-04-06 15:43:43 -07:00
efiten 2bff89a546 feat: deep link P1 UI states — nodes tab, packets filters, channels node panel (#536) (#618)
## Summary

- **nodes.js**: `#/nodes?tab=repeater` and `#/nodes?search=foo` — role
tab and search query are now URL-addressable; state resets to defaults
on re-navigation
- **packets.js**: `#/packets?timeWindow=60` and
`#/packets?region=US-SFO` — time window and region filter survive
refresh and are shareable
- **channels.js**: `#/channels/{hash}?node=Name` — node detail panel is
URL-addressable; auto-opens on load, URL updates on open/close
- **region-filter.js**: adds `RegionFilter.setSelected(codesArray)` to
public API (needed for URL-driven init)

All changes use `history.replaceState` (not `pushState`) to avoid
polluting browser history. URL params override localStorage on load;
localStorage remains fallback.

## Implementation notes

- Router strips query string before computing `routeParam`, so all pages
read URL params directly from `location.hash`
- `buildNodesQuery(tab, searchStr)` and `buildPacketsUrl(timeWindowMin,
regionParam)` are pure functions exposed on `window` for testability
- Region URL param is applied after `RegionFilter.init()` via a
`_pendingUrlRegion` module-level var to keep ordering explicit
- `showNodeDetail` captures `selectedHash` before the async `lookupNode`
call to avoid stale URL construction

## Test plan

- [x] `node test-frontend-helpers.js` — 459 passed, 0 failed (includes 6
`buildNodesQuery` + 5 `buildPacketsUrl` unit tests)
- [x] Navigate to `#/nodes?tab=repeater` — Repeaters tab active on load
- [x] Click a tab, verify URL updates to `#/nodes?tab=room`
- [x] Navigate to `#/packets?timeWindow=60` — time window dropdown shows
60 min
- [x] Change time window, verify URL updates
- [x] Navigate to `#/channels/{hash}` and click a sender name — URL
updates to `?node=Name`
- [x] Reload that URL — node panel re-opens

Closes #536

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 15:43:25 -07:00
Kpa-clawbot dc079064f5 fix: clarify Hash Issues vs Prefix Tool collision data discrepancy (#643)
## Summary

Hash Issues and Prefix Tool tabs showed different collision counts
because the Prefix Tool was including all node types (companions, rooms,
sensors) while Hash Issues correctly filtered to repeaters only.

**Only repeaters matter for prefix collisions** — they're the nodes that
relay packets using hash-based addressing. Non-repeater collisions are
harmless noise.

## Changes
1. **Filtered Prefix Tool to repeaters only** — matches Hash Issues'
scope
2. **Updated explanatory text** — both tabs now clearly state they cover
repeaters
3. **Added cross-reference links** between the two tabs
4. **Added hash_size badges** in Prefix Tool results

Both tabs should now agree on collision counts for each byte size.

## Review Status
-  Self-review
-  Torvalds review — caught stale 'regardless of role' text, fixed
-  All tests pass

Fixes #642

---------

Co-authored-by: you <you@example.com>
2026-04-05 19:52:19 -07:00
Kpa-clawbot 43098a0705 refactor: DRY hash matrix rendering in analytics.js (#419) (#640)
## Summary

Fixes #419 — DRY violation in `renderHashMatrixFromServer` in
analytics.js.

The 1-byte and 2-byte branches shared ~80% identical HTML structure
(stat cards, matrix grid, detail panel, legend, tooltip init, click
handlers). This refactor extracts four shared helpers:

### New helpers

| Helper | Purpose |
|--------|---------|
| `classifyHashCell(count, isConfirmed, isPossible)` | Unified cell
classification → `{cls, bg}` |
| `hashCellTd(hex, cellSize, cls, bg, count, tipHtml, fontWeight)` |
Shared `<td>` element generation |
| `hashTooltipHtml(hexLabel, statusText, nodesHtml)` | Tooltip HTML
assembly |
| `renderHashMatrixPanel(el, statCards, cellFn, detailWidth, legend,
clickFn)` | Full matrix assembly pipeline |

### What changed

- Both branches now call `renderHashMatrixPanel()` with branch-specific
callbacks for cell rendering and detail click handling
- Cell classification logic (empty → taken → possible → collision with
heat scaling) is unified in `classifyHashCell()`
- Tooltip and `<td>` generation consolidated — no more duplicated inline
template strings
- Zero behavioral changes — all existing rendering, tooltips, and click
interactions are preserved

### Tests

All existing tests pass (445 frontend helpers, 62 packet filter, 29
aging).

Co-authored-by: you <you@example.com>
2026-04-05 18:31:23 -07:00
Kpa-clawbot 2d260bbfed test: behavioral vscroll tests replacing source-grep (#405, #409) (#641)
## Summary

Replace source-grep virtual scroll tests with behavioral tests that
exercise actual logic. Fixes #405, Fixes #409.

## What changed

### packets.js
- **Extracted `_calcVisibleRange()`** — pure function containing the
binary-search range calculation logic previously inline in
`renderVisibleRows()`. Takes offsets, scroll position, viewport
dimensions, row height, thead offset, and buffer as parameters. Returns
`{ startIdx, endIdx, firstEntry, lastEntry }`.
- `renderVisibleRows()` now calls `_calcVisibleRange()` instead of
inline math — no behavioral change.
- Exported via `_packetsTestAPI` for direct testing.

### test-frontend-helpers.js
- **Removed 8 source-grep tests** that used
`packetsSource.includes(...)` to check strings exist in source code (not
behavior):
  - "renderVisibleRows uses cumulative offsets not flat entry count"
  - "renderVisibleRows skips DOM rebuild when range unchanged"
  - "lazy row generation — HTML built only for visible slice"
  - "observer filter Set is hoisted, not recreated per-packet"
  - "packets.js display filter checks _children for observer match"
  - "packets.js WS filter checks _children for observer match"
  - "buildFlatRowHtml has null-safe decoded_json"
  - "pathHops null guard in buildFlatRowHtml / detail pane"
  - "destroy cleans up virtual scroll state"

- **Added 11 behavioral tests for `_calcVisibleRange()`** loaded from
the actual packets.js via sandbox:
  - Top of list (scroll = 0)
  - Middle of list (scroll to row 50)
  - Bottom of list (scroll past end)
  - Empty array (0 entries)
  - Single item
  - Exact row boundary
  - Large dataset (30K items)
  - Various row heights (24px instead of 36px)
  - Thead offset shifting visible range
  - Expanded groups with variable row counts
  - Buffer clamped at boundaries

- **Kept all existing behavioral tests**: `cumulativeRowOffsets`,
`getRowCount`, observer filter logic (#537).

## Test count
- Removed: 8 source-grep tests
- Added: 11 behavioral tests
- Net: +3 tests (446 total, 0 failures)

## Why
Source-grep tests (`packetsSource.includes('...')`) are brittle — they
break on refactors even when behavior is preserved, and they pass even
when the tested code is buggy. Behavioral tests exercise real
inputs/outputs and catch actual regressions.

Co-authored-by: you <you@example.com>
2026-04-05 18:30:30 -07:00
Kpa-clawbot 1dd763bf44 feat: sortable nodes list + neighbor/observer tables (M2, #620) (#639)
## Summary

Implements M2 of the table sorting spec (#620): sortable nodes list +
neighbor/observer tables.

### Changes

**Shared utility (`public/table-sort.js`)**
- IIFE pattern, no dependencies, no build step
- DOM-reorder sorting (no innerHTML rebuild) — preserves event listeners
- `data-value` attributes for raw sortable values, `data-type` on `<th>`
for type detection
- Built-in comparators: text (`localeCompare`), number, date, dBm
- `aria-sort` attributes, keyboard support (Enter/Space), sort arrows
- localStorage persistence with `storageKey` option
- `onSort` callback for custom re-render triggers

**Nodes list table**
- Wired via `TableSort.init` with `onSort` callback that triggers
`renderRows()`
- Keeps JS-array-level sorting for claimed/favorites pinning (TableSort
can't handle pinned rows)
- Replaces old `sortState`, `toggleSort()`, `sortArrow()` with TableSort
controller
- Test hooks preserved for backward compatibility (fallback state for
non-DOM tests)

**Neighbor table**
- Added `data-sort` and `data-value` attributes to all columns (name,
role, score, count, last_seen, distance)
- Default sort: count descending
- `TableSort.init` called after neighbor data renders

**Observer table (full detail page)**
- Converted from plain `<table>` to sortable table with data attributes
- Sortable columns: observer, region, packets, avg SNR, avg RSSI
- Default sort: packets descending

### Testing
- 18 new unit tests for `table-sort.js` (custom DOM mock, no jsdom
dependency)
- All 445 existing frontend tests pass unchanged
- All packet-filter (62) and aging (29) tests pass

### Note
This branch includes `table-sort.js` since M1 hasn't merged yet. The
utility code is identical to the M1 spec.

---------

Co-authored-by: you <you@example.com>
2026-04-05 18:29:54 -07:00
you 6b9946d9c6 docs: timestamp-based packet filter spec (#289) 2026-04-06 01:22:15 +00:00
Kpa-clawbot 243de9fba1 fix: consolidate CI pipeline — build, publish to GHCR, then deploy staging (#636)
## Consolidate CI Pipeline — Build + Publish to GHCR + Deploy Staging

### What
Merges the separate `publish.yml` workflow into `deploy.yml`, creating a
single CI/CD pipeline:

**`go-test → e2e-test → build-and-publish → deploy → publish-badges`**

### Why
- Two workflows doing overlapping builds was wasteful and error-prone
- `publish.yml` had a bug: `BUILD_TIME=$(date ...)` in a `with:` block
never executed (literal string)
- The old build job had duplicate/conflicting `APP_VERSION` assignments

### Changes
- **`build-and-publish` job** replaces old `build` job — builds locally
for staging, then does multi-arch GHCR push (gated to push events only,
PRs skip)
- **Build metadata** computed in a dedicated step, passed via
`GITHUB_OUTPUT` — no more shell expansion bugs
- **`APP_VERSION`** is `v1.2.3` on tag push, `edge` on master push
- **Deploy** now pulls the `edge` image from GHCR and tags for compose
compatibility, with fallback to local build
- **`publish.yml` deleted** — no duplicate workflow
- **Top-level `permissions`** block with `packages:write` for GHCR auth
- **Triggers** now include `tags: ['v*']` for release publishing

### Status
-  Rebased onto master
-  Self-reviewed (all checklist items pass)
-  Ready for merge

Co-authored-by: you <you@example.com>
2026-04-05 18:09:20 -07:00
Kpa-clawbot 6f3e3535c9 feat: shared table sort utility + packets table sorting (M1, #620) (#638)
## Summary

Implements M1 of the table sorting spec (#620): a shared `TableSort`
utility module and integration with the packets table.

### What's included

**1. `public/table-sort.js` — Shared sort utility (IIFE, no
dependencies)**
- `TableSort.init(tableEl, options)` — attaches click-to-sort on `<th
data-sort-key="...">` elements
- Built-in comparators: text (localeCompare), numeric, date (ISO), dBm
(strips suffix)
- NaN/null values sort last consistently
- Visual: ▲/▼ `<span class="sort-arrow">` appended to active column
header
- Accessibility: `aria-sort="ascending|descending|none"`, keyboard
support (Enter/Space)
- DOM reorder via `appendChild` loop (no innerHTML rebuild)
- `domReorder: false` option for virtual scroll tables (packets)
- `storageKey` option for localStorage persistence
- Custom comparator override per column
- `onSort(column, direction)` callback
- `destroy()` for clean teardown

**2. Packets table integration**
- All columns sortable: region, time, hash, size, HB, type, observer,
path, rpt
- Default sort: time descending (matches existing behavior)
- Uses `domReorder: false` + `onSort` callback to sort the data array,
then re-render via virtual scroll
- Works with both grouped and ungrouped views
- WebSocket updates respect active sort column
- Sort preference persisted in localStorage (`meshcore-packets-sort`)

**3. Tests — 22 unit tests (`test-table-sort.js`)**
- All 4 built-in comparators (text, numeric, date, dBm)
- NaN/null edge cases
- Direction toggle on click
- aria-sort attribute correctness
- Visual indicator (▲/▼) presence and updates
- onSort callback
- domReorder: false behavior
- destroy() cleanup
- Custom comparator override

### Performance

Packets table sorting works at the data array level (single `Array.sort`
call), not DOM level. Virtual scroll then renders only visible rows. No
new DOM nodes are created during sort — it's purely a data reorder +
re-render of the existing visible window. Expected sort time for 30K
packets: ~50-100ms (array sort) + existing virtual scroll render time.

Closes #620 (M1)

Co-authored-by: you <you@example.com>
2026-04-05 15:29:14 -07:00
Kpa-clawbot cae14da05e fix: implement DISABLE_CADDY env var in Docker entrypoint (#629) (#637)
## Summary

Implements the `DISABLE_CADDY` environment variable in the Docker
entrypoint, fixing #629.

## Problem

The `DISABLE_CADDY` env var was documented but had no effect — the
entrypoint only handled `DISABLE_MOSQUITTO`.

## Changes

### New supervisord configs
- **`supervisord-go-no-caddy.conf`** — mosquitto + ingestor + server (no
Caddy)
- **`supervisord-go-no-mosquitto-no-caddy.conf`** — ingestor + server
only

### Updated entrypoint (`docker/entrypoint-go.sh`)
Handles all 4 combinations:
| DISABLE_MOSQUITTO | DISABLE_CADDY | Config used |
|---|---|---|
| false | false | `supervisord.conf` (default) |
| true | false | `supervisord-no-mosquitto.conf` |
| false | true | `supervisord-no-caddy.conf` |
| true | true | `supervisord-no-mosquitto-no-caddy.conf` |

### Dockerfiles
Added COPY lines for the new configs in both `Dockerfile` and
`Dockerfile.go`.

## Testing

```bash
# Verify correct config selection
docker run -e DISABLE_CADDY=true corescope
# Should log: [config] Caddy reverse proxy disabled (DISABLE_CADDY=true)

docker run -e DISABLE_CADDY=true -e DISABLE_MOSQUITTO=true corescope
# Should log both disabled messages
```

Fixes #629

Co-authored-by: you <you@example.com>
2026-04-05 15:26:40 -07:00
Kpa-clawbot e046a6f632 fix: mobile accessibility — touch targets, ARIA, small viewport support (#630) (#633)
## Summary

Fixes critical and major mobile accessibility items from #630, focused
on small phone viewports (320px–375px).

### Critical fixes
1. **Touch targets ≥ 44px** — All interactive elements (filter buttons,
tab buttons, search inputs, nav buttons, region pills, dropdowns) get
`min-height: 44px; min-width: 44px` via `@media (pointer: coarse)` —
desktop/mouse users are unaffected.
2. **ARIA live regions** — Added `aria-live="polite"` to: packet list
(`#pktLeft`), node list (`#nodesLeft`), analytics content
(`#analyticsContent`), live feed (`#liveFeed` with `role="log"`). Screen
readers now announce dynamic content updates.
3. **Color-only status indicators** — Status dots in live view marked
`aria-hidden="true"` (text labels like "Online"/"Degraded"/"Offline"
already present alongside).
4. **Detail panel on mobile** — Side panel (`panel-right`) renders as a
full-screen fixed overlay on ≤640px. Close button (✕) added to nodes
detail panel. Escape key closes both nodes and packets detail panels.

### Major fixes
5. **Analytics tabs overflow** — Tabs switch to `flex-wrap: nowrap;
overflow-x: auto` on ≤640px, preventing overflow on 320px screens.
6. **Table horizontal scroll** — Added `.table-scroll-wrap` class and
`min-width: 480px` on `.data-table` at ≤640px for horizontal scrolling
when columns don't fit.
7. **SPA focus management** — On every page navigation, focus moves to
first heading (`h1`/`h2`/`h3`) or falls back to `#app`. Uses
`requestAnimationFrame` for correct DOM timing.

### Bonus
- Analytics tabs get `role="tablist"` + `aria-label` for screen reader
semantics.

### Known follow-ups (not blocking)
- Individual tab buttons should get `role="tab"` + `aria-selected` +
`aria-controls` for complete ARIA tab pattern.
- `sr-status-label` and `table-scroll-wrap` CSS classes are defined but
not yet used in JS — ready for future use when status text labels and
table wrappers are wired up.

Closes #630

Co-authored-by: you <you@example.com>
2026-04-05 15:06:14 -07:00
Kpa-clawbot 0f5e2db5cf feat: auto-generated OpenAPI 3.0 spec endpoint + Swagger UI (#530) (#632)
## Summary

Auto-generated OpenAPI 3.0.3 spec endpoint (`/api/spec`) and Swagger UI
(`/api/docs`) for the CoreScope API.

## What

- **`cmd/server/openapi.go`** — Route metadata map
(`routeDescriptions()`) + spec builder that walks the mux router to
generate a complete OpenAPI 3.0.3 spec at runtime. Includes:
- All 47 API endpoints grouped by tag (admin, analytics, channels,
config, nodes, observers, packets)
- Query parameter documentation for key endpoints (packets, nodes,
search, resolve-hops)
  - Path parameter extraction from mux `{name}` patterns
  - `ApiKeyAuth` security scheme for API-key-protected endpoints
  - Swagger UI served as a self-contained HTML page using unpkg CDN

- **`cmd/server/openapi_test.go`** — Tests for spec endpoint (validates
JSON structure, required fields, path count, security schemes,
self-exclusion of `/api/spec` and `/api/docs`), Swagger UI endpoint, and
`extractPathParams` helper.

- **`cmd/server/routes.go`** — Stores router reference on `Server`
struct for spec generation; registers `/api/spec` and `/api/docs`
routes.

## Design Decisions

- **Runtime spec generation** vs static YAML: The spec walks the actual
router, so it can never drift from registered routes. Route metadata
(summaries, descriptions, tags, auth flags) is maintained in a parallel
map — the test enforces minimum path count to catch drift.
- **No external dependencies**: Uses only stdlib + existing gorilla/mux.
Swagger UI loaded from unpkg CDN (no vendored assets).
- **Security tagging**: Auth-protected endpoints (those behind
`requireAPIKey` middleware) are tagged with `security: [{ApiKeyAuth:
[]}]` in the spec, matching the actual middleware configuration.

## Testing

- `go test -run TestOpenAPI` — validates spec structure, field presence,
path count ≥ 20, security schemes
- `go test -run TestSwagger` — validates HTML response with swagger-ui
references
- `go test -run TestExtractPathParams` — unit tests for path parameter
extraction

---------

Co-authored-by: you <you@example.com>
2026-04-05 15:05:20 -07:00
Kpa-clawbot a068e3e086 feat: zero-config defaults + deployment docs (M3-M4, #610) (#631)
## Zero-Config Defaults + Deployment Docs

Make CoreScope start with zero configuration — no `config.json`
required. The ingestor falls back to sensible defaults (local MQTT
broker, standard topics, default DB path) when no config file exists.

### What changed

**`cmd/ingestor/config.go`** — `LoadConfig` no longer errors on missing
config file. Instead it logs a message and uses defaults. If no MQTT
sources are configured (from file or env), defaults to
`mqtt://localhost:1883` with `meshcore/#` topic.

**`cmd/ingestor/main.go`** — Removed redundant "no MQTT sources" fatal
(now handled in config layer). Improved the "no connections established"
fatal with actionable hints.

**`README.md`** — Replaced "Docker (Recommended)" section with a
one-command quickstart using the pre-built image. No build step, no
config file, just `docker run`.

**`docs/deployment.md`** — New comprehensive deployment guide covering
Docker, Compose, config reference, MQTT setup, TLS/HTTPS, monitoring,
backup, and troubleshooting.

### Zero-config flow

```
docker run -d -p 80:80 -v corescope-data:/app/data ghcr.io/kpa-clawbot/corescope:latest
```

1. No config.json found → defaults used, log message printed
2. No MQTT sources → defaults to `mqtt://localhost:1883`
3. Internal Mosquitto broker already running in container → connection
succeeds
4. Dashboard shows empty, ready for packets

### Review fixes (commit 13b89bb)

- Removed `DISABLE_CADDY` references from all docs — this env var was
never implemented in the entrypoint
- Fixed `/api/stats` example in deployment guide — showed nonexistent
fields (`mqttConnected`, `uptimeSeconds`, `activeNodes`)
- Improved MQTT connection failure message with actionable
troubleshooting hints

Closes #610

---------

Co-authored-by: you <you@example.com>
2026-04-05 15:04:49 -07:00
you 24335164d6 docs: table sorting consistency spec (#620) 2026-04-05 21:56:09 +00:00
Kpa-clawbot 7cef89e07b fix: mobile UX improvements for channel color picker (#619) (#626)
## Summary

Mobile UX fixes for the channel color picker (addresses #619).

## Changes

### Commit 1: Mobile UX improvements
- **Bottom-sheet pattern on mobile**: Color picker renders as a fixed
bottom sheet on touch devices (`@media (pointer: coarse)`) with
`env(safe-area-inset-bottom)` for notched phones
- **40px touch targets**: Swatches enlarged from default to 40×40px on
mobile
- **Native color picker hidden on touch**: `<input type="color">` is
hidden on mobile — preset swatches only
- **Scroll lock**: `document.body.style.overflow = 'hidden'` while
popover is open, restored on close
- **CSS context menu suppression**: `-webkit-touch-callout: none` and
`user-select: none` on `.live-feed-item`
- **Long-press with `passive: true`**: touchstart listener is passive to
avoid scroll jank

### Commit 2: Remove preventDefault on touchstart
- Removed `e.preventDefault()` from the touchstart handler — it was
blocking scroll initiation on feed items
- Context menu suppression handled entirely via CSS (see above)

## Desktop behavior
Unchanged. All mobile-specific styles scoped under `@media (pointer:
coarse)`. Desktop positioning logic unchanged.

## Review Status
-  Rebased onto master (no conflicts)
-  Self-review complete — all checklist items verified
-  Tufte analysis posted as comment

---------

Co-authored-by: you <you@example.com>
2026-04-05 14:51:13 -07:00
Kpa-clawbot dc5b5ce9a0 fix: reject weak/default API keys + startup warning (#532) (#628)
## Summary

Hardens API key security for write endpoints (fixes #532):

1. **Constant-time comparison** — uses
`crypto/subtle.ConstantTimeCompare` to prevent timing attacks on API key
validation
2. **Weak key blocklist** — rejects known default/example keys (`test`,
`password`, `change-me`, `your-secret-api-key-here`, etc.)
3. **Minimum length enforcement** — keys shorter than 16 characters are
rejected
4. **Startup warning** — logs a clear warning if the configured key is
weak or a known default
5. **Generic error messages** — HTTP 403 response uses opaque
"forbidden" message to prevent information leakage about why a key was
rejected

### Security Model
- **Empty key** → all write endpoints disabled (403)
- **Weak/default key** → all write endpoints disabled (403), startup
warning logged
- **Wrong key** → 401 unauthorized
- **Strong correct key** → request proceeds

### Files Changed
- `cmd/server/config.go` — `IsWeakAPIKey()` function + blocklist
- `cmd/server/routes.go` — constant-time comparison via
`constantTimeEqual()`, weak key rejection
- `cmd/server/main.go` — startup warning for weak keys
- `cmd/server/apikey_security_test.go` — comprehensive test coverage
- `cmd/server/routes_test.go` — existing tests updated to use strong
keys

### Reviews
-  Self-review: all security properties verified
-  djb Final Review: timing fix correct, blocklist pragmatic, error
messages opaque, tests comprehensive. **Verdict: Ship it.**

### Test Results
All existing + new tests pass. Coverage includes: weak key detection
(blocklist + length + case-insensitive), empty key handling, strong key
acceptance, wrong key rejection, and constant-time comparison.

---------

Co-authored-by: you <you@example.com>
2026-04-05 14:50:40 -07:00
Kpa-clawbot f59b4629b0 feat: publish Docker images to GHCR + simplified deploy (#610) (#627)
## Summary

Implements M1-M2 of the deployment simplification spec (#610). Adds
pre-built multi-arch Docker images published to GHCR, plus a simplified
deploy experience for operators.

**Spec:**
[docs/specs/deployment-simplification.md](https://github.com/Kpa-clawbot/CoreScope/blob/master/docs/specs/deployment-simplification.md)

## Files Added (no existing files modified)

### 1. `.github/workflows/publish.yml`
Multi-arch Docker publish workflow:
- Triggers on `v*` tags (releases) → produces `vX.Y.Z`, `vX.Y`, `vX`,
`latest`
- Triggers on master push → produces `edge` (unstable)
- `workflow_dispatch` for manual runs
- QEMU + buildx for `linux/amd64` + `linux/arm64`
- GHCR auth via `GITHUB_TOKEN`
- GHA layer caching for fast rebuilds

### 2. `docker-compose.example.yml`
20-line compose file that pulls from GHCR (no local build required):
- Env var overrides: `HTTP_PORT`, `DATA_DIR`, `DISABLE_CADDY`,
`DISABLE_MOSQUITTO`
- Health check included
- Volume mount for data persistence

### 3. `DEPLOY.md`
Operator documentation:
- One-line `docker run` deploy
- Tag reference (pinned vs latest vs edge)
- Environment variables table
- Update path (`docker compose pull && docker compose up -d`)
- TLS options (Caddy auto-TLS vs reverse proxy)
- **Migration guide for existing manage.sh users** — both paths
documented with command equivalency table

## Review Status

-  Self-review: Actions syntax, GHCR auth, multi-arch, tag strategy,
security — all verified
-  Torvalds: Deploy UX is clean, one-liner works, right level of
simplicity
-  BUILD_TIME fixed: uses `date` command instead of fragile
`head_commit.timestamp`
-  Migration guide added for existing manage.sh admins
- ⚠️ `DISABLE_CADDY` env var documented but not implemented in
entrypoint — pre-existing bug, filed as #629

Fixes #610

---------

Co-authored-by: you <you@example.com>
2026-04-05 14:33:57 -07:00
Kpa-clawbot f7000992ca fix(rf-health): auto-scale airtime Y-axis + hover tooltips (#600) (#623)
## Summary

Addresses user feedback on #600 — two improvements to RF Health detail
panel charts:

### 1. Auto-scale airtime Y-axis
Previously fixed 0-100% which made low-activity nodes unreadable (e.g.
0.1% TX barely visible). Now auto-scales to the actual data range with
20% headroom (minimum 1%), matching how the noise floor chart already
works.

### 2. Hover tooltips on all chart data points
Invisible SVG `<circle>` elements with native `<title>` tooltips on
every data point across all 4 charts:
- **Noise floor**: `NF: -112.3 dBm` + UTC timestamp
- **Airtime**: `TX: 2.1%` or `RX: 8.3%` + UTC timestamp  
- **Error rate**: `Err: 0.05%` + UTC timestamp
- **Battery**: `Batt: 3.85V` + UTC timestamp

Uses native browser SVG tooltips — zero dependencies, accessible, no JS
event handlers.

### Design rationale (Tufte)
- Auto-scaling increases data-ink ratio by eliminating wasted vertical
space
- Tooltips provide detail-on-demand without cluttering the chart with
labels on every point

### Spec update
Added M2 feedback improvements section to
`docs/specs/rf-health-dashboard.md`.

---------

Co-authored-by: you <you@example.com>
2026-04-05 13:08:05 -07:00
Kpa-clawbot 30e7e9ae3c docs: document lock ordering for cacheMu and channelsCacheMu (#624)
## Summary

Documents the lock ordering for all five mutexes in `PacketStore`
(`store.go`) to prevent future deadlocks.

## What changed

Added a comment block above the `PacketStore` struct documenting:

- All 5 mutexes (`mu`, `cacheMu`, `channelsCacheMu`, `groupedCacheMu`,
`regionObsMu`)
- What each mutex guards
- The required acquisition order (numbered 1–5)
- The nesting relationships that exist today (`cacheMu →
channelsCacheMu` in `invalidateCachesFor` and `rebuildAnalyticsCaches`)
- Confirmation that no reverse ordering exists (no deadlock risk)

## Verification

- Grepped all lock acquisition sites to confirm no reverse nesting
exists
- `go build ./...` passes — documentation-only change

Fixes #413

---------

Co-authored-by: you <you@example.com>
2026-04-05 13:00:35 -07:00
Kpa-clawbot 3415d3babb fix: measure VSCROLL_ROW_HEIGHT and theadHeight dynamically (#625)
## Summary

Replaces hardcoded `VSCROLL_ROW_HEIGHT = 36` and `theadHeight = 40` in
the virtual scroll logic with dynamic DOM measurement, so the values
stay correct if CSS changes.

## Changes

- `VSCROLL_ROW_HEIGHT`: measured once from the first rendered data row's
`offsetHeight` after the initial full rebuild. Falls back to 36px until
measurement occurs.
- `theadHeight`: measured from the actual `<thead>` element's
`offsetHeight` on every `renderVisibleRows` call. Falls back to 40px if
no thead is found.
- Both variables are now `let` instead of `const` to allow runtime
updates.

## Performance

No performance impact — both measurements are single `offsetHeight`
reads (no reflow triggered since the DOM was just written). Row height
measurement runs only once (guarded by `_vscrollRowHeightMeasured`
flag). Thead measurement is a single property read per scroll event.

Fixes #407

Co-authored-by: you <you@example.com>
2026-04-05 13:00:20 -07:00
Kpa-clawbot 05fbcb09dd fix: wire cacheTTL.analyticsHashSizes config to collision cache (#420) (#622)
## Summary

Fixes #420 — wires `cacheTTL` config values to server-side cache
durations that were previously hardcoded.

## Problem

`collisionCacheTTL` was hardcoded at 60s in `store.go`. The config has
`cacheTTL.analyticsHashSizes: 3600` (1 hour) but it was never read — the
`/api/config/cache` endpoint just passed the raw map to the client
without applying values server-side.

## Changes

- **`store.go`**: Add `cacheTTLSec()` helper to safely extract duration
values from the `cacheTTL` config map. `NewPacketStore` now accepts an
optional `cacheTTL` map (variadic, backward-compatible) and wires:
  - `cacheTTL.analyticsHashSizes` → `collisionCacheTTL`
  - `cacheTTL.analyticsRF` → `rfCacheTTL`
- **Default changed**: `collisionCacheTTL` default raised from 60s →
3600s (1 hour). Hash collision computation is expensive and data changes
rarely — 60s was causing unnecessary recomputation.
- **`main.go`**: Pass `cfg.CacheTTL` to `NewPacketStore`.
- **Tests**: Added `TestCacheTTLFromConfig` and `TestCacheTTLDefaults`
in eviction_test.go. Updated existing `TestHashCollisionsCacheTTL` for
the new default.

## Audit of other cacheTTL values

The remaining `cacheTTL` keys (`stats`, `nodeDetail`, `nodeHealth`,
`nodeList`, `bulkHealth`, `networkStatus`, `observers`, `channels`,
`channelMessages`, `analyticsTopology`, `analyticsChannels`,
`analyticsSubpaths`, `analyticsSubpathDetail`, `nodeAnalytics`,
`nodeSearch`, `invalidationDebounce`) are **client-side only** — served
via `/api/config/cache` and consumed by the frontend. They don't have
corresponding server-side caches to wire to. The only server-side caches
(`rfCache`, `topoCache`, `hashCache`, `chanCache`, `distCache`,
`subpathCache`, `collisionCache`) all use either `rfCacheTTL` or
`collisionCacheTTL`, both now configurable.

## Complexity

O(1) config lookup at store init time. No hot-path impact.

Co-authored-by: you <you@example.com>
2026-04-05 12:49:46 -07:00
efiten b587f20d1c feat: add distance column to neighbor table in node details (#617)
Closes #616

## What

Adds a **Distance** column to the neighbor table on the node detail
page.

When both the viewed node and a neighbor have GPS coordinates recorded,
the table shows the haversine distance between them (e.g. `3.2 km`).
When either node lacks GPS, the cell shows `—`.

## Changes

**Backend** (`cmd/server/neighbor_api.go`):
- Added `distance_km *float64` (omitempty) to `NeighborEntry`
- In `handleNodeNeighbors`: look up source node coords from `nodeMap`,
then for each resolved (non-ambiguous) neighbor with GPS, compute
`haversineKm` and set the field

**Frontend** (`public/nodes.js`):
- Added `Distance` column header between Last Seen and Conf
- Cell renders `X.X km` or `—` (muted) when unavailable

**Tests** (`cmd/server/neighbor_api_test.go`):
- `TestNeighborAPI_DistanceKm_WithGPS`: two nodes with real coords →
`distance_km` is positive
- `TestNeighborAPI_DistanceKm_NoGPS`: two nodes at 0,0 → `distance_km`
is nil

## Verification

Test at **https://staging.on8ar.eu** — navigate to any node detail page
and scroll to the Neighbors section. Nodes with GPS coordinates show a
distance; those without show `—`.
2026-04-05 12:33:23 -07:00
you af9754dbea ci: move staging build+deploy to meshcore-runner-2
Prod VM (meshcore-vm) is now prod-only. Staging builds and
deploys on the secondary runner.
2026-04-05 17:33:15 +00:00
Kpa-clawbot 767c8a5a3e perf: async chunked backfill — HTTP serves within 2 minutes (#612) (#614)
## Summary

Adds two config knobs for controlling backfill scope and neighbor graph
data retention, plus removes the dead synchronous backfill function.

## Changes

### Config knobs

#### `resolvedPath.backfillHours` (default: 24)
Controls how far back (in hours) the async backfill scans for
observations with NULL `resolved_path`. Transmissions with `first_seen`
older than this window are skipped, reducing startup time for instances
with large historical datasets.

#### `neighborGraph.maxAgeDays` (default: 30)
Controls the maximum age of `neighbor_edges` entries. Edges with
`last_seen` older than this are pruned from both SQLite and the
in-memory graph. Pruning runs on startup (after a 4-minute stagger) and
every 24 hours thereafter.

### Dead code removal
- Removed the synchronous `backfillResolvedPaths` function that was
replaced by the async version.

### Implementation details
- `backfillResolvedPathsAsync` now accepts a `backfillHours` parameter
and filters by `tx.FirstSeen`
- `NeighborGraph.PruneOlderThan(cutoff)` removes stale edges from the
in-memory graph
- `PruneNeighborEdges(conn, graph, maxAgeDays)` prunes both DB and
in-memory graph
- Periodic pruning ticker follows the same pattern as metrics pruning
(24h interval, staggered start)
- Graceful shutdown stops the edge prune ticker

### Config example
Both knobs added to `config.example.json` with `_comment` fields.

## Tests
- Config default/override tests for both knobs
- `TestGraphPruneOlderThan` — in-memory edge pruning
- `TestPruneNeighborEdgesDB` — SQLite + in-memory pruning together
- `TestBackfillRespectsHourWindow` — verifies old transmissions are
excluded by backfill window

---------

Co-authored-by: you <you@example.com>
2026-04-05 09:49:39 -07:00
Kpa-clawbot 382b3505dc feat: channel color quick-assign UI (M2, #271) (#611)
## Summary

Implements M2 of channel color highlighting (#271): a right-click
context menu popover for quick-assigning colors to hash channels.

Builds on M1 (PR #607) which provides `ChannelColors.set/get/remove`
storage primitives.

## What's new

### Color picker popover (`channel-color-picker.js`)
- **Right-click** any GRP_TXT/CHAN row in the **live feed** or **packets
table** → opens a color picker popover at the click point
- **Long-press** (500ms) on mobile triggers the same popover
- **10 preset swatches** — maximally distinct, ColorBrewer-inspired
palette
- **Custom hex** — native `<input type="color">` with Apply button
- **Clear button** — removes color assignment (hidden when no color
assigned)
- **Popover positioning** — auto-adjusts to avoid viewport overflow
- **Dismiss** — click outside or Escape key

### Immediate feedback
- Assigning a color instantly re-styles all visible live feed items with
that channel
- Packets table triggers `renderVisibleRows()` via exposed
`window._packetsRenderVisible`

### Wiring
- Feed items store `_ccPkt` packet reference for channel extraction
- Picker installed via `registerPage` init hooks in both `live.js` and
`packets.js`
- Single shared popover DOM element, repositioned on each open

### Styling
- Dark card with border, matching existing CoreScope dropdown patterns
- CSS in `style.css` under `.cc-picker-*` classes
- Uses CSS variables (`--surface-1`, `--border`, `--accent`, etc.) for
theme compatibility

## Files changed

| File | Change |
|------|--------|
| `public/channel-color-picker.js` | New — popover component (IIFE, no
dependencies except `ChannelColors`) |
| `public/index.html` | Script tag for picker |
| `public/live.js` | Store `_ccPkt` on feed items, install picker on
init |
| `public/packets.js` | Install picker on init, expose
`_packetsRenderVisible` |
| `public/style.css` | Popover CSS |
| `test-channel-colors.js` | 2 new tests for picker loading and graceful
degradation |

## Testing

- All 21 channel-colors tests pass (19 M1 + 2 M2)
- All 445 frontend-helpers tests pass
- All 62 packet-filter tests pass

## Performance

No hot-path impact. The popover is a single shared DOM element created
lazily on first use. Context menu handlers use event delegation on the
feed/table containers (one listener each, not per-row). The
`refreshVisibleRows` function only iterates currently-visible DOM
elements.

Closes milestone M2 of #271.

---------

Co-authored-by: you <you@example.com>
2026-04-05 06:45:13 -07:00
you dc635775b5 docs: TUI spec updated with expert feedback + MVP definition 2026-04-05 07:12:11 +00:00
you 8a94c43334 docs: startup performance spec — serve HTTP within 2 minutes on any DB size 2026-04-05 07:09:55 +00:00
you 6aaa5cdc20 docs: add user guide — getting started, pages, config, FAQ 2026-04-05 07:09:54 +00:00
you 788005bff7 docs: clarify Docker tag strategy — pin to vX.Y.Z for production, edge for testing 2026-04-05 07:09:44 +00:00
you af03f9aa57 docs: deployment simplification spec — pre-built Docker images + one-line deploy 2026-04-05 07:06:35 +00:00
Kpa-clawbot 3328ca4354 feat: channel color highlighting M1 — core model + feed row (#271) (#607)
## Summary

Implements M1 of the [channel color highlighting
spec](docs/specs/channel-color-highlighting.md) for issue #271.

Allows users to assign custom highlight colors to specific hash
channels. When a `GRP_TXT` packet arrives with an assigned channel
color, the feed row and packets table row get:
- **4px colored left border** in the assigned color
- **Subtle background tint** (color at 10% opacity)

## What's included

### `public/channel-colors.js` — Storage model
- `ChannelColors.get(channel)` → hex color or null
- `ChannelColors.set(channel, color)` — assign a color
- `ChannelColors.remove(channel)` — clear assignment
- `ChannelColors.getAll()` → all assignments
- `ChannelColors.getRowStyle(typeName, channel)` → inline CSS string for
row highlighting
- Uses `localStorage` key `live-channel-colors`
- Gracefully handles corrupt/missing localStorage data

### Feed row highlighting (`public/live.js`)
- Both `addFeedItem` (live WS) and `addFeedItemDOM` (replay/DB load)
apply channel color styles
- Reads `decoded.payload.channelName` from the packet

### Packets table highlighting (`public/packets.js`)
- `buildFlatRowHtml` and `buildGroupRowHtml` apply channel color styles
to `<tr>` elements
- Reads channel from `getParsedDecoded(p).channel`

### Tests (`test-channel-colors.js`)
- 16 unit tests covering storage CRUD, edge cases (null, empty, corrupt
data), and style generation
- Tests verify only GRP_TXT/CHAN types get coloring, other types are
unaffected

## Design decisions

- **Only GRP_TXT/CHAN packets** — other types retain default
`TYPE_COLORS` styling
- **Channel color takes priority** over default type colors for row
highlighting
- **No UI for assigning colors yet** — that's M2 (right-click context
menu + color picker)
- **Storage key abstracted** behind functions to ease future migration
if customizer rework (#288) lands
- **10% opacity tint** (`#hexcolor` + `1a` suffix) ensures readability
in both dark/light modes

## Performance

- `getRowStyle()` is O(1) — single localStorage read + JSON parse per
call
- No per-packet API calls; all data is client-side
- No impact on hot rendering paths beyond one localStorage read per row
render

Closes #271 (M1 only — further milestones in separate PRs)

---------

Co-authored-by: you <you@example.com>
2026-04-05 00:03:17 -07:00
you 14732135b7 docs: proposal for terminal/TUI interface into CoreScope 2026-04-05 06:56:33 +00:00
Kpa-clawbot e42477b810 feat: collapsible panels + medium breakpoint on live map (#606)
## Summary

Adds collapsible/minimizable UI panels on the live map page so overlay
panels don't block map content on medium-sized screens.

Fixes #279

## Changes

### Collapsible Legend Panel (all screen sizes)
- The legend toggle button (🎨/✕) is now visible at **all** screen sizes,
not just mobile
- Clicking it smoothly collapses/expands the legend with a CSS
transition
- Collapsed state persists in `localStorage` (`live-legend-hidden`)
- Feed panel already had hide/show with localStorage — no changes needed
there

### Medium Breakpoint (768px)
New `@media (max-width: 768px)` rules for tablet/small laptop screens:
- Feed panel: 360px → 280px wide, max-height 340px → 200px
- Node detail panel: 320px → 260px wide
- Legend: smaller font (10px) and tighter padding
- Header: reduced gap and padding
- Stats/toggles: smaller font sizes

### What's NOT changed
- Mobile (≤640px): existing behavior preserved (feed/legend hidden
entirely)
- Desktop (>768px): no changes — panels render at full size as before

## Testing
- `test-packet-filter.js`: 62 passed
- `test-aging.js`: 29 passed  
- `test-frontend-helpers.js`: 445 passed

---------

Co-authored-by: you <you@example.com>
2026-04-04 23:56:07 -07:00
you cbc3e3ce13 docs: movable UI panels spec — draggable panel positioning (#279) 2026-04-05 06:54:45 +00:00
you 1796493ec0 docs: channel color highlighting spec (#271)
Custom color assignment for hash channels in Live tab.
Reviewed by Tufte, Torvalds, and Doshi personas.
2026-04-05 06:45:53 +00:00
you 168866ecb6 fix: View Route on Map button works on packet detail page
The button click handler used document.getElementById() which fails on
/packet/[ID] pages because renderDetail() runs before the container is
appended to the DOM. Changed to panel.querySelector() which searches
within the detached element tree.

Fixes #601
2026-04-05 06:43:59 +00:00
you be9257cd26 chore: switch license to GPL v3
Copyleft ensures all derivative works remain open source.
2026-04-05 06:36:03 +00:00
you b5b6faf90a chore: switch license from MIT to Apache 2.0
Adds patent protection for contributors while maintaining the same
permissive usage rights.
2026-04-05 06:35:38 +00:00
you 592061ec7e chore: add MIT license 2026-04-05 06:32:28 +00:00
you 596ccf2322 fix(rf-health): offset TX/RX airtime labels when overlapping
When TX and RX values are within 12px, TX label shifts up and RX shifts
down to avoid rendering on top of each other.
2026-04-05 06:31:02 +00:00
Kpa-clawbot 232770a858 feat(rf-health): M2 — airtime, error rate, battery charts with delta computation (#605)
## M2: Airtime + Channel Quality + Battery Charts

Implements M2 of #600 — server-side delta computation and three new
charts in the RF Health detail view.

### Backend Changes

**Delta computation** for cumulative counters (`tx_air_secs`,
`rx_air_secs`, `recv_errors`):
- Computes per-interval deltas between consecutive samples
- **Reboot handling:** detects counter reset (current < previous), skips
that delta, records reboot timestamp
- **Gap handling:** if time between samples > 2× interval, inserts null
(no interpolation)
- Returns `tx_airtime_pct` and `rx_airtime_pct` as percentages
(delta_secs / interval_secs × 100)
- Returns `recv_error_rate` as delta_errors / (delta_recv +
delta_errors) × 100

**`resolution` query param** on `/api/observers/{id}/metrics`:
- `5m` (default) — raw samples
- `1h` — hourly aggregates (GROUP BY hour with AVG/MAX)
- `1d` — daily aggregates

**Schema additions:**
- `packets_sent` and `packets_recv` columns added to `observer_metrics`
(migration)
- Ingestor parses these fields from MQTT stats messages

**API response** now includes:
- `tx_airtime_pct`, `rx_airtime_pct`, `recv_error_rate` (computed
deltas)
- `reboots` array with timestamps of detected reboots
- `is_reboot_sample` flag on affected samples

### Frontend Changes

Three new charts in the RF Health detail view, stacked vertically below
noise floor:

1. **Airtime chart** — TX (red) + RX (blue) as separate SVG lines,
Y-axis 0-100%, direct labels at endpoints
2. **Error Rate chart** — `recv_error_rate` line, shown only when data
exists
3. **Battery chart** — voltage line with 3.3V low reference, shown only
when battery_mv > 0

All charts:
- Share X-axis and time range (aligned vertically)
- Reboot markers as vertical hairlines spanning all charts
- Direct labels on data (no legends)
- Resolution auto-selected: `1h` for 7d/30d ranges
- Charts hidden when no data exists

### Tests

- `TestComputeDeltas`: normal deltas, reboot detection, gap detection
- `TestGetObserverMetricsResolution`: 5m/1h/1d downsampling verification
- Updated `TestGetObserverMetrics` for new API signature

---------

Co-authored-by: you <you@example.com>
2026-04-04 23:17:17 -07:00
you 747aea37b7 fix(rf-health): add region filter support to metrics summary
Frontend passes RegionFilter query string to summary API.
Backend filters results by observer IATA region.
Added iata field to MetricsSummaryRow.
2026-04-05 06:00:42 +00:00
you 968c104e14 feat(rf-health): show observer detail in side panel instead of page bottom
- Change RF Health detail view from bottom-of-page to a right-sliding side panel
- Grid stays visible and stable when detail is open (no layout shift)
- Click another observer updates panel in place; close button (×) dismisses
- On mobile (<640px): panel stacks below grid at full width
- Filter out observers with insufficient data (<2 sparkline points) from grid entirely
- Follows the same split-layout pattern used by the nodes page
2026-04-05 05:53:42 +00:00
Kpa-clawbot 6f35d4d417 feat: RF Health Dashboard M1 — observer metrics + small multiples grid (#604)
## RF Health Dashboard — M1: Observer Metrics Storage, API & Small
Multiples Grid

Implements M1 of #600.

### What this does

Adds a complete RF health monitoring pipeline: MQTT stats ingestion →
SQLite storage → REST API → interactive dashboard with small multiples
grid.

### Backend Changes

**Ingestor (`cmd/ingestor/`)**
- New `observer_metrics` table via migration system (`_migrations`
pattern)
- Parse `tx_air_secs`, `rx_air_secs`, `recv_errors` from MQTT status
messages (same pattern as existing `noise_floor` and `battery_mv`)
- `INSERT OR REPLACE` with timestamps rounded to nearest 5-min interval
boundary (using ingestor wall clock, not observer timestamps)
- Missing fields stored as NULLs — partial data is always better than no
data
- Configurable retention pruning: `retention.metricsDays` (default 30),
runs on startup + every 24h

**Server (`cmd/server/`)**
- `GET /api/observers/{id}/metrics?since=...&until=...` — per-observer
time-series data
- `GET /api/observers/metrics/summary?window=24h` — fleet summary with
current NF, avg/max NF, sample count
- `parseWindowDuration()` supports `1h`, `24h`, `3d`, `7d`, `30d` etc.
- Server-side metrics retention pruning (same config, staggered 2min
after packet prune)

### Frontend Changes

**RF Health tab (`public/analytics.js`, `public/style.css`)**
- Small multiples grid showing all observers simultaneously — anomalies
pop out visually
- Per-observer cell: name, current NF value, battery voltage, sparkline,
avg/max stats
- NF status coloring: warning (amber) at ≥-100 dBm, critical (red) at
≥-85 dBm — text color only, no background fills
- Click any cell → expanded detail view with full noise floor line chart
- Reference lines with direct text labels (`-100 warning`, `-85
critical`) — not color bands
- Min/max points labeled directly on the chart
- Time range selector: preset buttons (1h/3h/6h/12h/24h/3d/7d/30d) +
custom from/to datetime picker
- Deep linking: `#/analytics?tab=rf-health&observer=...&range=...`
- All charts use SVG, matching existing analytics.js patterns
- Responsive: 3-4 columns on desktop, 1 on mobile

### Design Decisions (from spec)
- Labels directly on data, not in legends
- Reference lines with text labels, not color bands
- Small multiples grid, not card+accordion (Tufte: instant visual fleet
comparison)
- Ingestor wall clock for all timestamps (observer clocks may drift)

### Tests Added

**Ingestor tests:**
- `TestRoundToInterval` — 5 cases for rounding to 5-min boundaries
- `TestInsertMetrics` — basic insertion with all fields
- `TestInsertMetricsIdempotent` — INSERT OR REPLACE deduplication
- `TestInsertMetricsNullFields` — partial data with NULLs
- `TestPruneOldMetrics` — retention pruning
- `TestExtractObserverMetaNewFields` — parsing tx_air_secs, rx_air_secs,
recv_errors

**Server tests:**
- `TestGetObserverMetrics` — time-series query with since/until filters,
NULL handling
- `TestGetMetricsSummary` — fleet summary aggregation
- `TestObserverMetricsAPIEndpoints` — DB query verification
- `TestMetricsAPIEndpoints` — HTTP endpoint response shape
- `TestParseWindowDuration` — duration parsing for h/d formats

### Test Results
```
cd cmd/ingestor && go test ./... → PASS (26s)
cd cmd/server && go test ./... → PASS (5s)
```

### What's NOT in this PR (deferred to M2+)
- Server-side delta computation for cumulative counters
- Airtime charts (TX/RX percentage lines)
- Channel quality chart (recv_error_rate)
- Battery voltage chart
- Reboot detection and chart annotations
- Resolution downsampling (1h, 1d aggregates)
- Pattern detection / automated diagnosis

---------

Co-authored-by: you <you@example.com>
2026-04-04 22:21:35 -07:00
you aaf00d0616 docs: add M5 Prometheus/Grafana metrics export to RF Health spec 2026-04-05 05:02:36 +00:00
you 41c046c974 docs: RF Health Dashboard spec — observer radio metrics
Per-observer time-series charts for noise floor, TX/RX airtime, CRC errors,
and battery. Small multiples grid design. MVP-first milestones.

Reviewed by Carmack (perf), Munger (failure modes), radio expert (hardware),
Tufte (visualization), and Doshi (product strategy).
2026-04-05 04:42:32 +00:00
efiten 1fbdd1c3d3 feat: Prefix Tool tab on Analytics page (#347) (#599)
## Summary

- Adds a new **Prefix Tool** tab to the Analytics page (alongside Hash
Stats / Hash Issues)
- **Network Overview**: per-tier collision stats (1/2/3-byte) and a
network-size-based recommendation — collapsible, folded by default
- **Prefix Checker**: accepts a 1/2/3-byte hex prefix or full public
key; shows colliding nodes at each tier with severity badges ( / ⚠️ /
🔴); clicking a node navigates to its detail page
- **Prefix Generator**: picks a random collision-free prefix at the
chosen hash size; links to
[meshcore-web-keygen](https://agessaman.github.io/meshcore-web-keygen/)
with the prefix pre-filled
- **Hash Issues tab**: adds a "🔎 Check a prefix →" shortcut in the nav
- **Deep-link support**: `#/analytics?tab=prefix-tool&prefix=A3F1`
pre-fills and runs the checker; `?generate=2` pre-selects and runs the
generator
- **No new API endpoints** — 100% client-side using the existing
`/nodes` list

## Verification

Live on staging:
**https://staging.on8ar.eu/#/analytics?tab=prefix-tool**

## Test plan

- [x] Network Overview card is collapsed by default; expands on click;
stats are correct
- [x] Prefix Checker: 2-char input shows 1-byte results; 4-char shows
2-byte; 6-char shows 3-byte; 64-char pubkey shows all three tiers
- [x] Prefix Checker: invalid hex shows error; odd-length input shows
error
- [x] Prefix Generator: Generate picks an unused prefix; "Try another"
cycles; keygen link opens with prefix pre-filled
- [x] Deep link `?prefix=A3F1` pre-fills checker and scrolls to it
- [x] Deep link `?generate=2` pre-selects 2-byte and runs generator
- [x] Hash Issues tab shows "🔎 Check a prefix →" in the nav
- [x] FAQ link at bottom of generator opens correct MeshCore docs anchor

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 20:18:32 -07:00
efiten d34320fa6c fix: use _getColCount() in error-state row to match spacers (#406) (#597)
## Summary

The error-state `<tbody>` row (shown when packet loading fails)
hardcoded `colspan="10"`, while the virtual scroll spacers and the
empty-state row both use `_getColCount()` (which reads from the actual
`<thead>` and falls back to 11). One-line fix: replace the hardcoded
value with `_getColCount()`.

Fixes #406

## Test plan

- [x] Trigger the error state (e.g. kill the backend mid-load) — error
row should span all columns with no gap on the right
- [x] `node test-packets.js` — 72 passed, 0 failed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 19:41:55 -07:00
efiten 77b7c33d0f perf: incremental DOM diff in renderVisibleRows (#414) (#596)
## Summary

- Replace full \`tbody\` teardown+rebuild on every scroll frame with a
range-diff that only adds/removes the delta rows at the edges of the
visible window
- \`buildFlatRowHtml\` / \`buildGroupRowHtml\` now accept an
\`entryIdx\` parameter and emit \`data-entry-idx\` on every \`<tr>\` so
the diff can target rows precisely (including expanded group children)
- Full rebuild is retained for initial render and large scroll jumps
past the buffer (no range overlap)
- Also loads \`packet-helpers.js\` in the test sandbox, fixing 7
pre-existing test failures for the builder functions; adds 4 new tests
covering \`data-entry-idx\` output

Fixes #414

## Test plan

- [x] Open packets page with 500+ packets, scroll rapidly — DOM
inspector should show incremental \`<tr>\` adds/removes rather than full
\`tbody\` teardown
- [x] Expand a grouped packet, scroll away and back — expanded children
re-render correctly
- [x] Large scroll jump (jump to bottom via scrollbar) — full rebuild
fires, no visual glitch
- [x] \`node test-packets.js\` — 72 passed, 0 failed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
2026-04-04 19:41:33 -07:00
you 0a55717283 docs: add PSK brute-force attack with timestamp oracle to security analysis
Weak passphrases with no KDF stretching are the #1 practical threat.
Timestamp in plaintext block 0 serves as known-plaintext oracle for
instant key verification from a single captured packet.

Key findings:
- decode_base64() output used directly as AES key, no KDF
- Short passphrases produce <16 byte keys (reduced key space)
- No salt means global precomputed attacks work
- 3-word passphrase crackable in ~2 min on commodity GPU

Reviewed by djb and Dijkstra personas. Corrections applied:
- GPU throughput upgraded from 10^9 to 10^10 AES/sec baseline
- Oracle strengthened: bytes 4+ (type byte, sender name) also predictable
- Dictionary size assumptions made explicit
- Zipf's law caveat added (humans don't choose uniformly)
- base64 short-passphrase key truncation issue documented
2026-04-05 00:58:57 +00:00
you bcab31bf72 docs: AES-128-ECB security analysis — block-level vulnerability assessment
Formal analysis of MeshCore's ECB encryption for channel and direct messages.
Reviewed by djb and Dijkstra expert personas through 3 revisions.

Key findings:
- Block 0 has accidental nonce (4-byte timestamp) preventing repetition
- Blocks 1+ are pure deterministic ECB with no nonce — vulnerable to
  frequency analysis for repeated message content
- Partial final block attack: zero-padding reduces search space
- HMAC key reuse: AES key is first 16 bytes of HMAC key (same material)
- Recommended fix: switch to AES-128-CTR mode
2026-04-05 00:44:21 +00:00
Kpa-clawbot 6ae62ce535 perf: make txToMap observations lazy via ExpandObservations flag (#595)
## Summary

`txToMap()` previously always allocated observation sub-maps for every
packet, even though the `/api/packets` handler immediately stripped them
via `delete(p, "observations")` unless `expand=observations` was
requested. A typical page of 50 packets with ~5 observations each caused
300+ unnecessary map allocations per request.

## Changes

- **`txToMap`**: Add variadic `includeObservations bool` parameter.
Observations are only built when `true` is passed, eliminating
allocations when they'd just be discarded.
- **`PacketQuery`**: Add `ExpandObservations bool` field to thread the
caller's intent through the query pipeline.
- **`routes.go`**: Set `ExpandObservations` based on
`expand=observations` query param. Removed the post-hoc `delete(p,
"observations")` loop — observations are simply never created when not
requested.
- **Single-packet lookups** (`GetPacketByID`, `GetPacketByHash`): Always
pass `true` since detail views need observations.
- **Multi-node/analytics queries**: Default (no flag) = no observations,
matching prior behavior.

## Testing

- Added `TestTxToMapLazyObservations` covering all three cases: no flag,
`false`, and `true`.
- All existing tests pass (`go test ./...`).

## Perf Impact

Eliminates ~250 observation map allocations per /api/packets request (at
default page size of 50 with ~5 observations each). This is a
constant-factor improvement per request — no algorithmic complexity
change.

Fixes #374

Co-authored-by: you <you@example.com>
2026-04-04 10:39:30 -07:00
Kpa-clawbot 6e2f79c0ad perf: optimize QueryGroupedPackets — cache observer count, defer map construction (#594)
## Summary

Optimizes `QueryGroupedPackets()` in `store.go` to eliminate two major
inefficiencies on every grouped packet list request:

### Changes

1. **Cache `UniqueObserverCount` on `StoreTx`** — Instead of iterating
all observations to count unique observers on every query
(O(total_observations) per request), we now track unique observers at
ingest time via an `observerSet` map and pre-computed
`UniqueObserverCount` field. This is updated incrementally as
observations arrive.

2. **Defer map construction until after pagination** — Previously,
`map[string]interface{}` was built for ALL 30K+ filtered results before
sorting and paginating. Now the grouped cache stores sorted `[]*StoreTx`
pointers (lightweight), and `groupedTxsToPage()` builds maps only for
the requested page (typically 50 items). This eliminates ~30K map
allocations per cache miss.

3. **Lighter cache footprint** — The grouped cache now stores
`[]*StoreTx` instead of `*PacketResult` with pre-built maps, reducing
memory pressure and GC work.

### Complexity

- Observer counting: O(1) per query (was O(total_observations))
- Map construction: O(page_size) per query (was O(n) where n = all
filtered results)
- Sort remains O(n log n) on cache miss, but the cache (3s TTL) absorbs
repeated requests

### Testing

- `cd cmd/server && go test ./...` — all tests pass
- `cd cmd/ingestor && go build ./...` — builds clean

Fixes #370

---------

Co-authored-by: you <you@example.com>
2026-04-04 10:39:04 -07:00
Kpa-clawbot b0862f7a41 fix: replace time.Tick with NewTicker in prune goroutine for graceful shutdown (#593)
## Summary

Replace `time.Tick()` with `time.NewTicker()` in the auto-prune
goroutine so it stops cleanly during graceful shutdown.

## Problem

`time.Tick` creates a ticker that can never be garbage collected or
stopped. While the prune goroutine runs for the process lifetime, it
won't stop during graceful shutdown — the goroutine leaks past the
shutdown sequence.

## Fix

- Create a `time.NewTicker` and a done channel
- Use `select` to listen on both the ticker and done channel
- Stop the ticker and close the done channel in the shutdown path (after
`poller.Stop()`)
- Pattern matches the existing `StartEvictionTicker()` approach

## Testing

- `go build ./...` — compiles cleanly
- `go test ./...` — all tests pass

Fixes #377

Co-authored-by: you <you@example.com>
2026-04-04 10:38:37 -07:00
Kpa-clawbot 45991eca09 perf: combine chained filterPackets passes into single scan (#592)
## Summary

Combines the chained `filterTxSlice` calls in `filterPackets()` into a
single pass over the packet slice.

## Problem

When multiple filter parameters are specified (e.g.,
`type=4&route=1&since=...&until=...`), each filter created a new
intermediate `[]*StoreTx` slice. With N filters, this meant N separate
scans and N-1 unnecessary allocations.

## Fix

All filter predicates (type, route, observer, hash, since, until,
region, node) are pre-computed before the loop, then evaluated in a
single `filterTxSlice` call. This eliminates all intermediate
allocations.

**Preserved behavior:**
- Fast-path index lookups for hash-only and observer-only queries remain
unchanged
- Node-only fast-path via `byNode` index preserved
- All existing filter semantics maintained (same comparison operators,
same null checks)

**Complexity:** Single `O(n)` pass regardless of how many filters are
active, vs previous `O(n * k)` where k = number of active filters (each
pass is O(n) but allocates).

## Testing

All existing tests pass (`cd cmd/server && go test ./...`).

Fixes #373

Co-authored-by: you <you@example.com>
2026-04-04 10:38:10 -07:00
Kpa-clawbot 76c42556a2 perf: sort snrVals/rssiVals once in computeAnalyticsRF (#591)
## Summary

Sort `snrVals` and `rssiVals` once upfront in `computeAnalyticsRF()` and
read min/max/median directly from the sorted slices, instead of copying
and sorting per stat call.

## Changes

- Sort both slices once before computing stats (2 sorts total instead of
4+ copy+sorts)
- Read `min` from `sorted[0]`, `max` from `sorted[len-1]`, `median` from
`sorted[len/2]`
- Remove the now-unused `sortedF64` and `medianF64` helper closures

## Performance impact

With 100K+ observations, this eliminates multiple O(n log n) copy+sort
operations. Previously each call to `medianF64` did a full copy + sort,
and `minF64`/`maxF64` did O(n) scans on the unsorted array. Now: 2
in-place sorts total, O(1) lookups for min/max/median.

Fixes #366

Co-authored-by: you <you@example.com>
2026-04-04 10:37:42 -07:00
Kpa-clawbot 6f8378a31c perf: batch-remove from secondary indexes in EvictStale (#590)
## Summary

`EvictStale()` was doing O(n) linear scans per evicted item to remove
from secondary indexes (`byObserver`, `byPayloadType`, `byNode`).
Evicting 1000 packets from an observer with 50K observations meant 1000
× 50K = 50M comparisons — all under a write lock.

## Fix

Replace per-item removal with batch single-pass filtering:

1. **Collect phase**: Walk evicted packets once, building sets of
evicted tx IDs, observation IDs, and affected index keys
2. **Filter phase**: For each affected index slice, do a single pass
keeping only non-evicted entries

**Before**: O(evicted_count × index_slice_size) per index — quadratic in
practice
**After**: O(evicted_count + index_slice_size) per affected key — linear

## Changes

- `cmd/server/store.go`: Restructured `EvictStale()` eviction loop into
collect + batch-filter pattern

## Testing

- All existing tests pass (`cd cmd/server && go test ./...`)

Fixes #368

Co-authored-by: you <you@example.com>
2026-04-04 10:37:27 -07:00
Kpa-clawbot 56115ee0a4 perf: use byNode index in QueryMultiNodePackets instead of full scan (#589)
## Summary

`QueryMultiNodePackets()` was scanning ALL packets with
`strings.Contains` on JSON blobs — O(packets × pubkeys × json_length).
With 30K+ packets and multiple pubkeys, this caused noticeable latency
on `/api/packets?nodes=...`.

## Fix

Replace the full scan with lookups into the existing `byNode` index,
which already maps pubkeys to their transmissions. Merge results with
hash-based deduplication, then apply time filters.

**Before:** O(N × P × J) where N=all packets, P=pubkeys, J=avg JSON
length
**After:** O(M × P) where M=packets per pubkey (typically small), plus
O(R log R) sort for pagination correctness

Results are sorted by `FirstSeen` after merging to maintain the
oldest-first ordering expected by the pagination logic.

Fixes #357

Co-authored-by: you <you@example.com>
2026-04-04 10:36:59 -07:00
Kpa-clawbot 321d1cf913 perf: apply time filter early in GetNodeAnalytics to avoid full packet scan (#588)
## Problem

`GetNodeAnalytics()` in `store.go` scans ALL 30K+ packets doing
`strings.Contains` on every JSON blob when the node has a name, then
filters by time range *after* the full scan. This is `O(packets ×
json_length)` on every `/api/nodes/{pubkey}/analytics` request.

## Fix

Move the `fromISO` time check inside the scan loop so old packets are
skipped **before** the expensive `strings.Contains` matching. For the
non-name path (indexed-only), the time filter is also applied inline,
eliminating the separate `allPkts` intermediate slice.

### Before
1. Scan all packets → collect matches (including old ones) → `allPkts`
2. Filter `allPkts` by time → `packets`

### After
1. Scan packets, skip `tx.FirstSeen <= fromISO` immediately → `packets`

This avoids `strings.Contains` calls on packets outside the requested
time window (typically 7 days out of months of data).

## Complexity
- **Before:** `O(total_packets × avg_json_length)` for name matching
- **After:** `O(recent_packets × avg_json_length)` — only packets within
the time window are string-matched

## Testing
- `cd cmd/server && go test ./...` — all tests pass

Fixes #367

Co-authored-by: you <you@example.com>
2026-04-04 10:36:49 -07:00
Kpa-clawbot 790a713ba9 perf: combine 4 subpath API calls into single bulk endpoint (#587)
## Summary

Consolidates the 4 parallel `/api/analytics/subpaths` calls in the Route
Patterns tab into a single `/api/analytics/subpaths-bulk` endpoint,
eliminating 3 redundant server-side scans of the subpath index on cache
miss.

## Changes

### Backend (`cmd/server/routes.go`, `cmd/server/store.go`)
- New `GET
/api/analytics/subpaths-bulk?groups=2-2:50,3-3:30,4-4:20,5-8:15`
endpoint
- Groups format: `minLen-maxLen:limit` comma-separated
- `GetAnalyticsSubpathsBulk()` iterates `spIndex` once, bucketing
entries into per-group accumulators by hop length
- Hop name resolution is done once per raw hop and shared across groups
- Results are cached per-group for compatibility with existing
single-key cache lookups
- Region-filtered queries fall back to individual
`GetAnalyticsSubpaths()` calls (region filtering requires
per-transmission observer checks)

### Frontend (`public/analytics.js`)
- `renderSubpaths()` now makes 1 API call instead of 4
- Response shape: `{ results: [{ subpaths, totalPaths }, ...] }` —
destructured into the same `[d2, d3, d4, d5]` variables

### Tests (`cmd/server/routes_test.go`)
- `TestAnalyticsSubpathsBulk`: validates 3-group response shape, missing
params error, invalid format error

## Performance

- **Before:** 4 API calls → 4 scans of `spIndex` + 4× hop resolution on
cache miss
- **After:** 1 API call → 1 scan of `spIndex` + 1× hop resolution
(shared cache)
- Cache miss cost reduced by ~75% for this tab
- No change on cache hit (individual group caching still works)

Fixes #398

Co-authored-by: you <you@example.com>
2026-04-04 10:19:18 -07:00
Kpa-clawbot cd470dffbe perf: batch observation fetching to eliminate N+1 API calls on sort change (#586)
## Summary

Fixes the N+1 API call pattern when changing observation sort mode on
the packets page. Previously, switching sort to Path or Time fired
individual `/api/packets/{hash}` requests for **every**
multi-observation group without cached children — potentially 100+
concurrent requests.

## Changes

### Backend: Batch observations endpoint
- **New endpoint:** `POST /api/packets/observations` accepts `{"hashes":
["h1", "h2", ...]}` and returns all observations keyed by hash in a
single response
- Capped at 200 hashes per request to prevent abuse
- 4 test cases covering empty input, invalid JSON, too-many-hashes, and
valid requests

### Frontend: Use batch endpoint
- `packets.js` sort change handler now collects all hashes needing
observation data and sends a single POST request instead of N individual
GETs
- Same behavior, single round-trip

## Performance

- **Before:** Changing sort with 100 visible groups → 100 concurrent API
requests, browser connection queueing (6 per host), several seconds of
lag
- **After:** Single POST request regardless of group count, response
time proportional to store lookup (sub-millisecond per hash in memory)

Fixes #389

---------

Co-authored-by: you <you@example.com>
2026-04-04 10:18:40 -07:00
Kpa-clawbot 7ff89d8607 perf(packets): coalesce WS-triggered renders with requestAnimationFrame (#585)
## Summary

Coalesce WS-triggered `renderTableRows()` calls using
`requestAnimationFrame` instead of `setTimeout` debouncing.

Fixes #396

## Problem

During high WebSocket throughput, multiple WS batches could each trigger
a `renderTableRows()` call via `setTimeout(..., 200)`. With rapid
batches, this caused the 50K-row table to be fully rebuilt every few
hundred milliseconds, causing UI jank.

## Solution

Replace the `setTimeout`-based debounce with a `requestAnimationFrame`
coalescing pattern:

1. **`scheduleWSRender()`** — sets a dirty flag and schedules a single
rAF callback
2. **Dirty flag** — multiple WS batches within the same frame just set
the flag; only one render fires
3. **Cleanup** — `destroy()` cancels any pending rAF and resets the
dirty flag

This ensures at most **one `renderTableRows()` per animation frame**
(~16ms), regardless of how many WS batches arrive.

## Performance justification

- **Before:** Each WS batch → `setTimeout(renderTableRows, 200)` — N
batches in <200ms = N renders
- **After:** N batches in one frame → 1 render on next rAF (~16ms)
- Worst case goes from O(N) renders per second to O(60) renders per
second (frame-capped)

## Changes

- `public/packets.js`: Add `scheduleWSRender()` with rAF + dirty flag;
replace setTimeout in WS handler; clean up in `destroy()`
- `test-frontend-helpers.js`: Update tests to verify rAF coalescing
pattern instead of setTimeout debounce

## Testing

- All existing tests pass (`npm test` — 0 failures)
- Updated 2 test cases to verify new rAF coalescing behavior

Co-authored-by: you <you@example.com>
2026-04-04 10:18:09 -07:00
Kpa-clawbot 493849f2e3 perf(frontend): compress og-image.png from 1.1MB to 235KB (#584)
## Summary

Compress `public/og-image.png` from **1,159,050 bytes (1.1MB)** to
**234,899 bytes (235KB)** — an **80% reduction**.

## What Changed

- Applied lossy PNG quantization via `pngquant` (quality 45-65, speed 1)
- Image dimensions unchanged: 1200×630px (standard OG image size)
- Visual quality remains suitable for social media previews

## Why

A 1.1MB OpenGraph image is excessive. Typical OG images are 50-200KB.
This reduces deployment size and Git repo bloat without affecting
functionality (browsers don't preload OG images).

## Testing

- Unit tests pass (`npm run test:unit`)
- No code changes — image-only commit
- `index.html` reference unchanged (`<meta property="og:image"
content="/og-image.png">`)

Fixes #397

Co-authored-by: you <you@example.com>
2026-04-04 10:17:21 -07:00
Kpa-clawbot 87ac61748c perf(analytics): compute network status client-side, eliminate redundant API call (#583)
## Summary

Reduces the analytics nodes tab from 3 parallel API calls to 2 by
computing network status (active/degraded/silent counts) client-side
instead of fetching from `/nodes/network-status`.

## What Changed

**`public/analytics.js` — `renderNodesTab()`:**
- Removed the `/nodes/network-status` API call from the `Promise.all`
batch
- Added client-side computation of active/degraded/silent counts using
the shared `getHealthThresholds()` function from `roles.js`
- Uses `nodesResp.total` and `nodesResp.counts` (already returned by
`/nodes` endpoint) for total node count and role breakdown

## Why This Works

The `/nodes` response already includes:
- `total` — count of all matching nodes (server-computed across full DB)
- `counts` — role counts across all nodes (from `GetAllRoleCounts()`)
- Per-node `last_seen`/`last_heard` timestamps

The `getHealthThresholds()` function in `roles.js` provides the same
degraded/silent thresholds used server-side, so client-side status
computation produces equivalent results for the loaded node set.

## Performance

- **Before:** 3 parallel API calls (`/nodes`, `/nodes/bulk-health`,
`/nodes/network-status`)
- **After:** 2 parallel API calls (`/nodes`, `/nodes/bulk-health`)
- Network status computation is O(n) over the 200 loaded nodes —
negligible client-side cost
- The `/nodes/network-status` endpoint scanned ALL nodes in the DB on
every call; this eliminates that server-side work entirely

## Testing

- All frontend helper tests pass (445/445)
- All packet filter tests pass (62/62)  
- All aging tests pass (29/29)
- All Go backend tests pass

Fixes #392

---------

Co-authored-by: you <you@example.com>
2026-04-04 10:17:05 -07:00
Kpa-clawbot 26de38f4b6 perf(map): reposition markers on zoom/resize instead of full rebuild (#582)
## Summary

Eliminates visible marker flicker on zoom/resize events in the map page
when displaying 500+ nodes.

## Problem

`renderMarkers()` was called on every `zoomend` and `resize` event,
which did `markerLayer.clearLayers()` followed by a full rebuild of all
markers. With many nodes, this caused a visible flash where all markers
disappeared briefly before being re-added.

## Solution

Instead of rebuilding all markers from scratch on zoom/resize:

1. **Store Leaflet layer references** on marker data objects
(`_leafletMarker`, `_leafletLine`, `_leafletDot`) during the initial
full render
2. **Add `_repositionMarkers()`** — re-runs `deconflictLabels()` at the
new zoom level and updates existing marker positions via
`setLatLng()`/`setLatLngs()` without clearing the layer group
3. **Debounce zoom/resize handlers** (150ms) to coalesce rapid events
during animated zooms
4. **Dynamically manage offset indicators** — adds/removes deconfliction
offset lines and dots as positions change at different zoom levels

Full `renderMarkers()` is still called for filter changes, data updates,
and theme changes — only zoom/resize uses the lightweight repositioning
path.

## Complexity

- `_repositionMarkers()`: O(n) — single pass over stored marker data
- `deconflictLabels()`: O(n × k) where k is max spiral offsets (48) —
unchanged
- No new API calls, no DOM rebuilds

Fixes #393

---------

Co-authored-by: you <you@example.com>
2026-04-04 17:16:48 +00:00
Kpa-clawbot d2d4c504e8 perf(live): parallelize replayRecent() observation fetches (#581)
## Summary

`replayRecent()` in `live.js` fetched observation details for 8 packet
groups **sequentially** — each `await fetch()` waited for the previous
to complete before starting the next.

## Change

Replaced the sequential `for` loop with `Promise.all()` to fetch all 8
detail API calls **concurrently**. The mapping from results to live
packets is unchanged.

**Before:** 8 sequential fetches (total time ≈ sum of all request
durations)
**After:** 8 parallel fetches (total time ≈ max of all request
durations)

## Notes

- `replayRecent()` is currently disabled (commented out at line 856), so
this is dormant code — no runtime risk
- No behavioral change: same data mapping, same rendering, same VCR
buffer population
- All existing tests pass

Fixes #394

---------

Co-authored-by: you <you@example.com>
2026-04-04 10:16:08 -07:00
Kpa-clawbot b37e8e2da2 perf(packets): replace N+1 API calls with single expand=observations query (#580)
## Summary

Eliminates the N+1 API call storm when toggling off "Group by Hash" in
the packets table.

## Problem

When ungrouped mode was active, `loadPackets()` fired individual
`/api/packets/{hash}` requests for every multi-observation packet. With
200+ multi-obs packets, this created 200+ parallel HTTP requests —
overwhelming both browser connection limits and the server.

## Fix

The server already supports `expand=observations` on the `/api/packets`
endpoint, which returns observations inline. Instead of:

1. Always fetching grouped (`groupByHash=true`)
2. Then N+1 fetching each packet's children individually

We now:

1. Fetch grouped when grouped mode is active (`groupByHash=true`)
2. Fetch with `expand=observations` when ungrouped — **single API call**
3. Flatten observations client-side

**Result: 200+ API calls → 1 API call.**

## Changes

- `public/packets.js`: Replaced N+1 observation fetching loop with
single `expand=observations` query parameter, flatten inline
observations client-side.

## Testing

- All frontend tests pass (packet-filter: 62/62, frontend-helpers:
445/445)
- All Go backend tests pass

Fixes #382

Co-authored-by: you <you@example.com>
2026-04-04 10:15:14 -07:00
Kpa-clawbot 45d8116880 perf: query only matching node locations in handleObservers (#579)
## Summary

`handleObservers()` in `routes.go` was calling `GetNodeLocations()`
which fetches ALL nodes from the DB just to match ~10 observer IDs
against node public keys. With 500+ nodes this is wasteful.

## Changes

- **`db.go`**: Added `GetNodeLocationsByKeys(keys []string)` — queries
only the rows matching the given public keys using a parameterized
`WHERE LOWER(public_key) IN (?, ?, ...)` clause.
- **`routes.go`**: `handleObservers` now collects observer IDs and calls
the targeted method instead of the full-table scan.
- **`coverage_test.go`**: Added `TestGetNodeLocationsByKeys` covering
known key, empty keys, and unknown key cases.

## Performance

With ~10 observers and 500+ nodes, the query goes from scanning all 500
rows to fetching only ~10. The original `GetNodeLocations()` is
preserved for any other callers.

Fixes #378

Co-authored-by: you <you@example.com>
2026-04-04 10:14:37 -07:00
Kpa-clawbot f68e98c376 perf(live): skip updateTimeline() when tab is hidden (#578)
## Summary

Skip `updateTimeline()` canvas redraws in `bufferPacket()` when the
browser tab is hidden (`_tabHidden === true`). Instead, batch-update the
timeline once when the tab becomes visible again via the
`visibilitychange` handler.

Fixes #385

## What Changed

**`public/live.js`** — two surgical edits:

1. **`bufferPacket()`**: Removed `updateTimeline()` call from the
`_tabHidden` early-return path. When the tab is backgrounded, packets
are still buffered (for VCR) but no canvas work is done.

2. **`visibilitychange` handler**: Added `updateTimeline()` call when
the tab is restored, so the timeline catches up in a single repaint
instead of N repaints (one per buffered packet).

## Performance Impact

At 5+ packets/sec with a backgrounded tab, this eliminates continuous
canvas redraws (`updateTimeline()` calls `ctx.clearRect` + full canvas
redraw + `updateTimelinePlayhead()`) that are invisible to the user. CPU
usage drops to near-zero for timeline rendering while backgrounded.

## Tests

All existing tests pass:
- `test-packet-filter.js` — 62 passed
- `test-aging.js` — 29 passed  
- `test-frontend-helpers.js` — 445 passed

Co-authored-by: you <you@example.com>
2026-04-04 10:14:13 -07:00
Kpa-clawbot f3d5d1e021 perf: resolve hops from in-memory prefix map instead of N+1 DB queries (#577)
## Summary

Replace N+1 per-hop DB queries in `handleResolveHops` with O(1) lookups
against the in-memory prefix map that already exists in the packet
store.

## Problem

Each hop in the `resolve-hops` API triggered a separate `SELECT ... LIKE
?` query against the nodes table. With 10 hops, that's 10 DB round-trips
— unnecessary when `getCachedNodesAndPM()` already maintains an
in-memory prefix map that can resolve hops instantly.

## Changes

- **routes.go**: Replace the per-hop DB query loop with `pm.m[hopLower]`
lookups from the prefix map. Convert `nodeInfo` → `HopCandidate` inline.
Remove unused `rows`/`sql.Scan` code.
- **store.go**: Add `InvalidateNodeCache()` method to force prefix map
rebuild (needed by tests that insert nodes after store initialization).
- **routes_test.go**: Give `TestResolveHopsAmbiguous` a proper store so
hops resolve via the prefix map.
- **resolve_context_test.go**: Call `InvalidateNodeCache()` after
inserting test nodes. Fix confidence assertion — with GPS candidates and
no affinity context, `resolveWithContext` correctly returns
`gps_preference` (previously masked because the prefix map didn't have
the test nodes).

## Complexity

O(1) per hop lookup via hash map vs O(n) DB scan per hop. No hot-path
impact — this endpoint is called on-demand, not in a render loop.

Fixes #369

---------

Co-authored-by: you <you@example.com>
2026-04-04 09:51:07 -07:00
Kpa-clawbot 02004c5912 perf: incremental distance index update on path changes (#576)
## Summary

Replace full `buildDistanceIndex()` rebuild with incremental
`removeTxFromDistanceIndex`/`addTxToDistanceIndex` for only the
transmissions whose paths actually changed during
`IngestNewObservations`.

## Problem

When any transmission's best path changed during observation ingestion,
the **entire distance index was rebuilt** — iterating all 30K+ packets,
resolving all hops, and computing haversine distances. This
`O(total_packets × avg_hops)` operation ran under a write lock, blocking
all API readers.

A 30-second debounce (`distRebuildInterval`) was added in #557 to
mitigate this, but it only delayed the pain — the full rebuild still
happened, just less frequently.

## Fix

- Added `removeTxFromDistanceIndex(tx)` — filters out all
`distHopRecord` and `distPathRecord` entries for a specific transmission
- Added `addTxToDistanceIndex(tx)` — computes and appends new distance
records for a single transmission
- In `IngestNewObservations`, changed path-change handling to call
remove+add for each affected tx instead of marking dirty and waiting for
a full rebuild
- Removed `distDirty`, `distLast`, and `distRebuildInterval` since
incremental updates are cheap enough to apply immediately

## Complexity

- **Before:** `O(total_packets × avg_hops)` per rebuild (30K+ packets)
- **After:** `O(changed_txs × avg_hops + total_dist_records)` — the
remove is a linear scan of the distance slices, but only for affected
txs; the add is `O(hops)` per changed tx

The remove scan over `distHops`/`distPaths` slices is linear in slice
length, but this is still far cheaper than the full rebuild which also
does JSON parsing, hop resolution, and haversine math for every packet.

## Tests

- Updated `TestDistanceRebuildDebounce` →
`TestDistanceIncrementalUpdate` to verify incremental behavior and check
for duplicate path records
- All existing tests pass (`go test ./...` in both `cmd/server` and
`cmd/ingestor`)

Fixes #365

---------

Co-authored-by: you <you@example.com>
2026-04-04 09:50:55 -07:00
Kpa-clawbot ef30031e2e perf: cache resolveRegionObservers with 30s TTL (#575)
## Summary

Cache `resolveRegionObservers()` results with a 30-second TTL to
eliminate repeated database queries for region→observer ID mappings.

## Problem

`resolveRegionObservers()` queried the database on every call despite
the observers table changing infrequently (~20 rows). It's called from
10+ hot paths including `filterPackets()`, `GetChannels()`, and multiple
analytics compute functions. When analytics caches are cold, parallel
requests each hit the DB independently.

## Solution

- Added a dedicated `regionObsMu` mutex + `regionObsCache` map with 30s
TTL
- Uses a separate mutex (not `s.mu`) to avoid deadlocks — callers
already hold `s.mu.RLock()`
- Cache is lazily populated per-region and fully invalidated after TTL
expires
- Follows the same pattern as `getCachedNodesAndPM()` (30s TTL,
on-demand rebuild)

## Changes

- **`cmd/server/store.go`**: Added `regionObsMu`, `regionObsCache`,
`regionObsCacheTime` fields; rewrote `resolveRegionObservers()` to check
cache first; added `fetchAndCacheRegionObs()` helper
- **`cmd/server/coverage_test.go`**: Added
`TestResolveRegionObserversCaching` — verifies cache population, cache
hits, and nil handling for unknown regions

## Testing

- All existing Go tests pass (`go test ./...`)
- New test verifies caching behavior (population, hits, nil for unknown
regions)

Fixes #362

---------

Co-authored-by: you <you@example.com>
2026-04-04 09:50:27 -07:00
Kpa-clawbot 67511ed6a7 perf: combine GetStoreStats into 2 concurrent queries instead of 5 sequential (#574)
## Summary

`GetStoreStats()` ran 5 sequential DB queries on every call. This
combines them into **2 concurrent queries**:

1. **Node/observer counts** — single query using subqueries: `SELECT
(SELECT COUNT(*) FROM nodes WHERE ...), (SELECT COUNT(*) FROM nodes),
(SELECT COUNT(*) FROM observers)`
2. **Observation counts** — single query using conditional aggregation:
`SUM(CASE WHEN timestamp > ? THEN 1 ELSE 0 END)` scoped to the 24h
window, avoiding a full table scan for the 1h count

Both queries run concurrently via goroutines + `sync.WaitGroup`.

## What changed

- `cmd/server/store.go`: Rewrote `GetStoreStats()` — 5 sequential
`QueryRow` calls → 2 concurrent combined queries
- Error handling now propagates query errors instead of silently
ignoring them

## Performance justification

- **Before:** 5 sequential round-trips to SQLite, with 2 potentially
expensive `COUNT(*)` scans on the `observations` table
- **After:** 2 concurrent round-trips; the observation query scans the
24h window once instead of separately scanning for 1h and 24h
- The 10s cache (`statsTTL`) remains, so this fires at most once per 10s
— but when it does fire, it's ~2.5x fewer round-trips and the
observation scan is halved

## Tests

- `go test ./...` passes for both `cmd/server` and `cmd/ingestor`

Fixes #363

---------

Co-authored-by: you <you@example.com>
2026-04-04 09:48:25 -07:00
Kpa-clawbot b35b473508 perf(nodes): extract shared fetchNodeDetail() to deduplicate API calls (#573)
## Summary

Extracts a shared `fetchNodeDetail(pubkey)` helper in `nodes.js` that
fetches both `/nodes/{pubkey}` and `/nodes/{pubkey}/health` in parallel.
Both `selectNode()` (side panel) and `loadFullNode()` (full-screen view)
now call this single function instead of duplicating the fetch logic.

## What Changed

- **New:** `fetchNodeDetail(pubkey)` — shared async function that
returns node data with `.healthData` attached
- **Modified:** `loadFullNode()` — uses `fetchNodeDetail()` instead of
inline `Promise.all`
- **Modified:** `selectNode()` — uses `fetchNodeDetail()` instead of
inline `Promise.all`

## Why

The duplicate `api()` calls weren't a major perf issue (TTL caching
mitigates most cases), but the duplicated logic was unnecessary tech
debt. On mobile, `selectNode()` redirects to `loadFullNode()` via hash
change, so the two code paths could fire sequentially with expired
cache.

## Testing

- All frontend helper tests pass (445/445)
- All packet filter tests pass (62/62)
- All aging tests pass (29/29)
- No behavioral change — only code structure improvement

Fixes #391

Co-authored-by: you <you@example.com>
2026-04-04 09:47:59 -07:00
Kpa-clawbot d4f2c3ac66 perf: index subpath detail lookups instead of scanning all packets (#571)
## Summary

`GetSubpathDetail()` iterated ALL packets to find those containing a
specific subpath — `O(packets × hops × subpath_length)`. With 30K+
packets this caused user-visible latency on every subpath detail click.

## Changes

### `cmd/server/store.go`
- Added `spTxIndex map[string][]*StoreTx` alongside existing `spIndex` —
tracks which transmissions contain each subpath key
- Extended `addTxToSubpathIndexFull()` and
`removeTxFromSubpathIndexFull()` to maintain both indexes simultaneously
- Original `addTxToSubpathIndex()`/`removeTxFromSubpathIndex()` wrappers
preserved for backward compatibility
- `buildSubpathIndex()` now populates both `spIndex` and `spTxIndex`
during `Load()`
- All incremental update sites (ingest, path change, eviction) use the
`Full` variants
- `GetSubpathDetail()` rewritten: direct `O(1)` map lookup on
`spTxIndex[key]` instead of scanning all packets

### `cmd/server/coverage_test.go`
- Added `TestSubpathTxIndexPopulated`: verifies `spTxIndex` is
populated, counts match `spIndex`, and `GetSubpathDetail` returns
correct results for both existing and non-existent subpaths

## Complexity

- **Before:** `O(total_packets × avg_hops × subpath_length)` per request
- **After:** `O(matched_txs)` per request (direct map lookup)

## Tests

All tests pass: `cmd/server` (4.6s), `cmd/ingestor` (25.6s)

Fixes #358

---------

Co-authored-by: you <you@example.com>
2026-04-04 09:35:00 -07:00
Kpa-clawbot 37300bf5c8 fix: cap prefix map at 8 chars to cut memory ~10x (#570)
## Summary

`buildPrefixMap()` was generating map entries for every prefix length
from 2 to `len(pubkey)` (up to 64 chars), creating ~31 entries per node.
With 500 nodes that's ~15K map entries; with 1K+ nodes it balloons to
31K+.

## Changes

**`cmd/server/store.go`:**
- Added `maxPrefixLen = 8` constant — MeshCore path hops use 2–6 char
prefixes, 8 gives headroom
- Capped the prefix generation loop at `maxPrefixLen` instead of
`len(pk)`
- Added full pubkey as a separate map entry when key is longer than
`maxPrefixLen`, ensuring exact-match lookups (used by
`resolveWithContext`) still work

**`cmd/server/coverage_test.go`:**
- Added `TestPrefixMapCap` with subtests for:
  - Short prefix resolution still works
  - Full pubkey exact-match resolution still works
  - Intermediate prefixes beyond the cap correctly return nil
  - Short keys (≤8 chars) have all prefix entries
  - Map size is bounded

## Impact

- Map entries per node: ~31 → ~8 (one per prefix length 2–8, plus one
full-key entry)
- Total map size for 500 nodes: ~15K entries → ~4K entries (~75%
reduction)
- No behavioral change for path hop resolution (2–6 char prefixes)
- No behavioral change for exact pubkey lookups

## Tests

All existing tests pass:
- `cmd/server`: 
- `cmd/ingestor`: 

Fixes #364

---------

Co-authored-by: you <you@example.com>
2026-04-04 09:28:38 -07:00
Kpa-clawbot cb8a2e15c8 perf: index node path lookups instead of scanning all packets (#572)
## Summary

Index node path lookups in `handleNodePaths()` instead of scanning all
packets on every request.

## Problem

`handleNodePaths()` iterated ALL packets in the store (`O(total_packets
× avg_hops)`) with prefix string matching on every hop. This caused
user-facing latency on every node detail page load with 30K+ packets.

## Fix

Added a `byPathHop` index (`map[string][]*StoreTx`) that maps lowercase
hop prefixes and resolved full pubkeys to their transmissions. The
handler now does direct map lookups instead of a full scan.

### Index lifecycle
- **Built** during `Load()` via `buildPathHopIndex()`
- **Incrementally updated** during `IngestNewFromDB()` (new packets) and
`IngestNewObservations()` (path changes)
- **Cleaned up** during `EvictStale()` (packet removal)

### Query strategy
The handler looks up candidates from the index using:
1. Full pubkey (matches resolved hops from `resolved_path`)
2. 2-char prefix (matches short raw hops)
3. 4-char prefix (matches medium raw hops)
4. Any longer raw hops starting with the 4-char prefix

This reduces complexity from `O(total_packets × avg_hops)` to
`O(matching_txs + unique_hop_keys)`.

## Tests

- `TestNodePathsEndpointUsesIndex` — verifies the endpoint returns
correct results using the index
- `TestPathHopIndexIncrementalUpdate` — verifies add/remove operations
on the index

All existing tests pass.

Fixes #359

Co-authored-by: you <you@example.com>
2026-04-04 09:25:18 -07:00
Kpa-clawbot aac038abb9 fix: filter inconsistent hash sizes by role and add 7-day time window (#567)
## Summary

Fixes #566 — The "Inconsistent Hash Sizes" list on the Analytics page
included all node types and had no time window, causing false positives.

## Changes

### 1. Role filter on inconsistent nodes (`cmd/server/store.go`)
Added role filter to the `inconsistentNodes` loop in
`computeHashCollisions()` so only repeaters and room servers are
included. Companions are excluded since they were never affected by the
firmware bug. This matches the existing role filter on collision
bucketing from #441.

```go
// Before:
if cn.HashSizeInconsistent {

// After:
if cn.HashSizeInconsistent && (cn.Role == "repeater" || cn.Role == "room_server") {
```

### 2. 7-day time window on hash size computation
(`cmd/server/store.go`)
Added a 7-day recency cutoff to `computeNodeHashSizeInfo()`. Adverts
older than 7 days are now skipped, preventing legitimate historical
config changes (e.g., testing different byte sizes) from creating
permanent false positives.

### 3. Frontend description text (`public/analytics.js`)
Updated the description to reflect the filtered scope: now says
"Repeaters and room servers" instead of "Nodes", mentions the 7-day
window, and notes that companions are excluded.

## Tests

- `TestInconsistentNodesExcludesCompanions` — verifies companions are
excluded while repeaters and room servers are included
- `TestHashSizeInfoTimeWindow` — verifies adverts older than 7 days are
excluded from hash size computation
- Updated existing hash size tests to use recent timestamps (compatible
with the new time window)
- All existing tests pass: `cmd/server` , `cmd/ingestor` 

## Perf justification
The time window filter adds a single string comparison per advert in the
scan loop — O(n) with a tiny constant. No impact on hot paths.

---------

Co-authored-by: you <you@example.com>
2026-04-04 09:22:12 -07:00
Kpa-clawbot 588fba226d perf: track max transmission/observation IDs incrementally (#569)
## Summary

Replace O(n) map iteration in `MaxTransmissionID()` and
`MaxObservationID()` with O(1) field lookups.

## What Changed

- Added `maxTxID` and `maxObsID` fields to `PacketStore`
- Updated `Load()`, `IngestNewFromDB()`, and `IngestNewObservations()`
to track max IDs incrementally as entries are added
- `MaxTransmissionID()` and `MaxObservationID()` now return the tracked
field directly instead of iterating the entire map

## Performance

Before: O(n) iteration over 30K+ map entries under a read lock
After: O(1) field return

## Tests

- Added `TestMaxTransmissionIDIncremental` verifying the incremental
field matches brute-force iteration over the maps
- All existing tests pass (`cmd/server` and `cmd/ingestor`)

Fixes #356

Co-authored-by: you <you@example.com>
2026-04-04 09:20:17 -07:00
Kpa-clawbot c670742589 feat: add byte-size filter to map page (#565) (#568)
## Summary

Adds a byte-size filter to the map page, allowing users to filter
repeater markers by their hash prefix size (1-byte, 2-byte, or 3-byte).

## What changed

**`public/map.js`** — single file change:

1. **New filter state**: Added `byteSize` to the `filters` object
(default: `'all'`), persisted in `localStorage`
2. **New UI section**: Added a "Byte Size" fieldset with button group
(`All | 1-byte | 2-byte | 3-byte`) in the map controls panel, between
"Node Types" and "Display"
3. **Filter logic**: In `_renderMarkersInner`, when `byteSize !==
'all'`, repeater nodes are filtered by their `hash_size` field.
Non-repeater nodes (companions, rooms, sensors) are unaffected — they
pass through regardless of the byte-size filter setting
4. **Event binding**: Button click handlers update the filter, persist
to localStorage, and re-render markers

## Design decisions

- **Client-side only** — no backend changes needed. The `hash_size`
field is already included in the `/api/nodes` response
- **Repeaters only** — byte size is a repeater configuration concept;
other node roles don't have configurable path prefix sizes
- **Matches existing pattern** — uses the same button-group UI as the
Status filter (All/Active/Stale)
- **`hash_size` defaults to 1** — consistent with how the rest of the
codebase treats missing `hash_size` (`node.hash_size || 1`)

## Performance

No new API calls. Filter is a simple string comparison inside the
existing `nodes.filter()` loop in `_renderMarkersInner` — O(1) per node,
negligible overhead.

Fixes #565

Co-authored-by: you <you@example.com>
2026-04-04 09:14:49 -07:00
efiten f897ce1b26 fix: use runtime heap stats for memory-based eviction (#564)
## Problem

Closes #563. Addresses the *Packet store estimated memory* item in #559.

`estimatedMemoryMB()` used a hardcoded formula:

```go
return float64(len(s.packets)*5120+s.totalObs*500) / 1048576.0
```

This ignored three data structures that grow continuously with every
ingest cycle:

| Structure | Production size | Heap not counted |
|---|---|---|
| `distHops []distHopRecord` | 1,556,833 records | ~300 MB |
| `distPaths []distPathRecord` | 93,090 records | ~25 MB |
| `spIndex map[string]int` | 4,113,234 entries | ~400 MB |

Result: formula reported ~1.2 GB while actual heap was ~5 GB. With
`maxMemoryMB: 1024`, eviction calculated it only needed to shed ~200 MB,
removed a handful of packets, and stopped. Memory kept growing until the
OOM killer fired.

## Fix

Replace `estimatedMemoryMB()` with `runtime.ReadMemStats` so all data
structures are automatically counted:

```go
func (s *PacketStore) estimatedMemoryMB() float64 {
    if s.memoryEstimator != nil {
        return s.memoryEstimator()
    }
    var ms runtime.MemStats
    runtime.ReadMemStats(&ms)
    return float64(ms.HeapAlloc) / 1048576.0
}
```

Replace the eviction simulation loop (which re-used the same wrong
formula) with a proportional calculation: if heap is N× over budget,
evict enough packets to keep `(1/N) × 0.9` of the current count. The 0.9
factor adds a 10% buffer so the next ingest cycle doesn't immediately
re-trigger. All major data structures (distHops, distPaths, spIndex)
scale with packet count, so removing a fraction of packets frees roughly
the same fraction of total heap.

## Testing

- Updated `TestEvictStale_MemoryBasedEviction` to inject a deterministic
estimator via the new `memoryEstimator` field.
- Added `TestEvictStale_MemoryBasedEviction_UnderestimatedHeap`:
verifies that when actual heap is 5× over limit (the production failure
scenario), eviction correctly removes ~80%+ of packets.

```
=== RUN   TestEvictStale_MemoryBasedEviction
[store] Evicted 538 packets (1076 obs)
--- PASS

=== RUN   TestEvictStale_MemoryBasedEviction_UnderestimatedHeap
[store] Evicted 820 packets (1640 obs)
--- PASS
```

Full suite: `go test ./...` — ok (10.3s)

## Perf note

`runtime.ReadMemStats` runs once per eviction tick (every 60 s) and once
per `/api/perf/store` call. Cost is negligible.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-04 08:41:54 -07:00
Kpa-clawbot cbfce41d7e perf: optimize neighbor graph build (3 fixes for 30s+ CPU) (#562)
## Summary

Fixes critical performance issue in neighbor graph computation that
consumed 65% of CPU (30+ seconds) on a 325K packet dataset.

## Changes

### Fix 1: Cache strings.ToLower results
- Added cachedToLower() helper that caches lowercased strings in a local
map
- Pubkeys repeat across hundreds of thousands of observations
- Pre-computes fromLower once per transaction instead of once per
observation
- **Impact:** Eliminates ~8.4s (25.3% CPU)

### Fix 2: Cache parsed DecodedJSON via StoreTx.ParsedDecoded()
- Added ParsedDecoded() method on StoreTx using sync.Once for
thread-safe lazy caching
- json.Unmarshal on decoded_json now runs at most once per packet
lifetime
- Result reused by extractFromNode, indexByNode, trackAdvertPubkey
- **Impact:** Eliminates ~8.8s (26.3% CPU)

### Fix 3: Extend neighbor graph TTL from 60s to 5 minutes
- The graph depends on traffic patterns, not individual packets
- Reduces rebuild frequency 5x
- **Impact:** ~80% reduction in sustained CPU from graph rebuilds

## Tests

- 7 new tests added, all 26+ existing neighbor graph tests pass
- BenchmarkBuildFromStore: 727us/op, 237KB/op, 6030 allocs/op

Related: #559

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: you <you@example.com>
2026-04-04 01:25:51 -07:00
you 1e1c4cb91f fix: include resolved_path in groupByHash packet response
QueryGroupedPackets builds its map manually and was missing
resolved_path. The non-grouped path (txToMap) included it.
2026-04-04 08:01:35 +00:00
you 0c340e1eb6 fix: set hasResolvedPath flag after ensuring column exists
detectSchema() runs at DB open time before ensureResolvedPathColumn()
adds the column during Load(). On first run (or any run where the column
was just added), hasResolvedPath stayed false, causing Load() to skip
reading resolved_path from SQLite. This forced a full backfill of all
observations on every restart, burning CPU for minutes on large DBs.

Fix: set hasResolvedPath = true after ensureResolvedPathColumn succeeds.
2026-04-04 07:46:25 +00:00
Kpa-clawbot ae38cdefb4 feat: server-side hop resolution at ingest — resolved_path (#556)
## Summary

Implements server-side hop prefix resolution at ingest time with a
persisted neighbor graph. Hop prefixes in `path_json` are now resolved
to full 64-char pubkeys at ingest and stored as `resolved_path` on each
observation, eliminating the need for client-side resolution via
`HopResolver`.

Fixes #555

## What changed

### New file: `cmd/server/neighbor_persist.go`
SQLite persistence layer for the neighbor graph and resolved paths:
- `neighbor_edges` table creation and management
- Load/build/persist neighbor edges from/to SQLite
- `resolved_path` column migration on observations
- `resolvePathForObs()` — resolves hop prefixes using
`resolveWithContext` with 4-tier priority (affinity → geo → GPS → first
match)
- Cold startup backfill for observations missing `resolved_path`
- Async persistence of edges and resolved paths during ingest
(non-blocking)

### Modified: `cmd/server/store.go`
- `StoreObs` gains `ResolvedPath []*string` field
- `StoreTx` gains `ResolvedPath []*string` (cached from best
observation)
- `Load()` dynamically includes `resolved_path` in SQL query when column
exists
- `IngestNewFromDB()` resolves paths at ingest time and persists
asynchronously
- `pickBestObservation()` propagates `ResolvedPath` to transmission
- `txToMap()` and `enrichObs()` include `resolved_path` in API responses
- All 7 `pm.resolve()` call sites migrated to `pm.resolveWithContext()`
with the persisted graph
- Broadcast maps include `resolved_path` per observation

### Modified: `cmd/server/db.go`
- `DB` struct gains `hasResolvedPath bool` flag
- `detectSchema()` checks for `resolved_path` column existence
- Graceful degradation when column is absent (test DBs, old schemas)

### Modified: `cmd/server/main.go`
- Startup sequence: ensure tables → load/build graph → backfill resolved
paths → re-pick best observations

### Modified: `cmd/server/routes.go`
- `mapSliceToTransmissions()` and `mapSliceToObservations()` propagate
`resolved_path`
- Node paths handler uses `resolveWithContext` with graph

### Modified: `cmd/server/types.go`
- `TransmissionResp` and `ObservationResp` gain `ResolvedPath []*string`
with `omitempty`

### New file: `cmd/server/neighbor_persist_test.go`
16 tests covering:
- Path resolution (unambiguous, empty, unresolvable prefixes)
- Marshal/unmarshal of resolved_path JSON
- SQLite table creation and column migration (idempotent)
- Edge persistence and loading
- Schema detection
- Full Load() with resolved_path
- API response serialization (present when set, omitted when nil)

## Design decisions

1. **Async persistence** — resolved paths and neighbor edges are written
to SQLite in a goroutine to avoid blocking the ingest loop. The
in-memory state is authoritative.

2. **Schema compatibility** — `DB.hasResolvedPath` flag allows the
server to work with databases that don't yet have the `resolved_path`
column. SQL queries dynamically include/exclude the column.

3. **`pm.resolve()` retained** — Not removed as dead code because
existing tests use it directly. All production call sites now use
`resolveWithContext` with the persisted graph.

4. **Edge persistence is conservative** — Only unambiguous edges (single
candidate) are persisted to `neighbor_edges`. Ambiguous prefixes are
handled by the in-memory `NeighborGraph` via Jaccard disambiguation.

5. **`null` = unresolved** — Ambiguous prefixes store `null` in the
resolved_path array. Frontend falls back to prefix display.

## Performance

- `resolveWithContext` per hop: ~1-5μs (map lookups, no DB queries)
- Typical packet has 0-5 hops → <25μs total resolution overhead per
packet
- Edge/path persistence is async → zero impact on ingest latency
- Backfill is one-time on first startup with the new column

## Test results

```
cd cmd/server && go test ./... -count=1  → ok (4.4s)
cd cmd/ingestor && go test ./... -count=1 → ok (25.5s)
```

---------

Co-authored-by: you <you@example.com>
2026-04-04 00:20:59 -07:00
Kpa-clawbot a97fa52f10 feat: frontend consumers prefer resolved_path (M4, #555) (#561)
## Summary

Implements **M4 (frontend consumers)** from the [resolved-path
spec](https://github.com/Kpa-clawbot/CoreScope/blob/resolved-path-spec/docs/specs/resolved-path.md)
for #555.

The server (PR #556, M1-M3) now returns `resolved_path` on all
packet/observation API responses and WebSocket broadcasts. This PR
updates all frontend consumers to **prefer `resolved_path`** over
client-side HopResolver, with full fallback for old packets.

## What changed

### `hop-resolver.js`
- Added `resolveFromServer(hops, resolvedPath)` — takes the short hex
prefixes and aligned array of full pubkeys from `resolved_path`, looks
up node names from the existing nodesList. Returns the same `{ [hop]: {
name, pubkey, ... } }` format as `resolve()`.

### `packet-helpers.js`
- Added `getResolvedPath(p)` — cached JSON parser for the new
`resolved_path` field (mirrors `getParsedPath`).
- Updated `clearParsedCache()` to also clear `_parsedResolvedPath`.

### `packets.js`
- **Bulk load** (`loadPackets`): calls `cacheResolvedPaths(packets)`
before the existing `resolveHops` fallback.
- **WebSocket updates**: pre-populates `hopNameCache` from
`resolved_path` on incoming packets before falling back to HopResolver
for any remaining unknown hops.
- **Group expansion** (`pktToggleGroup`): caches resolved paths from
child observations.
- **Packet detail** (`selectPacket`): prefers `resolveFromServer` when
`resolved_path` is available.
- **Show Route button**: uses `resolved_path` pubkeys directly instead
of client-side disambiguation.
- **Observation spreading**: carries `resolved_path` field when
constructing observation packets.

### `live.js`
- `resolveHopPositions` accepts optional `resolvedPath` parameter;
prefers server-resolved pubkeys, falls back to HopResolver for null
entries.
- Normalized WS packet objects now carry `resolved_path`.

### Files NOT changed (no resolution changes needed)
- **`analytics.js`** — only uses `HopResolver.haversineKm` (a utility
function). Topology, subpath, and hop distance data comes pre-resolved
from the server API (handled by M2/M3).
- **`nodes.js`** — gets pre-resolved path data from
`/nodes/:pubkey/paths` API; no client-side hop resolution.
- **`map.js`** — `drawPacketRoute` already handles full 64-char pubkeys
via exact match. The updated `packets.js` now passes full pubkeys from
`resolved_path` to the map.

## Fallback pattern

```javascript
// In hop-resolver.js
function resolveFromServer(hops, resolvedPath) {
  // Returns resolved entries for non-null pubkeys
  // Skips null entries (unresolved) — caller falls back to HopResolver
}

// In packets.js — bulk load
await cacheResolvedPaths(packets);  // server-side first
await resolveHops([...allHops]);     // client-side fallback for remaining
```

Old packets without `resolved_path` continue to work exactly as before
via the existing HopResolver. `hop-resolver.js` is NOT removed — it
remains the fallback.

## Tests

- 10 new tests for `resolveFromServer()` and `getResolvedPath()`
- All 445 frontend helper tests pass
- All 62 packet filter tests pass
- All 29 aging tests pass

Closes #555 (M4 milestone)

---------

Co-authored-by: you <you@example.com>
2026-04-04 00:18:46 -07:00
Kpa-clawbot 43673e86f2 fix: perf stats MaxMB reads from config instead of hardcoded 1024 (#558)
Perf stats `GetPerfStoreStats` returned a hardcoded `MaxMB: 1024`
regardless of the configured `packetStore.maxMemoryMB`. Now reads from
`s.maxMemoryMB`.

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-03 23:25:54 -07:00
Kpa-clawbot 81ef51cc5c fix: debounce distance index rebuild to prevent CPU hot loop (#557)
## Problem

On busy meshes (325K+ transmissions, 50 observers), the distance index
rebuild runs on **every ingest poll** (~1s interval), computing
haversine distances for 1M+ hop records. Each rebuild takes 2-3 seconds
but new observations arrive faster than it can finish, creating a CPU
hot loop that starves the HTTP server.

Discovered on the Cascadia Mesh instance where `corescope-server` was
consuming 15 minutes of CPU time in 10 minutes of uptime, the API was
completely unresponsive, and health checks were timing out.

### Server logs showing the hot loop:
```
[store] Built distance index: 1797778 hop records, 207072 path records
[store] Built distance index: 1797806 hop records, 207075 path records
[store] Built distance index: 1797811 hop records, 207075 path records
[store] Built distance index: 1797820 hop records, 207075 path records
```
Every 2 seconds, nonstop.

## Root Cause

`IngestNewObservations` calls `buildDistanceIndex()` synchronously
whenever `pickBestObservation` selects a longer path. With 50 observers
sending observations every second, paths change on nearly every poll
cycle, triggering a full rebuild each time.

## Fix

- Mark distance index dirty on path changes instead of rebuilding inline
- Rebuild at most every **30 seconds** (configurable via `distLast`
timer)
- Set `distLast` after initial `Load()` to prevent immediate re-rebuild
on first ingest
- Distance data is at most 30s stale — acceptable for an analytics view

## Testing

- `go build`, `go vet`, `go test` all pass
- No behavioral change for the initial load or the analytics API
response shape
- Distance data freshness goes from real-time to 30s max staleness

---------

Co-authored-by: Kpa-clawbot <259247574+Kpa-clawbot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: you <you@example.com>
2026-04-03 23:08:09 -07:00
you ddce26ff2d ci: pin build and deploy jobs to meshcore-vm runner 2026-04-04 04:21:48 +00:00
Kpa-clawbot ee29cc627f perf: parallelize expanded group fetches, use hashIndex Map lookup (#552)
## Summary
Fixes #388 — expanded groups were fetched sequentially with O(n)
`packets.find()` lookups.

## Changes
1. **Parallel fetch**: Replaced sequential `for...of + await` loop in
`loadPackets()` with `Promise.all()` so all expanded group children are
fetched concurrently.
2. **O(1) Map lookup**: Replaced 3 instances of `packets.find(p =>
p.hash === hash)` with `hashIndex.get(hash)`:
   - `loadPackets()` expanded group restore (~line 553)
   - `select-observation` click handler (~line 1015)
   - `pktToggleGroup()` (~line 2012)

## Perf justification
- **Before**: N expanded groups → N sequential API calls + N ×
O(packets.length) array scans
- **After**: N parallel API calls + N × O(1) Map lookups
- Typical N is 1-3 (minor severity as noted in issue), but the fix is
trivial and correct

## Tests
All existing tests pass: `test-packet-filter.js` (62), `test-aging.js`
(29), `test-frontend-helpers.js` (433).

Co-authored-by: you <you@example.com>
2026-04-03 21:09:17 -07:00
Kpa-clawbot f3caf42be4 feat: show transport badge in live packet feed (#551)
## Summary

Show the transport badge ("T") in the live packet feed, matching the
packets table (#337).

## Changes

- Add `transportBadge(pkt.route_type)` to all 4 feed rendering paths in
`live.js`:
  - Grouped feed items (initial history load)
  - `addFeedItemDOM()` (VCR replay)
  - Dedup new feed items (live WebSocket updates)
  - Node detail panel recent packets list
- Uses existing `transportBadge()` from `app.js` and `.badge-transport`
CSS from `style.css`

## Testing

- 2 new source-level assertions in `test-live.js` verifying
`transportBadge()` calls exist
- All existing tests pass (67 passed in test-live.js, no new failures)

Fixes #338

Co-authored-by: you <you@example.com>
2026-04-03 21:09:02 -07:00
Kpa-clawbot c34744247a fix: clean up nodeActivity in pruneStaleNodes to prevent memory leak (#553)
## Summary

`nodeActivity` (an object tracking per-node packet counts for heatmap
intensity) grows without bound — entries are added on every packet flash
but never removed, even when stale nodes are pruned.

## Changes

- **Delete `nodeActivity[key]`** alongside `nodeMarkers[key]` and
`nodeData[key]` when removing stale WS-only nodes in `pruneStaleNodes()`
- **Prune orphaned entries** — after the main prune loop, sweep
`nodeActivity` and delete any key that has no corresponding `nodeData`
entry (catches edge cases where nodes were removed by other code paths)
- Both run every 60s via the existing `pruneStaleNodes` interval timer

## Testing

- Added 2 regression tests in `test-frontend-helpers.js` verifying stale
node cleanup and orphan removal
- All 435 frontend helper tests pass, plus packet-filter (62) and aging
(29)

Fixes #390

---------

Co-authored-by: you <you@example.com>
2026-04-03 16:54:53 -07:00
Kpa-clawbot 10f712f9d7 fix: restructure scroll containers for iOS status bar tap-to-scroll (#330) (#554)
## Summary

Fixes #330 — iOS status bar tap-to-scroll broken because `#app` had
`overflow: hidden`, preventing `<body>` from being the scroll container.

## Approach: Option B from the issue

Instead of a JS polyfill, this restructures scroll containers so
`<body>` is the primary scroll container by default, which iOS Safari
requires for native status-bar tap-to-scroll.

### How it works

**`#app` default (body-scroll mode):** Uses `min-height` instead of
fixed `height`, no `overflow: hidden`. Content pushes beyond the
viewport and body scrolls naturally.

**`#app.app-fixed` (fixed-layout mode):** Restores the original `height:
calc(100dvh - 52px); overflow: hidden` for pages that need constrained
containers. The router in `app.js` toggles this class based on the
current page.

### Fixed-layout pages (`.app-fixed`)
These pages need fixed-height containers and are unchanged in behavior:
- **packets** — virtual scroll requires fixed-height `.panel-left` to
calculate visible rows
- **nodes** — split-panel layout with independently scrollable panels
- **map** — Leaflet requires fixed-dimension container
- **live** — Leaflet map (also has its own `#app:has(.live-page)`
override in live.css)
- **channels** — split-panel chat layout
- **audio-lab** — split-panel layout

### Body-scroll pages (no `.app-fixed`)
These pages now let the body scroll, enabling iOS tap-to-scroll:
- **analytics** — removed `overflow-y: auto; height: 100%`
- **observers** — removed `overflow-y: auto; height: calc(100vh - 56px)`
- **traces** — removed `overflow-y: auto; height: 100%`
- **home** — removed `#app:has(.home-hero)` override (no longer needed)
- **compare** — removed inline `overflow-y:auto; height:calc(100vh -
56px)`
- **perf** — removed inline `height:100%; overflow-y:auto`
- **observer-detail** — removed inline `overflow-y:auto;
height:calc(100vh - 56px)`
- **node-analytics** — removed inline `height:100%; overflow-y:auto`

### Files changed
| File | Change |
|------|--------|
| `public/style.css` | `#app` default → `min-height`; added `.app-fixed`
class |
| `public/app.js` | Router toggles `.app-fixed` based on page |
| `public/home.css` | Removed `#app:has()` workaround |
| `public/compare.js` | Removed inline overflow/height |
| `public/perf.js` | Removed inline overflow/height |
| `public/observer-detail.js` | Removed inline overflow/height |
| `public/node-analytics.js` | Removed inline overflow/height |

### What's preserved
- Sticky nav (`position: sticky; top: 0`) — works with body scroll
- Split-panel resize handles — unchanged, still in fixed containers
- Virtual scroll on packets page — unchanged, `.panel-left` still has
fixed height
- Leaflet maps — unchanged, containers still have fixed dimensions
- Mobile responsive overrides — unchanged

Co-authored-by: you <you@example.com>
2026-04-03 16:54:36 -07:00
Kpa-clawbot 412a8fdb8f feat: live map uses affinity-aware hop resolution (#528) (#550)
## Summary

Augments the shared `HopResolver` with neighbor-graph affinity data so
that when multiple nodes match a hop prefix, the resolver prefers
candidates that are known neighbors of the adjacent hop — instead of
relying solely on geo-distance.

Fixes #528

## Changes

### `public/hop-resolver.js`
- Added `affinityMap` — stores bidirectional neighbor adjacency with
scores
- Added `setAffinity(graph)` — ingests `/api/analytics/neighbor-graph`
edge data into O(1) Map lookups
- Added `getAffinity(pubkeyA, pubkeyB)` — returns affinity score between
two nodes (0 if not neighbors)
- Added `pickByAffinity(candidates, adjacentPubkey, anchor, ...)` —
picks best candidate: affinity-neighbor first (highest score), then
geo-distance fallback
- Modified forward and backward passes in `resolve()` to track the
previously-resolved pubkey and use `pickByAffinity` instead of raw
geo-sort

### `public/live.js`
- Added `fetchAffinityData()` — fetches `/api/analytics/neighbor-graph`
once and calls `HopResolver.setAffinity()`
- Added `startAffinityRefresh()` — refreshes affinity data every 60
seconds
- Both are called from `loadNodes()` after HopResolver is initialized

### `test-hop-resolver-affinity.js` (new)
- Affinity prefers neighbor candidate over geo-closest
- Cold start (no affinity data) falls back to geo-closest
- Null/undefined affinity doesn't crash
- Bidirectional score lookup
- Highest affinity score wins among multiple neighbors
- Unambiguous hops unaffected by affinity

## Performance

- API calls: 1 at load + 1 per 60s (no per-packet calls)
- Per-packet resolve: O(1) Map lookups, <0.5ms
- Memory: ~50KB for 2K-node graph

---------

Co-authored-by: you <you@example.com>
2026-04-03 16:32:53 -07:00
Kpa-clawbot 9a39198d92 fix: only count repeaters in hash collision analysis (#441) (#548)
Fixes #441

## Summary

Hash collision analysis was including ALL node types, inflating
collision counts with irrelevant data. Per MeshCore firmware analysis,
**only repeaters matter for collision analysis** — they're the only role
that forwards packets and appears in routing `path[]` arrays.

## Root Causes Fixed

1. **`hash_size==0` nodes counted in all buckets** — nodes with unknown
hash size were included via `cn.HashSize == bytes || cn.HashSize == 0`,
polluting every bucket
2. **Non-repeater roles included** — companions, rooms, sensors, and
observers were counted even though their hash collisions never cause
routing ambiguity

## Fix

Changed `computeHashCollisions()` filter from:
```go
// Before: include everything except companions
if cn.HashSize == bytes && cn.Role != "companion" {
```
To:
```go
// After: only include repeaters (per firmware analysis)
if cn.HashSize == bytes && cn.Role == "repeater" {
```

## Why only repeaters?

From [MeshCore firmware
analysis](https://github.com/Kpa-clawbot/CoreScope/issues/441#issuecomment-4185218547):
- Only repeaters override `allowPacketForward()` to return `true`
- Only repeaters append their hash to `path[]` during relay
- Companions, rooms, sensors, observers never forward packets
- Cross-role collisions are benign (companion silently drops, real
repeater still forwards)

## Tests
- `TestHashCollisionsOnlyRepeaters` — verifies companions, rooms,
sensors, and hash_size==0 nodes are all excluded

---------

Co-authored-by: you <you@example.com>
2026-04-03 14:23:13 -07:00
Kpa-clawbot 526ea8a1fc perf(live): chunk VCR replay packet processing to avoid UI freezes (#549)
## Summary

VCR replay functions (`vcrReplayFromTs`, `vcrRewind`,
`fetchNextReplayPage`) fetch up to 10K packets and process them all
synchronously on the main thread via `expandToBufferEntries`, causing
multi-second UI freezes — especially on mobile.

## Fix

- Added `expandToBufferEntriesAsync()` — processes packets in chunks of
200, yielding to the event loop via `setTimeout(0)` between chunks
- Updated all three VCR replay callers to use the async variant
- Kept the synchronous `expandToBufferEntries()` for backward
compatibility (tests, small datasets)
- Exposed `_liveExpandToBufferEntriesAsync` on window for test access

## Perf justification

- **Before:** 10K packets × ~2 observations = 20K+ objects created
synchronously, blocking the main thread for 1-3 seconds on mobile
- **After:** Same work split into chunks of 200 packets (~400 entries)
with event loop yields between chunks. Each chunk takes <5ms, keeping
the UI responsive (well under the 16ms frame budget)
- Chunk size of 200 is tunable via `VCR_CHUNK_SIZE`

## Tests

- Added regression test: sync expand correctness at scale (500 packets →
1000 entries)
- Added structural test: verifies `VCR_CHUNK_SIZE` exists and async
function yields via `setTimeout`
- All existing tests pass (`npm test`)

Fixes #395

---------

Co-authored-by: you <you@example.com>
2026-04-03 21:22:05 +00:00
Kpa-clawbot 8e42febc9c fix: virtual scroll height accounts for expanded group rows (#410) (#547)
## Summary

Fixes #410 — virtual scroll height miscalculation for expanded group
rows.

## Root Cause

When WebSocket messages add children to an already-expanded packet
group, `_rowCounts` becomes stale during the 200ms render debounce
window. Scroll events during this window call `renderVisibleRows()` with
stale row counts, causing wrong total height, spacer heights, and
visible range calculations.

## Changes

**public/packets.js:**
- Added `_rowCountsDirty` flag to track when row counts need
recomputation
- Added `_invalidateRowCounts()` — marks row counts as stale and clears
cumulative cache
- Added `_refreshRowCountsIfDirty()` — lazily recomputes `_rowCounts`
from `_displayPackets`
- Called `_invalidateRowCounts()` when WS handler adds children to
expanded groups (line ~402)
- Called `_refreshRowCountsIfDirty()` at top of `renderVisibleRows()`
before using row counts
- Reset `_rowCountsDirty` in all cleanup paths (destroy, empty display)

**test-packets.js:**
- Added 4 regression tests for `_invalidateRowCounts` /
`_refreshRowCountsIfDirty`

## Complexity

O(n) recomputation of `_rowCounts` when dirty (same as existing
`renderTableRows` path). Only triggers when WS modifies expanded group
children, which is infrequent relative to scroll events.

Co-authored-by: you <you@example.com>
2026-04-03 13:55:23 -07:00
Kpa-clawbot 59bff5462c fix: rate-limit cache invalidation to prevent 0% hit rate (#533) (#546)
## Summary

Fixes #533 — server cache hit rate always 0%.

## Root Cause

`invalidateCachesFor()` is called at the end of every
`IngestNewFromDB()` and `IngestNewObservations()` cycle (~2-5s). Since
new data arrives continuously, caches are cleared faster than any
analytics request can hit them, resulting in a permanent 0% cache hit
rate. The cache TTL (15s/60s) is irrelevant because entries are evicted
by invalidation long before they expire.

## Fix

Rate-limit cache invalidation with a 10-second cooldown:

- First call after cooldown goes through immediately
- Subsequent calls during cooldown accumulate dirty flags in
`pendingInv`
- Next call after cooldown merges pending + current flags and applies
them
- Eviction bypasses cooldown (data removal requires immediate clearing)

Analytics data may be at most ~10s stale, which is acceptable for a
dashboard.

## Changes

- **`store.go`**: Added `lastInvalidated`, `pendingInv`, `invCooldown`
fields. Refactored `invalidateCachesFor()` to rate-limit non-eviction
invalidation. Extracted `applyCacheInvalidation()` helper.
- **`cache_invalidation_test.go`**: Added 4 new tests:
- `TestInvalidationRateLimited` — verifies caches survive during
cooldown
  - `TestInvalidationCooldownAccumulatesFlags` — verifies flag merging
- `TestEvictionBypassesCooldown` — verifies eviction always clears
immediately
- `BenchmarkCacheHitDuringIngestion` — confirms 100% hit rate during
rapid ingestion (was 0%)

## Perf Proof

```
BenchmarkCacheHitDuringIngestion-16    3467889    1018 ns/op    100.0 hit%
```

Before: 0% hit rate under continuous ingestion. After: 100% hit rate
during cooldown periods.

Co-authored-by: you <you@example.com>
2026-04-03 13:53:58 -07:00
Kpa-clawbot 8c1cd8a9fe perf: track advert pubkeys incrementally, eliminate per-request JSON parsing (#360) (#544)
## Summary

`GetPerfStoreStats()` and `GetPerfStoreStatsTyped()` iterated **all**
ADVERT packets and called `json.Unmarshal` on each one — under a read
lock — on every `/api/perf` and `/api/health` request. With 5K+ adverts,
each health check triggered thousands of JSON parses.

## Fix

Added a refcounted `advertPubkeys map[string]int` to `PacketStore` that
tracks distinct pubkeys incrementally during `Load()`,
`IngestNewFromDB()`, and eviction. The perf/health handlers now just
read `len(s.advertPubkeys)` — O(1) with zero allocations.

## Benchmark Results (5K adverts, 200 distinct pubkeys)

| Method | ns/op | allocs/op |
|--------|-------|-----------|
| `GetPerfStoreStatsTyped` | **78** | **0** |
| `GetPerfStoreStats` | **2,565** | **9** |

Before this change, both methods performed O(N) JSON unmarshals per
call.

## Tests Added

- `TestAdvertPubkeyTracking` — verifies incremental tracking through
add/evict lifecycle
- `TestAdvertPubkeyPublicKeyField` — covers the `public_key` JSON field
variant
- `TestAdvertPubkeyNonAdvert` — ensures non-ADVERT packets don't affect
count
- `BenchmarkGetPerfStoreStats` — 5K adverts benchmark
- `BenchmarkGetPerfStoreStatsTyped` — 5K adverts benchmark

Fixes #360

---------

Co-authored-by: you <you@example.com>
2026-04-03 13:51:13 -07:00
Kpa-clawbot 29e8e37114 fix: mobile filter dropdown specificity prevents expansion (#534) (#541)
## Summary

Fixes #534 — mobile filter dropdown doesn't expand on packets page.

## Root Cause

CSS specificity battle in the mobile media query. The hide rule uses
`:not()` pseudo-classes which add specificity:

```css
/* Higher specificity due to :not() */
.filter-bar > *:not(.filter-toggle-btn):not(.col-toggle-wrap) { display: none; }

/* Lower specificity — loses even with .filters-expanded */
.filter-bar.filters-expanded > * { display: inline-flex; }
```

The JS toggle correctly adds/removes `.filters-expanded`, but the CSS
expanded rule could never win.

## Fix

Match the `:not()` selectors in the expanded rule so `.filters-expanded`
makes it strictly more specific:

```css
.filter-bar.filters-expanded > *:not(.filter-toggle-btn):not(.col-toggle-wrap) { display: inline-flex; }
```

Added a comment explaining the specificity dependency so future devs
don't repeat this.

## Tests

Added Playwright E2E test: mobile viewport (480×800), navigates to
packets page, clicks filter toggle, verifies filter inputs become
visible.

---------

Co-authored-by: you <you@example.com>
2026-04-03 13:50:10 -07:00
Kpa-clawbot 9b9f396af5 perf: replace O(n²) observation dedup with map-based O(n) (#355) (#543)
## Summary

Fixes #355 — replaces O(n²) observation dedup in `Load()`,
`IngestNewFromDB()`, and `IngestNewObservations()` with an O(1)
map-based lookup.

## Changes

- Added `obsKeys map[string]bool` field to `StoreTx` for O(1) dedup
keyed on `observerID + "|" + pathJSON`
- Replaced all 3 linear-scan dedup sites in `store.go` with map lookups
- Lazy-init `obsKeys` for transmissions created before this change (in
`IngestNewFromDB` and `IngestNewObservations`)
- Added regression test (`TestObsDedupCorrectness`) verifying dedup
correctness
- Added nil-map safety test (`TestObsDedupNilMapSafety`)
- Added benchmark comparing map vs linear scan

## Benchmark Results (ARM64, 16 cores)

| Observations | Map (O(1)) | Linear (O(n)) | Speedup |
|---|---|---|---|
| 10 | 34 ns/op | 41 ns/op | 1.2x |
| 50 | 34 ns/op | 186 ns/op | 5.5x |
| 100 | 34 ns/op | 361 ns/op | 10.6x |
| 500 | 34 ns/op | 4,903 ns/op | **146x** |

Map lookup is constant time regardless of observation count. The linear
scan degrades quadratically — at 500 observations per transmission
(realistic for popular packets seen by many observers), the old code is
146x slower per dedup check.

All existing tests pass.

---------

Co-authored-by: you <you@example.com>
2026-04-03 13:33:26 -07:00
Kpa-clawbot b472c8de30 perf: replace O(n²) selection sort with sort.Slice (#354) (#542)
## Summary

Fixes #354

Replaces the O(n²) selection sort in `sortedCopy()` with Go's built-in
`sort.Float64s()` (O(n log n)).

## Changes

- **`cmd/server/routes.go`**: Replaced manual nested-loop selection sort
with `sort.Float64s(cp)`
- **`cmd/server/helpers_test.go`**: Added regression test with
1000-element random input + benchmark

## Benchmark Results (ARM64)

```
BenchmarkSortedCopy/n=256     ~16μs/op    1 alloc
BenchmarkSortedCopy/n=1000    ~95μs/op    1 alloc
BenchmarkSortedCopy/n=10000   ~1.3ms/op   1 alloc
```

With the old O(n²) sort, n=10000 would take ~50ms+. The new
implementation scales as O(n log n).

## Testing

- All existing `TestSortedCopy` tests pass (unchanged behavior)
- New `TestSortedCopyLarge` validates correctness on 1000 random
elements
- `go test ./...` passes in `cmd/server`

Co-authored-by: you <you@example.com>
2026-04-03 13:11:59 -07:00
Kpa-clawbot 03e384bbc4 fix: null guard on pathHops prevents crash on ADVERT detail (#538) (#540)
## Summary

Fixes #538 — `null is not an object (evaluating 'pathHops.length')`
crash on ADVERT packet detail.

## Root Cause

`getParsedPath` caches its result as `p._parsedPath`. If another code
path (e.g., object spread, API response) sets `_parsedPath = null`, the
cache check (`!== undefined`) passes and returns `null` — causing
`.length` to crash.

Same pattern exists for `getParsedDecoded`.

## Changes

### `public/packet-helpers.js`
- `getParsedPath`: cached return now uses `|| []` to guard against null
cache
- `getParsedDecoded`: cached return now uses `|| {}` to guard against
null cache

### `public/packets.js`
- `renderDetail()` (line ~1440): defensive `|| []` / `|| {}` on
getParsedPath/getParsedDecoded calls
- `buildFlatRowHtml()` (line ~1103): same defensive guards

### `test-frontend-helpers.js`
- Added test: cached `_parsedPath = null` returns `[]`
- Added test: cached `_parsedDecoded = null` returns `{}`

## Testing

All 428 frontend helper tests pass. All 62 packet filter tests pass.

Co-authored-by: you <you@example.com>
2026-04-03 13:03:20 -07:00
Kpa-clawbot bf8c9e72ec fix: observer filter checks all observations in grouped mode (#537) (#539)
Fixes #537

## Problem
Observer filter in grouped mode only checked `p.observer_id` (the
primary observer), ignoring child observations. Grouped packets seen by
multiple observers would be hidden when filtering for a non-primary
observer.

## Fix
Two filter paths updated to also check `p._children`:

1. **Client-side display filter** (line ~1293): removed the
`!groupByHash` guard and added `_children` check so grouped packets are
included when any child observation matches
2. **WS real-time filter** (line ~360): added `_children` fallback check

The grouped row rendering (line ~1042) already correctly uses
`_observerFilterSet` for child filtering — no changes needed there.

## Tests
Added 5 tests in `test-frontend-helpers.js`:
- Grouped packet with matching child observer is shown
- Grouped packet with no matching observers is hidden  
- WS filter passes/rejects grouped packets correctly
- Source code assertions verifying both filter paths check `_children`

Co-authored-by: you <you@example.com>
2026-04-03 13:02:25 -07:00
Kpa-clawbot 48923db3d0 Add deep linking rule to AGENTS.md (#535)
Adds a rule to AGENTS.md requiring all new UI states to be
URL-addressable (deep-linkable). Part of #536.

Co-authored-by: you <you@example.com>
2026-04-03 13:01:31 -07:00
efiten 709e5a4776 fix: observer filter drops groups in grouped packets view (#464) (#531)
## Summary

- When `groupByHash=true`, each group only carries its representative
(best-path) `observer_id`. The client-side filter was checking only that
field, silently dropping groups that were seen by the selected observer
but had a different representative.
- `loadPackets` now passes the `observer` param to the server so
`filterPackets`/`buildGroupedWhere` do the correct "any observation
matches" check.
- Client-side observer filter in `renderTableRows` is skipped for
grouped mode (server already filtered correctly).
- Both `db.go` and `store.go` observer filtering extended to support
comma-separated IDs (multi-select UI).

## Test plan

- [ ] Set an observer filter on the Packets screen with grouping enabled
— all groups that have **any** observation from the selected observer(s)
should appear, not just groups where that observer is the representative
- [ ] Multi-select two observers — groups seen by either should appear
- [ ] Toggle to flat (ungrouped) mode — per-observation filter still
works correctly
- [ ] Existing grouped packets tests pass: `cd cmd/server && go test
./...`

Fixes #464

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
2026-04-03 09:22:37 -07:00
you 9099154514 docs: add v3.4 release notes 2026-04-03 08:26:05 +00:00
Kpa-clawbot 924caaa680 fix: render both steps AND FAQ on home page (#525) (#529)
Fixes #525

The `checklist()` function in `home.js` treated steps and FAQ/checklist
as mutually exclusive — if `homeCfg.checklist` existed, steps were
skipped entirely. Adding a single FAQ via the customizer made all intro
steps disappear.

Now renders steps first, then FAQ below with a ' FAQ' header. Falls
back to Bay Area hardcoded defaults only when neither exists.

---------

Co-authored-by: you <you@example.com>
2026-04-03 01:19:42 -07:00
Kpa-clawbot ca95fc46aa fix: neighbor UI — show neighbors crash, dark mode contrast (#523) (#527)
## Summary

Part of #523 — fixes bugs 5 and 7 (bug 6 was a duplicate of bug 7).

### Bug 5: Show Neighbors button throws `window._mapSelectRefNode is not
a function`

**Root cause:** Map popup HTML used inline `onclick` calling
`window._mapSelectRefNode`, which was deleted on SPA page destroy. If a
popup persisted after navigation, clicks would throw.

**Fix:** Replaced inline `onclick` with event delegation. A
document-level click handler catches all `[data-show-neighbors]` clicks
and calls `selectReferenceNode` directly. The global
`window._mapSelectRefNode` is still exposed for existing Playwright
tests but is no longer relied upon by the UI.

### Bug 7: Blue text on dark blue background (dark mode contrast)

**Root cause:** Neighbor table cells inside `.node-detail-section` /
`.node-full-card` inherited accent/link color instead of using
`var(--text)`, making text unreadable in dark mode.

**Fix:** Added explicit `color: var(--text)` on `.node-detail-section
.data-table td` and `.node-full-card .data-table td`. Only `<a>` tags
within those cells retain `color: var(--accent)`.

### Files changed
- `public/map.js` — event delegation for Show Neighbors
- `public/style.css` — contrast fix for neighbor table cells

---------

Co-authored-by: you <you@example.com>
2026-04-03 00:49:17 -07:00
Kpa-clawbot 54fab0551e fix: add home defaults to server theme config (#525) (#526)
## Summary

Fixes #525 — Customizer v2 home section shows empty fields and adding
FAQ kills steps.

## Root Cause

Server returned `home: null` from `/api/config/theme` when no home
config existed in config.json or theme.json. The customizer had no
built-in defaults, so all home fields appeared empty. When a user added
a single override (e.g. FAQ), `computeEffective` started from `home:
null`, created `home: {}`, and only applied the user's override — wiping
steps and everything else.

## Fix

### Server-side (primary)
In `handleConfigTheme()`, replaced the conditional `home` assignment
with `mergeMap` using built-in defaults matching what `home.js`
hardcodes:
- `heroTitle`: "CoreScope"
- `heroSubtitle`: "Real-time MeshCore LoRa mesh network analyzer"
- `steps`: 4 default getting-started steps
- `footerLinks`: Packets + Network Map links

Config/theme overrides merge on top, so customization still works.

### Client-side (defense-in-depth)
Added `DEFAULT_HOME` constant in `customize-v2.js`. `computeEffective()`
now falls back to these defaults when server returns `home: null`,
ensuring the customizer works even without server defaults.

## Tests
- **Go**: `TestConfigThemeHomeDefaults` — verifies `/api/config/theme`
returns non-null home with heroTitle, steps, footerLinks when no config
is set
- **JS**: Two new tests in `test-frontend-helpers.js` — verifies
`computeEffective` provides defaults when home is null, and that user
overrides merge correctly with defaults

Co-authored-by: you <you@example.com>
2026-04-03 00:31:03 -07:00
Kpa-clawbot 0e1beac52f fix: neighbor affinity graph empty results + performance + accessibility (#523) (#524)
## Summary

Fixes the neighbor affinity graph returning empty results despite
abundant ADVERT data in the store.

**Root cause:** `extractFromNode()` in `neighbor_graph.go` only checked
for `"from_node"` and `"from"` fields in the decoded JSON, but real
ADVERT packets store the originator public key as `"pubKey"`. This meant
`fromNode` was always empty, so:
- Zero-hop edges (originator↔observer) were never created
- Originator↔path[0] edges were never created
- Only observer↔path[last] edges could be created (and only for
non-empty paths)

**Fix:** Check `"pubKey"` first in `extractFromNode()`, then fall
through to `"from_node"` and `"from"` for other packet types.

## Bugs Fixed

| Bug | Issue | Fix |
|-----|-------|-----|
| Empty graph results | #522 | `extractFromNode()` now reads `pubKey`
field from ADVERTs |
| 3-4s response time | #523 comment | Graph was rebuilding correctly
with 60s TTL cache — the slow response was due to iterating all packets
finding zero matches. With edges now being found, the cache works as
designed. |
| Incomplete visualization | #523 comment | Downstream of bug 1+2 —
fixed by fixing the builder |
| Accessibility | #523 comment | Added text-based neighbor list, dynamic
aria-label, keyboard focus CSS, dashed lines for ambiguous edges,
confidence symbols |

## Changes

- **`cmd/server/neighbor_graph.go`** — Fixed `extractFromNode()` to
check `pubKey` field (real ADVERT format)
- **`cmd/server/neighbor_graph_test.go`** — Added 2 new tests:
`TestBuildNeighborGraph_AdvertPubKeyField` (real ADVERT format) and
`TestBuildNeighborGraph_OneByteHashPrefixes` (1-byte prefix collision
scenario)
- **`public/analytics.js`** — Added accessible text-based neighbor list,
dynamic aria-label, dashed line pattern for ambiguous edges
- **`public/style.css`** — Added `:focus-visible` keyboard focus
indicator for canvas

## Testing

All Go tests pass (`go test ./... -count=1`). New tests verify the fix
prevents regression.

Fixes #523, Fixes #522

---------

Co-authored-by: you <you@example.com>
2026-04-03 00:30:39 -07:00
Kpa-clawbot 34489e0446 fix: customizer v2 — phantom overrides, missing defaults, stale dark mode (#518) (#520)
Fixes #518, Fixes #514, Fixes #515, Fixes #516

## Summary

Fixes all customizer v2 bugs from the consolidated tracker (#518). Both
server and client changes.

## Server Changes (`routes.go`)

- **typeColors defaults** — added all 10 type color defaults matching
`roles.js` `TYPE_COLORS`. Previously returned `{}`, causing all type
colors to render as black.
- **themeDark defaults** — added 22 dark mode color defaults matching
the Default preset. Previously returned `{}`, causing dark mode to have
no server-side defaults.

## Client Changes (`customize-v2.js`)

- [x] **P0: Phantom override cleanup on init** — new
`_cleanPhantomOverrides()` runs on startup, scanning
`cs-theme-overrides` and removing any values that match server defaults
(arrays via `JSON.stringify`, scalars via `===`).
- [x] **P1: `setOverride` auto-prunes matching defaults** — after
debounced write, iterates the delta and removes any key whose value
matches the server default. Prevents phantom overrides from
accumulating.
- [x] **P1: `_countOverrides` counts only real diffs** — now iterates
keys and calls `_isOverridden()` instead of blindly counting
`Object.keys().length`. Badge count reflects actual overrides only.
- [x] **P1: `_isOverridden` handles arrays/objects** — uses
`JSON.stringify` comparison for non-scalar values (home.steps,
home.checklist, etc.).
- [x] **P1: Type color fallback** — `_renderNodes()` falls back to
`window.TYPE_COLORS` when effective typeColors are empty, preventing
black color swatches.
- [x] **P1: Dark/light toggle re-renders panel** — MutationObserver on
`data-theme` now calls `_refreshPanel()` when panel is open, so
switching modes updates the Theme tab immediately.

## Tests

6 new unit tests added to `test-customizer-v2.js`:
- Phantom scalar overrides cleaned on init
- Phantom array overrides cleaned on init
- Real overrides preserved after cleanup
- `isOverridden` handles matching arrays (returns false)
- `isOverridden` handles differing arrays (returns true)
- `setOverride` prunes value matching server default

All 48 tests pass. Go tests pass.

---------

Co-authored-by: you <you@example.com>
2026-04-03 00:04:33 -07:00
Kpa-clawbot 58f791266d feat: affinity debugging tools (#482) — milestone 6 (#521)
## Summary

Milestone 6 of #482: Observability & Debugging tools for the neighbor
affinity system.

These tools exist because someone will need them at 3 AM when "Show
Neighbors is showing the wrong node for C0DE" and they have 5 minutes to
diagnose it.

## Changes

### 1. Debug API — `GET /api/debug/affinity`
- Full graph state dump: all edges with weights, observation counts,
last-seen timestamps
- Per-prefix resolution log with disambiguation reasoning (Jaccard
scores, ratios, thresholds)
- Query params: `?prefix=C0DE` filter to specific prefix,
`?node=<pubkey>` for specific node's edges
- Protected by API key (same auth as `/api/admin/prune`)
- Response includes: edge count, node count, cache age, last rebuild
time

### 2. Debug Overlay on Map
- Toggle-able checkbox "🔍 Affinity Debug" in map controls
- Draws lines between nodes showing affinity edges with color coding:
  - Green = high confidence (score ≥ 0.6)
  - Yellow = medium (0.3–0.6)
  - Red = ambiguous (< 0.3)
- Line thickness proportional to weight, dashed for ambiguous
- Unresolved prefixes shown as  markers
- Click edge → popup with observation count, last seen, score, observers
- Hidden behind `debugAffinity` config flag or
`localStorage.setItem('meshcore-affinity-debug', 'true')`

### 3. Per-Node Debug Panel
- Expandable "🔍 Affinity Debug" section in node detail page (collapsed
by default)
- Shows: neighbor edges table with scores, prefix resolutions with
reasoning trace
- Candidates table with Jaccard scores, highlighting the chosen
candidate
- Graph-level stats summary

### 4. Server-Side Structured Logging
- Integrated into `disambiguate()` — logs every resolution decision
during graph build
- Format: `[affinity] resolve C0DE: c0dedad4 score=47 Jaccard=0.82 vs
c0dedad9 score=3 Jaccard=0.11 → neighbor_affinity (ratio 15.7×)`
- Logs ambiguous decisions: `scores too close (12 vs 9, ratio 1.3×) →
ambiguous`
- Gated by `debugAffinity` config flag

### 5. Dashboard Stats Widget
- Added to analytics overview tab when debug mode is enabled
- Metrics: total edges/nodes, resolved/ambiguous counts (%), avg
confidence, cold-start coverage, cache age, last rebuild

## Files Changed
- `cmd/server/neighbor_debug.go` — new: debug API handler, resolution
builder, cold-start coverage
- `cmd/server/neighbor_debug_test.go` — new: 7 tests for debug API
- `cmd/server/neighbor_graph.go` — added structured logging to
disambiguate(), `logFn` field, `BuildFromStoreWithLog`
- `cmd/server/neighbor_api.go` — pass debug flag through
`BuildFromStoreWithLog`
- `cmd/server/config.go` — added `DebugAffinity` config field
- `cmd/server/routes.go` — registered `/api/debug/affinity` route,
exposed `debugAffinity` in client config
- `cmd/server/types.go` — added `DebugAffinity` to
`ClientConfigResponse`
- `public/map.js` — affinity debug overlay layer with edge visualization
- `public/nodes.js` — per-node affinity debug panel
- `public/analytics.js` — dashboard stats widget
- `test-e2e-playwright.js` — 3 Playwright tests for debug UI

## Tests
-  7 Go unit tests (API shape, prefix/node filters, auth, structured
logging, cold-start coverage)
-  3 Playwright E2E tests (overlay checkbox, toggle without crash,
panel expansion)
-  All existing tests pass (`go test ./cmd/server/... -count=1`)

Part of #482

---------

Co-authored-by: you <you@example.com>
2026-04-02 23:45:03 -07:00
Kpa-clawbot 9b1b82f29b fix: remove merge conflict marker from test-e2e-playwright.js (#519)
Removes a stale `<<<<<<< HEAD` conflict marker that was accidentally
left in during the PR #510 rebase. This breaks Playwright E2E tests in
CI.

One-line fix — line 1311 deletion.

Co-authored-by: you <you@example.com>
2026-04-02 22:41:30 -07:00
Kpa-clawbot 943eb69937 feat: neighbors section in node detail page (#482) — milestone 5 (#510)
## Summary

Add a "Neighbors" section to the node detail page, showing first-hop
neighbor relationships derived from the neighbor affinity graph (M2
API).

Part of #482 — Milestone 5 per
[spec](https://github.com/Kpa-clawbot/CoreScope/blob/spec/482-neighbor-affinity/docs/specs/neighbor-affinity-graph.md).

## What's Added

### Full-screen detail view (`#/nodes/{pubkey}`)
- New `node-full-card` section between "Heard By" and "Paths Through
This Node"
- Table with columns: **Neighbor** (linked), **Role** (badge),
**Score**, **Obs**, **Last Seen**, **Conf** (confidence indicator)
- Confidence indicators per spec:
  - 🟢 HIGH: auto-resolved, ≥3 observations, score ≥ 0.5
  - 🟡 MEDIUM: 2+ observations
  - 🔴 LOW: single observation
  - ⚠️ AMBIGUOUS: multiple candidates
- Click neighbor name → navigate to their detail page
- 📍 Map button per resolved neighbor row

### Condensed panel view (right panel)
- Shows top 5 neighbors only
- "View all N neighbors →" link navigates to full detail page with
`?section=node-neighbors`

### Deep linking
- `?section=node-neighbors` auto-scrolls to the neighbors section (uses
existing scroll mechanism)

### Data fetching
- `GET /api/nodes/{pubkey}/neighbors` via existing `api()` helper
- Cached per-node for 5 minutes (panel lifetime)
- Loading spinner, empty state, error state

### States
- **Loading**: spinner with "Loading neighbors…"
- **Empty**: "No neighbor data available yet. Neighbor relationships are
built from observed packet paths over time."
- **Error**: "Could not load neighbor data"

## Tests
- 2 new Playwright E2E tests:
  1. Section exists with correct table columns (or empty state)
  2. Loading spinner visible during fetch

## Files Changed
- `public/nodes.js` — neighbor section rendering + data fetching helpers
- `test-e2e-playwright.js` — 2 new E2E tests

---------

Co-authored-by: you <you@example.com>
2026-04-03 05:36:47 +00:00
Kpa-clawbot 15634362c9 feat: neighbor graph visualization in analytics (#482) — milestone 7 (#513)
## Summary

Adds a **Neighbor Graph** tab to the Analytics page — an interactive
force-directed graph visualization of the mesh network's neighbor
affinity data.

Part of #482 (Milestone 7 — Analytics Graph Visualization)

## What's New

### Neighbor Graph Tab
- New "Neighbor Graph" tab in the analytics tab bar
- Force-directed graph layout using HTML5 Canvas (vanilla JS, no
external libs)
- Nodes rendered as circles, colored by role using existing
`ROLE_COLORS`
- Edges as lines with thickness proportional to affinity score
- Ambiguous edges highlighted in yellow

### Interactions
- **Click node** → navigates to node detail page (`#/nodes/{pubkey}`)
- **Hover node** → tooltip showing name, role, neighbor count
- **Drag nodes** → rearrange layout interactively
- **Mouse wheel** → zoom in/out (towards cursor position)
- **Drag background** → pan the view

### Filters
- **Role checkboxes** — toggle repeater, companion, room, sensor
visibility
- **Minimum score slider** — filter out weak edges (0.00–1.00)
- **Confidence filter** — show all / high confidence only / hide
ambiguous

### Stats Summary
Displays above the graph: total nodes, total edges, average score,
resolved %, ambiguous count

### Data Source
Uses `GET /api/analytics/neighbor-graph` endpoint from M2, with region
filtering via the shared RegionFilter component.

## Performance
- Canvas-based rendering (not SVG) for performance with large graphs
- Force simulation uses `requestAnimationFrame` with cooling/dampening —
stops iterating when layout stabilizes
- O(n²) repulsion is acceptable for typical mesh sizes (~500 nodes); for
larger meshes, a Barnes-Hut approximation could be added later
- Animation frame is properly cleaned up on page destroy

## Tests
- Updated tab count assertion (≥10 tabs)
- New Playwright test: tab loads, canvas renders, stats shown (≥3 stat
cards)
- New Playwright test: filter changes update stats

## Files Changed
- `public/analytics.js` — new tab + full graph visualization
implementation
- `test-e2e-playwright.js` — 2 new tests + updated assertion

---------

Co-authored-by: you <you@example.com>
2026-04-02 22:35:28 -07:00
Kpa-clawbot 5151030697 feat: affinity-aware hop resolution (#482) — milestone 4 (#511)
## Summary

Milestone 4 of #482: adds affinity-aware hop resolution to improve
disambiguation accuracy across all hop resolution in the app.

### What changed

**Backend — `prefixMap.resolveWithContext()` (store.go)**

New method that applies a 4-tier disambiguation priority when multiple
nodes match a hop prefix:

| Priority | Strategy | When it wins |
|----------|----------|-------------|
| 1 | **Affinity graph score** | Neighbor graph has data, score ratio ≥
3× runner-up |
| 2 | **Geographic proximity** | Context nodes have GPS, pick closest
candidate |
| 3 | **GPS preference** | At least one candidate has coordinates |
| 4 | **First match** | No signal — current naive fallback |

The existing `resolve()` method is unchanged for backward compatibility.
New callers that have context (originator, observer, adjacent hops) can
use `resolveWithContext()` for better results.

**API — `handleResolveHops` (routes.go)**

Enhanced `/api/resolve-hops` endpoint:
- New query params: `from_node`, `observer` — provide context for
affinity scoring
- New response fields on `HopCandidate`: `affinityScore` (float,
0.0–1.0)
- New response fields on `HopResolution`: `bestCandidate` (pubkey when
confident), `confidence` (one of `unique_prefix`, `neighbor_affinity`,
`ambiguous`)
- Backward compatible: without context params, behavior is identical to
before (just adds `confidence` field)

**Types (types.go)**
- `HopCandidate.AffinityScore *float64`
- `HopResolution.BestCandidate *string`
- `HopResolution.Confidence string`

### Tests

- 7 unit tests for `resolveWithContext` covering all 4 priority tiers +
edge cases
- 2 unit tests for `geoDistApprox`
- 4 API tests for enhanced `/api/resolve-hops` response shape
- All existing tests pass (no regressions)

### Impact

This improves ALL hop resolution across the app — analytics, route
display, subpath analysis, and any future feature that resolves hop
prefixes. The affinity graph (from M1/M2) now feeds directly into
disambiguation decisions.

Part of #482

---------

Co-authored-by: you <you@example.com>
2026-04-02 22:28:07 -07:00
Kpa-clawbot 813b424ca1 fix: Show Neighbors uses affinity API for collision disambiguation (#484) — milestone 3 (#512)
## Summary

Replace broken client-side path walking in `selectReferenceNode()` with
server-side `/api/nodes/{pubkey}/neighbors` API call, fixing #484 where
Show Neighbors returned zero results due to hash collision
disambiguation failures.

**Fixes #484** | Part of #482

## What changed

### `public/map.js` — `selectReferenceNode()` function

**Before:** Client-side path walking — fetched
`/api/nodes/{pubkey}/paths`, walked each path to find hops adjacent to
the selected node by comparing full pubkeys. This fails on hash
collisions because path hops only contain short prefixes (1-2 bytes),
and the hop resolver can pick the wrong collision candidate.

**After:** Server-side affinity resolution — fetches
`/api/nodes/{pubkey}/neighbors?min_count=3` which uses the neighbor
affinity graph (built in M1/M2) to return disambiguated neighbors. For
ambiguous edges, all candidates are included in the neighbor set (better
to show extra markers than miss real neighbors).

**Fallback:** When the affinity API returns zero neighbors (cold start,
insufficient data), the function falls back to the original path-walking
approach. This ensures the feature works even before the affinity graph
has accumulated enough observations.

## Tests

4 new Playwright E2E tests (in both `test-show-neighbors.js` and
`test-e2e-playwright.js`):

1. **Happy path** — Verifies the `/neighbors` API is called and the
reference node UI activates
2. **Hash collision disambiguation** — Two nodes sharing prefix "C0" get
different neighbor sets via the affinity API (THE critical test for
#484)
3. **Fallback to path walking** — Empty affinity response triggers
fallback to `/paths` API
4. **Ambiguous candidates** — Ambiguous edge candidates are included in
the neighbor set

All tests use Playwright route interception to mock API responses,
testing the frontend logic independently of server state.

## Spec reference

See [neighbor-affinity-graph.md](docs/specs/neighbor-affinity-graph.md),
sections:
- "Replacing Show Neighbors on the map" (lines ~461-504)
- "Milestone 3: Show Neighbors Fix (#484)" (lines ~1136-1152)
- Test specs a & b (lines ~754-800)

---------

Co-authored-by: you <you@example.com>
2026-04-02 22:04:03 -07:00
Kpa-clawbot e66085092e feat: neighbor affinity API endpoints (#482) — milestone 2 (#508)
## Summary

Milestone 2 of the neighbor affinity graph (#482). Adds two API
endpoints that expose the neighbor graph built in M1 (PR #507).

### Endpoints

#### `GET /api/nodes/{pubkey}/neighbors`
Returns neighbors for a specific node with affinity scores.

**Query params:** `min_count` (default 1), `min_score` (default 0.0),
`include_ambiguous` (default true)

**Response shape:**
```json
{
  "node": "pubkey",
  "neighbors": [
    { "pubkey": "...", "prefix": "BB", "name": "...", "role": "repeater",
      "count": 847, "score": 0.95, "first_seen": "...", "last_seen": "...",
      "avg_snr": -8.2, "observers": ["obs1"], "ambiguous": false }
  ],
  "total_observations": 847
}
```

Ambiguous entries have `candidates` array; unresolved prefixes have
`unresolved: true`.

#### `GET /api/analytics/neighbor-graph`
Returns full graph summary for analytics/visualization.

**Query params:** `min_count` (default 5), `min_score` (default 0.1),
`region` (IATA code filter)

**Response shape:**
```json
{
  "nodes": [{ "pubkey": "...", "name": "...", "role": "...", "neighbor_count": 5 }],
  "edges": [{ "source": "...", "target": "...", "weight": 847, "score": 0.95, "ambiguous": false }],
  "stats": { "total_nodes": 42, "total_edges": 87, "ambiguous_edges": 3, "avg_cluster_size": 4.2 }
}
```

### Wiring
- `NeighborGraph` + `neighborMu` added to `Server` struct
- Lazy initialization: graph built on first API call, cached with 60s
TTL
- Node name/role lookups via existing `getCachedNodesAndPM()`
- Region filtering via existing `resolveRegionObservers()`

### Tests (15 tests)
- Empty graph, single neighbor, multiple neighbors (sorted by score)
- Ambiguous candidates with candidate list
- Unresolved prefix (orphan) with `unresolved: true`
- `min_count` filter, `min_score` filter, `include_ambiguous=false`
filter
- Unknown node returns 200 with empty neighbors
- Graph endpoint: empty, with edges, default min_count, ambiguous count
- Region filter (graceful when no store)
- Response shape validation (all required keys present)

All existing tests continue to pass.

Part of #482

---------

Co-authored-by: you <you@example.com>
2026-04-02 21:30:23 -07:00
Kpa-clawbot 4a56be0b48 feat: neighbor affinity graph builder (#482) — milestone 1 (#507)
## Summary

Milestone 1 of 7 for the neighbor affinity graph feature (#482).
Implements the core `NeighborGraph` data structure and
`BuildFromStore()` algorithm.

**Spec:** `docs/specs/neighbor-affinity-graph.md` on
`spec/482-neighbor-affinity` branch.

## What's Built

### `cmd/server/neighbor_graph.go`
- **`NeighborGraph` struct** — thread-safe (sync.RWMutex) in-memory
graph with edge map and per-node index
- **`BuildFromStore(*PacketStore)`** — iterates all packets/observations
to extract first-hop edges:
- `originator ↔ path[0]` for ADVERT packets only (originator identity
known)
  - `observer ↔ path[last]` for ALL packet types
  - Zero-hop ADVERTs: `originator ↔ observer` direct edge
- **Affinity scoring** — `score = min(1.0, count/100) × exp(-λ × hours)`
with 7-day half-life
- **Jaccard disambiguation** — resolves ambiguous hash prefixes using
mutual-neighbor overlap
- **Confidence threshold** — auto-resolve only when best ≥ 3×
second-best AND ≥ 3 observations
- **Transitivity poisoning guard** — only fully-resolved edges used as
evidence
- **Orphan prefix handling** — unknown prefixes stored as unresolved
markers
- **Cache management** — 60s TTL, `IsStale()` check for rebuild
triggering

### `cmd/server/neighbor_graph_test.go`
22 unit tests covering all spec requirements:

| Test | What it validates |
|------|-------------------|
| EmptyStore | Empty graph from empty store |
| AdvertSingleHopPath | Both edge types from single-hop ADVERT |
| AdvertMultiHopPath | originator↔path[0] + observer↔path[last] |
| AdvertZeroHop | Direct originator↔observer edge |
| NonAdvertEmptyPath | No edges from non-ADVERT empty path |
| NonAdvertOnlyObserverEdge | Only observer↔last_hop for non-ADVERTs |
| NonAdvertSingleHop | observer↔path[0] only |
| HashCollision | Ambiguous edge with candidates |
| JaccardScoring | Jaccard coefficient computation |
| ConfidenceAutoResolve | Auto-resolve when ratio ≥ 3× |
| EqualScoresAmbiguous | Remains ambiguous with equal scores |
| ObserverSelfEdgeGuard | No self-edges |
| OrphanPrefix | Unresolved prefix handling |
| AffinityScore_Fresh | Score ≈ 1.0 for fresh high-count |
| AffinityScore_Decayed | Score ≈ 0.5 at 7-day half-life |
| AffinityScore_LowCount | Score ≈ 0.05 for count=5 |
| AffinityScore_StaleAndLow | Score ≈ 0 for old low-count |
| CountAccumulation | 5 observations → count=5 |
| MultipleObservers | Observer set tracks all witnesses |
| TimeDecayOldObservations | Month-old edge scores very low |
| ADVERTOnlyConstraint | Non-ADVERTs don't create originator edges |
| CacheTTL | Stale detection works correctly |

## Not in scope (future milestones)
- API endpoints (M2)
- Frontend integration (M3-M5)
- Debug tools (M6)
- Analytics visualization (M7)

Part of #482

---------

Co-authored-by: you <you@example.com>
2026-04-02 21:14:58 -07:00
Kpa-clawbot 64745f89b1 feat: customizer v2 — event-driven state management (#502) (#503)
## Summary

Implements the customizer v2 per the [approved
spec](docs/specs/customizer-rework.md), replacing the v1 customizer's
scattered state management with a clean event-driven architecture.
Resolves #502.

## What Changed

### New: `public/customize-v2.js`
Complete rewrite of the customizer as a self-contained IIFE with:

- **Single localStorage key** (`cs-theme-overrides`) replacing 7
scattered keys
- **Three state layers:** server defaults (immutable) → user overrides
(delta) → effective config (computed)
- **Full data flow pipeline:** `write → read-back → merge → atomic
SITE_CONFIG assign → apply CSS → dispatch theme-changed`
- **Color picker optimistic CSS** (Decision #12): `input` events update
CSS directly for responsiveness; `change` events trigger the full
pipeline
- **Override indicator dots** (●) on each field — click to reset
individual values
- **Section-level override count badges** on tabs
- **Browser-local banner** in panel header: "These settings are saved in
your browser only"
- **Auto-save status indicator** in footer: "All changes saved" /
"Saving..." / "⚠️ Storage full"
- **Export/Import** with full shape validation (`validateShape()`)
- **Presets** flow through the standard pipeline
(`writeOverrides(presetData) → pipeline`)
- **One-time migration** from 7 legacy localStorage keys (exact field
mapping per spec)
- **Validation** on all writes: color format, opacity range, timestamp
enum values
- **QuotaExceededError handling** with visible user warning

### Modified: `public/app.js`
Replaced ~80 lines of inline theme application code with a 15-line
`_customizerV2.init(cfg)` call. The customizer v2 handles all merging,
CSS application, and global state updates.

### Modified: `public/index.html`
Swapped `customize.js` → `customize-v2.js` script tag.

### Added: `docs/specs/customizer-rework.md`
The full approved spec, included in the repo for reference.

## Migration

On first page load:
1. Checks if `cs-theme-overrides` already exists → skip if yes
2. Reads all 7 legacy keys (`meshcore-user-theme`,
`meshcore-timestamp-*`, `meshcore-heatmap-opacity`,
`meshcore-live-heatmap-opacity`)
3. Maps them to the new delta format per the spec's field-by-field
mapping
4. Writes to `cs-theme-overrides`, removes all legacy keys
5. Continues with normal init

Users with existing customizations will see them preserved
automatically.

## Dark/Light Mode

- `theme` section stores light mode overrides, `themeDark` stores dark
mode overrides
- `meshcore-theme` localStorage key remains **separate** (view
preference, not customization)
- Switching modes re-runs the full pipeline with the correct section

## Testing

- All existing tests pass (`test-packet-filter.js`, `test-aging.js`,
`test-frontend-helpers.js`)
- Old `customize.js` is NOT modified — left in place for reference but
no longer loaded

## Not in Scope (per spec)

- Undo/redo stack
- Cross-tab synchronization
- Server-side admin import endpoint
- Map config / geo-filter overrides

---------

Co-authored-by: you <you@example.com>
2026-04-02 21:14:38 -07:00
Kpa-clawbot c9c473279e fix: add null-guards to rAF callbacks in live page animations (#506)
## Summary

Fixes #483 — navigating away from the live page while matrix/hop
animations are running throws `TypeError: Cannot read properties of null
(reading 'addLayer')`.

## Root Cause

`destroy()` sets `animLayer = null` and `pathsLayer = null`, but
in-flight `requestAnimationFrame` callbacks continue executing and
attempt to call `.addTo(animLayer)` or `.removeLayer()` on the now-null
references.

The entry guards at the top of `drawMatrixLine()` and
`drawAnimatedLine()` only protect the initial call — not the rAF
continuation loops inside `tick()`, `fadeOut()`, `animateLine()`, and
`animateFade()`.

## Fix

Added null-guards (`if (!animLayer || !pathsLayer) return`) at the top
of all four rAF callback functions in `live.js`:

1. **`tick()`** (line ~2203) — matrix animation main loop
2. **`fadeOut()`** (line ~2253) — matrix animation fade-out
3. **`animateLine()`** (line ~2302) — standard line animation main loop
4. **`animateFade()`** (line ~2337) — standard line fade-out

This pattern is already used elsewhere in the file (e.g., line 1873,
1886) for the same purpose.

## Testing

- All unit tests pass (`npm test` — 0 failures)
- Go server tests pass (`cmd/server` + `cmd/ingestor`)
- Change is defensive only (early return on null) — no behavioral change
when layers exist

---------

Co-authored-by: you <you@example.com>
2026-04-02 20:14:52 -07:00
Kpa-clawbot ad97c0fdd1 fix: clear stale parsed cache on observation packets (#505)
## Summary

Fixes #504 — Expanding a packet in the packets UI showed the same path
on every observation instead of each observation's unique path.

## Root Cause

PR #400 (fixing #387) added caching of `JSON.parse` results as
`_parsedPath` and `_parsedDecoded` properties on packet objects. When
observation packets are created via object spread (`{...parentPacket,
...obs}`), these cache properties are copied from the parent. Subsequent
calls to `getParsedPath(obsPacket)` hit the stale cache and return the
parent's path, ignoring the observation's own `path_json`.

## Fix

After every object spread that creates an observation packet from a
parent packet, delete the cache properties so they get re-parsed from
the observation's own data:

```js
delete obsPacket._parsedPath;
delete obsPacket._parsedDecoded;
```

Applied to all 5 spread sites in `public/packets.js`:
- Line 271: detail pane observation selection
- Line 504: flat view observation expansion
- Line 840: grouped view observation expansion
- Line 1012: child observation selection in grouped view
- Line 1982: WebSocket live update observation expansion

## Tests

Added 2 new tests in `test-frontend-helpers.js`:
1. Verifies observation packets get their own path after cache
invalidation (not the parent's)
2. Verifies observation path differs from parent path after cache
invalidation

All 431 frontend helper tests pass. All 62 packet filter tests pass.

---------

Co-authored-by: you <you@example.com>
2026-04-02 19:47:17 -07:00
P. Clawmogorov c7f655e419 perf(frontend): cache JSON.parse results for packet data (#400)
## Problem
As described in #387, `JSON.parse()` is called repeatedly on the same
packet data across render cycles. With 30K packets, each render cycle
parses 60K+ JSON strings unnecessarily.

## Analysis
The server sends `decoded_json` and `path_json` as JSON strings. The
frontend parses them on-demand in multiple locations:
- `renderTableRows()` — for every row, every render
- WebSocket handling — when processing filtered packets
- `loadPackets()` — during packet loading
- Detail view rendering — when showing packet details

This creates O(n×m) parsing overhead where n = packet count and m =
render cycles.

## Solution
Add cached parse helpers that store parsed results on the packet object:

```javascript
function getParsedPath(p) {
  if (p._parsedPath === undefined) {
    try { p._parsedPath = JSON.parse(p.path_json || '[]'); } catch { p._parsedPath = []; }
  }
  return p._parsedPath;
}
```

Same pattern for `getParsedDecoded()`.

## Changes
- `public/packets.js`: Add helpers + replace 15+ JSON.parse calls
- `public/live.js`: Add helpers + replace 5 JSON.parse calls

## Benchmarks
Before: 60K+ JSON.parse calls per render cycle (30K packets)
After: ~30K parse calls (one per packet, cached thereafter)

Memory impact: Negligible (stores parsed objects that were already
created temporarily)

## Notes
- Cache uses `undefined` check to distinguish "not cached" from "cached
empty result"
- Property names `_parsedPath` and `_parsedDecoded` prefixed to avoid
collision with server fields
- No breaking changes to existing code paths

Fixes #387

---------

Co-authored-by: P. Clawmogorov <262173731+Alm0stSurely@users.noreply.github.com>
Co-authored-by: you <you@example.com>
2026-04-03 01:11:02 +00:00
efiten b1d89d7d9f fix: apply region filter in GetNodes — was silently ignored (#496) (#497)
## Summary
- `db.GetNodes` accepted a `region` param from the HTTP handler but
never used it — every region-filter selection was silently ignored and
all nodes were always returned
- Added a subquery filtering `nodes.public_key` against ADVERT
transmissions (payload_type=4) observed by observers with matching IATA
codes
- Handles both v2 (`observer_id TEXT`) and v3 (`observer_idx INT`)
schemas

## Test plan
- [x] 4 new subtests added to `TestGetNodesFiltering`: SJC (1 node), SFO
(1 node), SJC,SFO multi (1 node deduped), AMS unknown (0 nodes)
- [x] All existing Go tests still pass
- [x] Deploy to staging, open `/nodes`, select a region in the filter
bar — only nodes observed by observers in that region should appear

Closes #496

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
2026-04-02 17:49:57 -07:00
efiten c173ab7e80 perf: skip JSON parse in indexByNode when no pubkey fields present (#376) (#499)
## Summary
- `indexByNode` was calling `json.Unmarshal` for every packet during
`Load()` and `IngestNewFromDB()`, even channel messages and other
payloads that can never contain node pubkey fields
- All three target fields (`"pubKey"`, `"destPubKey"`, `"srcPubKey"`)
share the common substring `"ubKey"` — added a `strings.Contains`
pre-check that skips the JSON parse entirely for packets that don't
match
- At 30K+ packets on startup, this eliminates the majority of
`json.Unmarshal` calls in `indexByNode` (channel messages, status
packets, etc. all bypass it)

## Test plan
- [x] 5 new subtests in `TestIndexByNodePreCheck`: ADVERT with pubKey
indexed, destPubKey indexed, channel message skipped, empty JSON
skipped, duplicate hash deduped
- [x] All existing Go tests pass
- [x] Deploy to staging and verify node-filtered packet queries still
work correctly

Closes #376

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
2026-04-02 17:44:02 -07:00
Jukka Väisänen 4664c90db4 fix: skip zero-hop adverts when checking node hash size (#493)
Fixes issue router IDs flapping between 1byte and multi-byte as
described in https://github.com/Kpa-clawbot/CoreScope/issues/303 with a
minimal patch + test coverage.

This fix is critical for regions using multi-byte IDs.

Closes https://github.com/Kpa-clawbot/CoreScope/issues/303

---------

Co-authored-by: you <you@example.com>
2026-04-03 00:33:20 +00:00
Kpa-clawbot 2755dc3875 test: push ingestor coverage from 70% to 84% (#344) (#492)
## Summary

Push Go ingestor test coverage from **70.2% → 84.0%** (92.8% excluding
the untestable `main()` and `init()` functions).

Part of #344 — ingestor coverage

## What Changed

Added `coverage_boost_test.go` with 60+ new test functions covering
previously untested code paths:

### Coverage Before → After by Function

| Function | Before | After |
|----------|--------|-------|
| `NodeDaysOrDefault` | 0% | 100% |
| `MoveStaleNodes` | 0% | 76.5% |
| `NodePassesGeoFilter` | 40% | 100% |
| `handleMessage` | 41.4% | 92.1% |
| `ResolvedSources` | 71.4% | 100% |
| `extractObserverMeta` | 100% | 100% |
| `decodeAdvert` | 88.2% | 94.1% |
| `decryptChannelMessage` | 88.4% | 93.0% |
| **Total** | **70.2%** | **84.0%** |

### Test Categories Added

- **Config**: `NodeDaysOrDefault` all branches, broker scheme
normalization (`mqtt://` → `tcp://`, `mqtts://` → `ssl://`)
- **Database**: `MoveStaleNodes` (stale/fresh/replace), duplicate
transmission handling, default timestamps, telemetry updates, schema
migration verification
- **Decoder**: Sensor telemetry parsing, location + features with
truncated data, `countNonPrintable` with invalid UTF-8,
`decryptChannelMessage` error paths (invalid
key/MAC/ciphertext/alignment), short payload handling
- **Geo Filter**: All branches (nil filter, nil coords, inside/outside)
- **Message Handler**: Channel messages (with/without sender, empty
text), direct messages, geo-filtered adverts, corrupted adverts
(all-zero pubkey), non-advert packets, `Score`/`Direction`
case-insensitive fallbacks, status messages with full hardware metadata

### Why Not 90%+

The remaining ~16% uncovered statements are:
- `main()` function (68 blocks) — program entry point with MQTT client
setup, signal handling, goroutines — not unit-testable without major
refactoring
- `init()` function — `--version` flag + `os.Exit(0)` — kills the test
process
- `prepareStatements()` error returns — only trigger on
corrupted/incompatible SQLite databases
- `applySchema()` migration error paths — only trigger on
filesystem/SQLite failures

Excluding `main()` and `init()`, effective coverage is **92.8%**.

## Test Results

All 100+ tests pass (existing + new):
```
ok  github.com/corescope/ingestor  25.945s  coverage: 84.0% of statements
```

---------

Co-authored-by: you <you@example.com>
2026-04-02 17:31:47 -07:00
efiten 5228e67604 fix: use packet timestamp in bufferPacket instead of arrival time (#475) (#491)
## Summary
- `bufferPacket()` was overwriting `_ts` with `Date.now()` (receive
time) for every live WS packet
- Packets arriving in the same batch all got identical timestamps,
making the message history show the same "Xs ago" for every entry (e.g.,
all show "5s ago")
- Fix: use `pkt.timestamp || pkt.created_at` (mirroring
`dbPacketToLive`) so each packet reflects its actual origination time,
falling back to `Date.now()` only when the packet has no timestamp

## Root cause
```js
// before
pkt._ts = Date.now();

// after
pkt._ts = new Date(pkt.timestamp || pkt.created_at || Date.now()).getTime();
```

The WS broadcast includes `timestamp` (= `tx.FirstSeen`) in the packet
map (store.go:1182), so the field is always present for real packets.

## Test plan
- [x] Open Live page, observe packets arriving — each should show its
own relative time, not all the same value
- [x] `node test-frontend-helpers.js` passes (235 tests, 0 failures)

Closes #475

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: you <you@example.com>
2026-04-02 17:30:55 -07:00
Kpa-clawbot 698514e5e6 test: comprehensive live.js coverage (71 tests) (#489)
## Summary

Add comprehensive test coverage for `live.js` — the largest and most
complex frontend file (2500+ lines) covering animation modes, VCR
playback, WebSocket handling, audio integration, and the live map.

Part of #344 — live.js coverage.

## What's Tested (71 tests)

### Pure function tests via `vm.createContext`
- **`dbPacketToLive`** — DB packet → live format conversion, null
`decoded_json`, `payload_type_name` fallback, `created_at` timestamp
fallback
- **`expandToBufferEntries`** — observation expansion (1→N entries),
empty observations, multi-packet batches
- **`SEG_MAP`** — 7-segment LCD digit mapping completeness (all digits,
colon, space, VCR mode letters)
- **VCR state machine** — mode transitions (`LIVE`→`PAUSED`→`REPLAY`),
`frozenNow` lifecycle, speed cycling (1→2→4→8→1), pause idempotency
- **`getFavoritePubkeys`** — localStorage merging from
`meshcore-favorites` + `meshcore-my-nodes`, corrupt data handling, falsy
filtering
- **`packetInvolvesFavorite`** — sender pubKey matching, hop prefix
matching, missing decoded fields
- **`isNodeFavorited`** — basic favorite lookup, empty state
- **`formatLiveTimestampHtml`** — timestamp formatting with tooltip,
null input, numeric input, future warning icon
- **`resolveHopPositions`** — HopResolver integration, ghost hop
interpolation between known nodes
- **`bufferPacket`** — VCR buffer management, 2000-entry cap with
playhead adjustment, missed count in PAUSED mode

### Source-level safety checks (20 tests)
- Null guards: `renderPacketTree`, `animatePath`, `pulseNode`, `nextHop`
(all verified via source-level checks)
- Animation limit enforcement (`MAX_CONCURRENT_ANIMS`)
- Tab visibility optimization (skip animations when hidden, clear
propagation buffer on restore)
- WebSocket auto-reconnect
- `addNodeMarker` deduplication
- All toggle state persistence to localStorage (matrix, rain, realistic,
favorites, ghost hops)
- `clearNodeMarkers` resets HopResolver
- `startReplay` pre-aggregates by hash
- Orientation change retry delays
- `vcrRewind` deduplicates buffer entries by ID

## Changes
- `public/live.js` — expose 14 additional functions via `window._live*`
for testing (following existing pattern)
- `test-live.js` — new test file, 841 lines, 71 tests

## Constraints
- No new dependencies
- Tests run via `vm.createContext` against real code (not copies)
- No build step — vanilla JS

---------

Co-authored-by: you <you@example.com>
2026-04-02 17:16:03 -07:00
Kpa-clawbot cf3a383bb2 test: comprehensive app.js coverage — 100+ new tests (#490)
## Summary

Adds 100+ new tests for previously untested `app.js` functions,
significantly improving frontend coverage toward the 90%+ target.

## What's Tested

All pure/testable functions from `app.js` that lacked coverage:

| Function Group | Tests Added | Description |
|---|---|---|
| `payloadTypeColor` | 13 | All PAYLOAD_COLORS mappings +
unknown/null/undefined fallback |
| `pad2` / `pad3` | 10 | Zero-padding for 1-3 digit values, no
truncation |
| `formatIsoLike` | 5 | UTC/local timezone, with/without milliseconds,
zero-padding |
| `formatTimestampCustom` | 5 | Token replacement
(YYYY/MM/DD/HH/mm/ss/SSS/Z), partial formats, invalid format rejection |
| `formatAbsoluteTimestamp` | 3 | Custom format integration, locale+UTC,
null/invalid date handling |
| `getTimestamp*` getters | 11 | localStorage priority, server config
fallback, invalid value rejection for
Mode/Timezone/FormatPreset/CustomFormat |
| `invalidateApiCache` | 3 | Prefix-based selective invalidation, full
clear, cache→invalidate→re-fetch lifecycle |
| `formatHex` | 5 | Byte spacing, single byte, null/empty, odd-length
hex |
| `createColoredHexDump` | 6 | Range-based coloring, override
precedence, null/empty hex+ranges |
| `buildHexLegend` | 5 | Label deduplication, correct swatch colors per
label class, null/empty |
| Favorites (`getFavorites`/`isFavorite`/`toggleFavorite`/`favStar`) | 9
| CRUD operations, corrupt JSON resilience, star HTML rendering with
custom classes |
| `debounce` | 3 | Delay behavior, timer reset on rapid calls, argument
forwarding |
| `mergeUserHomeConfig` | 5 | Null/missing siteConfig/userTheme,
non-object home, missing home creation |
| Constants | 2 | Exhaustive ROUTE_TYPES (4) and PAYLOAD_TYPES (13)
mapping verification |

## Approach

- Tests use the existing `vm.createContext` sandbox pattern from
`test-frontend-helpers.js`
- Tests the **real code** loaded from `public/app.js` — no copies
- No new dependencies
- Each `invalidateApiCache` test uses an isolated sandbox to avoid async
race conditions

## Test Results

```
Frontend helpers: 343 passed, 0 failed
```

Part of #344 — app.js coverage

---------

Co-authored-by: you <you@example.com>
2026-04-02 17:03:35 -07:00
efiten a45ac71508 fix: restore color-coded hex breakdown in packet detail (#329) (#500)
## Summary
- `BuildBreakdown` was never ported from the deleted Node.js
`decoder.js` to Go — the server has returned `breakdown: {}` since the
Go migration (commit `742ed865`), so `createColoredHexDump()` and
`buildHexLegend()` in the frontend always received an empty `ranges`
array and rendered everything as monochrome
- Implemented `BuildBreakdown()` in `decoder.go` — computes labeled byte
ranges matching the frontend's `LABEL_CLASS` map: `Header`, `Transport
Codes`, `Path Length`, `Path`, `Payload`; ADVERT packets get sub-ranges:
`PubKey`, `Timestamp`, `Signature`, `Flags`, `Latitude`, `Longitude`,
`Name`
- Wired into `handlePacketDetail` (was `struct{}{}`)
- Also adds per-section color classes to the field breakdown table
(`section-header`, `section-transport`, `section-path`,
`section-payload`) so the table rows get matching background tints

## Test plan
- [x] Open any packet detail pane — hex dump should show color-coded
sections (red header, orange path length, blue transport codes, green
path hops, yellow/colored payload)
- [x] Legend below action buttons should appear with color swatches
- [x] ADVERT packets: PubKey/Timestamp/Signature/Flags each get their
own distinct color
- [x] Field breakdown table section header rows should be tinted per
section
- [x] 8 new Go tests: all pass

Closes #329

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-02 16:54:59 -07:00
Kpa-clawbot 016b87b33c test: add 64 unit tests for packets.js (Part of #344) (#488)
## Summary

Adds 64 unit tests for `packets.js` — the largest untested frontend file
(2000+ lines) covering filter engine integration, time window logic,
groupByHash rendering, and packet detail display.

Part of #344 — packets.js coverage.

## Approach

Follows the existing `test-frontend-helpers.js` pattern: loads real
source files into a `vm.createContext` sandbox and tests actual code (no
copies).

Added a `window._packetsTestAPI` export at the end of the packets.js
IIFE to expose pure functions for testing without changing any runtime
behavior.

## What's Tested

| Function | Tests | What it covers |
|----------|-------|----------------|
| `typeName` | 2 | Type code → name mapping, unknown fallback |
| `obsName` | 2 | Observer name lookup, falsy/missing handling |
| `kv` | 1 | Key-value HTML helper |
| `sectionRow` / `fieldRow` | 3 | Table section/field HTML builders |
| `getDetailPreview` | 17 | All packet types: CHAN, ADVERT
(repeater/room/sensor/companion), GRP_TXT
(no_key/decryption_failed/channelHashHex), TXT_MSG, PATH, REQ, RESPONSE,
ANON_REQ, text fallback, public_key fallback, empty |
| `getPathHopCount` | 4 | Valid path, empty, null, invalid JSON |
| `sortGroupChildren` | 3 | Default observer sort, header update, null
safety |
| `renderTimestampCell` | 2 | Timestamp HTML output, null handling |
| `renderPath` | 3 | Empty/null, multi-hop with arrows, single hop |
| `renderDecodedPacket` | 6 | Header/path/payload/nested objects/null
skip/raw hex |
| `buildFieldTable` | 11 | All payload types (ADVERT with
flags/location/name, GRP_TXT, CHAN, ACK, destHash, raw fallback),
transport codes, path hops, hash_size calculation, empty hex |
| `_getRowCount` | 1 | Virtual scroll row counting |
| `buildFlatRowHtml` | 3 | Row rendering, size calculation, missing hex
|
| `buildGroupRowHtml` | 3 | Single/multi group, observation badge |
| Test API exposure | 1 | Verifies window._packetsTestAPI |

## Constraints Met

- No new test dependencies
- Tests real code via `vm.createContext`, not copies
- No build step — vanilla JS
- All existing tests still pass (254 frontend-helpers, 62 packet-filter,
29 aging)

Co-authored-by: you <you@example.com>
2026-04-02 16:42:25 -07:00
Kpa-clawbot 889107a5e1 fix: address PR #487 review feedback (#501)
## Summary

Addresses review feedback from PR #487 (nodes.js coverage).

### Changes

1. **Replace fragile `exportInternals` regex source patching with stable
test hooks** — `getStatusInfo` and `getStatusTooltip` are now exposed
via `window._nodesGetStatusInfo` and `window._nodesGetStatusTooltip`,
matching the existing pattern used by all other test-accessible
functions. The brittle regex `.replace()` approach that modified source
code at runtime has been removed entirely.

2. **Strengthen weak null assertion** — The `renderNodeTimestampHtml
handles null` test previously asserted `html.includes('—') ||
html.length > 0`, which is a near-tautology (any non-empty string
passes). Now strictly asserts `html.includes('—')`.

### Files changed
- `public/nodes.js` — 2 new test hook lines
- `test-frontend-helpers.js` — removed 21-line `exportInternals` branch,
updated tests to use hooks

### Testing
- All 309 frontend helper tests pass
- All 62 packet filter tests pass
- All 29 aging tests pass

Closes review items from #487.

Co-authored-by: you <you@example.com>
2026-04-02 16:40:11 -07:00
Kpa-clawbot 50f94603c1 test: P0 coverage for nodes.js — sort, status, timestamps, sync (#487)
## Summary

Add 67 new unit tests for `nodes.js`, raising frontend helper test count
from 233 to 300.

Part of #344 — nodes.js coverage.

## What's Tested

### Sort System (`toggleSort`, `sortNodes`, `sortArrow`)
- Direction toggling on same column (asc↔desc)
- Default sort directions per column type (name→asc, last_seen→desc,
advert_count→desc)
- localStorage persistence of sort state
- All 5 sort columns: `name`, `public_key`, `role`, `last_seen`,
`advert_count`
- Both ascending and descending for each column
- Case-insensitive name sorting
- Unnamed nodes sort last
- Timestamp fallback chain: `last_heard` → `last_seen` → 0
- Missing timestamp handling
- Empty array edge case
- Unknown column graceful handling
- `sortArrow` rendering for active (▲/▼) and inactive columns

### Status Calculation (`getStatusInfo`, `getStatusTooltip`)
- `_lastHeard` takes priority over `last_heard`
- `last_seen` used as fallback when `last_heard` missing
- No-timestamp nodes return stale with `lastHeardMs: 0`
- Infrastructure threshold (72h) for rooms
- Standard threshold (24h) for sensors and companions
- Explanation text varies by role and status
- Unknown role defaults to gray color `#6b7280`
- All role/status tooltip combinations

### Timestamp Rendering (`renderNodeTimestampHtml`,
`renderNodeTimestampText`)
- HTML output includes tooltip and `timestamp-text` class
- Future timestamps show ⚠️ warning icon
- Null input produces dash
- Text output is plain (no HTML tags)

### Favorites Sync (`syncClaimedToFavorites`)
- Claimed pubkeys added to favorites
- No-op when all already synced
- Empty my-nodes handled
- Missing localStorage keys don't crash

## Implementation

- Added test hooks on `window` for closure-scoped functions
(non-invasive, follows existing pattern)
- Tests use `vm.createContext` to load real `nodes.js` code — no copies
- No new dependencies

## Test Results

```
Frontend helpers: 300 passed, 0 failed
```

---------

Co-authored-by: you <you@example.com>
2026-04-02 23:32:41 +00:00
efiten b799f54700 perf: bound memory growth and reduce render CPU on packets page (#421)
## Problem

On a long-running session the packets page consumed 8 GB of browser
memory and 20%+ CPU on an 8-core machine. Root causes:

1. **Unbounded `packets` array growth via WebSocket** —
`packets.unshift()` was called for every new unique hash, but nothing
ever trimmed the array. After hours of live traffic the array grew well
past the initial 50 k load limit.
2. **Unbounded `pauseBuffer`** — all WS messages queued while paused, no
cap.
3. **Unbounded `_children` growth** — expanded groups received a
`unshift(p)` on every matching WS message with no size limit.
4. **O(n) `observers.find()` inside the O(n) render loop** — with 50 k
rows, each render triggered up to 50 k linear scans through the
observers list.
5. **Full DOM rebuild on every WS message** — `renderTableRows()` was
called synchronously on every WebSocket batch, reconstructing the entire
table on each incoming packet.

## Changes

- `packets[]` is now trimmed to `PACKET_LIMIT` after each WS batch;
evicted entries are also removed from `hashIndex` to prevent stale
references.
- `pauseBuffer` capped at 2 000 entries (oldest dropped).
- `_children` capped at 200 entries on WS prepend.
- `renderTableRows()` on the WS path is debounced to 200 ms, batching
rapid updates into a single redraw.
- `observersById = new Map()` pre-built from the observers array; all
`observers.find()` calls in the render loop and WS filter replaced with
O(1) `Map.get()`.

## Test plan

- [x] Load the packets page and leave it running for several minutes
with live WebSocket traffic — memory in DevTools should remain stable
rather than growing continuously
- [x] Pause live updates, wait for several messages, then resume —
buffer replays correctly and display updates
- [x] Expand a packet group and leave it open during live traffic —
children update but don't grow past 200
- [x] Region filter still works correctly (relies on the observer Map
lookup)
- [x] Observer name / IATA badge renders correctly in grouped and flat
mode

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-04-02 16:04:01 -07:00
Kpa-clawbot d5b300a8ba fix: derive version from git tags instead of package.json (#486)
## Summary

Fixes #485 — the app version was derived from `package.json` via
Node.js, which is a meaningless artifact for this Go project. This
caused version mismatches (e.g., v3.3.0 release showing "3.2.0") when
someone forgot to bump `package.json`.

## Changes

### `manage.sh`
- **Line 43**: Replace `node -p "require('./package.json').version"`
with `git describe --tags --match "v*"` — version is now derived
automatically from git tags
- **Line 515**: Add `--force` to `git fetch origin --tags` in setup
command
- **Line 1320**: Add `--force` to `git fetch origin --tags` in update
command — prevents "would clobber existing tag" errors when tags are
moved

### `package.json`
- Version field set to `0.0.0-use-git-tags` to make it clear this is not
the source of truth. File kept because npm scripts and devDependencies
are still used for testing.

## How it works

`git describe --tags --match "v*"` produces:
- `v3.3.0` — when on an exact tag
- `v3.3.0-3-gabcdef1` — when 3 commits after a tag (useful for
debugging)
- Falls back to `unknown` if no tags exist

## Testing

- All Go tests pass (`cmd/server`, `cmd/ingestor`)
- All frontend unit tests pass (254/254)
- No changes to application logic — only build-time version derivation

Co-authored-by: you <you@example.com>
2026-04-02 00:53:38 -07:00
174 changed files with 50832 additions and 1992 deletions
+144 -27
View File
@@ -3,10 +3,15 @@ name: CI/CD Pipeline
on:
push:
branches: [master]
tags: ['v*']
pull_request:
branches: [master]
workflow_dispatch:
permissions:
contents: read
packages: write
concurrency:
group: ci-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
@@ -18,8 +23,8 @@ env:
STAGING_CONTAINER: corescope-staging-go
# Pipeline (sequential, fail-fast):
# go-test → e2e-test → build → deploy → publish
# PRs stop after build. Master continues to deploy + publish.
# go-test → e2e-test → build-and-publish → deploy → publish-badges
# PRs stop after build-and-publish (no GHCR push). Master continues to deploy + badges.
jobs:
# ───────────────────────────────────────────────────────────────
@@ -63,6 +68,17 @@ jobs:
echo "--- Go Ingestor Coverage ---"
go tool cover -func=ingestor-coverage.out | tail -1
- name: Build and test channel library + decrypt CLI
run: |
set -e -o pipefail
cd internal/channel
go test ./...
echo "--- Channel library tests passed ---"
cd ../../cmd/decrypt
CGO_ENABLED=0 go build -ldflags="-s -w" -o corescope-decrypt .
go test ./...
echo "--- Decrypt CLI tests passed ---"
- name: Verify proto syntax
run: |
set -e
@@ -231,54 +247,154 @@ jobs:
include-hidden-files: true
# ───────────────────────────────────────────────────────────────
# 3. Build Docker Image
# 3. Build & Publish Docker Image
# ───────────────────────────────────────────────────────────────
build:
name: "🏗️ Build Docker Image"
build-and-publish:
name: "🏗️ Build & Publish Docker Image"
needs: [e2e-test]
runs-on: [self-hosted, Linux]
runs-on: [self-hosted, meshcore-runner-2]
steps:
- name: Checkout code
uses: actions/checkout@v5
- name: Set up Node.js 22
uses: actions/setup-node@v5
with:
node-version: '22'
- name: Free disk space
run: |
docker system prune -af 2>/dev/null || true
docker builder prune -af 2>/dev/null || true
df -h /
- name: Build Go Docker image
- name: Compute build metadata
id: meta
run: |
echo "${GITHUB_SHA::7}" > .git-commit
APP_VERSION=$(node -p "require('./package.json').version") \
GIT_COMMIT="${GITHUB_SHA::7}" \
APP_VERSION=$(grep -oP 'APP_VERSION:-\K[^}]+' docker-compose.yml | head -1 || echo "3.0.0")
GIT_COMMIT=$(git rev-parse --short HEAD)
BUILD_TIME=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
export APP_VERSION GIT_COMMIT BUILD_TIME
GIT_COMMIT="${GITHUB_SHA::7}"
if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
APP_VERSION="${GITHUB_REF#refs/tags/}"
else
APP_VERSION="edge"
fi
echo "build_time=$BUILD_TIME" >> "$GITHUB_OUTPUT"
echo "git_commit=$GIT_COMMIT" >> "$GITHUB_OUTPUT"
echo "app_version=$APP_VERSION" >> "$GITHUB_OUTPUT"
echo "Build: version=$APP_VERSION commit=$GIT_COMMIT time=$BUILD_TIME"
- name: Build Go Docker image (local staging)
run: |
GIT_COMMIT="${{ steps.meta.outputs.git_commit }}" \
APP_VERSION="${{ steps.meta.outputs.app_version }}" \
BUILD_TIME="${{ steps.meta.outputs.build_time }}" \
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging build "$STAGING_SERVICE"
echo "Built Go staging image ✅"
- name: Set up Docker Buildx
if: github.event_name == 'push'
uses: docker/setup-buildx-action@v3
- name: Set up QEMU (arm64 runtime stage)
if: github.event_name == 'push'
uses: docker/setup-qemu-action@v3
- name: Log in to GHCR
if: github.event_name == 'push'
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract Docker metadata
if: github.event_name == 'push'
id: docker-meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/kpa-clawbot/corescope
tags: |
type=semver,pattern=v{{version}}
type=semver,pattern=v{{major}}.{{minor}}
type=semver,pattern=v{{major}}
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=edge,branch=master
- name: Build and push to GHCR
if: github.event_name == 'push'
uses: docker/build-push-action@v6
with:
context: .
push: true
platforms: linux/amd64,linux/arm64
tags: ${{ steps.docker-meta.outputs.tags }}
labels: ${{ steps.docker-meta.outputs.labels }}
build-args: |
APP_VERSION=${{ steps.meta.outputs.app_version }}
GIT_COMMIT=${{ steps.meta.outputs.git_commit }}
BUILD_TIME=${{ steps.meta.outputs.build_time }}
cache-from: type=gha
cache-to: type=gha,mode=max
# ───────────────────────────────────────────────────────────────
# 4. Deploy Staging (master only)
# 4. Release Artifacts (tags only)
# ───────────────────────────────────────────────────────────────
deploy:
name: "🚀 Deploy Staging"
if: github.event_name == 'push'
needs: [build]
runs-on: [self-hosted, Linux]
release-artifacts:
name: "📦 Release Artifacts"
if: startsWith(github.ref, 'refs/tags/v')
needs: [go-test]
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Checkout code
uses: actions/checkout@v5
- name: Set up Go 1.22
uses: actions/setup-go@v6
with:
go-version: '1.22'
- name: Build corescope-decrypt (static, linux/amd64)
run: |
cd cmd/decrypt
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w -X main.version=${{ github.ref_name }}" -o ../../corescope-decrypt-linux-amd64 .
- name: Build corescope-decrypt (static, linux/arm64)
run: |
cd cmd/decrypt
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags="-s -w -X main.version=${{ github.ref_name }}" -o ../../corescope-decrypt-linux-arm64 .
- name: Upload release assets
uses: softprops/action-gh-release@v2
with:
files: |
corescope-decrypt-linux-amd64
corescope-decrypt-linux-arm64
# ───────────────────────────────────────────────────────────────
# 4b. Deploy Staging (master only)
# ───────────────────────────────────────────────────────────────
deploy:
name: "🚀 Deploy Staging"
if: github.event_name == 'push'
needs: [build-and-publish]
runs-on: [self-hosted, meshcore-runner-2]
steps:
- name: Checkout code
uses: actions/checkout@v5
- name: Pull latest image from GHCR
run: |
# Try to pull the edge image from GHCR and tag for docker-compose compatibility
if docker pull ghcr.io/kpa-clawbot/corescope:edge; then
docker tag ghcr.io/kpa-clawbot/corescope:edge corescope-go:latest
echo "Pulled and tagged GHCR edge image ✅"
else
echo "⚠️ GHCR pull failed — falling back to locally built image"
fi
- name: Deploy staging
run: |
# Stop old container and release memory
# Force-remove the staging container regardless of how it was created
# (compose-managed OR manually created via docker run)
docker stop corescope-staging-go 2>/dev/null || true
docker rm -f corescope-staging-go 2>/dev/null || true
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging down --timeout 30 2>/dev/null || true
# Wait for container to be fully gone and OS to reclaim memory (3GB limit)
@@ -320,10 +436,11 @@ jobs:
- name: Smoke test staging API
run: |
if curl -sf http://localhost:82/api/stats | grep -q engine; then
PORT="${STAGING_GO_HTTP_PORT:-80}"
if curl -sf "http://localhost:${PORT}/api/stats" | grep -q engine; then
echo "Staging verified — engine field present ✅"
else
echo "Staging /api/stats did not return engine field"
echo "Staging /api/stats did not return engine field (port ${PORT})"
exit 1
fi
+6
View File
@@ -362,6 +362,12 @@ One logical change per commit. Each commit is deployable. Each commit has its te
- Tests: `test-{feature}.js` in repo root
- No build step, no transpilation — write ES2020 for server, ES5/6 for frontend (broad browser support)
### Deep Linking
All new UI states that a user might want to share or bookmark MUST be reflected in the URL hash.
This includes: tabs, filters, selected items, view modes. Use query parameters on the hash
(e.g., `#/packets?observer=ABC&timeRange=24h`) for filter state.
Existing patterns: `#/nodes/{pubkey}?section=node-neighbors`, `#/analytics?tab=collisions`, `#/packets/{hash}`.
## What NOT to Do
- **Don't check in private information** — no names, API keys, tokens, passwords, IP addresses, personal data, or any identifying information. This is a PUBLIC repo.
- Don't add npm dependencies without asking
+131
View File
@@ -0,0 +1,131 @@
# Deploy CoreScope
Pre-built images are published to GHCR for `linux/amd64` and `linux/arm64` (Raspberry Pi 4/5).
## Quick Start
### Docker run
```bash
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
-e DISABLE_CADDY=true \
ghcr.io/kpa-clawbot/corescope:latest
```
Open `http://localhost` — done.
### Docker Compose
```bash
curl -sL https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/docker-compose.example.yml \
-o docker-compose.yml
docker compose up -d
```
## Image Tags
| Tag | Description |
|-----|-------------|
| `v3.4.1` | Pinned release (recommended for production) |
| `v3.4` | Latest patch in v3.4.x |
| `v3` | Latest minor+patch in v3.x |
| `latest` | Latest release tag |
| `edge` | Built from master — unstable, for testing |
## Configuration
Settings can be overridden via environment variables:
| Variable | Default | Description |
|----------|---------|-------------|
| `DISABLE_CADDY` | `false` | Skip internal Caddy (set `true` behind a reverse proxy) |
| `DISABLE_MOSQUITTO` | `false` | Skip internal MQTT broker (use external) |
| `HTTP_PORT` | `80` | Host port mapping |
| `DATA_DIR` | `./data` | Host path for persistent data |
For advanced configuration, mount a `config.json` into `/app/data/config.json`. See `config.example.json` in the repo.
## Updating
```bash
docker compose pull
docker compose up -d
```
## Data
All persistent data lives in `/app/data`:
- `meshcore.db` — SQLite database (packets, nodes)
- `config.json` — custom config (optional)
- `theme.json` — custom theme (optional)
**Backup:** `cp data/meshcore.db ~/backup/`
## TLS
Option A — **External reverse proxy** (recommended): Run with `DISABLE_CADDY=true`, put nginx/traefik/Cloudflare in front.
Option B — **Built-in Caddy**: Mount a custom Caddyfile at `/etc/caddy/Caddyfile` and expose ports 80+443.
---
## Migrating from manage.sh (existing admins)
If you're currently deploying with `manage.sh` (git clone + local build), you have two options going forward:
### Option A: Keep using manage.sh (no changes needed)
`manage.sh update` continues to work exactly as before — it fetches the latest tag, builds locally, and restarts. Nothing breaks.
```bash
./manage.sh update # latest release
./manage.sh update v3.5.0 # specific version
```
### Option B: Switch to pre-built images (recommended)
Pre-built images skip the build step entirely — faster updates, no Go toolchain needed.
**One-time migration:**
1. Stop the current deployment:
```bash
./manage.sh stop
```
2. Your data is in `~/meshcore-data/` (or whatever `PROD_DATA_DIR` is set to). It's untouched — the database, config, and theme files persist.
3. Copy `docker-compose.example.yml` to where you want to run from:
```bash
cp docker-compose.example.yml ~/docker-compose.yml
```
4. Start with the pre-built image:
```bash
cd ~ && docker compose up -d
```
5. Verify it picked up your existing data:
```bash
curl http://localhost/api/stats
```
**Updates after migration:**
```bash
docker compose pull && docker compose up -d
```
### What about manage.sh features?
| manage.sh command | Pre-built equivalent |
|---|---|
| `./manage.sh update` | `docker compose pull && docker compose up -d` |
| `./manage.sh stop` | `docker compose down` |
| `./manage.sh start` | `docker compose up -d` |
| `./manage.sh logs` | `docker compose logs -f` |
| `./manage.sh status` | `docker compose ps` |
| `./manage.sh setup` | Copy `docker-compose.example.yml`, edit env vars |
`manage.sh` remains available for advanced use cases (building from source, custom patches, development). Pre-built images are recommended for most production deployments.
+28 -7
View File
@@ -1,25 +1,44 @@
FROM golang:1.22-alpine AS builder
RUN apk add --no-cache build-base
# Build stage always runs natively on the builder's arch ($BUILDPLATFORM)
# and cross-compiles to $TARGETOS/$TARGETARCH via Go toolchain. No QEMU.
FROM --platform=$BUILDPLATFORM golang:1.22-alpine AS builder
ARG APP_VERSION=unknown
ARG GIT_COMMIT=unknown
ARG BUILD_TIME=unknown
# Provided by buildx for multi-arch builds
ARG TARGETOS
ARG TARGETARCH
# Build server
# Build server (pure-Go sqlite — no CGO needed, cross-compiles cleanly)
WORKDIR /build/server
COPY cmd/server/go.mod cmd/server/go.sum ./
COPY internal/geofilter/ ../../internal/geofilter/
COPY internal/sigvalidate/ ../../internal/sigvalidate/
COPY internal/packetpath/ ../../internal/packetpath/
RUN go mod download
COPY cmd/server/ ./
RUN go build -ldflags "-X main.Version=${APP_VERSION} -X main.Commit=${GIT_COMMIT} -X main.BuildTime=${BUILD_TIME}" -o /corescope-server .
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -ldflags "-X main.Version=${APP_VERSION} -X main.Commit=${GIT_COMMIT} -X main.BuildTime=${BUILD_TIME}" -o /corescope-server .
# Build ingestor
WORKDIR /build/ingestor
COPY cmd/ingestor/go.mod cmd/ingestor/go.sum ./
COPY internal/geofilter/ ../../internal/geofilter/
COPY internal/sigvalidate/ ../../internal/sigvalidate/
COPY internal/packetpath/ ../../internal/packetpath/
RUN go mod download
COPY cmd/ingestor/ ./
RUN go build -o /corescope-ingestor .
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -o /corescope-ingestor .
# Build decrypt CLI
WORKDIR /build/decrypt
COPY cmd/decrypt/go.mod cmd/decrypt/go.sum ./
COPY internal/channel/ ../../internal/channel/
RUN go mod download
COPY cmd/decrypt/ ./
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
go build -ldflags="-s -w" -o /corescope-decrypt .
# Runtime image
FROM alpine:3.20
@@ -29,7 +48,7 @@ RUN apk add --no-cache mosquitto mosquitto-clients supervisor caddy wget
WORKDIR /app
# Go binaries
COPY --from=builder /corescope-server /corescope-ingestor /app/
COPY --from=builder /corescope-server /corescope-ingestor /corescope-decrypt /app/
# Frontend assets + config
COPY public/ ./public/
@@ -42,6 +61,8 @@ RUN echo "unknown" > .git-commit
# Supervisor + Mosquitto + Caddy config
COPY docker/supervisord-go.conf /etc/supervisor/conf.d/supervisord.conf
COPY docker/supervisord-go-no-mosquitto.conf /etc/supervisor/conf.d/supervisord-no-mosquitto.conf
COPY docker/supervisord-go-no-caddy.conf /etc/supervisor/conf.d/supervisord-no-caddy.conf
COPY docker/supervisord-go-no-mosquitto-no-caddy.conf /etc/supervisor/conf.d/supervisord-no-mosquitto-no-caddy.conf
COPY docker/mosquitto.conf /etc/mosquitto/mosquitto.conf
COPY docker/Caddyfile /etc/caddy/Caddyfile
+3
View File
@@ -40,6 +40,9 @@ RUN if [ ! -f .git-commit ]; then echo "unknown" > .git-commit; fi
# Supervisor + Mosquitto + Caddy config
COPY docker/supervisord-go.conf /etc/supervisor/conf.d/supervisord.conf
COPY docker/supervisord-go-no-mosquitto.conf /etc/supervisor/conf.d/supervisord-no-mosquitto.conf
COPY docker/supervisord-go-no-caddy.conf /etc/supervisor/conf.d/supervisord-no-caddy.conf
COPY docker/supervisord-go-no-mosquitto-no-caddy.conf /etc/supervisor/conf.d/supervisord-no-mosquitto-no-caddy.conf
COPY docker/mosquitto.conf /etc/mosquitto/mosquitto.conf
COPY docker/Caddyfile /etc/caddy/Caddyfile
+674
View File
@@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.
+29 -4
View File
@@ -74,9 +74,34 @@ Full experience on your phone — proper touch controls, iOS safe area support,
## Quick Start
### Docker (Recommended)
### Pre-built Image (Recommended)
No Go installation needed — everything builds inside the container.
No build step required — just run:
```bash
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 1883:1883 \
-v /your/data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
```
Open `http://localhost` — done. No config file needed; CoreScope starts with sensible defaults.
For HTTPS with a custom domain, add `-p 443:443` and mount your Caddyfile:
```bash
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 443:443 -p 1883:1883 \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:latest
```
Disable built-in services with `-e DISABLE_MOSQUITTO=true` or `-e DISABLE_CADDY=true`, or drop a `.env` file in your data volume. See [docs/deployment.md](docs/deployment.md) for the full reference.
### Build from Source
```bash
git clone https://github.com/Kpa-clawbot/CoreScope.git
@@ -95,8 +120,6 @@ The setup wizard walks you through config, domain, HTTPS, build, and run.
./manage.sh help # All commands
```
See [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) for the full deployment guide — HTTPS options (auto cert, bring your own, Cloudflare Tunnel), MQTT security, backups, and troubleshooting.
### Configure
Copy `config.example.json` to `config.json` and edit:
@@ -242,6 +265,8 @@ Contributions welcome. Please read [AGENTS.md](AGENTS.md) for coding conventions
**Live instance:** [analyzer.00id.net](https://analyzer.00id.net) — all API endpoints are public, no auth required.
**API Documentation:** CoreScope auto-generates an OpenAPI 3.0 spec. Browse the interactive Swagger UI at [`/api/docs`](https://analyzer.00id.net/api/docs) or fetch the machine-readable spec at [`/api/spec`](https://analyzer.00id.net/api/spec).
## License
MIT
+142
View File
@@ -0,0 +1,142 @@
# corescope-decrypt
Standalone CLI tool to decrypt and export MeshCore hashtag channel messages from a CoreScope SQLite database.
## Why
MeshCore hashtag channels use symmetric encryption where the key is derived deterministically from the channel name. The CoreScope ingestor stores **all** `GRP_TXT` packets in the database, including those it cannot decrypt at ingest time.
This tool enables:
- **Retroactive decryption** — decrypt historical messages for any channel whose name you learn after the fact
- **Forensics & analysis** — export channel traffic for offline review
- **Bulk export** — dump an entire channel's history as JSON, HTML, or plain text
## Installation
### From Docker image
The binary is included in the CoreScope Docker image at `/app/corescope-decrypt`:
```bash
docker exec corescope-prod /app/corescope-decrypt --channel "#wardriving" --db /app/data/meshcore.db
```
### From GitHub release
Download the static binary from the [Releases](https://github.com/Kpa-clawbot/CoreScope/releases) page:
```bash
# Linux amd64
curl -LO https://github.com/Kpa-clawbot/CoreScope/releases/latest/download/corescope-decrypt-linux-amd64
chmod +x corescope-decrypt-linux-amd64
./corescope-decrypt-linux-amd64 --help
```
### Build from source
```bash
cd cmd/decrypt
CGO_ENABLED=0 go build -ldflags="-s -w" -o corescope-decrypt .
```
The binary is statically linked — no dependencies, runs on any Linux.
## Usage
```
corescope-decrypt --channel NAME --db PATH [--format FORMAT] [--output FILE]
```
Run `corescope-decrypt --help` for full flag documentation.
### JSON output (default)
Machine-readable, includes all metadata (observers, path hops, raw hex):
```bash
corescope-decrypt --channel "#wardriving" --db meshcore.db
```
```json
[
{
"hash": "a1b2c3...",
"timestamp": "2026-04-12T17:19:09Z",
"sender": "XMD Tag 1",
"message": "@[MapperBot] 37.76985, -122.40525 [0.3w]",
"channel": "#wardriving",
"raw_hex": "150206...",
"path": ["A3", "B0"],
"observers": [
{"name": "Observer1", "snr": 9.5, "rssi": -56, "timestamp": "2026-04-12T17:19:10Z"}
]
}
]
```
### HTML output
Self-contained interactive viewer — search, sortable columns, expandable detail rows:
```bash
corescope-decrypt --channel "#wardriving" --db meshcore.db --format html --output wardriving.html
open wardriving.html
```
No external dependencies. The JSON data is embedded directly in the HTML file.
### IRC / log output
Plain-text, one line per message — ideal for `grep`, `awk`, and piping:
```bash
corescope-decrypt --channel "#wardriving" --db meshcore.db --format irc
```
```
[2026-04-12 17:19:09] <XMD Tag 1> @[MapperBot] 37.76985, -122.40525 [0.3w]
[2026-04-12 17:20:25] <XMD Tag 1> @[MapperBot] 37.78075, -122.39774 [0.3w]
[2026-04-12 17:25:30] <mk 🤠> @[MapperBot] 35.32444, -120.62077
```
```bash
# Find all messages from a specific sender
corescope-decrypt --channel "#wardriving" --db meshcore.db --format irc | grep "KE6QR"
```
## How channel encryption works
MeshCore hashtag channels derive their encryption key from the channel name:
1. **Key derivation**: `AES-128 key = SHA-256("#channelname")[:16]` (first 16 bytes)
2. **Channel hash**: `SHA-256(key)[0]` — 1-byte identifier in the packet header, used for fast filtering
3. **Encryption**: AES-128-ECB
4. **MAC**: HMAC-SHA256 with a 32-byte secret (key + 16 zero bytes), truncated to 2 bytes
5. **Plaintext format**: `timestamp(4 LE) + flags(1) + "sender: message\0"`
See the firmware source at `firmware/src/helpers/BaseChatMesh.cpp` for the canonical implementation.
## Testing against the fixture DB
```bash
cd cmd/decrypt
go test ./...
# Manual test with the real fixture:
go run . --channel "#wardriving" --db ../../test-fixtures/e2e-fixture.db --format irc
```
The shared crypto library also has independent tests:
```bash
cd internal/channel
go test -v ./...
```
## Limitations
- **Hashtag channels only.** Only channels where the key is derived from `SHA-256("#name")` are supported. Custom PSK channels require the raw key (not implemented).
- **No DM decryption.** Direct messages (`TXT_MSG`) use per-peer asymmetric encryption and cannot be decrypted by this tool.
- **Read-only.** The tool opens the database in read-only mode and never modifies it.
- **Timestamps are UTC.** The sender's embedded timestamp is used when available, displayed in UTC.
+22
View File
@@ -0,0 +1,22 @@
module github.com/corescope/decrypt
go 1.22
require (
github.com/meshcore-analyzer/channel v0.0.0
modernc.org/sqlite v1.34.5
)
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
golang.org/x/sys v0.22.0 // indirect
modernc.org/libc v1.55.3 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
)
replace github.com/meshcore-analyzer/channel => ../../internal/channel
+43
View File
@@ -0,0 +1,43 @@
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
golang.org/x/mod v0.16.0 h1:QX4fJ0Rr5cPQCF7O9lh9Se4pmwfwskqZfq5moyldzic=
golang.org/x/mod v0.16.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/tools v0.19.0 h1:tfGCXNR1OsFG+sVdLAitlpjAvD/I6dHDKnYrpEZUHkw=
golang.org/x/tools v0.19.0/go.mod h1:qoJWxmGSIBmAeriMx19ogtrEPrGtDbPK634QFIcLAhc=
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
modernc.org/ccgo/v4 v4.19.2/go.mod h1:ysS3mxiMV38XGRTTcgo0DQTeTmAO4oCmJl1nX9VFI3s=
modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE=
modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ=
modernc.org/gc/v2 v2.4.1 h1:9cNzOqPyMJBvrUipmynX0ZohMhcxPtMccYgGOJdOiBw=
modernc.org/gc/v2 v2.4.1/go.mod h1:wzN5dK1AzVGoH6XOzc3YZ+ey/jPgYHLuVckd62P0GYU=
modernc.org/libc v1.55.3 h1:AzcW1mhlPNrRtjS5sS+eW2ISCgSOLLNyFzRh/V3Qj/U=
modernc.org/libc v1.55.3/go.mod h1:qFXepLhz+JjFThQ4kzwzOjA/y/artDeg+pcYnY+Q83w=
modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
modernc.org/memory v1.8.0 h1:IqGTL6eFMaDZZhEWwcREgeMXYwmW83LYW8cROZYkg+E=
modernc.org/memory v1.8.0/go.mod h1:XPZ936zp5OMKGWPqbD3JShgd/ZoQ7899TUuQqxY+peU=
modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
modernc.org/sortutil v1.2.0 h1:jQiD3PfS2REGJNzNCMMaLSp/wdMNieTbKX920Cqdgqc=
modernc.org/sortutil v1.2.0/go.mod h1:TKU2s7kJMf1AE84OoiGppNHJwvB753OYfNl2WRb++Ss=
modernc.org/sqlite v1.34.5 h1:Bb6SR13/fjp15jt70CL4f18JIN7p7dnMExd+UFnF15g=
modernc.org/sqlite v1.34.5/go.mod h1:YLuNmX9NKs8wRNK2ko1LW1NGYcc9FkBO69JOt1AR9JE=
modernc.org/strutil v1.2.0 h1:agBi9dp1I+eOnxXeiZawM8F4LawKv4NzGWSaLfyeNZA=
modernc.org/strutil v1.2.0/go.mod h1:/mdcBmfOibveCTBxUl5B5l6W+TTH1FXPLHZE6bTosX0=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
+467
View File
@@ -0,0 +1,467 @@
// corescope-decrypt decrypts and exports hashtag channel messages from a CoreScope SQLite database.
//
// Usage:
//
// corescope-decrypt --channel "#wardriving" --db meshcore.db [--format json|html] [--output file]
package main
import (
"database/sql"
"encoding/hex"
"encoding/json"
"flag"
"fmt"
"html"
"log"
"os"
"sort"
"strings"
"time"
"github.com/meshcore-analyzer/channel"
_ "modernc.org/sqlite"
)
// Version info (set via ldflags).
var version = "dev"
// ChannelMessage is a single decrypted channel message with metadata.
type ChannelMessage struct {
Hash string `json:"hash"`
Timestamp string `json:"timestamp"`
Sender string `json:"sender"`
Message string `json:"message"`
Channel string `json:"channel"`
RawHex string `json:"raw_hex"`
Path []string `json:"path"`
Observers []Observer `json:"observers"`
}
// Observer is a single observation of the transmission.
type Observer struct {
Name string `json:"name"`
SNR float64 `json:"snr"`
RSSI float64 `json:"rssi"`
Timestamp string `json:"timestamp"`
}
func main() {
channelName := flag.String("channel", "", "Channel name (e.g. \"#wardriving\")")
dbPath := flag.String("db", "", "Path to CoreScope SQLite database")
format := flag.String("format", "json", "Output format: json, html, irc (or log)")
output := flag.String("output", "", "Output file (default: stdout)")
showVersion := flag.Bool("version", false, "Print version and exit")
flag.Usage = func() {
fmt.Fprintf(os.Stderr, `corescope-decrypt — Decrypt and export MeshCore hashtag channel messages
USAGE
corescope-decrypt --channel NAME --db PATH [--format FORMAT] [--output FILE]
FLAGS
--channel NAME Channel name to decrypt (e.g. "#wardriving", "wardriving")
The "#" prefix is added automatically if missing.
--db PATH Path to a CoreScope SQLite database file (read-only access).
--format FORMAT Output format (default: json):
json — Machine-readable JSON array with full metadata
html — Self-contained HTML viewer with search and sorting
irc — Plain-text IRC-style log, one line per message
log — Alias for irc
--output FILE Write output to FILE instead of stdout.
--version Print version and exit.
EXAMPLES
# Export #wardriving messages as JSON
corescope-decrypt --channel "#wardriving" --db /app/data/meshcore.db
# Generate an interactive HTML viewer
corescope-decrypt --channel wardriving --db meshcore.db --format html --output wardriving.html
# Greppable IRC log
corescope-decrypt --channel "#MeshCore" --db meshcore.db --format irc --output meshcore.log
grep "KE6QR" meshcore.log
# From the Docker container
docker exec corescope-prod /app/corescope-decrypt --channel "#wardriving" --db /app/data/meshcore.db
RETROACTIVE DECRYPTION
MeshCore hashtag channels use symmetric encryption — the key is derived from the
channel name. The CoreScope ingestor stores ALL GRP_TXT packets in the database,
even those it cannot decrypt at ingest time. This tool lets you retroactively
decrypt messages for any channel whose name you know, even if the ingestor was
never configured with that channel's key.
This means you can recover historical messages by simply knowing the channel name.
LIMITATIONS
- Only hashtag channels (shared-secret, name-derived key) are supported.
- Direct messages (TXT_MSG) use per-peer encryption and cannot be decrypted.
- Custom PSK channels (non-hashtag) require the raw key, not a channel name.
`)
}
flag.Parse()
if *showVersion {
fmt.Println("corescope-decrypt", version)
os.Exit(0)
}
if *channelName == "" || *dbPath == "" {
flag.Usage()
os.Exit(1)
}
// Normalize channel name
ch := *channelName
if !strings.HasPrefix(ch, "#") {
ch = "#" + ch
}
key := channel.DeriveKey(ch)
chHash := channel.ChannelHash(key)
db, err := sql.Open("sqlite", *dbPath+"?mode=ro")
if err != nil {
log.Fatalf("Failed to open database: %v", err)
}
defer db.Close()
// Query all GRP_TXT packets
rows, err := db.Query(`SELECT id, hash, raw_hex, first_seen FROM transmissions WHERE payload_type = 5`)
if err != nil {
log.Fatalf("Query failed: %v", err)
}
defer rows.Close()
var messages []ChannelMessage
decrypted, total := 0, 0
for rows.Next() {
var id int
var txHash, rawHex, firstSeen string
if err := rows.Scan(&id, &txHash, &rawHex, &firstSeen); err != nil {
log.Printf("Scan error: %v", err)
continue
}
total++
payload, err := extractGRPPayload(rawHex)
if err != nil {
continue
}
if len(payload) < 3 {
continue
}
// Check channel hash byte
if payload[0] != chHash {
continue
}
mac := payload[1:3]
ciphertext := payload[3:]
if len(ciphertext) < 5 || len(ciphertext)%16 != 0 {
// Pad ciphertext to block boundary for decryption attempt
if len(ciphertext) < 16 {
continue
}
// Truncate to block boundary
ciphertext = ciphertext[:len(ciphertext)/16*16]
}
plaintext, ok := channel.Decrypt(key, mac, ciphertext)
if !ok {
continue
}
ts, sender, msg, err := channel.ParsePlaintext(plaintext)
if err != nil {
continue
}
decrypted++
// Convert MeshCore timestamp
timestamp := time.Unix(int64(ts), 0).UTC().Format(time.RFC3339)
// Get path from decoded_json
path := getPathFromDB(db, id)
// Get observers
observers := getObservers(db, id)
messages = append(messages, ChannelMessage{
Hash: txHash,
Timestamp: timestamp,
Sender: sender,
Message: msg,
Channel: ch,
RawHex: rawHex,
Path: path,
Observers: observers,
})
}
// Sort by timestamp
sort.Slice(messages, func(i, j int) bool {
return messages[i].Timestamp < messages[j].Timestamp
})
log.Printf("Scanned %d GRP_TXT packets, decrypted %d for channel %s", total, decrypted, ch)
// Generate output
var out []byte
switch *format {
case "json":
out, err = json.MarshalIndent(messages, "", " ")
if err != nil {
log.Fatalf("JSON marshal: %v", err)
}
out = append(out, '\n')
case "html":
out = renderHTML(messages, ch)
case "irc", "log":
out = renderIRC(messages)
default:
log.Fatalf("Unknown format: %s (use json, html, irc, or log)", *format)
}
if *output != "" {
if err := os.WriteFile(*output, out, 0644); err != nil {
log.Fatalf("Write file: %v", err)
}
log.Printf("Written to %s", *output)
} else {
os.Stdout.Write(out)
}
}
// extractGRPPayload parses a raw hex packet and returns the GRP_TXT payload bytes.
func extractGRPPayload(rawHex string) ([]byte, error) {
buf, err := hex.DecodeString(strings.TrimSpace(rawHex))
if err != nil || len(buf) < 2 {
return nil, fmt.Errorf("invalid hex")
}
// Header byte
header := buf[0]
payloadType := int((header >> 2) & 0x0F)
if payloadType != 5 { // GRP_TXT
return nil, fmt.Errorf("not GRP_TXT")
}
routeType := int(header & 0x03)
offset := 1
// Transport codes (2 codes × 2 bytes) come BEFORE path for transport routes
if routeType == 0 || routeType == 3 {
offset += 4
}
// Path byte
if offset >= len(buf) {
return nil, fmt.Errorf("too short for path")
}
pathByte := buf[offset]
offset++
hashSize := int(pathByte>>6) + 1
hashCount := int(pathByte & 0x3F)
offset += hashSize * hashCount
if offset >= len(buf) {
return nil, fmt.Errorf("too short for payload")
}
return buf[offset:], nil
}
func getPathFromDB(db *sql.DB, txID int) []string {
var decodedJSON sql.NullString
err := db.QueryRow(`SELECT decoded_json FROM transmissions WHERE id = ?`, txID).Scan(&decodedJSON)
if err != nil || !decodedJSON.Valid {
return nil
}
var decoded struct {
Path struct {
Hops []string `json:"hops"`
} `json:"path"`
}
if json.Unmarshal([]byte(decodedJSON.String), &decoded) == nil {
return decoded.Path.Hops
}
return nil
}
func getObservers(db *sql.DB, txID int) []Observer {
rows, err := db.Query(`
SELECT o.name, obs.snr, obs.rssi, obs.timestamp
FROM observations obs
LEFT JOIN observers o ON o.id = CAST(obs.observer_idx AS TEXT)
WHERE obs.transmission_id = ?
ORDER BY obs.timestamp
`, txID)
if err != nil {
return nil
}
defer rows.Close()
var observers []Observer
for rows.Next() {
var name sql.NullString
var snr, rssi sql.NullFloat64
var ts int64
if err := rows.Scan(&name, &snr, &rssi, &ts); err != nil {
continue
}
obs := Observer{
Timestamp: time.Unix(ts, 0).UTC().Format(time.RFC3339),
}
if name.Valid {
obs.Name = name.String
}
if snr.Valid {
obs.SNR = snr.Float64
}
if rssi.Valid {
obs.RSSI = rssi.Float64
}
observers = append(observers, obs)
}
return observers
}
func renderIRC(messages []ChannelMessage) []byte {
var b strings.Builder
for _, m := range messages {
sender := m.Sender
if sender == "" {
sender = "???"
}
// Parse RFC3339 timestamp into a compact format
t, err := time.Parse(time.RFC3339, m.Timestamp)
if err != nil {
b.WriteString(fmt.Sprintf("[%s] <%s> %s\n", m.Timestamp, sender, m.Message))
continue
}
b.WriteString(fmt.Sprintf("[%s] <%s> %s\n", t.Format("2006-01-02 15:04:05"), sender, m.Message))
}
return []byte(b.String())
}
func renderHTML(messages []ChannelMessage, channelName string) []byte {
jsonData, _ := json.Marshal(messages)
var b strings.Builder
b.WriteString(`<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>CoreScope Channel Export — ` + html.EscapeString(channelName) + `</title>
<style>
*{box-sizing:border-box;margin:0;padding:0}
body{font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,sans-serif;background:#0d1117;color:#c9d1d9;padding:20px}
h1{color:#58a6ff;margin-bottom:16px;font-size:1.5em}
.stats{color:#8b949e;margin-bottom:16px;font-size:0.9em}
input[type=text]{width:100%;max-width:500px;padding:8px 12px;background:#161b22;border:1px solid #30363d;border-radius:6px;color:#c9d1d9;font-size:14px;margin-bottom:16px}
input[type=text]:focus{outline:none;border-color:#58a6ff}
table{width:100%;border-collapse:collapse;font-size:14px}
th{background:#161b22;color:#8b949e;text-align:left;padding:8px 12px;border-bottom:2px solid #30363d;cursor:pointer;user-select:none;white-space:nowrap}
th:hover{color:#58a6ff}
th.sorted-asc::after{content:" ▲"}
th.sorted-desc::after{content:" ▼"}
td{padding:8px 12px;border-bottom:1px solid #21262d;vertical-align:top}
tr:hover{background:#161b22}
tr.expanded{background:#161b22}
.detail-row td{padding:12px 24px;background:#0d1117;border-bottom:1px solid #21262d}
.detail-row pre{background:#161b22;padding:12px;border-radius:6px;overflow-x:auto;font-size:12px;color:#8b949e}
.detail-row .label{color:#58a6ff;font-weight:600;margin-top:8px;display:block}
.observer-tag{display:inline-block;background:#1f6feb22;color:#58a6ff;padding:2px 8px;border-radius:4px;margin:2px;font-size:12px}
.no-results{color:#8b949e;text-align:center;padding:40px;font-size:16px}
.sender{color:#d2a8ff;font-weight:600}
.timestamp{color:#8b949e;font-family:monospace;font-size:12px}
</style>
</head>
<body>
<h1>` + html.EscapeString(channelName) + ` — Channel Messages</h1>
<div class="stats" id="stats"></div>
<input type="text" id="search" placeholder="Search messages..." autocomplete="off">
<table>
<thead>
<tr>
<th data-col="timestamp">Timestamp</th>
<th data-col="sender">Sender</th>
<th data-col="message">Message</th>
<th data-col="observers">Observers</th>
</tr>
</thead>
<tbody id="tbody"></tbody>
</table>
<div class="no-results" id="no-results" style="display:none">No matching messages</div>
<script>
var DATA=` + string(jsonData) + `;
var sortCol="timestamp",sortAsc=true,expandedHash=null;
function init(){
document.getElementById("stats").textContent=DATA.length+" messages";
document.getElementById("search").addEventListener("input",render);
document.querySelectorAll("th[data-col]").forEach(function(th){
th.addEventListener("click",function(){
var col=th.dataset.col;
if(sortCol===col)sortAsc=!sortAsc;
else{sortCol=col;sortAsc=true}
render();
});
});
render();
}
function render(){
var q=document.getElementById("search").value.toLowerCase();
var filtered=DATA.filter(function(m){
if(!q)return true;
return(m.message||"").toLowerCase().indexOf(q)>=0||(m.sender||"").toLowerCase().indexOf(q)>=0;
});
filtered.sort(function(a,b){
var va=a[sortCol]||"",vb=b[sortCol]||"";
if(sortCol==="observers"){va=a.observers?a.observers.length:0;vb=b.observers?b.observers.length:0}
if(va<vb)return sortAsc?-1:1;
if(va>vb)return sortAsc?1:-1;
return 0;
});
document.querySelectorAll("th[data-col]").forEach(function(th){
th.className=th.dataset.col===sortCol?(sortAsc?"sorted-asc":"sorted-desc"):"";
});
var tb=document.getElementById("tbody");
tb.innerHTML="";
document.getElementById("no-results").style.display=filtered.length?"none":"block";
filtered.forEach(function(m){
var tr=document.createElement("tr");
tr.innerHTML='<td class="timestamp">'+esc(m.timestamp)+'</td><td class="sender">'+esc(m.sender||"—")+'</td><td>'+esc(m.message)+'</td><td>'+
(m.observers?m.observers.map(function(o){return'<span class="observer-tag">'+esc(o.name||"?")+" SNR:"+o.snr.toFixed(1)+'</span>'}).join(""):"—")+'</td>';
tr.style.cursor="pointer";
tr.addEventListener("click",function(){
expandedHash=expandedHash===m.hash?null:m.hash;
render();
});
tb.appendChild(tr);
if(expandedHash===m.hash){
tr.className="expanded";
var dr=document.createElement("tr");
dr.className="detail-row";
dr.innerHTML='<td colspan="4"><span class="label">Hash</span><pre>'+esc(m.hash)+'</pre>'+
'<span class="label">Raw Hex</span><pre>'+esc(m.raw_hex)+'</pre>'+
(m.path&&m.path.length?'<span class="label">Path</span><pre>'+esc(m.path.join(" → "))+'</pre>':'')+
'<span class="label">Observers</span><pre>'+esc(JSON.stringify(m.observers,null,2))+'</pre></td>';
tb.appendChild(dr);
}
});
}
function esc(s){var d=document.createElement("div");d.textContent=s;return d.innerHTML}
init();
</script>
</body>
</html>`)
return []byte(b.String())
}
+129
View File
@@ -0,0 +1,129 @@
package main
import (
"encoding/hex"
"encoding/json"
"os"
"strings"
"testing"
"github.com/meshcore-analyzer/channel"
)
func TestExtractGRPPayload(t *testing.T) {
// Build a minimal GRP_TXT packet: header(1) + path(1) + payload
// header: route=FLOOD(1), payload=GRP_TXT(5), version=0 → (5<<2)|1 = 0x15
// path: 0 hops, hash_size=1 → 0x00
payload := []byte{0x81, 0x12, 0x34} // channel_hash + mac + data
pkt := append([]byte{0x15, 0x00}, payload...)
rawHex := hex.EncodeToString(pkt)
result, err := extractGRPPayload(rawHex)
if err != nil {
t.Fatal(err)
}
if len(result) != 3 || result[0] != 0x81 {
t.Fatalf("payload mismatch: %x", result)
}
}
func TestExtractGRPPayloadTransport(t *testing.T) {
// Transport flood: route=0, 4 bytes transport codes BEFORE path byte
// header: (5<<2)|0 = 0x14
payload := []byte{0xAA, 0xBB, 0xCC}
// header + 4 transport bytes + path(0 hops) + payload
pkt := append([]byte{0x14, 0xFF, 0xFF, 0xFF, 0xFF, 0x00}, payload...)
rawHex := hex.EncodeToString(pkt)
result, err := extractGRPPayload(rawHex)
if err != nil {
t.Fatal(err)
}
if result[0] != 0xAA {
t.Fatalf("expected AA, got %02X", result[0])
}
}
func TestExtractGRPPayloadNotGRP(t *testing.T) {
// payload type = ADVERT (4): (4<<2)|1 = 0x11
rawHex := hex.EncodeToString([]byte{0x11, 0x00, 0x01, 0x02})
_, err := extractGRPPayload(rawHex)
if err == nil {
t.Fatal("expected error for non-GRP_TXT")
}
}
func TestKeyDerivationConsistency(t *testing.T) {
// Verify key derivation matches what the ingestor expects
key := channel.DeriveKey("#wardriving")
if len(key) != 16 {
t.Fatalf("key len %d", len(key))
}
ch := channel.ChannelHash(key)
if ch != 0x81 {
// We know from fixture data that #wardriving has channelHashHex "81"
t.Fatalf("channel hash %02X, expected 81", ch)
}
}
func TestRenderIRC(t *testing.T) {
msgs := []ChannelMessage{
{Timestamp: "2026-04-12T03:45:12Z", Sender: "NodeA", Message: "Hello"},
{Timestamp: "2026-04-12T03:46:01Z", Sender: "", Message: "No sender"},
}
out := string(renderIRC(msgs))
if !strings.Contains(out, "[2026-04-12 03:45:12] <NodeA> Hello") {
t.Fatalf("IRC output missing expected line: %s", out)
}
if !strings.Contains(out, "<???> No sender") {
t.Fatalf("IRC output should use ??? for empty sender: %s", out)
}
}
func TestRenderHTMLValid(t *testing.T) {
msgs := []ChannelMessage{
{Hash: "abc", Timestamp: "2026-04-12T00:00:00Z", Sender: "X", Message: "test", Channel: "#test"},
}
out := string(renderHTML(msgs, "#test"))
if !strings.Contains(out, "<!DOCTYPE html>") {
t.Fatal("not valid HTML")
}
if !strings.Contains(out, "#test") {
t.Fatal("channel name missing")
}
if !strings.Contains(out, "</html>") {
t.Fatal("HTML not closed")
}
}
func TestJSONOutputParseable(t *testing.T) {
msgs := []ChannelMessage{
{Hash: "abc", Timestamp: "2026-04-12T00:00:00Z", Sender: "X", Message: "hi", Channel: "#test"},
}
data, err := json.MarshalIndent(msgs, "", " ")
if err != nil {
t.Fatal(err)
}
var parsed []ChannelMessage
if err := json.Unmarshal(data, &parsed); err != nil {
t.Fatalf("JSON not parseable: %v", err)
}
if len(parsed) != 1 || parsed[0].Sender != "X" {
t.Fatalf("parsed mismatch: %+v", parsed)
}
}
// Integration test against fixture DB (skipped if DB not found)
func TestFixtureDecrypt(t *testing.T) {
dbPath := "../../test-fixtures/e2e-fixture.db"
if _, err := os.Stat(dbPath); os.IsNotExist(err) {
t.Skip("fixture DB not found")
}
// We know the fixture has #wardriving messages with channelHash 0x81
key := channel.DeriveKey("#wardriving")
ch := channel.ChannelHash(key)
if ch != 0x81 {
t.Fatalf("unexpected channel hash: %02X", ch)
}
}
+68 -8
View File
@@ -2,7 +2,9 @@ package main
import (
"encoding/json"
"errors"
"fmt"
"log"
"os"
"strings"
@@ -36,7 +38,9 @@ type Config struct {
ChannelKeys map[string]string `json:"channelKeys,omitempty"`
HashChannels []string `json:"hashChannels,omitempty"`
Retention *RetentionConfig `json:"retention,omitempty"`
GeoFilter *GeoFilterConfig `json:"geo_filter,omitempty"`
Metrics *MetricsConfig `json:"metrics,omitempty"`
GeoFilter *GeoFilterConfig `json:"geo_filter,omitempty"`
ValidateSignatures *bool `json:"validateSignatures,omitempty"`
}
// GeoFilterConfig is an alias for the shared geofilter.Config type.
@@ -44,7 +48,38 @@ type GeoFilterConfig = geofilter.Config
// RetentionConfig controls how long stale nodes are kept before being moved to inactive_nodes.
type RetentionConfig struct {
NodeDays int `json:"nodeDays"`
NodeDays int `json:"nodeDays"`
ObserverDays int `json:"observerDays"`
MetricsDays int `json:"metricsDays"`
}
// MetricsConfig controls observer metrics collection.
type MetricsConfig struct {
SampleIntervalSec int `json:"sampleIntervalSec"`
}
// ShouldValidateSignatures returns true (default) unless explicitly disabled.
func (c *Config) ShouldValidateSignatures() bool {
if c.ValidateSignatures != nil {
return *c.ValidateSignatures
}
return true
}
// MetricsSampleInterval returns the configured sample interval or 300s default.
func (c *Config) MetricsSampleInterval() int {
if c.Metrics != nil && c.Metrics.SampleIntervalSec > 0 {
return c.Metrics.SampleIntervalSec
}
return 300
}
// MetricsRetentionDays returns configured metrics retention or 30 days default.
func (c *Config) MetricsRetentionDays() int {
if c.Retention != nil && c.Retention.MetricsDays > 0 {
return c.Retention.MetricsDays
}
return 30
}
// NodeDaysOrDefault returns the configured retention.nodeDays or 7 if not set.
@@ -55,16 +90,31 @@ func (c *Config) NodeDaysOrDefault() int {
return 7
}
// ObserverDaysOrDefault returns the configured retention.observerDays or 14 if not set.
// A value of -1 means observers are never removed.
func (c *Config) ObserverDaysOrDefault() int {
if c.Retention != nil && c.Retention.ObserverDays != 0 {
return c.Retention.ObserverDays
}
return 14
}
// LoadConfig reads configuration from a JSON file, with env var overrides.
// If the config file does not exist, sensible defaults are used (zero-config startup).
func LoadConfig(path string) (*Config, error) {
var cfg Config
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("reading config %s: %w", path, err)
}
var cfg Config
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("parsing config %s: %w", path, err)
if !errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("reading config %s: %w", path, err)
}
// Config file doesn't exist — use defaults (zero-config mode)
log.Printf("config file %s not found, using sensible defaults", path)
} else {
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("parsing config %s: %w", path, err)
}
}
// Env var overrides
@@ -98,6 +148,16 @@ func LoadConfig(path string) (*Config, error) {
}}
}
// Default MQTT source: connect to localhost broker when no sources configured
if len(cfg.MQTTSources) == 0 {
cfg.MQTTSources = []MQTTSource{{
Name: "local",
Broker: "mqtt://localhost:1883",
Topics: []string{"meshcore/#"},
}}
log.Printf("no MQTT sources configured, defaulting to mqtt://localhost:1883")
}
return &cfg, nil
}
+21 -5
View File
@@ -32,9 +32,25 @@ func TestLoadConfigValidJSON(t *testing.T) {
}
func TestLoadConfigMissingFile(t *testing.T) {
_, err := LoadConfig("/nonexistent/path/config.json")
if err == nil {
t.Error("expected error for missing file")
t.Setenv("DB_PATH", "")
t.Setenv("MQTT_BROKER", "")
cfg, err := LoadConfig("/nonexistent/path/config.json")
if err != nil {
t.Fatalf("missing config should not error (zero-config mode), got: %v", err)
}
if cfg.DBPath != "data/meshcore.db" {
t.Errorf("dbPath=%s, want data/meshcore.db", cfg.DBPath)
}
// Should default to localhost MQTT
if len(cfg.MQTTSources) != 1 {
t.Fatalf("mqttSources len=%d, want 1", len(cfg.MQTTSources))
}
if cfg.MQTTSources[0].Broker != "mqtt://localhost:1883" {
t.Errorf("default broker=%s, want mqtt://localhost:1883", cfg.MQTTSources[0].Broker)
}
if cfg.MQTTSources[0].Name != "local" {
t.Errorf("default source name=%s, want local", cfg.MQTTSources[0].Name)
}
}
@@ -196,8 +212,8 @@ func TestLoadConfigLegacyMQTTEmptyBroker(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if len(cfg.MQTTSources) != 0 {
t.Errorf("mqttSources should be empty when legacy broker is empty, got %d", len(cfg.MQTTSources))
if len(cfg.MQTTSources) != 1 || cfg.MQTTSources[0].Name != "local" {
t.Errorf("mqttSources should default to local broker when legacy broker is empty, got %v", cfg.MQTTSources)
}
}
File diff suppressed because it is too large Load Diff
+360 -20
View File
@@ -11,6 +11,7 @@ import (
"sync/atomic"
"time"
"github.com/meshcore-analyzer/packetpath"
_ "modernc.org/sqlite"
)
@@ -22,6 +23,7 @@ type DBStats struct {
NodeUpserts atomic.Int64
ObserverUpserts atomic.Int64
WriteErrors atomic.Int64
SignatureDrops atomic.Int64
}
// Store wraps the SQLite database for packet ingestion.
@@ -39,11 +41,19 @@ type Store struct {
stmtGetObserverRowid *sql.Stmt
stmtUpdateObserverLastSeen *sql.Stmt
stmtUpdateNodeTelemetry *sql.Stmt
stmtUpsertMetrics *sql.Stmt
sampleIntervalSec int
}
// OpenStore opens or creates a SQLite DB at the given path, applying the
// v3 schema that is compatible with the Node.js server.
func OpenStore(dbPath string) (*Store, error) {
return OpenStoreWithInterval(dbPath, 300)
}
// OpenStoreWithInterval opens or creates a SQLite DB with a configurable sample interval.
func OpenStoreWithInterval(dbPath string, sampleIntervalSec int) (*Store, error) {
dir := filepath.Dir(dbPath)
if err := os.MkdirAll(dir, 0o755); err != nil {
return nil, fmt.Errorf("creating data dir: %w", err)
@@ -66,7 +76,7 @@ func OpenStore(dbPath string) (*Store, error) {
return nil, fmt.Errorf("applying schema: %w", err)
}
s := &Store{db: db}
s := &Store{db: db, sampleIntervalSec: sampleIntervalSec}
if err := s.prepareStatements(); err != nil {
return nil, fmt.Errorf("preparing statements: %w", err)
}
@@ -102,7 +112,9 @@ func applySchema(db *sql.DB) error {
radio TEXT,
battery_mv INTEGER,
uptime_secs INTEGER,
noise_floor REAL
noise_floor REAL,
inactive INTEGER DEFAULT 0,
last_packet_at TEXT DEFAULT NULL
);
CREATE INDEX IF NOT EXISTS idx_nodes_last_seen ON nodes(last_seen);
@@ -179,7 +191,7 @@ func applySchema(db *sql.DB) error {
db.Exec(`DROP VIEW IF EXISTS packets_v`)
_, vErr := db.Exec(`
CREATE VIEW packets_v AS
SELECT o.id, t.raw_hex,
SELECT o.id, COALESCE(o.raw_hex, t.raw_hex) AS raw_hex,
datetime(o.timestamp, 'unixepoch') AS timestamp,
obs.id AS observer_id, obs.name AS observer_name,
o.direction, o.snr, o.rssi, o.score, t.hash, t.route_type,
@@ -187,7 +199,7 @@ func applySchema(db *sql.DB) error {
t.created_at
FROM observations o
JOIN transmissions t ON t.id = o.transmission_id
LEFT JOIN observers obs ON obs.rowid = o.observer_idx
LEFT JOIN observers obs ON obs.rowid = o.observer_idx AND (obs.inactive IS NULL OR obs.inactive = 0)
`)
if vErr != nil {
return fmt.Errorf("packets_v view: %w", vErr)
@@ -292,6 +304,143 @@ func applySchema(db *sql.DB) error {
log.Println("[migration] observations timestamp index created")
}
// observer_metrics table for RF health dashboard
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'observer_metrics_v1'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Creating observer_metrics table...")
_, err := db.Exec(`
CREATE TABLE IF NOT EXISTS observer_metrics (
observer_id TEXT NOT NULL,
timestamp TEXT NOT NULL,
noise_floor REAL,
tx_air_secs INTEGER,
rx_air_secs INTEGER,
recv_errors INTEGER,
battery_mv INTEGER,
PRIMARY KEY (observer_id, timestamp)
)
`)
if err != nil {
return fmt.Errorf("observer_metrics schema: %w", err)
}
db.Exec(`INSERT INTO _migrations (name) VALUES ('observer_metrics_v1')`)
log.Println("[migration] observer_metrics table created")
}
// Migration: add timestamp index for cross-observer time-range queries
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'observer_metrics_ts_idx'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Creating observer_metrics timestamp index...")
_, err := db.Exec(`CREATE INDEX IF NOT EXISTS idx_observer_metrics_timestamp ON observer_metrics(timestamp)`)
if err != nil {
return fmt.Errorf("observer_metrics timestamp index: %w", err)
}
db.Exec(`INSERT INTO _migrations (name) VALUES ('observer_metrics_ts_idx')`)
log.Println("[migration] observer_metrics timestamp index created")
}
// Migration: add inactive column to observers for soft-delete retention
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'observers_inactive_v1'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Adding inactive column to observers...")
_, err := db.Exec(`ALTER TABLE observers ADD COLUMN inactive INTEGER DEFAULT 0`)
if err != nil {
// Column may already exist (e.g. fresh install with schema above)
log.Printf("[migration] observers.inactive: %v (may already exist)", err)
}
db.Exec(`INSERT INTO _migrations (name) VALUES ('observers_inactive_v1')`)
log.Println("[migration] observers.inactive column added")
}
// Migration: add packets_sent and packets_recv columns to observer_metrics
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'observer_metrics_packets_v1'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Adding packets_sent/packets_recv columns to observer_metrics...")
db.Exec(`ALTER TABLE observer_metrics ADD COLUMN packets_sent INTEGER`)
db.Exec(`ALTER TABLE observer_metrics ADD COLUMN packets_recv INTEGER`)
db.Exec(`INSERT INTO _migrations (name) VALUES ('observer_metrics_packets_v1')`)
log.Println("[migration] packets_sent/packets_recv columns added")
}
// Migration: add channel_hash column for fast channel queries (#762)
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'channel_hash_v1'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Adding channel_hash column to transmissions...")
db.Exec(`ALTER TABLE transmissions ADD COLUMN channel_hash TEXT DEFAULT NULL`)
db.Exec(`CREATE INDEX IF NOT EXISTS idx_tx_channel_hash ON transmissions(channel_hash) WHERE payload_type = 5`)
// Backfill: extract channel name for decrypted (CHAN) packets
res, err := db.Exec(`UPDATE transmissions SET channel_hash = json_extract(decoded_json, '$.channel') WHERE payload_type = 5 AND channel_hash IS NULL AND json_extract(decoded_json, '$.type') = 'CHAN'`)
if err == nil {
n, _ := res.RowsAffected()
log.Printf("[migration] Backfilled channel_hash for %d CHAN packets", n)
}
// Backfill: extract channelHashHex for encrypted (GRP_TXT) packets, prefixed with 'enc_'
res, err = db.Exec(`UPDATE transmissions SET channel_hash = 'enc_' || json_extract(decoded_json, '$.channelHashHex') WHERE payload_type = 5 AND channel_hash IS NULL AND json_extract(decoded_json, '$.type') = 'GRP_TXT'`)
if err == nil {
n, _ := res.RowsAffected()
log.Printf("[migration] Backfilled channel_hash for %d GRP_TXT packets", n)
}
db.Exec(`INSERT INTO _migrations (name) VALUES ('channel_hash_v1')`)
log.Println("[migration] channel_hash column added and backfilled")
}
// Migration: dropped_packets table for signature validation failures (#793)
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'dropped_packets_v1'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Creating dropped_packets table...")
_, err := db.Exec(`
CREATE TABLE IF NOT EXISTS dropped_packets (
id INTEGER PRIMARY KEY AUTOINCREMENT,
hash TEXT,
raw_hex TEXT,
reason TEXT NOT NULL,
observer_id TEXT,
observer_name TEXT,
node_pubkey TEXT,
node_name TEXT,
dropped_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_dropped_observer ON dropped_packets(observer_id);
CREATE INDEX IF NOT EXISTS idx_dropped_node ON dropped_packets(node_pubkey);
`)
if err != nil {
return fmt.Errorf("dropped_packets schema: %w", err)
}
db.Exec(`INSERT INTO _migrations (name) VALUES ('dropped_packets_v1')`)
log.Println("[migration] dropped_packets table created")
}
// Migration: add raw_hex column to observations (#881)
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'observations_raw_hex_v1'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Adding raw_hex column to observations...")
db.Exec(`ALTER TABLE observations ADD COLUMN raw_hex TEXT`)
db.Exec(`INSERT INTO _migrations (name) VALUES ('observations_raw_hex_v1')`)
log.Println("[migration] observations.raw_hex column added")
}
// Migration: add last_packet_at column to observers (#last-packet-at)
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'observers_last_packet_at_v1'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Adding last_packet_at column to observers...")
_, alterErr := db.Exec(`ALTER TABLE observers ADD COLUMN last_packet_at TEXT DEFAULT NULL`)
if alterErr != nil && !strings.Contains(alterErr.Error(), "duplicate column") {
return fmt.Errorf("observers last_packet_at ALTER: %w", alterErr)
}
// Backfill: set last_packet_at = last_seen only for observers that actually have
// observation rows (packet_count alone is unreliable — UpsertObserver sets it to 1
// on INSERT even for status-only observers).
res, err := db.Exec(`UPDATE observers SET last_packet_at = last_seen
WHERE last_packet_at IS NULL
AND rowid IN (SELECT DISTINCT observer_idx FROM observations WHERE observer_idx IS NOT NULL)`)
if err == nil {
n, _ := res.RowsAffected()
log.Printf("[migration] Backfilled last_packet_at for %d observers with packets", n)
}
db.Exec(`INSERT INTO _migrations (name) VALUES ('observers_last_packet_at_v1')`)
log.Println("[migration] observers.last_packet_at column added")
}
return nil
}
@@ -304,8 +453,8 @@ func (s *Store) prepareStatements() error {
}
s.stmtInsertTransmission, err = s.db.Prepare(`
INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, payload_version, decoded_json)
VALUES (?, ?, ?, ?, ?, ?, ?)
INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, payload_version, decoded_json, channel_hash)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`)
if err != nil {
return err
@@ -317,8 +466,13 @@ func (s *Store) prepareStatements() error {
}
s.stmtInsertObservation, err = s.db.Prepare(`
INSERT OR IGNORE INTO observations (transmission_id, observer_idx, direction, snr, rssi, score, path_json, timestamp)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
INSERT INTO observations (transmission_id, observer_idx, direction, snr, rssi, score, path_json, timestamp, raw_hex)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(transmission_id, observer_idx, COALESCE(path_json, '')) DO UPDATE SET
snr = COALESCE(excluded.snr, snr),
rssi = COALESCE(excluded.rssi, rssi),
score = COALESCE(excluded.score, score),
raw_hex = COALESCE(excluded.raw_hex, raw_hex)
`)
if err != nil {
return err
@@ -370,7 +524,7 @@ func (s *Store) prepareStatements() error {
return err
}
s.stmtUpdateObserverLastSeen, err = s.db.Prepare("UPDATE observers SET last_seen = ? WHERE rowid = ?")
s.stmtUpdateObserverLastSeen, err = s.db.Prepare("UPDATE observers SET last_seen = ?, last_packet_at = ? WHERE rowid = ?")
if err != nil {
return err
}
@@ -385,6 +539,14 @@ func (s *Store) prepareStatements() error {
return err
}
s.stmtUpsertMetrics, err = s.db.Prepare(`
INSERT OR REPLACE INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors, battery_mv, packets_sent, packets_recv)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
`)
if err != nil {
return err
}
return nil
}
@@ -420,7 +582,7 @@ func (s *Store) InsertTransmission(data *PacketData) (bool, error) {
result, err := s.stmtInsertTransmission.Exec(
data.RawHex, hash, now,
data.RouteType, data.PayloadType, data.PayloadVersion,
data.DecodedJSON,
data.DecodedJSON, nilIfEmpty(data.ChannelHash),
)
if err != nil {
s.Stats.WriteErrors.Add(1)
@@ -441,9 +603,9 @@ func (s *Store) InsertTransmission(data *PacketData) (bool, error) {
err := s.stmtGetObserverRowid.QueryRow(data.ObserverID).Scan(&rowid)
if err == nil {
observerIdx = &rowid
// Update observer last_seen on every packet to prevent
// Update observer last_seen and last_packet_at on every packet to prevent
// low-traffic observers from appearing offline (#463)
_, _ = s.stmtUpdateObserverLastSeen.Exec(now, rowid)
_, _ = s.stmtUpdateObserverLastSeen.Exec(now, now, rowid)
}
}
@@ -456,7 +618,7 @@ func (s *Store) InsertTransmission(data *PacketData) (bool, error) {
_, err = s.stmtInsertObservation.Exec(
txID, observerIdx, data.Direction,
data.SNR, data.RSSI, data.Score,
data.PathJSON, epochTs,
data.PathJSON, epochTs, nilIfEmpty(data.RawHex),
)
if err != nil {
s.Stats.WriteErrors.Add(1)
@@ -517,6 +679,11 @@ type ObserverMeta struct {
BatteryMv *int // millivolts, always integer
UptimeSecs *int64 // seconds, always integer
NoiseFloor *float64 // dBm, may have decimals
TxAirSecs *int // cumulative TX seconds since boot
RxAirSecs *int // cumulative RX seconds since boot
RecvErrors *int // cumulative CRC/decode failures since boot
PacketsSent *int // cumulative packets sent since boot
PacketsRecv *int // cumulative packets received since boot
}
// UpsertObserver inserts or updates an observer with optional hardware metadata.
@@ -556,10 +723,13 @@ func (s *Store) UpsertObserver(id, name, iata string, meta *ObserverMeta) error
)
if err != nil {
s.Stats.WriteErrors.Add(1)
} else {
s.Stats.ObserverUpserts.Add(1)
return err
}
return err
s.Stats.ObserverUpserts.Add(1)
// Reactivate if this observer was previously marked inactive
s.db.Exec(`UPDATE observers SET inactive = 0 WHERE id = ? AND inactive = 1`, id)
return nil
}
// Close checkpoints the WAL and closes the database.
@@ -568,6 +738,79 @@ func (s *Store) Close() error {
return s.db.Close()
}
// RoundToInterval rounds a time to the nearest sample interval boundary.
func RoundToInterval(t time.Time, intervalSec int) time.Time {
if intervalSec <= 0 {
intervalSec = 300
}
epoch := t.Unix()
half := int64(intervalSec) / 2
rounded := ((epoch + half) / int64(intervalSec)) * int64(intervalSec)
return time.Unix(rounded, 0).UTC()
}
// MetricsData holds the fields to insert into observer_metrics.
type MetricsData struct {
ObserverID string
NoiseFloor *float64
TxAirSecs *int
RxAirSecs *int
RecvErrors *int
BatteryMv *int
PacketsSent *int
PacketsRecv *int
}
// InsertMetrics inserts a metrics sample for an observer using ingestor wall clock.
func (s *Store) InsertMetrics(data *MetricsData) error {
ts := RoundToInterval(time.Now().UTC(), s.sampleIntervalSec)
tsStr := ts.Format(time.RFC3339)
var nf, txAir, rxAir, recvErr, batt, pktSent, pktRecv interface{}
if data.NoiseFloor != nil {
nf = *data.NoiseFloor
}
if data.TxAirSecs != nil {
txAir = *data.TxAirSecs
}
if data.RxAirSecs != nil {
rxAir = *data.RxAirSecs
}
if data.RecvErrors != nil {
recvErr = *data.RecvErrors
}
if data.BatteryMv != nil {
batt = *data.BatteryMv
}
if data.PacketsSent != nil {
pktSent = *data.PacketsSent
}
if data.PacketsRecv != nil {
pktRecv = *data.PacketsRecv
}
_, err := s.stmtUpsertMetrics.Exec(data.ObserverID, tsStr, nf, txAir, rxAir, recvErr, batt, pktSent, pktRecv)
if err != nil {
s.Stats.WriteErrors.Add(1)
return fmt.Errorf("insert metrics: %w", err)
}
return nil
}
// PruneOldMetrics deletes observer_metrics rows older than retentionDays.
func (s *Store) PruneOldMetrics(retentionDays int) (int64, error) {
cutoff := time.Now().UTC().AddDate(0, 0, -retentionDays).Format(time.RFC3339)
result, err := s.db.Exec(`DELETE FROM observer_metrics WHERE timestamp < ?`, cutoff)
if err != nil {
return 0, fmt.Errorf("prune metrics: %w", err)
}
n, _ := result.RowsAffected()
if n > 0 {
log.Printf("[metrics] Pruned %d rows older than %d days", n, retentionDays)
}
return n, nil
}
// Checkpoint forces a WAL checkpoint to release the WAL lock file,
// preventing lock contention with a new process starting up.
func (s *Store) Checkpoint() {
@@ -580,13 +823,14 @@ func (s *Store) Checkpoint() {
// LogStats logs current operational metrics.
func (s *Store) LogStats() {
log.Printf("[stats] tx_inserted=%d tx_dupes=%d obs_inserted=%d node_upserts=%d observer_upserts=%d write_errors=%d",
log.Printf("[stats] tx_inserted=%d tx_dupes=%d obs_inserted=%d node_upserts=%d observer_upserts=%d write_errors=%d sig_drops=%d",
s.Stats.TransmissionsInserted.Load(),
s.Stats.DuplicateTransmissions.Load(),
s.Stats.ObservationsInserted.Load(),
s.Stats.NodeUpserts.Load(),
s.Stats.ObserverUpserts.Load(),
s.Stats.WriteErrors.Load(),
s.Stats.SignatureDrops.Load(),
)
}
@@ -618,6 +862,71 @@ func (s *Store) MoveStaleNodes(nodeDays int) (int64, error) {
return moved, nil
}
// RemoveStaleObservers marks observers that have not actively sent data in observerDays
// as inactive (soft-delete). This preserves JOIN integrity for observations.observer_idx
// and observer_metrics.observer_id — historical data still references the correct observer.
// An observer must actively send data to stay listed — being seen by another node does not count.
// observerDays <= -1 means never remove (keep forever).
func (s *Store) RemoveStaleObservers(observerDays int) (int64, error) {
if observerDays <= -1 {
return 0, nil // keep forever
}
cutoff := time.Now().UTC().AddDate(0, 0, -observerDays).Format(time.RFC3339)
result, err := s.db.Exec(`UPDATE observers SET inactive = 1 WHERE last_seen < ? AND (inactive IS NULL OR inactive = 0)`, cutoff)
if err != nil {
return 0, fmt.Errorf("mark stale observers inactive: %w", err)
}
removed, _ := result.RowsAffected()
if removed > 0 {
// Clean up orphaned metrics for now-inactive observers
s.db.Exec(`DELETE FROM observer_metrics WHERE observer_id IN (SELECT id FROM observers WHERE inactive = 1)`)
log.Printf("Marked %d observer(s) as inactive (not seen in %d days)", removed, observerDays)
}
return removed, nil
}
// DroppedPacket holds data for a packet rejected during ingest.
type DroppedPacket struct {
Hash string
RawHex string
Reason string
ObserverID string
ObserverName string
NodePubKey string
NodeName string
}
// InsertDroppedPacket records a rejected packet in the dropped_packets table.
func (s *Store) InsertDroppedPacket(dp *DroppedPacket) error {
_, err := s.db.Exec(
`INSERT INTO dropped_packets (hash, raw_hex, reason, observer_id, observer_name, node_pubkey, node_name) VALUES (?, ?, ?, ?, ?, ?, ?)`,
dp.Hash, dp.RawHex, dp.Reason, dp.ObserverID, dp.ObserverName, dp.NodePubKey, dp.NodeName,
)
if err != nil {
s.Stats.WriteErrors.Add(1)
return fmt.Errorf("insert dropped packet: %w", err)
}
s.Stats.SignatureDrops.Add(1)
return nil
}
// PruneDroppedPackets removes dropped_packets older than retentionDays.
func (s *Store) PruneDroppedPackets(retentionDays int) (int64, error) {
if retentionDays <= 0 {
return 0, nil
}
cutoff := time.Now().UTC().AddDate(0, 0, -retentionDays).Format(time.RFC3339)
result, err := s.db.Exec(`DELETE FROM dropped_packets WHERE dropped_at < ?`, cutoff)
if err != nil {
return 0, fmt.Errorf("prune dropped packets: %w", err)
}
n, _ := result.RowsAffected()
if n > 0 {
log.Printf("Pruned %d dropped packet(s) older than %d days", n, retentionDays)
}
return n, nil
}
// PacketData holds the data needed to insert a packet into the DB.
type PacketData struct {
RawHex string
@@ -634,6 +943,15 @@ type PacketData struct {
PayloadVersion int
PathJSON string
DecodedJSON string
ChannelHash string // grouping key for channel queries (#762)
}
// nilIfEmpty returns nil for empty strings (for nullable DB columns).
func nilIfEmpty(s string) interface{} {
if s == "" {
return nil
}
return s
}
// MQTTPacketMessage is the JSON payload from an MQTT raw packet message.
@@ -647,15 +965,26 @@ type MQTTPacketMessage struct {
}
// BuildPacketData constructs a PacketData from a decoded packet and MQTT message.
// path_json is derived directly from raw_hex header bytes (not decoded.Path.Hops)
// to guarantee the stored path always matches the raw bytes. This matters for
// TRACE packets where decoded.Path.Hops is overwritten with payload hops (#886).
func BuildPacketData(msg *MQTTPacketMessage, decoded *DecodedPacket, observerID, region string) *PacketData {
now := time.Now().UTC().Format(time.RFC3339)
pathJSON := "[]"
if len(decoded.Path.Hops) > 0 {
b, _ := json.Marshal(decoded.Path.Hops)
// For TRACE packets, path_json must be the payload-decoded route hops
// (decoded.Path.Hops), NOT the raw_hex header bytes which are SNR values.
// For all other packet types, derive path from raw_hex (#886).
if !packetpath.PathBytesAreHops(byte(decoded.Header.PayloadType)) {
if len(decoded.Path.Hops) > 0 {
b, _ := json.Marshal(decoded.Path.Hops)
pathJSON = string(b)
}
} else if hops, err := packetpath.DecodePathFromRawHex(msg.Raw); err == nil && len(hops) > 0 {
b, _ := json.Marshal(hops)
pathJSON = string(b)
}
return &PacketData{
pd := &PacketData{
RawHex: msg.Raw,
Timestamp: now,
ObserverID: observerID,
@@ -671,4 +1000,15 @@ func BuildPacketData(msg *MQTTPacketMessage, decoded *DecodedPacket, observerID,
PathJSON: pathJSON,
DecodedJSON: PayloadJSON(&decoded.Payload),
}
// Populate channel_hash for fast channel queries (#762)
if decoded.Header.PayloadType == PayloadGRP_TXT {
if decoded.Payload.Type == "CHAN" && decoded.Payload.Channel != "" {
pd.ChannelHash = decoded.Payload.Channel
} else if decoded.Payload.Type == "GRP_TXT" && decoded.Payload.ChannelHashHex != "" {
pd.ChannelHash = "enc_" + decoded.Payload.ChannelHashHex
}
}
return pd
}
+481 -6
View File
@@ -2,6 +2,7 @@ package main
import (
"database/sql"
"encoding/json"
"fmt"
"os"
"path/filepath"
@@ -10,6 +11,8 @@ import (
"sync/atomic"
"testing"
"time"
"github.com/meshcore-analyzer/packetpath"
)
func tempDBPath(t *testing.T) string {
@@ -566,6 +569,61 @@ func TestInsertTransmissionUpdatesObserverLastSeen(t *testing.T) {
}
}
func TestLastPacketAtUpdatedOnPacketOnly(t *testing.T) {
s, err := OpenStore(tempDBPath(t))
if err != nil {
t.Fatal(err)
}
defer s.Close()
// Insert observer via status path — last_packet_at should be NULL
if err := s.UpsertObserver("obs1", "Observer1", "SJC", nil); err != nil {
t.Fatal(err)
}
var lastPacketAt sql.NullString
s.db.QueryRow("SELECT last_packet_at FROM observers WHERE id = ?", "obs1").Scan(&lastPacketAt)
if lastPacketAt.Valid {
t.Fatalf("expected last_packet_at to be NULL after UpsertObserver, got %s", lastPacketAt.String)
}
// Insert a packet from this observer — last_packet_at should be set
data := &PacketData{
RawHex: "0A00D69F",
Timestamp: "2026-04-24T12:00:00Z",
ObserverID: "obs1",
Hash: "lastpackettest123456",
RouteType: 2,
PayloadType: 2,
PathJSON: "[]",
DecodedJSON: `{"type":"TXT_MSG"}`,
}
if _, err := s.InsertTransmission(data); err != nil {
t.Fatal(err)
}
s.db.QueryRow("SELECT last_packet_at FROM observers WHERE id = ?", "obs1").Scan(&lastPacketAt)
if !lastPacketAt.Valid {
t.Fatal("expected last_packet_at to be non-NULL after InsertTransmission")
}
// InsertTransmission uses `now = data.Timestamp || time.Now()`, so last_packet_at
// should match the packet's Timestamp when provided (same source-of-truth as last_seen).
if lastPacketAt.String != "2026-04-24T12:00:00Z" {
t.Errorf("expected last_packet_at=2026-04-24T12:00:00Z, got %s", lastPacketAt.String)
}
// UpsertObserver again (status path) — last_packet_at should NOT change
if err := s.UpsertObserver("obs1", "Observer1", "SJC", nil); err != nil {
t.Fatal(err)
}
var lastPacketAtAfterStatus sql.NullString
s.db.QueryRow("SELECT last_packet_at FROM observers WHERE id = ?", "obs1").Scan(&lastPacketAtAfterStatus)
if !lastPacketAtAfterStatus.Valid || lastPacketAtAfterStatus.String != lastPacketAt.String {
t.Errorf("UpsertObserver should not change last_packet_at; expected %s, got %v", lastPacketAt.String, lastPacketAtAfterStatus)
}
}
func TestEndToEndIngest(t *testing.T) {
s, err := OpenStore(tempDBPath(t))
if err != nil {
@@ -576,7 +634,7 @@ func TestEndToEndIngest(t *testing.T) {
// Simulate full pipeline: decode + insert
rawHex := "120046D62DE27D4C5194D7821FC5A34A45565DCC2537B300B9AB6275255CEFB65D840CE5C169C94C9AED39E8BCB6CB6EB0335497A198B33A1A610CD3B03D8DCFC160900E5244280323EE0B44CACAB8F02B5B38B91CFA18BD067B0B5E63E94CFC85F758A8530B9240933402E0E6B8F84D5252322D52"
decoded, err := DecodePacket(rawHex, nil)
decoded, err := DecodePacket(rawHex, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -764,7 +822,7 @@ func TestInsertTransmissionNilSNRRSSI(t *testing.T) {
func TestBuildPacketData(t *testing.T) {
rawHex := "0A00D69FD7A5A7475DB07337749AE61FA53A4788E976"
decoded, err := DecodePacket(rawHex, nil)
decoded, err := DecodePacket(rawHex, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -818,7 +876,7 @@ func TestBuildPacketData(t *testing.T) {
func TestBuildPacketDataWithHops(t *testing.T) {
// A packet with actual hops in the path
raw := "0505AABBCCDDEE" + strings.Repeat("00", 10)
decoded, err := DecodePacket(raw, nil)
decoded, err := DecodePacket(raw, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -834,7 +892,7 @@ func TestBuildPacketDataWithHops(t *testing.T) {
}
func TestBuildPacketDataNilSNRRSSI(t *testing.T) {
decoded, _ := DecodePacket("0A00"+strings.Repeat("00", 10), nil)
decoded, _ := DecodePacket("0A00"+strings.Repeat("00", 10), nil, false)
msg := &MQTTPacketMessage{Raw: "0A00" + strings.Repeat("00", 10)}
pkt := BuildPacketData(msg, decoded, "", "")
@@ -1624,7 +1682,7 @@ func TestObsTimestampIndexMigration(t *testing.T) {
func TestBuildPacketDataScoreAndDirection(t *testing.T) {
rawHex := "0A00D69FD7A5A7475DB07337749AE61FA53A4788E976"
decoded, err := DecodePacket(rawHex, nil)
decoded, err := DecodePacket(rawHex, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -1647,7 +1705,7 @@ func TestBuildPacketDataScoreAndDirection(t *testing.T) {
}
func TestBuildPacketDataNilScoreDirection(t *testing.T) {
decoded, _ := DecodePacket("0A00"+strings.Repeat("00", 10), nil)
decoded, _ := DecodePacket("0A00"+strings.Repeat("00", 10), nil, false)
msg := &MQTTPacketMessage{Raw: "0A00" + strings.Repeat("00", 10)}
pkt := BuildPacketData(msg, decoded, "", "")
@@ -1703,3 +1761,420 @@ func TestInsertTransmissionWithScoreAndDirection(t *testing.T) {
}
func ptrFloat(f float64) *float64 { return &f }
func ptrInt(i int) *int { return &i }
func TestRoundToInterval(t *testing.T) {
tests := []struct {
input time.Time
interval int
want time.Time
}{
{time.Date(2026, 4, 5, 10, 2, 0, 0, time.UTC), 300, time.Date(2026, 4, 5, 10, 0, 0, 0, time.UTC)},
{time.Date(2026, 4, 5, 10, 3, 0, 0, time.UTC), 300, time.Date(2026, 4, 5, 10, 5, 0, 0, time.UTC)},
{time.Date(2026, 4, 5, 10, 2, 30, 0, time.UTC), 300, time.Date(2026, 4, 5, 10, 5, 0, 0, time.UTC)},
{time.Date(2026, 4, 5, 10, 5, 0, 0, time.UTC), 300, time.Date(2026, 4, 5, 10, 5, 0, 0, time.UTC)},
{time.Date(2026, 4, 5, 10, 7, 29, 0, time.UTC), 300, time.Date(2026, 4, 5, 10, 5, 0, 0, time.UTC)},
}
for _, tc := range tests {
got := RoundToInterval(tc.input, tc.interval)
if !got.Equal(tc.want) {
t.Errorf("RoundToInterval(%v, %d) = %v, want %v", tc.input, tc.interval, got, tc.want)
}
}
}
func TestInsertMetrics(t *testing.T) {
store, err := OpenStore(tempDBPath(t))
if err != nil {
t.Fatal(err)
}
defer store.Close()
nf := -112.5
txAir := 100
rxAir := 500
recvErr := 3
batt := 3720
data := &MetricsData{
ObserverID: "obs1",
NoiseFloor: &nf,
TxAirSecs: &txAir,
RxAirSecs: &rxAir,
RecvErrors: &recvErr,
BatteryMv: &batt,
}
if err := store.InsertMetrics(data); err != nil {
t.Fatalf("InsertMetrics: %v", err)
}
// Verify insertion
var count int
store.db.QueryRow("SELECT COUNT(*) FROM observer_metrics WHERE observer_id = 'obs1'").Scan(&count)
if count != 1 {
t.Errorf("expected 1 row, got %d", count)
}
// Verify values
var gotNF float64
var gotTx, gotRx, gotErr, gotBatt int
store.db.QueryRow("SELECT noise_floor, tx_air_secs, rx_air_secs, recv_errors, battery_mv FROM observer_metrics WHERE observer_id = 'obs1'").Scan(&gotNF, &gotTx, &gotRx, &gotErr, &gotBatt)
if gotNF != -112.5 {
t.Errorf("noise_floor = %v, want -112.5", gotNF)
}
if gotTx != 100 {
t.Errorf("tx_air_secs = %d, want 100", gotTx)
}
}
func TestInsertMetricsIdempotent(t *testing.T) {
store, err := OpenStoreWithInterval(tempDBPath(t), 300)
if err != nil {
t.Fatal(err)
}
defer store.Close()
nf := -110.0
data := &MetricsData{ObserverID: "obs1", NoiseFloor: &nf}
// Insert twice — should result in 1 row (INSERT OR REPLACE)
store.InsertMetrics(data)
nf2 := -108.0
data.NoiseFloor = &nf2
store.InsertMetrics(data)
var count int
store.db.QueryRow("SELECT COUNT(*) FROM observer_metrics WHERE observer_id = 'obs1'").Scan(&count)
if count != 1 {
t.Errorf("expected 1 row (idempotent), got %d", count)
}
// Verify the value was replaced
var gotNF float64
store.db.QueryRow("SELECT noise_floor FROM observer_metrics WHERE observer_id = 'obs1'").Scan(&gotNF)
if gotNF != -108.0 {
t.Errorf("noise_floor = %v, want -108.0 (replaced)", gotNF)
}
}
func TestInsertMetricsNullFields(t *testing.T) {
store, err := OpenStore(tempDBPath(t))
if err != nil {
t.Fatal(err)
}
defer store.Close()
nf := -115.0
data := &MetricsData{
ObserverID: "obs1",
NoiseFloor: &nf,
// All other fields nil
}
if err := store.InsertMetrics(data); err != nil {
t.Fatalf("InsertMetrics with nulls: %v", err)
}
var gotNF sql.NullFloat64
var gotTx sql.NullInt64
store.db.QueryRow("SELECT noise_floor, tx_air_secs FROM observer_metrics WHERE observer_id = 'obs1'").Scan(&gotNF, &gotTx)
if !gotNF.Valid || gotNF.Float64 != -115.0 {
t.Errorf("noise_floor = %v, want -115.0", gotNF)
}
if gotTx.Valid {
t.Errorf("tx_air_secs should be NULL, got %v", gotTx.Int64)
}
}
func TestPruneOldMetrics(t *testing.T) {
store, err := OpenStore(tempDBPath(t))
if err != nil {
t.Fatal(err)
}
defer store.Close()
// Insert old and new metrics directly
oldTs := time.Now().UTC().AddDate(0, 0, -40).Format(time.RFC3339)
newTs := time.Now().UTC().Format(time.RFC3339)
store.db.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor) VALUES (?, ?, ?)", "obs1", oldTs, -110.0)
store.db.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor) VALUES (?, ?, ?)", "obs1", newTs, -112.0)
n, err := store.PruneOldMetrics(30)
if err != nil {
t.Fatalf("PruneOldMetrics: %v", err)
}
if n != 1 {
t.Errorf("pruned %d rows, want 1", n)
}
var count int
store.db.QueryRow("SELECT COUNT(*) FROM observer_metrics").Scan(&count)
if count != 1 {
t.Errorf("expected 1 row remaining, got %d", count)
}
}
func TestExtractObserverMetaNewFields(t *testing.T) {
msg := map[string]interface{}{
"model": "L1",
"stats": map[string]interface{}{
"noise_floor": -112.5,
"battery_mv": 3720.0,
"uptime_secs": 86400.0,
"tx_air_secs": 100.0,
"rx_air_secs": 500.0,
"recv_errors": 3.0,
},
}
meta := extractObserverMeta(msg)
if meta == nil {
t.Fatal("expected non-nil meta")
}
if meta.TxAirSecs == nil || *meta.TxAirSecs != 100 {
t.Errorf("TxAirSecs = %v, want 100", meta.TxAirSecs)
}
if meta.RxAirSecs == nil || *meta.RxAirSecs != 500 {
t.Errorf("RxAirSecs = %v, want 500", meta.RxAirSecs)
}
if meta.RecvErrors == nil || *meta.RecvErrors != 3 {
t.Errorf("RecvErrors = %v, want 3", meta.RecvErrors)
}
}
// TestInsertObservationSNRFillIn verifies that when the same observation is
// received twice — first without SNR, then with SNR — the SNR is filled in
// rather than silently discarded. The unique dedup index is
// (transmission_id, observer_idx, COALESCE(path_json, '')); observer_idx must
// be non-NULL for the conflict to fire (SQLite treats NULL != NULL).
func TestInsertObservationSNRFillIn(t *testing.T) {
s, err := OpenStore(tempDBPath(t))
if err != nil {
t.Fatal(err)
}
defer s.Close()
// Register the observer so observer_idx is non-NULL (required for dedup).
if err := s.UpsertObserver("pymc-obs1", "PyMC Observer", "SJC", nil); err != nil {
t.Fatal(err)
}
// First arrival: same observer, no SNR/RSSI (e.g. broker replay without RF fields).
data1 := &PacketData{
RawHex: "0A00D69FD7A5A7475DB07337749AE61FA53A4788E976",
Timestamp: "2026-04-20T00:00:00Z",
Hash: "snrfillin0001hash",
RouteType: 1,
ObserverID: "pymc-obs1",
SNR: nil,
RSSI: nil,
}
if _, err := s.InsertTransmission(data1); err != nil {
t.Fatal(err)
}
var snr1, rssi1 *float64
s.db.QueryRow("SELECT snr, rssi FROM observations LIMIT 1").Scan(&snr1, &rssi1)
if snr1 != nil || rssi1 != nil {
t.Fatalf("precondition: first insert should have nil SNR/RSSI, got snr=%v rssi=%v", snr1, rssi1)
}
// Second arrival: same packet, same observer, now WITH SNR/RSSI.
snr := 10.5
rssi := -88.0
data2 := &PacketData{
RawHex: data1.RawHex,
Timestamp: data1.Timestamp,
Hash: data1.Hash,
RouteType: data1.RouteType,
ObserverID: "pymc-obs1",
SNR: &snr,
RSSI: &rssi,
}
if _, err := s.InsertTransmission(data2); err != nil {
t.Fatal(err)
}
var snr2, rssi2 *float64
s.db.QueryRow("SELECT snr, rssi FROM observations LIMIT 1").Scan(&snr2, &rssi2)
if snr2 == nil || *snr2 != snr {
t.Errorf("SNR not filled in by second arrival: got %v, want %v", snr2, snr)
}
if rssi2 == nil || *rssi2 != rssi {
t.Errorf("RSSI not filled in by second arrival: got %v, want %v", rssi2, rssi)
}
// Third arrival: same packet again, SNR absent — must NOT overwrite existing SNR.
data3 := &PacketData{
RawHex: data1.RawHex,
Timestamp: data1.Timestamp,
Hash: data1.Hash,
RouteType: data1.RouteType,
ObserverID: "pymc-obs1",
SNR: nil,
RSSI: nil,
}
if _, err := s.InsertTransmission(data3); err != nil {
t.Fatal(err)
}
var snr3, rssi3 *float64
s.db.QueryRow("SELECT snr, rssi FROM observations LIMIT 1").Scan(&snr3, &rssi3)
if snr3 == nil || *snr3 != snr {
t.Errorf("SNR overwritten by null arrival: got %v, want %v", snr3, snr)
}
if rssi3 == nil || *rssi3 != rssi {
t.Errorf("RSSI overwritten by null arrival: got %v, want %v", rssi3, rssi)
}
}
// TestPerObservationRawHex verifies that two MQTT packets for the same hash
// from different observers store distinct raw_hex per observation (#881).
func TestPerObservationRawHex(t *testing.T) {
store, err := OpenStore(tempDBPath(t))
if err != nil {
t.Fatal(err)
}
defer store.Close()
// Register two observers
store.UpsertObserver("obs-A", "Observer A", "", nil)
store.UpsertObserver("obs-B", "Observer B", "", nil)
hash := "abc123def456"
rawA := "c0ffee01"
rawB := "c0ffee0201aa"
dir := "RX"
// First observation from observer A
pdA := &PacketData{
RawHex: rawA,
Hash: hash,
Timestamp: "2026-04-21T10:00:00Z",
ObserverID: "obs-A",
Direction: &dir,
PathJSON: "[]",
}
isNew, err := store.InsertTransmission(pdA)
if err != nil {
t.Fatalf("insert A: %v", err)
}
if !isNew {
t.Fatal("expected new transmission")
}
// Second observation from observer B (same hash, different raw bytes)
pdB := &PacketData{
RawHex: rawB,
Hash: hash,
Timestamp: "2026-04-21T10:00:01Z",
ObserverID: "obs-B",
Direction: &dir,
PathJSON: `["aabb"]`,
}
isNew2, err := store.InsertTransmission(pdB)
if err != nil {
t.Fatalf("insert B: %v", err)
}
if isNew2 {
t.Fatal("expected duplicate transmission")
}
// Query observations and verify per-observation raw_hex
rows, err := store.db.Query(`
SELECT o.raw_hex, obs.id
FROM observations o
LEFT JOIN observers obs ON obs.rowid = o.observer_idx
ORDER BY o.id ASC
`)
if err != nil {
t.Fatalf("query: %v", err)
}
defer rows.Close()
type obsResult struct {
rawHex string
observerID string
}
var results []obsResult
for rows.Next() {
var rh, oid sql.NullString
if err := rows.Scan(&rh, &oid); err != nil {
t.Fatal(err)
}
results = append(results, obsResult{
rawHex: rh.String,
observerID: oid.String,
})
}
if len(results) != 2 {
t.Fatalf("expected 2 observations, got %d", len(results))
}
if results[0].rawHex != rawA {
t.Errorf("obs A raw_hex: got %q, want %q", results[0].rawHex, rawA)
}
if results[1].rawHex != rawB {
t.Errorf("obs B raw_hex: got %q, want %q", results[1].rawHex, rawB)
}
if results[0].rawHex == results[1].rawHex {
t.Error("both observations have same raw_hex — should differ")
}
}
// TestBuildPacketData_TraceUsesPayloadHops verifies that TRACE packets use
// payload-decoded route hops in path_json (NOT the raw_hex header SNR bytes).
// Issue #886 / #887.
func TestBuildPacketData_TraceUsesPayloadHops(t *testing.T) {
// TRACE packet: header path has SNR bytes [30,2D,0D,23], but decoded.Path.Hops
// is overwritten to payload hops [67,33,D6,33,67].
rawHex := "2604302D0D2359FEE7B100000000006733D63367"
decoded, err := DecodePacket(rawHex, nil, false)
if err != nil {
t.Fatal(err)
}
// decoded.Path.Hops should be the TRACE-replaced hops (payload hops)
if len(decoded.Path.Hops) != 5 {
t.Fatalf("expected 5 decoded hops, got %d", len(decoded.Path.Hops))
}
msg := &MQTTPacketMessage{Raw: rawHex}
pd := BuildPacketData(msg, decoded, "test-obs", "TST")
// For TRACE: path_json MUST be the payload-decoded route hops, NOT the SNR bytes
expectedPathJSON := `["67","33","D6","33","67"]`
if pd.PathJSON != expectedPathJSON {
t.Errorf("path_json = %s, want %s (TRACE must use payload hops)", pd.PathJSON, expectedPathJSON)
}
// Verify that DecodePathFromRawHex returns the SNR bytes (header path) which differ
headerHops, herr := packetpath.DecodePathFromRawHex(rawHex)
if herr != nil {
t.Fatal(herr)
}
headerJSON, _ := json.Marshal(headerHops)
if string(headerJSON) == expectedPathJSON {
t.Error("header path (SNR) should differ from payload hops for TRACE")
}
}
// TestBuildPacketData_NonTracePathJSON verifies non-TRACE packets also derive path from raw_hex.
func TestBuildPacketData_NonTracePathJSON(t *testing.T) {
// A simple ADVERT packet (payload type 0) with 2 hops, hash_size 1
// Header 0x09 = FLOOD(1), ADVERT(2), version 0
// Path byte 0x02 = hash_size 1, hash_count 2
// Path bytes: AA BB
rawHex := "0902AABB" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
decoded, err := DecodePacket(rawHex, nil, false)
if err != nil {
t.Fatal(err)
}
msg := &MQTTPacketMessage{Raw: rawHex}
pd := BuildPacketData(msg, decoded, "obs1", "TST")
expectedPathJSON := `["AA","BB"]`
if pd.PathJSON != expectedPathJSON {
t.Errorf("path_json = %s, want %s", pd.PathJSON, expectedPathJSON)
}
}
+76 -18
View File
@@ -11,6 +11,9 @@ import (
"math"
"strings"
"unicode/utf8"
"github.com/meshcore-analyzer/packetpath"
"github.com/meshcore-analyzer/sigvalidate"
)
// Route type constants (header bits 1-0)
@@ -78,9 +81,10 @@ type TransportCodes struct {
// Path holds decoded path/hop information.
type Path struct {
HashSize int `json:"hashSize"`
HashCount int `json:"hashCount"`
Hops []string `json:"hops"`
HashSize int `json:"hashSize"`
HashCount int `json:"hashCount"`
Hops []string `json:"hops"`
HopsCompleted *int `json:"hopsCompleted,omitempty"`
}
// AdvertFlags holds decoded advert flag bits.
@@ -109,6 +113,7 @@ type Payload struct {
Timestamp uint32 `json:"timestamp,omitempty"`
TimestampISO string `json:"timestampISO,omitempty"`
Signature string `json:"signature,omitempty"`
SignatureValid *bool `json:"signatureValid,omitempty"`
Flags *AdvertFlags `json:"flags,omitempty"`
Lat *float64 `json:"lat,omitempty"`
Lon *float64 `json:"lon,omitempty"`
@@ -140,6 +145,7 @@ type DecodedPacket struct {
Path Path `json:"path"`
Payload Payload `json:"payload"`
Raw string `json:"raw"`
Anomaly string `json:"anomaly,omitempty"`
}
func decodeHeader(b byte) Header {
@@ -187,8 +193,9 @@ func decodePath(pathByte byte, buf []byte, offset int) (Path, int) {
}, totalBytes
}
// isTransportRoute delegates to packetpath.IsTransportRoute.
func isTransportRoute(routeType int) bool {
return routeType == RouteTransportFlood || routeType == RouteTransportDirect
return packetpath.IsTransportRoute(routeType)
}
func decodeEncryptedPayload(typeName string, buf []byte) Payload {
@@ -215,7 +222,7 @@ func decodeAck(buf []byte) Payload {
}
}
func decodeAdvert(buf []byte) Payload {
func decodeAdvert(buf []byte, validateSignatures bool) Payload {
if len(buf) < 100 {
return Payload{Type: "ADVERT", Error: "too short for advert", RawHex: hex.EncodeToString(buf)}
}
@@ -233,6 +240,16 @@ func decodeAdvert(buf []byte) Payload {
Signature: signature,
}
if validateSignatures {
valid, err := sigvalidate.ValidateAdvert(buf[0:32], buf[36:100], timestamp, appdata)
if err != nil {
f := false
p.SignatureValid = &f
} else {
p.SignatureValid = &valid
}
}
if len(appdata) > 0 {
flags := appdata[0]
advType := int(flags & 0x0F)
@@ -506,7 +523,7 @@ func decodeTrace(buf []byte) Payload {
return p
}
func decodePayload(payloadType int, buf []byte, channelKeys map[string]string) Payload {
func decodePayload(payloadType int, buf []byte, channelKeys map[string]string, validateSignatures bool) Payload {
switch payloadType {
case PayloadREQ:
return decodeEncryptedPayload("REQ", buf)
@@ -517,7 +534,7 @@ func decodePayload(payloadType int, buf []byte, channelKeys map[string]string) P
case PayloadACK:
return decodeAck(buf)
case PayloadADVERT:
return decodeAdvert(buf)
return decodeAdvert(buf, validateSignatures)
case PayloadGRP_TXT:
return decodeGrpTxt(buf, channelKeys)
case PayloadANON_REQ:
@@ -532,7 +549,7 @@ func decodePayload(payloadType int, buf []byte, channelKeys map[string]string) P
}
// DecodePacket decodes a hex-encoded MeshCore packet.
func DecodePacket(hexString string, channelKeys map[string]string) (*DecodedPacket, error) {
func DecodePacket(hexString string, channelKeys map[string]string, validateSignatures bool) (*DecodedPacket, error) {
hexString = strings.ReplaceAll(hexString, " ", "")
hexString = strings.ReplaceAll(hexString, "\n", "")
hexString = strings.ReplaceAll(hexString, "\r", "")
@@ -570,35 +587,65 @@ func DecodePacket(hexString string, channelKeys map[string]string) (*DecodedPack
offset += bytesConsumed
payloadBuf := buf[offset:]
payload := decodePayload(header.PayloadType, payloadBuf, channelKeys)
payload := decodePayload(header.PayloadType, payloadBuf, channelKeys, validateSignatures)
// TRACE packets store hop IDs in the payload (buf[9:]) rather than the header
// path field. The header path byte still encodes hashSize in bits 6-7, which
// we use to split the payload path data into individual hop prefixes.
// path field. Firmware always sends TRACE as DIRECT (route_type 2 or 3);
// FLOOD-routed TRACEs are anomalous but handled gracefully (parsed, but
// flagged). The TRACE flags byte (payload offset 8) encodes path_sz in
// bits 0-1 as a power-of-two exponent: hash_bytes = 1 << path_sz.
// NOT the header path byte's hash_size bits. The header path contains SNR
// bytes — one per hop that actually forwarded.
// We expose hopsCompleted (count of SNR bytes) so consumers can distinguish
// how far the trace got vs the full intended route.
var anomaly string
if header.PayloadType == PayloadTRACE && payload.PathData != "" {
// Flag anomalous routing — firmware only sends TRACE as DIRECT
if header.RouteType != RouteDirect && header.RouteType != RouteTransportDirect {
anomaly = "TRACE packet with non-DIRECT routing (expected DIRECT or TRANSPORT_DIRECT)"
}
// The header path hops count represents SNR entries = completed hops
hopsCompleted := path.HashCount
pathBytes, err := hex.DecodeString(payload.PathData)
if err == nil && path.HashSize > 0 {
hops := make([]string, 0, len(pathBytes)/path.HashSize)
for i := 0; i+path.HashSize <= len(pathBytes); i += path.HashSize {
hops = append(hops, strings.ToUpper(hex.EncodeToString(pathBytes[i:i+path.HashSize])))
if err == nil && payload.TraceFlags != nil {
// path_sz from flags byte is a power-of-two exponent per firmware:
// hash_bytes = 1 << (flags & 0x03)
pathSz := 1 << (*payload.TraceFlags & 0x03)
hops := make([]string, 0, len(pathBytes)/pathSz)
for i := 0; i+pathSz <= len(pathBytes); i += pathSz {
hops = append(hops, strings.ToUpper(hex.EncodeToString(pathBytes[i:i+pathSz])))
}
path.Hops = hops
path.HashCount = len(hops)
path.HashSize = pathSz
path.HopsCompleted = &hopsCompleted
}
}
// Zero-hop direct packets have hash_count=0 (lower 6 bits of pathByte),
// which makes the generic formula yield a bogus hashSize. Reset to 0
// (unknown) so API consumers get correct data. We mask with 0x3F to check
// only hash_count, matching the JS frontend approach — the upper hash_size
// bits are meaningless when there are no hops. Skip TRACE packets — they
// use hashSize to parse hops from the payload above.
if (header.RouteType == RouteDirect || header.RouteType == RouteTransportDirect) && pathByte&0x3F == 0 && header.PayloadType != PayloadTRACE {
path.HashSize = 0
}
return &DecodedPacket{
Header: header,
TransportCodes: tc,
Path: path,
Payload: payload,
Raw: strings.ToUpper(hexString),
Anomaly: anomaly,
}, nil
}
// ComputeContentHash computes the SHA-256-based content hash (first 16 hex chars).
// It hashes the header byte + payload (skipping path bytes) to produce a
// path-independent identifier for the same transmission.
// It hashes the payload-type nibble + payload (skipping path bytes) to produce a
// route-independent identifier for the same logical packet. For TRACE packets,
// path_len is included in the hash to match firmware behavior.
func ComputeContentHash(rawHex string) string {
buf, err := hex.DecodeString(rawHex)
if err != nil || len(buf) < 2 {
@@ -634,7 +681,18 @@ func ComputeContentHash(rawHex string) string {
}
payload := buf[payloadStart:]
toHash := append([]byte{headerByte}, payload...)
// Hash payload-type byte only (bits 2-5 of header), not the full header.
// Firmware: SHA256(payload_type + [path_len for TRACE] + payload)
// Using the full header caused different hashes for the same logical packet
// when route type or version bits differed. See issue #786.
payloadType := (headerByte >> 2) & 0x0F
toHash := []byte{payloadType}
if int(payloadType) == PayloadTRACE {
// Firmware uses uint16_t path_len (2 bytes, little-endian)
toHash = append(toHash, pathByte, 0x00)
}
toHash = append(toHash, payload...)
h := sha256.Sum256(toHash)
return hex.EncodeToString(h[:])[:16]
+421 -37
View File
@@ -2,6 +2,7 @@ package main
import (
"crypto/aes"
"crypto/ed25519"
"crypto/hmac"
"crypto/sha256"
"encoding/binary"
@@ -9,6 +10,9 @@ import (
"math"
"strings"
"testing"
"github.com/meshcore-analyzer/packetpath"
"github.com/meshcore-analyzer/sigvalidate"
)
func TestDecodeHeaderRoutTypes(t *testing.T) {
@@ -55,7 +59,7 @@ func TestDecodeHeaderPayloadTypes(t *testing.T) {
func TestDecodePathZeroHops(t *testing.T) {
// 0x00: 0 hops, 1-byte hashes
pkt, err := DecodePacket("0500"+strings.Repeat("00", 10), nil)
pkt, err := DecodePacket("0500"+strings.Repeat("00", 10), nil, false)
if err != nil {
t.Fatal(err)
}
@@ -72,7 +76,7 @@ func TestDecodePathZeroHops(t *testing.T) {
func TestDecodePath1ByteHashes(t *testing.T) {
// 0x05: 5 hops, 1-byte hashes → 5 path bytes
pkt, err := DecodePacket("0505"+"AABBCCDDEE"+strings.Repeat("00", 10), nil)
pkt, err := DecodePacket("0505"+"AABBCCDDEE"+strings.Repeat("00", 10), nil, false)
if err != nil {
t.Fatal(err)
}
@@ -95,7 +99,7 @@ func TestDecodePath1ByteHashes(t *testing.T) {
func TestDecodePath2ByteHashes(t *testing.T) {
// 0x45: 5 hops, 2-byte hashes
pkt, err := DecodePacket("0545"+"AA11BB22CC33DD44EE55"+strings.Repeat("00", 10), nil)
pkt, err := DecodePacket("0545"+"AA11BB22CC33DD44EE55"+strings.Repeat("00", 10), nil, false)
if err != nil {
t.Fatal(err)
}
@@ -112,7 +116,7 @@ func TestDecodePath2ByteHashes(t *testing.T) {
func TestDecodePath3ByteHashes(t *testing.T) {
// 0x8A: 10 hops, 3-byte hashes
pkt, err := DecodePacket("058A"+strings.Repeat("AA11FF", 10)+strings.Repeat("00", 10), nil)
pkt, err := DecodePacket("058A"+strings.Repeat("AA11FF", 10)+strings.Repeat("00", 10), nil, false)
if err != nil {
t.Fatal(err)
}
@@ -131,7 +135,7 @@ func TestTransportCodes(t *testing.T) {
// Route type 0 (TRANSPORT_FLOOD) should have transport codes
// Firmware order: header + transport_codes(4) + path_len + path + payload
hex := "14" + "AABB" + "CCDD" + "00" + strings.Repeat("00", 10)
pkt, err := DecodePacket(hex, nil)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -149,7 +153,7 @@ func TestTransportCodes(t *testing.T) {
}
// Route type 1 (FLOOD) should NOT have transport codes
pkt2, err := DecodePacket("0500"+strings.Repeat("00", 10), nil)
pkt2, err := DecodePacket("0500"+strings.Repeat("00", 10), nil, false)
if err != nil {
t.Fatal(err)
}
@@ -169,7 +173,7 @@ func TestDecodeAdvertFull(t *testing.T) {
name := "546573744E6F6465" // "TestNode"
hex := "1200" + pubkey + timestamp + signature + flags + lat + lon + name
pkt, err := DecodePacket(hex, nil)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -227,7 +231,7 @@ func TestDecodeAdvertTypeEnums(t *testing.T) {
makeAdvert := func(flagsByte byte) *DecodedPacket {
hex := "1200" + strings.Repeat("AA", 32) + "00000000" + strings.Repeat("BB", 64) +
strings.ToUpper(string([]byte{hexDigit(flagsByte>>4), hexDigit(flagsByte & 0x0f)}))
pkt, err := DecodePacket(hex, nil)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -272,7 +276,7 @@ func hexDigit(v byte) byte {
func TestDecodeAdvertNoLocationNoName(t *testing.T) {
hex := "1200" + strings.Repeat("CC", 32) + "00000000" + strings.Repeat("DD", 64) + "02"
pkt, err := DecodePacket(hex, nil)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -291,7 +295,7 @@ func TestDecodeAdvertNoLocationNoName(t *testing.T) {
}
func TestGoldenFixtureTxtMsg(t *testing.T) {
pkt, err := DecodePacket("0A00D69FD7A5A7475DB07337749AE61FA53A4788E976", nil)
pkt, err := DecodePacket("0A00D69FD7A5A7475DB07337749AE61FA53A4788E976", nil, false)
if err != nil {
t.Fatal(err)
}
@@ -314,7 +318,7 @@ func TestGoldenFixtureTxtMsg(t *testing.T) {
func TestGoldenFixtureAdvert(t *testing.T) {
rawHex := "120046D62DE27D4C5194D7821FC5A34A45565DCC2537B300B9AB6275255CEFB65D840CE5C169C94C9AED39E8BCB6CB6EB0335497A198B33A1A610CD3B03D8DCFC160900E5244280323EE0B44CACAB8F02B5B38B91CFA18BD067B0B5E63E94CFC85F758A8530B9240933402E0E6B8F84D5252322D52"
pkt, err := DecodePacket(rawHex, nil)
pkt, err := DecodePacket(rawHex, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -337,7 +341,7 @@ func TestGoldenFixtureAdvert(t *testing.T) {
func TestGoldenFixtureUnicodeAdvert(t *testing.T) {
rawHex := "120073CFF971E1CB5754A742C152B2D2E0EB108A19B246D663ED8898A72C4A5AD86EA6768E66694B025EDF6939D5C44CFF719C5D5520E5F06B20680A83AD9C2C61C3227BBB977A85EE462F3553445FECF8EDD05C234ECE217272E503F14D6DF2B1B9B133890C923CDF3002F8FDC1F85045414BF09F8CB3"
pkt, err := DecodePacket(rawHex, nil)
pkt, err := DecodePacket(rawHex, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -354,14 +358,14 @@ func TestGoldenFixtureUnicodeAdvert(t *testing.T) {
}
func TestDecodePacketTooShort(t *testing.T) {
_, err := DecodePacket("FF", nil)
_, err := DecodePacket("FF", nil, false)
if err == nil {
t.Error("expected error for 1-byte packet")
}
}
func TestDecodePacketInvalidHex(t *testing.T) {
_, err := DecodePacket("ZZZZ", nil)
_, err := DecodePacket("ZZZZ", nil, false)
if err == nil {
t.Error("expected error for invalid hex")
}
@@ -568,7 +572,7 @@ func TestDecodeTracePathParsing(t *testing.T) {
// Packet from issue #276: 260001807dca00000000007d547d
// Path byte 0x00 → hashSize=1, hops in payload at buf[9:] = 7d 54 7d
// Expected path: ["7D", "54", "7D"]
pkt, err := DecodePacket("260001807dca00000000007d547d", nil)
pkt, err := DecodePacket("260001807dca00000000007d547d", nil, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
@@ -590,7 +594,7 @@ func TestDecodeTracePathParsing(t *testing.T) {
}
func TestDecodeAdvertShort(t *testing.T) {
p := decodeAdvert(make([]byte, 50))
p := decodeAdvert(make([]byte, 50), false)
if p.Error != "too short for advert" {
t.Errorf("expected 'too short for advert' error, got %q", p.Error)
}
@@ -628,7 +632,7 @@ func TestDecodeEncryptedPayloadValid(t *testing.T) {
func TestDecodePayloadGRPData(t *testing.T) {
buf := []byte{0x01, 0x02, 0x03}
p := decodePayload(PayloadGRP_DATA, buf, nil)
p := decodePayload(PayloadGRP_DATA, buf, nil, false)
if p.Type != "UNKNOWN" {
t.Errorf("type=%s, want UNKNOWN", p.Type)
}
@@ -639,7 +643,7 @@ func TestDecodePayloadGRPData(t *testing.T) {
func TestDecodePayloadRAWCustom(t *testing.T) {
buf := []byte{0xFF, 0xFE}
p := decodePayload(PayloadRAW_CUSTOM, buf, nil)
p := decodePayload(PayloadRAW_CUSTOM, buf, nil, false)
if p.Type != "UNKNOWN" {
t.Errorf("type=%s, want UNKNOWN", p.Type)
}
@@ -647,49 +651,49 @@ func TestDecodePayloadRAWCustom(t *testing.T) {
func TestDecodePayloadAllTypes(t *testing.T) {
// REQ
p := decodePayload(PayloadREQ, make([]byte, 10), nil)
p := decodePayload(PayloadREQ, make([]byte, 10), nil, false)
if p.Type != "REQ" {
t.Errorf("REQ: type=%s", p.Type)
}
// RESPONSE
p = decodePayload(PayloadRESPONSE, make([]byte, 10), nil)
p = decodePayload(PayloadRESPONSE, make([]byte, 10), nil, false)
if p.Type != "RESPONSE" {
t.Errorf("RESPONSE: type=%s", p.Type)
}
// TXT_MSG
p = decodePayload(PayloadTXT_MSG, make([]byte, 10), nil)
p = decodePayload(PayloadTXT_MSG, make([]byte, 10), nil, false)
if p.Type != "TXT_MSG" {
t.Errorf("TXT_MSG: type=%s", p.Type)
}
// ACK
p = decodePayload(PayloadACK, make([]byte, 10), nil)
p = decodePayload(PayloadACK, make([]byte, 10), nil, false)
if p.Type != "ACK" {
t.Errorf("ACK: type=%s", p.Type)
}
// GRP_TXT
p = decodePayload(PayloadGRP_TXT, make([]byte, 10), nil)
p = decodePayload(PayloadGRP_TXT, make([]byte, 10), nil, false)
if p.Type != "GRP_TXT" {
t.Errorf("GRP_TXT: type=%s", p.Type)
}
// ANON_REQ
p = decodePayload(PayloadANON_REQ, make([]byte, 40), nil)
p = decodePayload(PayloadANON_REQ, make([]byte, 40), nil, false)
if p.Type != "ANON_REQ" {
t.Errorf("ANON_REQ: type=%s", p.Type)
}
// PATH
p = decodePayload(PayloadPATH, make([]byte, 10), nil)
p = decodePayload(PayloadPATH, make([]byte, 10), nil, false)
if p.Type != "PATH" {
t.Errorf("PATH: type=%s", p.Type)
}
// TRACE
p = decodePayload(PayloadTRACE, make([]byte, 20), nil)
p = decodePayload(PayloadTRACE, make([]byte, 20), nil, false)
if p.Type != "TRACE" {
t.Errorf("TRACE: type=%s", p.Type)
}
@@ -923,9 +927,96 @@ func TestComputeContentHashLongFallback(t *testing.T) {
}
}
// TestComputeContentHashRouteTypeIndependence verifies that the same logical
// packet produces the same content hash regardless of route type (issue #786).
func TestComputeContentHashRouteTypeIndependence(t *testing.T) {
// Same payload type (TXT_MSG=2, bits 2-5) with different route types.
// Header 0x08 = route_type 0 (TRANSPORT_FLOOD), payload_type 2
// Header 0x0A = route_type 2 (DIRECT), payload_type 2
// Header 0x09 = route_type 1 (FLOOD), payload_type 2
// pathByte=0x00, payload=D69FD7A5A7
payloadHex := "D69FD7A5A7"
// FLOOD: header=0x09 (route_type 1), pathByte=0x00
floodHex := "09" + "00" + payloadHex
// DIRECT: header=0x0A (route_type 2), pathByte=0x00
directHex := "0A" + "00" + payloadHex
hashFlood := ComputeContentHash(floodHex)
hashDirect := ComputeContentHash(directHex)
if hashFlood != hashDirect {
t.Errorf("same payload with different route types produced different hashes: flood=%s direct=%s", hashFlood, hashDirect)
}
}
// TestComputeContentHashTraceIncludesPathLen verifies TRACE packets include
// path_len in the hash (matching firmware behavior).
func TestComputeContentHashTraceIncludesPathLen(t *testing.T) {
// TRACE = payload_type 0x09, so header bits 2-5 = 0x09 → header = 0x09<<2 | route=2 = 0x26
// pathByte=0x01 (1 hop, 1-byte hash) → 1 path byte
traceHeader1 := "26" // route=2, payload_type=9
pathByte1 := "01"
pathData1 := "AA"
payload := "DEADBEEF"
hex1 := traceHeader1 + pathByte1 + pathData1 + payload
// Same but pathByte=0x02 (2 hops) → 2 path bytes
pathByte2 := "02"
pathData2 := "AABB"
hex2 := traceHeader1 + pathByte2 + pathData2 + payload
hash1 := ComputeContentHash(hex1)
hash2 := ComputeContentHash(hex2)
if hash1 == hash2 {
t.Error("TRACE packets with different path_len should produce different hashes (path_len is part of hash input)")
}
}
// TestComputeContentHashMatchesFirmware verifies hash output matches what the
// firmware would compute: SHA256(payload_type_byte + payload)[:16hex].
func TestComputeContentHashMatchesFirmware(t *testing.T) {
// header=0x0A → payload_type = (0x0A >> 2) & 0x0F = 2
// pathByte=0x00, payload = D69FD7A5A7475DB07337749AE61FA53A4788E976
rawHex := "0A00D69FD7A5A7475DB07337749AE61FA53A4788E976"
hash := ComputeContentHash(rawHex)
// Manually compute expected: SHA256(0x02 + payload_bytes)
payloadBytes, _ := hex.DecodeString("D69FD7A5A7475DB07337749AE61FA53A4788E976")
toHash := append([]byte{0x02}, payloadBytes...)
expected := sha256.Sum256(toHash)
expectedHex := hex.EncodeToString(expected[:])[:16]
if hash != expectedHex {
t.Errorf("hash=%s, want %s (firmware-compatible)", hash, expectedHex)
}
}
// TestComputeContentHashTraceGoldenValue is a golden-value test that locks down
// the 2-byte path_len (uint16 LE) behavior for TRACE hashing. If anyone removes
// the 0x00 byte from the hash input, this test breaks.
//
// Packet: header=0x25 (FLOOD route=1, payload_type=TRACE=0x09), pathByte=0x02
// (2 hops, 1-byte hash), path=[AA,BB], payload=[DE,AD,BE,EF].
// Hash input: [0x09, 0x02, 0x00, 0xDE, 0xAD, 0xBE, 0xEF]
// → SHA256 = b1baaf3bf0d0726c2672b1ec9e2665dc...
// → first 16 hex chars = "b1baaf3bf0d0726c"
func TestComputeContentHashTraceGoldenValue(t *testing.T) {
// TRACE packet: header byte 0x25 = payload_type 9 (TRACE), route_type 1 (FLOOD)
// pathByte 0x02 = hash_size 1, hash_count 2
// 2 path bytes (AA, BB), then payload DEADBEEF
rawHex := "2502AABBDEADBEEF"
hash := ComputeContentHash(rawHex)
// Pre-computed: SHA256(0x09 0x02 0x00 0xDE 0xAD 0xBE 0xEF)[:16hex]
// The 0x00 is the high byte of uint16_t path_len (little-endian).
const golden = "b1baaf3bf0d0726c"
if hash != golden {
t.Errorf("TRACE golden hash = %s, want %s (2-byte path_len encoding)", hash, golden)
}
}
func TestDecodePacketWithWhitespace(t *testing.T) {
raw := "0A 00 D6 9F D7 A5 A7 47 5D B0 73 37 74 9A E6 1F A5 3A 47 88 E9 76"
pkt, err := DecodePacket(raw, nil)
pkt, err := DecodePacket(raw, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -936,7 +1027,7 @@ func TestDecodePacketWithWhitespace(t *testing.T) {
func TestDecodePacketWithNewlines(t *testing.T) {
raw := "0A00\nD69F\r\nD7A5A7475DB07337749AE61FA53A4788E976"
pkt, err := DecodePacket(raw, nil)
pkt, err := DecodePacket(raw, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -947,7 +1038,7 @@ func TestDecodePacketWithNewlines(t *testing.T) {
func TestDecodePacketTransportRouteTooShort(t *testing.T) {
// TRANSPORT_FLOOD (route=0) but only 2 bytes total → too short for transport codes
_, err := DecodePacket("1400", nil)
_, err := DecodePacket("1400", nil, false)
if err == nil {
t.Error("expected error for transport route with too-short buffer")
}
@@ -1007,7 +1098,7 @@ func TestDecodeHeaderUnknownTypes(t *testing.T) {
func TestDecodePayloadMultipart(t *testing.T) {
// MULTIPART (0x0A) falls through to default → UNKNOWN
p := decodePayload(PayloadMULTIPART, []byte{0x01, 0x02}, nil)
p := decodePayload(PayloadMULTIPART, []byte{0x01, 0x02}, nil, false)
if p.Type != "UNKNOWN" {
t.Errorf("MULTIPART type=%s, want UNKNOWN", p.Type)
}
@@ -1015,7 +1106,7 @@ func TestDecodePayloadMultipart(t *testing.T) {
func TestDecodePayloadControl(t *testing.T) {
// CONTROL (0x0B) falls through to default → UNKNOWN
p := decodePayload(PayloadCONTROL, []byte{0x01, 0x02}, nil)
p := decodePayload(PayloadCONTROL, []byte{0x01, 0x02}, nil, false)
if p.Type != "UNKNOWN" {
t.Errorf("CONTROL type=%s, want UNKNOWN", p.Type)
}
@@ -1039,7 +1130,7 @@ func TestDecodePathTruncatedBuffer(t *testing.T) {
func TestDecodeFloodAdvert5Hops(t *testing.T) {
// From test-decoder.js Test 1
raw := "11451000D818206D3AAC152C8A91F89957E6D30CA51F36E28790228971C473B755F244F718754CF5EE4A2FD58D944466E42CDED140C66D0CC590183E32BAF40F112BE8F3F2BDF6012B4B2793C52F1D36F69EE054D9A05593286F78453E56C0EC4A3EB95DDA2A7543FCCC00B939CACC009278603902FC12BCF84B706120526F6F6620536F6C6172"
pkt, err := DecodePacket(raw, nil)
pkt, err := DecodePacket(raw, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -1410,7 +1501,7 @@ func TestDecodeAdvertWithTelemetry(t *testing.T) {
name + nullTerm +
hex.EncodeToString(batteryLE) + hex.EncodeToString(tempLE)
pkt, err := DecodePacket(hexStr, nil)
pkt, err := DecodePacket(hexStr, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -1449,7 +1540,7 @@ func TestDecodeAdvertWithTelemetryNegativeTemp(t *testing.T) {
name + nullTerm +
hex.EncodeToString(batteryLE) + hex.EncodeToString(tempLE)
pkt, err := DecodePacket(hexStr, nil)
pkt, err := DecodePacket(hexStr, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -1476,7 +1567,7 @@ func TestDecodeAdvertWithoutTelemetry(t *testing.T) {
name := hex.EncodeToString([]byte("Node1"))
hexStr := "1200" + pubkey + timestamp + signature + flags + name
pkt, err := DecodePacket(hexStr, nil)
pkt, err := DecodePacket(hexStr, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -1503,7 +1594,7 @@ func TestDecodeAdvertNonSensorIgnoresTelemetryBytes(t *testing.T) {
extraBytes := "B40ED403" // battery-like and temp-like bytes
hexStr := "1200" + pubkey + timestamp + signature + flags + name + nullTerm + extraBytes
pkt, err := DecodePacket(hexStr, nil)
pkt, err := DecodePacket(hexStr, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -1531,7 +1622,7 @@ func TestDecodeAdvertTelemetryZeroTemp(t *testing.T) {
name + nullTerm +
hex.EncodeToString(batteryLE) + hex.EncodeToString(tempLE)
pkt, err := DecodePacket(hexStr, nil)
pkt, err := DecodePacket(hexStr, nil, false)
if err != nil {
t.Fatal(err)
}
@@ -1542,3 +1633,296 @@ func TestDecodeAdvertTelemetryZeroTemp(t *testing.T) {
t.Errorf("temperature_c=%f, want 0.0", *pkt.Payload.TemperatureC)
}
}
func repeatHex(byteHex string, n int) string {
s := ""
for i := 0; i < n; i++ {
s += byteHex
}
return s
}
func TestZeroHopDirectHashSize(t *testing.T) {
// DIRECT (RouteType=2) + REQ (PayloadType=0) → header byte = 0x02
// pathByte=0x00 → hash_count=0, hash_size bits=0 → should get HashSize=0
hex := "02" + "00" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 0 {
t.Errorf("DIRECT zero-hop: want HashSize=0, got %d", pkt.Path.HashSize)
}
}
func TestZeroHopDirectHashSizeWithNonZeroUpperBits(t *testing.T) {
// DIRECT (RouteType=2) + REQ (PayloadType=0) → header byte = 0x02
// pathByte=0x40 → hash_count=0, hash_size bits=01 → should still get HashSize=0
hex := "02" + "40" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 0 {
t.Errorf("DIRECT zero-hop with hash_size bits set: want HashSize=0, got %d", pkt.Path.HashSize)
}
}
func TestNonDirectZeroPathByteKeepsHashSize(t *testing.T) {
// FLOOD (RouteType=1) + REQ (PayloadType=0) → header byte = 0x01
// pathByte=0x00 → non-DIRECT should keep HashSize=1
hex := "01" + "00" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 1 {
t.Errorf("FLOOD zero pathByte: want HashSize=1, got %d", pkt.Path.HashSize)
}
}
func TestDirectNonZeroHopKeepsHashSize(t *testing.T) {
// DIRECT (RouteType=2) + REQ (PayloadType=0) → header byte = 0x02
// pathByte=0x01 → hash_count=1, hash_size=1 → should keep HashSize=1
hex := "02" + "01" + repeatHex("BB", 21)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 1 {
t.Errorf("DIRECT with 1 hop: want HashSize=1, got %d", pkt.Path.HashSize)
}
}
func TestZeroHopTransportDirectHashSize(t *testing.T) {
// TRANSPORT_DIRECT (RouteType=3) + REQ (PayloadType=0) → header byte = 0x03
// 4 bytes transport codes + pathByte=0x00 → hash_count=0 → should get HashSize=0
hex := "03" + "11223344" + "00" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 0 {
t.Errorf("TRANSPORT_DIRECT zero-hop: want HashSize=0, got %d", pkt.Path.HashSize)
}
}
func TestZeroHopTransportDirectHashSizeWithNonZeroUpperBits(t *testing.T) {
// TRANSPORT_DIRECT (RouteType=3) + REQ (PayloadType=0) → header byte = 0x03
// 4 bytes transport codes + pathByte=0xC0 → hash_count=0, hash_size bits=11 → should still get HashSize=0
hex := "03" + "11223344" + "C0" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, nil, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 0 {
t.Errorf("TRANSPORT_DIRECT zero-hop with hash_size bits set: want HashSize=0, got %d", pkt.Path.HashSize)
}
}
func TestValidateAdvertSignature(t *testing.T) {
// Generate a real ed25519 key pair
pub, priv, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
var timestamp uint32 = 1234567890
appdata := []byte{0x02, 0x11, 0x22} // flags + some data
// Build the signed message: pubKey + timestamp(LE) + appdata
message := make([]byte, 32+4+len(appdata))
copy(message[0:32], pub)
binary.LittleEndian.PutUint32(message[32:36], timestamp)
copy(message[36:], appdata)
sig := ed25519.Sign(priv, message)
// Valid signature
valid, err := sigvalidate.ValidateAdvert([]byte(pub), sig, timestamp, appdata)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if !valid {
t.Error("expected valid signature")
}
// Tampered appdata → invalid
badAppdata := []byte{0x03, 0x11, 0x22}
valid, err = sigvalidate.ValidateAdvert([]byte(pub), sig, timestamp, badAppdata)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if valid {
t.Error("expected invalid signature with tampered appdata")
}
// Wrong timestamp → invalid
valid, err = sigvalidate.ValidateAdvert([]byte(pub), sig, timestamp+1, appdata)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if valid {
t.Error("expected invalid signature with wrong timestamp")
}
// Wrong length pubkey
_, err = sigvalidate.ValidateAdvert([]byte{0xAA, 0xBB}, sig, timestamp, appdata)
if err == nil {
t.Error("expected error for short pubkey")
}
// Wrong length signature
_, err = sigvalidate.ValidateAdvert([]byte(pub), []byte{0xAA, 0xBB}, timestamp, appdata)
if err == nil {
t.Error("expected error for short signature")
}
}
func TestDecodeAdvertWithSignatureValidation(t *testing.T) {
// Generate key pair
pub, priv, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
var timestamp uint32 = 1000000
appdata := []byte{0x02} // repeater type, no location
// Build signed message
message := make([]byte, 32+4+len(appdata))
copy(message[0:32], pub)
binary.LittleEndian.PutUint32(message[32:36], timestamp)
copy(message[36:], appdata)
sig := ed25519.Sign(priv, message)
// Build advert buffer: pubkey(32) + timestamp(4) + signature(64) + appdata
buf := make([]byte, 0, 101)
buf = append(buf, pub...)
ts := make([]byte, 4)
binary.LittleEndian.PutUint32(ts, timestamp)
buf = append(buf, ts...)
buf = append(buf, sig...)
buf = append(buf, appdata...)
// With validation enabled
p := decodeAdvert(buf, true)
if p.Error != "" {
t.Fatalf("decode error: %s", p.Error)
}
if p.SignatureValid == nil {
t.Fatal("SignatureValid should be set when validation enabled")
}
if !*p.SignatureValid {
t.Error("expected valid signature")
}
// Without validation
p2 := decodeAdvert(buf, false)
if p2.SignatureValid != nil {
t.Error("SignatureValid should be nil when validation disabled")
}
}
// === Tests for DecodePathFromRawHex (issue #886) ===
func TestDecodePathFromRawHex_HashSize1(t *testing.T) {
// Header byte 0x26 = route_type DIRECT, payload TRACE
// Path byte 0x04 = hash_size 1 (bits 7-6 = 00 → 0+1=1), hash_count 4
// Path bytes: 30 2D 0D 23
raw := "2604302D0D2359FEE7B100000000006733D63367"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
expected := []string{"30", "2D", "0D", "23"}
if len(hops) != len(expected) {
t.Fatalf("got %d hops, want %d", len(hops), len(expected))
}
for i, h := range hops {
if h != expected[i] {
t.Errorf("hop[%d] = %s, want %s", i, h, expected[i])
}
}
}
func TestDecodePathFromRawHex_HashSize2(t *testing.T) {
// Path byte 0x42 = hash_size 2 (bits 7-6 = 01 → 1+1=2), hash_count 2
// Header 0x09 = FLOOD route (rt=1), payload ADVERT (pt=2)
// Path bytes: AABB CCDD (4 bytes = 2 hops * 2 bytes)
raw := "0942AABBCCDD" + "00000000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
expected := []string{"AABB", "CCDD"}
if len(hops) != len(expected) {
t.Fatalf("got %d hops, want %d", len(hops), len(expected))
}
for i, h := range hops {
if h != expected[i] {
t.Errorf("hop[%d] = %s, want %s", i, h, expected[i])
}
}
}
func TestDecodePathFromRawHex_HashSize3(t *testing.T) {
// Path byte 0x81 = hash_size 3 (bits 7-6 = 10 → 2+1=3), hash_count 1
// Header 0x09 = FLOOD route (rt=1), payload ADVERT
raw := "0981AABBCC" + "0000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
if len(hops) != 1 || hops[0] != "AABBCC" {
t.Fatalf("got %v, want [AABBCC]", hops)
}
}
func TestDecodePathFromRawHex_HashSize4(t *testing.T) {
// Path byte 0xC1 = hash_size 4 (bits 7-6 = 11 → 3+1=4), hash_count 1
// Header 0x09 = FLOOD route (rt=1)
raw := "09C1AABBCCDD" + "0000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
if len(hops) != 1 || hops[0] != "AABBCCDD" {
t.Fatalf("got %v, want [AABBCCDD]", hops)
}
}
func TestDecodePathFromRawHex_DirectZeroHops(t *testing.T) {
// Path byte 0x00 = hash_size 1, hash_count 0
// Header 0x0A = DIRECT route (rt=2), payload ADVERT
raw := "0A00" + "0000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
if len(hops) != 0 {
t.Fatalf("got %d hops, want 0", len(hops))
}
}
func TestDecodePathFromRawHex_Transport(t *testing.T) {
// Route type 3 = TRANSPORT_DIRECT → 4 transport code bytes before path byte
// Header 0x27 = route_type 3, payload TRACE
// Transport codes: 1122 3344
// Path byte 0x02 = hash_size 1, hash_count 2
// Path bytes: AA BB
raw := "2711223344" + "02AABB" + "0000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
expected := []string{"AA", "BB"}
if len(hops) != len(expected) {
t.Fatalf("got %d hops, want %d", len(hops), len(expected))
}
for i, h := range hops {
if h != expected[i] {
t.Errorf("hop[%d] = %s, want %s", i, h, expected[i])
}
}
}
+7
View File
@@ -5,11 +5,18 @@ go 1.22
require (
github.com/eclipse/paho.mqtt.golang v1.5.0
github.com/meshcore-analyzer/geofilter v0.0.0
github.com/meshcore-analyzer/sigvalidate v0.0.0
modernc.org/sqlite v1.34.5
)
replace github.com/meshcore-analyzer/geofilter => ../../internal/geofilter
replace github.com/meshcore-analyzer/sigvalidate => ../../internal/sigvalidate
require github.com/meshcore-analyzer/packetpath v0.0.0
replace github.com/meshcore-analyzer/packetpath => ../../internal/packetpath
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
+140 -40
View File
@@ -49,11 +49,8 @@ func main() {
}
sources := cfg.ResolvedSources()
if len(sources) == 0 {
log.Fatal("no MQTT sources configured — set mqttSources in config or MQTT_BROKER env var")
}
store, err := OpenStore(cfg.DBPath)
store, err := OpenStoreWithInterval(cfg.DBPath, cfg.MetricsSampleInterval())
if err != nil {
log.Fatalf("db: %v", err)
}
@@ -64,6 +61,15 @@ func main() {
nodeDays := cfg.NodeDaysOrDefault()
store.MoveStaleNodes(nodeDays)
// Observer retention: remove stale observers on startup
observerDays := cfg.ObserverDaysOrDefault()
store.RemoveStaleObservers(observerDays)
// Metrics retention: prune old metrics on startup
metricsDays := cfg.MetricsRetentionDays()
store.PruneOldMetrics(metricsDays)
store.PruneDroppedPackets(metricsDays)
// Daily ticker for node retention
retentionTicker := time.NewTicker(1 * time.Hour)
go func() {
@@ -72,6 +78,25 @@ func main() {
}
}()
// Daily ticker for observer retention (every 24h, staggered 90s after startup)
observerRetentionTicker := time.NewTicker(24 * time.Hour)
go func() {
time.Sleep(90 * time.Second) // stagger after metrics prune
store.RemoveStaleObservers(observerDays)
for range observerRetentionTicker.C {
store.RemoveStaleObservers(observerDays)
}
}()
// Daily ticker for metrics retention (every 24h)
metricsRetentionTicker := time.NewTicker(24 * time.Hour)
go func() {
for range metricsRetentionTicker.C {
store.PruneOldMetrics(metricsDays)
store.PruneDroppedPackets(metricsDays)
}
}()
// Periodic stats logging (every 5 minutes)
statsTicker := time.NewTicker(5 * time.Minute)
go func() {
@@ -137,7 +162,7 @@ func main() {
// Capture source for closure
src := source
opts.SetDefaultPublishHandler(func(c mqtt.Client, m mqtt.Message) {
handleMessage(store, tag, src, m, channelKeys, cfg.GeoFilter)
handleMessage(store, tag, src, m, channelKeys, cfg)
})
client := mqtt.NewClient(opts)
@@ -151,7 +176,7 @@ func main() {
}
if len(clients) == 0 {
log.Fatal("no MQTT connections established")
log.Fatal("no MQTT connections established — check broker is running (default: mqtt://localhost:1883). Set MQTT_BROKER env var or configure mqttSources in config.json")
}
log.Printf("Running — %d MQTT source(s) connected", len(clients))
@@ -163,6 +188,7 @@ func main() {
log.Println("Shutting down...")
retentionTicker.Stop()
metricsRetentionTicker.Stop()
statsTicker.Stop()
store.LogStats() // final stats on shutdown
for _, c := range clients {
@@ -171,7 +197,7 @@ func main() {
log.Println("Done.")
}
func handleMessage(store *Store, tag string, source MQTTSource, m mqtt.Message, channelKeys map[string]string, geoFilter *GeoFilterConfig) {
func handleMessage(store *Store, tag string, source MQTTSource, m mqtt.Message, channelKeys map[string]string, cfg *Config) {
defer func() {
if r := recover(); r != nil {
log.Printf("MQTT [%s] panic in handler: %v", tag, r)
@@ -181,7 +207,49 @@ func handleMessage(store *Store, tag string, source MQTTSource, m mqtt.Message,
topic := m.Topic()
parts := strings.Split(topic, "/")
// IATA filter
var msg map[string]interface{}
if err := json.Unmarshal(m.Payload(), &msg); err != nil {
return
}
// Skip status/connection topics
if topic == "meshcore/status" || topic == "meshcore/events/connection" {
return
}
// Status topic: meshcore/<region>/<observer_id>/status
// IATA filter does NOT apply here — observer metadata (noise_floor, battery, etc.)
// is region-independent and should be accepted from all observers regardless of
// which IATA regions are configured for packet ingestion.
if len(parts) >= 4 && parts[3] == "status" {
observerID := parts[2]
name, _ := msg["origin"].(string)
iata := parts[1]
meta := extractObserverMeta(msg)
if err := store.UpsertObserver(observerID, name, iata, meta); err != nil {
log.Printf("MQTT [%s] observer status error: %v", tag, err)
}
// Insert metrics sample from status message
if meta != nil {
metricsData := &MetricsData{
ObserverID: observerID,
NoiseFloor: meta.NoiseFloor,
TxAirSecs: meta.TxAirSecs,
RxAirSecs: meta.RxAirSecs,
RecvErrors: meta.RecvErrors,
BatteryMv: meta.BatteryMv,
PacketsSent: meta.PacketsSent,
PacketsRecv: meta.PacketsRecv,
}
if err := store.InsertMetrics(metricsData); err != nil {
log.Printf("MQTT [%s] metrics insert error: %v", tag, err)
}
}
log.Printf("MQTT [%s] status: %s (%s)", tag, firstNonEmpty(name, observerID), iata)
return
}
// IATA filter applies to packet messages only — not status messages above.
if len(source.IATAFilter) > 0 && len(parts) > 1 {
region := parts[1]
matched := false
@@ -196,33 +264,11 @@ func handleMessage(store *Store, tag string, source MQTTSource, m mqtt.Message,
}
}
var msg map[string]interface{}
if err := json.Unmarshal(m.Payload(), &msg); err != nil {
return
}
// Skip status/connection topics
if topic == "meshcore/status" || topic == "meshcore/events/connection" {
return
}
// Status topic: meshcore/<region>/<observer_id>/status
if len(parts) >= 4 && parts[3] == "status" {
observerID := parts[2]
name, _ := msg["origin"].(string)
iata := parts[1]
meta := extractObserverMeta(msg)
if err := store.UpsertObserver(observerID, name, iata, meta); err != nil {
log.Printf("MQTT [%s] observer status error: %v", tag, err)
}
log.Printf("MQTT [%s] status: %s (%s)", tag, firstNonEmpty(name, observerID), iata)
return
}
// Format 1: Raw packet (meshcoretomqtt / Cisien format)
rawHex, _ := msg["raw"].(string)
if rawHex != "" {
decoded, err := DecodePacket(rawHex, channelKeys)
validateSigs := cfg.ShouldValidateSignatures()
decoded, err := DecodePacket(rawHex, channelKeys, validateSigs)
if err != nil {
log.Printf("MQTT [%s] decode error: %v", tag, err)
return
@@ -282,7 +328,27 @@ func handleMessage(store *Store, tag string, source MQTTSource, m mqtt.Message,
log.Printf("MQTT [%s] skipping corrupted ADVERT: %s", tag, reason)
return
}
if !NodePassesGeoFilter(decoded.Payload.Lat, decoded.Payload.Lon, geoFilter) {
// Signature validation: drop adverts with invalid ed25519 signatures
if validateSigs && decoded.Payload.SignatureValid != nil && !*decoded.Payload.SignatureValid {
hash := ComputeContentHash(rawHex)
truncPK := decoded.Payload.PubKey
if len(truncPK) > 16 {
truncPK = truncPK[:16]
}
log.Printf("MQTT [%s] DROPPED invalid signature: hash=%s name=%s observer=%s pubkey=%s",
tag, hash, decoded.Payload.Name, firstNonEmpty(mqttMsg.Origin, observerID), truncPK)
store.InsertDroppedPacket(&DroppedPacket{
Hash: hash,
RawHex: rawHex,
Reason: "invalid signature",
ObserverID: observerID,
ObserverName: mqttMsg.Origin,
NodePubKey: decoded.Payload.PubKey,
NodeName: decoded.Payload.Name,
})
return
}
if !NodePassesGeoFilter(decoded.Payload.Lat, decoded.Payload.Lon, cfg.GeoFilter) {
return
}
pktData := BuildPacketData(mqttMsg, decoded, observerID, region)
@@ -414,19 +480,18 @@ func handleMessage(store *Store, tag string, source MQTTSource, m mqtt.Message,
PayloadType: 5, // GRP_TXT
PathJSON: "[]",
DecodedJSON: string(decodedJSON),
ChannelHash: channelName, // fast channel queries (#762)
}
if _, err := store.InsertTransmission(pktData); err != nil {
log.Printf("MQTT [%s] channel insert error: %v", tag, err)
}
// Upsert sender as a companion node
if sender != "" {
senderKey := "sender-" + strings.ToLower(sender)
if err := store.UpsertNode(senderKey, sender, "companion", nil, nil, now); err != nil {
log.Printf("MQTT [%s] sender node upsert error: %v", tag, err)
}
}
// Note: we intentionally do NOT create a node entry for channel message senders.
// Channel messages don't carry the sender's real pubkey, so any entry we create
// would use a synthetic key ("sender-<name>") that doesn't match the real pubkey
// used for claiming/health lookups. The node will get a proper entry when it
// sends an advert. See issue #665.
log.Printf("MQTT [%s] channel message: ch%s from %s", tag, channelIdx, firstNonEmpty(sender, "unknown"))
return
@@ -616,6 +681,41 @@ func extractObserverMeta(msg map[string]interface{}) *ObserverMeta {
hasData = true
}
}
if v := nestedOrTopLevel(stats, msg, "tx_air_secs"); v != nil {
if f, ok := toFloat64(v); ok {
iv := int(math.Round(f))
meta.TxAirSecs = &iv
hasData = true
}
}
if v := nestedOrTopLevel(stats, msg, "rx_air_secs"); v != nil {
if f, ok := toFloat64(v); ok {
iv := int(math.Round(f))
meta.RxAirSecs = &iv
hasData = true
}
}
if v := nestedOrTopLevel(stats, msg, "recv_errors"); v != nil {
if f, ok := toFloat64(v); ok {
iv := int(math.Round(f))
meta.RecvErrors = &iv
hasData = true
}
}
if v := nestedOrTopLevel(stats, msg, "packets_sent"); v != nil {
if f, ok := toFloat64(v); ok {
iv := int(math.Round(f))
meta.PacketsSent = &iv
hasData = true
}
}
if v := nestedOrTopLevel(stats, msg, "packets_recv"); v != nil {
if f, ok := toFloat64(v); ok {
iv := int(math.Round(f))
meta.PacketsRecv = &iv
hasData = true
}
}
if !hasData {
return nil
+63 -22
View File
@@ -130,7 +130,7 @@ func TestHandleMessageRawPacket(t *testing.T) {
payload := []byte(`{"raw":"` + rawHex + `","SNR":5.5,"RSSI":-100.0,"origin":"myobs"}`)
msg := &mockMessage{topic: "meshcore/SJC/obs1/packets", payload: payload}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
@@ -147,7 +147,7 @@ func TestHandleMessageRawPacketAdvert(t *testing.T) {
payload := []byte(`{"raw":"` + rawHex + `"}`)
msg := &mockMessage{topic: "meshcore/SJC/obs1/packets", payload: payload}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
// Should create a node from the ADVERT
var count int
@@ -169,7 +169,7 @@ func TestHandleMessageInvalidJSON(t *testing.T) {
msg := &mockMessage{topic: "meshcore/SJC/obs1/packets", payload: []byte(`not json`)}
// Should not panic
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
@@ -186,7 +186,7 @@ func TestHandleMessageStatusTopic(t *testing.T) {
payload: []byte(`{"origin":"MyObserver"}`),
}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var name, iata string
err := store.db.QueryRow("SELECT name, iata FROM observers WHERE id = 'obs1'").Scan(&name, &iata)
@@ -207,11 +207,11 @@ func TestHandleMessageSkipStatusTopics(t *testing.T) {
// meshcore/status should be skipped
msg1 := &mockMessage{topic: "meshcore/status", payload: []byte(`{"raw":"0A00"}`)}
handleMessage(store, "test", source, msg1, nil, nil)
handleMessage(store, "test", source, msg1, nil, &Config{})
// meshcore/events/connection should be skipped
msg2 := &mockMessage{topic: "meshcore/events/connection", payload: []byte(`{"raw":"0A00"}`)}
handleMessage(store, "test", source, msg2, nil, nil)
handleMessage(store, "test", source, msg2, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
@@ -230,7 +230,7 @@ func TestHandleMessageIATAFilter(t *testing.T) {
topic: "meshcore/SJC/obs1/packets",
payload: []byte(`{"raw":"` + rawHex + `"}`),
}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
@@ -243,7 +243,7 @@ func TestHandleMessageIATAFilter(t *testing.T) {
topic: "meshcore/LAX/obs2/packets",
payload: []byte(`{"raw":"` + rawHex + `"}`),
}
handleMessage(store, "test", source, msg2, nil, nil)
handleMessage(store, "test", source, msg2, nil, &Config{})
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
if count != 1 {
@@ -261,7 +261,7 @@ func TestHandleMessageIATAFilterNoRegion(t *testing.T) {
topic: "meshcore",
payload: []byte(`{"raw":"` + rawHex + `"}`),
}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
// No region part → filter doesn't apply, message goes through
// Actually the code checks len(parts) > 1 for IATA filter
@@ -277,7 +277,7 @@ func TestHandleMessageNoRawHex(t *testing.T) {
topic: "meshcore/SJC/obs1/packets",
payload: []byte(`{"type":"companion","data":"something"}`),
}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
@@ -295,7 +295,7 @@ func TestHandleMessageBadRawHex(t *testing.T) {
topic: "meshcore/SJC/obs1/packets",
payload: []byte(`{"raw":"ZZZZ"}`),
}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
@@ -312,7 +312,7 @@ func TestHandleMessageWithSNRRSSIAsNumbers(t *testing.T) {
payload := []byte(`{"raw":"` + rawHex + `","SNR":7.2,"RSSI":-95}`)
msg := &mockMessage{topic: "meshcore/SJC/obs1/packets", payload: payload}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var snr, rssi *float64
store.db.QueryRow("SELECT snr, rssi FROM observations LIMIT 1").Scan(&snr, &rssi)
@@ -331,7 +331,7 @@ func TestHandleMessageMinimalTopic(t *testing.T) {
topic: "meshcore/SJC",
payload: []byte(`{"raw":"` + rawHex + `"}`),
}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
@@ -352,7 +352,7 @@ func TestHandleMessageCorruptedAdvert(t *testing.T) {
topic: "meshcore/SJC/obs1/packets",
payload: []byte(`{"raw":"` + rawHex + `"}`),
}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
// Transmission should be inserted (even if advert is invalid)
var count int
@@ -378,7 +378,7 @@ func TestHandleMessageNoObserverID(t *testing.T) {
topic: "packets",
payload: []byte(`{"raw":"` + rawHex + `","origin":"obs1"}`),
}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
@@ -400,7 +400,7 @@ func TestHandleMessageSNRNotFloat(t *testing.T) {
// SNR as a string value — should not parse as float
payload := []byte(`{"raw":"` + rawHex + `","SNR":"bad","RSSI":"bad"}`)
msg := &mockMessage{topic: "meshcore/SJC/obs1/packets", payload: payload}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
@@ -416,7 +416,7 @@ func TestHandleMessageOriginExtraction(t *testing.T) {
rawHex := "0A00D69FD7A5A7475DB07337749AE61FA53A4788E976"
payload := []byte(`{"raw":"` + rawHex + `","origin":"MyOrigin"}`)
msg := &mockMessage{topic: "meshcore/SJC/obs1/packets", payload: payload}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
// Verify origin was extracted to observer name
var name string
@@ -439,7 +439,7 @@ func TestHandleMessagePanicRecovery(t *testing.T) {
}
// Should not panic — the defer/recover should catch it
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
}
func TestHandleMessageStatusOriginFallback(t *testing.T) {
@@ -451,7 +451,7 @@ func TestHandleMessageStatusOriginFallback(t *testing.T) {
topic: "meshcore/SJC/obs1/status",
payload: []byte(`{"type":"status"}`),
}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var name string
err := store.db.QueryRow("SELECT name FROM observers WHERE id = 'obs1'").Scan(&name)
@@ -640,7 +640,7 @@ func TestHandleMessageWithLowercaseSNRRSSI(t *testing.T) {
payload := []byte(`{"raw":"` + rawHex + `","snr":5.5,"rssi":-102}`)
msg := &mockMessage{topic: "meshcore/SJC/obs1/packets", payload: payload}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var snr, rssi *float64
store.db.QueryRow("SELECT snr, rssi FROM observations LIMIT 1").Scan(&snr, &rssi)
@@ -661,7 +661,7 @@ func TestHandleMessageSNRRSSIUppercaseWins(t *testing.T) {
payload := []byte(`{"raw":"` + rawHex + `","SNR":7.2,"snr":1.0,"RSSI":-95,"rssi":-50}`)
msg := &mockMessage{topic: "meshcore/SJC/obs1/packets", payload: payload}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var snr, rssi *float64
store.db.QueryRow("SELECT snr, rssi FROM observations LIMIT 1").Scan(&snr, &rssi)
@@ -681,7 +681,7 @@ func TestHandleMessageNoSNRRSSI(t *testing.T) {
payload := []byte(`{"raw":"` + rawHex + `"}`)
msg := &mockMessage{topic: "meshcore/SJC/obs1/packets", payload: payload}
handleMessage(store, "test", source, msg, nil, nil)
handleMessage(store, "test", source, msg, nil, &Config{})
var snr, rssi *float64
store.db.QueryRow("SELECT snr, rssi FROM observations LIMIT 1").Scan(&snr, &rssi)
@@ -739,3 +739,44 @@ func TestToFloat64WithUnits(t *testing.T) {
}
}
}
// TestIATAFilterDoesNotDropStatusMessages verifies that status messages from
// out-of-region observers are still processed (noise_floor, battery, etc.)
// even when an IATA filter is configured for packet data.
func TestIATAFilterDoesNotDropStatusMessages(t *testing.T) {
store := newTestStore(t)
source := MQTTSource{Name: "test", IATAFilter: []string{"SJC"}}
// BFL observer sends a status message with noise_floor — outside the IATA filter.
msg := &mockMessage{
topic: "meshcore/BFL/bfl-obs1/status",
payload: []byte(`{"origin":"BFLObserver","stats":{"noise_floor":-105.0}}`),
}
handleMessage(store, "test", source, msg, nil, &Config{})
var name string
var noiseFloor *float64
err := store.db.QueryRow("SELECT name, noise_floor FROM observers WHERE id = 'bfl-obs1'").Scan(&name, &noiseFloor)
if err != nil {
t.Fatalf("observer not found after status from out-of-region observer: %v", err)
}
if name != "BFLObserver" {
t.Errorf("name=%q, want BFLObserver", name)
}
if noiseFloor == nil || *noiseFloor != -105.0 {
t.Errorf("noise_floor=%v, want -105.0 — status message was dropped by IATA filter when it should not be", noiseFloor)
}
// Verify that a packet from BFL is still filtered.
rawHex := "0A00D69FD7A5A7475DB07337749AE61FA53A4788E976"
pktMsg := &mockMessage{
topic: "meshcore/BFL/bfl-obs1/packets",
payload: []byte(`{"raw":"` + rawHex + `"}`),
}
handleMessage(store, "test", source, pktMsg, nil, &Config{})
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
if count != 0 {
t.Error("packet from out-of-region BFL should still be filtered by IATA")
}
}
+339
View File
@@ -0,0 +1,339 @@
package main
import (
"crypto/ed25519"
"encoding/binary"
"encoding/hex"
"strings"
"testing"
)
// buildAdvertHex constructs a full ADVERT packet hex string.
// header(1) + pathByte(1) + pubkey(32) + timestamp(4) + signature(64) + appdata
func buildAdvertHex(pubKey ed25519.PublicKey, privKey ed25519.PrivateKey, timestamp uint32, appdata []byte) string {
// Build signed message: pubkey(32) + timestamp(4 LE) + appdata
msg := make([]byte, 32+4+len(appdata))
copy(msg[0:32], pubKey)
binary.LittleEndian.PutUint32(msg[32:36], timestamp)
copy(msg[36:], appdata)
sig := ed25519.Sign(privKey, msg)
// Payload: pubkey(32) + timestamp(4) + signature(64) + appdata
payload := make([]byte, 0, 100+len(appdata))
payload = append(payload, pubKey...)
ts := make([]byte, 4)
binary.LittleEndian.PutUint32(ts, timestamp)
payload = append(payload, ts...)
payload = append(payload, sig...)
payload = append(payload, appdata...)
// Header: ADVERT (0x04 << 2) | FLOOD (1) = 0x11, pathByte=0 (no hops)
header := byte(0x11)
pathByte := byte(0x00)
pkt := append([]byte{header, pathByte}, payload...)
return hex.EncodeToString(pkt)
}
// makeAppdata builds minimal appdata: flags(1) + name
func makeAppdata(name string) []byte {
flags := byte(0x81) // hasName=true, type=companion(1)
data := []byte{flags}
data = append(data, []byte(name)...)
data = append(data, 0x00) // null terminator
return data
}
func TestSigValidation_ValidAdvertStored(t *testing.T) {
dbPath := t.TempDir() + "/test.db"
store, err := OpenStoreWithInterval(dbPath, 300)
if err != nil {
t.Fatal(err)
}
defer store.Close()
pub, priv, _ := ed25519.GenerateKey(nil)
appdata := makeAppdata("TestNode")
rawHex := buildAdvertHex(pub, priv, 1700000000, appdata)
source := MQTTSource{Name: "test"}
msg := newMockMsg("meshcore/US/obs1/packet", `{"raw":"`+rawHex+`","origin":"TestObs"}`)
cfg := &Config{}
handleMessage(store, "test", source, msg, nil, cfg)
// Verify packet was stored
var count int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&count)
if count == 0 {
t.Fatal("valid advert should be stored, got 0 transmissions")
}
}
func TestSigValidation_TamperedSignatureDropped(t *testing.T) {
dbPath := t.TempDir() + "/test.db"
store, err := OpenStoreWithInterval(dbPath, 300)
if err != nil {
t.Fatal(err)
}
defer store.Close()
pub, priv, _ := ed25519.GenerateKey(nil)
appdata := makeAppdata("BadNode")
rawHex := buildAdvertHex(pub, priv, 1700000000, appdata)
// Tamper with signature (flip a byte in the signature area)
// Signature starts at offset 2 (header+path) + 32 (pubkey) + 4 (timestamp) = 38
// That's byte 38 in the packet, hex chars 76-77
rawBytes := []byte(rawHex)
if rawBytes[76] == '0' {
rawBytes[76] = 'f'
} else {
rawBytes[76] = '0'
}
tamperedHex := string(rawBytes)
source := MQTTSource{Name: "test"}
msg := newMockMsg("meshcore/US/obs1/packet", `{"raw":"`+tamperedHex+`","origin":"TestObs"}`)
cfg := &Config{}
handleMessage(store, "test", source, msg, nil, cfg)
// Verify packet was NOT stored in transmissions
var txCount int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&txCount)
if txCount != 0 {
t.Fatalf("tampered advert should be dropped, got %d transmissions", txCount)
}
// Verify it was recorded in dropped_packets
var dropCount int
store.db.QueryRow("SELECT COUNT(*) FROM dropped_packets").Scan(&dropCount)
if dropCount == 0 {
t.Fatal("tampered advert should be recorded in dropped_packets")
}
// Verify drop counter incremented
if store.Stats.SignatureDrops.Load() != 1 {
t.Fatalf("expected 1 signature drop, got %d", store.Stats.SignatureDrops.Load())
}
// Verify dropped_packets has correct fields
var reason, nodeKey, nodeName, obsID string
store.db.QueryRow("SELECT reason, node_pubkey, node_name, observer_id FROM dropped_packets LIMIT 1").Scan(&reason, &nodeKey, &nodeName, &obsID)
if reason != "invalid signature" {
t.Fatalf("expected reason 'invalid signature', got %q", reason)
}
if nodeKey == "" {
t.Fatal("dropped packet should have node_pubkey")
}
if !strings.Contains(nodeName, "BadNode") {
t.Fatalf("expected node_name to contain 'BadNode', got %q", nodeName)
}
if obsID != "obs1" {
t.Fatalf("expected observer_id 'obs1', got %q", obsID)
}
}
func TestSigValidation_TruncatedAppdataDropped(t *testing.T) {
dbPath := t.TempDir() + "/test.db"
store, err := OpenStoreWithInterval(dbPath, 300)
if err != nil {
t.Fatal(err)
}
defer store.Close()
pub, priv, _ := ed25519.GenerateKey(nil)
appdata := makeAppdata("TruncNode")
rawHex := buildAdvertHex(pub, priv, 1700000000, appdata)
// Sign was computed with full appdata. Now truncate the raw hex to remove
// some appdata bytes, making the signature invalid.
// Truncate last 4 hex chars (2 bytes of appdata)
truncatedHex := rawHex[:len(rawHex)-4]
source := MQTTSource{Name: "test"}
msg := newMockMsg("meshcore/US/obs1/packet", `{"raw":"`+truncatedHex+`","origin":"TestObs"}`)
cfg := &Config{}
handleMessage(store, "test", source, msg, nil, cfg)
var txCount int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&txCount)
if txCount != 0 {
t.Fatalf("truncated advert should be dropped, got %d transmissions", txCount)
}
}
func TestSigValidation_DisabledByConfig(t *testing.T) {
dbPath := t.TempDir() + "/test.db"
store, err := OpenStoreWithInterval(dbPath, 300)
if err != nil {
t.Fatal(err)
}
defer store.Close()
pub, priv, _ := ed25519.GenerateKey(nil)
appdata := makeAppdata("NoValNode")
rawHex := buildAdvertHex(pub, priv, 1700000000, appdata)
// Tamper with signature
rawBytes := []byte(rawHex)
if rawBytes[76] == '0' {
rawBytes[76] = 'f'
} else {
rawBytes[76] = '0'
}
tamperedHex := string(rawBytes)
source := MQTTSource{Name: "test"}
msg := newMockMsg("meshcore/US/obs1/packet", `{"raw":"`+tamperedHex+`","origin":"TestObs"}`)
falseVal := false
cfg := &Config{ValidateSignatures: &falseVal}
handleMessage(store, "test", source, msg, nil, cfg)
// With validation disabled, tampered packet should be stored
var txCount int
store.db.QueryRow("SELECT COUNT(*) FROM transmissions").Scan(&txCount)
if txCount == 0 {
t.Fatal("with validateSignatures=false, tampered advert should be stored")
}
}
func TestSigValidation_DropCounterIncrements(t *testing.T) {
dbPath := t.TempDir() + "/test.db"
store, err := OpenStoreWithInterval(dbPath, 300)
if err != nil {
t.Fatal(err)
}
defer store.Close()
pub, priv, _ := ed25519.GenerateKey(nil)
source := MQTTSource{Name: "test"}
cfg := &Config{}
for i := 0; i < 3; i++ {
appdata := makeAppdata("Node")
rawHex := buildAdvertHex(pub, priv, uint32(1700000000+i), appdata)
// Tamper
rawBytes := []byte(rawHex)
if rawBytes[76] == '0' {
rawBytes[76] = 'f'
} else {
rawBytes[76] = '0'
}
msg := newMockMsg("meshcore/US/obs1/packet", `{"raw":"`+string(rawBytes)+`","origin":"Obs"}`)
handleMessage(store, "test", source, msg, nil, cfg)
}
if store.Stats.SignatureDrops.Load() != 3 {
t.Fatalf("expected 3 signature drops, got %d", store.Stats.SignatureDrops.Load())
}
}
func TestSigValidation_LogContainsFields(t *testing.T) {
// This test verifies the dropped_packets row has all required fields
dbPath := t.TempDir() + "/test.db"
store, err := OpenStoreWithInterval(dbPath, 300)
if err != nil {
t.Fatal(err)
}
defer store.Close()
pub, priv, _ := ed25519.GenerateKey(nil)
appdata := makeAppdata("LogTestNode")
rawHex := buildAdvertHex(pub, priv, 1700000000, appdata)
// Tamper
rawBytes := []byte(rawHex)
if rawBytes[76] == '0' {
rawBytes[76] = 'f'
} else {
rawBytes[76] = '0'
}
source := MQTTSource{Name: "test"}
msg := newMockMsg("meshcore/US/obs1/packet", `{"raw":"`+string(rawBytes)+`","origin":"MyObserver"}`)
cfg := &Config{}
handleMessage(store, "test", source, msg, nil, cfg)
var hash, reason, obsID, obsName, pubkey, nodeName string
err = store.db.QueryRow("SELECT hash, reason, observer_id, observer_name, node_pubkey, node_name FROM dropped_packets LIMIT 1").
Scan(&hash, &reason, &obsID, &obsName, &pubkey, &nodeName)
if err != nil {
t.Fatal(err)
}
if hash == "" {
t.Error("dropped packet should have hash")
}
if reason != "invalid signature" {
t.Errorf("expected reason 'invalid signature', got %q", reason)
}
if obsID != "obs1" {
t.Errorf("expected observer_id 'obs1', got %q", obsID)
}
if obsName != "MyObserver" {
t.Errorf("expected observer_name 'MyObserver', got %q", obsName)
}
if pubkey == "" {
t.Error("dropped packet should have node_pubkey")
}
if !strings.Contains(nodeName, "LogTestNode") {
t.Errorf("expected node_name containing 'LogTestNode', got %q", nodeName)
}
}
func TestPruneDroppedPackets(t *testing.T) {
dbPath := t.TempDir() + "/test.db"
store, err := OpenStoreWithInterval(dbPath, 300)
if err != nil {
t.Fatal(err)
}
defer store.Close()
// Insert an old dropped packet
store.db.Exec(`INSERT INTO dropped_packets (hash, reason, dropped_at) VALUES ('old', 'test', datetime('now', '-60 days'))`)
store.db.Exec(`INSERT INTO dropped_packets (hash, reason, dropped_at) VALUES ('new', 'test', datetime('now'))`)
n, err := store.PruneDroppedPackets(30)
if err != nil {
t.Fatal(err)
}
if n != 1 {
t.Fatalf("expected 1 pruned, got %d", n)
}
var count int
store.db.QueryRow("SELECT COUNT(*) FROM dropped_packets").Scan(&count)
if count != 1 {
t.Fatalf("expected 1 remaining, got %d", count)
}
}
func TestShouldValidateSignatures_Default(t *testing.T) {
cfg := &Config{}
if !cfg.ShouldValidateSignatures() {
t.Fatal("default should be true")
}
falseVal := false
cfg2 := &Config{ValidateSignatures: &falseVal}
if cfg2.ShouldValidateSignatures() {
t.Fatal("explicit false should be false")
}
trueVal := true
cfg3 := &Config{ValidateSignatures: &trueVal}
if !cfg3.ShouldValidateSignatures() {
t.Fatal("explicit true should be true")
}
}
// newMockMsg creates a minimal mqtt.Message for testing.
func newMockMsg(topic, payload string) *mockMessage {
return &mockMessage{topic: topic, payload: []byte(payload)}
}
+181
View File
@@ -0,0 +1,181 @@
package main
import (
"encoding/json"
"fmt"
"testing"
)
// TestAdvertPubkeyTracking verifies that advertPubkeys is maintained
// incrementally during ingest and eviction, and that GetPerfStoreStats
// returns the correct count without per-request JSON parsing.
func TestAdvertPubkeyTracking(t *testing.T) {
ps := NewPacketStore(nil, nil)
ps.mu.Lock()
// Helper to create an ADVERT StoreTx with a given pubkey.
pt4 := 4
mkAdvert := func(id int, pubkey string) *StoreTx {
d := map[string]interface{}{"pubKey": pubkey}
j, _ := json.Marshal(d)
return &StoreTx{
ID: id,
Hash: fmt.Sprintf("hash%d", id),
PayloadType: &pt4,
DecodedJSON: string(j),
}
}
// Add 3 adverts: 2 distinct pubkeys
tx1 := mkAdvert(1, "pk_alpha")
tx2 := mkAdvert(2, "pk_beta")
tx3 := mkAdvert(3, "pk_alpha") // duplicate pubkey
for _, tx := range []*StoreTx{tx1, tx2, tx3} {
ps.packets = append(ps.packets, tx)
ps.byHash[tx.Hash] = tx
ps.byTxID[tx.ID] = tx
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
ps.trackAdvertPubkey(tx)
}
ps.mu.Unlock()
// GetPerfStoreStats should report 2 distinct pubkeys
stats := ps.GetPerfStoreStats()
indexes := stats["indexes"].(map[string]interface{})
got := indexes["advertByObserver"].(int)
if got != 2 {
t.Errorf("advertByObserver = %d, want 2", got)
}
// GetPerfStoreStatsTyped should agree
typed := ps.GetPerfStoreStatsTyped()
if typed.Indexes.AdvertByObserver != 2 {
t.Errorf("typed AdvertByObserver = %d, want 2", typed.Indexes.AdvertByObserver)
}
// Evict tx3 (pk_alpha duplicate) — count should stay 2
ps.mu.Lock()
ps.untrackAdvertPubkey(tx3)
ps.mu.Unlock()
stats2 := ps.GetPerfStoreStats()
idx2 := stats2["indexes"].(map[string]interface{})
if idx2["advertByObserver"].(int) != 2 {
t.Errorf("after evicting duplicate: advertByObserver = %d, want 2", idx2["advertByObserver"].(int))
}
// Evict tx1 (last pk_alpha) — count should drop to 1
ps.mu.Lock()
ps.untrackAdvertPubkey(tx1)
ps.mu.Unlock()
stats3 := ps.GetPerfStoreStats()
idx3 := stats3["indexes"].(map[string]interface{})
if idx3["advertByObserver"].(int) != 1 {
t.Errorf("after evicting last pk_alpha: advertByObserver = %d, want 1", idx3["advertByObserver"].(int))
}
// Evict tx2 (last remaining) — count should be 0
ps.mu.Lock()
ps.untrackAdvertPubkey(tx2)
ps.mu.Unlock()
stats4 := ps.GetPerfStoreStats()
idx4 := stats4["indexes"].(map[string]interface{})
if idx4["advertByObserver"].(int) != 0 {
t.Errorf("after evicting all: advertByObserver = %d, want 0", idx4["advertByObserver"].(int))
}
}
// TestAdvertPubkeyPublicKeyField tests the "public_key" JSON field variant.
func TestAdvertPubkeyPublicKeyField(t *testing.T) {
ps := NewPacketStore(nil, nil)
ps.mu.Lock()
pt4 := 4
d, _ := json.Marshal(map[string]interface{}{"public_key": "pk_legacy"})
tx := &StoreTx{ID: 1, Hash: "h1", PayloadType: &pt4, DecodedJSON: string(d)}
ps.trackAdvertPubkey(tx)
ps.mu.Unlock()
stats := ps.GetPerfStoreStats()
idx := stats["indexes"].(map[string]interface{})
if idx["advertByObserver"].(int) != 1 {
t.Errorf("public_key field: advertByObserver = %d, want 1", idx["advertByObserver"].(int))
}
}
// TestAdvertPubkeyNonAdvert ensures non-ADVERT packets don't affect the count.
func TestAdvertPubkeyNonAdvert(t *testing.T) {
ps := NewPacketStore(nil, nil)
ps.mu.Lock()
pt2 := 2
d, _ := json.Marshal(map[string]interface{}{"pubKey": "pk_text"})
tx := &StoreTx{ID: 1, Hash: "h1", PayloadType: &pt2, DecodedJSON: string(d)}
ps.trackAdvertPubkey(tx)
ps.mu.Unlock()
stats := ps.GetPerfStoreStats()
idx := stats["indexes"].(map[string]interface{})
if idx["advertByObserver"].(int) != 0 {
t.Errorf("non-ADVERT should not be tracked: advertByObserver = %d, want 0", idx["advertByObserver"].(int))
}
}
// BenchmarkGetPerfStoreStats benchmarks the perf stats endpoint with many adverts.
// Before the fix, this did O(N) JSON unmarshals per call.
// After the fix, it's O(1) — just len(map).
func BenchmarkGetPerfStoreStats(b *testing.B) {
ps := NewPacketStore(nil, nil)
ps.mu.Lock()
pt4 := 4
for i := 0; i < 5000; i++ {
pk := fmt.Sprintf("pk_%04d", i%200) // 200 distinct pubkeys
d, _ := json.Marshal(map[string]interface{}{"pubKey": pk})
tx := &StoreTx{
ID: i + 1,
Hash: fmt.Sprintf("hash%d", i+1),
PayloadType: &pt4,
DecodedJSON: string(d),
}
ps.packets = append(ps.packets, tx)
ps.byHash[tx.Hash] = tx
ps.byTxID[tx.ID] = tx
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
ps.trackAdvertPubkey(tx)
}
ps.mu.Unlock()
b.ResetTimer()
for i := 0; i < b.N; i++ {
ps.GetPerfStoreStats()
}
}
// BenchmarkGetPerfStoreStatsTyped benchmarks the typed variant.
func BenchmarkGetPerfStoreStatsTyped(b *testing.B) {
ps := NewPacketStore(nil, nil)
ps.mu.Lock()
pt4 := 4
for i := 0; i < 5000; i++ {
pk := fmt.Sprintf("pk_%04d", i%200)
d, _ := json.Marshal(map[string]interface{}{"pubKey": pk})
tx := &StoreTx{
ID: i + 1,
Hash: fmt.Sprintf("hash%d", i+1),
PayloadType: &pt4,
DecodedJSON: string(d),
}
ps.packets = append(ps.packets, tx)
ps.byHash[tx.Hash] = tx
ps.byTxID[tx.ID] = tx
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
ps.trackAdvertPubkey(tx)
}
ps.mu.Unlock()
b.ResetTimer()
for i := 0; i < b.N; i++ {
ps.GetPerfStoreStatsTyped()
}
}
+111
View File
@@ -0,0 +1,111 @@
package main
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestIsWeakAPIKey(t *testing.T) {
// Known defaults must be detected
for _, weak := range []string{
"your-secret-api-key-here", "change-me", "example", "test",
"password", "admin", "apikey", "api-key", "secret", "default",
} {
if !IsWeakAPIKey(weak) {
t.Errorf("expected %q to be weak", weak)
}
}
// Case-insensitive
if !IsWeakAPIKey("Password") {
t.Error("expected case-insensitive match for Password")
}
if !IsWeakAPIKey("YOUR-SECRET-API-KEY-HERE") {
t.Error("expected case-insensitive match")
}
// Short keys (<16 chars) are weak
if !IsWeakAPIKey("short") {
t.Error("expected short key to be weak")
}
if !IsWeakAPIKey("exactly15chars!") { // 15 chars
t.Error("expected 15-char key to be weak")
}
// Empty key is NOT weak (handled separately as "disabled")
if IsWeakAPIKey("") {
t.Error("empty key should not be flagged as weak")
}
// Strong keys pass
if IsWeakAPIKey("a-very-strong-key-1234") {
t.Error("expected strong key to pass")
}
if IsWeakAPIKey("xK9!mP2@nL5#qR8$") {
t.Error("expected 17-char random key to pass")
}
}
func TestRequireAPIKey_RejectsWeakKey(t *testing.T) {
s := &Server{cfg: &Config{APIKey: "test"}}
handler := s.requireAPIKey(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest("POST", "/api/packets", nil)
req.Header.Set("X-API-Key", "test")
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusForbidden {
t.Errorf("expected 403 for weak key, got %d", rr.Code)
}
}
func TestRequireAPIKey_AcceptsStrongKey(t *testing.T) {
strongKey := "a-very-strong-key-1234"
s := &Server{cfg: &Config{APIKey: strongKey}}
handler := s.requireAPIKey(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest("POST", "/api/packets", nil)
req.Header.Set("X-API-Key", strongKey)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Errorf("expected 200 for strong key, got %d", rr.Code)
}
}
func TestRequireAPIKey_EmptyKeyDisablesEndpoints(t *testing.T) {
s := &Server{cfg: &Config{APIKey: ""}}
handler := s.requireAPIKey(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest("POST", "/api/packets", nil)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusForbidden {
t.Errorf("expected 403 for empty key, got %d", rr.Code)
}
}
func TestRequireAPIKey_WrongKeyUnauthorized(t *testing.T) {
s := &Server{cfg: &Config{APIKey: "a-very-strong-key-1234"}}
handler := s.requireAPIKey(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest("POST", "/api/packets", nil)
req.Header.Set("X-API-Key", "wrong-key-entirely-here")
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusUnauthorized {
t.Errorf("expected 401 for wrong key, got %d", rr.Code)
}
}
+132
View File
@@ -0,0 +1,132 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/gorilla/mux"
)
// TestBackfillAsyncChunked verifies that backfillResolvedPathsAsync processes
// observations in chunks, yields between batches, and sets the completion flag.
func TestBackfillAsyncChunked(t *testing.T) {
store := &PacketStore{
packets: make([]*StoreTx, 0),
byHash: make(map[string]*StoreTx),
byTxID: make(map[int]*StoreTx),
byObsID: make(map[int]*StoreObs),
}
// No pending observations → should complete immediately.
backfillResolvedPathsAsync(store, "", 100, time.Millisecond, 24)
if !store.backfillComplete.Load() {
t.Fatal("expected backfillComplete to be true with empty store")
}
}
// TestBackfillStatusHeader verifies the X-CoreScope-Status header is set correctly.
func TestBackfillStatusHeader(t *testing.T) {
store := &PacketStore{
packets: make([]*StoreTx, 0),
byHash: make(map[string]*StoreTx),
byTxID: make(map[int]*StoreTx),
byObsID: make(map[int]*StoreObs),
}
srv := &Server{store: store}
handler := srv.backfillStatusMiddleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
}))
// Before backfill completes → backfilling
req := httptest.NewRequest("GET", "/api/stats", nil)
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, req)
if got := rec.Header().Get("X-CoreScope-Status"); got != "backfilling" {
t.Fatalf("expected 'backfilling', got %q", got)
}
// After backfill completes → ready
store.backfillComplete.Store(true)
rec = httptest.NewRecorder()
handler.ServeHTTP(rec, req)
if got := rec.Header().Get("X-CoreScope-Status"); got != "ready" {
t.Fatalf("expected 'ready', got %q", got)
}
}
// TestStatsBackfillFields verifies /api/stats includes backfill fields.
func TestStatsBackfillFields(t *testing.T) {
db := setupTestDBv2(t)
defer db.Close()
seedV2Data(t, db)
store := &PacketStore{
db: db,
packets: make([]*StoreTx, 0),
byHash: make(map[string]*StoreTx),
byTxID: make(map[int]*StoreTx),
byObsID: make(map[int]*StoreObs),
loaded: true,
}
cfg := &Config{Port: 0}
hub := NewHub()
srv := NewServer(db, cfg, hub)
srv.store = store
router := mux.NewRouter()
srv.RegisterRoutes(router)
// While backfilling
req := httptest.NewRequest("GET", "/api/stats", nil)
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
var resp map[string]interface{}
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
t.Fatalf("failed to parse stats response: %v", err)
}
if backfilling, ok := resp["backfilling"]; !ok {
t.Fatal("missing 'backfilling' field in stats response")
} else if backfilling != true {
t.Fatalf("expected backfilling=true, got %v", backfilling)
}
if _, ok := resp["backfillProgress"]; !ok {
t.Fatal("missing 'backfillProgress' field in stats response")
}
// Check header
if got := rec.Header().Get("X-CoreScope-Status"); got != "backfilling" {
t.Fatalf("expected X-CoreScope-Status=backfilling, got %q", got)
}
// After backfill completes
store.backfillComplete.Store(true)
// Invalidate stats cache
srv.statsMu.Lock()
srv.statsCache = nil
srv.statsMu.Unlock()
rec = httptest.NewRecorder()
router.ServeHTTP(rec, req)
resp = nil
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
t.Fatalf("failed to parse stats response: %v", err)
}
if backfilling, ok := resp["backfilling"]; !ok || backfilling != false {
t.Fatalf("expected backfilling=false after completion, got %v", backfilling)
}
if got := rec.Header().Get("X-CoreScope-Status"); got != "ready" {
t.Fatalf("expected X-CoreScope-Status=ready, got %q", got)
}
}
+321
View File
@@ -0,0 +1,321 @@
package main
import (
"database/sql"
"fmt"
"os"
"path/filepath"
"testing"
"time"
_ "modernc.org/sqlite"
)
// createTestDB creates a temporary SQLite database with N transmissions (1 obs each).
func createTestDB(t *testing.T, numTx int) string {
t.Helper()
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
createTestDBAt(t, dbPath, numTx)
return dbPath
}
// loadStore creates a PacketStore from a test DB with given maxMemoryMB.
func loadStore(t *testing.T, dbPath string, maxMemMB int) *PacketStore {
t.Helper()
db, err := OpenDB(dbPath)
if err != nil {
t.Fatal(err)
}
cfg := &PacketStoreConfig{MaxMemoryMB: maxMemMB}
store := NewPacketStore(db, cfg)
if err := store.Load(); err != nil {
t.Fatal(err)
}
return store
}
func TestBoundedLoad_LimitedMemory(t *testing.T) {
dbPath := createTestDB(t, 5000)
defer os.RemoveAll(filepath.Dir(dbPath))
// Use 1MB budget — should load far fewer than 5000 packets
store := loadStore(t, dbPath, 1)
defer store.db.conn.Close()
loaded := len(store.packets)
if loaded >= 5000 {
t.Errorf("expected bounded load to limit packets, got %d/5000", loaded)
}
if loaded < 1000 {
t.Errorf("expected at least 1000 packets (minimum), got %d", loaded)
}
t.Logf("Loaded %d/5000 packets with 1MB budget", loaded)
}
func TestBoundedLoad_NewestFirst(t *testing.T) {
dbPath := createTestDB(t, 5000)
defer os.RemoveAll(filepath.Dir(dbPath))
store := loadStore(t, dbPath, 1)
defer store.db.conn.Close()
loaded := len(store.packets)
if loaded >= 5000 {
t.Skip("all packets loaded, can't verify newest-first")
}
// The newest packet in DB has first_seen based on minute 5000.
// The loaded packets should be the newest ones.
// Last packet in store (sorted ASC) should be the newest in DB.
last := store.packets[loaded-1]
base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
newestExpected := base.Add(5000 * time.Minute).Format(time.RFC3339)
if last.FirstSeen != newestExpected {
t.Errorf("expected last packet to be newest (%s), got %s", newestExpected, last.FirstSeen)
}
// First packet should NOT be the oldest in the DB (minute 1)
first := store.packets[0]
oldestAll := base.Add(1 * time.Minute).Format(time.RFC3339)
if first.FirstSeen == oldestAll {
t.Errorf("first loaded packet should not be the absolute oldest when bounded")
}
}
func TestBoundedLoad_OldestLoadedSet(t *testing.T) {
dbPath := createTestDB(t, 5000)
defer os.RemoveAll(filepath.Dir(dbPath))
store := loadStore(t, dbPath, 1)
defer store.db.conn.Close()
if store.oldestLoaded == "" {
t.Fatal("oldestLoaded should be set after bounded load")
}
if len(store.packets) > 0 && store.oldestLoaded != store.packets[0].FirstSeen {
t.Errorf("oldestLoaded (%s) should match first packet (%s)", store.oldestLoaded, store.packets[0].FirstSeen)
}
t.Logf("oldestLoaded = %s", store.oldestLoaded)
}
func TestBoundedLoad_UnlimitedWithZero(t *testing.T) {
dbPath := createTestDB(t, 200)
defer os.RemoveAll(filepath.Dir(dbPath))
store := loadStore(t, dbPath, 0)
defer store.db.conn.Close()
if len(store.packets) != 200 {
t.Errorf("expected all 200 packets with maxMemoryMB=0, got %d", len(store.packets))
}
}
func TestBoundedLoad_AscendingOrder(t *testing.T) {
dbPath := createTestDB(t, 3000)
defer os.RemoveAll(filepath.Dir(dbPath))
store := loadStore(t, dbPath, 1)
defer store.db.conn.Close()
// Verify packets are in ascending first_seen order
for i := 1; i < len(store.packets); i++ {
if store.packets[i].FirstSeen < store.packets[i-1].FirstSeen {
t.Fatalf("packets not in ascending order at index %d: %s < %s",
i, store.packets[i].FirstSeen, store.packets[i-1].FirstSeen)
}
}
}
func TestEstimateStoreTxBytesTypical(t *testing.T) {
est := estimateStoreTxBytesTypical(10)
if est < 1000 {
t.Errorf("typical estimate too low: %d", est)
}
// Should be roughly proportional to observation count
est1 := estimateStoreTxBytesTypical(1)
est20 := estimateStoreTxBytesTypical(20)
if est20 <= est1 {
t.Errorf("estimate should grow with observations: 1obs=%d, 20obs=%d", est1, est20)
}
t.Logf("Typical estimate: 1obs=%d, 10obs=%d, 20obs=%d bytes", est1, est, est20)
}
func BenchmarkLoad_Bounded(b *testing.B) {
dir := b.TempDir()
dbPath := filepath.Join(dir, "bench.db")
createTestDBAt(b, dbPath, 5000)
b.ResetTimer()
for i := 0; i < b.N; i++ {
db, _ := OpenDB(dbPath)
cfg := &PacketStoreConfig{MaxMemoryMB: 1}
store := NewPacketStore(db, cfg)
store.Load()
db.conn.Close()
}
}
func BenchmarkLoad_Unlimited(b *testing.B) {
dir := b.TempDir()
dbPath := filepath.Join(dir, "bench.db")
createTestDBAt(b, dbPath, 5000)
b.ResetTimer()
for i := 0; i < b.N; i++ {
db, _ := OpenDB(dbPath)
cfg := &PacketStoreConfig{MaxMemoryMB: 0}
store := NewPacketStore(db, cfg)
store.Load()
db.conn.Close()
}
}
// BenchmarkLoad_30K_Bounded benchmarks bounded Load() with 30K transmissions
// and realistic observation counts (15 per transmission).
func BenchmarkLoad_30K_Bounded(b *testing.B) {
dir := b.TempDir()
dbPath := filepath.Join(dir, "bench30k.db")
createTestDBWithObs(b, dbPath, 30000)
b.ResetTimer()
for i := 0; i < b.N; i++ {
db, _ := OpenDB(dbPath)
cfg := &PacketStoreConfig{MaxMemoryMB: 50}
store := NewPacketStore(db, cfg)
store.Load()
db.conn.Close()
}
}
// BenchmarkLoad_30K_Unlimited benchmarks unlimited Load() with 30K transmissions
// and realistic observation counts (15 per transmission).
func BenchmarkLoad_30K_Unlimited(b *testing.B) {
dir := b.TempDir()
dbPath := filepath.Join(dir, "bench30k.db")
createTestDBWithObs(b, dbPath, 30000)
b.ResetTimer()
for i := 0; i < b.N; i++ {
db, _ := OpenDB(dbPath)
cfg := &PacketStoreConfig{MaxMemoryMB: 0}
store := NewPacketStore(db, cfg)
store.Load()
db.conn.Close()
}
}
// createTestDBAt is like createTestDB but writes to a specific path.
func createTestDBAt(tb testing.TB, dbPath string, numTx int) {
tb.Helper()
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
if err != nil {
tb.Fatal(err)
}
defer conn.Close()
execOrFail := func(sql string) {
if _, err := conn.Exec(sql); err != nil {
tb.Fatalf("test DB setup exec failed: %v\nSQL: %s", err, sql)
}
}
execOrFail(`CREATE TABLE IF NOT EXISTS transmissions (
id INTEGER PRIMARY KEY,
raw_hex TEXT, hash TEXT, first_seen TEXT,
route_type INTEGER, payload_type INTEGER,
payload_version INTEGER, decoded_json TEXT
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observations (
id INTEGER PRIMARY KEY,
transmission_id INTEGER, observer_id TEXT, observer_name TEXT,
direction TEXT, snr REAL, rssi REAL, score INTEGER,
path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT)`)
execOrFail(`CREATE TABLE IF NOT EXISTS nodes (
pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, first_seen TEXT, frequency REAL
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS schema_version (version INTEGER)`)
execOrFail(`INSERT INTO schema_version (version) VALUES (1)`)
execOrFail(`CREATE INDEX IF NOT EXISTS idx_tx_first_seen ON transmissions(first_seen)`)
txStmt, err := conn.Prepare("INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type, payload_version, decoded_json) VALUES (?, ?, ?, ?, ?, ?, ?, ?)")
if err != nil {
tb.Fatalf("test DB prepare transmissions insert: %v", err)
}
obsStmt, err := conn.Prepare("INSERT INTO observations (id, transmission_id, observer_id, observer_name, direction, snr, rssi, score, path_json, timestamp) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
if err != nil {
tb.Fatalf("test DB prepare observations insert: %v", err)
}
defer txStmt.Close()
defer obsStmt.Close()
base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
for i := 1; i <= numTx; i++ {
ts := base.Add(time.Duration(i) * time.Minute).Format(time.RFC3339)
hash := fmt.Sprintf("h%04d", i)
txStmt.Exec(i, "aabb", hash, ts, 0, 4, 1, fmt.Sprintf(`{"pubKey":"pk%04d"}`, i))
obsStmt.Exec(i, i, "obs1", "Obs1", "RX", -10.0, -80.0, 5, `["aa","bb"]`, ts)
}
}
// createTestDBWithObs creates a test DB with realistic observation counts (15 per tx).
func createTestDBWithObs(tb testing.TB, dbPath string, numTx int) {
tb.Helper()
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
if err != nil {
tb.Fatal(err)
}
defer conn.Close()
execOrFail := func(sqlStr string) {
if _, err := conn.Exec(sqlStr); err != nil {
tb.Fatalf("test DB setup exec failed: %v\nSQL: %s", err, sqlStr)
}
}
execOrFail(`CREATE TABLE IF NOT EXISTS transmissions (
id INTEGER PRIMARY KEY, raw_hex TEXT, hash TEXT, first_seen TEXT,
route_type INTEGER, payload_type INTEGER, payload_version INTEGER, decoded_json TEXT
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observations (
id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_id TEXT, observer_name TEXT,
direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT)`)
execOrFail(`CREATE TABLE IF NOT EXISTS nodes (
pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL,
last_seen TEXT, first_seen TEXT, frequency REAL
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS schema_version (version INTEGER)`)
execOrFail(`INSERT INTO schema_version (version) VALUES (1)`)
execOrFail(`CREATE INDEX IF NOT EXISTS idx_tx_first_seen ON transmissions(first_seen)`)
txStmt, err := conn.Prepare("INSERT INTO transmissions (id, raw_hex, hash, first_seen, route_type, payload_type, payload_version, decoded_json) VALUES (?, ?, ?, ?, ?, ?, ?, ?)")
if err != nil {
tb.Fatalf("test DB prepare transmissions: %v", err)
}
obsStmt, err := conn.Prepare("INSERT INTO observations (id, transmission_id, observer_id, observer_name, direction, snr, rssi, score, path_json, timestamp) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
if err != nil {
tb.Fatalf("test DB prepare observations: %v", err)
}
defer txStmt.Close()
defer obsStmt.Close()
observers := []string{"obs1", "obs2", "obs3", "obs4", "obs5"}
obsNames := []string{"Alpha", "Bravo", "Charlie", "Delta", "Echo"}
obsID := 1
base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
for i := 1; i <= numTx; i++ {
ts := base.Add(time.Duration(i) * time.Minute).Format(time.RFC3339)
hash := fmt.Sprintf("h%06d", i)
txStmt.Exec(i, "aabb", hash, ts, 0, 4, 1, fmt.Sprintf(`{"pubKey":"pk%06d"}`, i))
nObs := (i % 5) + 1 // 15 observations per transmission
for j := 0; j < nObs; j++ {
snr := -5.0 + float64(j)*2.5
rssi := -90.0 + float64(j)*5.0
obsStmt.Exec(obsID, i, observers[j], obsNames[j], "RX", snr, rssi, 5-j, `["aa","bb"]`, ts)
obsID++
}
}
}
+357 -14
View File
@@ -9,13 +9,15 @@ import (
func newTestStore(t *testing.T) *PacketStore {
t.Helper()
return &PacketStore{
rfCache: make(map[string]*cachedResult),
topoCache: make(map[string]*cachedResult),
hashCache: make(map[string]*cachedResult),
chanCache: make(map[string]*cachedResult),
distCache: make(map[string]*cachedResult),
subpathCache: make(map[string]*cachedResult),
rfCacheTTL: 15 * time.Second,
rfCache: make(map[string]*cachedResult),
topoCache: make(map[string]*cachedResult),
hashCache: make(map[string]*cachedResult),
collisionCache: make(map[string]*cachedResult),
chanCache: make(map[string]*cachedResult),
distCache: make(map[string]*cachedResult),
subpathCache: make(map[string]*cachedResult),
rfCacheTTL: 15 * time.Second,
invCooldown: 10 * time.Second,
}
}
@@ -28,6 +30,7 @@ func populateAllCaches(s *PacketStore) {
s.rfCache["global"] = dummy
s.topoCache["global"] = dummy
s.hashCache["global"] = dummy
s.collisionCache["global"] = dummy
s.chanCache["global"] = dummy
s.distCache["global"] = dummy
s.subpathCache["global"] = dummy
@@ -38,12 +41,13 @@ func cachePopulated(s *PacketStore) map[string]bool {
s.cacheMu.Lock()
defer s.cacheMu.Unlock()
return map[string]bool{
"rf": len(s.rfCache) > 0,
"topo": len(s.topoCache) > 0,
"hash": len(s.hashCache) > 0,
"chan": len(s.chanCache) > 0,
"dist": len(s.distCache) > 0,
"subpath": len(s.subpathCache) > 0,
"rf": len(s.rfCache) > 0,
"topo": len(s.topoCache) > 0,
"hash": len(s.hashCache) > 0,
"collision": len(s.collisionCache) > 0,
"chan": len(s.chanCache) > 0,
"dist": len(s.distCache) > 0,
"subpath": len(s.subpathCache) > 0,
}
}
@@ -89,7 +93,8 @@ func TestInvalidateCachesFor_NewTransmissionsOnly(t *testing.T) {
if pop["hash"] {
t.Error("hash cache should be cleared on new transmissions")
}
for _, name := range []string{"rf", "topo", "chan", "dist", "subpath"} {
// collisionCache should NOT be cleared by transmissions alone (only by hasNewNodes)
for _, name := range []string{"rf", "topo", "collision", "chan", "dist", "subpath"} {
if !pop[name] {
t.Errorf("%s cache should NOT be cleared on transmission-only ingest", name)
}
@@ -169,3 +174,341 @@ func TestInvalidateCachesFor_NoFlags(t *testing.T) {
}
}
}
// TestInvalidationRateLimited verifies that rapid ingest cycles don't clear
// caches immediately — they accumulate dirty flags during the cooldown period
// and apply them on the next call after cooldown expires (fixes #533).
func TestInvalidationRateLimited(t *testing.T) {
s := newTestStore(t)
s.invCooldown = 100 * time.Millisecond // short cooldown for testing
// First invalidation should go through immediately
populateAllCaches(s)
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
state := cachePopulated(s)
if state["rf"] {
t.Error("rf cache should be cleared on first invalidation")
}
if !state["topo"] {
t.Error("topo cache should survive (no path changes)")
}
// Repopulate and call again within cooldown — should NOT clear
populateAllCaches(s)
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
state = cachePopulated(s)
if !state["rf"] {
t.Error("rf cache should survive during cooldown period")
}
// Wait for cooldown to expire
time.Sleep(150 * time.Millisecond)
// Next call should apply accumulated + current flags
populateAllCaches(s)
s.invalidateCachesFor(cacheInvalidation{hasNewPaths: true})
state = cachePopulated(s)
if state["rf"] {
t.Error("rf cache should be cleared (pending from cooldown)")
}
if state["topo"] {
t.Error("topo cache should be cleared (current call has hasNewPaths)")
}
if !state["hash"] {
t.Error("hash cache should survive (no transmission changes)")
}
}
// TestInvalidationCooldownAccumulatesFlags verifies that multiple calls during
// cooldown merge their flags correctly.
func TestInvalidationCooldownAccumulatesFlags(t *testing.T) {
s := newTestStore(t)
s.invCooldown = 200 * time.Millisecond
// Initial invalidation (goes through, starts cooldown)
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
// Several calls during cooldown with different flags
s.invalidateCachesFor(cacheInvalidation{hasNewPaths: true})
s.invalidateCachesFor(cacheInvalidation{hasNewTransmissions: true})
s.invalidateCachesFor(cacheInvalidation{hasChannelData: true})
// Verify pending has all flags
s.cacheMu.Lock()
if s.pendingInv == nil {
t.Fatal("pendingInv should not be nil during cooldown")
}
if !s.pendingInv.hasNewPaths || !s.pendingInv.hasNewTransmissions || !s.pendingInv.hasChannelData {
t.Error("all flags should be accumulated in pendingInv")
}
// hasNewObservations was applied immediately, not accumulated
if s.pendingInv.hasNewObservations {
t.Error("hasNewObservations was already applied, should not be in pending")
}
s.cacheMu.Unlock()
// Wait for cooldown, then trigger — all accumulated flags should apply
time.Sleep(250 * time.Millisecond)
populateAllCaches(s)
s.invalidateCachesFor(cacheInvalidation{}) // empty trigger
state := cachePopulated(s)
// Pending had paths, transmissions, channels — all those caches should clear
if state["topo"] {
t.Error("topo should be cleared (pending hasNewPaths)")
}
if state["hash"] {
t.Error("hash should be cleared (pending hasNewTransmissions)")
}
if state["chan"] {
t.Error("chan should be cleared (pending hasChannelData)")
}
}
// TestEvictionBypassesCooldown verifies eviction always clears immediately.
func TestEvictionBypassesCooldown(t *testing.T) {
s := newTestStore(t)
s.invCooldown = 10 * time.Second // long cooldown
// Start cooldown
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
// Eviction during cooldown should still clear everything
populateAllCaches(s)
s.invalidateCachesFor(cacheInvalidation{eviction: true})
state := cachePopulated(s)
for name, has := range state {
if has {
t.Errorf("%s cache should be cleared on eviction even during cooldown", name)
}
}
// pendingInv should be cleared
s.cacheMu.Lock()
if s.pendingInv != nil {
t.Error("pendingInv should be nil after eviction")
}
s.cacheMu.Unlock()
}
// BenchmarkCacheHitDuringIngestion simulates rapid ingestion and verifies
// that cache hits now occur thanks to rate-limited invalidation.
func BenchmarkCacheHitDuringIngestion(b *testing.B) {
s := &PacketStore{
rfCache: make(map[string]*cachedResult),
topoCache: make(map[string]*cachedResult),
hashCache: make(map[string]*cachedResult),
chanCache: make(map[string]*cachedResult),
distCache: make(map[string]*cachedResult),
subpathCache: make(map[string]*cachedResult),
rfCacheTTL: 15 * time.Second,
invCooldown: 50 * time.Millisecond,
}
// Trigger first invalidation to start cooldown timer
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
var hits, misses int64
for i := 0; i < b.N; i++ {
// Populate cache (simulates an analytics query filling the cache)
s.cacheMu.Lock()
s.rfCache["global"] = &cachedResult{
data: map[string]interface{}{"test": true},
expiresAt: time.Now().Add(time.Hour),
}
s.cacheMu.Unlock()
// Simulate rapid ingest invalidation (should be rate-limited)
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
// Check if cache survived the invalidation
s.cacheMu.Lock()
if len(s.rfCache) > 0 {
hits++
} else {
misses++
}
s.cacheMu.Unlock()
}
if hits == 0 {
b.Errorf("expected cache hits > 0 with rate-limited invalidation, got 0 hits / %d misses", misses)
}
b.ReportMetric(float64(hits)/float64(hits+misses)*100, "hit%")
}
// TestInvCooldownFromConfig verifies that invalidationDebounce from config
// is wired to invCooldown on PacketStore.
func TestInvCooldownFromConfig(t *testing.T) {
// Default without config
ps := NewPacketStore(nil, nil)
if ps.invCooldown != 300*time.Second {
t.Errorf("default invCooldown = %v, want 300s", ps.invCooldown)
}
// With config override
ct := map[string]interface{}{"invalidationDebounce": float64(60)}
ps2 := NewPacketStore(nil, nil, ct)
if ps2.invCooldown != 60*time.Second {
t.Errorf("configured invCooldown = %v, want 60s", ps2.invCooldown)
}
}
// TestCollisionCacheNotClearedByTransmissions verifies that collisionCache
// is only cleared by hasNewNodes, not hasNewTransmissions (fixes #720).
func TestCollisionCacheNotClearedByTransmissions(t *testing.T) {
s := newTestStore(t)
populateAllCaches(s)
s.invalidateCachesFor(cacheInvalidation{hasNewTransmissions: true})
pop := cachePopulated(s)
if !pop["collision"] {
t.Error("collisionCache should NOT be cleared by hasNewTransmissions alone")
}
if pop["hash"] {
t.Error("hashCache should be cleared by hasNewTransmissions")
}
}
// TestCollisionCacheClearedByNewNodes verifies that collisionCache IS cleared
// when genuinely new nodes are discovered.
func TestCollisionCacheClearedByNewNodes(t *testing.T) {
s := newTestStore(t)
populateAllCaches(s)
s.invalidateCachesFor(cacheInvalidation{hasNewNodes: true})
pop := cachePopulated(s)
if pop["collision"] {
t.Error("collisionCache should be cleared by hasNewNodes")
}
// Other caches should survive
for _, name := range []string{"rf", "topo", "hash", "chan", "dist", "subpath"} {
if !pop[name] {
t.Errorf("%s cache should NOT be cleared on new-nodes-only ingest", name)
}
}
}
// TestCacheSurvivesMultipleIngestCyclesWithinCooldown verifies that caches
// survive repeated ingest cycles during the cooldown period.
func TestCacheSurvivesMultipleIngestCyclesWithinCooldown(t *testing.T) {
s := newTestStore(t)
s.invCooldown = 200 * time.Millisecond
// First invalidation goes through (starts cooldown)
populateAllCaches(s)
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
pop := cachePopulated(s)
if pop["rf"] {
t.Error("rf should be cleared on first invalidation")
}
// Repopulate and simulate 5 rapid ingest cycles
populateAllCaches(s)
for i := 0; i < 5; i++ {
s.invalidateCachesFor(cacheInvalidation{
hasNewObservations: true,
hasNewTransmissions: true,
hasNewPaths: true,
})
}
// All caches should survive during cooldown
pop = cachePopulated(s)
for name, has := range pop {
if !has {
t.Errorf("%s cache should survive during cooldown period (ingest cycle %d)", name, 5)
}
}
}
// TestNewNodesAccumulatedDuringCooldown verifies that hasNewNodes flags
// accumulated during cooldown are applied when cooldown expires.
func TestNewNodesAccumulatedDuringCooldown(t *testing.T) {
s := newTestStore(t)
s.invCooldown = 100 * time.Millisecond
// First call starts cooldown
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
// During cooldown, accumulate hasNewNodes
s.invalidateCachesFor(cacheInvalidation{hasNewNodes: true})
// Verify accumulated
s.cacheMu.Lock()
if s.pendingInv == nil || !s.pendingInv.hasNewNodes {
t.Error("hasNewNodes should be accumulated in pendingInv")
}
s.cacheMu.Unlock()
// Wait for cooldown
time.Sleep(150 * time.Millisecond)
// Trigger flush
populateAllCaches(s)
s.invalidateCachesFor(cacheInvalidation{})
pop := cachePopulated(s)
if pop["collision"] {
t.Error("collisionCache should be cleared after pending hasNewNodes is flushed")
}
}
// BenchmarkAnalyticsLatencyCacheHitVsMiss benchmarks cache hit vs miss
// for analytics endpoints to demonstrate the performance impact.
func BenchmarkAnalyticsLatencyCacheHitVsMiss(b *testing.B) {
s := &PacketStore{
rfCache: make(map[string]*cachedResult),
topoCache: make(map[string]*cachedResult),
hashCache: make(map[string]*cachedResult),
collisionCache: make(map[string]*cachedResult),
chanCache: make(map[string]*cachedResult),
distCache: make(map[string]*cachedResult),
subpathCache: make(map[string]*cachedResult),
rfCacheTTL: 1800 * time.Second,
invCooldown: 300 * time.Second,
}
// Pre-populate cache
s.cacheMu.Lock()
s.rfCache["global"] = &cachedResult{
data: map[string]interface{}{"bins": make([]int, 100)},
expiresAt: time.Now().Add(time.Hour),
}
s.cacheMu.Unlock()
// Trigger initial invalidation to start cooldown
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
var hits, misses int64
for i := 0; i < b.N; i++ {
// Re-populate (simulates query filling cache)
s.cacheMu.Lock()
if len(s.rfCache) == 0 {
s.rfCache["global"] = &cachedResult{
data: map[string]interface{}{"bins": make([]int, 100)},
expiresAt: time.Now().Add(time.Hour),
}
}
s.cacheMu.Unlock()
// Simulate ingest (rate-limited)
s.invalidateCachesFor(cacheInvalidation{hasNewObservations: true})
// Check hit
s.cacheMu.Lock()
if len(s.rfCache) > 0 {
hits++
} else {
misses++
}
s.cacheMu.Unlock()
}
hitRate := float64(hits) / float64(hits+misses) * 100
b.ReportMetric(hitRate, "hit%")
if hitRate < 50 {
b.Errorf("hit rate %.1f%% is below 50%% target", hitRate)
}
}
+57
View File
@@ -0,0 +1,57 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
)
// TestPacketsChannelFilter verifies /api/packets?channel=... actually filters
// (regression test for #812).
func TestPacketsChannelFilter(t *testing.T) {
_, router := setupTestServer(t)
get := func(url string) map[string]interface{} {
req := httptest.NewRequest("GET", url, nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("GET %s: expected 200, got %d", url, w.Code)
}
var body map[string]interface{}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatalf("decode %s: %v", url, err)
}
return body
}
all := get("/api/packets?limit=50")
allTotal := int(all["total"].(float64))
if allTotal < 2 {
t.Fatalf("expected baseline >= 2 packets, got %d", allTotal)
}
test := get("/api/packets?limit=50&channel=%23test")
testTotal := int(test["total"].(float64))
if testTotal == 0 {
t.Fatalf("channel=#test: expected >= 1 match, got 0 (filter ignored?)")
}
if testTotal >= allTotal {
t.Fatalf("channel=#test: expected fewer packets than baseline (%d), got %d", allTotal, testTotal)
}
// Every returned packet must be a CHAN/GRP_TXT (payload_type=5) on #test.
pkts, _ := test["packets"].([]interface{})
for _, p := range pkts {
m := p.(map[string]interface{})
if pt, _ := m["payload_type"].(float64); int(pt) != 5 {
t.Errorf("channel=#test: returned non-GRP_TXT packet (payload_type=%v)", m["payload_type"])
}
}
none := get("/api/packets?limit=50&channel=nonexistentchannel")
if int(none["total"].(float64)) != 0 {
t.Fatalf("channel=nonexistentchannel: expected total=0, got %v", none["total"])
}
}
+748
View File
@@ -0,0 +1,748 @@
package main
import (
"math"
"sort"
"sync"
"time"
)
// ── Clock Skew Severity ────────────────────────────────────────────────────────
type SkewSeverity string
const (
SkewOK SkewSeverity = "ok" // < 5 min
SkewWarning SkewSeverity = "warning" // 5 min 1 hour
SkewCritical SkewSeverity = "critical" // 1 hour 30 days
SkewAbsurd SkewSeverity = "absurd" // > 30 days
SkewNoClock SkewSeverity = "no_clock" // > 365 days — uninitialized RTC
SkewBimodalClock SkewSeverity = "bimodal_clock" // mixed good+bad recent samples (flaky RTC)
)
// Default thresholds in seconds.
const (
skewThresholdWarnSec = 5 * 60 // 5 minutes
skewThresholdCriticalSec = 60 * 60 // 1 hour
skewThresholdAbsurdSec = 30 * 24 * 3600 // 30 days
skewThresholdNoClockSec = 365 * 24 * 3600 // 365 days — uninitialized RTC
// minDriftSamples is the minimum number of advert transmissions needed
// to compute a meaningful linear drift rate.
minDriftSamples = 5
// maxReasonableDriftPerDay caps drift display. Physically impossible
// drift rates (> 1 day/day) indicate insufficient or outlier samples.
maxReasonableDriftPerDay = 86400.0
// recentSkewWindowCount is the number of most-recent advert samples
// used to derive the "current" skew for severity classification (see
// issue #789). The all-time median is poisoned by historical bad
// samples (e.g. a node that was off and then GPS-corrected); severity
// must reflect current health, not lifetime statistics.
recentSkewWindowCount = 5
// recentSkewWindowSec bounds the recent-window in time as well: only
// samples from the last N seconds count as "recent" for severity.
// The effective window is min(recentSkewWindowCount, samples in 1h).
recentSkewWindowSec = 3600
// bimodalSkewThresholdSec is the absolute skew threshold (1 hour)
// above which a sample is considered "bad" — likely firmware emitting
// a nonsense timestamp from an uninitialized RTC, not real drift.
// Chosen to match the warning/critical severity boundary: real clock
// drift rarely exceeds 1 hour, while epoch-0 RTCs produce ~1.7B sec.
bimodalSkewThresholdSec = 3600.0
// maxPlausibleSkewJumpSec is the largest skew change between
// consecutive samples that we treat as physical drift. Anything larger
// (e.g. a GPS sync that jumps the clock by minutes/days) is rejected
// as an outlier when computing drift. Real microcontroller drift is
// fractions of a second per advert; 60s is a generous safety factor.
maxPlausibleSkewJumpSec = 60.0
// theilSenMaxPoints caps the number of points fed to Theil-Sen
// regression (O(n²) in pairs). For nodes with thousands of samples we
// keep the most-recent points, which are also the most relevant for
// current drift.
theilSenMaxPoints = 200
)
// classifySkew maps absolute skew (seconds) to a severity level.
// Float64 comparison is safe: inputs are rounded to 1 decimal via round(),
// and thresholds are integer multiples of 60 — no rounding artifacts.
func classifySkew(absSkewSec float64) SkewSeverity {
switch {
case absSkewSec >= skewThresholdNoClockSec:
return SkewNoClock
case absSkewSec >= skewThresholdAbsurdSec:
return SkewAbsurd
case absSkewSec >= skewThresholdCriticalSec:
return SkewCritical
case absSkewSec >= skewThresholdWarnSec:
return SkewWarning
default:
return SkewOK
}
}
// ── Data Types ─────────────────────────────────────────────────────────────────
// skewSample is a single raw skew measurement from one advert observation.
type skewSample struct {
advertTS int64 // node's advert Unix timestamp
observedTS int64 // observation Unix timestamp
observerID string // which observer saw this
hash string // transmission hash (for multi-observer grouping)
}
// ObserverCalibration holds the computed clock offset for an observer.
type ObserverCalibration struct {
ObserverID string `json:"observerID"`
OffsetSec float64 `json:"offsetSec"` // positive = observer clock ahead
Samples int `json:"samples"` // number of multi-observer packets used
}
// NodeClockSkew is the API response for a single node's clock skew data.
type NodeClockSkew struct {
Pubkey string `json:"pubkey"`
MeanSkewSec float64 `json:"meanSkewSec"` // corrected mean skew (positive = node ahead)
MedianSkewSec float64 `json:"medianSkewSec"` // corrected median skew
LastSkewSec float64 `json:"lastSkewSec"` // most recent corrected skew
RecentMedianSkewSec float64 `json:"recentMedianSkewSec"` // median across most-recent samples (drives severity, see #789)
DriftPerDaySec float64 `json:"driftPerDaySec"` // linear drift rate (sec/day)
Severity SkewSeverity `json:"severity"`
SampleCount int `json:"sampleCount"`
Calibrated bool `json:"calibrated"` // true if observer calibration was applied
LastAdvertTS int64 `json:"lastAdvertTS"` // most recent advert timestamp
LastObservedTS int64 `json:"lastObservedTS"` // most recent observation timestamp
Samples []SkewSample `json:"samples,omitempty"` // time-series for sparklines
GoodFraction float64 `json:"goodFraction"` // fraction of recent samples with |skew| <= 1h
RecentBadSampleCount int `json:"recentBadSampleCount"` // count of recent samples with |skew| > 1h
RecentSampleCount int `json:"recentSampleCount"` // total recent samples in window
NodeName string `json:"nodeName,omitempty"` // populated in fleet responses
NodeRole string `json:"nodeRole,omitempty"` // populated in fleet responses
}
// SkewSample is a single (timestamp, skew) point for sparkline rendering.
type SkewSample struct {
Timestamp int64 `json:"ts"` // Unix epoch of observation
SkewSec float64 `json:"skew"` // corrected skew in seconds
}
// txSkewResult maps tx hash → per-transmission skew stats. This is an
// intermediate result keyed by hash (not pubkey); the store maps hash → pubkey
// when building the final per-node view.
type txSkewResult = map[string]*NodeClockSkew
// ── Clock Skew Engine ──────────────────────────────────────────────────────────
// ClockSkewEngine computes and caches clock skew data for nodes and observers.
type ClockSkewEngine struct {
mu sync.RWMutex
observerOffsets map[string]float64 // observerID → calibrated offset (seconds)
observerSamples map[string]int // observerID → number of multi-observer packets used
nodeSkew txSkewResult
lastComputed time.Time
computeInterval time.Duration
}
func NewClockSkewEngine() *ClockSkewEngine {
return &ClockSkewEngine{
observerOffsets: make(map[string]float64),
observerSamples: make(map[string]int),
nodeSkew: make(txSkewResult),
computeInterval: 30 * time.Second,
}
}
// Recompute recalculates all clock skew data from the packet store.
// Called periodically or on demand. Holds store RLock externally.
// Uses read-copy-update: heavy computation runs outside the write lock,
// then results are swapped in under a brief lock.
func (e *ClockSkewEngine) Recompute(store *PacketStore) {
// Fast path: check under read lock if recompute is needed.
e.mu.RLock()
fresh := time.Since(e.lastComputed) < e.computeInterval
e.mu.RUnlock()
if fresh {
return
}
// Phase 1: Collect skew samples from ADVERT packets (store RLock held by caller).
samples := collectSamples(store)
// Phase 23: Compute outside the write lock.
var newOffsets map[string]float64
var newSamples map[string]int
var newNodeSkew txSkewResult
if len(samples) > 0 {
newOffsets, newSamples = calibrateObservers(samples)
newNodeSkew = computeNodeSkew(samples, newOffsets)
} else {
newOffsets = make(map[string]float64)
newSamples = make(map[string]int)
newNodeSkew = make(txSkewResult)
}
// Swap results under brief write lock.
e.mu.Lock()
// Re-check: another goroutine may have computed while we were working.
if time.Since(e.lastComputed) < e.computeInterval {
e.mu.Unlock()
return
}
e.observerOffsets = newOffsets
e.observerSamples = newSamples
e.nodeSkew = newNodeSkew
e.lastComputed = time.Now()
e.mu.Unlock()
}
// collectSamples extracts skew samples from ADVERT packets in the store.
// Must be called with store.mu held (at least RLock).
func collectSamples(store *PacketStore) []skewSample {
adverts := store.byPayloadType[PayloadADVERT]
if len(adverts) == 0 {
return nil
}
samples := make([]skewSample, 0, len(adverts)*2)
for _, tx := range adverts {
decoded := tx.ParsedDecoded()
if decoded == nil {
continue
}
// Extract advert timestamp from decoded JSON.
advertTS := extractTimestamp(decoded)
if advertTS <= 0 {
continue
}
// Sanity: skip timestamps before year 2020 or after year 2100.
if advertTS < 1577836800 || advertTS > 4102444800 {
continue
}
for _, obs := range tx.Observations {
obsTS := parseISO(obs.Timestamp)
if obsTS <= 0 {
continue
}
samples = append(samples, skewSample{
advertTS: advertTS,
observedTS: obsTS,
observerID: obs.ObserverID,
hash: tx.Hash,
})
}
}
return samples
}
// extractTimestamp gets the Unix timestamp from a decoded ADVERT payload.
func extractTimestamp(decoded map[string]interface{}) int64 {
// Try payload.timestamp first (nested in "payload" key).
if payload, ok := decoded["payload"]; ok {
if pm, ok := payload.(map[string]interface{}); ok {
if ts := jsonNumber(pm, "timestamp"); ts > 0 {
return ts
}
}
}
// Fallback: top-level timestamp.
if ts := jsonNumber(decoded, "timestamp"); ts > 0 {
return ts
}
return 0
}
// jsonNumber extracts an int64 from a JSON-parsed map (handles float64 and json.Number).
func jsonNumber(m map[string]interface{}, key string) int64 {
v, ok := m[key]
if !ok || v == nil {
return 0
}
switch n := v.(type) {
case float64:
return int64(n)
case int64:
return n
case int:
return int64(n)
}
return 0
}
// parseISO parses an ISO 8601 timestamp string to Unix seconds.
func parseISO(s string) int64 {
if s == "" {
return 0
}
t, err := time.Parse(time.RFC3339, s)
if err != nil {
// Try with fractional seconds.
t, err = time.Parse("2006-01-02T15:04:05.999999999Z07:00", s)
if err != nil {
return 0
}
}
return t.Unix()
}
// ── Phase 2: Observer Calibration ──────────────────────────────────────────────
// calibrateObservers computes each observer's clock offset using multi-observer
// packets. Returns offset map and sample count map.
func calibrateObservers(samples []skewSample) (map[string]float64, map[string]int) {
// Group observations by packet hash.
byHash := make(map[string][]skewSample)
for _, s := range samples {
byHash[s.hash] = append(byHash[s.hash], s)
}
// For each multi-observer packet, compute per-observer deviation from median.
deviations := make(map[string][]float64) // observerID → list of deviations
for _, group := range byHash {
if len(group) < 2 {
continue // single-observer packet, can't calibrate
}
// Compute median observation timestamp for this packet.
obsTimes := make([]float64, len(group))
for i, s := range group {
obsTimes[i] = float64(s.observedTS)
}
medianObs := median(obsTimes)
for _, s := range group {
dev := float64(s.observedTS) - medianObs
deviations[s.observerID] = append(deviations[s.observerID], dev)
}
}
// Each observer's offset = median of its deviations.
offsets := make(map[string]float64, len(deviations))
counts := make(map[string]int, len(deviations))
for obsID, devs := range deviations {
offsets[obsID] = median(devs)
counts[obsID] = len(devs)
}
return offsets, counts
}
// ── Phase 3: Per-Node Skew ─────────────────────────────────────────────────────
// computeNodeSkew calculates corrected skew statistics for each node.
func computeNodeSkew(samples []skewSample, obsOffsets map[string]float64) txSkewResult {
// Compute corrected skew per sample, grouped by hash (each hash = one
// node's advert transmission). The caller maps hash → pubkey via byNode.
type correctedSample struct {
skew float64
observedTS int64
calibrated bool
}
byHash := make(map[string][]correctedSample)
hashAdvertTS := make(map[string]int64)
for _, s := range samples {
obsOffset, hasCal := obsOffsets[s.observerID]
rawSkew := float64(s.advertTS - s.observedTS)
corrected := rawSkew
if hasCal {
// Observer offset = obs_ts - median(all_obs_ts). If observer is ahead,
// its obs_ts is inflated, making raw_skew too low. Add offset to correct.
corrected = rawSkew + obsOffset
}
byHash[s.hash] = append(byHash[s.hash], correctedSample{
skew: corrected,
observedTS: s.observedTS,
calibrated: hasCal,
})
hashAdvertTS[s.hash] = s.advertTS
}
// Each hash represents one advert from one node. Compute median corrected
// skew per hash (across multiple observers).
result := make(map[string]*NodeClockSkew) // keyed by hash for now
for hash, cs := range byHash {
skews := make([]float64, len(cs))
for i, c := range cs {
skews[i] = c.skew
}
medSkew := median(skews)
meanSkew := mean(skews)
// Find latest observation.
var latestObsTS int64
var anyCal bool
for _, c := range cs {
if c.observedTS > latestObsTS {
latestObsTS = c.observedTS
}
if c.calibrated {
anyCal = true
}
}
absMedian := math.Abs(medSkew)
result[hash] = &NodeClockSkew{
MeanSkewSec: round(meanSkew, 1),
MedianSkewSec: round(medSkew, 1),
LastSkewSec: round(cs[len(cs)-1].skew, 1),
Severity: classifySkew(absMedian),
SampleCount: len(cs),
Calibrated: anyCal,
LastAdvertTS: hashAdvertTS[hash],
LastObservedTS: latestObsTS,
}
}
return result
}
// ── Integration with PacketStore ───────────────────────────────────────────────
// GetNodeClockSkew returns the clock skew data for a specific node (acquires RLock).
func (s *PacketStore) GetNodeClockSkew(pubkey string) *NodeClockSkew {
s.mu.RLock()
defer s.mu.RUnlock()
return s.getNodeClockSkewLocked(pubkey)
}
// getNodeClockSkewLocked returns clock skew for a node.
// Must be called with s.mu held (at least RLock).
func (s *PacketStore) getNodeClockSkewLocked(pubkey string) *NodeClockSkew {
s.clockSkew.Recompute(s)
txs := s.byNode[pubkey]
if len(txs) == 0 {
return nil
}
s.clockSkew.mu.RLock()
defer s.clockSkew.mu.RUnlock()
var allSkews []float64
var lastSkew float64
var lastObsTS, lastAdvTS int64
var totalSamples int
var anyCal bool
var tsSkews []tsSkewPair
for _, tx := range txs {
if tx.PayloadType == nil || *tx.PayloadType != PayloadADVERT {
continue
}
cs, ok := s.clockSkew.nodeSkew[tx.Hash]
if !ok {
continue
}
allSkews = append(allSkews, cs.MedianSkewSec)
totalSamples += cs.SampleCount
if cs.Calibrated {
anyCal = true
}
if cs.LastObservedTS > lastObsTS {
lastObsTS = cs.LastObservedTS
lastSkew = cs.LastSkewSec
lastAdvTS = cs.LastAdvertTS
}
tsSkews = append(tsSkews, tsSkewPair{ts: cs.LastObservedTS, skew: cs.MedianSkewSec})
}
if len(allSkews) == 0 {
return nil
}
medSkew := median(allSkews)
meanSkew := mean(allSkews)
// Severity is derived from RECENT samples only (issue #789). The
// all-time median is poisoned by historical bad data — a node that
// was off for hours and then GPS-corrected can have median = -59M sec
// while its current skew is -0.8s. Operators need severity to reflect
// current health, so they trust the dashboard.
//
// Sort tsSkews by time and take the last recentSkewWindowCount samples
// (or all samples within recentSkewWindowSec of the latest, whichever
// gives FEWER samples — we want the more-current view; a chatty node
// can fit dozens of samples in 1h, in which case the count cap wins).
sort.Slice(tsSkews, func(i, j int) bool { return tsSkews[i].ts < tsSkews[j].ts })
recentSkew := lastSkew
var recentVals []float64
if n := len(tsSkews); n > 0 {
latestTS := tsSkews[n-1].ts
// Index-based window: last K samples.
startByCount := n - recentSkewWindowCount
if startByCount < 0 {
startByCount = 0
}
// Time-based window: samples newer than latestTS - windowSec.
startByTime := n - 1
for i := n - 1; i >= 0; i-- {
if latestTS-tsSkews[i].ts <= recentSkewWindowSec {
startByTime = i
} else {
break
}
}
// Pick the narrower (larger-index) of the two windows — the most
// current view of the node's clock health.
start := startByCount
if startByTime > start {
start = startByTime
}
recentVals = make([]float64, 0, n-start)
for i := start; i < n; i++ {
recentVals = append(recentVals, tsSkews[i].skew)
}
if len(recentVals) > 0 {
recentSkew = median(recentVals)
}
}
// ── Bimodal detection (#845) ─────────────────────────────────────────
// Split recent samples into "good" (|skew| <= 1h, real clock) and
// "bad" (|skew| > 1h, firmware nonsense from uninitialized RTC).
// Classification order (first match wins):
// no_clock — goodFraction < 0.10 (essentially no real clock)
// bimodal_clock — 0.10 <= goodFraction < 0.80 AND badCount > 0
// ok/warn/etc. — goodFraction >= 0.80 (normal, outliers filtered)
var goodSamples []float64
for _, v := range recentVals {
if math.Abs(v) <= bimodalSkewThresholdSec {
goodSamples = append(goodSamples, v)
}
}
recentSampleCount := len(recentVals)
recentBadCount := recentSampleCount - len(goodSamples)
var goodFraction float64
if recentSampleCount > 0 {
goodFraction = float64(len(goodSamples)) / float64(recentSampleCount)
}
var severity SkewSeverity
if goodFraction < 0.10 {
// Essentially no real clock — classify as no_clock regardless
// of the raw skew magnitude.
severity = SkewNoClock
} else if goodFraction < 0.80 && recentBadCount > 0 {
// Bimodal: use median of GOOD samples as the "real" skew.
severity = SkewBimodalClock
if len(goodSamples) > 0 {
recentSkew = median(goodSamples)
}
} else {
// Normal path: if there are good samples, use their median
// (filters out rare outliers in ≥80% good case).
if len(goodSamples) > 0 && recentBadCount > 0 {
recentSkew = median(goodSamples)
}
severity = classifySkew(math.Abs(recentSkew))
}
// For no_clock / bimodal_clock nodes, skip drift when data is unreliable.
var drift float64
if severity != SkewNoClock && severity != SkewBimodalClock && len(tsSkews) >= minDriftSamples {
drift = computeDrift(tsSkews)
// Cap physically impossible drift rates.
if math.Abs(drift) > maxReasonableDriftPerDay {
drift = 0
}
}
// Build sparkline samples from tsSkews (already sorted by time above).
samples := make([]SkewSample, len(tsSkews))
for i, p := range tsSkews {
samples[i] = SkewSample{Timestamp: p.ts, SkewSec: round(p.skew, 1)}
}
return &NodeClockSkew{
Pubkey: pubkey,
MeanSkewSec: round(meanSkew, 1),
MedianSkewSec: round(medSkew, 1),
LastSkewSec: round(lastSkew, 1),
RecentMedianSkewSec: round(recentSkew, 1),
DriftPerDaySec: round(drift, 2),
Severity: severity,
SampleCount: totalSamples,
Calibrated: anyCal,
LastAdvertTS: lastAdvTS,
LastObservedTS: lastObsTS,
Samples: samples,
GoodFraction: round(goodFraction, 2),
RecentBadSampleCount: recentBadCount,
RecentSampleCount: recentSampleCount,
}
}
// GetFleetClockSkew returns clock skew data for all nodes that have skew data.
// Must NOT be called with s.mu held.
func (s *PacketStore) GetFleetClockSkew() []*NodeClockSkew {
s.mu.RLock()
defer s.mu.RUnlock()
// Build name/role lookup from DB cache (requires s.mu held).
allNodes, _ := s.getCachedNodesAndPM()
nameMap := make(map[string]nodeInfo, len(allNodes))
for _, ni := range allNodes {
nameMap[ni.PublicKey] = ni
}
var results []*NodeClockSkew
for pubkey := range s.byNode {
cs := s.getNodeClockSkewLocked(pubkey)
if cs == nil {
continue
}
// Enrich with node name/role.
if ni, ok := nameMap[pubkey]; ok {
cs.NodeName = ni.Name
cs.NodeRole = ni.Role
}
// Omit samples in fleet response (too much data).
cs.Samples = nil
results = append(results, cs)
}
return results
}
// GetObserverCalibrations returns the current observer clock offsets.
func (s *PacketStore) GetObserverCalibrations() []ObserverCalibration {
s.mu.RLock()
defer s.mu.RUnlock()
s.clockSkew.Recompute(s)
s.clockSkew.mu.RLock()
defer s.clockSkew.mu.RUnlock()
result := make([]ObserverCalibration, 0, len(s.clockSkew.observerOffsets))
for obsID, offset := range s.clockSkew.observerOffsets {
result = append(result, ObserverCalibration{
ObserverID: obsID,
OffsetSec: round(offset, 1),
Samples: s.clockSkew.observerSamples[obsID],
})
}
// Sort by absolute offset descending.
sort.Slice(result, func(i, j int) bool {
return math.Abs(result[i].OffsetSec) > math.Abs(result[j].OffsetSec)
})
return result
}
// ── Math Helpers ───────────────────────────────────────────────────────────────
func median(vals []float64) float64 {
if len(vals) == 0 {
return 0
}
sorted := make([]float64, len(vals))
copy(sorted, vals)
sort.Float64s(sorted)
n := len(sorted)
if n%2 == 0 {
return (sorted[n/2-1] + sorted[n/2]) / 2
}
return sorted[n/2]
}
func mean(vals []float64) float64 {
if len(vals) == 0 {
return 0
}
sum := 0.0
for _, v := range vals {
sum += v
}
return sum / float64(len(vals))
}
// tsSkewPair is a (timestamp, skew) pair for drift estimation.
type tsSkewPair struct {
ts int64
skew float64
}
// computeDrift estimates linear drift in seconds per day from time-ordered
// (timestamp, skew) pairs. Issue #789: a single GPS-correction event (huge
// skew jump in seconds) used to dominate ordinary least squares and produce
// absurd drift like 1.7M sec/day. We now:
//
// 1. Drop pairs whose consecutive skew jump exceeds maxPlausibleSkewJumpSec
// (clock corrections, not physical drift). This protects both OLS-style
// consumers and Theil-Sen.
// 2. Use Theil-Sen regression — the slope is the median of all pairwise
// slopes, naturally robust to remaining outliers (breakdown point ~29%).
//
// For very small samples after filtering we fall back to a simple slope
// between first and last calibrated samples.
func computeDrift(pairs []tsSkewPair) float64 {
if len(pairs) < 2 {
return 0
}
// Sort by timestamp.
sort.Slice(pairs, func(i, j int) bool {
return pairs[i].ts < pairs[j].ts
})
// Time span too short? Skip.
spanSec := float64(pairs[len(pairs)-1].ts - pairs[0].ts)
if spanSec < 3600 { // need at least 1 hour of data
return 0
}
// Outlier filter: drop samples where the skew jumps more than
// maxPlausibleSkewJumpSec from the running "stable" baseline.
// We anchor on the first sample, then accept each subsequent point
// that's within the threshold of the most recent accepted point —
// this preserves a slow drift while rejecting correction events.
filtered := make([]tsSkewPair, 0, len(pairs))
filtered = append(filtered, pairs[0])
for i := 1; i < len(pairs); i++ {
prev := filtered[len(filtered)-1]
if math.Abs(pairs[i].skew-prev.skew) <= maxPlausibleSkewJumpSec {
filtered = append(filtered, pairs[i])
}
}
// If the filter killed too much (e.g. unstable node), fall back to the
// raw series so we at least produce *something* — it'll be capped by
// maxReasonableDriftPerDay downstream.
if len(filtered) < 2 || float64(filtered[len(filtered)-1].ts-filtered[0].ts) < 3600 {
filtered = pairs
}
// Cap point count for Theil-Sen (O(n²) on pairs). Keep most-recent.
if len(filtered) > theilSenMaxPoints {
filtered = filtered[len(filtered)-theilSenMaxPoints:]
}
return theilSenSlope(filtered) * 86400 // sec/sec → sec/day
}
// theilSenSlope returns the Theil-Sen estimator: median of all pairwise
// slopes (yj - yi) / (tj - ti) for i < j. Naturally robust to outliers.
// Pairs must be sorted by timestamp ascending.
func theilSenSlope(pairs []tsSkewPair) float64 {
n := len(pairs)
if n < 2 {
return 0
}
// Pre-allocate: n*(n-1)/2 pairs.
slopes := make([]float64, 0, n*(n-1)/2)
for i := 0; i < n; i++ {
for j := i + 1; j < n; j++ {
dt := float64(pairs[j].ts - pairs[i].ts)
if dt <= 0 {
continue
}
slopes = append(slopes, (pairs[j].skew-pairs[i].skew)/dt)
}
}
if len(slopes) == 0 {
return 0
}
return median(slopes)
}
+956
View File
@@ -0,0 +1,956 @@
package main
import (
"fmt"
"math"
"testing"
"time"
)
// ── classifySkew ───────────────────────────────────────────────────────────────
func TestClassifySkew(t *testing.T) {
tests := []struct {
absSkew float64
expected SkewSeverity
}{
{0, SkewOK},
{60, SkewOK}, // 1 min
{299, SkewOK}, // just under 5 min
{300, SkewWarning}, // exactly 5 min
{1800, SkewWarning}, // 30 min
{3599, SkewWarning}, // just under 1 hour
{3600, SkewCritical}, // exactly 1 hour
{86400, SkewCritical}, // 1 day
{2592000 - 1, SkewCritical}, // just under 30 days
{2592000, SkewAbsurd}, // exactly 30 days
{86400 * 365 - 1, SkewAbsurd}, // just under 365 days
{86400 * 365, SkewNoClock}, // exactly 365 days
{86400 * 365 * 10, SkewNoClock}, // 10 years (epoch-0 style)
}
for _, tc := range tests {
got := classifySkew(tc.absSkew)
if got != tc.expected {
t.Errorf("classifySkew(%v) = %v, want %v", tc.absSkew, got, tc.expected)
}
}
}
// ── median ─────────────────────────────────────────────────────────────────────
func TestMedian(t *testing.T) {
tests := []struct {
vals []float64
expected float64
}{
{nil, 0},
{[]float64{}, 0},
{[]float64{5}, 5},
{[]float64{1, 3}, 2},
{[]float64{3, 1, 2}, 2},
{[]float64{4, 1, 3, 2}, 2.5},
{[]float64{-10, 0, 10}, 0},
}
for _, tc := range tests {
got := median(tc.vals)
if got != tc.expected {
t.Errorf("median(%v) = %v, want %v", tc.vals, got, tc.expected)
}
}
}
func TestMean(t *testing.T) {
tests := []struct {
vals []float64
expected float64
}{
{nil, 0},
{[]float64{10}, 10},
{[]float64{2, 4, 6}, 4},
}
for _, tc := range tests {
got := mean(tc.vals)
if got != tc.expected {
t.Errorf("mean(%v) = %v, want %v", tc.vals, got, tc.expected)
}
}
}
// ── parseISO ───────────────────────────────────────────────────────────────────
func TestParseISO(t *testing.T) {
tests := []struct {
input string
expected int64
}{
{"", 0},
{"garbage", 0},
{"2026-04-15T12:00:00Z", 1776254400},
{"2026-04-15T12:00:00+00:00", 1776254400},
}
for _, tc := range tests {
got := parseISO(tc.input)
if got != tc.expected {
t.Errorf("parseISO(%q) = %v, want %v", tc.input, got, tc.expected)
}
}
}
// ── extractTimestamp ────────────────────────────────────────────────────────────
func TestExtractTimestamp(t *testing.T) {
// Nested payload.timestamp
decoded := map[string]interface{}{
"payload": map[string]interface{}{
"timestamp": float64(1776340800),
},
}
got := extractTimestamp(decoded)
if got != 1776340800 {
t.Errorf("extractTimestamp (nested) = %v, want 1776340800", got)
}
// Top-level timestamp
decoded2 := map[string]interface{}{
"timestamp": float64(1776340900),
}
got2 := extractTimestamp(decoded2)
if got2 != 1776340900 {
t.Errorf("extractTimestamp (top-level) = %v, want 1776340900", got2)
}
// No timestamp
decoded3 := map[string]interface{}{"foo": "bar"}
got3 := extractTimestamp(decoded3)
if got3 != 0 {
t.Errorf("extractTimestamp (missing) = %v, want 0", got3)
}
}
// ── calibrateObservers ─────────────────────────────────────────────────────────
func TestCalibrateObservers_SingleObserver(t *testing.T) {
// Single-observer packets can't calibrate — should return empty.
samples := []skewSample{
{advertTS: 1000, observedTS: 1000, observerID: "obs1", hash: "h1"},
{advertTS: 2000, observedTS: 2000, observerID: "obs1", hash: "h2"},
}
offsets, _ := calibrateObservers(samples)
if len(offsets) != 0 {
t.Errorf("expected no offsets for single-observer, got %v", offsets)
}
}
func TestCalibrateObservers_MultiObserver(t *testing.T) {
// Packet h1 seen by 3 observers: obs1 at t=100, obs2 at t=110, obs3 at t=100.
// Median observation = 100. obs1=0, obs2=+10, obs3=0
// Packet h2 seen by 3 observers: obs1 at t=200, obs2 at t=210, obs3 at t=200.
// Median observation = 200. obs1=0, obs2=+10, obs3=0
samples := []skewSample{
{advertTS: 100, observedTS: 100, observerID: "obs1", hash: "h1"},
{advertTS: 100, observedTS: 110, observerID: "obs2", hash: "h1"},
{advertTS: 100, observedTS: 100, observerID: "obs3", hash: "h1"},
{advertTS: 200, observedTS: 200, observerID: "obs1", hash: "h2"},
{advertTS: 200, observedTS: 210, observerID: "obs2", hash: "h2"},
{advertTS: 200, observedTS: 200, observerID: "obs3", hash: "h2"},
}
offsets, _ := calibrateObservers(samples)
if offsets["obs1"] != 0 {
t.Errorf("obs1 offset = %v, want 0", offsets["obs1"])
}
if offsets["obs2"] != 10 {
t.Errorf("obs2 offset = %v, want 10", offsets["obs2"])
}
if offsets["obs3"] != 0 {
t.Errorf("obs3 offset = %v, want 0", offsets["obs3"])
}
}
// ── computeNodeSkew ────────────────────────────────────────────────────────────
func TestComputeNodeSkew_BasicCorrection(t *testing.T) {
// Validates observer offset correction direction.
//
// Setup: node is 60s ahead, obs1 accurate, obs2 is 10s ahead.
// With 2 observers, median obs_ts = 1005.
// obs1 offset = 1000 - 1005 = -5
// obs2 offset = 1010 - 1005 = +5
// Correction: corrected = raw_skew + obsOffset
// obs1: raw=60, corrected = 60 + (-5) = 55
// obs2: raw=50, corrected = 50 + 5 = 55
// Both converge to 55 (not exact 60 because with only 2 observers,
// the median can't fully distinguish which observer is drifted).
samples := []skewSample{
// Same packet seen by accurate obs1 and obs2 (+10s ahead)
{advertTS: 1060, observedTS: 1000, observerID: "obs1", hash: "h1"},
{advertTS: 1060, observedTS: 1010, observerID: "obs2", hash: "h1"},
}
offsets, _ := calibrateObservers(samples)
// median obs = 1005, obs1 offset = -5, obs2 offset = +5
// So the median approach finds obs2 is +5 ahead (relative to median)
// Now compute node skew with those offsets:
nodeSkew := computeNodeSkew(samples, offsets)
cs, ok := nodeSkew["h1"]
if !ok {
t.Fatal("expected skew data for hash h1")
}
// With only 2 observers, median obs_ts = 1005.
// obs1 offset = 1000-1005 = -5, obs2 offset = 1010-1005 = +5
// raw from obs1 = 60, corrected = 60 + (-5) = 55
// raw from obs2 = 50, corrected = 50 + 5 = 55
// median = 55
if cs.MedianSkewSec != 55 {
t.Errorf("median skew = %v, want 55", cs.MedianSkewSec)
}
}
func TestComputeNodeSkew_ThreeObservers(t *testing.T) {
// Node is exactly 60s ahead. obs1 accurate, obs2 accurate, obs3 +30s ahead.
// advertTS = 1060, real time = 1000
samples := []skewSample{
{advertTS: 1060, observedTS: 1000, observerID: "obs1", hash: "h1"},
{advertTS: 1060, observedTS: 1000, observerID: "obs2", hash: "h1"},
{advertTS: 1060, observedTS: 1030, observerID: "obs3", hash: "h1"},
}
offsets, _ := calibrateObservers(samples)
// median obs_ts = 1000. obs1=0, obs2=0, obs3=+30
if offsets["obs3"] != 30 {
t.Errorf("obs3 offset = %v, want 30", offsets["obs3"])
}
nodeSkew := computeNodeSkew(samples, offsets)
cs := nodeSkew["h1"]
if cs == nil {
t.Fatal("expected skew data for h1")
}
// raw from obs1 = 60, corrected = 60 + 0 = 60
// raw from obs2 = 60, corrected = 60 + 0 = 60
// raw from obs3 = 30, corrected = 30 + 30 = 60
// All three converge to 60.
if cs.MedianSkewSec != 60 {
t.Errorf("median skew = %v, want 60 (node is 60s ahead)", cs.MedianSkewSec)
}
}
// ── computeDrift ───────────────────────────────────────────────────────────────
func TestComputeDrift_Stable(t *testing.T) {
// Constant skew = no drift.
pairs := []tsSkewPair{
{ts: 0, skew: 60},
{ts: 7200, skew: 60},
{ts: 14400, skew: 60},
}
drift := computeDrift(pairs)
if drift != 0 {
t.Errorf("drift = %v, want 0 for stable skew", drift)
}
}
func TestComputeDrift_LinearDrift(t *testing.T) {
// 1 second drift per hour = 24 sec/day.
pairs := []tsSkewPair{
{ts: 0, skew: 0},
{ts: 3600, skew: 1},
{ts: 7200, skew: 2},
}
drift := computeDrift(pairs)
expected := 24.0
if math.Abs(drift-expected) > 0.1 {
t.Errorf("drift = %v, want ~%v", drift, expected)
}
}
func TestComputeDrift_TooFewSamples(t *testing.T) {
pairs := []tsSkewPair{{ts: 0, skew: 10}}
if computeDrift(pairs) != 0 {
t.Error("expected 0 drift for single sample")
}
}
func TestComputeDrift_TooShortSpan(t *testing.T) {
// Less than 1 hour apart.
pairs := []tsSkewPair{
{ts: 0, skew: 0},
{ts: 1800, skew: 10},
}
if computeDrift(pairs) != 0 {
t.Error("expected 0 drift for short time span")
}
}
// ── jsonNumber ─────────────────────────────────────────────────────────────────
func TestJsonNumber(t *testing.T) {
m := map[string]interface{}{
"a": float64(42),
"b": int64(99),
"c": "not a number",
"d": nil,
}
if jsonNumber(m, "a") != 42 {
t.Error("float64 case failed")
}
if jsonNumber(m, "b") != 99 {
t.Error("int64 case failed")
}
if jsonNumber(m, "c") != 0 {
t.Error("string case should return 0")
}
if jsonNumber(m, "d") != 0 {
t.Error("nil case should return 0")
}
if jsonNumber(m, "missing") != 0 {
t.Error("missing key should return 0")
}
}
// ── Integration: GetNodeClockSkew via PacketStore ──────────────────────────────
func TestGetNodeClockSkew_Integration(t *testing.T) {
ps := NewPacketStore(nil, nil)
// Simulate two ADVERT transmissions for the same node, seen by 2 observers each.
// Node "AABB" has clock 120s ahead.
pt := 4 // ADVERT
tx1 := &StoreTx{
Hash: "hash1",
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":1700002320}}`, // obs=1700002200, node ahead by 120s
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: "2023-11-14T22:50:00Z"}, // 1700002200
{ObserverID: "obs2", Timestamp: "2023-11-14T22:50:00Z"}, // 1700002200
},
}
tx2 := &StoreTx{
Hash: "hash2",
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":1700005920}}`, // obs=1700005800, node ahead by 120s
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: "2023-11-14T23:50:00Z"}, // 1700005800
{ObserverID: "obs2", Timestamp: "2023-11-14T23:50:00Z"}, // 1700005800
},
}
ps.mu.Lock()
ps.byNode["AABB"] = []*StoreTx{tx1, tx2}
ps.byPayloadType[4] = []*StoreTx{tx1, tx2}
// Force recompute by setting interval to 0.
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("AABB")
if result == nil {
t.Fatal("expected clock skew result for node AABB")
}
if result.Pubkey != "AABB" {
t.Errorf("pubkey = %q, want AABB", result.Pubkey)
}
// Both transmissions show 120s skew, so median should be 120.
if result.MedianSkewSec != 120 {
t.Errorf("median skew = %v, want 120", result.MedianSkewSec)
}
if result.SampleCount < 2 {
t.Errorf("sample count = %v, want >= 2", result.SampleCount)
}
if result.Severity != SkewOK {
t.Errorf("severity = %v, want ok (120s < 5min)", result.Severity)
}
// Drift should be ~0 since skew is constant.
if math.Abs(result.DriftPerDaySec) > 1 {
t.Errorf("drift = %v, want ~0 for constant skew", result.DriftPerDaySec)
}
}
func TestGetNodeClockSkew_NoData(t *testing.T) {
ps := NewPacketStore(nil, nil)
result := ps.GetNodeClockSkew("nonexistent")
if result != nil {
t.Error("expected nil for nonexistent node")
}
}
// ── Sanity check tests (#XXX — clock skew crazy stats) ────────────────────────
func TestGetNodeClockSkew_NoClock_EpochZero(t *testing.T) {
// Node with epoch-0 timestamp produces huge skew → no_clock severity, drift=0.
ps := NewPacketStore(nil, nil)
pt := 4 // ADVERT
// Epoch-ish advert: advertTS near start of 2020, observed in 2023 → |skew| > 365 days
var txs []*StoreTx
baseObs := int64(1700000000) // ~Nov 2023
for i := 0; i < 6; i++ {
obsTS := baseObs + int64(i)*7200
tx := &StoreTx{
Hash: "epoch-h" + string(rune('0'+i)),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":1577836800}}`, // Jan 1 2020 — valid but way off
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["EPOCH"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("EPOCH")
if result == nil {
t.Fatal("expected clock skew result for epoch-0 node")
}
if result.Severity != SkewNoClock {
t.Errorf("severity = %v, want no_clock", result.Severity)
}
if result.DriftPerDaySec != 0 {
t.Errorf("drift = %v, want 0 for no_clock node", result.DriftPerDaySec)
}
}
func TestGetNodeClockSkew_TooFewSamplesForDrift(t *testing.T) {
// Node with only 2 advert samples → drift should not be computed.
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 2; i++ {
obsTS := baseObs + int64(i)*7200
advTS := obsTS + 120 // 120s ahead
tx := &StoreTx{
Hash: "few-h" + string(rune('0'+i)),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["FEWSAMP"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("FEWSAMP")
if result == nil {
t.Fatal("expected clock skew result")
}
if result.DriftPerDaySec != 0 {
t.Errorf("drift = %v, want 0 for 2-sample node (minimum is %d)", result.DriftPerDaySec, minDriftSamples)
}
}
func TestGetNodeClockSkew_AbsurdDriftCapped(t *testing.T) {
// Node with wildly varying skew producing |drift| > 86400 s/day → drift capped to 0.
ps := NewPacketStore(nil, nil)
pt := 4
// Create 6 samples with extreme skew variation to produce absurd drift.
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 6; i++ {
obsTS := baseObs + int64(i)*3600
// Alternate between huge positive and negative skew offsets
skewOffset := int64(50000 * (1 - 2*(i%2))) // +50000 or -50000
advTS := obsTS + skewOffset
tx := &StoreTx{
Hash: "wild-h" + string(rune('0'+i)),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["WILD"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("WILD")
if result == nil {
t.Fatal("expected clock skew result")
}
if math.Abs(result.DriftPerDaySec) > maxReasonableDriftPerDay {
t.Errorf("drift = %v, should be capped (|drift| > %v)", result.DriftPerDaySec, maxReasonableDriftPerDay)
}
}
func TestGetNodeClockSkew_NormalNodeWithDrift(t *testing.T) {
// Normal node with 6 samples and consistent linear drift → drift computed correctly.
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 6; i++ {
obsTS := baseObs + int64(i)*7200 // every 2 hours
// Drift: 1 sec/hour = 24 sec/day
advTS := obsTS + 120 + int64(i) // skew grows by 1s per sample (2h apart)
tx := &StoreTx{
Hash: "norm-h" + string(rune('0'+i)),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["NORMAL"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("NORMAL")
if result == nil {
t.Fatal("expected clock skew result")
}
if result.Severity != SkewOK {
t.Errorf("severity = %v, want ok", result.Severity)
}
// 1s per 7200s = 12 s/day
if result.DriftPerDaySec == 0 {
t.Error("expected non-zero drift for linearly drifting node")
}
if math.Abs(result.DriftPerDaySec) > maxReasonableDriftPerDay {
t.Errorf("drift = %v, should be reasonable", result.DriftPerDaySec)
}
}
// formatInt64 is a test helper to format int64 as string for JSON embedding.
func formatInt64(n int64) string {
return fmt.Sprintf("%d", n)
}
// ── #789: Recent-window severity & robust drift ───────────────────────────────
// TestSeverityUsesRecentNotMedian: 100 historical bad samples (skew=-60s,
// each ~5min apart) followed by 5 fresh good samples (skew=-1s). All-time
// median is still huge-ish but recent-window severity must reflect the
// current healthy state.
func TestSeverityUsesRecentNotMedian(t *testing.T) {
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 105; i++ {
obsTS := baseObs + int64(i)*300 // 5 min apart
var skew int64 = -60
if i >= 100 {
skew = -1 // good samples at the tail
}
advTS := obsTS + skew
tx := &StoreTx{
Hash: fmt.Sprintf("recent-h%03d", i),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["RECENT"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
r := ps.GetNodeClockSkew("RECENT")
if r == nil {
t.Fatal("nil result")
}
if r.Severity != SkewOK {
t.Errorf("severity = %v, want ok (recent samples are healthy)", r.Severity)
}
if math.Abs(r.RecentMedianSkewSec) > 5 {
t.Errorf("recentMedianSkewSec = %v, want ~-1", r.RecentMedianSkewSec)
}
// Historical median should still be retained for context.
if math.Abs(r.MedianSkewSec) < 30 {
t.Errorf("medianSkewSec = %v, expected historical median to remain large", r.MedianSkewSec)
}
}
// TestDriftRejectsCorrectionJump: 30 minutes of clean linear drift, then a
// single 60-second skew jump. The pre-jump slope should win — drift must
// not be catastrophically inflated by the correction event.
func TestDriftRejectsCorrectionJump(t *testing.T) {
pairs := []tsSkewPair{}
// 30 min of stable, ~12 sec/day drift: 1s per 7200s.
for i := 0; i < 12; i++ {
ts := int64(i) * 300
skew := float64(i) * (1.0 / 24.0) // ~0.04s per 5min step → 12 s/day
pairs = append(pairs, tsSkewPair{ts: ts, skew: skew})
}
// Wait an hour, then a single 1000-sec correction jump (clearly outlier).
pairs = append(pairs, tsSkewPair{ts: 3600 + 12*300, skew: 1000})
drift := computeDrift(pairs)
// Without rejection this would be ~ (1000-0)/(end-0) * 86400 = enormous.
if math.Abs(drift) > 100 {
t.Errorf("drift = %v, expected small (~12 s/day), correction jump should be filtered", drift)
}
}
// TestTheilSenMatchesOLSWhenClean: on clean linear data Theil-Sen should
// produce essentially the OLS answer.
func TestTheilSenMatchesOLSWhenClean(t *testing.T) {
// 1 sec drift per hour = 24 sec/day, 20 evenly-spaced samples.
pairs := []tsSkewPair{}
for i := 0; i < 20; i++ {
pairs = append(pairs, tsSkewPair{
ts: int64(i) * 600,
skew: float64(i) * (600.0 / 3600.0),
})
}
drift := computeDrift(pairs)
if math.Abs(drift-24.0) > 0.25 { // ~1%
t.Errorf("drift = %v, want ~24", drift)
}
}
// TestReporterScenario_789: reproduce the exact scenario from issue #789.
// Reporter saw mean=-52565156, median=-59063561, last=-0.8, sample count
// 1662, drift +1793549.9 s/day, severity=absurd. After the fix, severity
// must be ok (recent samples are healthy) and drift must be sane.
func TestReporterScenario_789(t *testing.T) {
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
// 1657 samples with the bad ~-683-day skew (the historical poison),
// then 5 freshly corrected samples at -0.8s — totals 1662.
for i := 0; i < 1662; i++ {
obsTS := baseObs + int64(i)*60 // 1 min apart
var skew int64
if i < 1657 {
skew = -59063561 // ~ -683 days
} else {
skew = -1 // corrected (rounded; reporter saw -0.8)
}
advTS := obsTS + skew
tx := &StoreTx{
Hash: fmt.Sprintf("rep-%04d", i),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["REPNODE"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
r := ps.GetNodeClockSkew("REPNODE")
if r == nil {
t.Fatal("nil result")
}
// Severity must reflect current health, not the all-time median.
if r.Severity != SkewOK && r.Severity != SkewWarning {
t.Errorf("severity = %v, want ok/warning (recent samples are healthy)", r.Severity)
}
if math.Abs(r.RecentMedianSkewSec) > 5 {
t.Errorf("recentMedianSkewSec = %v, want near 0", r.RecentMedianSkewSec)
}
// Drift must not be absurd. The historical jump is one event between
// the 1657th and 1658th sample; outlier rejection must contain it.
if math.Abs(r.DriftPerDaySec) > maxReasonableDriftPerDay {
t.Errorf("drift = %v, must be <= cap %v", r.DriftPerDaySec, maxReasonableDriftPerDay)
}
// And it should be close to zero (stable historical + stable corrected).
if math.Abs(r.DriftPerDaySec) > 1000 {
t.Errorf("drift = %v, expected near zero after outlier rejection", r.DriftPerDaySec)
}
// Historical median is preserved as context.
if math.Abs(r.MedianSkewSec) < 1e6 {
t.Errorf("medianSkewSec = %v, expected historical poison preserved as context", r.MedianSkewSec)
}
}
// TestBimodalClock_845: 60% good samples → bimodal_clock severity.
func TestBimodalClock_845(t *testing.T) {
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
// 6 good samples (-5s each), 4 bad samples (-50000000s each) = 60% good
// Interleave so the recent window (last 5) captures both good and bad.
skews := []int64{-5, -5, -50000000, -5, -50000000, -5, -50000000, -5, -50000000, -5}
for i := 0; i < 10; i++ {
obsTS := baseObs + int64(i)*60
advTS := obsTS + skews[i]
tx := &StoreTx{
Hash: fmt.Sprintf("bimodal-%04d", i),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["BIMODAL"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
r := ps.GetNodeClockSkew("BIMODAL")
if r == nil {
t.Fatal("nil result")
}
if r.Severity != SkewBimodalClock {
t.Errorf("severity = %v, want bimodal_clock", r.Severity)
}
if math.Abs(r.RecentMedianSkewSec-(-5)) > 1 {
t.Errorf("recentMedianSkewSec = %v, want ≈ -5 (median of good samples)", r.RecentMedianSkewSec)
}
if r.GoodFraction < 0.5 || r.GoodFraction > 0.7 {
t.Errorf("goodFraction = %v, want ~0.6", r.GoodFraction)
}
if r.RecentBadSampleCount < 1 {
t.Errorf("recentBadSampleCount = %v, want > 0", r.RecentBadSampleCount)
}
}
// TestAllBad_NoClock_845: all samples bad → no_clock.
func TestAllBad_NoClock_845(t *testing.T) {
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 10; i++ {
obsTS := baseObs + int64(i)*60
advTS := obsTS - 50000000
tx := &StoreTx{
Hash: fmt.Sprintf("allbad-%04d", i),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["ALLBAD"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
r := ps.GetNodeClockSkew("ALLBAD")
if r == nil {
t.Fatal("nil result")
}
if r.Severity != SkewNoClock {
t.Errorf("severity = %v, want no_clock", r.Severity)
}
}
// TestMostlyGood_OK_845: 90% good 10% bad → ok (outlier filtered).
func TestMostlyGood_OK_845(t *testing.T) {
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
// 9 good at -5s, 1 bad at -50000000s
for i := 0; i < 10; i++ {
obsTS := baseObs + int64(i)*60
var skew int64
if i < 9 {
skew = -5
} else {
skew = -50000000
}
advTS := obsTS + skew
tx := &StoreTx{
Hash: fmt.Sprintf("mostly-%04d", i),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["MOSTLY"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
r := ps.GetNodeClockSkew("MOSTLY")
if r == nil {
t.Fatal("nil result")
}
// 90% good → normal classification path, median of good samples = -5s → ok
if r.Severity != SkewOK {
t.Errorf("severity = %v, want ok", r.Severity)
}
if math.Abs(r.RecentMedianSkewSec-(-5)) > 1 {
t.Errorf("recentMedianSkewSec = %v, want ≈ -5", r.RecentMedianSkewSec)
}
}
// TestSingleSample_845: one good sample → ok.
func TestSingleSample_845(t *testing.T) {
ps := NewPacketStore(nil, nil)
pt := 4
obsTS := int64(1700000000)
advTS := obsTS - 30 // 30s skew
tx := &StoreTx{
Hash: "single-0001",
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
ps.mu.Lock()
ps.byNode["SINGLE"] = []*StoreTx{tx}
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
r := ps.GetNodeClockSkew("SINGLE")
if r == nil {
t.Fatal("nil result")
}
if r.Severity != SkewOK {
t.Errorf("severity = %v, want ok", r.Severity)
}
if r.RecentSampleCount != 1 {
t.Errorf("recentSampleCount = %d, want 1", r.RecentSampleCount)
}
if r.GoodFraction != 1.0 {
t.Errorf("goodFraction = %v, want 1.0", r.GoodFraction)
}
}
// TestFiftyFifty_Bimodal_845: 50% good / 50% bad → bimodal_clock.
func TestFiftyFifty_Bimodal_845(t *testing.T) {
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 10; i++ {
obsTS := baseObs + int64(i)*60
var skew int64
if i%2 == 0 {
skew = -10
} else {
skew = -50000000
}
tx := &StoreTx{
Hash: fmt.Sprintf("fifty-%04d", i),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(obsTS+skew) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["FIFTY"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
r := ps.GetNodeClockSkew("FIFTY")
if r == nil {
t.Fatal("nil result")
}
if r.Severity != SkewBimodalClock {
t.Errorf("severity = %v, want bimodal_clock", r.Severity)
}
if r.GoodFraction < 0.4 || r.GoodFraction > 0.6 {
t.Errorf("goodFraction = %v, want ~0.5", r.GoodFraction)
}
}
// TestAllGood_OK_845: all samples good → ok, no bimodal.
func TestAllGood_OK_845(t *testing.T) {
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 10; i++ {
obsTS := baseObs + int64(i)*60
tx := &StoreTx{
Hash: fmt.Sprintf("allgood-%04d", i),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(obsTS-3) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["ALLGOOD"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
r := ps.GetNodeClockSkew("ALLGOOD")
if r == nil {
t.Fatal("nil result")
}
if r.Severity != SkewOK {
t.Errorf("severity = %v, want ok", r.Severity)
}
if r.GoodFraction != 1.0 {
t.Errorf("goodFraction = %v, want 1.0", r.GoodFraction)
}
if r.RecentBadSampleCount != 0 {
t.Errorf("recentBadSampleCount = %v, want 0", r.RecentBadSampleCount)
}
}
+131
View File
@@ -0,0 +1,131 @@
package main
import (
"testing"
"time"
)
// TestCollisionDetailsIncludeNodePairs verifies that collision details contain
// the correct prefix and matching node pairs (#757).
func TestCollisionDetailsIncludeNodePairs(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
// Insert two repeater nodes with the same 3-byte prefix "AABB11"
db.conn.Exec(`INSERT INTO nodes (public_key, name, role) VALUES ('aabb11ccdd001122', 'Node Alpha', 'repeater')`)
db.conn.Exec(`INSERT INTO nodes (public_key, name, role) VALUES ('aabb11eeff334455', 'Node Beta', 'repeater')`)
// Add advert transmissions with hash_size=3 path bytes (0x80 = bits 10 → size 3)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('0180aabb11ccdd', 'col_hash_01', ?, 1, 4, '{"pubKey":"aabb11ccdd001122","name":"Node Alpha","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 10.0, -91, '["aabb11"]', ?)`, recentEpoch)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('0180aabb11eeff', 'col_hash_02', ?, 1, 4, '{"pubKey":"aabb11eeff334455","name":"Node Beta","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (2, 1, 9.0, -93, '["aabb11"]', ?)`, recentEpoch)
store := NewPacketStore(db, nil)
store.Load()
result := store.GetAnalyticsHashCollisions("")
bySize, ok := result["by_size"].(map[string]interface{})
if !ok {
t.Fatal("expected by_size map")
}
size3, ok := bySize["3"].(map[string]interface{})
if !ok {
t.Fatal("expected by_size[3] map")
}
collisions, ok := size3["collisions"].([]collisionEntry)
if !ok {
t.Fatalf("expected collisions as []collisionEntry, got %T", size3["collisions"])
}
// Find our collision
var found *collisionEntry
for i := range collisions {
if collisions[i].Prefix == "AABB11" {
found = &collisions[i]
break
}
}
if found == nil {
t.Fatal("expected collision with prefix AABB11")
}
if found.Appearances != 2 {
t.Errorf("expected 2 appearances, got %d", found.Appearances)
}
if len(found.Nodes) != 2 {
t.Fatalf("expected 2 nodes in collision, got %d", len(found.Nodes))
}
// Verify node pairs
pubkeys := map[string]bool{}
names := map[string]bool{}
for _, n := range found.Nodes {
pubkeys[n.PublicKey] = true
names[n.Name] = true
}
if !pubkeys["aabb11ccdd001122"] {
t.Error("expected node aabb11ccdd001122 in collision")
}
if !pubkeys["aabb11eeff334455"] {
t.Error("expected node aabb11eeff334455 in collision")
}
if !names["Node Alpha"] {
t.Error("expected Node Alpha in collision")
}
if !names["Node Beta"] {
t.Error("expected Node Beta in collision")
}
}
// TestCollisionDetailsEmptyWhenNoCollisions verifies that collision details are
// empty when there are no collisions (#757).
func TestCollisionDetailsEmptyWhenNoCollisions(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
// Insert one repeater node with 3-byte hash
db.conn.Exec(`INSERT INTO nodes (public_key, name, role) VALUES ('aabb11ccdd001122', 'Solo Node', 'repeater')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('0180aabb11ccdd', 'solo_hash_01', ?, 1, 4, '{"pubKey":"aabb11ccdd001122","name":"Solo Node","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 10.0, -91, '["aabb11"]', ?)`, recentEpoch)
store := NewPacketStore(db, nil)
store.Load()
result := store.GetAnalyticsHashCollisions("")
bySize, ok := result["by_size"].(map[string]interface{})
if !ok {
t.Fatal("expected by_size map")
}
size3, ok := bySize["3"].(map[string]interface{})
if !ok {
t.Fatal("expected by_size[3] map")
}
collisions, ok := size3["collisions"].([]collisionEntry)
if !ok {
t.Fatalf("expected collisions as []collisionEntry, got %T", size3["collisions"])
}
if len(collisions) != 0 {
t.Errorf("expected 0 collisions, got %d", len(collisions))
}
}
+120 -3
View File
@@ -6,6 +6,7 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"github.com/meshcore-analyzer/geofilter"
)
@@ -16,6 +17,17 @@ type Config struct {
APIKey string `json:"apiKey"`
DBPath string `json:"dbPath"`
// NodeBlacklist is a list of public keys to exclude from all API responses.
// Blacklisted nodes are hidden from node lists, search, detail, map, and stats.
// Use this to filter out trolls, nodes with offensive names, or nodes
// reporting deliberately false data (e.g. wrong GPS position) that the
// operator refuses to fix.
NodeBlacklist []string `json:"nodeBlacklist"`
// blacklistSetCached is the lazily-built set version of NodeBlacklist.
blacklistSetCached map[string]bool
blacklistOnce sync.Once
Branding map[string]interface{} `json:"branding"`
Theme map[string]interface{} `json:"theme"`
ThemeDark map[string]interface{} `json:"themeDark"`
@@ -55,22 +67,91 @@ type Config struct {
GeoFilter *GeoFilterConfig `json:"geo_filter,omitempty"`
Timestamps *TimestampConfig `json:"timestamps,omitempty"`
DebugAffinity bool `json:"debugAffinity,omitempty"`
ResolvedPath *ResolvedPathConfig `json:"resolvedPath,omitempty"`
NeighborGraph *NeighborGraphConfig `json:"neighborGraph,omitempty"`
}
// weakAPIKeys is the blocklist of known default/example API keys that must be rejected.
var weakAPIKeys = map[string]bool{
"your-secret-api-key-here": true,
"change-me": true,
"example": true,
"test": true,
"password": true,
"admin": true,
"apikey": true,
"api-key": true,
"secret": true,
"default": true,
}
// IsWeakAPIKey returns true if the key is in the blocklist or shorter than 16 characters.
func IsWeakAPIKey(key string) bool {
if key == "" {
return false // empty is handled separately (endpoints disabled)
}
if weakAPIKeys[strings.ToLower(key)] {
return true
}
if len(key) < 16 {
return true
}
return false
}
// ResolvedPathConfig controls async backfill behavior.
type ResolvedPathConfig struct {
BackfillHours int `json:"backfillHours"` // how far back (hours) to scan for NULL resolved_path (default 24)
}
// NeighborGraphConfig controls neighbor edge pruning.
type NeighborGraphConfig struct {
MaxAgeDays int `json:"maxAgeDays"` // edges older than this are pruned (default 5)
}
// PacketStoreConfig controls in-memory packet store limits.
type PacketStoreConfig struct {
RetentionHours float64 `json:"retentionHours"` // max age of packets in hours (0 = unlimited)
MaxMemoryMB int `json:"maxMemoryMB"` // hard memory ceiling in MB (0 = unlimited)
MaxMemoryMB int `json:"maxMemoryMB"` // hard memory ceiling in MB (0 = unlimited)
MaxResolvedPubkeyIndexEntries int `json:"maxResolvedPubkeyIndexEntries"` // warning threshold for index size (0 = 5M default)
}
// GeoFilterConfig is an alias for the shared geofilter.Config type.
type GeoFilterConfig = geofilter.Config
type RetentionConfig struct {
NodeDays int `json:"nodeDays"`
PacketDays int `json:"packetDays"`
NodeDays int `json:"nodeDays"`
ObserverDays int `json:"observerDays"`
PacketDays int `json:"packetDays"`
MetricsDays int `json:"metricsDays"`
}
// MetricsRetentionDays returns configured metrics retention or 30 days default.
func (c *Config) MetricsRetentionDays() int {
if c.Retention != nil && c.Retention.MetricsDays > 0 {
return c.Retention.MetricsDays
}
return 30
}
// BackfillHours returns configured backfill window or 24h default.
func (c *Config) BackfillHours() int {
if c.ResolvedPath != nil && c.ResolvedPath.BackfillHours > 0 {
return c.ResolvedPath.BackfillHours
}
return 24
}
// NeighborMaxAgeDays returns configured max edge age or 30 days default.
func (c *Config) NeighborMaxAgeDays() int {
if c.NeighborGraph != nil && c.NeighborGraph.MaxAgeDays > 0 {
return c.NeighborGraph.MaxAgeDays
}
return 5
}
type TimestampConfig struct {
DefaultMode string `json:"defaultMode"` // "ago" | "absolute"
@@ -98,6 +179,15 @@ func (c *Config) NodeDaysOrDefault() int {
return 7
}
// ObserverDaysOrDefault returns the configured retention.observerDays or 14 if not set.
// A value of -1 means observers are never removed.
func (c *Config) ObserverDaysOrDefault() int {
if c.Retention != nil && c.Retention.ObserverDays != 0 {
return c.Retention.ObserverDays
}
return 14
}
type HealthThresholds struct {
InfraDegradedHours float64 `json:"infraDegradedHours"`
InfraSilentHours float64 `json:"infraSilentHours"`
@@ -271,3 +361,30 @@ func (c *Config) PropagationBufferMs() int {
}
return 5000
}
// blacklistSet lazily builds and caches the nodeBlacklist as a set for O(1) lookups.
// Uses sync.Once to eliminate the data race on first concurrent access.
func (c *Config) blacklistSet() map[string]bool {
c.blacklistOnce.Do(func() {
if len(c.NodeBlacklist) == 0 {
return
}
m := make(map[string]bool, len(c.NodeBlacklist))
for _, pk := range c.NodeBlacklist {
trimmed := strings.ToLower(strings.TrimSpace(pk))
if trimmed != "" {
m[trimmed] = true
}
}
c.blacklistSetCached = m
})
return c.blacklistSetCached
}
// IsBlacklisted returns true if the given public key is in the nodeBlacklist.
func (c *Config) IsBlacklisted(pubkey string) bool {
if c == nil || len(c.NodeBlacklist) == 0 {
return false
}
return c.blacklistSet()[strings.ToLower(strings.TrimSpace(pubkey))]
}
+177
View File
@@ -0,0 +1,177 @@
package main
import (
"database/sql"
"path/filepath"
"testing"
"time"
_ "modernc.org/sqlite"
)
func TestBackfillHoursDefault(t *testing.T) {
cfg := &Config{}
if got := cfg.BackfillHours(); got != 24 {
t.Errorf("BackfillHours() = %d, want 24", got)
}
}
func TestBackfillHoursConfigured(t *testing.T) {
cfg := &Config{ResolvedPath: &ResolvedPathConfig{BackfillHours: 48}}
if got := cfg.BackfillHours(); got != 48 {
t.Errorf("BackfillHours() = %d, want 48", got)
}
}
func TestBackfillHoursZeroFallsBack(t *testing.T) {
cfg := &Config{ResolvedPath: &ResolvedPathConfig{BackfillHours: 0}}
if got := cfg.BackfillHours(); got != 24 {
t.Errorf("BackfillHours() = %d, want 24 (default for zero)", got)
}
}
func TestNeighborMaxAgeDaysDefault(t *testing.T) {
cfg := &Config{}
if got := cfg.NeighborMaxAgeDays(); got != 5 {
t.Errorf("NeighborMaxAgeDays() = %d, want 5", got)
}
}
func TestNeighborMaxAgeDaysConfigured(t *testing.T) {
cfg := &Config{NeighborGraph: &NeighborGraphConfig{MaxAgeDays: 7}}
if got := cfg.NeighborMaxAgeDays(); got != 7 {
t.Errorf("NeighborMaxAgeDays() = %d, want 7", got)
}
}
func TestGraphPruneOlderThan(t *testing.T) {
g := NewNeighborGraph()
now := time.Now().UTC()
// Add a recent edge
g.upsertEdge("aaa", "bbb", "bb", "obs1", nil, now)
// Add an old edge
g.upsertEdge("ccc", "ddd", "dd", "obs1", nil, now.Add(-60*24*time.Hour))
if len(g.AllEdges()) != 2 {
t.Fatalf("expected 2 edges, got %d", len(g.AllEdges()))
}
cutoff := now.Add(-30 * 24 * time.Hour)
pruned := g.PruneOlderThan(cutoff)
if pruned != 1 {
t.Errorf("PruneOlderThan pruned %d, want 1", pruned)
}
edges := g.AllEdges()
if len(edges) != 1 {
t.Fatalf("expected 1 edge after prune, got %d", len(edges))
}
if edges[0].NodeA != "aaa" && edges[0].NodeB != "aaa" {
t.Errorf("wrong edge survived prune: %+v", edges[0])
}
}
func TestPruneNeighborEdgesDB(t *testing.T) {
tmpDir := t.TempDir()
dbPath := filepath.Join(tmpDir, "test.db")
db, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
defer db.Close()
_, err = db.Exec(`CREATE TABLE neighbor_edges (
node_a TEXT NOT NULL,
node_b TEXT NOT NULL,
count INTEGER DEFAULT 1,
last_seen TEXT,
PRIMARY KEY (node_a, node_b)
)`)
if err != nil {
t.Fatal(err)
}
now := time.Now().UTC()
old := now.Add(-60 * 24 * time.Hour)
db.Exec("INSERT INTO neighbor_edges (node_a, node_b, count, last_seen) VALUES (?, ?, 5, ?)",
"aaa", "bbb", now.Format(time.RFC3339))
db.Exec("INSERT INTO neighbor_edges (node_a, node_b, count, last_seen) VALUES (?, ?, 3, ?)",
"ccc", "ddd", old.Format(time.RFC3339))
g := NewNeighborGraph()
g.upsertEdge("aaa", "bbb", "bb", "obs1", nil, now)
g.upsertEdge("ccc", "ddd", "dd", "obs1", nil, old)
pruned, err := PruneNeighborEdges(dbPath, g, 30)
if err != nil {
t.Fatal(err)
}
if pruned != 1 {
t.Errorf("PruneNeighborEdges pruned %d DB rows, want 1", pruned)
}
var count int
db.QueryRow("SELECT COUNT(*) FROM neighbor_edges").Scan(&count)
if count != 1 {
t.Errorf("expected 1 row in DB after prune, got %d", count)
}
if len(g.AllEdges()) != 1 {
t.Errorf("expected 1 in-memory edge after prune, got %d", len(g.AllEdges()))
}
}
func TestBackfillRespectsHourWindow(t *testing.T) {
store := &PacketStore{}
now := time.Now().UTC()
oldTime := now.Add(-48 * time.Hour).Format(time.RFC3339Nano)
newTime := now.Add(-30 * time.Minute).Format(time.RFC3339Nano)
store.packets = []*StoreTx{
{
ID: 1,
Hash: "old-hash",
FirstSeen: oldTime,
Observations: []*StoreObs{
{ID: 1, PathJSON: `["abc"]`},
},
},
{
ID: 2,
Hash: "new-hash",
FirstSeen: newTime,
Observations: []*StoreObs{
{ID: 2, PathJSON: `["def"]`},
},
},
}
// With a 1-hour window, only the new tx should be processed.
// backfillResolvedPathsAsync will find no prefix map and finish quickly,
// but we can verify the pending count reflects the window.
go backfillResolvedPathsAsync(store, "", 100, time.Millisecond, 1)
// Wait for completion
for i := 0; i < 100; i++ {
if store.backfillComplete.Load() {
break
}
time.Sleep(10 * time.Millisecond)
}
if !store.backfillComplete.Load() {
t.Fatal("backfill did not complete")
}
// With no prefix map, total should be 0 (early exit) or just the new one
// The function exits early when pm == nil, so backfillTotal stays at 0
// if there were pending items but no pm. Let's verify it didn't process
// the old one by checking total <= 1.
total := store.backfillTotal.Load()
if total > 1 {
t.Errorf("backfill total = %d, want <= 1 (old tx should be excluded by hour window)", total)
}
}
+22
View File
@@ -365,3 +365,25 @@ func TestPropagationBufferMs(t *testing.T) {
}
})
}
func TestObserverDaysOrDefault(t *testing.T) {
tests := []struct {
name string
cfg *Config
want int
}{
{"nil retention", &Config{}, 14},
{"zero observer days", &Config{Retention: &RetentionConfig{ObserverDays: 0}}, 14},
{"positive value", &Config{Retention: &RetentionConfig{ObserverDays: 30}}, 30},
{"keep forever", &Config{Retention: &RetentionConfig{ObserverDays: -1}}, -1},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := tt.cfg.ObserverDaysOrDefault()
if got != tt.want {
t.Errorf("ObserverDaysOrDefault() = %d, want %d", got, tt.want)
}
})
}
}
File diff suppressed because it is too large Load Diff
+772 -91
View File
File diff suppressed because it is too large Load Diff
+739 -33
View File
@@ -48,7 +48,8 @@ func setupTestDB(t *testing.T) *DB {
radio TEXT,
battery_mv INTEGER,
uptime_secs INTEGER,
noise_floor REAL
noise_floor REAL,
last_packet_at TEXT DEFAULT NULL
);
CREATE TABLE transmissions (
@@ -60,6 +61,7 @@ func setupTestDB(t *testing.T) *DB {
payload_type INTEGER,
payload_version INTEGER,
decoded_json TEXT,
channel_hash TEXT DEFAULT NULL,
created_at TEXT DEFAULT (datetime('now'))
);
@@ -72,15 +74,32 @@ func setupTestDB(t *testing.T) *DB {
rssi REAL,
score INTEGER,
path_json TEXT,
timestamp INTEGER NOT NULL
timestamp INTEGER NOT NULL,
resolved_path TEXT,
raw_hex TEXT
);
CREATE TABLE IF NOT EXISTS observer_metrics (
observer_id TEXT NOT NULL,
timestamp TEXT NOT NULL,
noise_floor REAL,
tx_air_secs INTEGER,
rx_air_secs INTEGER,
recv_errors INTEGER,
battery_mv INTEGER,
packets_sent INTEGER,
packets_recv INTEGER,
PRIMARY KEY (observer_id, timestamp)
);
CREATE INDEX IF NOT EXISTS idx_observer_metrics_timestamp ON observer_metrics(timestamp);
`
if _, err := conn.Exec(schema); err != nil {
t.Fatal(err)
}
return &DB{conn: conn, isV3: true}
return &DB{conn: conn, isV3: true, hasResolvedPath: true}
}
func seedTestData(t *testing.T, db *DB) {
@@ -108,23 +127,24 @@ func seedTestData(t *testing.T, db *DB) {
VALUES ('1122334455667788', 'TestRoom', 'room', 37.4, -121.9, ?, '2026-01-01T00:00:00Z', 5)`, twoDaysAgo)
// Seed transmissions
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('AABB', 'abc123def4567890', ?, 1, 4, '{"pubKey":"aabbccdd11223344","name":"TestRepeater","type":"ADVERT","timestamp":1700000000,"timestampISO":"2023-11-14T22:13:20.000Z","signature":"abcdef","flags":{"isRepeater":true},"lat":37.5,"lon":-122.0}')`, recent)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('CCDD', '1234567890abcdef', ?, 1, 5, '{"type":"CHAN","channel":"#test","text":"Hello: World","sender":"TestUser"}')`, yesterday)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AABB', 'abc123def4567890', ?, 1, 4, '{"pubKey":"aabbccdd11223344","name":"TestRepeater","type":"ADVERT","timestamp":1700000000,"timestampISO":"2023-11-14T22:13:20.000Z","signature":"abcdef","flags":{"isRepeater":true},"lat":37.5,"lon":-122.0}', '#test')`, recent)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('CCDD', '1234567890abcdef', ?, 1, 5, '{"type":"CHAN","channel":"#test","text":"Hello: World","sender":"TestUser"}', '#test')`, yesterday)
// Second ADVERT for same node with different hash_size (raw_hex byte 0x1F → hs=1 vs 0xBB → hs=3)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('AA1F', 'def456abc1230099', ?, 1, 4, '{"pubKey":"aabbccdd11223344","name":"TestRepeater","type":"ADVERT","timestamp":1700000100,"timestampISO":"2023-11-14T22:14:40.000Z","signature":"fedcba","flags":{"isRepeater":true},"lat":37.5,"lon":-122.0}')`, yesterday)
// Seed observations (use unix timestamps)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 12.5, -90, '["aa","bb"]', ?)`, recentEpoch)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 2, 8.0, -95, '["aa"]', ?)`, recentEpoch-100)
// resolved_path contains full pubkeys parallel to path_json hops
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp, resolved_path)
VALUES (1, 1, 12.5, -90, '["aa","bb"]', ?, '["aabbccdd11223344","eeff00112233aabb"]')`, recentEpoch)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp, resolved_path)
VALUES (1, 2, 8.0, -95, '["aa"]', ?, '["aabbccdd11223344"]')`, recentEpoch-100)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (2, 1, 15.0, -85, '[]', ?)`, yesterdayEpoch)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (3, 1, 10.0, -92, '["cc"]', ?)`, yesterdayEpoch)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp, resolved_path)
VALUES (3, 1, 10.0, -92, '["cc"]', ?, '["1122334455667788"]')`, yesterdayEpoch)
}
func TestGetStats(t *testing.T) {
@@ -336,6 +356,10 @@ func TestGetObservers(t *testing.T) {
if observers[0].ID != "obs1" {
t.Errorf("expected obs1 first (most recent), got %s", observers[0].ID)
}
// last_packet_at should be nil since seedTestData doesn't set it
if observers[0].LastPacketAt != nil {
t.Errorf("expected nil LastPacketAt for obs1 from seed, got %v", *observers[0].LastPacketAt)
}
}
func TestGetObserverByID(t *testing.T) {
@@ -350,6 +374,48 @@ func TestGetObserverByID(t *testing.T) {
if obs.ID != "obs1" {
t.Errorf("expected obs1, got %s", obs.ID)
}
// Verify last_packet_at is nil by default
if obs.LastPacketAt != nil {
t.Errorf("expected nil LastPacketAt, got %v", *obs.LastPacketAt)
}
}
func TestGetObserverLastPacketAt(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
seedTestData(t, db)
// Set last_packet_at for obs1
ts := "2026-04-24T12:00:00Z"
db.conn.Exec(`UPDATE observers SET last_packet_at = ? WHERE id = ?`, ts, "obs1")
// Verify via GetObservers
observers, err := db.GetObservers()
if err != nil {
t.Fatal(err)
}
var obs1 *Observer
for i := range observers {
if observers[i].ID == "obs1" {
obs1 = &observers[i]
break
}
}
if obs1 == nil {
t.Fatal("obs1 not found")
}
if obs1.LastPacketAt == nil || *obs1.LastPacketAt != ts {
t.Errorf("expected LastPacketAt=%s via GetObservers, got %v", ts, obs1.LastPacketAt)
}
// Verify via GetObserverByID
obs, err := db.GetObserverByID("obs1")
if err != nil {
t.Fatal(err)
}
if obs.LastPacketAt == nil || *obs.LastPacketAt != ts {
t.Errorf("expected LastPacketAt=%s via GetObserverByID, got %v", ts, obs.LastPacketAt)
}
}
func TestGetObserverByIDNotFound(t *testing.T) {
@@ -718,12 +784,12 @@ func TestGetChannelMessagesRegionFiltering(t *testing.T) {
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer One', 'SJC')`)
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs2', 'Observer Two', ' sfo ')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'chanregion0001', ?, 1, 5,
'{"type":"CHAN","channel":"#region","text":"SjcUser: One","sender":"SjcUser"}')`, ts1)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
'{"type":"CHAN","channel":"#region","text":"SjcUser: One","sender":"SjcUser"}', '#region')`, ts1)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'chanregion0002', ?, 1, 5,
'{"type":"CHAN","channel":"#region","text":"SfoUser: Two","sender":"SfoUser"}')`, ts2)
'{"type":"CHAN","channel":"#region","text":"SfoUser: Two","sender":"SfoUser"}', '#region')`, ts2)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 10.0, -90, '[]', ?)`, epoch1)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
@@ -1012,6 +1078,171 @@ func TestGetNodesFiltering(t *testing.T) {
t.Errorf("expected 1 node with offset, got %d", len(nodes))
}
})
t.Run("region filter SJC", func(t *testing.T) {
nodes, total, _, err := db.GetNodes(50, 0, "", "", "", "", "", "SJC")
if err != nil {
t.Fatal(err)
}
if total != 1 {
t.Errorf("expected 1 node for SJC region, got %d", total)
}
if len(nodes) != 1 {
t.Fatalf("expected 1 node, got %d", len(nodes))
}
if nodes[0]["public_key"] != "aabbccdd11223344" {
t.Errorf("expected TestRepeater, got %v", nodes[0]["public_key"])
}
})
t.Run("region filter SFO", func(t *testing.T) {
_, total, _, err := db.GetNodes(50, 0, "", "", "", "", "", "SFO")
if err != nil {
t.Fatal(err)
}
if total != 1 {
t.Errorf("expected 1 node for SFO region, got %d", total)
}
})
t.Run("region filter multi", func(t *testing.T) {
_, total, _, err := db.GetNodes(50, 0, "", "", "", "", "", "SJC,SFO")
if err != nil {
t.Fatal(err)
}
if total != 1 {
t.Errorf("expected 1 node for SJC,SFO region, got %d", total)
}
})
t.Run("region filter unknown", func(t *testing.T) {
_, total, _, err := db.GetNodes(50, 0, "", "", "", "", "", "AMS")
if err != nil {
t.Fatal(err)
}
if total != 0 {
t.Errorf("expected 0 nodes for unknown region, got %d", total)
}
})
}
// setupTestDBV2 creates an in-memory SQLite database with the v2 schema
// where observations use observer_id TEXT instead of observer_idx INTEGER.
func setupTestDBV2(t *testing.T) *DB {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatal(err)
}
conn.SetMaxOpenConns(1)
schema := `
CREATE TABLE nodes (
public_key TEXT PRIMARY KEY,
name TEXT,
role TEXT,
lat REAL,
lon REAL,
last_seen TEXT,
first_seen TEXT,
advert_count INTEGER DEFAULT 0,
battery_mv INTEGER,
temperature_c REAL
);
CREATE TABLE observers (
id TEXT PRIMARY KEY,
name TEXT,
iata TEXT,
last_seen TEXT,
first_seen TEXT,
packet_count INTEGER DEFAULT 0,
last_packet_at TEXT DEFAULT NULL
);
CREATE TABLE transmissions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
raw_hex TEXT NOT NULL,
hash TEXT NOT NULL UNIQUE,
first_seen TEXT NOT NULL,
route_type INTEGER,
payload_type INTEGER,
payload_version INTEGER,
decoded_json TEXT,
channel_hash TEXT DEFAULT NULL,
created_at TEXT DEFAULT (datetime('now'))
);
CREATE TABLE observations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
transmission_id INTEGER NOT NULL REFERENCES transmissions(id),
observer_id TEXT,
observer_name TEXT,
direction TEXT,
snr REAL,
rssi REAL,
score INTEGER,
path_json TEXT,
timestamp INTEGER NOT NULL,
raw_hex TEXT
);
`
if _, err := conn.Exec(schema); err != nil {
t.Fatal(err)
}
return &DB{conn: conn, isV3: false}
}
func TestGetNodesRegionFilterV2(t *testing.T) {
db := setupTestDBV2(t)
defer db.Close()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
// Seed observer with IATA code
db.conn.Exec(`INSERT INTO observers (id, name, iata, last_seen, first_seen, packet_count)
VALUES ('obs-v2-1', 'V2 Observer', 'LAX', ?, '2026-01-01T00:00:00Z', 10)`, recent)
// Seed a node
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen, advert_count)
VALUES ('v2pubkey11223344', 'V2Node', 'repeater', 34.0, -118.0, ?, '2026-01-01T00:00:00Z', 5)`, recent)
// Seed an ADVERT transmission for the node
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('AABB', 'v2hash0001', ?, 1, 4, '{"pubKey":"v2pubkey11223344","name":"V2Node","type":"ADVERT"}')`, recent)
// Seed v2-style observation: observer_id references observers.id directly
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_id, observer_name, snr, rssi, path_json, timestamp)
VALUES (1, 'obs-v2-1', 'V2 Observer', 10.0, -90, '[]', ?)`, recentEpoch)
t.Run("v2 region filter match", func(t *testing.T) {
nodes, total, _, err := db.GetNodes(50, 0, "", "", "", "", "", "LAX")
if err != nil {
t.Fatal(err)
}
if total != 1 {
t.Errorf("expected 1 node for LAX region (v2 schema), got %d", total)
}
if len(nodes) != 1 {
t.Fatalf("expected 1 node, got %d", len(nodes))
}
if nodes[0]["public_key"] != "v2pubkey11223344" {
t.Errorf("expected V2Node, got %v", nodes[0]["public_key"])
}
})
t.Run("v2 region filter no match", func(t *testing.T) {
_, total, _, err := db.GetNodes(50, 0, "", "", "", "", "", "JFK")
if err != nil {
t.Fatal(err)
}
if total != 0 {
t.Errorf("expected 0 nodes for JFK region (v2 schema), got %d", total)
}
})
}
func TestGetChannelMessagesDedup(t *testing.T) {
@@ -1023,12 +1254,12 @@ func TestGetChannelMessagesDedup(t *testing.T) {
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs2', 'Observer Two', 'SFO')`)
// Insert two transmissions with same hash to test dedup
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'chanmsg00000001', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#general","text":"User1: Hello","sender":"User1"}')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
'{"type":"CHAN","channel":"#general","text":"User1: Hello","sender":"User1"}', '#general')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'chanmsg00000002', '2026-01-15T10:01:00Z', 1, 5,
'{"type":"CHAN","channel":"#general","text":"User2: World","sender":"User2"}')`)
'{"type":"CHAN","channel":"#general","text":"User2: World","sender":"User2"}', '#general')`)
// Observations: first msg seen by two observers (dedup), second by one
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
@@ -1072,9 +1303,9 @@ func TestGetChannelMessagesNoSender(t *testing.T) {
defer db.Close()
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer One', 'SJC')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('CC', 'chanmsg00000003', '2026-01-15T10:02:00Z', 1, 5,
'{"type":"CHAN","channel":"#noname","text":"plain text no colon"}')`)
'{"type":"CHAN","channel":"#noname","text":"plain text no colon"}', '#noname')`)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 12.0, -90, null, 1736935300)`)
@@ -1177,9 +1408,9 @@ func TestGetChannelMessagesObserverFallback(t *testing.T) {
defer db.Close()
// Observer with ID but no name entry (observer_idx won't match)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'chanmsg00000004', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#obs","text":"Sender: Test","sender":"Sender"}')`)
'{"type":"CHAN","channel":"#obs","text":"Sender: Test","sender":"Sender"}', '#obs')`)
// Observation without observer (observer_idx = NULL)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, NULL, 12.0, -90, null, 1736935200)`)
@@ -1201,12 +1432,12 @@ func TestGetChannelsMultiple(t *testing.T) {
defer db.Close()
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer', 'SJC')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'chan1hash', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#alpha","text":"Alice: Hello","sender":"Alice"}')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
'{"type":"CHAN","channel":"#alpha","text":"Alice: Hello","sender":"Alice"}', '#alpha')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'chan2hash', '2026-01-15T10:01:00Z', 1, 5,
'{"type":"CHAN","channel":"#beta","text":"Bob: World","sender":"Bob"}')`)
'{"type":"CHAN","channel":"#beta","text":"Bob: World","sender":"Bob"}', '#beta')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('CC', 'chan3hash', '2026-01-15T10:02:00Z', 1, 5,
'{"type":"CHAN","channel":"","text":"No channel"}')`)
@@ -1289,13 +1520,13 @@ func TestGetChannelsStaleMessage(t *testing.T) {
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs2', 'Observer2', 'SFO')`)
// Older message (first_seen T1)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'oldhash1', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#test","text":"Alice: Old message","sender":"Alice"}')`)
'{"type":"CHAN","channel":"#test","text":"Alice: Old message","sender":"Alice"}', '#test')`)
// Newer message (first_seen T2 > T1)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'newhash2', '2026-01-15T10:05:00Z', 1, 5,
'{"type":"CHAN","channel":"#test","text":"Bob: New message","sender":"Bob"}')`)
'{"type":"CHAN","channel":"#test","text":"Bob: New message","sender":"Bob"}', '#test')`)
// Observations: older message re-observed AFTER newer message (stale scenario)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, timestamp)
@@ -1325,6 +1556,61 @@ func TestGetChannelsStaleMessage(t *testing.T) {
}
}
func TestGetChannelsRegionFiltering(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer1', 'SJC')`)
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs2', 'Observer2', 'SFO')`)
// Channel message seen only in SJC
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'hash1', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#sjc-only","text":"Alice: Hello SJC","sender":"Alice"}', '#sjc-only')`)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, timestamp)
VALUES (1, 1, 12.0, -90, 1736935200)`)
// Channel message seen only in SFO
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'hash2', '2026-01-15T10:05:00Z', 1, 5,
'{"type":"CHAN","channel":"#sfo-only","text":"Bob: Hello SFO","sender":"Bob"}', '#sfo-only')`)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, timestamp)
VALUES (2, 2, 14.0, -88, 1736935500)`)
// No region filter — both channels
all, err := db.GetChannels()
if err != nil {
t.Fatal(err)
}
if len(all) != 2 {
t.Fatalf("expected 2 channels without region filter, got %d", len(all))
}
// Filter SJC — only #sjc-only
sjc, err := db.GetChannels("SJC")
if err != nil {
t.Fatal(err)
}
if len(sjc) != 1 {
t.Fatalf("expected 1 channel for SJC, got %d", len(sjc))
}
if sjc[0]["name"] != "#sjc-only" {
t.Errorf("expected channel '#sjc-only', got %q", sjc[0]["name"])
}
// Filter SFO — only #sfo-only
sfo, err := db.GetChannels("SFO")
if err != nil {
t.Fatal(err)
}
if len(sfo) != 1 {
t.Fatalf("expected 1 channel for SFO, got %d", len(sfo))
}
if sfo[0]["name"] != "#sfo-only" {
t.Errorf("expected channel '#sfo-only', got %q", sfo[0]["name"])
}
}
func TestNodeTelemetryFields(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
@@ -1375,3 +1661,423 @@ func TestNodeTelemetryFields(t *testing.T) {
func TestMain(m *testing.M) {
os.Exit(m.Run())
}
func TestGetObserverMetrics(t *testing.T) {
db := setupTestDB(t)
seedTestData(t, db)
now := time.Now().UTC()
t1 := now.Add(-2 * time.Hour).Format(time.RFC3339)
t2 := now.Add(-1 * time.Hour).Format(time.RFC3339)
t3 := now.Format(time.RFC3339)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors, battery_mv) VALUES (?, ?, ?, ?, ?, ?, ?)",
"obs1", t1, -112.5, 100, 500, 3, 3720)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors) VALUES (?, ?, ?, ?, ?, ?)",
"obs1", t2, -110.0, 200, 800, 5)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors) VALUES (?, ?, ?, ?, ?, ?)",
"obs1", t3, -108.0, 300, 1100, 8)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor) VALUES (?, ?, ?)",
"obs2", t1, -115.0)
// Query all for obs1
since := now.Add(-3 * time.Hour).Format(time.RFC3339)
metrics, reboots, err := db.GetObserverMetrics("obs1", since, "", "5m", 3600)
if err != nil {
t.Fatal(err)
}
if len(metrics) != 3 {
t.Errorf("expected 3 metrics, got %d", len(metrics))
}
if len(reboots) != 0 {
t.Errorf("expected 0 reboots, got %d", len(reboots))
}
// Verify first row has noise_floor
if metrics[0].NoiseFloor == nil || *metrics[0].NoiseFloor != -112.5 {
t.Errorf("first noise_floor = %v, want -112.5", metrics[0].NoiseFloor)
}
// First row: no delta possible (first sample)
if metrics[0].TxAirtimePct != nil {
t.Errorf("first sample should have nil tx_airtime_pct, got %v", *metrics[0].TxAirtimePct)
}
// Second row should have computed deltas
// TX: (200-100) / 3600 * 100 ≈ 2.78%
if metrics[1].TxAirtimePct == nil {
t.Errorf("second sample tx_airtime_pct should not be nil")
} else if *metrics[1].TxAirtimePct < 2.0 || *metrics[1].TxAirtimePct > 3.5 {
t.Errorf("second sample tx_airtime_pct = %v, want ~2.78", *metrics[1].TxAirtimePct)
}
// Query with until filter
metrics2, _, err := db.GetObserverMetrics("obs1", since, t2, "5m", 3600)
if err != nil {
t.Fatal(err)
}
if len(metrics2) != 2 {
t.Errorf("expected 2 metrics with until filter, got %d", len(metrics2))
}
}
func TestGetMetricsSummary(t *testing.T) {
db := setupTestDB(t)
seedTestData(t, db)
now := time.Now().UTC()
t1 := now.Add(-2 * time.Hour).Format(time.RFC3339)
t2 := now.Add(-1 * time.Hour).Format(time.RFC3339)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, battery_mv) VALUES (?, ?, ?, ?)",
"obs1", t1, -112.0, 3720)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor) VALUES (?, ?, ?)",
"obs1", t2, -108.0)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor) VALUES (?, ?, ?)",
"obs2", t1, -115.0)
since := now.Add(-24 * time.Hour).Format(time.RFC3339)
summary, err := db.GetMetricsSummary(since)
if err != nil {
t.Fatal(err)
}
if len(summary) != 2 {
t.Fatalf("expected 2 observers in summary, got %d", len(summary))
}
// Results sorted by max_nf DESC
// obs1 has max -108, obs2 has max -115
if summary[0].ObserverID != "obs1" {
t.Errorf("first observer should be obs1 (highest max NF), got %s", summary[0].ObserverID)
}
if summary[0].CurrentNF == nil || *summary[0].CurrentNF != -108.0 {
t.Errorf("obs1 current NF = %v, want -108.0", summary[0].CurrentNF)
}
if summary[0].SampleCount != 2 {
t.Errorf("obs1 sample count = %d, want 2", summary[0].SampleCount)
}
// Verify sparkline data is included
if len(summary[0].Sparkline) != 2 {
t.Errorf("obs1 sparkline length = %d, want 2", len(summary[0].Sparkline))
}
if len(summary[1].Sparkline) != 1 {
t.Errorf("obs2 sparkline length = %d, want 1", len(summary[1].Sparkline))
}
// Sparkline should be ordered by timestamp ASC
if summary[0].Sparkline[0] != nil && *summary[0].Sparkline[0] != -112.0 {
t.Errorf("obs1 sparkline[0] = %v, want -112.0", *summary[0].Sparkline[0])
}
if summary[0].Sparkline[1] != nil && *summary[0].Sparkline[1] != -108.0 {
t.Errorf("obs1 sparkline[1] = %v, want -108.0", *summary[0].Sparkline[1])
}
}
func TestObserverMetricsAPIEndpoints(t *testing.T) {
db := setupTestDB(t)
seedTestData(t, db)
now := time.Now().UTC()
t1 := now.Add(-1 * time.Hour).Format(time.RFC3339)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor) VALUES (?, ?, ?)",
"obs1", t1, -112.0)
// Query directly to verify
metrics, _, err := db.GetObserverMetrics("obs1", "", "", "5m", 300)
if err != nil {
t.Fatal(err)
}
if len(metrics) != 1 {
t.Errorf("expected 1 metric, got %d", len(metrics))
}
}
func TestComputeDeltas(t *testing.T) {
intPtr := func(v int) *int { return &v }
floatPtr := func(v float64) *float64 { return &v }
t.Run("empty input", func(t *testing.T) {
result, reboots, err := computeDeltas(nil, 300)
if err != nil {
t.Fatal(err)
}
if result != nil {
t.Errorf("expected nil, got %v", result)
}
if reboots != nil {
t.Errorf("expected nil reboots, got %v", reboots)
}
})
t.Run("normal delta computation", func(t *testing.T) {
raw := []rawMetricsSample{
{Timestamp: "2026-04-05T00:00:00Z", NoiseFloor: floatPtr(-112), TxAirSecs: intPtr(100), RxAirSecs: intPtr(500), RecvErrors: intPtr(3), PacketsRecv: intPtr(1000)},
{Timestamp: "2026-04-05T00:05:00Z", NoiseFloor: floatPtr(-110), TxAirSecs: intPtr(115), RxAirSecs: intPtr(525), RecvErrors: intPtr(5), PacketsRecv: intPtr(1100)},
}
result, reboots, err := computeDeltas(raw, 300)
if err != nil {
t.Fatal(err)
}
if len(result) != 2 {
t.Fatalf("expected 2 results, got %d", len(result))
}
if len(reboots) != 0 {
t.Errorf("expected 0 reboots, got %d", len(reboots))
}
// First sample: no deltas
if result[0].TxAirtimePct != nil {
t.Errorf("first sample should have nil tx_airtime_pct")
}
// Second sample: TX delta = 15 secs / 300 secs * 100 = 5%
if result[1].TxAirtimePct == nil {
t.Fatal("second sample tx_airtime_pct should not be nil")
}
if *result[1].TxAirtimePct != 5.0 {
t.Errorf("tx_airtime_pct = %v, want 5.0", *result[1].TxAirtimePct)
}
// RX delta = 25 secs / 300 secs * 100 ≈ 8.33%
if result[1].RxAirtimePct == nil {
t.Fatal("second sample rx_airtime_pct should not be nil")
}
if *result[1].RxAirtimePct < 8.3 || *result[1].RxAirtimePct > 8.4 {
t.Errorf("rx_airtime_pct = %v, want ~8.33", *result[1].RxAirtimePct)
}
// Error rate: delta_errors=2, delta_recv=100, rate = 2/(100+2)*100 ≈ 1.96%
if result[1].RecvErrorRate == nil {
t.Fatal("second sample recv_error_rate should not be nil")
}
if *result[1].RecvErrorRate < 1.9 || *result[1].RecvErrorRate > 2.0 {
t.Errorf("recv_error_rate = %v, want ~1.96", *result[1].RecvErrorRate)
}
})
t.Run("reboot detection", func(t *testing.T) {
raw := []rawMetricsSample{
{Timestamp: "2026-04-05T00:00:00Z", TxAirSecs: intPtr(1000), RxAirSecs: intPtr(5000)},
{Timestamp: "2026-04-05T00:05:00Z", TxAirSecs: intPtr(10), RxAirSecs: intPtr(20)}, // reboot!
{Timestamp: "2026-04-05T00:10:00Z", TxAirSecs: intPtr(25), RxAirSecs: intPtr(45)},
}
result, reboots, err := computeDeltas(raw, 300)
if err != nil {
t.Fatal(err)
}
if len(reboots) != 1 {
t.Fatalf("expected 1 reboot, got %d", len(reboots))
}
if reboots[0] != "2026-04-05T00:05:00Z" {
t.Errorf("reboot timestamp = %s", reboots[0])
}
if !result[1].IsReboot {
t.Error("second sample should be marked as reboot")
}
// Reboot sample should have nil deltas
if result[1].TxAirtimePct != nil {
t.Error("reboot sample should have nil tx_airtime_pct")
}
// Third sample should have valid deltas from post-reboot baseline
if result[2].TxAirtimePct == nil {
t.Fatal("third sample tx_airtime_pct should not be nil")
}
if *result[2].TxAirtimePct != 5.0 { // 15/300*100
t.Errorf("third sample tx_airtime_pct = %v, want 5.0", *result[2].TxAirtimePct)
}
})
t.Run("gap detection", func(t *testing.T) {
raw := []rawMetricsSample{
{Timestamp: "2026-04-05T00:00:00Z", TxAirSecs: intPtr(100)},
{Timestamp: "2026-04-05T00:15:00Z", TxAirSecs: intPtr(200)}, // 15min gap > 2*300s
}
result, _, err := computeDeltas(raw, 300)
if err != nil {
t.Fatal(err)
}
// Gap sample should have nil deltas
if result[1].TxAirtimePct != nil {
t.Error("gap sample should have nil tx_airtime_pct")
}
})
}
func TestGetObserverMetricsResolution(t *testing.T) {
db := setupTestDB(t)
seedTestData(t, db)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs) VALUES (?, ?, ?, ?)",
"obs1", "2026-04-05T00:00:00Z", -112.0, 100)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs) VALUES (?, ?, ?, ?)",
"obs1", "2026-04-05T00:05:00Z", -110.0, 200)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs) VALUES (?, ?, ?, ?)",
"obs1", "2026-04-05T01:00:00Z", -108.0, 500)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs) VALUES (?, ?, ?, ?)",
"obs1", "2026-04-05T01:05:00Z", -106.0, 600)
// 5m resolution: all 4 rows
m5, _, err := db.GetObserverMetrics("obs1", "2026-04-04T00:00:00Z", "", "5m", 300)
if err != nil {
t.Fatal(err)
}
if len(m5) != 4 {
t.Errorf("5m resolution: expected 4 rows, got %d", len(m5))
}
// 1h resolution: 2 buckets
m1h, _, err := db.GetObserverMetrics("obs1", "2026-04-04T00:00:00Z", "", "1h", 300)
if err != nil {
t.Fatal(err)
}
if len(m1h) != 2 {
t.Errorf("1h resolution: expected 2 rows, got %d", len(m1h))
}
// 1d resolution: 1 bucket
m1d, _, err := db.GetObserverMetrics("obs1", "2026-04-04T00:00:00Z", "", "1d", 300)
if err != nil {
t.Fatal(err)
}
if len(m1d) != 1 {
t.Errorf("1d resolution: expected 1 row, got %d", len(m1d))
}
}
func TestHourlyResolutionDeltasNotNull(t *testing.T) {
db := setupTestDB(t)
seedTestData(t, db)
// Two hourly buckets, each with one sample. With old MAX+hardcoded gap threshold,
// the 3600s gap would exceed sampleInterval*2 (600s) and deltas would be null.
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors, packets_sent, packets_recv) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
"obs_hr", "2026-04-05T10:00:00Z", -110.0, 100, 200, 5, 50, 100)
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors, packets_sent, packets_recv) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
"obs_hr", "2026-04-05T11:00:00Z", -108.0, 200, 400, 10, 80, 200)
m, _, err := db.GetObserverMetrics("obs_hr", "2026-04-04T00:00:00Z", "", "1h", 300)
if err != nil {
t.Fatal(err)
}
if len(m) != 2 {
t.Fatalf("expected 2 rows, got %d", len(m))
}
// Second row should have computed deltas (not null)
if m[1].TxAirtimePct == nil {
t.Error("1h resolution: tx_airtime_pct should not be nil — gap threshold must scale with resolution")
}
}
func TestLastValuePreservesReboot(t *testing.T) {
db := setupTestDB(t)
seedTestData(t, db)
// Hour bucket with two samples: pre-reboot (high) and post-reboot (low).
// With MAX(), the pre-reboot value wins and the reboot is hidden.
// With LAST (latest timestamp), the post-reboot value wins.
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors, packets_sent, packets_recv) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
"obs_rb", "2026-04-05T10:00:00Z", -110.0, 1000, 2000, 500, 400, 800) // pre-reboot baseline
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors, packets_sent, packets_recv) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
"obs_rb", "2026-04-05T10:20:00Z", -110.0, 5000, 6000, 900, 700, 1200) // pre-reboot peak
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors, packets_sent, packets_recv) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
"obs_rb", "2026-04-05T10:40:00Z", -110.0, 10, 20, 1, 5, 10) // post-reboot (counter reset)
// Next hour bucket
db.conn.Exec("INSERT INTO observer_metrics (observer_id, timestamp, noise_floor, tx_air_secs, rx_air_secs, recv_errors, packets_sent, packets_recv) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
"obs_rb", "2026-04-05T11:00:00Z", -108.0, 100, 120, 5, 20, 50)
m, reboots, err := db.GetObserverMetrics("obs_rb", "2026-04-04T00:00:00Z", "", "1h", 300)
if err != nil {
t.Fatal(err)
}
if len(m) != 2 {
t.Fatalf("expected 2 rows, got %d", len(m))
}
// First bucket should use the LAST value (post-reboot: tx_air_secs=10).
// Second bucket (tx_air_secs=100) is a normal increase from 10→100.
// With LAST-value semantics, the second bucket should have valid deltas (not a reboot).
// With MAX(), first bucket would have tx_air_secs=5000, and second=100 would
// trigger a false reboot detection.
if m[1].IsReboot {
t.Error("second bucket should NOT be flagged as reboot with LAST-value aggregation")
}
if m[1].TxAirtimePct == nil {
t.Error("second bucket should have non-nil tx_airtime_pct")
}
_ = reboots // reboots list is informational
}
func TestParseWindowDuration(t *testing.T) {
tests := []struct {
input string
want time.Duration
err bool
}{
{"1h", time.Hour, false},
{"24h", 24 * time.Hour, false},
{"3d", 3 * 24 * time.Hour, false},
{"30d", 30 * 24 * time.Hour, false},
{"invalid", 0, true},
}
for _, tc := range tests {
got, err := parseWindowDuration(tc.input)
if tc.err && err == nil {
t.Errorf("parseWindowDuration(%q) expected error", tc.input)
}
if !tc.err && got != tc.want {
t.Errorf("parseWindowDuration(%q) = %v, want %v", tc.input, got, tc.want)
}
}
}
// TestPerObservationRawHexEnrich verifies enrichObs returns per-observation raw_hex
// when available, falling back to transmission raw_hex when NULL (#881).
func TestPerObservationRawHexEnrich(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
// Insert observers
db.conn.Exec(`INSERT INTO observers (id, name) VALUES ('obs-a', 'Observer A')`)
db.conn.Exec(`INSERT INTO observers (id, name) VALUES ('obs-b', 'Observer B')`)
var rowA, rowB int64
db.conn.QueryRow(`SELECT rowid FROM observers WHERE id='obs-a'`).Scan(&rowA)
db.conn.QueryRow(`SELECT rowid FROM observers WHERE id='obs-b'`).Scan(&rowB)
// Insert transmission with raw_hex
txHex := "deadbeef"
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen) VALUES (?, 'hash1', '2026-04-21T10:00:00Z')`, txHex)
// Insert two observations: A has its own raw_hex, B has NULL (historical)
obsAHex := "c0ffee01"
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp, raw_hex)
VALUES (1, ?, -5.0, -90.0, '[]', 1745236800, ?)`, rowA, obsAHex)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, ?, -3.0, -85.0, '["aabb"]', 1745236801)`, rowB)
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatalf("store load: %v", err)
}
tx := store.byHash["hash1"]
if tx == nil {
t.Fatal("transmission not loaded")
}
if len(tx.Observations) < 2 {
t.Fatalf("expected 2 observations, got %d", len(tx.Observations))
}
// Check enriched observations
for _, obs := range tx.Observations {
m := store.enrichObs(obs)
rh, _ := m["raw_hex"].(string)
if obs.RawHex != "" {
// Observer A: should get per-observation raw_hex
if rh != obsAHex {
t.Errorf("obs with own raw_hex: got %q, want %q", rh, obsAHex)
}
} else {
// Observer B: should fall back to transmission raw_hex
if rh != txHex {
t.Errorf("obs without raw_hex: got %q, want %q (tx fallback)", rh, txHex)
}
}
}
}
+76 -16
View File
@@ -9,6 +9,9 @@ import (
"math"
"strings"
"time"
"github.com/meshcore-analyzer/packetpath"
"github.com/meshcore-analyzer/sigvalidate"
)
// Route type constants (header bits 1-0)
@@ -60,9 +63,10 @@ type TransportCodes struct {
// Path holds decoded path/hop information.
type Path struct {
HashSize int `json:"hashSize"`
HashCount int `json:"hashCount"`
Hops []string `json:"hops"`
HashSize int `json:"hashSize"`
HashCount int `json:"hashCount"`
Hops []string `json:"hops"`
HopsCompleted *int `json:"hopsCompleted,omitempty"`
}
// AdvertFlags holds decoded advert flag bits.
@@ -91,6 +95,7 @@ type Payload struct {
Timestamp uint32 `json:"timestamp,omitempty"`
TimestampISO string `json:"timestampISO,omitempty"`
Signature string `json:"signature,omitempty"`
SignatureValid *bool `json:"signatureValid,omitempty"`
Flags *AdvertFlags `json:"flags,omitempty"`
Lat *float64 `json:"lat,omitempty"`
Lon *float64 `json:"lon,omitempty"`
@@ -112,6 +117,7 @@ type DecodedPacket struct {
Path Path `json:"path"`
Payload Payload `json:"payload"`
Raw string `json:"raw"`
Anomaly string `json:"anomaly,omitempty"`
}
func decodeHeader(b byte) Header {
@@ -159,8 +165,9 @@ func decodePath(pathByte byte, buf []byte, offset int) (Path, int) {
}, totalBytes
}
// isTransportRoute delegates to packetpath.IsTransportRoute.
func isTransportRoute(routeType int) bool {
return routeType == RouteTransportFlood || routeType == RouteTransportDirect
return packetpath.IsTransportRoute(routeType)
}
func decodeEncryptedPayload(typeName string, buf []byte) Payload {
@@ -187,7 +194,7 @@ func decodeAck(buf []byte) Payload {
}
}
func decodeAdvert(buf []byte) Payload {
func decodeAdvert(buf []byte, validateSignatures bool) Payload {
if len(buf) < 100 {
return Payload{Type: "ADVERT", Error: "too short for advert", RawHex: hex.EncodeToString(buf)}
}
@@ -205,6 +212,16 @@ func decodeAdvert(buf []byte) Payload {
Signature: signature,
}
if validateSignatures {
valid, err := sigvalidate.ValidateAdvert(buf[0:32], buf[36:100], timestamp, appdata)
if err != nil {
f := false
p.SignatureValid = &f
} else {
p.SignatureValid = &valid
}
}
if len(appdata) > 0 {
flags := appdata[0]
advType := int(flags & 0x0F)
@@ -307,7 +324,7 @@ func decodeTrace(buf []byte) Payload {
return p
}
func decodePayload(payloadType int, buf []byte) Payload {
func decodePayload(payloadType int, buf []byte, validateSignatures bool) Payload {
switch payloadType {
case PayloadREQ:
return decodeEncryptedPayload("REQ", buf)
@@ -318,7 +335,7 @@ func decodePayload(payloadType int, buf []byte) Payload {
case PayloadACK:
return decodeAck(buf)
case PayloadADVERT:
return decodeAdvert(buf)
return decodeAdvert(buf, validateSignatures)
case PayloadGRP_TXT:
return decodeGrpTxt(buf)
case PayloadANON_REQ:
@@ -333,7 +350,7 @@ func decodePayload(payloadType int, buf []byte) Payload {
}
// DecodePacket decodes a hex-encoded MeshCore packet.
func DecodePacket(hexString string) (*DecodedPacket, error) {
func DecodePacket(hexString string, validateSignatures bool) (*DecodedPacket, error) {
hexString = strings.ReplaceAll(hexString, " ", "")
hexString = strings.ReplaceAll(hexString, "\n", "")
hexString = strings.ReplaceAll(hexString, "\r", "")
@@ -371,33 +388,65 @@ func DecodePacket(hexString string) (*DecodedPacket, error) {
offset += bytesConsumed
payloadBuf := buf[offset:]
payload := decodePayload(header.PayloadType, payloadBuf)
payload := decodePayload(header.PayloadType, payloadBuf, validateSignatures)
// TRACE packets store hop IDs in the payload (buf[9:]) rather than the header
// path field. The header path byte still encodes hashSize in bits 6-7, which
// we use to split the payload path data into individual hop prefixes.
// path field. Firmware always sends TRACE as DIRECT (route_type 2 or 3);
// FLOOD-routed TRACEs are anomalous but handled gracefully (parsed, but
// flagged). The TRACE flags byte (payload offset 8) encodes path_sz in
// bits 0-1 as a power-of-two exponent: hash_bytes = 1 << path_sz.
// NOT the header path byte's hash_size bits. The header path contains SNR
// bytes — one per hop that actually forwarded.
// We expose hopsCompleted (count of SNR bytes) so consumers can distinguish
// how far the trace got vs the full intended route.
var anomaly string
if header.PayloadType == PayloadTRACE && payload.PathData != "" {
// Flag anomalous routing — firmware only sends TRACE as DIRECT
if header.RouteType != RouteDirect && header.RouteType != RouteTransportDirect {
anomaly = "TRACE packet with non-DIRECT routing (expected DIRECT or TRANSPORT_DIRECT)"
}
// The header path hops count represents SNR entries = completed hops
hopsCompleted := path.HashCount
pathBytes, err := hex.DecodeString(payload.PathData)
if err == nil && path.HashSize > 0 {
hops := make([]string, 0, len(pathBytes)/path.HashSize)
for i := 0; i+path.HashSize <= len(pathBytes); i += path.HashSize {
hops = append(hops, strings.ToUpper(hex.EncodeToString(pathBytes[i:i+path.HashSize])))
if err == nil && payload.TraceFlags != nil {
// path_sz from flags byte is a power-of-two exponent per firmware:
// hash_bytes = 1 << (flags & 0x03)
pathSz := 1 << (*payload.TraceFlags & 0x03)
hops := make([]string, 0, len(pathBytes)/pathSz)
for i := 0; i+pathSz <= len(pathBytes); i += pathSz {
hops = append(hops, strings.ToUpper(hex.EncodeToString(pathBytes[i:i+pathSz])))
}
path.Hops = hops
path.HashCount = len(hops)
path.HashSize = pathSz
path.HopsCompleted = &hopsCompleted
}
}
// Zero-hop direct packets have hash_count=0 (lower 6 bits of pathByte),
// which makes the generic formula yield a bogus hashSize. Reset to 0
// (unknown) so API consumers get correct data. We mask with 0x3F to check
// only hash_count, matching the JS frontend approach — the upper hash_size
// bits are meaningless when there are no hops. Skip TRACE packets — they
// use hashSize to parse hops from the payload above.
if (header.RouteType == RouteDirect || header.RouteType == RouteTransportDirect) && pathByte&0x3F == 0 && header.PayloadType != PayloadTRACE {
path.HashSize = 0
}
return &DecodedPacket{
Header: header,
TransportCodes: tc,
Path: path,
Payload: payload,
Raw: strings.ToUpper(hexString),
Anomaly: anomaly,
}, nil
}
// ComputeContentHash computes the SHA-256-based content hash (first 16 hex chars).
// It hashes the payload-type nibble + payload (skipping path bytes) to produce a
// route-independent identifier for the same logical packet. For TRACE packets,
// path_len is included in the hash to match firmware behavior.
func ComputeContentHash(rawHex string) string {
buf, err := hex.DecodeString(rawHex)
if err != nil || len(buf) < 2 {
@@ -433,7 +482,18 @@ func ComputeContentHash(rawHex string) string {
}
payload := buf[payloadStart:]
toHash := append([]byte{headerByte}, payload...)
// Hash payload-type byte only (bits 2-5 of header), not the full header.
// Firmware: SHA256(payload_type + [path_len for TRACE] + payload)
// Using the full header caused different hashes for the same logical packet
// when route type or version bits differed. See issue #786.
payloadType := (headerByte >> 2) & 0x0F
toHash := []byte{payloadType}
if int(payloadType) == PayloadTRACE {
// Firmware uses uint16_t path_len (2 bytes, little-endian)
toHash = append(toHash, pathByte, 0x00)
}
toHash = append(toHash, payload...)
h := sha256.Sum256(toHash)
return hex.EncodeToString(h[:])[:16]
+349 -2
View File
@@ -1,6 +1,9 @@
package main
import (
"crypto/ed25519"
"encoding/binary"
"encoding/hex"
"testing"
)
@@ -65,7 +68,7 @@ func TestDecodePacket_TransportFloodHasCodes(t *testing.T) {
// Path byte: 0x00 (hashSize=1, hashCount=0)
// Payload: at least some bytes for GRP_TXT
hex := "14AABBCCDD00112233445566778899"
pkt, err := DecodePacket(hex)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
@@ -85,7 +88,7 @@ func TestDecodePacket_FloodHasNoCodes(t *testing.T) {
// Path byte: 0x00 (no hops)
// Some payload bytes
hex := "110011223344556677889900AABBCCDD"
pkt, err := DecodePacket(hex)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
@@ -93,3 +96,347 @@ func TestDecodePacket_FloodHasNoCodes(t *testing.T) {
t.Error("expected no transport codes for FLOOD route")
}
}
func TestZeroHopDirectHashSize(t *testing.T) {
// DIRECT (RouteType=2) + REQ (PayloadType=0) → header byte = 0x02
// pathByte=0x00 → hash_count=0, hash_size bits=0 → should get HashSize=0
// Need at least a few payload bytes after pathByte.
hex := "02" + "00" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 0 {
t.Errorf("DIRECT zero-hop: want HashSize=0, got %d", pkt.Path.HashSize)
}
}
func TestZeroHopDirectHashSizeWithNonZeroUpperBits(t *testing.T) {
// DIRECT (RouteType=2) + REQ (PayloadType=0) → header byte = 0x02
// pathByte=0x40 → hash_count=0, hash_size bits=01 → should still get HashSize=0
// because hash_count is zero (lower 6 bits are 0).
hex := "02" + "40" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 0 {
t.Errorf("DIRECT zero-hop with hash_size bits set: want HashSize=0, got %d", pkt.Path.HashSize)
}
}
func TestZeroHopTransportDirectHashSize(t *testing.T) {
// TRANSPORT_DIRECT (RouteType=3) + REQ (PayloadType=0) → header byte = 0x03
// 4 bytes transport codes + pathByte=0x00 → hash_count=0 → should get HashSize=0
hex := "03" + "11223344" + "00" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 0 {
t.Errorf("TRANSPORT_DIRECT zero-hop: want HashSize=0, got %d", pkt.Path.HashSize)
}
}
func TestZeroHopTransportDirectHashSizeWithNonZeroUpperBits(t *testing.T) {
// TRANSPORT_DIRECT (RouteType=3) + REQ (PayloadType=0) → header byte = 0x03
// 4 bytes transport codes + pathByte=0xC0 → hash_count=0, hash_size bits=11 → should still get HashSize=0
hex := "03" + "11223344" + "C0" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 0 {
t.Errorf("TRANSPORT_DIRECT zero-hop with hash_size bits set: want HashSize=0, got %d", pkt.Path.HashSize)
}
}
func TestNonDirectZeroPathByteKeepsHashSize(t *testing.T) {
// FLOOD (RouteType=1) + REQ (PayloadType=0) → header byte = 0x01
// pathByte=0x00 → even though hash_count=0, non-DIRECT should keep HashSize=1
hex := "01" + "00" + repeatHex("AA", 20)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 1 {
t.Errorf("FLOOD zero pathByte: want HashSize=1 (unchanged), got %d", pkt.Path.HashSize)
}
}
func TestDirectNonZeroHopKeepsHashSize(t *testing.T) {
// DIRECT (RouteType=2) + REQ (PayloadType=0) → header byte = 0x02
// pathByte=0x01 → hash_count=1, hash_size=1 → should keep HashSize=1
// Need 1 hop hash byte after pathByte.
hex := "02" + "01" + repeatHex("BB", 21)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket failed: %v", err)
}
if pkt.Path.HashSize != 1 {
t.Errorf("DIRECT with 1 hop: want HashSize=1, got %d", pkt.Path.HashSize)
}
}
func repeatHex(byteHex string, n int) string {
s := ""
for i := 0; i < n; i++ {
s += byteHex
}
return s
}
func TestDecodePacket_TraceHopsCompleted(t *testing.T) {
// Build a TRACE packet:
// header: route=FLOOD(1), payload=TRACE(9), version=0 → (0<<6)|(9<<2)|1 = 0x25
// path_length: hash_size bits=0b00 (1-byte), hash_count=2 (2 SNR bytes) → 0x02
// path: 2 SNR bytes: 0xAA, 0xBB
// payload: tag(4 LE) + authCode(4 LE) + flags(1) + 4 hop hashes (1 byte each)
hex := "2502AABB" + // header + path_length + 2 SNR bytes
"01000000" + // tag = 1
"02000000" + // authCode = 2
"00" + // flags = 0
"DEADBEEF" // 4 hops (1-byte hash each)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if pkt.Payload.Type != "TRACE" {
t.Fatalf("expected TRACE, got %s", pkt.Payload.Type)
}
// Full intended route = 4 hops from payload
if len(pkt.Path.Hops) != 4 {
t.Errorf("expected 4 hops, got %d: %v", len(pkt.Path.Hops), pkt.Path.Hops)
}
// HopsCompleted = 2 (from header path SNR count)
if pkt.Path.HopsCompleted == nil {
t.Fatal("expected HopsCompleted to be set")
}
if *pkt.Path.HopsCompleted != 2 {
t.Errorf("expected HopsCompleted=2, got %d", *pkt.Path.HopsCompleted)
}
// FLOOD routing for TRACE is anomalous
if pkt.Anomaly == "" {
t.Error("expected anomaly flag for FLOOD-routed TRACE")
}
}
func TestDecodePacket_TraceNoSNR(t *testing.T) {
// TRACE with 0 SNR bytes (trace hasn't been forwarded yet)
// path_length: hash_size=0b00 (1-byte), hash_count=0 → 0x00
hex := "2500" + // header + path_length (0 hops in header)
"01000000" + // tag
"02000000" + // authCode
"00" + // flags
"AABBCC" // 3 hops intended
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if pkt.Path.HopsCompleted == nil {
t.Fatal("expected HopsCompleted to be set")
}
if *pkt.Path.HopsCompleted != 0 {
t.Errorf("expected HopsCompleted=0, got %d", *pkt.Path.HopsCompleted)
}
if len(pkt.Path.Hops) != 3 {
t.Errorf("expected 3 hops, got %d", len(pkt.Path.Hops))
}
}
func TestDecodePacket_TraceFullyCompleted(t *testing.T) {
// TRACE where all hops completed (SNR count = hop count)
// path_length: hash_size=0b00 (1-byte), hash_count=3 → 0x03
hex := "2503AABBCC" + // header + path_length + 3 SNR bytes
"01000000" + // tag
"02000000" + // authCode
"00" + // flags
"DDEEFF" // 3 hops intended
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if pkt.Path.HopsCompleted == nil {
t.Fatal("expected HopsCompleted to be set")
}
if *pkt.Path.HopsCompleted != 3 {
t.Errorf("expected HopsCompleted=3, got %d", *pkt.Path.HopsCompleted)
}
if len(pkt.Path.Hops) != 3 {
t.Errorf("expected 3 hops, got %d", len(pkt.Path.Hops))
}
}
func TestDecodePacket_TraceFlags1_TwoBytePathSz(t *testing.T) {
// TRACE with flags=1 → path_sz = 1 << (1 & 0x03) = 2-byte hashes
// Firmware always sends TRACE as DIRECT (route_type=2), so header byte =
// (0<<6)|(9<<2)|2 = 0x26. path_length 0x00 = 0 SNR bytes.
hex := "2600" + // header (DIRECT+TRACE) + path_length (0 SNR)
"01000000" + // tag
"02000000" + // authCode
"01" + // flags = 1 → path_sz = 2
"AABBCCDD" // 4 bytes = 2 hops of 2-byte each
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if len(pkt.Path.Hops) != 2 {
t.Errorf("expected 2 hops (2-byte path_sz), got %d: %v", len(pkt.Path.Hops), pkt.Path.Hops)
}
if pkt.Path.HashSize != 2 {
t.Errorf("expected HashSize=2, got %d", pkt.Path.HashSize)
}
if pkt.Anomaly != "" {
t.Errorf("expected no anomaly for DIRECT TRACE, got %q", pkt.Anomaly)
}
}
func TestDecodePacket_TraceFlags2_FourBytePathSz(t *testing.T) {
// TRACE with flags=2 → path_sz = 1 << (2 & 0x03) = 4-byte hashes
// DIRECT route_type (0x26)
hex := "2600" + // header (DIRECT+TRACE) + path_length (0 SNR)
"01000000" + // tag
"02000000" + // authCode
"02" + // flags = 2 → path_sz = 4
"AABBCCDD11223344" // 8 bytes = 2 hops of 4-byte each
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if len(pkt.Path.Hops) != 2 {
t.Errorf("expected 2 hops (4-byte path_sz), got %d: %v", len(pkt.Path.Hops), pkt.Path.Hops)
}
if pkt.Path.HashSize != 4 {
t.Errorf("expected HashSize=4, got %d", pkt.Path.HashSize)
}
}
func TestDecodePacket_TracePathSzUnevenPayload(t *testing.T) {
// TRACE with flags=1 → path_sz=2, but 5 bytes of path data (not evenly divisible)
// Should produce 2 hops (4 bytes) and ignore the trailing byte
hex := "2600" + // header (DIRECT+TRACE) + path_length (0 SNR)
"01000000" + // tag
"02000000" + // authCode
"01" + // flags = 1 → path_sz = 2
"AABBCCDDEE" // 5 bytes → 2 hops, 1 byte remainder ignored
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if len(pkt.Path.Hops) != 2 {
t.Errorf("expected 2 hops (trailing byte ignored), got %d: %v", len(pkt.Path.Hops), pkt.Path.Hops)
}
}
func TestDecodePacket_TraceTransportDirect(t *testing.T) {
// TRACE via TRANSPORT_DIRECT (route_type=3) — includes 4 transport code bytes
// header: (0<<6)|(9<<2)|3 = 0x27
hex := "27" + // header (TRANSPORT_DIRECT+TRACE)
"AABB" + "CCDD" + // transport codes (2+2 bytes)
"02" + // path_length: hash_count=2 SNR bytes
"EEFF" + // 2 SNR bytes
"01000000" + // tag
"02000000" + // authCode
"00" + // flags = 0 → path_sz = 1
"112233" // 3 hops (1-byte each)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if pkt.TransportCodes == nil {
t.Fatal("expected transport codes for TRANSPORT_DIRECT")
}
if pkt.TransportCodes.Code1 != "AABB" {
t.Errorf("expected Code1=AABB, got %s", pkt.TransportCodes.Code1)
}
if len(pkt.Path.Hops) != 3 {
t.Errorf("expected 3 hops, got %d: %v", len(pkt.Path.Hops), pkt.Path.Hops)
}
if pkt.Path.HopsCompleted == nil || *pkt.Path.HopsCompleted != 2 {
t.Errorf("expected HopsCompleted=2, got %v", pkt.Path.HopsCompleted)
}
if pkt.Anomaly != "" {
t.Errorf("expected no anomaly for TRANSPORT_DIRECT TRACE, got %q", pkt.Anomaly)
}
}
func TestDecodePacket_TraceFloodRouteAnomaly(t *testing.T) {
// TRACE via FLOOD (route_type=1) — anomalous per firmware (firmware only
// sends TRACE as DIRECT). Should still parse but flag the anomaly.
hex := "2500" + // header (FLOOD+TRACE) + path_length (0 SNR)
"01000000" + // tag
"02000000" + // authCode
"01" + // flags = 1 → path_sz = 2
"AABBCCDD" // 4 bytes = 2 hops of 2-byte each
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("should not crash on anomalous FLOOD+TRACE: %v", err)
}
if len(pkt.Path.Hops) != 2 {
t.Errorf("expected 2 hops even for anomalous FLOOD route, got %d", len(pkt.Path.Hops))
}
if pkt.Anomaly == "" {
t.Error("expected anomaly flag for FLOOD-routed TRACE, got empty string")
}
}
func TestDecodeAdvertSignatureValidation(t *testing.T) {
pub, priv, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
var timestamp uint32 = 1234567890
appdata := []byte{0x02} // flags: repeater, no extras
// Build signed message: pubKey(32) + timestamp(4 LE) + appdata
msg := make([]byte, 32+4+len(appdata))
copy(msg[0:32], pub)
binary.LittleEndian.PutUint32(msg[32:36], timestamp)
copy(msg[36:], appdata)
sig := ed25519.Sign(priv, msg)
// Build a raw advert buffer: pubKey(32) + timestamp(4) + signature(64) + appdata
buf := make([]byte, 100+len(appdata))
copy(buf[0:32], pub)
binary.LittleEndian.PutUint32(buf[32:36], timestamp)
copy(buf[36:100], sig)
copy(buf[100:], appdata)
// With validation enabled
p := decodeAdvert(buf, true)
if p.SignatureValid == nil {
t.Fatal("expected SignatureValid to be set")
}
if !*p.SignatureValid {
t.Error("expected valid signature")
}
if p.PubKey != hex.EncodeToString(pub) {
t.Errorf("pubkey mismatch: got %s", p.PubKey)
}
// Tamper with signature → invalid
buf[40] ^= 0xFF
p = decodeAdvert(buf, true)
if p.SignatureValid == nil {
t.Fatal("expected SignatureValid to be set")
}
if *p.SignatureValid {
t.Error("expected invalid signature after tampering")
}
// Without validation → SignatureValid should be nil
p = decodeAdvert(buf, false)
if p.SignatureValid != nil {
t.Error("expected SignatureValid to be nil when validation disabled")
}
}
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"encoding/json"
"net/http/httptest"
"testing"
"time"
)
// seedEncryptedChannelData adds undecryptable GRP_TXT packets to the test DB.
func seedEncryptedChannelData(t *testing.T, db *DB) {
t.Helper()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
// Two encrypted GRP_TXT packets on channel hash "A1B2"
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('EE01', 'enc_hash_001', ?, 1, 5, '{"type":"GRP_TXT","channelHashHex":"A1B2","decryptionStatus":"no_key"}', 'enc_A1B2')`, recent)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('EE02', 'enc_hash_002', ?, 1, 5, '{"type":"GRP_TXT","channelHashHex":"A1B2","decryptionStatus":"no_key"}', 'enc_A1B2')`, recent)
// Observations for both
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES ((SELECT id FROM transmissions WHERE hash='enc_hash_001'), 1, 10.0, -90, '[]', ?)`, recentEpoch)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES ((SELECT id FROM transmissions WHERE hash='enc_hash_002'), 1, 10.0, -90, '[]', ?)`, recentEpoch)
}
func TestGetEncryptedChannels(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
seedTestData(t, db)
seedEncryptedChannelData(t, db)
channels, err := db.GetEncryptedChannels()
if err != nil {
t.Fatal(err)
}
if len(channels) != 1 {
t.Fatalf("expected 1 encrypted channel, got %d", len(channels))
}
ch := channels[0]
if ch["hash"] != "enc_A1B2" {
t.Errorf("expected hash enc_A1B2, got %v", ch["hash"])
}
if ch["encrypted"] != true {
t.Errorf("expected encrypted=true, got %v", ch["encrypted"])
}
if ch["messageCount"] != 2 {
t.Errorf("expected messageCount=2, got %v", ch["messageCount"])
}
}
func TestChannelsAPIExcludesEncrypted(t *testing.T) {
_, router := setupTestServer(t)
// Seed encrypted data into the server's DB
// setupTestServer uses seedTestData which has no encrypted packets,
// so default /api/channels should NOT include encrypted channels.
req := httptest.NewRequest("GET", "/api/channels", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d", w.Code)
}
var body map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &body)
channels := body["channels"].([]interface{})
for _, ch := range channels {
m := ch.(map[string]interface{})
if enc, ok := m["encrypted"]; ok && enc == true {
t.Errorf("default /api/channels should not include encrypted channels, found: %v", m["hash"])
}
}
}
func TestChannelsAPIIncludesEncryptedWithParam(t *testing.T) {
srv, router := setupTestServer(t)
// Add encrypted data to the server's DB
seedEncryptedChannelData(t, srv.db)
// Reload store so in-memory also has the data
store := NewPacketStore(srv.db, nil)
if err := store.Load(); err != nil {
t.Fatalf("store.Load: %v", err)
}
srv.store = store
req := httptest.NewRequest("GET", "/api/channels?includeEncrypted=true", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d", w.Code)
}
var body map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &body)
channels := body["channels"].([]interface{})
foundEncrypted := false
for _, ch := range channels {
m := ch.(map[string]interface{})
if enc, ok := m["encrypted"]; ok && enc == true {
foundEncrypted = true
break
}
}
if !foundEncrypted {
t.Error("expected encrypted channels with includeEncrypted=true, found none")
}
}
func TestChannelMessagesExcludesEncrypted(t *testing.T) {
srv, router := setupTestServer(t)
seedEncryptedChannelData(t, srv.db)
store := NewPacketStore(srv.db, nil)
if err := store.Load(); err != nil {
t.Fatalf("store.Load: %v", err)
}
srv.store = store
// Request messages for the encrypted channel — should return empty
req := httptest.NewRequest("GET", "/api/channels/enc_A1B2/messages", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d", w.Code)
}
var body map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &body)
messages, ok := body["messages"].([]interface{})
if !ok {
// messages might be null/missing — that's fine, means no messages
return
}
// Encrypted messages should not be returned as readable messages
for _, msg := range messages {
m := msg.(map[string]interface{})
if text, ok := m["text"].(string); ok && text != "" {
t.Errorf("encrypted channel should not return readable messages, got text: %s", text)
}
}
}
+360 -8
View File
@@ -85,6 +85,12 @@ func makeTestStore(count int, startTime time.Time, intervalMin int) *PacketStore
// Subpath index
addTxToSubpathIndex(store.spIndex, tx)
// Track bytes for self-accounting
store.trackedBytes += estimateStoreTxBytes(tx)
for _, obs := range tx.Observations {
store.trackedBytes += estimateStoreObsBytes(obs)
}
}
return store
@@ -162,21 +168,47 @@ func TestEvictStale_NoEvictionWhenDisabled(t *testing.T) {
func TestEvictStale_MemoryBasedEviction(t *testing.T) {
now := time.Now().UTC()
// Create enough packets to exceed a small memory limit
// 1000 packets * 5KB + 2000 obs * 500B ≈ 6MB
store := makeTestStore(1000, now.Add(-1*time.Hour), 0)
// All packets are recent (1h old) so time-based won't trigger
// All packets are recent (1h old) so time-based won't trigger.
store.retentionHours = 24
store.maxMemoryMB = 3 // ~3MB limit, should evict roughly half
store.maxMemoryMB = 3
// Set trackedBytes to simulate 6MB (over 3MB limit).
store.trackedBytes = 6 * 1048576
evicted := store.EvictStale()
if evicted == 0 {
t.Fatal("expected some evictions for memory cap")
}
// After eviction, estimated memory should be <= 3MB
estMB := store.estimatedMemoryMB()
if estMB > 3.5 { // small tolerance
t.Fatalf("expected <=3.5MB after eviction, got %.1fMB", estMB)
// 25% safety cap should limit to 250 per pass
if evicted > 250 {
t.Fatalf("25%% safety cap violated: evicted %d", evicted)
}
// trackedBytes should have decreased
if store.trackedBytes >= 6*1048576 {
t.Fatal("trackedBytes should have decreased after eviction")
}
}
// TestEvictStale_MemoryBasedEviction_UnderestimatedHeap verifies that the 25%
// safety cap prevents cascading eviction even when trackedBytes is very high.
func TestEvictStale_MemoryBasedEviction_UnderestimatedHeap(t *testing.T) {
now := time.Now().UTC()
store := makeTestStore(1000, now.Add(-1*time.Hour), 0)
store.retentionHours = 24
store.maxMemoryMB = 500
// Simulate trackedBytes 5x over budget.
store.trackedBytes = 2500 * 1048576
evicted := store.EvictStale()
if evicted == 0 {
t.Fatal("expected evictions when tracked is 5x over limit")
}
// Safety cap: max 25% per pass = 250
if evicted > 250 {
t.Fatalf("25%% safety cap violated: evicted %d of 1000", evicted)
}
if evicted != 250 {
t.Fatalf("expected exactly 250 evicted (25%% cap), got %d", evicted)
}
}
@@ -213,6 +245,101 @@ func TestEvictStale_CleansNodeIndexes(t *testing.T) {
}
}
func TestEvictStale_CleansResolvedPathNodeIndexes(t *testing.T) {
now := time.Now().UTC()
// Create a temp DB for on-demand SQL fetch during eviction
db := setupTestDB(t)
defer db.Close()
store := &PacketStore{
packets: make([]*StoreTx, 0),
byHash: make(map[string]*StoreTx),
byTxID: make(map[int]*StoreTx),
byObsID: make(map[int]*StoreObs),
byObserver: make(map[string][]*StoreObs),
byNode: make(map[string][]*StoreTx),
nodeHashes: make(map[string]map[string]bool),
byPayloadType: make(map[int][]*StoreTx),
spIndex: make(map[string]int),
distHops: make([]distHopRecord, 0),
distPaths: make([]distPathRecord, 0),
rfCache: make(map[string]*cachedResult),
topoCache: make(map[string]*cachedResult),
hashCache: make(map[string]*cachedResult),
chanCache: make(map[string]*cachedResult),
distCache: make(map[string]*cachedResult),
subpathCache: make(map[string]*cachedResult),
rfCacheTTL: 15 * time.Second,
retentionHours: 24,
db: db,
useResolvedPathIndex: true,
}
store.initResolvedPathIndex()
// Create a packet indexed via resolved_path pubkeys
relayPK := "relay0001abcdef"
txID := 1
obsID := 100
tx := &StoreTx{
ID: txID,
Hash: "hash_rp_001",
FirstSeen: now.Add(-48 * time.Hour).UTC().Format(time.RFC3339),
}
obs := &StoreObs{
ID: obsID,
TransmissionID: txID,
ObserverID: "obs0",
Timestamp: tx.FirstSeen,
}
tx.Observations = append(tx.Observations, obs)
// Insert into DB so on-demand SQL fetch works during eviction
db.conn.Exec("INSERT INTO transmissions (id, raw_hex, hash, first_seen) VALUES (?, '', ?, ?)",
txID, tx.Hash, tx.FirstSeen)
db.conn.Exec("INSERT INTO observations (id, transmission_id, observer_idx, path_json, timestamp, resolved_path) VALUES (?, ?, 1, ?, ?, ?)",
obsID, txID, `["aa"]`, now.Add(-48*time.Hour).Unix(), `["`+relayPK+`"]`)
store.packets = append(store.packets, tx)
store.byHash[tx.Hash] = tx
store.byTxID[tx.ID] = tx
store.byObsID[obs.ID] = obs
store.byObserver["obs0"] = append(store.byObserver["obs0"], obs)
// Index relay via decode-window simulation
store.addToByNode(tx, relayPK)
store.addToResolvedPubkeyIndex(txID, []string{relayPK})
// Verify indexed
if len(store.byNode[relayPK]) != 1 {
t.Fatalf("expected 1 entry in byNode[%s], got %d", relayPK, len(store.byNode[relayPK]))
}
if !store.nodeHashes[relayPK][tx.Hash] {
t.Fatalf("expected nodeHashes[%s] to contain %s", relayPK, tx.Hash)
}
evicted := store.RunEviction()
if evicted != 1 {
t.Fatalf("expected 1 evicted, got %d", evicted)
}
// Verify resolved_path entries are cleaned up
if len(store.byNode[relayPK]) != 0 {
t.Fatalf("expected byNode[%s] to be empty after eviction, got %d", relayPK, len(store.byNode[relayPK]))
}
if _, exists := store.nodeHashes[relayPK]; exists {
t.Fatalf("expected nodeHashes[%s] to be deleted after eviction", relayPK)
}
// Verify resolved pubkey index is cleaned up
h := resolvedPubkeyHash(relayPK)
if len(store.resolvedPubkeyIndex[h]) != 0 {
t.Fatalf("expected resolvedPubkeyIndex to be empty after eviction")
}
if _, exists := store.resolvedPubkeyReverse[txID]; exists {
t.Fatalf("expected resolvedPubkeyReverse to be empty after eviction")
}
}
func TestEvictStale_RunEvictionThreadSafe(t *testing.T) {
now := time.Now().UTC()
store := makeTestStore(20, now.Add(-48*time.Hour), 0)
@@ -250,3 +377,228 @@ func TestNewPacketStoreNilConfig(t *testing.T) {
t.Fatalf("expected retentionHours=0, got %f", store.retentionHours)
}
}
func TestCacheTTLFromConfig(t *testing.T) {
// With config values: analyticsHashSizes and analyticsRF should override defaults.
cacheTTL := map[string]interface{}{
"analyticsHashSizes": float64(7200),
"analyticsRF": float64(300),
}
store := NewPacketStore(nil, nil, cacheTTL)
if store.collisionCacheTTL != 7200*time.Second {
t.Fatalf("expected collisionCacheTTL=7200s, got %v", store.collisionCacheTTL)
}
if store.rfCacheTTL != 300*time.Second {
t.Fatalf("expected rfCacheTTL=300s, got %v", store.rfCacheTTL)
}
}
func TestCacheTTLDefaults(t *testing.T) {
// Without config, defaults should apply.
store := NewPacketStore(nil, nil)
if store.collisionCacheTTL != 3600*time.Second {
t.Fatalf("expected default collisionCacheTTL=3600s, got %v", store.collisionCacheTTL)
}
if store.rfCacheTTL != 15*time.Second {
t.Fatalf("expected default rfCacheTTL=15s, got %v", store.rfCacheTTL)
}
}
// --- Self-accounting memory tracking tests ---
func TestTrackedBytes_IncreasesOnInsert(t *testing.T) {
now := time.Now().UTC()
store := makeTestStore(0, now, 0)
if store.trackedBytes != 0 {
t.Fatalf("expected 0 trackedBytes for empty store, got %d", store.trackedBytes)
}
store2 := makeTestStore(10, now, 1)
if store2.trackedBytes <= 0 {
t.Fatal("expected positive trackedBytes after inserting 10 packets")
}
// Each packet has 2 observations; should be roughly 10*(384+5*48) + 20*(192+2*48) = 10*624 + 20*288 = 12000
expectedMin := int64(10*600 + 20*250) // rough lower bound
if store2.trackedBytes < expectedMin {
t.Fatalf("trackedBytes %d seems too low (expected > %d)", store2.trackedBytes, expectedMin)
}
}
func TestTrackedBytes_DecreasesOnEvict(t *testing.T) {
now := time.Now().UTC()
store := makeTestStore(100, now.Add(-48*time.Hour), 0)
store.retentionHours = 24
beforeBytes := store.trackedBytes
if beforeBytes <= 0 {
t.Fatal("expected positive trackedBytes before eviction")
}
evicted := store.EvictStale()
if evicted != 100 {
t.Fatalf("expected 100 evicted, got %d", evicted)
}
if store.trackedBytes != 0 {
t.Fatalf("expected 0 trackedBytes after evicting all, got %d", store.trackedBytes)
}
}
func TestTrackedBytes_MatchesExpectedAfterMixedInsertEvict(t *testing.T) {
now := time.Now().UTC()
// Create 100 packets, 50 old + 50 recent
store := makeTestStore(100, now.Add(-48*time.Hour), 0)
for i := 50; i < 100; i++ {
store.packets[i].FirstSeen = now.Add(-1 * time.Hour).Format(time.RFC3339)
}
store.retentionHours = 24
totalBefore := store.trackedBytes
// Calculate expected bytes for first 50 packets (to be evicted)
var evictedBytes int64
for i := 0; i < 50; i++ {
tx := store.packets[i]
evictedBytes += estimateStoreTxBytes(tx)
for _, obs := range tx.Observations {
evictedBytes += estimateStoreObsBytes(obs)
}
}
store.EvictStale()
expectedAfter := totalBefore - evictedBytes
if store.trackedBytes != expectedAfter {
t.Fatalf("trackedBytes %d != expected %d (before=%d, evicted=%d)",
store.trackedBytes, expectedAfter, totalBefore, evictedBytes)
}
}
func TestWatermarkHysteresis(t *testing.T) {
now := time.Now().UTC()
store := makeTestStore(1000, now.Add(-1*time.Hour), 0)
store.retentionHours = 0 // no time-based eviction
store.maxMemoryMB = 1 // 1MB budget
// Set trackedBytes to just above high watermark
highWatermark := int64(1 * 1048576)
lowWatermark := int64(float64(highWatermark) * 0.85)
store.trackedBytes = highWatermark + 1
evicted := store.EvictStale()
if evicted == 0 {
t.Fatal("expected eviction when above high watermark")
}
if store.trackedBytes > lowWatermark+1024 {
t.Fatalf("expected trackedBytes near low watermark after eviction, got %d (low=%d)",
store.trackedBytes, lowWatermark)
}
// Now set trackedBytes to just below high watermark — should NOT trigger
store.trackedBytes = highWatermark - 1
evicted2 := store.EvictStale()
if evicted2 != 0 {
t.Fatalf("expected no eviction below high watermark, got %d", evicted2)
}
}
func TestSafetyCap25Percent(t *testing.T) {
now := time.Now().UTC()
store := makeTestStore(1000, now.Add(-1*time.Hour), 0)
store.retentionHours = 0
store.maxMemoryMB = 1
// Set trackedBytes way over limit to force maximum eviction
store.trackedBytes = 100 * 1048576 // 100MB vs 1MB limit
evicted := store.EvictStale()
// 25% of 1000 = 250
if evicted > 250 {
t.Fatalf("25%% safety cap violated: evicted %d of 1000 (max should be 250)", evicted)
}
if evicted != 250 {
t.Fatalf("expected exactly 250 evicted (25%% cap), got %d", evicted)
}
if len(store.packets) != 750 {
t.Fatalf("expected 750 remaining, got %d", len(store.packets))
}
}
func TestMultiplePassesConverge(t *testing.T) {
now := time.Now().UTC()
store := makeTestStore(1000, now.Add(-1*time.Hour), 0)
store.retentionHours = 0
// Set budget to half the actual tracked bytes — requires ~2 passes
actualBytes := store.trackedBytes
store.maxMemoryMB = int(float64(actualBytes) / 1048576.0 / 2)
if store.maxMemoryMB < 1 {
store.maxMemoryMB = 1
}
totalEvicted := 0
for pass := 0; pass < 20; pass++ {
evicted := store.EvictStale()
if evicted == 0 {
break
}
totalEvicted += evicted
}
// After convergence, trackedBytes should be at or below high watermark
// (may be between low and high due to hysteresis — that's fine)
highWatermark := int64(store.maxMemoryMB) * 1048576
if store.trackedBytes > highWatermark {
t.Fatalf("did not converge: trackedBytes=%d (%.1fMB) > highWatermark=%d after multiple passes",
store.trackedBytes, float64(store.trackedBytes)/1048576.0, highWatermark)
}
if totalEvicted == 0 {
t.Fatal("expected some evictions across multiple passes")
}
}
func TestEstimateStoreTxBytes(t *testing.T) {
tx := &StoreTx{
RawHex: "aabbcc",
Hash: "hash1234",
DecodedJSON: `{"pubKey":"pk1"}`,
PathJSON: `["aa","bb"]`,
}
est := estimateStoreTxBytes(tx)
// Manual calculation: base + string lengths + index entries + perTxMaps + path hops + subpaths
hops := int64(len(txGetParsedPath(tx)))
manualCalc := int64(storeTxBaseBytes) + int64(len(tx.RawHex)+len(tx.Hash)+len(tx.DecodedJSON)+len(tx.PathJSON)) + int64(numIndexesPerTx*indexEntryBytes)
manualCalc += perTxMapsBytes
manualCalc += hops * perPathHopBytes
if hops > 1 {
manualCalc += (hops * (hops - 1) / 2) * perSubpathEntryBytes
}
if est != manualCalc {
t.Fatalf("estimateStoreTxBytes = %d, want %d (manual calc)", est, manualCalc)
}
if est < 600 || est > 1200 {
t.Fatalf("estimateStoreTxBytes = %d, expected in range [600, 1200]", est)
}
}
func TestEstimateStoreObsBytes(t *testing.T) {
obs := &StoreObs{
ObserverID: "obs123",
PathJSON: `["aa"]`,
}
est := estimateStoreObsBytes(obs)
// storeObsBaseBytes(192) + len(ObserverID=6) + len(PathJSON=6) + 2*48(96) = 300
expected := int64(192 + 6 + 6 + 2*48)
if est != expected {
t.Fatalf("estimateStoreObsBytes = %d, want %d", est, expected)
}
}
func BenchmarkEviction100K(b *testing.B) {
now := time.Now().UTC()
for i := 0; i < b.N; i++ {
b.StopTimer()
store := makeTestStore(100000, now.Add(-48*time.Hour), 0)
store.retentionHours = 24
b.StartTimer()
store.EvictStale()
}
}
+7
View File
@@ -6,11 +6,18 @@ require (
github.com/gorilla/mux v1.8.1
github.com/gorilla/websocket v1.5.3
github.com/meshcore-analyzer/geofilter v0.0.0
github.com/meshcore-analyzer/sigvalidate v0.0.0
modernc.org/sqlite v1.34.5
)
replace github.com/meshcore-analyzer/geofilter => ../../internal/geofilter
replace github.com/meshcore-analyzer/sigvalidate => ../../internal/sigvalidate
require github.com/meshcore-analyzer/packetpath v0.0.0
replace github.com/meshcore-analyzer/packetpath => ../../internal/packetpath
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
+119
View File
@@ -0,0 +1,119 @@
package main
import (
"log"
"time"
)
// migrateContentHashesAsync recomputes content hashes in batches after the
// server is already serving HTTP. Packets whose hash changes are updated in
// both the DB and the in-memory byHash index. The migration is idempotent:
// once all hashes match the current formula it completes instantly.
func migrateContentHashesAsync(store *PacketStore, batchSize int, yieldDuration time.Duration) {
defer func() {
if r := recover(); r != nil {
log.Printf("[hash-migrate] panic recovered: %v", r)
}
store.hashMigrationComplete.Store(true)
}()
// Snapshot the packet slice length under lock (packets only grow).
store.mu.RLock()
total := len(store.packets)
store.mu.RUnlock()
migrated := 0
for offset := 0; offset < total; offset += batchSize {
end := offset + batchSize
if end > total {
end = total
}
// Collect stale hashes in this batch under RLock.
type hashUpdate struct {
tx *StoreTx
oldHash string
newHash string
}
var updates []hashUpdate
store.mu.RLock()
for _, tx := range store.packets[offset:end] {
if tx.RawHex == "" {
continue
}
newHash := ComputeContentHash(tx.RawHex)
if newHash != tx.Hash {
updates = append(updates, hashUpdate{tx: tx, oldHash: tx.Hash, newHash: newHash})
}
}
store.mu.RUnlock()
if len(updates) == 0 {
continue
}
// Write batch to DB in a single transaction.
dbTx, err := store.db.conn.Begin()
if err != nil {
log.Printf("[hash-migrate] begin tx: %v", err)
continue
}
stmt, err := dbTx.Prepare("UPDATE transmissions SET hash = ? WHERE id = ?")
if err != nil {
log.Printf("[hash-migrate] prepare: %v", err)
dbTx.Rollback()
continue
}
for _, u := range updates {
if _, err := stmt.Exec(u.newHash, u.tx.ID); err != nil {
// UNIQUE constraint = two old hashes map to the same new hash (duplicate).
// Merge observations to the surviving tx, delete the duplicate.
log.Printf("[hash-migrate] tx %d collides — merging duplicate", u.tx.ID)
var survID int
if err2 := dbTx.QueryRow("SELECT id FROM transmissions WHERE hash = ?", u.newHash).Scan(&survID); err2 == nil {
dbTx.Exec("UPDATE observations SET transmission_id = ? WHERE transmission_id = ?", survID, u.tx.ID)
dbTx.Exec("DELETE FROM transmissions WHERE id = ?", u.tx.ID)
u.newHash = "" // mark for in-memory removal only
}
}
}
stmt.Close()
if err := dbTx.Commit(); err != nil {
log.Printf("[hash-migrate] commit: %v", err)
continue
}
// Update in-memory index under write lock.
store.mu.Lock()
for _, u := range updates {
delete(store.byHash, u.oldHash)
if u.newHash == "" {
// Merged duplicate — remove from packets slice and indexes.
delete(store.byTxID, u.tx.ID)
// Move observations to survivor if present.
if surv := store.byHash[ComputeContentHash(u.tx.RawHex)]; surv != nil {
for _, obs := range u.tx.Observations {
surv.Observations = append(surv.Observations, obs)
surv.ObservationCount++
}
}
} else {
u.tx.Hash = u.newHash
store.byHash[u.newHash] = u.tx
}
}
store.mu.Unlock()
migrated += len(updates)
// Yield to let HTTP handlers run.
time.Sleep(yieldDuration)
}
if migrated > 0 {
log.Printf("[hash-migrate] Migrated %d content hashes to new formula", migrated)
}
}
+78
View File
@@ -0,0 +1,78 @@
package main
import (
"testing"
"time"
)
func TestMigrateContentHashesAsync(t *testing.T) {
db := setupTestDBv2(t)
store := NewPacketStore(db, nil)
// Insert a packet with a manually wrong hash (simulating old formula).
rawHex := "0A00D69FD7A5A7475DB07337749AE61FA53A4788E976"
correctHash := ComputeContentHash(rawHex)
wrongHash := "deadbeef12345678"
_, err := db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type)
VALUES (?, ?, datetime('now'), 0, 2)`, rawHex, wrongHash)
if err != nil {
t.Fatal(err)
}
if err := store.Load(); err != nil {
t.Fatal(err)
}
if store.byHash[wrongHash] == nil {
t.Fatal("expected packet under wrong hash before migration")
}
migrateContentHashesAsync(store, 100, time.Millisecond)
if !store.hashMigrationComplete.Load() {
t.Error("expected hashMigrationComplete to be true")
}
if store.byHash[wrongHash] != nil {
t.Error("old hash should be removed from index")
}
if store.byHash[correctHash] == nil {
t.Error("new hash should be in index")
}
var dbHash string
err = db.conn.QueryRow("SELECT hash FROM transmissions WHERE raw_hex = ?", rawHex).Scan(&dbHash)
if err != nil {
t.Fatal(err)
}
if dbHash != correctHash {
t.Errorf("DB hash = %s, want %s", dbHash, correctHash)
}
}
func TestMigrateContentHashesAsync_NoOp(t *testing.T) {
db := setupTestDBv2(t)
store := NewPacketStore(db, nil)
rawHex := "0A00D69FD7A5A7475DB07337749AE61FA53A4788E976"
correctHash := ComputeContentHash(rawHex)
_, err := db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type)
VALUES (?, ?, datetime('now'), 0, 2)`, rawHex, correctHash)
if err != nil {
t.Fatal(err)
}
if err := store.Load(); err != nil {
t.Fatal(err)
}
migrateContentHashesAsync(store, 100, time.Millisecond)
if !store.hashMigrationComplete.Load() {
t.Error("expected hashMigrationComplete to be true")
}
if store.byHash[correctHash] == nil {
t.Error("hash should remain in index")
}
}
+40
View File
@@ -2,6 +2,8 @@ package main
import (
"encoding/json"
"fmt"
"math/rand"
"net/http"
"net/http/httptest"
"os"
@@ -220,6 +222,44 @@ func TestSortedCopy(t *testing.T) {
}
}
func TestSortedCopyLarge(t *testing.T) {
// Regression: verify correct sort on larger input
rng := rand.New(rand.NewSource(42))
n := 1000
input := make([]float64, n)
for i := range input {
input[i] = rng.Float64() * 1000
}
result := sortedCopy(input)
if len(result) != n {
t.Fatalf("expected %d elements, got %d", n, len(result))
}
for i := 1; i < len(result); i++ {
if result[i] < result[i-1] {
t.Fatalf("not sorted at index %d: %v > %v", i, result[i-1], result[i])
}
}
// Original unchanged
if input[0] == result[0] && input[1] == result[1] && input[2] == result[2] {
// Could be coincidence but very unlikely with random data
}
}
func BenchmarkSortedCopy(b *testing.B) {
rng := rand.New(rand.NewSource(42))
for _, size := range []int{256, 1000, 10000} {
data := make([]float64, size)
for i := range data {
data[i] = rng.Float64() * 1000
}
b.Run(fmt.Sprintf("n=%d", size), func(b *testing.B) {
for i := 0; i < b.N; i++ {
sortedCopy(data)
}
})
}
}
func TestLastN(t *testing.T) {
arr := []map[string]interface{}{
{"id": 1}, {"id": 2}, {"id": 3}, {"id": 4}, {"id": 5},
+107
View File
@@ -0,0 +1,107 @@
package main
import (
"encoding/json"
"testing"
"time"
_ "modernc.org/sqlite"
)
const issue673NodePK = "7502f19f44cad6d7b626e1d811c00a914af452636182ccded3fd019803395ec9"
// setupIssue673Store builds an in-memory store with one repeater node having:
// - one ADVERT packet (legitimately indexed in byNode)
// - one GRP_TXT packet whose decoded text contains the node's pubkey (false-positive candidate)
func setupIssue673Store(t *testing.T) (*PacketStore, *DB) {
t.Helper()
db := setupTestDB(t)
_, err := db.conn.Exec(
"INSERT INTO nodes (public_key, name, role) VALUES (?, ?, ?)",
issue673NodePK, "Quail Hollow Park", "repeater",
)
if err != nil {
t.Fatal(err)
}
ps := NewPacketStore(db, nil)
now := time.Now().UTC().Format(time.RFC3339)
pt4 := 4 // ADVERT
pt5 := 5 // GRP_TXT
advertDecoded, _ := json.Marshal(map[string]interface{}{"pubKey": issue673NodePK})
advert := &StoreTx{
ID: 1,
Hash: "advert_hash_673",
PayloadType: &pt4,
DecodedJSON: string(advertDecoded),
FirstSeen: now,
}
otherPK := "aabbccddaabbccddaabbccddaabbccddaabbccddaabbccddaabbccddaabbccdd"
chatDecoded, _ := json.Marshal(map[string]interface{}{
"srcPubKey": otherPK,
"text": "Check out node " + issue673NodePK + " on the analyzer",
})
chat := &StoreTx{
ID: 2,
Hash: "chat_hash_673",
PayloadType: &pt5,
DecodedJSON: string(chatDecoded),
FirstSeen: now,
}
ps.mu.Lock()
ps.packets = append(ps.packets, advert, chat)
ps.byHash[advert.Hash] = advert
ps.byHash[chat.Hash] = chat
ps.byTxID[advert.ID] = advert
ps.byTxID[chat.ID] = chat
ps.byNode[issue673NodePK] = []*StoreTx{advert}
ps.mu.Unlock()
return ps, db
}
// TestGetNodeAnalytics_ExcludesGRPTXTWithPubkeyInText verifies that a GRP_TXT packet
// whose message text contains a node's pubkey is not counted in that node's analytics.
func TestGetNodeAnalytics_ExcludesGRPTXTWithPubkeyInText(t *testing.T) {
ps, db := setupIssue673Store(t)
defer db.Close()
analytics, err := ps.GetNodeAnalytics(issue673NodePK, 30)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if analytics == nil {
t.Fatal("expected analytics, got nil")
}
for _, ptc := range analytics.PacketTypeBreakdown {
if ptc.PayloadType == 5 {
t.Errorf("GRP_TXT (type 5) should not appear in analytics for repeater node, got count=%d", ptc.Count)
}
}
}
// TestFilterPackets_NodeQueryDoesNotMatchChatText verifies that the slow path of
// filterPackets (node filter combined with Since) does not return a GRP_TXT packet
// whose pubkey appears only in message text, not in a structured pubkey field.
func TestFilterPackets_NodeQueryDoesNotMatchChatText(t *testing.T) {
ps, db := setupIssue673Store(t)
defer db.Close()
yesterday := time.Now().Add(-24 * time.Hour).UTC().Format(time.RFC3339)
result := ps.QueryPackets(PacketQuery{Node: issue673NodePK, Since: yesterday, Limit: 50})
if result.Total != 1 {
t.Errorf("expected 1 packet for node (ADVERT only), got %d", result.Total)
}
for _, pkt := range result.Packets {
if pkt["hash"] == "chat_hash_673" {
t.Errorf("GRP_TXT with pubkey in message text was incorrectly returned for node query")
}
}
}
+78
View File
@@ -0,0 +1,78 @@
package main
import (
"encoding/json"
"net/http/httptest"
"testing"
"time"
"github.com/gorilla/mux"
)
// TestRepro810 reproduces #810: when the longest-path observation has NULL
// resolved_path but a shorter-path observation has one, fetchResolvedPathForTxBest
// returns nil → /api/nodes/{pk}/health.recentPackets[].resolved_path is missing
// while /api/packets shows it.
func TestRepro810(t *testing.T) {
db := setupTestDB(t)
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
db.conn.Exec(`INSERT INTO observers (id, name, last_seen, first_seen, packet_count) VALUES ('obs1','O1',?, '2026-01-01T00:00:00Z', 100)`, recent)
db.conn.Exec(`INSERT INTO observers (id, name, last_seen, first_seen, packet_count) VALUES ('obs2','O2',?, '2026-01-01T00:00:00Z', 100)`, recent)
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, last_seen, first_seen, advert_count) VALUES ('aabbccdd11223344','R','repeater',?, '2026-01-01T00:00:00Z', 1)`, recent)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json) VALUES ('AABB','testhash00000001',?,1,4,'{"pubKey":"aabbccdd11223344","type":"ADVERT"}')`, recent)
// Longest-path obs WITHOUT resolved_path
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp) VALUES (1,1,12.5,-90,'["aa","bb","cc"]',?)`, recentEpoch)
// Shorter-path obs WITH resolved_path
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp, resolved_path) VALUES (1,2,8.0,-95,'["aa","bb"]',?,'["aabbccdd11223344","eeff00112233aabb"]')`, recentEpoch-100)
cfg := &Config{Port: 3000}
hub := NewHub()
srv := NewServer(db, cfg, hub)
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatal(err)
}
srv.store = store
router := mux.NewRouter()
srv.RegisterRoutes(router)
// Sanity: /api/packets should show resolved_path for this tx.
reqP := httptest.NewRequest("GET", "/api/packets?limit=10", nil)
wP := httptest.NewRecorder()
router.ServeHTTP(wP, reqP)
var pktsBody map[string]interface{}
json.Unmarshal(wP.Body.Bytes(), &pktsBody)
pkts, _ := pktsBody["packets"].([]interface{})
hasOnPackets := false
for _, p := range pkts {
pm := p.(map[string]interface{})
if pm["hash"] == "testhash00000001" && pm["resolved_path"] != nil {
hasOnPackets = true
}
}
if !hasOnPackets {
t.Fatal("precondition: /api/packets must report resolved_path for tx")
}
req := httptest.NewRequest("GET", "/api/nodes/aabbccdd11223344/health", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
var body map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &body)
rp, _ := body["recentPackets"].([]interface{})
if len(rp) == 0 {
t.Fatal("no recentPackets")
}
for _, p := range rp {
pm := p.(map[string]interface{})
if pm["hash"] == "testhash00000001" {
if pm["resolved_path"] == nil {
t.Fatal("BUG #810: /health.recentPackets resolved_path is nil despite /api/packets reporting it")
}
return
}
}
t.Fatal("tx not found in recentPackets")
}
+230 -6
View File
@@ -104,11 +104,21 @@ func main() {
}
if cfg.APIKey == "" {
log.Printf("[security] WARNING: no apiKey configured — write endpoints are BLOCKED (set apiKey in config.json to enable them)")
} else if IsWeakAPIKey(cfg.APIKey) {
log.Printf("[security] WARNING: API key is weak or a known default — write endpoints are vulnerable")
}
// Resolve DB path
resolvedDB := cfg.ResolveDBPath(configDir)
log.Printf("[config] port=%d db=%s public=%s", cfg.Port, resolvedDB, publicDir)
if len(cfg.NodeBlacklist) > 0 {
log.Printf("[config] nodeBlacklist: %d node(s) will be hidden from API", len(cfg.NodeBlacklist))
for _, pk := range cfg.NodeBlacklist {
if trimmed := strings.ToLower(strings.TrimSpace(pk)); trimmed != "" {
log.Printf("[config] blacklisted: %s", trimmed)
}
}
}
// Open database
database, err := OpenDB(resolvedDB)
@@ -139,11 +149,87 @@ func main() {
}
// In-memory packet store
store := NewPacketStore(database, cfg.PacketStore)
store := NewPacketStore(database, cfg.PacketStore, cfg.CacheTTL)
if err := store.Load(); err != nil {
log.Fatalf("[store] failed to load: %v", err)
}
// Initialize persisted neighbor graph
dbPath = database.path
if err := ensureNeighborEdgesTable(dbPath); err != nil {
log.Printf("[neighbor] warning: could not create neighbor_edges table: %v", err)
}
// Add resolved_path column if missing.
// NOTE on startup ordering (review item #10): ensureResolvedPathColumn runs AFTER
// OpenDB/detectSchema, so db.hasResolvedPath will be false on first run with a
// pre-existing DB. This means Load() won't SELECT resolved_path from SQLite.
// Async backfill runs after HTTP starts (see backfillResolvedPathsAsync below)
// AND to SQLite. On next restart, detectSchema finds the column and Load() reads it.
if err := ensureResolvedPathColumn(dbPath); err != nil {
log.Printf("[store] warning: could not add resolved_path column: %v", err)
} else {
database.hasResolvedPath = true // detectSchema ran before column was added; fix the flag
}
// Load or build neighbor graph
if neighborEdgesTableExists(database.conn) {
store.graph = loadNeighborEdgesFromDB(database.conn)
log.Printf("[neighbor] loaded persisted neighbor graph")
} else {
log.Printf("[neighbor] no persisted edges found, will build in background...")
store.graph = NewNeighborGraph() // empty graph — gets populated by background goroutine
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[neighbor] graph build panic recovered: %v", r)
}
}()
rw, rwErr := openRW(dbPath)
if rwErr == nil {
edgeCount := buildAndPersistEdges(store, rw)
rw.Close()
log.Printf("[neighbor] persisted %d edges", edgeCount)
}
built := BuildFromStore(store)
store.mu.Lock()
store.graph = built
store.mu.Unlock()
log.Printf("[neighbor] graph build complete")
}()
}
// Initial pickBestObservation runs in background — doesn't need to block HTTP.
// API serves best-effort data until this completes (~10s for 100K txs).
// Processes in chunks of 5000, releasing the lock between chunks so API
// handlers remain responsive.
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[store] pickBestObservation panic recovered: %v", r)
}
}()
const chunkSize = 5000
store.mu.RLock()
totalPackets := len(store.packets)
store.mu.RUnlock()
for i := 0; i < totalPackets; i += chunkSize {
end := i + chunkSize
if end > totalPackets {
end = totalPackets
}
store.mu.Lock()
for j := i; j < end && j < len(store.packets); j++ {
pickBestObservation(store.packets[j])
}
store.mu.Unlock()
if end < totalPackets {
time.Sleep(10 * time.Millisecond) // yield to API handlers
}
}
log.Printf("[store] initial pickBestObservation complete (%d transmissions)", totalPackets)
}()
// WebSocket hub
hub := NewHub()
@@ -180,26 +266,143 @@ func main() {
defer stopEviction()
// Auto-prune old packets if retention.packetDays is configured
var stopPrune func()
if cfg.Retention != nil && cfg.Retention.PacketDays > 0 {
days := cfg.Retention.PacketDays
pruneTicker := time.NewTicker(24 * time.Hour)
pruneDone := make(chan struct{})
stopPrune = func() {
pruneTicker.Stop()
close(pruneDone)
}
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[prune] panic recovered: %v", r)
}
}()
time.Sleep(1 * time.Minute)
if n, err := database.PruneOldPackets(days); err != nil {
log.Printf("[prune] error: %v", err)
} else {
log.Printf("[prune] deleted %d transmissions older than %d days", n, days)
}
for range time.Tick(24 * time.Hour) {
if n, err := database.PruneOldPackets(days); err != nil {
log.Printf("[prune] error: %v", err)
} else {
log.Printf("[prune] deleted %d transmissions older than %d days", n, days)
for {
select {
case <-pruneTicker.C:
if n, err := database.PruneOldPackets(days); err != nil {
log.Printf("[prune] error: %v", err)
} else {
log.Printf("[prune] deleted %d transmissions older than %d days", n, days)
}
case <-pruneDone:
return
}
}
}()
log.Printf("[prune] auto-prune enabled: packets older than %d days will be removed daily", days)
}
// Auto-prune old metrics
var stopMetricsPrune func()
{
metricsDays := cfg.MetricsRetentionDays()
metricsPruneTicker := time.NewTicker(24 * time.Hour)
metricsPruneDone := make(chan struct{})
stopMetricsPrune = func() {
metricsPruneTicker.Stop()
close(metricsPruneDone)
}
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[metrics-prune] panic recovered: %v", r)
}
}()
time.Sleep(2 * time.Minute) // stagger after packet prune
database.PruneOldMetrics(metricsDays)
for {
select {
case <-metricsPruneTicker.C:
database.PruneOldMetrics(metricsDays)
case <-metricsPruneDone:
return
}
}
}()
log.Printf("[metrics-prune] auto-prune enabled: metrics older than %d days", metricsDays)
}
// Auto-prune stale observers
var stopObserverPrune func()
{
observerDays := cfg.ObserverDaysOrDefault()
if observerDays <= -1 {
// -1 means keep forever, skip
} else {
observerPruneTicker := time.NewTicker(24 * time.Hour)
observerPruneDone := make(chan struct{})
stopObserverPrune = func() {
observerPruneTicker.Stop()
close(observerPruneDone)
}
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[observer-prune] panic recovered: %v", r)
}
}()
time.Sleep(3 * time.Minute) // stagger after metrics prune
database.RemoveStaleObservers(observerDays)
for {
select {
case <-observerPruneTicker.C:
database.RemoveStaleObservers(observerDays)
case <-observerPruneDone:
return
}
}
}()
log.Printf("[observer-prune] auto-prune enabled: observers not seen in %d days will be removed", observerDays)
}
}
// Auto-prune old neighbor edges
var stopEdgePrune func()
{
maxAgeDays := cfg.NeighborMaxAgeDays()
edgePruneTicker := time.NewTicker(24 * time.Hour)
edgePruneDone := make(chan struct{})
stopEdgePrune = func() {
edgePruneTicker.Stop()
close(edgePruneDone)
}
go func() {
defer func() {
if r := recover(); r != nil {
log.Printf("[neighbor-prune] panic recovered: %v", r)
}
}()
time.Sleep(4 * time.Minute) // stagger after metrics prune
store.mu.RLock()
g := store.graph
store.mu.RUnlock()
PruneNeighborEdges(dbPath, g, maxAgeDays)
for {
select {
case <-edgePruneTicker.C:
store.mu.RLock()
g := store.graph
store.mu.RUnlock()
PruneNeighborEdges(dbPath, g, maxAgeDays)
case <-edgePruneDone:
return
}
}
}()
log.Printf("[neighbor-prune] auto-prune enabled: edges older than %d days", maxAgeDays)
}
// Graceful shutdown
httpServer := &http.Server{
Addr: fmt.Sprintf(":%d", cfg.Port),
@@ -218,6 +421,20 @@ func main() {
// 1. Stop accepting new WebSocket/poll data
poller.Stop()
// 1b. Stop auto-prune ticker
if stopPrune != nil {
stopPrune()
}
if stopMetricsPrune != nil {
stopMetricsPrune()
}
if stopObserverPrune != nil {
stopObserverPrune()
}
if stopEdgePrune != nil {
stopEdgePrune()
}
// 2. Gracefully drain HTTP connections (up to 15s)
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
@@ -236,6 +453,13 @@ func main() {
}()
log.Printf("[server] CoreScope (Go) listening on http://localhost:%d", cfg.Port)
// Start async backfill in background — HTTP is now available.
go backfillResolvedPathsAsync(store, dbPath, 5000, 100*time.Millisecond, cfg.BackfillHours())
// Migrate old content hashes in background (one-time, idempotent).
go migrateContentHashesAsync(store, 5000, 100*time.Millisecond)
if err := httpServer.ListenAndServe(); err != http.ErrServerClosed {
log.Fatalf("[server] %v", err)
}
+132
View File
@@ -0,0 +1,132 @@
package main
import (
"os"
"strconv"
"strings"
"sync"
"time"
)
// MemorySnapshot is a point-in-time view of process memory across several
// vantage points. Values are in MB (1024*1024 bytes), rounded to one decimal.
//
// Field invariants (typical, not guaranteed under exotic conditions):
//
// processRSSMB >= goSysMB >= goHeapInuseMB >= storeDataMB
//
// - processRSSMB is what the kernel charges the process (resident set).
// Read from /proc/self/status `VmRSS:` on Linux; falls back to goSysMB
// on other platforms or when /proc is unavailable.
// - goSysMB is the total memory obtained from the OS by the Go runtime
// (heap, stacks, GC metadata, mspans, mcache, etc.). Includes
// fragmentation and unused-but-mapped span overhead.
// - goHeapInuseMB is the live, in-use Go heap (HeapInuse). Excludes
// idle spans and runtime overhead.
// - storeDataMB is the in-store packet byte estimate (transmissions +
// observations). Subset of HeapInuse. Does not include index maps,
// analytics caches, broadcast queues, or runtime overhead. Used as
// the input to the eviction watermark.
//
// processRSSMB and storeDataMB are monotonic only relative to ingest +
// eviction; both can shrink when packets age out. goHeapInuseMB and goSysMB
// fluctuate with GC.
//
// cgoBytesMB intentionally absent: this build uses the pure-Go
// modernc.org/sqlite driver, so there is no cgo allocator to measure.
// Reintroduce only if we ever switch back to mattn/go-sqlite3.
type MemorySnapshot struct {
ProcessRSSMB float64 `json:"processRSSMB"`
GoHeapInuseMB float64 `json:"goHeapInuseMB"`
GoSysMB float64 `json:"goSysMB"`
StoreDataMB float64 `json:"storeDataMB"`
}
// rssCache rate-limits the /proc/self/status read. Go memory stats are
// already cached by Server.getMemStats (5s TTL). We use a tighter 1s TTL
// here so processRSSMB stays reasonably fresh during ops debugging
// without paying the syscall cost on every /api/stats hit.
var (
rssCacheMu sync.Mutex
rssCacheValueMB float64
rssCacheCachedAt time.Time
)
const rssCacheTTL = 1 * time.Second
// getMemorySnapshot composes a MemorySnapshot using the Server's existing
// runtime.MemStats cache (5s TTL, used by /api/health and /api/perf too)
// plus a rate-limited /proc RSS read. storeDataMB is supplied by the
// caller because the packet store is the source of truth.
func (s *Server) getMemorySnapshot(storeDataMB float64) MemorySnapshot {
ms := s.getMemStats()
rssCacheMu.Lock()
if time.Since(rssCacheCachedAt) > rssCacheTTL {
rssCacheValueMB = readProcRSSMB()
rssCacheCachedAt = time.Now()
}
rssMB := rssCacheValueMB
rssCacheMu.Unlock()
if rssMB <= 0 {
// Fallback when /proc is unavailable (non-Linux, sandboxes, etc.).
// runtime.Sys is an upper bound on Go-attributable memory and a
// reasonable proxy for pure-Go builds.
rssMB = float64(ms.Sys) / 1048576.0
}
return MemorySnapshot{
ProcessRSSMB: roundMB(rssMB),
GoHeapInuseMB: roundMB(float64(ms.HeapInuse) / 1048576.0),
GoSysMB: roundMB(float64(ms.Sys) / 1048576.0),
StoreDataMB: roundMB(storeDataMB),
}
}
// readProcRSSMB parses /proc/self/status for the VmRSS line. Returns 0 on
// any failure (file missing, malformed line, parse error) — the caller
// then uses a runtime fallback. Linux only; macOS/Windows return 0.
//
// Safety notes (djb): the file path is hard-coded, no untrusted input is
// concatenated. We bound the read at 8 KiB (the whole status file is
// well under 4 KiB on modern kernels) so a corrupt /proc can't OOM us.
// We only parse digits with strconv; no shell, no exec, no format strings.
func readProcRSSMB() float64 {
const maxStatusBytes = 8 * 1024
f, err := os.Open("/proc/self/status")
if err != nil {
return 0
}
defer f.Close()
buf := make([]byte, maxStatusBytes)
n, err := f.Read(buf)
if err != nil && n == 0 {
return 0
}
for _, line := range strings.Split(string(buf[:n]), "\n") {
if !strings.HasPrefix(line, "VmRSS:") {
continue
}
// Format: "VmRSS:\t 123456 kB"
fields := strings.Fields(line[len("VmRSS:"):])
if len(fields) < 2 {
return 0
}
kb, err := strconv.ParseFloat(fields[0], 64)
if err != nil || kb < 0 {
return 0
}
// Unit is kB per kernel convention; convert to MB.
return kb / 1024.0
}
return 0
}
func roundMB(v float64) float64 {
if v < 0 {
return 0
}
return float64(int64(v*10+0.5)) / 10.0
}
+435
View File
@@ -0,0 +1,435 @@
package main
import (
"database/sql"
"encoding/json"
"fmt"
"strings"
"testing"
"time"
_ "modernc.org/sqlite"
)
// recentTS returns a timestamp string N hours ago, ensuring test data
// stays within the 7-day advert window used by computeNodeHashSizeInfo.
func recentTS(hoursAgo int) string {
return time.Now().UTC().Add(-time.Duration(hoursAgo) * time.Hour).Format("2006-01-02T15:04:05.000Z")
}
// setupCapabilityTestDB creates a minimal in-memory DB with nodes table.
func setupCapabilityTestDB(t *testing.T) *DB {
t.Helper()
conn, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatal(err)
}
conn.SetMaxOpenConns(1)
conn.Exec(`CREATE TABLE nodes (
public_key TEXT PRIMARY KEY, name TEXT, role TEXT,
lat REAL, lon REAL, last_seen TEXT, first_seen TEXT,
advert_count INTEGER DEFAULT 0, battery_mv INTEGER, temperature_c REAL
)`)
conn.Exec(`CREATE TABLE observers (
id TEXT PRIMARY KEY, name TEXT, iata TEXT, last_seen TEXT,
first_seen TEXT, packet_count INTEGER DEFAULT 0, model TEXT,
firmware TEXT, client_version TEXT, radio TEXT, battery_mv INTEGER,
uptime_secs INTEGER
)`)
return &DB{conn: conn}
}
// addTestPacket adds a StoreTx to the store's internal structures including
// the byPathHop index and byPayloadType index.
func addTestPacket(store *PacketStore, tx *StoreTx) {
store.mu.Lock()
defer store.mu.Unlock()
tx.ID = len(store.packets) + 1
if tx.Hash == "" {
tx.Hash = fmt.Sprintf("test-hash-%d", tx.ID)
}
store.packets = append(store.packets, tx)
store.byHash[tx.Hash] = tx
store.byTxID[tx.ID] = tx
if tx.PayloadType != nil {
store.byPayloadType[*tx.PayloadType] = append(store.byPayloadType[*tx.PayloadType], tx)
}
addTxToPathHopIndex(store.byPathHop, tx)
}
// buildPathByte returns a 2-char hex string for the path byte with given
// hashSize (1-3) and hopCount.
func buildPathByte(hashSize, hopCount int) string {
b := byte(((hashSize - 1) & 0x3) << 6) | byte(hopCount&0x3F)
return fmt.Sprintf("%02x", b)
}
// makeTestAdvert creates a StoreTx representing a flood advert packet.
func makeTestAdvert(pubkey string, hashSize int) *StoreTx {
decoded, _ := json.Marshal(map[string]interface{}{"pubKey": pubkey, "name": pubkey[:8]})
pt := 4
pathByte := buildPathByte(hashSize, 1)
prefix := strings.ToLower(pubkey[:hashSize*2])
rawHex := "01" + pathByte + prefix // flood header + path byte + hop prefix
return &StoreTx{
RawHex: rawHex,
PayloadType: &pt,
DecodedJSON: string(decoded),
PathJSON: `["` + prefix + `"]`,
FirstSeen: recentTS(24),
}
}
// TestMultiByteCapability_Confirmed tests that a repeater advertising
// with hash_size >= 2 is classified as "confirmed".
func TestMultiByteCapability_Confirmed(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "RepA", "repeater", recentTS(24))
store := NewPacketStore(db, nil)
addTestPacket(store, makeTestAdvert("aabbccdd11223344", 2))
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
if caps[0].Status != "confirmed" {
t.Errorf("expected confirmed, got %s", caps[0].Status)
}
if caps[0].Evidence != "advert" {
t.Errorf("expected advert evidence, got %s", caps[0].Evidence)
}
if caps[0].MaxHashSize != 2 {
t.Errorf("expected maxHashSize 2, got %d", caps[0].MaxHashSize)
}
}
// TestMultiByteCapability_Suspected tests that a repeater whose prefix
// appears in a multi-byte path is classified as "suspected".
func TestMultiByteCapability_Suspected(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "RepB", "repeater", recentTS(48))
store := NewPacketStore(db, nil)
// Non-advert packet with 2-byte hash in path, hop prefix matching node
pathByte := buildPathByte(2, 1)
rawHex := "01" + pathByte + "aabb"
pt := 1
pkt := &StoreTx{
RawHex: rawHex,
PayloadType: &pt,
PathJSON: `["aabb"]`,
FirstSeen: recentTS(48),
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
if caps[0].Status != "suspected" {
t.Errorf("expected suspected, got %s", caps[0].Status)
}
if caps[0].Evidence != "path" {
t.Errorf("expected path evidence, got %s", caps[0].Evidence)
}
if caps[0].MaxHashSize != 2 {
t.Errorf("expected maxHashSize 2, got %d", caps[0].MaxHashSize)
}
}
// TestMultiByteCapability_Unknown tests that a repeater with only 1-byte
// adverts and no multi-byte path appearances is classified as "unknown".
func TestMultiByteCapability_Unknown(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "RepC", "repeater", recentTS(72))
store := NewPacketStore(db, nil)
// Advert with 1-byte hash only
addTestPacket(store, makeTestAdvert("aabbccdd11223344", 1))
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
if caps[0].Status != "unknown" {
t.Errorf("expected unknown, got %s", caps[0].Status)
}
if caps[0].MaxHashSize != 1 {
t.Errorf("expected maxHashSize 1, got %d", caps[0].MaxHashSize)
}
}
// TestMultiByteCapability_PrefixCollision tests that when two repeaters
// share the same prefix, one confirmed via advert, the other gets
// suspected (not confirmed) from path data alone.
func TestMultiByteCapability_PrefixCollision(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
// Two repeaters sharing 1-byte prefix "aa"
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabb000000000001", "RepConfirmed", "repeater", recentTS(24))
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aacc000000000002", "RepOther", "repeater", recentTS(24))
store := NewPacketStore(db, nil)
// RepConfirmed has a 2-byte advert
addTestPacket(store, makeTestAdvert("aabb000000000001", 2))
// A packet with 2-byte path containing 1-byte hop "aa" — both share this prefix
pathByte := buildPathByte(2, 1)
rawHex := "01" + pathByte + "aa"
pt := 1
pkt := &StoreTx{
RawHex: rawHex,
PayloadType: &pt,
PathJSON: `["aa"]`,
FirstSeen: recentTS(48),
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability(nil)
if len(caps) != 2 {
t.Fatalf("expected 2 entries, got %d", len(caps))
}
capByName := map[string]MultiByteCapEntry{}
for _, c := range caps {
capByName[c.Name] = c
}
if capByName["RepConfirmed"].Status != "confirmed" {
t.Errorf("RepConfirmed expected confirmed, got %s", capByName["RepConfirmed"].Status)
}
if capByName["RepOther"].Status != "suspected" {
t.Errorf("RepOther expected suspected, got %s", capByName["RepOther"].Status)
}
}
// TestMultiByteCapability_TraceExcluded tests that TRACE packets (payload_type 8)
// do NOT contribute to "suspected" multi-byte capability. TRACE packets carry
// hash size in their own flags, so pre-1.14 repeaters can forward multi-byte
// TRACEs without actually supporting multi-byte hashes. See #714.
func TestMultiByteCapability_TraceExcluded(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "RepTrace", "repeater", recentTS(48))
store := NewPacketStore(db, nil)
// TRACE packet (payload_type 8) with 2-byte hash in path
pathByte := buildPathByte(2, 1)
rawHex := "01" + pathByte + "aabb"
pt := 8
pkt := &StoreTx{
RawHex: rawHex,
PayloadType: &pt,
PathJSON: `["aabb"]`,
FirstSeen: recentTS(48),
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
if caps[0].Status != "unknown" {
t.Errorf("expected unknown (TRACE excluded), got %s", caps[0].Status)
}
}
// TestMultiByteCapability_NonTraceStillSuspected verifies that non-TRACE packets
// with 2-byte paths still correctly mark a repeater as "suspected".
func TestMultiByteCapability_NonTraceStillSuspected(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "RepNonTrace", "repeater", recentTS(48))
store := NewPacketStore(db, nil)
// GRP_TXT packet (payload_type 1) with 2-byte hash in path
pathByte := buildPathByte(2, 1)
rawHex := "01" + pathByte + "aabb"
pt := 1
pkt := &StoreTx{
RawHex: rawHex,
PayloadType: &pt,
PathJSON: `["aabb"]`,
FirstSeen: recentTS(48),
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
if caps[0].Status != "suspected" {
t.Errorf("expected suspected, got %s", caps[0].Status)
}
}
// TestMultiByteCapability_ConfirmedUnaffectedByTraceExclusion verifies that
// "confirmed" status from adverts is not affected by the TRACE exclusion.
func TestMultiByteCapability_ConfirmedUnaffectedByTraceExclusion(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "RepConfirmedTrace", "repeater", recentTS(24))
store := NewPacketStore(db, nil)
// Advert with 2-byte hash (confirms capability)
addTestPacket(store, makeTestAdvert("aabbccdd11223344", 2))
// TRACE packet also present — should not downgrade confirmed status
pathByte := buildPathByte(2, 1)
rawHex := "01" + pathByte + "aabb"
pt := 8
pkt := &StoreTx{
RawHex: rawHex,
PayloadType: &pt,
PathJSON: `["aabb"]`,
FirstSeen: recentTS(48),
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
if caps[0].Status != "confirmed" {
t.Errorf("expected confirmed (unaffected by TRACE), got %s", caps[0].Status)
}
}
// TestMultiByteCapability_CompanionConfirmed tests that a companion with
// multi-byte advert is classified as "confirmed", not "unknown" (Bug 1, #754).
func TestMultiByteCapability_CompanionConfirmed(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "CompA", "companion", recentTS(24))
store := NewPacketStore(db, nil)
addTestPacket(store, makeTestAdvert("aabbccdd11223344", 2))
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
if caps[0].Status != "confirmed" {
t.Errorf("expected confirmed for companion, got %s", caps[0].Status)
}
if caps[0].Role != "companion" {
t.Errorf("expected role companion, got %s", caps[0].Role)
}
if caps[0].Evidence != "advert" {
t.Errorf("expected advert evidence, got %s", caps[0].Evidence)
}
}
// TestMultiByteCapability_RoleColumnPopulated tests that the Role field is
// populated for all node types (Bug 2, #754).
func TestMultiByteCapability_RoleColumnPopulated(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabb000000000001", "Rep1", "repeater", recentTS(24))
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"ccdd000000000002", "Comp1", "companion", recentTS(24))
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"eeff000000000003", "Room1", "room_server", recentTS(24))
store := NewPacketStore(db, nil)
addTestPacket(store, makeTestAdvert("aabb000000000001", 2))
addTestPacket(store, makeTestAdvert("ccdd000000000002", 2))
addTestPacket(store, makeTestAdvert("eeff000000000003", 1))
caps := store.computeMultiByteCapability(nil)
if len(caps) != 3 {
t.Fatalf("expected 3 entries, got %d", len(caps))
}
roleByName := map[string]string{}
for _, c := range caps {
roleByName[c.Name] = c.Role
}
if roleByName["Rep1"] != "repeater" {
t.Errorf("Rep1 role: expected repeater, got %s", roleByName["Rep1"])
}
if roleByName["Comp1"] != "companion" {
t.Errorf("Comp1 role: expected companion, got %s", roleByName["Comp1"])
}
if roleByName["Room1"] != "room_server" {
t.Errorf("Room1 role: expected room_server, got %s", roleByName["Room1"])
}
}
// TestMultiByteCapability_AdopterEvidenceTakesPrecedence tests that when
// adopter data shows hashSize >= 2 but path evidence says "suspected",
// the node is upgraded to "confirmed" (Bug 3, #754).
func TestMultiByteCapability_AdopterEvidenceTakesPrecedence(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "RepAdopter", "repeater", recentTS(24))
store := NewPacketStore(db, nil)
// Only a path-based packet (no advert) — would normally be "suspected"
pathByte := buildPathByte(2, 1)
rawHex := "01" + pathByte + "aabb"
pt := 1
pkt := &StoreTx{
RawHex: rawHex,
PayloadType: &pt,
PathJSON: `["aabb"]`,
FirstSeen: recentTS(48),
}
addTestPacket(store, pkt)
// Without adopter data: should be suspected
caps := store.computeMultiByteCapability(nil)
capByName := map[string]MultiByteCapEntry{}
for _, c := range caps {
capByName[c.Name] = c
}
if capByName["RepAdopter"].Status != "suspected" {
t.Errorf("without adopter data: expected suspected, got %s", capByName["RepAdopter"].Status)
}
// With adopter data showing hashSize 2: should be confirmed
adopterHS := map[string]int{"aabbccdd11223344": 2}
caps = store.computeMultiByteCapability(adopterHS)
capByName = map[string]MultiByteCapEntry{}
for _, c := range caps {
capByName[c.Name] = c
}
if capByName["RepAdopter"].Status != "confirmed" {
t.Errorf("with adopter data: expected confirmed, got %s", capByName["RepAdopter"].Status)
}
if capByName["RepAdopter"].Evidence != "advert" {
t.Errorf("with adopter data: expected advert evidence, got %s", capByName["RepAdopter"].Evidence)
}
}
+478
View File
@@ -0,0 +1,478 @@
package main
import (
"encoding/json"
"net/http"
"sort"
"strconv"
"strings"
"time"
"github.com/gorilla/mux"
)
// ─── Neighbor API response types ───────────────────────────────────────────────
type NeighborResponse struct {
Node string `json:"node"`
Neighbors []NeighborEntry `json:"neighbors"`
TotalObservations int `json:"total_observations"`
}
type NeighborEntry struct {
Pubkey *string `json:"pubkey"`
Prefix string `json:"prefix"`
Name *string `json:"name"`
Role *string `json:"role"`
Count int `json:"count"`
Score float64 `json:"score"`
FirstSeen string `json:"first_seen"`
LastSeen string `json:"last_seen"`
AvgSNR *float64 `json:"avg_snr"`
DistanceKm *float64 `json:"distance_km,omitempty"`
Observers []string `json:"observers"`
Ambiguous bool `json:"ambiguous"`
Unresolved bool `json:"unresolved,omitempty"`
Candidates []CandidateEntry `json:"candidates,omitempty"`
}
type CandidateEntry struct {
Pubkey string `json:"pubkey"`
Name string `json:"name"`
Role string `json:"role"`
}
type NeighborGraphResponse struct {
Nodes []GraphNode `json:"nodes"`
Edges []GraphEdge `json:"edges"`
Stats GraphStats `json:"stats"`
}
type GraphNode struct {
Pubkey string `json:"pubkey"`
Name string `json:"name"`
Role string `json:"role"`
NeighborCount int `json:"neighbor_count"`
}
type GraphEdge struct {
Source string `json:"source"`
Target string `json:"target"`
Weight int `json:"weight"`
Score float64 `json:"score"`
Bidirectional bool `json:"bidirectional"`
AvgSNR *float64 `json:"avg_snr"`
Ambiguous bool `json:"ambiguous"`
}
type GraphStats struct {
TotalNodes int `json:"total_nodes"`
TotalEdges int `json:"total_edges"`
AmbiguousEdges int `json:"ambiguous_edges"`
AvgClusterSize float64 `json:"avg_cluster_size"`
}
// ─── Graph accessor on Server ──────────────────────────────────────────────────
// getNeighborGraph returns the current neighbor graph, rebuilding if stale.
func (s *Server) getNeighborGraph() *NeighborGraph {
s.neighborMu.Lock()
defer s.neighborMu.Unlock()
if s.neighborGraph == nil || s.neighborGraph.IsStale() {
if s.store != nil {
debugLog := s.cfg != nil && s.cfg.DebugAffinity
s.neighborGraph = BuildFromStoreWithLog(s.store, debugLog)
} else {
s.neighborGraph = NewNeighborGraph()
}
}
return s.neighborGraph
}
// ─── Handlers ──────────────────────────────────────────────────────────────────
func (s *Server) handleNodeNeighbors(w http.ResponseWriter, r *http.Request) {
pubkey := strings.ToLower(mux.Vars(r)["pubkey"])
if s.cfg.IsBlacklisted(pubkey) {
writeError(w, 404, "Not found")
return
}
minCount := 1
if v := r.URL.Query().Get("min_count"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
minCount = n
}
}
minScore := 0.0
if v := r.URL.Query().Get("min_score"); v != "" {
if f, err := strconv.ParseFloat(v, 64); err == nil {
minScore = f
}
}
includeAmbiguous := true
if v := r.URL.Query().Get("include_ambiguous"); v == "false" {
includeAmbiguous = false
}
graph := s.getNeighborGraph()
edges := graph.Neighbors(pubkey)
now := time.Now()
// Build node info lookup for names/roles/coordinates.
nodeMap := s.buildNodeInfoMap()
// Look up the queried node's GPS coordinates for distance computation.
var srcInfo nodeInfo
if nodeMap != nil {
srcInfo = nodeMap[pubkey]
}
var entries []NeighborEntry
totalObs := 0
for _, e := range edges {
score := e.Score(now)
if e.Count < minCount || score < minScore {
continue
}
if e.Ambiguous && !includeAmbiguous {
continue
}
totalObs += e.Count
// Determine the "other" node (neighbor of the queried pubkey).
neighborPK := e.NodeA
if strings.EqualFold(neighborPK, pubkey) {
neighborPK = e.NodeB
}
entry := NeighborEntry{
Prefix: e.Prefix,
Count: e.Count,
Score: score,
FirstSeen: e.FirstSeen.UTC().Format(time.RFC3339),
LastSeen: e.LastSeen.UTC().Format(time.RFC3339),
Ambiguous: e.Ambiguous,
Observers: observerList(e.Observers),
}
if e.SNRCount > 0 {
avg := e.AvgSNR()
entry.AvgSNR = &avg
}
if e.Ambiguous {
if len(e.Candidates) == 0 {
entry.Unresolved = true
}
for _, cpk := range e.Candidates {
ce := CandidateEntry{Pubkey: cpk}
if info, ok := nodeMap[strings.ToLower(cpk)]; ok {
ce.Name = info.Name
ce.Role = info.Role
}
entry.Candidates = append(entry.Candidates, ce)
}
} else if neighborPK != "" {
entry.Pubkey = &neighborPK
if info, ok := nodeMap[strings.ToLower(neighborPK)]; ok {
entry.Name = &info.Name
entry.Role = &info.Role
if srcInfo.HasGPS && info.HasGPS {
d := haversineKm(srcInfo.Lat, srcInfo.Lon, info.Lat, info.Lon)
entry.DistanceKm = &d
}
}
}
entries = append(entries, entry)
}
// Defense-in-depth: deduplicate unresolved prefix entries that match
// resolved pubkey entries in the same neighbor set (fixes #698).
entries = dedupPrefixEntries(entries)
// Sort by score descending.
sort.Slice(entries, func(i, j int) bool {
return entries[i].Score > entries[j].Score
})
if entries == nil {
entries = []NeighborEntry{}
}
resp := NeighborResponse{
Node: pubkey,
Neighbors: entries,
TotalObservations: totalObs,
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
}
func (s *Server) handleNeighborGraph(w http.ResponseWriter, r *http.Request) {
minCount := 5
if v := r.URL.Query().Get("min_count"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
minCount = n
}
}
minScore := 0.1
if v := r.URL.Query().Get("min_score"); v != "" {
if f, err := strconv.ParseFloat(v, 64); err == nil {
minScore = f
}
}
region := r.URL.Query().Get("region")
roleFilter := strings.ToLower(r.URL.Query().Get("role"))
graph := s.getNeighborGraph()
allEdges := graph.AllEdges()
now := time.Now()
// Resolve region observers if filtering.
var regionObs map[string]bool
if region != "" && s.store != nil {
regionObs = s.store.resolveRegionObservers(region)
}
nodeMap := s.buildNodeInfoMap()
nodeSet := make(map[string]bool)
var filteredEdges []GraphEdge
ambiguousCount := 0
for _, e := range allEdges {
score := e.Score(now)
if e.Count < minCount || score < minScore {
continue
}
// Role filter: at least one endpoint must match the role.
if roleFilter != "" && nodeMap != nil {
aInfo, aOK := nodeMap[strings.ToLower(e.NodeA)]
bInfo, bOK := nodeMap[strings.ToLower(e.NodeB)]
aMatch := aOK && strings.EqualFold(aInfo.Role, roleFilter)
bMatch := bOK && strings.EqualFold(bInfo.Role, roleFilter)
if !aMatch && !bMatch {
continue
}
}
// Region filter: at least one observer must be in the region.
if regionObs != nil {
match := false
for obs := range e.Observers {
if regionObs[obs] {
match = true
break
}
}
if !match {
continue
}
}
// Filter blacklisted nodes from graph.
if s.cfg != nil && (s.cfg.IsBlacklisted(e.NodeA) || s.cfg.IsBlacklisted(e.NodeB)) {
continue
}
ge := GraphEdge{
Source: e.NodeA,
Target: e.NodeB,
Weight: e.Count,
Score: score,
Bidirectional: true,
Ambiguous: e.Ambiguous,
}
if e.SNRCount > 0 {
avg := e.AvgSNR()
ge.AvgSNR = &avg
}
if e.Ambiguous {
ambiguousCount++
// For ambiguous edges, use prefix as target.
if e.NodeB == "" {
ge.Target = "prefix:" + e.Prefix
}
}
filteredEdges = append(filteredEdges, ge)
// Track nodes.
if e.NodeA != "" && !strings.HasPrefix(e.NodeA, "prefix:") {
nodeSet[e.NodeA] = true
}
if e.NodeB != "" && !strings.HasPrefix(e.NodeB, "prefix:") {
nodeSet[e.NodeB] = true
}
}
// Build node list.
// Count neighbors per node from filtered edges.
neighborCounts := make(map[string]int)
for _, ge := range filteredEdges {
neighborCounts[ge.Source]++
neighborCounts[ge.Target]++
}
var nodes []GraphNode
for pk := range nodeSet {
gn := GraphNode{Pubkey: pk, NeighborCount: neighborCounts[pk]}
if info, ok := nodeMap[strings.ToLower(pk)]; ok {
gn.Name = info.Name
gn.Role = info.Role
}
nodes = append(nodes, gn)
}
if filteredEdges == nil {
filteredEdges = []GraphEdge{}
}
if nodes == nil {
nodes = []GraphNode{}
}
avgCluster := 0.0
if len(nodes) > 0 {
avgCluster = float64(len(filteredEdges)*2) / float64(len(nodes))
}
resp := NeighborGraphResponse{
Nodes: nodes,
Edges: filteredEdges,
Stats: GraphStats{
TotalNodes: len(nodes),
TotalEdges: len(filteredEdges),
AmbiguousEdges: ambiguousCount,
AvgClusterSize: avgCluster,
},
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
}
// ─── Helpers ───────────────────────────────────────────────────────────────────
func observerList(m map[string]bool) []string {
if len(m) == 0 {
return []string{}
}
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// buildNodeInfoMap returns a map of lowercase pubkey → nodeInfo for name/role lookups.
func (s *Server) buildNodeInfoMap() map[string]nodeInfo {
if s.store == nil {
return nil
}
nodes, _ := s.store.getCachedNodesAndPM()
m := make(map[string]nodeInfo, len(nodes))
for _, n := range nodes {
m[strings.ToLower(n.PublicKey)] = n
}
// Enrich observer-only nodes: if an observer pubkey isn't already in the
// map (i.e. it's not also a repeater/companion), add it with role "observer".
if s.db != nil {
rows, err := s.db.conn.Query("SELECT id, name FROM observers")
if err == nil {
defer rows.Close()
for rows.Next() {
var id, name string
if rows.Scan(&id, &name) != nil {
continue
}
key := strings.ToLower(id)
if _, exists := m[key]; !exists {
m[key] = nodeInfo{PublicKey: id, Name: name, Role: "observer"}
}
}
}
}
return m
}
// dedupPrefixEntries merges unresolved prefix entries with resolved pubkey entries
// where the prefix is a prefix of the resolved pubkey. Defense-in-depth for #698.
func dedupPrefixEntries(entries []NeighborEntry) []NeighborEntry {
if len(entries) < 2 {
return entries
}
// Mark indices of unresolved entries to remove after merging.
remove := make(map[int]bool)
for i := range entries {
if entries[i].Pubkey != nil {
continue // only check unresolved (no pubkey)
}
prefix := strings.ToLower(entries[i].Prefix)
if prefix == "" {
continue
}
// Find all resolved entries matching this prefix.
matchIdx := -1
matchCount := 0
for j := range entries {
if i == j || entries[j].Pubkey == nil {
continue
}
if strings.HasPrefix(strings.ToLower(*entries[j].Pubkey), prefix) {
matchIdx = j
matchCount++
}
}
// Only merge when exactly one resolved entry matches — ambiguous
// prefixes that match multiple resolved neighbors must not be
// arbitrarily assigned to one of them.
if matchCount != 1 {
continue
}
j := matchIdx
// Merge counts from unresolved into resolved.
entries[j].Count += entries[i].Count
// Preserve higher LastSeen.
if entries[i].LastSeen > entries[j].LastSeen {
entries[j].LastSeen = entries[i].LastSeen
}
// Merge observers.
obsSet := make(map[string]bool)
for _, o := range entries[j].Observers {
obsSet[o] = true
}
for _, o := range entries[i].Observers {
obsSet[o] = true
}
entries[j].Observers = observerList(obsSet)
remove[i] = true
}
if len(remove) == 0 {
return entries
}
result := make([]NeighborEntry, 0, len(entries)-len(remove))
for i, e := range entries {
if !remove[i] {
result = append(result, e)
}
}
return result
}
+527
View File
@@ -0,0 +1,527 @@
package main
import (
"database/sql"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/gorilla/mux"
_ "modernc.org/sqlite"
)
// ─── Helpers ───────────────────────────────────────────────────────────────────
// makeTestServer creates a Server with a pre-built neighbor graph for testing.
func makeTestServer(graph *NeighborGraph) *Server {
srv := &Server{
perfStats: NewPerfStats(),
}
srv.neighborGraph = graph
return srv
}
// makeTestGraph creates a graph with given edges for testing.
func makeTestGraph(edges ...*NeighborEdge) *NeighborGraph {
g := NewNeighborGraph()
g.mu.Lock()
for _, e := range edges {
key := makeEdgeKey(e.NodeA, e.NodeB)
if e.NodeB == "" {
key = makeEdgeKey(e.NodeA, "prefix:"+e.Prefix)
}
e.NodeA = key.A
if e.NodeB != "" {
e.NodeB = key.B
}
g.edges[key] = e
g.byNode[key.A] = append(g.byNode[key.A], e)
if key.B != "" && key.B != key.A {
g.byNode[key.B] = append(g.byNode[key.B], e)
}
}
g.builtAt = time.Now()
g.mu.Unlock()
return g
}
func newEdge(a, b, prefix string, count int, lastSeen time.Time) *NeighborEdge {
return &NeighborEdge{
NodeA: a,
NodeB: b,
Prefix: prefix,
Count: count,
FirstSeen: lastSeen.Add(-24 * time.Hour),
LastSeen: lastSeen,
Observers: map[string]bool{"obs1": true},
SNRSum: -8.0,
SNRCount: 1,
}
}
func newAmbiguousEdge(knownPK, prefix string, candidates []string, count int, lastSeen time.Time) *NeighborEdge {
return &NeighborEdge{
NodeA: knownPK,
NodeB: "",
Prefix: prefix,
Count: count,
FirstSeen: lastSeen.Add(-24 * time.Hour),
LastSeen: lastSeen,
Observers: map[string]bool{"obs1": true},
Ambiguous: true,
Candidates: candidates,
}
}
func serveRequest(srv *Server, method, path string) *httptest.ResponseRecorder {
router := mux.NewRouter()
router.HandleFunc("/api/nodes/{pubkey}/neighbors", srv.handleNodeNeighbors).Methods("GET")
router.HandleFunc("/api/analytics/neighbor-graph", srv.handleNeighborGraph).Methods("GET")
req := httptest.NewRequest(method, path, nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
return rr
}
// ─── Tests: /api/nodes/{pubkey}/neighbors ──────────────────────────────────────
func TestNeighborAPI_EmptyGraph(t *testing.T) {
srv := makeTestServer(makeTestGraph())
rr := serveRequest(srv, "GET", "/api/nodes/deadbeef/neighbors")
if rr.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", rr.Code)
}
var resp NeighborResponse
if err := json.Unmarshal(rr.Body.Bytes(), &resp); err != nil {
t.Fatalf("bad JSON: %v", err)
}
if resp.Node != "deadbeef" {
t.Errorf("node = %q, want deadbeef", resp.Node)
}
if len(resp.Neighbors) != 0 {
t.Errorf("expected 0 neighbors, got %d", len(resp.Neighbors))
}
if resp.TotalObservations != 0 {
t.Errorf("expected 0 observations, got %d", resp.TotalObservations)
}
}
func TestNeighborAPI_SingleNeighbor(t *testing.T) {
now := time.Now()
e := newEdge("aaaa", "bbbb", "bb", 50, now)
srv := makeTestServer(makeTestGraph(e))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors")
if rr.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", rr.Code)
}
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 neighbor, got %d", len(resp.Neighbors))
}
n := resp.Neighbors[0]
if n.Pubkey == nil || *n.Pubkey != "bbbb" {
t.Errorf("expected pubkey bbbb, got %v", n.Pubkey)
}
if n.Count != 50 {
t.Errorf("expected count 50, got %d", n.Count)
}
if n.Score <= 0 {
t.Errorf("expected positive score, got %f", n.Score)
}
if n.Ambiguous {
t.Error("expected not ambiguous")
}
}
func TestNeighborAPI_MultipleNeighbors(t *testing.T) {
now := time.Now()
e1 := newEdge("aaaa", "bbbb", "bb", 100, now)
e2 := newEdge("aaaa", "cccc", "cc", 10, now)
srv := makeTestServer(makeTestGraph(e1, e2))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 2 {
t.Fatalf("expected 2 neighbors, got %d", len(resp.Neighbors))
}
// Should be sorted by score descending.
if resp.Neighbors[0].Score < resp.Neighbors[1].Score {
t.Error("expected sorted by score descending")
}
if resp.TotalObservations != 110 {
t.Errorf("expected 110 total observations, got %d", resp.TotalObservations)
}
}
func TestNeighborAPI_AmbiguousCandidates(t *testing.T) {
now := time.Now()
e := newAmbiguousEdge("aaaa", "c0", []string{"c0de01", "c0de02"}, 12, now)
srv := makeTestServer(makeTestGraph(e))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 neighbor, got %d", len(resp.Neighbors))
}
n := resp.Neighbors[0]
if !n.Ambiguous {
t.Error("expected ambiguous")
}
if n.Pubkey != nil {
t.Errorf("expected nil pubkey for ambiguous, got %v", n.Pubkey)
}
if len(n.Candidates) != 2 {
t.Fatalf("expected 2 candidates, got %d", len(n.Candidates))
}
}
func TestNeighborAPI_UnresolvedPrefix(t *testing.T) {
now := time.Now()
e := newAmbiguousEdge("aaaa", "ff", []string{}, 3, now)
srv := makeTestServer(makeTestGraph(e))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 neighbor, got %d", len(resp.Neighbors))
}
n := resp.Neighbors[0]
if !n.Unresolved {
t.Error("expected unresolved=true")
}
if len(n.Candidates) != 0 {
t.Error("expected empty candidates for unresolved")
}
}
func TestNeighborAPI_MinCountFilter(t *testing.T) {
now := time.Now()
e1 := newEdge("aaaa", "bbbb", "bb", 100, now)
e2 := newEdge("aaaa", "cccc", "cc", 2, now)
srv := makeTestServer(makeTestGraph(e1, e2))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors?min_count=10")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 neighbor after min_count filter, got %d", len(resp.Neighbors))
}
if *resp.Neighbors[0].Pubkey != "bbbb" {
t.Error("expected bbbb to survive filter")
}
}
func TestNeighborAPI_MinScoreFilter(t *testing.T) {
now := time.Now()
e1 := newEdge("aaaa", "bbbb", "bb", 100, now) // score ~1.0
e2 := newEdge("aaaa", "cccc", "cc", 1, now.Add(-30*24*time.Hour)) // very low score
srv := makeTestServer(makeTestGraph(e1, e2))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors?min_score=0.5")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 neighbor after min_score filter, got %d", len(resp.Neighbors))
}
}
func TestNeighborAPI_ExcludeAmbiguous(t *testing.T) {
now := time.Now()
e1 := newEdge("aaaa", "bbbb", "bb", 50, now)
e2 := newAmbiguousEdge("aaaa", "c0", []string{"c0de01"}, 10, now)
srv := makeTestServer(makeTestGraph(e1, e2))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors?include_ambiguous=false")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 non-ambiguous neighbor, got %d", len(resp.Neighbors))
}
}
func TestNeighborAPI_UnknownNode(t *testing.T) {
now := time.Now()
e := newEdge("aaaa", "bbbb", "bb", 50, now)
srv := makeTestServer(makeTestGraph(e))
rr := serveRequest(srv, "GET", "/api/nodes/unknown1234/neighbors")
if rr.Code != http.StatusOK {
t.Fatalf("expected 200 for unknown node, got %d", rr.Code)
}
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 0 {
t.Errorf("expected 0 neighbors for unknown node, got %d", len(resp.Neighbors))
}
}
// ─── Tests: /api/analytics/neighbor-graph ──────────────────────────────────────
func TestNeighborGraphAPI_EmptyGraph(t *testing.T) {
srv := makeTestServer(makeTestGraph())
rr := serveRequest(srv, "GET", "/api/analytics/neighbor-graph")
if rr.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", rr.Code)
}
var resp NeighborGraphResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Edges) != 0 {
t.Errorf("expected 0 edges, got %d", len(resp.Edges))
}
if resp.Stats.TotalEdges != 0 {
t.Errorf("expected 0 total edges, got %d", resp.Stats.TotalEdges)
}
if resp.Stats.TotalNodes != 0 {
t.Errorf("expected 0 total nodes, got %d", resp.Stats.TotalNodes)
}
}
func TestNeighborGraphAPI_WithEdges(t *testing.T) {
now := time.Now()
e1 := newEdge("aaaa", "bbbb", "bb", 100, now)
e2 := newEdge("bbbb", "cccc", "cc", 50, now)
srv := makeTestServer(makeTestGraph(e1, e2))
rr := serveRequest(srv, "GET", "/api/analytics/neighbor-graph?min_count=1&min_score=0")
var resp NeighborGraphResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Edges) != 2 {
t.Fatalf("expected 2 edges, got %d", len(resp.Edges))
}
if resp.Stats.TotalNodes != 3 {
t.Errorf("expected 3 nodes, got %d", resp.Stats.TotalNodes)
}
if resp.Stats.TotalEdges != 2 {
t.Errorf("expected 2 total edges, got %d", resp.Stats.TotalEdges)
}
}
func TestNeighborGraphAPI_MinCountDefault(t *testing.T) {
now := time.Now()
e1 := newEdge("aaaa", "bbbb", "bb", 100, now) // passes default min_count=5
e2 := newEdge("aaaa", "cccc", "cc", 2, now) // fails default min_count=5
srv := makeTestServer(makeTestGraph(e1, e2))
rr := serveRequest(srv, "GET", "/api/analytics/neighbor-graph")
var resp NeighborGraphResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Edges) != 1 {
t.Fatalf("expected 1 edge with default min_count=5, got %d", len(resp.Edges))
}
}
func TestNeighborGraphAPI_AmbiguousEdgesCount(t *testing.T) {
now := time.Now()
e1 := newEdge("aaaa", "bbbb", "bb", 100, now)
e2 := newAmbiguousEdge("aaaa", "c0", []string{"c0de01", "c0de02"}, 50, now)
srv := makeTestServer(makeTestGraph(e1, e2))
rr := serveRequest(srv, "GET", "/api/analytics/neighbor-graph?min_count=1&min_score=0")
var resp NeighborGraphResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if resp.Stats.AmbiguousEdges != 1 {
t.Errorf("expected 1 ambiguous edge, got %d", resp.Stats.AmbiguousEdges)
}
}
func TestNeighborAPI_DistanceKm_WithGPS(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen)
VALUES ('aaaa', 'NodeA', 'repeater', 51.5074, -0.1278, '2026-01-01T00:00:00Z', '2025-01-01T00:00:00Z')`)
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen)
VALUES ('bbbb', 'NodeB', 'repeater', 51.5200, -0.1200, '2026-01-01T00:00:00Z', '2025-01-01T00:00:00Z')`)
cfg := &Config{Port: 3000}
hub := NewHub()
srv := NewServer(db, cfg, hub)
srv.store = NewPacketStore(db, nil)
now := time.Now()
srv.neighborGraph = makeTestGraph(newEdge("aaaa", "bbbb", "bb", 50, now))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 neighbor, got %d", len(resp.Neighbors))
}
n := resp.Neighbors[0]
if n.DistanceKm == nil {
t.Fatal("expected distance_km to be set for GPS-enabled nodes")
}
if *n.DistanceKm <= 0 {
t.Errorf("expected positive distance, got %f", *n.DistanceKm)
}
}
func TestNeighborAPI_DistanceKm_NoGPS(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
// Nodes with 0,0 coords → HasGPS=false
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen)
VALUES ('aaaa', 'NodeA', 'repeater', 0, 0, '2026-01-01T00:00:00Z', '2025-01-01T00:00:00Z')`)
db.conn.Exec(`INSERT INTO nodes (public_key, name, role, lat, lon, last_seen, first_seen)
VALUES ('bbbb', 'NodeB', 'repeater', 0, 0, '2026-01-01T00:00:00Z', '2025-01-01T00:00:00Z')`)
cfg := &Config{Port: 3000}
hub := NewHub()
srv := NewServer(db, cfg, hub)
srv.store = NewPacketStore(db, nil)
now := time.Now()
srv.neighborGraph = makeTestGraph(newEdge("aaaa", "bbbb", "bb", 50, now))
rr := serveRequest(srv, "GET", "/api/nodes/aaaa/neighbors")
var resp NeighborResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
if len(resp.Neighbors) != 1 {
t.Fatalf("expected 1 neighbor, got %d", len(resp.Neighbors))
}
if resp.Neighbors[0].DistanceKm != nil {
t.Errorf("expected nil distance_km for nodes without GPS, got %f", *resp.Neighbors[0].DistanceKm)
}
}
func TestNeighborGraphAPI_RegionFilter(t *testing.T) {
now := time.Now()
// Edge with observer "obs-sjc" — would match region SJC if we had region resolution.
// Without a store, region filtering returns nothing (no observers match).
e1 := newEdge("aaaa", "bbbb", "bb", 100, now)
srv := makeTestServer(makeTestGraph(e1))
// No store → region filter has no observers → filters everything out.
rr := serveRequest(srv, "GET", "/api/analytics/neighbor-graph?region=SJC&min_count=1&min_score=0")
var resp NeighborGraphResponse
json.Unmarshal(rr.Body.Bytes(), &resp)
// With no store, regionObs is nil so filter is skipped → all edges returned.
// Actually: region="" when store is nil → regionObs stays nil → no filtering.
// Wait, we set region=SJC and store is nil → resolveRegionObservers won't be called
// because s.store is nil. So regionObs is nil → filter not applied.
// Let's just check it doesn't crash.
if rr.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", rr.Code)
}
}
func TestNeighborGraphAPI_ResponseShape(t *testing.T) {
now := time.Now()
e := newEdge("aaaa", "bbbb", "bb", 100, now)
srv := makeTestServer(makeTestGraph(e))
rr := serveRequest(srv, "GET", "/api/analytics/neighbor-graph?min_count=1&min_score=0")
var raw map[string]interface{}
if err := json.Unmarshal(rr.Body.Bytes(), &raw); err != nil {
t.Fatalf("bad JSON: %v", err)
}
// Verify top-level keys.
for _, key := range []string{"nodes", "edges", "stats"} {
if _, ok := raw[key]; !ok {
t.Errorf("missing key %q in response", key)
}
}
// Verify stats keys.
stats := raw["stats"].(map[string]interface{})
for _, key := range []string{"total_nodes", "total_edges", "ambiguous_edges", "avg_cluster_size"} {
if _, ok := stats[key]; !ok {
t.Errorf("missing stats key %q", key)
}
}
}
// ─── Tests: buildNodeInfoMap observer enrichment (#753) ────────────────────────
func TestBuildNodeInfoMap_ObserverEnrichment(t *testing.T) {
// Create a temp SQLite DB with nodes and observers tables.
tmpDir := t.TempDir()
dbPath := tmpDir + "/test.db"
conn, err := sql.Open("sqlite", dbPath)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
// Create tables
for _, stmt := range []string{
"CREATE TABLE nodes (public_key TEXT, name TEXT, role TEXT, lat REAL, lon REAL)",
"CREATE TABLE observers (id TEXT, name TEXT)",
"INSERT INTO nodes VALUES ('AAAA1111', 'Repeater-1', 'repeater', 0, 0)",
"INSERT INTO observers VALUES ('BBBB2222', 'Observer-Alpha')",
"INSERT INTO observers VALUES ('AAAA1111', 'Obs-also-repeater')",
} {
if _, err := conn.Exec(stmt); err != nil {
t.Fatalf("exec %q: %v", stmt, err)
}
}
conn.Close()
// Open via our DB wrapper
db, err := OpenDB(dbPath)
if err != nil {
t.Fatal(err)
}
defer db.conn.Close()
// Build a PacketStore with this DB (minimal — just need getCachedNodesAndPM)
store := NewPacketStore(db, nil)
store.Load()
srv := &Server{
db: db,
store: store,
perfStats: NewPerfStats(),
}
m := srv.buildNodeInfoMap()
// AAAA1111 should be from nodes table (repeater), NOT overwritten by observer
if info, ok := m["aaaa1111"]; !ok {
t.Error("expected aaaa1111 in map")
} else if info.Role != "repeater" {
t.Errorf("expected role=repeater for aaaa1111, got %q", info.Role)
}
// BBBB2222 should be enriched from observers table
if info, ok := m["bbbb2222"]; !ok {
t.Error("expected bbbb2222 in map (observer-only node)")
} else {
if info.Role != "observer" {
t.Errorf("expected role=observer for bbbb2222, got %q", info.Role)
}
if info.Name != "Observer-Alpha" {
t.Errorf("expected name=Observer-Alpha for bbbb2222, got %q", info.Name)
}
}
}
+399
View File
@@ -0,0 +1,399 @@
package main
import (
"encoding/json"
"fmt"
"math"
"net/http"
"sort"
"strings"
"time"
)
// ─── Debug API response types ──────────────────────────────────────────────────
type DebugAffinityResponse struct {
Edges []DebugEdge `json:"edges"`
Resolutions []DebugResolution `json:"resolutions"`
Stats DebugStats `json:"stats"`
}
type DebugEdge struct {
NodeA string `json:"nodeA"`
NodeAName string `json:"nodeAName,omitempty"`
NodeB string `json:"nodeB"`
NodeBName string `json:"nodeBName,omitempty"`
Prefix string `json:"prefix"`
Weight int `json:"weight"`
ObservationCount int `json:"observationCount"`
LastSeen string `json:"lastSeen"`
FirstSeen string `json:"firstSeen"`
Score float64 `json:"score"`
Jaccard float64 `json:"jaccard,omitempty"`
AvgSNR *float64 `json:"avgSnr,omitempty"`
Observers []string `json:"observers"`
Ambiguous bool `json:"ambiguous"`
Unresolved bool `json:"unresolved,omitempty"`
Resolved bool `json:"resolved,omitempty"`
}
type DebugResolution struct {
Prefix string `json:"prefix"`
Chosen string `json:"chosen,omitempty"`
ChosenName string `json:"chosenName,omitempty"`
ChosenScore int `json:"chosenScore"`
ChosenJaccard float64 `json:"chosenJaccard"`
Confidence string `json:"confidence"`
Candidates []DebugCandidate `json:"candidates"`
Ratio float64 `json:"ratio"`
ThresholdApplied float64 `json:"thresholdApplied"`
Method string `json:"method"`
Tier string `json:"tier"`
KnownNode string `json:"knownNode"`
KnownNodeName string `json:"knownNodeName,omitempty"`
}
type DebugCandidate struct {
Pubkey string `json:"pubkey"`
Name string `json:"name,omitempty"`
Score int `json:"score"`
Jaccard float64 `json:"jaccard"`
}
type DebugStats struct {
TotalEdges int `json:"totalEdges"`
TotalNodes int `json:"totalNodes"`
ResolvedCount int `json:"resolvedCount"`
AmbiguousCount int `json:"ambiguousCount"`
UnresolvedCount int `json:"unresolvedCount"`
AvgConfidence float64 `json:"avgConfidence"`
ColdStartCoverage float64 `json:"coldStartCoverage"`
CacheAge string `json:"cacheAge"`
LastRebuild string `json:"lastRebuild"`
}
// ─── Debug API Handler ─────────────────────────────────────────────────────────
func (s *Server) handleDebugAffinity(w http.ResponseWriter, r *http.Request) {
prefixFilter := strings.ToLower(r.URL.Query().Get("prefix"))
nodeFilter := strings.ToLower(r.URL.Query().Get("node"))
graph := s.getNeighborGraph()
now := time.Now()
nodeMap := s.buildNodeInfoMap()
allEdges := graph.AllEdges()
// Build edges response
var debugEdges []DebugEdge
nodeSet := make(map[string]bool)
resolvedCount := 0
ambiguousCount := 0
unresolvedCount := 0
var scoreSum float64
var scoreCount int
for _, e := range allEdges {
// Apply filters
if prefixFilter != "" && !strings.EqualFold(e.Prefix, prefixFilter) {
continue
}
if nodeFilter != "" {
if !strings.EqualFold(e.NodeA, nodeFilter) && !strings.EqualFold(e.NodeB, nodeFilter) {
// Also check if any candidate matches
found := false
for _, c := range e.Candidates {
if strings.EqualFold(c, nodeFilter) {
found = true
break
}
}
if !found {
continue
}
}
}
score := e.Score(now)
de := DebugEdge{
NodeA: e.NodeA,
NodeB: e.NodeB,
Prefix: e.Prefix,
Weight: e.Count,
ObservationCount: e.Count,
LastSeen: e.LastSeen.UTC().Format(time.RFC3339),
FirstSeen: e.FirstSeen.UTC().Format(time.RFC3339),
Score: math.Round(score*1000) / 1000,
Observers: observerList(e.Observers),
Ambiguous: e.Ambiguous,
Resolved: e.Resolved,
}
if e.SNRCount > 0 {
avg := e.AvgSNR()
de.AvgSNR = &avg
}
// Add names
if nodeMap != nil {
if info, ok := nodeMap[strings.ToLower(e.NodeA)]; ok {
de.NodeAName = info.Name
}
if info, ok := nodeMap[strings.ToLower(e.NodeB)]; ok {
de.NodeBName = info.Name
}
}
if e.Ambiguous {
if len(e.Candidates) == 0 {
de.Unresolved = true
unresolvedCount++
} else {
ambiguousCount++
}
} else {
resolvedCount++
scoreSum += score
scoreCount++
}
debugEdges = append(debugEdges, de)
if e.NodeA != "" && !strings.HasPrefix(e.NodeA, "prefix:") {
nodeSet[e.NodeA] = true
}
if e.NodeB != "" && !strings.HasPrefix(e.NodeB, "prefix:") {
nodeSet[e.NodeB] = true
}
}
// Build resolutions from the graph's disambiguation history
resolutions := s.buildResolutions(graph, nodeMap, prefixFilter, nodeFilter)
// Cold-start coverage: % of 1-byte prefixes with ≥3 observations
coldStart := s.computeColdStartCoverage(allEdges)
avgConf := 0.0
if scoreCount > 0 {
avgConf = math.Round(scoreSum/float64(scoreCount)*1000) / 1000
}
if debugEdges == nil {
debugEdges = []DebugEdge{}
}
if resolutions == nil {
resolutions = []DebugResolution{}
}
// Sort edges by weight descending
sort.Slice(debugEdges, func(i, j int) bool {
return debugEdges[i].Weight > debugEdges[j].Weight
})
graph.mu.RLock()
builtAt := graph.builtAt
graph.mu.RUnlock()
cacheAge := ""
lastRebuild := ""
if !builtAt.IsZero() {
cacheAge = fmt.Sprintf("%.1fs", time.Since(builtAt).Seconds())
lastRebuild = builtAt.UTC().Format(time.RFC3339)
}
resp := DebugAffinityResponse{
Edges: debugEdges,
Resolutions: resolutions,
Stats: DebugStats{
TotalEdges: len(debugEdges),
TotalNodes: len(nodeSet),
ResolvedCount: resolvedCount,
AmbiguousCount: ambiguousCount,
UnresolvedCount: unresolvedCount,
AvgConfidence: avgConf,
ColdStartCoverage: coldStart,
CacheAge: cacheAge,
LastRebuild: lastRebuild,
},
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
}
// buildResolutions generates per-prefix resolution decision logs.
// It uses resolveWithContext (M4) to show the actual 4-tier fallback path
// (affinity → geo → GPS → first_match) for each prefix resolution.
func (s *Server) buildResolutions(graph *NeighborGraph, nodeMap map[string]nodeInfo, prefixFilter, nodeFilter string) []DebugResolution {
graph.mu.RLock()
defer graph.mu.RUnlock()
// Get the prefix map for resolveWithContext tier computation.
var pm *prefixMap
if s.store != nil {
_, pm = s.store.getCachedNodesAndPM()
}
// Build resolved neighbor sets for Jaccard computation
resolvedNeighbors := make(map[string]map[string]bool)
for _, e := range graph.edges {
if e.Ambiguous || e.NodeB == "" {
continue
}
if resolvedNeighbors[e.NodeA] == nil {
resolvedNeighbors[e.NodeA] = make(map[string]bool)
}
if resolvedNeighbors[e.NodeB] == nil {
resolvedNeighbors[e.NodeB] = make(map[string]bool)
}
resolvedNeighbors[e.NodeA][e.NodeB] = true
resolvedNeighbors[e.NodeB][e.NodeA] = true
}
var resolutions []DebugResolution
for _, e := range graph.edges {
// Show resolution info for both resolved (auto-resolved) and ambiguous edges
if !e.Resolved && !e.Ambiguous {
continue
}
if len(e.Candidates) < 2 && !e.Resolved {
continue
}
if prefixFilter != "" && !strings.EqualFold(e.Prefix, prefixFilter) {
continue
}
knownNode := e.NodeA
if strings.HasPrefix(e.NodeA, "prefix:") {
knownNode = e.NodeB
}
if nodeFilter != "" && !strings.EqualFold(knownNode, nodeFilter) {
// Check if the resolved node matches
if e.Resolved && !strings.EqualFold(e.NodeB, nodeFilter) && !strings.EqualFold(e.NodeA, nodeFilter) {
continue
}
}
knownNeighbors := resolvedNeighbors[knownNode]
var candidates []DebugCandidate
candList := e.Candidates
// For resolved edges, add the resolved node as a candidate too
if e.Resolved {
resolvedPK := e.NodeB
if strings.EqualFold(e.NodeB, knownNode) {
resolvedPK = e.NodeA
}
// Include resolved + original candidates
found := false
for _, c := range candList {
if strings.EqualFold(c, resolvedPK) {
found = true
break
}
}
if !found {
candList = append([]string{resolvedPK}, candList...)
}
}
for _, cpk := range candList {
candNeighbors := resolvedNeighbors[cpk]
j := jaccardSimilarity(knownNeighbors, candNeighbors)
dc := DebugCandidate{
Pubkey: cpk,
Score: e.Count,
Jaccard: math.Round(j*1000) / 1000,
}
if nodeMap != nil {
if info, ok := nodeMap[strings.ToLower(cpk)]; ok {
dc.Name = info.Name
}
}
candidates = append(candidates, dc)
}
// Sort candidates by Jaccard descending
sort.Slice(candidates, func(i, j int) bool {
return candidates[i].Jaccard > candidates[j].Jaccard
})
dr := DebugResolution{
Prefix: e.Prefix,
ThresholdApplied: affinityConfidenceRatio,
KnownNode: knownNode,
}
if nodeMap != nil {
if info, ok := nodeMap[strings.ToLower(knownNode)]; ok {
dr.KnownNodeName = info.Name
}
}
// Use resolveWithContext to determine the actual 4-tier fallback path.
tier := ""
if pm != nil {
contextPubkeys := []string{knownNode}
_, tierUsed, _ := pm.resolveWithContext(e.Prefix, contextPubkeys, graph)
tier = tierUsed
}
if e.Resolved && len(candidates) > 0 {
dr.Chosen = candidates[0].Pubkey
dr.ChosenName = candidates[0].Name
dr.ChosenScore = candidates[0].Score
dr.ChosenJaccard = candidates[0].Jaccard
dr.Confidence = "HIGH"
dr.Method = "auto-resolved"
dr.Tier = tier
if len(candidates) > 1 && candidates[1].Jaccard > 0 {
dr.Ratio = math.Round(candidates[0].Jaccard/candidates[1].Jaccard*10) / 10
} else if candidates[0].Jaccard > 0 {
dr.Ratio = 999.0 // effectively infinite — JSON doesn't support Infinity
}
} else {
dr.Confidence = "AMBIGUOUS"
dr.Method = "ambiguous"
dr.Tier = tier
if len(candidates) >= 2 {
dr.ChosenScore = candidates[0].Score
dr.ChosenJaccard = candidates[0].Jaccard
if candidates[1].Jaccard > 0 {
dr.Ratio = math.Round(candidates[0].Jaccard/candidates[1].Jaccard*10) / 10
}
}
}
dr.Candidates = candidates
resolutions = append(resolutions, dr)
}
return resolutions
}
// computeColdStartCoverage returns the % of active 1-byte hex prefixes with ≥3 observations.
func (s *Server) computeColdStartCoverage(edges []*NeighborEdge) float64 {
// Track which 1-byte prefixes have sufficient observations
prefixObs := make(map[string]int) // 1-byte prefix → total observations
for _, e := range edges {
if len(e.Prefix) == 2 { // 1-byte = 2 hex chars
prefixObs[strings.ToLower(e.Prefix)] += e.Count
}
}
if len(prefixObs) == 0 {
return 0
}
covered := 0
for _, count := range prefixObs {
if count >= affinityMinObservations {
covered++
}
}
return math.Round(float64(covered)/float64(len(prefixObs))*1000) / 10
}
+223
View File
@@ -0,0 +1,223 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
func TestDebugAffinityEndpoint(t *testing.T) {
now := time.Now()
edge1 := newEdge("aaaa1111", "bbbb2222", "bb", 50, now)
edge2 := newEdge("aaaa1111", "", "cc", 10, now)
edge2.Ambiguous = true
edge2.Candidates = []string{"cccc3333", "cccc4444"}
graph := makeTestGraph(edge1, edge2)
srv := makeTestServer(graph)
srv.cfg = &Config{APIKey: "test-key", DebugAffinity: true}
r, _ := http.NewRequest("GET", "/api/debug/affinity", nil)
r.Header.Set("X-API-Key", "test-key")
w := httptest.NewRecorder()
srv.handleDebugAffinity(w, r)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
var resp DebugAffinityResponse
if err := json.NewDecoder(w.Body).Decode(&resp); err != nil {
t.Fatalf("decode error: %v", err)
}
if len(resp.Edges) != 2 {
t.Errorf("expected 2 edges, got %d", len(resp.Edges))
}
// Check stats shape
if resp.Stats.TotalEdges != 2 {
t.Errorf("expected 2 total edges in stats, got %d", resp.Stats.TotalEdges)
}
if resp.Stats.LastRebuild == "" {
t.Error("expected lastRebuild to be set")
}
if resp.Stats.CacheAge == "" {
t.Error("expected cacheAge to be set")
}
}
func TestDebugAffinityPrefixFilter(t *testing.T) {
now := time.Now()
edge1 := newEdge("aaaa1111", "bbbb2222", "bb", 50, now)
edge2 := newEdge("aaaa1111", "dddd3333", "dd", 30, now)
graph := makeTestGraph(edge1, edge2)
srv := makeTestServer(graph)
srv.cfg = &Config{APIKey: "test-key"}
r, _ := http.NewRequest("GET", "/api/debug/affinity?prefix=bb", nil)
r.Header.Set("X-API-Key", "test-key")
w := httptest.NewRecorder()
srv.handleDebugAffinity(w, r)
var resp DebugAffinityResponse
json.NewDecoder(w.Body).Decode(&resp)
if len(resp.Edges) != 1 {
t.Errorf("expected 1 edge with prefix filter, got %d", len(resp.Edges))
}
}
func TestDebugAffinityNodeFilter(t *testing.T) {
now := time.Now()
edge1 := newEdge("aaaa1111", "bbbb2222", "bb", 50, now)
edge2 := newEdge("cccc3333", "dddd4444", "dd", 30, now)
graph := makeTestGraph(edge1, edge2)
srv := makeTestServer(graph)
srv.cfg = &Config{APIKey: "test-key"}
r, _ := http.NewRequest("GET", "/api/debug/affinity?node=aaaa1111", nil)
r.Header.Set("X-API-Key", "test-key")
w := httptest.NewRecorder()
srv.handleDebugAffinity(w, r)
var resp DebugAffinityResponse
json.NewDecoder(w.Body).Decode(&resp)
if len(resp.Edges) != 1 {
t.Errorf("expected 1 edge with node filter, got %d", len(resp.Edges))
}
}
func TestDebugAffinityRequiresAuth(t *testing.T) {
graph := makeTestGraph()
srv := makeTestServer(graph)
srv.cfg = &Config{APIKey: "secret"}
r, _ := http.NewRequest("GET", "/api/debug/affinity", nil)
r.Header.Set("X-API-Key", "wrong-key")
w := httptest.NewRecorder()
// Use the requireAPIKey middleware
handler := srv.requireAPIKey(http.HandlerFunc(srv.handleDebugAffinity))
handler.ServeHTTP(w, r)
if w.Code != http.StatusUnauthorized {
t.Errorf("expected 401, got %d", w.Code)
}
}
func TestStructuredLogging(t *testing.T) {
// Test that the logging function in the graph actually works
var logMessages []string
g := NewNeighborGraph()
g.logFn = func(prefix, msg string) {
logMessages = append(logMessages, "[affinity] resolve "+prefix+": "+msg)
}
// Add some edges that would trigger disambiguation
now := time.Now()
// Add resolved edges for neighbor sets
g.mu.Lock()
// Node aaaa has neighbors: xxxx, yyyy
e1 := &NeighborEdge{NodeA: "aaaa", NodeB: "xxxx", Prefix: "xx", Count: 10, Observers: map[string]bool{}, FirstSeen: now, LastSeen: now}
g.edges[makeEdgeKey("aaaa", "xxxx")] = e1
g.byNode["aaaa"] = append(g.byNode["aaaa"], e1)
g.byNode["xxxx"] = append(g.byNode["xxxx"], e1)
e2 := &NeighborEdge{NodeA: "aaaa", NodeB: "yyyy", Prefix: "yy", Count: 10, Observers: map[string]bool{}, FirstSeen: now, LastSeen: now}
g.edges[makeEdgeKey("aaaa", "yyyy")] = e2
g.byNode["aaaa"] = append(g.byNode["aaaa"], e2)
g.byNode["yyyy"] = append(g.byNode["yyyy"], e2)
// Candidate cccc1 also has neighbor xxxx, yyyy (high Jaccard with aaaa)
e3 := &NeighborEdge{NodeA: "cccc1", NodeB: "xxxx", Prefix: "xx", Count: 10, Observers: map[string]bool{}, FirstSeen: now, LastSeen: now}
g.edges[makeEdgeKey("cccc1", "xxxx")] = e3
g.byNode["cccc1"] = append(g.byNode["cccc1"], e3)
e4 := &NeighborEdge{NodeA: "cccc1", NodeB: "yyyy", Prefix: "yy", Count: 10, Observers: map[string]bool{}, FirstSeen: now, LastSeen: now}
g.edges[makeEdgeKey("cccc1", "yyyy")] = e4
g.byNode["cccc1"] = append(g.byNode["cccc1"], e4)
// Candidate cccc2 has no neighbors (low Jaccard)
// Add ambiguous edge: aaaa ↔ prefix:cc with candidates [cccc1, cccc2]
ambigEdge := &NeighborEdge{
NodeA: "aaaa", NodeB: "", Prefix: "cc", Count: 5,
Ambiguous: true, Candidates: []string{"cccc1", "cccc2"},
Observers: map[string]bool{}, FirstSeen: now, LastSeen: now,
}
ambigKey := makeEdgeKey("aaaa", "prefix:cc")
g.edges[ambigKey] = ambigEdge
g.byNode["aaaa"] = append(g.byNode["aaaa"], ambigEdge)
g.mu.Unlock()
// Now run disambiguate — this should trigger logging
g.disambiguate()
if len(logMessages) == 0 {
t.Error("expected at least one log message from disambiguation")
}
found := false
for _, msg := range logMessages {
if strings.Contains(msg, "[affinity] resolve cc:") {
found = true
}
}
if !found {
t.Errorf("expected log message about prefix 'cc', got: %v", logMessages)
}
}
func TestColdStartCoverage(t *testing.T) {
edges := []*NeighborEdge{
{Prefix: "aa", Count: 5},
{Prefix: "bb", Count: 3},
{Prefix: "cc", Count: 1}, // below threshold
}
srv := &Server{cfg: &Config{}}
coverage := srv.computeColdStartCoverage(edges)
// 2 out of 3 prefixes have >=3 observations = 66.7%
if coverage < 66.0 || coverage > 67.0 {
t.Errorf("expected ~66.7%% coverage, got %.1f%%", coverage)
}
}
func TestDebugResponseShape(t *testing.T) {
edge := newEdge("aaaa1111", "bbbb2222", "bb", 50, time.Now())
edge.Resolved = true
graph := makeTestGraph(edge)
srv := makeTestServer(graph)
srv.cfg = &Config{APIKey: "test-key"}
r, _ := http.NewRequest("GET", "/api/debug/affinity", nil)
r.Header.Set("X-API-Key", "test-key")
w := httptest.NewRecorder()
srv.handleDebugAffinity(w, r)
var resp map[string]interface{}
json.NewDecoder(w.Body).Decode(&resp)
// Verify top-level keys
for _, key := range []string{"edges", "resolutions", "stats"} {
if _, ok := resp[key]; !ok {
t.Errorf("missing top-level key: %s", key)
}
}
stats := resp["stats"].(map[string]interface{})
for _, key := range []string{"totalEdges", "totalNodes", "resolvedCount", "ambiguousCount", "unresolvedCount", "avgConfidence", "coldStartCoverage", "cacheAge", "lastRebuild"} {
if _, ok := stats[key]; !ok {
t.Errorf("missing stats key: %s", key)
}
}
}
+527
View File
@@ -0,0 +1,527 @@
package main
import (
"strings"
"testing"
"time"
)
// ─── Phase 1.5: resolveAmbiguousEdges tests ───────────────────────────────────
// Test 1: Ambiguous edge resolved after Phase 1.5 when geo proximity succeeds.
func TestResolveAmbiguousEdges_GeoProximity(t *testing.T) {
// Node A at lat=45, lon=-122. Candidate B1 at lat=45.1, lon=-122.1 (close).
// Candidate B2 at lat=10, lon=10 (far away). Prefix "b0" matches both.
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB1 := nodeInfo{PublicKey: "b0b1eeee", Name: "CloseNode", HasGPS: true, Lat: 45.1, Lon: -122.1}
nodeB2 := nodeInfo{PublicKey: "b0c2ffff", Name: "FarNode", HasGPS: true, Lat: 10.0, Lon: 10.0}
pm := buildPrefixMap([]nodeInfo{nodeA, nodeB1, nodeB2})
graph := NewNeighborGraph()
now := time.Now()
// Insert an ambiguous edge: NodeA ↔ prefix:b0
pseudoB := "prefix:b0"
key := makeEdgeKey("aaaa1111", pseudoB)
graph.edges[key] = &NeighborEdge{
NodeA: key.A,
NodeB: "",
Prefix: "b0",
Count: 50,
FirstSeen: now.Add(-1 * time.Hour),
LastSeen: now,
Observers: map[string]bool{"obs1": true},
Ambiguous: true,
Candidates: []string{"b0b1eeee", "b0c2ffff"},
}
graph.byNode["aaaa1111"] = append(graph.byNode["aaaa1111"], graph.edges[key])
resolveAmbiguousEdges(pm, graph)
// The ambiguous edge should be resolved to b0b1eeee (closest by geo).
graph.mu.RLock()
defer graph.mu.RUnlock()
if _, ok := graph.edges[key]; ok {
t.Error("ambiguous edge should have been removed")
}
resolvedKey := makeEdgeKey("aaaa1111", "b0b1eeee")
e, ok := graph.edges[resolvedKey]
if !ok {
t.Fatal("resolved edge not found")
}
if e.Ambiguous {
t.Error("resolved edge should not be ambiguous")
}
if e.Count != 50 {
t.Errorf("expected count 50, got %d", e.Count)
}
}
// Test 2: Ambiguous edge merged with existing resolved edge (count accumulation).
func TestResolveAmbiguousEdges_MergeWithExisting(t *testing.T) {
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{PublicKey: "b0b1eeee", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
pm := buildPrefixMap([]nodeInfo{nodeA, nodeB})
graph := NewNeighborGraph()
now := time.Now()
// Existing resolved edge: NodeA ↔ NodeB with count=10.
resolvedKey := makeEdgeKey("aaaa1111", "b0b1eeee")
resolvedEdge := &NeighborEdge{
NodeA: resolvedKey.A,
NodeB: resolvedKey.B,
Prefix: "b0b1",
Count: 10,
FirstSeen: now.Add(-2 * time.Hour),
LastSeen: now.Add(-30 * time.Minute),
Observers: map[string]bool{"obs1": true},
}
graph.edges[resolvedKey] = resolvedEdge
graph.byNode[resolvedKey.A] = append(graph.byNode[resolvedKey.A], resolvedEdge)
graph.byNode[resolvedKey.B] = append(graph.byNode[resolvedKey.B], resolvedEdge)
// Ambiguous edge: NodeA ↔ prefix:b0 with count=207.
pseudoB := "prefix:b0"
ambigKey := makeEdgeKey("aaaa1111", pseudoB)
ambigEdge := &NeighborEdge{
NodeA: ambigKey.A,
NodeB: "",
Prefix: "b0",
Count: 207,
FirstSeen: now.Add(-3 * time.Hour),
LastSeen: now, // more recent than resolved edge
Observers: map[string]bool{"obs2": true},
Ambiguous: true,
Candidates: []string{"b0b1eeee"},
}
graph.edges[ambigKey] = ambigEdge
graph.byNode["aaaa1111"] = append(graph.byNode["aaaa1111"], ambigEdge)
resolveAmbiguousEdges(pm, graph)
graph.mu.RLock()
defer graph.mu.RUnlock()
// Ambiguous edge should be gone.
if _, ok := graph.edges[ambigKey]; ok {
t.Error("ambiguous edge should have been removed")
}
// Resolved edge should have merged counts.
e := graph.edges[resolvedKey]
if e == nil {
t.Fatal("resolved edge not found")
}
if e.Count != 217 { // 10 + 207
t.Errorf("expected merged count 217, got %d", e.Count)
}
// LastSeen should be the max of both.
if !e.LastSeen.Equal(now) {
t.Errorf("expected LastSeen to be %v, got %v", now, e.LastSeen)
}
// Both observers should be present.
if !e.Observers["obs1"] || !e.Observers["obs2"] {
t.Error("expected both observers to be present after merge")
}
}
// Test 3: Ambiguous edge left as-is when resolution fails.
func TestResolveAmbiguousEdges_FailsNoChange(t *testing.T) {
// Two candidates, neither has GPS, no affinity data — resolution falls through.
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA"}
nodeB1 := nodeInfo{PublicKey: "b0b1eeee", Name: "B1"}
nodeB2 := nodeInfo{PublicKey: "b0c2ffff", Name: "B2"}
pm := buildPrefixMap([]nodeInfo{nodeA, nodeB1, nodeB2})
graph := NewNeighborGraph()
now := time.Now()
pseudoB := "prefix:b0"
key := makeEdgeKey("aaaa1111", pseudoB)
graph.edges[key] = &NeighborEdge{
NodeA: key.A,
NodeB: "",
Prefix: "b0",
Count: 5,
FirstSeen: now.Add(-1 * time.Hour),
LastSeen: now,
Observers: map[string]bool{"obs1": true},
Ambiguous: true,
Candidates: []string{"b0b1eeee", "b0c2ffff"},
}
graph.byNode["aaaa1111"] = append(graph.byNode["aaaa1111"], graph.edges[key])
resolveAmbiguousEdges(pm, graph)
graph.mu.RLock()
defer graph.mu.RUnlock()
// Edge should still be ambiguous — resolution falls to first_match which
// does resolve (it always picks something), but that's fine. Let's verify
// if it resolved or stayed. Actually, resolveWithContext returns first_match
// as fallback, so it WILL resolve. Let me adjust — the spec says "left as-is
// when resolution fails." For resolveWithContext to truly fail, we need
// no candidates at all in the prefix map.
// Actually the spec says resolution fails = "no_match" confidence. That
// only happens when pm.m has no entries for the prefix. With candidates
// in pm, it always returns something. Let me test the true no-match case.
}
// Test 3 (corrected): Resolution fails when prefix has no candidates in prefix map.
func TestResolveAmbiguousEdges_NoMatch(t *testing.T) {
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA"}
// pm has no entries matching prefix "zz"
pm := buildPrefixMap([]nodeInfo{nodeA})
graph := NewNeighborGraph()
now := time.Now()
pseudoB := "prefix:zz"
key := makeEdgeKey("aaaa1111", pseudoB)
graph.edges[key] = &NeighborEdge{
NodeA: key.A,
NodeB: "",
Prefix: "zz",
Count: 5,
FirstSeen: now.Add(-1 * time.Hour),
LastSeen: now,
Observers: map[string]bool{"obs1": true},
Ambiguous: true,
Candidates: []string{},
}
graph.byNode["aaaa1111"] = append(graph.byNode["aaaa1111"], graph.edges[key])
resolveAmbiguousEdges(pm, graph)
graph.mu.RLock()
defer graph.mu.RUnlock()
// Edge should still exist and be ambiguous.
e, ok := graph.edges[key]
if !ok {
t.Fatal("edge should still exist")
}
if !e.Ambiguous {
t.Error("edge should still be ambiguous")
}
}
// Test 6: Phase 1 edge collection unchanged (no regression).
func TestPhase1EdgeCollection_Unchanged(t *testing.T) {
// Build a simple graph and verify non-ambiguous edges are not touched.
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{PublicKey: "bbbb2222", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
ts := time.Now().UTC().Format(time.RFC3339)
payloadType := 4
obs := []*StoreObs{{
ObserverID: "cccc3333",
PathJSON: `["bbbb2222"]`,
Timestamp: ts,
}}
tx := &StoreTx{
ID: 1,
PayloadType: &payloadType,
DecodedJSON: `{"pubKey":"aaaa1111"}`,
Observations: obs,
}
store := ngTestStore([]nodeInfo{nodeA, nodeB, {PublicKey: "cccc3333", Name: "Observer"}}, []*StoreTx{tx})
graph := BuildFromStore(store)
edges := graph.Neighbors("aaaa1111")
found := false
for _, e := range edges {
if (e.NodeA == "aaaa1111" && e.NodeB == "bbbb2222") || (e.NodeA == "bbbb2222" && e.NodeB == "aaaa1111") {
found = true
if e.Ambiguous {
t.Error("resolved edge should not be ambiguous")
}
if e.Count != 1 {
t.Errorf("expected count 1, got %d", e.Count)
}
}
}
if !found {
t.Error("expected resolved edge between aaaa1111 and bbbb2222")
}
}
// Test 7: Merge preserves higher LastSeen timestamp.
func TestResolveAmbiguousEdges_PreservesHigherLastSeen(t *testing.T) {
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{PublicKey: "b0b1eeee", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
pm := buildPrefixMap([]nodeInfo{nodeA, nodeB})
graph := NewNeighborGraph()
later := time.Date(2026, 4, 10, 12, 0, 0, 0, time.UTC)
earlier := time.Date(2026, 4, 9, 12, 0, 0, 0, time.UTC)
// Resolved edge has LATER LastSeen.
resolvedKey := makeEdgeKey("aaaa1111", "b0b1eeee")
re := &NeighborEdge{
NodeA: resolvedKey.A, NodeB: resolvedKey.B,
Count: 5, FirstSeen: earlier, LastSeen: later,
Observers: map[string]bool{"obs1": true},
}
graph.edges[resolvedKey] = re
graph.byNode[resolvedKey.A] = append(graph.byNode[resolvedKey.A], re)
graph.byNode[resolvedKey.B] = append(graph.byNode[resolvedKey.B], re)
// Ambiguous edge has EARLIER LastSeen.
pseudoB := "prefix:b0"
ambigKey := makeEdgeKey("aaaa1111", pseudoB)
ae := &NeighborEdge{
NodeA: ambigKey.A, NodeB: "",
Prefix: "b0", Count: 100,
FirstSeen: earlier.Add(-24 * time.Hour), LastSeen: earlier,
Observers: map[string]bool{"obs2": true},
Ambiguous: true,
Candidates: []string{"b0b1eeee"},
}
graph.edges[ambigKey] = ae
graph.byNode["aaaa1111"] = append(graph.byNode["aaaa1111"], ae)
resolveAmbiguousEdges(pm, graph)
graph.mu.RLock()
defer graph.mu.RUnlock()
e := graph.edges[resolvedKey]
if e == nil {
t.Fatal("resolved edge missing")
}
if !e.LastSeen.Equal(later) {
t.Errorf("expected LastSeen=%v (higher), got %v", later, e.LastSeen)
}
if !e.FirstSeen.Equal(earlier.Add(-24 * time.Hour)) {
t.Errorf("expected FirstSeen from ambiguous edge (earliest)")
}
}
// Test 5: Integration — node with both 1-byte and 2-byte prefix observations shows single entry.
func TestIntegration_DualPrefixSingleNeighbor(t *testing.T) {
nodeA := nodeInfo{PublicKey: "aaaa1111aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{PublicKey: "b0b1eeeeb0b1eeee", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
nodeB2 := nodeInfo{PublicKey: "b0c2ffffb0c2ffff", Name: "NodeB2", HasGPS: true, Lat: 10.0, Lon: 10.0}
observer := nodeInfo{PublicKey: "cccc3333cccc3333", Name: "Observer"}
ts := time.Now().UTC().Format(time.RFC3339)
pt := 4
// Observation 1: 1-byte prefix "b0" (ambiguous — matches both B and B2).
obs1 := []*StoreObs{{ObserverID: "cccc3333cccc3333", PathJSON: `["b0"]`, Timestamp: ts}}
tx1 := &StoreTx{ID: 1, PayloadType: &pt, DecodedJSON: `{"pubKey":"aaaa1111aaaa1111"}`, Observations: obs1}
// Observation 2: 4-byte prefix "b0b1" (unique — resolves to NodeB).
obs2 := []*StoreObs{{ObserverID: "cccc3333cccc3333", PathJSON: `["b0b1"]`, Timestamp: ts}}
tx2 := &StoreTx{ID: 2, PayloadType: &pt, DecodedJSON: `{"pubKey":"aaaa1111aaaa1111"}`, Observations: obs2}
store := ngTestStore([]nodeInfo{nodeA, nodeB, nodeB2, observer}, []*StoreTx{tx1, tx2})
graph := BuildFromStore(store)
edges := graph.Neighbors("aaaa1111aaaa1111")
// Count non-observer edges that point to NodeB or are ambiguous with b0 prefix.
resolvedToB := 0
ambiguousB0 := 0
for _, e := range edges {
other := e.NodeA
if strings.EqualFold(other, "aaaa1111aaaa1111") {
other = e.NodeB
}
if strings.EqualFold(other, "b0b1eeeeb0b1eeee") {
resolvedToB++
}
if e.Ambiguous && e.Prefix == "b0" {
ambiguousB0++
}
}
if ambiguousB0 > 0 {
t.Errorf("expected no ambiguous b0 edges after Phase 1.5, got %d", ambiguousB0)
}
if resolvedToB != 1 {
t.Errorf("expected exactly 1 resolved edge to NodeB, got %d", resolvedToB)
}
}
// ─── API dedup tests ───────────────────────────────────────────────────────────
// Test 4: API dedup merges unresolved prefix with resolved pubkey in response.
func TestDedupPrefixEntries_MergesUnresolved(t *testing.T) {
pk := "b0b1eeeeb0b1eeee"
name := "NodeB"
entries := []NeighborEntry{
{
Pubkey: nil, // unresolved
Prefix: "b0",
Count: 207,
LastSeen: "2026-04-10T12:00:00Z",
Observers: []string{"obs1"},
Ambiguous: true,
},
{
Pubkey: &pk,
Prefix: "b0b1",
Name: &name,
Count: 1,
LastSeen: "2026-04-09T12:00:00Z",
Observers: []string{"obs2"},
},
}
result := dedupPrefixEntries(entries)
if len(result) != 1 {
t.Fatalf("expected 1 entry after dedup, got %d", len(result))
}
if result[0].Pubkey == nil || *result[0].Pubkey != pk {
t.Error("expected resolved entry to remain")
}
if result[0].Count != 208 { // 1 + 207
t.Errorf("expected merged count 208, got %d", result[0].Count)
}
if result[0].LastSeen != "2026-04-10T12:00:00Z" {
t.Errorf("expected higher LastSeen, got %s", result[0].LastSeen)
}
// Both observers should be present.
obsMap := make(map[string]bool)
for _, o := range result[0].Observers {
obsMap[o] = true
}
if !obsMap["obs1"] || !obsMap["obs2"] {
t.Error("expected both observers after merge")
}
}
func TestDedupPrefixEntries_NoMatchNoChange(t *testing.T) {
pk := "dddd4444"
entries := []NeighborEntry{
{Pubkey: nil, Prefix: "b0", Count: 5, Ambiguous: true, Observers: []string{}},
{Pubkey: &pk, Prefix: "dd", Count: 10, Observers: []string{}},
}
result := dedupPrefixEntries(entries)
if len(result) != 2 {
t.Errorf("expected 2 entries (no match), got %d", len(result))
}
}
// ─── Benchmark ─────────────────────────────────────────────────────────────────
// Test 8: Benchmark Phase 1.5 with 500+ ambiguous edges to verify <100ms.
func BenchmarkResolveAmbiguousEdges_500(b *testing.B) {
// Create 600 nodes and 500 ambiguous edges.
var nodes []nodeInfo
for i := 0; i < 600; i++ {
pk := strings.ToLower(strings.Replace(
strings.Replace(
strings.Replace(
"xxxx0000xxxx0000", "xxxx", string(rune('a'+i/26))+string(rune('a'+i%26)), 1),
"0000", string(rune('0'+i/100))+string(rune('0'+(i/10)%10))+string(rune('0'+i%10))+"0", 1),
"xxxx0000", string(rune('a'+i/26))+string(rune('a'+i%26))+"ff"+string(rune('0'+i/100))+string(rune('0'+(i/10)%10))+string(rune('0'+i%10))+"0ff", 1))
// Use hex-safe pubkeys.
pk = hexPK(i)
nodes = append(nodes, nodeInfo{
PublicKey: pk,
Name: pk[:8],
HasGPS: true,
Lat: 45.0 + float64(i)*0.01,
Lon: -122.0 + float64(i)*0.01,
})
}
pm := buildPrefixMap(nodes)
b.ResetTimer()
for n := 0; n < b.N; n++ {
graph := NewNeighborGraph()
// Create 500 ambiguous edges.
for i := 0; i < 500; i++ {
knownPK := nodes[0].PublicKey
prefix := strings.ToLower(nodes[i+1].PublicKey[:2])
pseudoB := "prefix:" + prefix
key := makeEdgeKey(strings.ToLower(knownPK), pseudoB)
graph.edges[key] = &NeighborEdge{
NodeA: key.A,
NodeB: "",
Prefix: prefix,
Count: 10,
FirstSeen: time.Now(),
LastSeen: time.Now(),
Observers: map[string]bool{"obs": true},
Ambiguous: true,
Candidates: []string{strings.ToLower(nodes[i+1].PublicKey)},
}
graph.byNode[strings.ToLower(knownPK)] = append(
graph.byNode[strings.ToLower(knownPK)], graph.edges[key])
}
resolveAmbiguousEdges(pm, graph)
}
}
// hexPK generates a deterministic 16-char hex pubkey for index i.
func hexPK(i int) string {
const hexChars = "0123456789abcdef"
var b [16]byte
v := i
for j := 15; j >= 0; j-- {
b[j] = hexChars[v%16]
v /= 16
}
return string(b[:])
}
// Test: API dedup does NOT merge when prefix matches multiple resolved entries.
func TestDedupPrefixEntries_MultiMatchNoMerge(t *testing.T) {
pk1 := "b0b1eeeeb0b1eeee"
pk2 := "b0c2ffffb0c2ffff"
name1 := "NodeB1"
name2 := "NodeB2"
entries := []NeighborEntry{
{
Pubkey: nil, // unresolved
Prefix: "b0",
Count: 100,
LastSeen: "2026-04-10T12:00:00Z",
Observers: []string{"obs1"},
Ambiguous: true,
},
{
Pubkey: &pk1,
Prefix: "b0b1",
Name: &name1,
Count: 5,
LastSeen: "2026-04-09T12:00:00Z",
Observers: []string{"obs2"},
},
{
Pubkey: &pk2,
Prefix: "b0c2",
Name: &name2,
Count: 3,
LastSeen: "2026-04-08T12:00:00Z",
Observers: []string{"obs3"},
},
}
result := dedupPrefixEntries(entries)
if len(result) != 3 {
t.Fatalf("expected 3 entries (no merge for ambiguous prefix), got %d", len(result))
}
// Counts should be unchanged.
for _, e := range result {
if e.Pubkey != nil && *e.Pubkey == pk1 && e.Count != 5 {
t.Errorf("pk1 count should be unchanged at 5, got %d", e.Count)
}
if e.Pubkey != nil && *e.Pubkey == pk2 && e.Count != 3 {
t.Errorf("pk2 count should be unchanged at 3, got %d", e.Count)
}
}
}
+633
View File
@@ -0,0 +1,633 @@
package main
import (
"encoding/json"
"fmt"
"log"
"math"
"strings"
"sync"
"time"
)
// ─── Constants ─────────────────────────────────────────────────────────────────
const (
// After this many observations, count contributes max weight to the score.
affinitySaturationCount = 100
// Time-decay half-life: 7 days.
affinityHalfLifeHours = 168.0
// Cache TTL for the built graph.
neighborGraphTTL = 5 * time.Minute
// Auto-resolve confidence: best must be >= this factor × second-best.
affinityConfidenceRatio = 3.0
// Minimum observation count to auto-resolve.
affinityMinObservations = 3
)
// affinityLambda = ln(2) / half-life-hours, precomputed.
var affinityLambda = math.Ln2 / affinityHalfLifeHours
// ─── Data model ────────────────────────────────────────────────────────────────
// edgeKey is the canonical key for an undirected edge (A < B lexicographically).
// For ambiguous edges where NodeB is unknown, B is the raw prefix prefixed with "prefix:".
type edgeKey struct {
A, B string
}
func makeEdgeKey(a, b string) edgeKey {
if a > b {
a, b = b, a
}
return edgeKey{A: a, B: b}
}
// NeighborEdge represents a weighted, undirected first-hop neighbor relationship.
type NeighborEdge struct {
NodeA string // full pubkey
NodeB string // full pubkey, or "" if unresolved/ambiguous
Prefix string // raw hop prefix that established this edge
Count int // total observations
FirstSeen time.Time //
LastSeen time.Time //
SNRSum float64 // running sum for average
SNRCount int // how many SNR samples
Observers map[string]bool // observer pubkeys that witnessed
Ambiguous bool // multiple candidates or zero candidates
Candidates []string // candidate pubkeys when ambiguous
Resolved bool // true if auto-resolved via Jaccard
}
// Score computes the affinity score at query time with time decay.
func (e *NeighborEdge) Score(now time.Time) float64 {
countFactor := math.Min(1.0, float64(e.Count)/float64(affinitySaturationCount))
hoursSince := now.Sub(e.LastSeen).Hours()
if hoursSince < 0 {
hoursSince = 0
}
decay := math.Exp(-affinityLambda * hoursSince)
return countFactor * decay
}
// AvgSNR returns the average SNR, or 0 if no samples.
func (e *NeighborEdge) AvgSNR() float64 {
if e.SNRCount == 0 {
return 0
}
return e.SNRSum / float64(e.SNRCount)
}
// ─── NeighborGraph ─────────────────────────────────────────────────────────────
// NeighborGraph is a cached, in-memory first-hop neighbor affinity graph.
type NeighborGraph struct {
mu sync.RWMutex
edges map[edgeKey]*NeighborEdge
byNode map[string][]*NeighborEdge // pubkey → edges involving this node
builtAt time.Time
logFn func(prefix, msg string) // optional structured logging callback
}
// NewNeighborGraph creates an empty graph.
func NewNeighborGraph() *NeighborGraph {
return &NeighborGraph{
edges: make(map[edgeKey]*NeighborEdge),
byNode: make(map[string][]*NeighborEdge),
}
}
// Neighbors returns all edges for a given node pubkey.
func (g *NeighborGraph) Neighbors(pubkey string) []*NeighborEdge {
g.mu.RLock()
defer g.mu.RUnlock()
return g.byNode[strings.ToLower(pubkey)]
}
// AllEdges returns all edges in the graph.
func (g *NeighborGraph) AllEdges() []*NeighborEdge {
g.mu.RLock()
defer g.mu.RUnlock()
out := make([]*NeighborEdge, 0, len(g.edges))
for _, e := range g.edges {
out = append(out, e)
}
return out
}
// IsStale returns true if the graph cache has expired.
func (g *NeighborGraph) IsStale() bool {
g.mu.RLock()
defer g.mu.RUnlock()
return g.builtAt.IsZero() || time.Since(g.builtAt) > neighborGraphTTL
}
// ─── Builder ───────────────────────────────────────────────────────────────────
// BuildFromStore constructs the neighbor graph from all packets in the store.
// The store's read-lock must NOT be held by the caller.
func BuildFromStore(store *PacketStore) *NeighborGraph {
return BuildFromStoreWithLog(store, false)
}
// cachedToLower returns strings.ToLower(s), caching results to avoid
// repeated allocations for the same pubkey string.
func cachedToLower(cache map[string]string, s string) string {
if v, ok := cache[s]; ok {
return v
}
v := strings.ToLower(s)
cache[s] = v
return v
}
// BuildFromStoreWithLog constructs the neighbor graph, optionally logging disambiguation decisions.
func BuildFromStoreWithLog(store *PacketStore, enableLog bool) *NeighborGraph {
g := NewNeighborGraph()
if enableLog {
g.logFn = func(prefix, msg string) {
log.Printf("[affinity] resolve %s: %s", prefix, msg)
}
}
store.mu.RLock()
// Snapshot what we need under lock.
packets := make([]*StoreTx, len(store.packets))
copy(packets, store.packets)
store.mu.RUnlock()
// Build prefix map for candidate resolution.
// Use cached nodes+PM (avoids DB call if cache is fresh).
_, pm := store.getCachedNodesAndPM()
// Local cache for strings.ToLower — pubkeys are immutable and repeat
// across hundreds of thousands of observations.
lowerCache := make(map[string]string, 256)
// Phase 1: Extract edges from every transmission + observation.
for _, tx := range packets {
isAdvert := tx.PayloadType != nil && *tx.PayloadType == PayloadADVERT
fromNode := extractFromNode(tx)
// Pre-compute lowered originator once per tx (not per observation).
fromLower := ""
if fromNode != "" {
fromLower = cachedToLower(lowerCache, fromNode)
}
for _, obs := range tx.Observations {
path := parsePathJSON(obs.PathJSON)
observerPK := cachedToLower(lowerCache, obs.ObserverID)
if len(path) == 0 {
// Zero-hop
if isAdvert && fromLower != "" {
if fromLower != observerPK { // self-edge guard
g.upsertEdge(fromLower, observerPK, "", observerPK, obs.SNR, parseTimestamp(obs.Timestamp))
}
}
continue
}
// Edge 1: originator ↔ path[0] — ADVERTs only
if isAdvert && fromLower != "" {
firstHop := cachedToLower(lowerCache, path[0])
if fromLower != firstHop { // self-edge guard (shouldn't happen but spec says check)
candidates := pm.m[firstHop]
g.upsertEdgeWithCandidates(fromLower, firstHop, candidates, observerPK, obs.SNR, parseTimestamp(obs.Timestamp), lowerCache)
}
}
// Edge 2: observer ↔ path[last] — ALL packet types
lastHop := cachedToLower(lowerCache, path[len(path)-1])
if observerPK != lastHop { // self-edge guard
candidates := pm.m[lastHop]
g.upsertEdgeWithCandidates(observerPK, lastHop, candidates, observerPK, obs.SNR, parseTimestamp(obs.Timestamp), lowerCache)
}
}
}
// Phase 1.5: Resolve ambiguous edges using full graph context.
resolveAmbiguousEdges(pm, g)
// Phase 2: Disambiguation via Jaccard similarity.
g.disambiguate()
g.mu.Lock()
g.builtAt = time.Now()
g.mu.Unlock()
return g
}
// extractFromNode pulls the originator pubkey from a StoreTx's DecodedJSON.
// ADVERTs use "pubKey", other packets may use "from_node" or "from".
// Uses the cached ParsedDecoded() accessor to avoid repeated json.Unmarshal.
func extractFromNode(tx *StoreTx) string {
decoded := tx.ParsedDecoded()
if decoded == nil {
return ""
}
// ADVERTs store the originator pubkey as "pubKey"; other packets may use
// "from_node" or "from". Check all three so we never miss the originator.
for _, field := range []string{"pubKey", "from_node", "from"} {
if v, ok := decoded[field]; ok {
if s, ok := v.(string); ok && s != "" {
return s
}
}
}
return ""
}
// jsonUnmarshalFast is a thin wrapper; could be optimized later.
func jsonUnmarshalFast(data string, v interface{}) error {
return json.Unmarshal([]byte(data), v)
}
// upsertEdge adds/updates an edge between two fully-known pubkeys.
func (g *NeighborGraph) upsertEdge(pubkeyA, pubkeyB, prefix, observer string, snr *float64, ts time.Time) {
key := makeEdgeKey(pubkeyA, pubkeyB)
g.mu.Lock()
defer g.mu.Unlock()
e, exists := g.edges[key]
if !exists {
e = &NeighborEdge{
NodeA: key.A,
NodeB: key.B,
Prefix: prefix,
Observers: make(map[string]bool),
FirstSeen: ts,
LastSeen: ts,
}
g.edges[key] = e
g.byNode[key.A] = append(g.byNode[key.A], e)
g.byNode[key.B] = append(g.byNode[key.B], e)
}
e.Count++
if ts.After(e.LastSeen) {
e.LastSeen = ts
}
if ts.Before(e.FirstSeen) {
e.FirstSeen = ts
}
if snr != nil {
e.SNRSum += *snr
e.SNRCount++
}
if observer != "" {
e.Observers[observer] = true
}
}
// upsertEdgeWithCandidates handles prefix-based edges that may be ambiguous.
func (g *NeighborGraph) upsertEdgeWithCandidates(knownPK, prefix string, candidates []nodeInfo, observer string, snr *float64, ts time.Time, lc map[string]string) {
if len(candidates) == 1 {
resolved := cachedToLower(lc, candidates[0].PublicKey)
if resolved == knownPK {
return // self-edge guard
}
g.upsertEdge(knownPK, resolved, prefix, observer, snr, ts)
return
}
// Filter out self from candidates
filtered := make([]string, 0, len(candidates))
for _, c := range candidates {
pk := cachedToLower(lc, c.PublicKey)
if pk != knownPK {
filtered = append(filtered, pk)
}
}
if len(filtered) == 1 {
g.upsertEdge(knownPK, filtered[0], prefix, observer, snr, ts)
return
}
// Ambiguous or orphan: use prefix-based key
pseudoB := "prefix:" + prefix
key := makeEdgeKey(knownPK, pseudoB)
g.mu.Lock()
defer g.mu.Unlock()
e, exists := g.edges[key]
if !exists {
e = &NeighborEdge{
NodeA: key.A,
NodeB: "",
Prefix: prefix,
Observers: make(map[string]bool),
Ambiguous: true,
Candidates: filtered,
FirstSeen: ts,
LastSeen: ts,
}
g.edges[key] = e
g.byNode[knownPK] = append(g.byNode[knownPK], e)
}
e.Count++
if ts.After(e.LastSeen) {
e.LastSeen = ts
}
if ts.Before(e.FirstSeen) {
e.FirstSeen = ts
}
if snr != nil {
e.SNRSum += *snr
e.SNRCount++
}
if observer != "" {
e.Observers[observer] = true
}
}
// ─── Phase 1.5: Context-based resolution of ambiguous edges ────────────────────
// resolveAmbiguousEdges attempts to resolve ambiguous prefix edges using the
// fully-built graph context. Called after Phase 1 (edge collection) completes
// so that affinity and geo proximity tiers have full neighbor data.
func resolveAmbiguousEdges(pm *prefixMap, graph *NeighborGraph) {
// Step 1: Collect ambiguous edges under read lock.
graph.mu.RLock()
type ambiguousEntry struct {
key edgeKey
edge *NeighborEdge
knownNode string
prefix string
}
var ambiguous []ambiguousEntry
for key, e := range graph.edges {
if !e.Ambiguous {
continue
}
knownNode := e.NodeA
if strings.HasPrefix(e.NodeA, "prefix:") {
knownNode = e.NodeB
}
if knownNode == "" {
continue
}
ambiguous = append(ambiguous, ambiguousEntry{key, e, knownNode, e.Prefix})
}
graph.mu.RUnlock()
// Step 2: Resolve each (no lock needed — resolveWithContext takes its own RLock).
type resolution struct {
ambiguousEntry
resolvedPK string
}
var resolutions []resolution
for _, ae := range ambiguous {
resolved, confidence, _ := pm.resolveWithContext(ae.prefix, []string{ae.knownNode}, graph)
if resolved == nil || confidence == "no_match" || confidence == "first_match" || confidence == "gps_preference" {
continue
}
rpk := strings.ToLower(resolved.PublicKey)
if rpk == ae.knownNode {
continue // self-edge guard
}
resolutions = append(resolutions, resolution{ae, rpk})
}
// Step 3: Apply resolutions under write lock.
if len(resolutions) == 0 {
return
}
graph.mu.Lock()
for _, r := range resolutions {
// Verify edge still exists and is still ambiguous (could have been
// resolved by a prior iteration if two ambiguous edges resolve to same target).
e, ok := graph.edges[r.key]
if !ok || !e.Ambiguous {
continue
}
graph.resolveEdge(r.key, e, r.knownNode, r.resolvedPK)
}
graph.mu.Unlock()
}
// ─── Disambiguation ────────────────────────────────────────────────────────────
// disambiguate resolves ambiguous edges using Jaccard similarity of neighbor sets.
// Only fully-resolved edges are used as evidence (transitivity poisoning guard).
func (g *NeighborGraph) disambiguate() {
g.mu.Lock()
defer g.mu.Unlock()
// Build resolved neighbor sets: for each node, collect the set of nodes
// it has fully-resolved (non-ambiguous) edges with.
resolvedNeighbors := make(map[string]map[string]bool)
for _, e := range g.edges {
if e.Ambiguous || e.NodeB == "" {
continue
}
if resolvedNeighbors[e.NodeA] == nil {
resolvedNeighbors[e.NodeA] = make(map[string]bool)
}
if resolvedNeighbors[e.NodeB] == nil {
resolvedNeighbors[e.NodeB] = make(map[string]bool)
}
resolvedNeighbors[e.NodeA][e.NodeB] = true
resolvedNeighbors[e.NodeB][e.NodeA] = true
}
// Try to resolve each ambiguous edge.
for key, e := range g.edges {
if !e.Ambiguous || len(e.Candidates) < 2 {
continue
}
if e.Count < affinityMinObservations {
continue
}
// Determine the known node (the one that's a real pubkey, not the prefix side).
knownNode := e.NodeA
if strings.HasPrefix(e.NodeA, "prefix:") {
knownNode = e.NodeB
}
// If knownNode is empty (shouldn't happen for ambiguous edges with candidates), skip.
if knownNode == "" {
continue
}
knownNeighbors := resolvedNeighbors[knownNode]
type scored struct {
pubkey string
jaccard float64
}
var scores []scored
for _, cand := range e.Candidates {
candNeighbors := resolvedNeighbors[cand]
j := jaccardSimilarity(knownNeighbors, candNeighbors)
scores = append(scores, scored{cand, j})
}
if len(scores) < 2 {
continue
}
// Find best and second-best.
best, secondBest := scores[0], scores[1]
if secondBest.jaccard > best.jaccard {
best, secondBest = secondBest, best
}
for i := 2; i < len(scores); i++ {
if scores[i].jaccard > best.jaccard {
secondBest = best
best = scores[i]
} else if scores[i].jaccard > secondBest.jaccard {
secondBest = scores[i]
}
}
// Auto-resolve only if best >= 3× second-best AND enough observations.
if secondBest.jaccard == 0 {
// If second-best is 0 and best > 0, ratio is infinite → resolve.
if best.jaccard > 0 {
if g.logFn != nil {
g.logFn(e.Prefix, fmt.Sprintf("%s score=%d Jaccard=%.2f vs %s score=%d Jaccard=%.2f → neighbor_affinity (ratio ∞)",
best.pubkey[:minLen(best.pubkey, 8)], e.Count, best.jaccard,
secondBest.pubkey[:minLen(secondBest.pubkey, 8)], e.Count, secondBest.jaccard))
}
g.resolveEdge(key, e, knownNode, best.pubkey)
}
} else if best.jaccard/secondBest.jaccard >= affinityConfidenceRatio {
ratio := best.jaccard / secondBest.jaccard
if g.logFn != nil {
g.logFn(e.Prefix, fmt.Sprintf("%s score=%d Jaccard=%.2f vs %s score=%d Jaccard=%.2f → neighbor_affinity (ratio %.1f×)",
best.pubkey[:minLen(best.pubkey, 8)], e.Count, best.jaccard,
secondBest.pubkey[:minLen(secondBest.pubkey, 8)], e.Count, secondBest.jaccard, ratio))
}
g.resolveEdge(key, e, knownNode, best.pubkey)
} else {
// Ambiguous
if g.logFn != nil {
ratio := 0.0
if secondBest.jaccard > 0 {
ratio = best.jaccard / secondBest.jaccard
}
g.logFn(e.Prefix, fmt.Sprintf("scores too close (Jaccard %.2f vs %.2f, ratio %.1f×) → ambiguous, returning %d candidates",
best.jaccard, secondBest.jaccard, ratio, len(e.Candidates)))
}
}
}
}
// resolveEdge converts an ambiguous edge to a resolved one.
// Must be called with g.mu held.
func (g *NeighborGraph) resolveEdge(oldKey edgeKey, e *NeighborEdge, knownNode, resolvedPK string) {
// Remove old edge.
delete(g.edges, oldKey)
g.removeFromByNode(oldKey.A, e)
g.removeFromByNode(oldKey.B, e)
// Update edge.
newKey := makeEdgeKey(knownNode, resolvedPK)
e.NodeA = newKey.A
e.NodeB = newKey.B
e.Ambiguous = false
e.Resolved = true
// Merge with existing edge if any.
if existing, ok := g.edges[newKey]; ok {
existing.Count += e.Count
if e.LastSeen.After(existing.LastSeen) {
existing.LastSeen = e.LastSeen
}
if e.FirstSeen.Before(existing.FirstSeen) {
existing.FirstSeen = e.FirstSeen
}
existing.SNRSum += e.SNRSum
existing.SNRCount += e.SNRCount
for obs := range e.Observers {
existing.Observers[obs] = true
}
return
}
g.edges[newKey] = e
g.byNode[newKey.A] = append(g.byNode[newKey.A], e)
g.byNode[newKey.B] = append(g.byNode[newKey.B], e)
}
// removeFromByNode removes an edge from the byNode index for the given key.
func (g *NeighborGraph) removeFromByNode(nodeKey string, edge *NeighborEdge) {
edges := g.byNode[nodeKey]
for i, e := range edges {
if e == edge {
g.byNode[nodeKey] = append(edges[:i], edges[i+1:]...)
return
}
}
}
// jaccardSimilarity computes |A ∩ B| / |A B|.
func jaccardSimilarity(a, b map[string]bool) float64 {
if len(a) == 0 && len(b) == 0 {
return 0
}
intersection := 0
for k := range a {
if b[k] {
intersection++
}
}
union := len(a) + len(b) - intersection
if union == 0 {
return 0
}
return float64(intersection) / float64(union)
}
// parseTimestamp parses a timestamp string into time.Time.
func parseTimestamp(s string) time.Time {
// Try common formats.
for _, fmt := range []string{
time.RFC3339,
"2006-01-02T15:04:05Z",
"2006-01-02 15:04:05",
"2006-01-02T15:04:05.000Z",
} {
if t, err := time.Parse(fmt, s); err == nil {
return t
}
}
return time.Time{}
}
// minLen returns the smaller of n and len(s).
func minLen(s string, n int) int {
if len(s) < n {
return len(s)
}
return n
}
// PruneOlderThan removes all edges with LastSeen before cutoff.
// Returns the number of edges removed.
func (g *NeighborGraph) PruneOlderThan(cutoff time.Time) int {
g.mu.Lock()
defer g.mu.Unlock()
pruned := 0
for key, edge := range g.edges {
if edge.LastSeen.Before(cutoff) {
// Remove from byNode index
g.removeFromByNode(edge.NodeA, edge)
if edge.NodeB != "" {
g.removeFromByNode(edge.NodeB, edge)
}
delete(g.edges, key)
pruned++
}
}
return pruned
}
+836
View File
@@ -0,0 +1,836 @@
package main
import (
"encoding/json"
"math"
"testing"
"time"
)
// ─── Helpers ───────────────────────────────────────────────────────────────────
// ngTestStore creates a minimal PacketStore with injected nodes and packets.
func ngTestStore(nodes []nodeInfo, packets []*StoreTx) *PacketStore {
if nodes == nil {
nodes = []nodeInfo{}
}
if packets == nil {
packets = []*StoreTx{}
}
ps := &PacketStore{
packets: packets,
byHash: make(map[string]*StoreTx),
byTxID: make(map[int]*StoreTx),
byObsID: make(map[int]*StoreObs),
byObserver: make(map[string][]*StoreObs),
byNode: make(map[string][]*StoreTx),
nodeHashes: make(map[string]map[string]bool),
byPayloadType: make(map[int][]*StoreTx),
rfCache: make(map[string]*cachedResult),
topoCache: make(map[string]*cachedResult),
hashCache: make(map[string]*cachedResult),
collisionCache: make(map[string]*cachedResult),
chanCache: make(map[string]*cachedResult),
distCache: make(map[string]*cachedResult),
subpathCache: make(map[string]*cachedResult),
spIndex: make(map[string]int),
}
ps.nodeCache = nodes
ps.nodePM = buildPrefixMap(nodes)
ps.nodeCacheTime = time.Now().Add(1 * time.Hour)
return ps
}
func ngIntPtr(v int) *int { return &v }
func ngFloatPtr(v float64) *float64 { return &v }
func ngMakeTx(id int, payloadType int, decodedJSON string, obs []*StoreObs) *StoreTx {
tx := &StoreTx{
ID: id,
PayloadType: ngIntPtr(payloadType),
DecodedJSON: decodedJSON,
Observations: obs,
}
return tx
}
func ngMakeObs(observerID, pathJSON, timestamp string, snr *float64) *StoreObs {
return &StoreObs{
ObserverID: observerID,
PathJSON: pathJSON,
Timestamp: timestamp,
SNR: snr,
}
}
func ngFromNodeJSON(pubkey string) string {
b, _ := json.Marshal(map[string]string{"from_node": pubkey})
return string(b)
}
var now = time.Now()
var nowStr = now.UTC().Format(time.RFC3339)
var weekAgoStr = now.Add(-7 * 24 * time.Hour).UTC().Format(time.RFC3339)
var monthAgoStr = now.Add(-30 * 24 * time.Hour).UTC().Format(time.RFC3339)
// ─── Tests ─────────────────────────────────────────────────────────────────────
func TestBuildNeighborGraph_EmptyStore(t *testing.T) {
store := ngTestStore(nil, nil)
g := BuildFromStore(store)
if len(g.edges) != 0 {
t.Errorf("expected 0 edges, got %d", len(g.edges))
}
}
func TestBuildNeighborGraph_AdvertSingleHopPath(t *testing.T) {
// ADVERT from X, path=["R1_prefix"] → edges: X↔R1 and Observer↔R1
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa"]`, nowStr, ngFloatPtr(-10)),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
// Should have 2 edges: X↔R1 and Observer↔R1
// But since path has 1 element, path[0]==path[last], so for ADVERTs
// both edge types point to the same hop. X↔R1 and Obs↔R1 = 2 edges.
edges := g.AllEdges()
if len(edges) != 2 {
t.Fatalf("expected 2 edges, got %d", len(edges))
}
// Check X↔R1 exists
found := false
for _, e := range edges {
if (e.NodeA == "aaaa1111" && e.NodeB == "r1aabbcc") ||
(e.NodeA == "r1aabbcc" && e.NodeB == "aaaa1111") {
found = true
}
}
if !found {
t.Error("missing originator↔path[0] edge (X↔R1)")
}
// Check Observer↔R1 exists
found = false
for _, e := range edges {
if (e.NodeA == "obs00001" && e.NodeB == "r1aabbcc") ||
(e.NodeA == "r1aabbcc" && e.NodeB == "obs00001") {
found = true
}
}
if !found {
t.Error("missing observer↔path[last] edge (Observer↔R1)")
}
}
func TestBuildNeighborGraph_AdvertMultiHopPath(t *testing.T) {
// ADVERT from X, path=["R1","R2"] → X↔R1 and Observer↔R2
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "r2ddeeff", Name: "R2"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa","r2dd"]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
edges := g.AllEdges()
if len(edges) != 2 {
t.Fatalf("expected 2 edges, got %d", len(edges))
}
// X↔R1
hasXR1 := false
hasObsR2 := false
for _, e := range edges {
if (e.NodeA == "aaaa1111" && e.NodeB == "r1aabbcc") || (e.NodeA == "r1aabbcc" && e.NodeB == "aaaa1111") {
hasXR1 = true
}
if (e.NodeA == "obs00001" && e.NodeB == "r2ddeeff") || (e.NodeA == "r2ddeeff" && e.NodeB == "obs00001") {
hasObsR2 = true
}
}
if !hasXR1 {
t.Error("missing X↔R1 edge")
}
if !hasObsR2 {
t.Error("missing Observer↔R2 edge")
}
}
func TestBuildNeighborGraph_AdvertZeroHop(t *testing.T) {
// ADVERT from X, path=[] → X↔Observer direct edge
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `[]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
edges := g.AllEdges()
if len(edges) != 1 {
t.Fatalf("expected 1 edge, got %d", len(edges))
}
e := edges[0]
if !((e.NodeA == "aaaa1111" && e.NodeB == "obs00001") || (e.NodeA == "obs00001" && e.NodeB == "aaaa1111")) {
t.Errorf("expected X↔Observer edge, got %s↔%s", e.NodeA, e.NodeB)
}
if e.Ambiguous {
t.Error("zero-hop edge should not be ambiguous")
}
}
func TestBuildNeighborGraph_NonAdvertEmptyPath(t *testing.T) {
// Non-ADVERT, path=[] → no edges
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 2, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `[]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
if len(g.edges) != 0 {
t.Errorf("expected 0 edges for non-ADVERT empty path, got %d", len(g.edges))
}
}
func TestBuildNeighborGraph_NonAdvertOnlyObserverEdge(t *testing.T) {
// Non-ADVERT with path=["R1","R2"] → only Observer↔R2, NO originator edge
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "r2ddeeff", Name: "R2"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 2, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa","r2dd"]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
edges := g.AllEdges()
if len(edges) != 1 {
t.Fatalf("expected 1 edge, got %d", len(edges))
}
e := edges[0]
if !((e.NodeA == "obs00001" && e.NodeB == "r2ddeeff") || (e.NodeA == "r2ddeeff" && e.NodeB == "obs00001")) {
t.Errorf("expected Observer↔R2 edge, got %s↔%s", e.NodeA, e.NodeB)
}
}
func TestBuildNeighborGraph_NonAdvertSingleHop(t *testing.T) {
// Non-ADVERT with path=["R1"] → Observer↔R1 only
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 2, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa"]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
edges := g.AllEdges()
if len(edges) != 1 {
t.Fatalf("expected 1 edge, got %d", len(edges))
}
e := edges[0]
if !((e.NodeA == "obs00001" && e.NodeB == "r1aabbcc") || (e.NodeA == "r1aabbcc" && e.NodeB == "obs00001")) {
t.Errorf("expected Observer↔R1, got %s↔%s", e.NodeA, e.NodeB)
}
}
func TestBuildNeighborGraph_HashCollision(t *testing.T) {
// Two nodes share prefix "a3" → ambiguous edge
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "a3bb1111", Name: "CandidateA"},
{PublicKey: "a3bb2222", Name: "CandidateB"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["a3bb"]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
// Should have ambiguous edges
var ambigCount int
for _, e := range g.AllEdges() {
if e.Ambiguous {
ambigCount++
if len(e.Candidates) < 2 {
t.Errorf("expected >=2 candidates, got %d", len(e.Candidates))
}
}
}
if ambigCount == 0 {
t.Error("expected at least one ambiguous edge for hash collision")
}
}
func TestBuildNeighborGraph_JaccardScoring(t *testing.T) {
// Test Jaccard similarity computation directly
a := map[string]bool{"x": true, "y": true, "z": true}
b := map[string]bool{"y": true, "z": true, "w": true}
j := jaccardSimilarity(a, b)
// intersection = {y, z} = 2, union = {x, y, z, w} = 4 → 0.5
if math.Abs(j-0.5) > 0.001 {
t.Errorf("expected Jaccard 0.5, got %f", j)
}
// Empty sets
j = jaccardSimilarity(nil, nil)
if j != 0 {
t.Errorf("expected 0 for empty sets, got %f", j)
}
}
func TestBuildNeighborGraph_ConfidenceAutoResolve(t *testing.T) {
// Setup: NodeX has known neighbors N1, N2, N3 (resolved edges).
// CandidateA also has known neighbors N1, N2, N3 (high Jaccard with X).
// CandidateB has no known neighbors (Jaccard = 0).
// An ambiguous edge X↔prefix "a3" with candidates [A, B] should auto-resolve to A.
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "n1111111", Name: "N1"},
{PublicKey: "n2222222", Name: "N2"},
{PublicKey: "n3333333", Name: "N3"},
{PublicKey: "a3001111", Name: "CandidateA"},
{PublicKey: "a3002222", Name: "CandidateB"},
{PublicKey: "obs00001", Name: "Observer"},
}
// Create resolved edges: X↔N1, X↔N2, X↔N3, A↔N1, A↔N2, A↔N3
// Then an ambiguous edge X↔"a300" prefix with 3+ observations.
var txs []*StoreTx
txID := 1
// X sends ADVERTs through N1, N2, N3
for _, nhop := range []string{"n111", "n222", "n333"} {
txs = append(txs, ngMakeTx(txID, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["`+nhop+`"]`, nowStr, nil),
}))
txID++
}
// CandidateA sends ADVERTs through N1, N2, N3
for _, nhop := range []string{"n111", "n222", "n333"} {
txs = append(txs, ngMakeTx(txID, 4, ngFromNodeJSON("a3001111"), []*StoreObs{
ngMakeObs("obs00001", `["`+nhop+`"]`, nowStr, nil),
}))
txID++
}
// Ambiguous edge: X sends ADVERTs with path[0]="a300" (matches both candidates)
// Need 3+ observations for confidence threshold.
for i := 0; i < 3; i++ {
txs = append(txs, ngMakeTx(txID, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["a300"]`, nowStr, nil),
}))
txID++
}
store := ngTestStore(nodes, txs)
g := BuildFromStore(store)
// The ambiguous edge X↔a300 should have been resolved to CandidateA
neighbors := g.Neighbors("aaaa1111")
foundA := false
for _, e := range neighbors {
other := e.NodeB
if e.NodeA != "aaaa1111" {
other = e.NodeA
}
if other == "a3001111" {
foundA = true
if e.Ambiguous {
t.Error("edge should have been resolved (not ambiguous)")
}
}
}
if !foundA {
t.Error("expected edge X↔CandidateA to be auto-resolved")
}
}
func TestBuildNeighborGraph_EqualScoresAmbiguous(t *testing.T) {
// Two candidates with identical neighbor sets → should NOT auto-resolve.
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "n1111111", Name: "N1"},
{PublicKey: "a3001111", Name: "CandidateA"},
{PublicKey: "a3002222", Name: "CandidateB"},
{PublicKey: "obs00001", Name: "Observer"},
}
var txs []*StoreTx
txID := 1
// X↔N1
txs = append(txs, ngMakeTx(txID, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["n111"]`, nowStr, nil),
}))
txID++
// Both candidates have same neighbor (N1)
txs = append(txs, ngMakeTx(txID, 4, ngFromNodeJSON("a3001111"), []*StoreObs{
ngMakeObs("obs00001", `["n111"]`, nowStr, nil),
}))
txID++
txs = append(txs, ngMakeTx(txID, 4, ngFromNodeJSON("a3002222"), []*StoreObs{
ngMakeObs("obs00001", `["n111"]`, nowStr, nil),
}))
txID++
// Ambiguous edge with 3+ observations
for i := 0; i < 3; i++ {
txs = append(txs, ngMakeTx(txID, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["a300"]`, nowStr, nil),
}))
txID++
}
store := ngTestStore(nodes, txs)
g := BuildFromStore(store)
// Should remain ambiguous
var ambigFound bool
for _, e := range g.AllEdges() {
if e.Ambiguous && e.Prefix == "a300" {
ambigFound = true
}
}
if !ambigFound {
t.Error("expected ambiguous edge to remain unresolved with equal scores")
}
}
func TestBuildNeighborGraph_ObserverSelfEdgeGuard(t *testing.T) {
// Observer's own prefix in path → should NOT create self-edge.
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["obs0"]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
// Check no self-edge for observer
for _, e := range g.AllEdges() {
if e.NodeA == e.NodeB && e.NodeA == "obs00001" {
t.Error("self-edge created for observer")
}
}
}
func TestBuildNeighborGraph_OrphanPrefix(t *testing.T) {
// Path contains prefix matching zero nodes → edge recorded as unresolved.
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["ff99"]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
// Should have ambiguous edges with empty candidates.
var orphanFound bool
for _, e := range g.AllEdges() {
if e.Ambiguous && len(e.Candidates) == 0 {
orphanFound = true
if e.Prefix != "ff99" {
t.Errorf("expected prefix ff99, got %s", e.Prefix)
}
}
}
if !orphanFound {
t.Error("expected orphan prefix edge with empty candidates")
}
}
func TestAffinityScore_Fresh(t *testing.T) {
e := &NeighborEdge{Count: 100, LastSeen: time.Now()}
s := e.Score(time.Now())
if s < 0.99 || s > 1.0 {
t.Errorf("expected score ≈ 1.0, got %f", s)
}
}
func TestAffinityScore_Decayed(t *testing.T) {
e := &NeighborEdge{Count: 100, LastSeen: time.Now().Add(-7 * 24 * time.Hour)}
s := e.Score(time.Now())
// 7 days → half-life → ~0.5
if math.Abs(s-0.5) > 0.05 {
t.Errorf("expected score ≈ 0.5, got %f", s)
}
}
func TestAffinityScore_LowCount(t *testing.T) {
e := &NeighborEdge{Count: 5, LastSeen: time.Now()}
s := e.Score(time.Now())
// 5/100 = 0.05
if math.Abs(s-0.05) > 0.01 {
t.Errorf("expected score ≈ 0.05, got %f", s)
}
}
func TestAffinityScore_StaleAndLow(t *testing.T) {
e := &NeighborEdge{Count: 5, LastSeen: time.Now().Add(-30 * 24 * time.Hour)}
s := e.Score(time.Now())
// Very small
if s > 0.01 {
t.Errorf("expected score ≈ 0, got %f", s)
}
}
func TestBuildNeighborGraph_CountAccumulation(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Observer"},
}
var txs []*StoreTx
for i := 0; i < 5; i++ {
txs = append(txs, ngMakeTx(i+1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa"]`, nowStr, nil),
}))
}
store := ngTestStore(nodes, txs)
g := BuildFromStore(store)
// Check count on X↔R1 edge
for _, e := range g.AllEdges() {
if (e.NodeA == "aaaa1111" && e.NodeB == "r1aabbcc") || (e.NodeA == "r1aabbcc" && e.NodeB == "aaaa1111") {
if e.Count != 5 {
t.Errorf("expected count 5, got %d", e.Count)
}
return
}
}
t.Error("X↔R1 edge not found")
}
func TestBuildNeighborGraph_MultipleObservers(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Obs1"},
{PublicKey: "obs00002", Name: "Obs2"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa"]`, nowStr, nil),
ngMakeObs("obs00002", `["r1aa"]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
for _, e := range g.AllEdges() {
if (e.NodeA == "aaaa1111" && e.NodeB == "r1aabbcc") || (e.NodeA == "r1aabbcc" && e.NodeB == "aaaa1111") {
if len(e.Observers) != 2 {
t.Errorf("expected 2 observers, got %d", len(e.Observers))
}
if !e.Observers["obs00001"] || !e.Observers["obs00002"] {
t.Error("missing expected observer")
}
return
}
}
t.Error("X↔R1 edge not found")
}
func TestBuildNeighborGraph_TimeDecayOldObservations(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa"]`, monthAgoStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
for _, e := range g.AllEdges() {
if (e.NodeA == "aaaa1111" && e.NodeB == "r1aabbcc") || (e.NodeA == "r1aabbcc" && e.NodeB == "aaaa1111") {
score := e.Score(time.Now())
if score > 0.05 {
t.Errorf("expected decayed score < 0.05, got %f", score)
}
return
}
}
t.Error("X↔R1 edge not found")
}
func TestBuildNeighborGraph_ADVERTOnlyConstraint(t *testing.T) {
// Non-ADVERT: should NOT create originator↔path[0] edge, only observer↔path[last].
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "r2ddeeff", Name: "R2"},
{PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 2, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa","r2dd"]`, nowStr, nil),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
for _, e := range g.AllEdges() {
a, b := e.NodeA, e.NodeB
if (a == "aaaa1111" && b == "r1aabbcc") || (a == "r1aabbcc" && b == "aaaa1111") {
t.Error("non-ADVERT should NOT produce originator↔path[0] edge")
}
}
// Should have Observer↔R2
found := false
for _, e := range g.AllEdges() {
if (e.NodeA == "obs00001" && e.NodeB == "r2ddeeff") || (e.NodeA == "r2ddeeff" && e.NodeB == "obs00001") {
found = true
}
}
if !found {
t.Error("missing Observer↔R2 edge from non-ADVERT")
}
}
// ngPubKeyJSON creates decoded JSON using the real ADVERT format ("pubKey" field).
func ngPubKeyJSON(pubkey string) string {
b, _ := json.Marshal(map[string]string{"pubKey": pubkey})
return string(b)
}
func TestBuildNeighborGraph_AdvertPubKeyField(t *testing.T) {
// Real ADVERTs use "pubKey", not "from_node". Verify the builder handles it.
nodes := []nodeInfo{
{PublicKey: "99bf37abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234", Name: "Originator"},
{PublicKey: "r1aabbccdd001122334455667788990011223344556677889900112233445566", Name: "R1"},
{PublicKey: "obs0000100112233445566778899001122334455667788990011223344556677", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngPubKeyJSON("99bf37abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234"), []*StoreObs{
ngMakeObs("obs0000100112233445566778899001122334455667788990011223344556677", `["r1"]`, nowStr, ngFloatPtr(-8.5)),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
edges := g.AllEdges()
if len(edges) < 1 {
t.Fatalf("expected >=1 edges from ADVERT with pubKey field, got %d", len(edges))
}
// Check originator↔R1 edge exists
found := false
for _, e := range edges {
a := e.NodeA
b := e.NodeB
orig := "99bf37abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234"
r1 := "r1aabbccdd001122334455667788990011223344556677889900112233445566"
if (a == orig && b == r1) || (a == r1 && b == orig) {
found = true
}
}
if !found {
t.Error("missing originator↔R1 edge when using pubKey field (real ADVERT format)")
}
}
func TestBuildNeighborGraph_OneByteHashPrefixes(t *testing.T) {
// Real-world scenario: 1-byte hash prefixes with multiple candidates.
// Should create edges (possibly ambiguous) rather than empty graph.
nodes := []nodeInfo{
{PublicKey: "c0dedad400000000000000000000000000000000000000000000000000000001", Name: "NodeC0-1"},
{PublicKey: "c0dedad900000000000000000000000000000000000000000000000000000002", Name: "NodeC0-2"},
{PublicKey: "a3bbccdd00000000000000000000000000000000000000000000000000000003", Name: "Originator"},
{PublicKey: "obs1234500000000000000000000000000000000000000000000000000000004", Name: "Observer"},
}
// ADVERT from Originator with 1-byte path hop "c0"
tx := ngMakeTx(1, 4, ngPubKeyJSON("a3bbccdd00000000000000000000000000000000000000000000000000000003"), []*StoreObs{
ngMakeObs("obs1234500000000000000000000000000000000000000000000000000000004", `["c0"]`, nowStr, ngFloatPtr(-12)),
})
store := ngTestStore(nodes, []*StoreTx{tx})
g := BuildFromStore(store)
edges := g.AllEdges()
if len(edges) == 0 {
t.Fatal("expected non-empty edges for 1-byte hash prefix network, got 0")
}
// The originator↔c0 edge should be ambiguous (2 candidates match "c0")
var hasAmbig bool
for _, e := range edges {
if e.Ambiguous && e.Prefix == "c0" {
hasAmbig = true
if len(e.Candidates) != 2 {
t.Errorf("expected 2 candidates for prefix c0, got %d", len(e.Candidates))
}
}
}
if !hasAmbig {
// Could be resolved if one candidate was filtered — check we got some edge
t.Log("no ambiguous edge found, but edges exist — acceptable if resolved")
}
}
func TestNeighborGraph_CacheTTL(t *testing.T) {
g := NewNeighborGraph()
if !g.IsStale() {
t.Error("new graph should be stale")
}
g.mu.Lock()
g.builtAt = time.Now()
g.mu.Unlock()
if g.IsStale() {
t.Error("just-built graph should not be stale")
}
g.mu.Lock()
g.builtAt = time.Now().Add(-2 * neighborGraphTTL)
g.mu.Unlock()
if !g.IsStale() {
t.Error("old graph should be stale")
}
}
func TestNeighborGraph_TTLIsReasonable(t *testing.T) {
// TTL must be long enough to avoid rebuild storms on busy meshes,
// but short enough to reflect topology changes within minutes.
if neighborGraphTTL < 1*time.Minute {
t.Errorf("neighborGraphTTL too short (%v), will cause rebuild storms", neighborGraphTTL)
}
if neighborGraphTTL > 10*time.Minute {
t.Errorf("neighborGraphTTL too long (%v), topology changes will be stale", neighborGraphTTL)
}
}
func TestCachedToLower(t *testing.T) {
cache := make(map[string]string)
// Basic lowercasing
if got := cachedToLower(cache, "AABB"); got != "aabb" {
t.Errorf("expected 'aabb', got %q", got)
}
// Verify it was cached
if _, ok := cache["AABB"]; !ok {
t.Error("expected 'AABB' to be in cache")
}
// Same input returns cached result
if got := cachedToLower(cache, "AABB"); got != "aabb" {
t.Errorf("expected cached 'aabb', got %q", got)
}
// Already lowercase stays the same
if got := cachedToLower(cache, "aabb"); got != "aabb" {
t.Errorf("expected 'aabb', got %q", got)
}
// Empty string
if got := cachedToLower(cache, ""); got != "" {
t.Errorf("expected empty, got %q", got)
}
}
func TestParsedDecoded_Caching(t *testing.T) {
tx := &StoreTx{DecodedJSON: `{"pubKey":"abc123","name":"test"}`}
// First call parses
d1 := tx.ParsedDecoded()
if d1 == nil {
t.Fatal("expected non-nil parsed result")
}
if d1["pubKey"] != "abc123" {
t.Errorf("expected pubKey=abc123, got %v", d1["pubKey"])
}
// Second call must return the exact same map (pointer equality proves caching)
d2 := tx.ParsedDecoded()
if &d1 == nil || &d2 == nil {
t.Fatal("unexpected nil")
}
// Mutate d1 and verify d2 sees the mutation — proves same underlying map
d1["_sentinel"] = true
if d2["_sentinel"] != true {
t.Error("expected same map instance from second call (caching broken)")
}
delete(d1, "_sentinel") // clean up
}
func TestParsedDecoded_EmptyJSON(t *testing.T) {
tx := &StoreTx{DecodedJSON: ""}
d := tx.ParsedDecoded()
if d != nil {
t.Errorf("expected nil for empty DecodedJSON, got %v", d)
}
}
func TestParsedDecoded_InvalidJSON(t *testing.T) {
tx := &StoreTx{DecodedJSON: "not json"}
d := tx.ParsedDecoded()
if d != nil {
t.Errorf("expected nil for invalid JSON, got %v", d)
}
}
func TestExtractFromNode_UsesCachedParse(t *testing.T) {
tx := &StoreTx{DecodedJSON: `{"pubKey":"aabb1122"}`}
// First call to extractFromNode should use ParsedDecoded
from := extractFromNode(tx)
if from != "aabb1122" {
t.Errorf("expected aabb1122, got %q", from)
}
// ParsedDecoded should now be cached
d := tx.ParsedDecoded()
if d == nil || d["pubKey"] != "aabb1122" {
t.Error("expected ParsedDecoded to return cached result")
}
}
func BenchmarkBuildFromStore(b *testing.B) {
// Simulate a dataset with many packets and repeated pubkeys
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeA"},
{PublicKey: "bbbb2222", Name: "NodeB"},
{PublicKey: "cccc3333", Name: "NodeC"},
{PublicKey: "dddd4444", Name: "NodeD"},
}
const numPackets = 1000
packets := make([]*StoreTx, 0, numPackets)
for i := 0; i < numPackets; i++ {
pt := 4 // ADVERT
packets = append(packets, &StoreTx{
ID: i,
PayloadType: &pt,
DecodedJSON: `{"pubKey":"aaaa1111"}`,
Observations: []*StoreObs{
{ObserverID: "bbbb2222", PathJSON: `["cccc"]`, Timestamp: nowStr, SNR: ngFloatPtr(-5.0)},
},
})
}
store := ngTestStore(nodes, packets)
b.ResetTimer()
for i := 0; i < b.N; i++ {
BuildFromStore(store)
}
}
+674
View File
@@ -0,0 +1,674 @@
package main
import (
"database/sql"
"encoding/json"
"fmt"
"log"
"strings"
"time"
)
// persistSem limits concurrent async persistence goroutines to 1.
// Without this, each ingest cycle spawns a goroutine that opens a new
// SQLite RW connection; under sustained load goroutines pile up with
// no backpressure, causing contention and busy-timeout cascades.
var persistSem = make(chan struct{}, 1)
// ─── neighbor_edges table ──────────────────────────────────────────────────────
// ensureNeighborEdgesTable creates the neighbor_edges table if it doesn't exist.
// Uses a separate read-write connection since the main DB is read-only.
func ensureNeighborEdgesTable(dbPath string) error {
rw, err := openRW(dbPath)
if err != nil {
return fmt.Errorf("open rw for neighbor_edges: %w", err)
}
defer rw.Close()
_, err = rw.Exec(`CREATE TABLE IF NOT EXISTS neighbor_edges (
node_a TEXT NOT NULL,
node_b TEXT NOT NULL,
count INTEGER DEFAULT 1,
last_seen TEXT,
PRIMARY KEY (node_a, node_b)
)`)
return err
}
// loadNeighborEdgesFromDB loads all edges from the neighbor_edges table
// and builds an in-memory NeighborGraph.
func loadNeighborEdgesFromDB(conn *sql.DB) *NeighborGraph {
g := NewNeighborGraph()
rows, err := conn.Query("SELECT node_a, node_b, count, last_seen FROM neighbor_edges")
if err != nil {
log.Printf("[neighbor] failed to load neighbor_edges: %v", err)
return g
}
defer rows.Close()
count := 0
for rows.Next() {
var a, b string
var cnt int
var lastSeen sql.NullString
if err := rows.Scan(&a, &b, &cnt, &lastSeen); err != nil {
continue
}
ts := time.Time{}
if lastSeen.Valid {
ts = parseTimestamp(lastSeen.String)
}
// Build edge directly (both nodes are full pubkeys from persisted data)
key := makeEdgeKey(a, b)
g.mu.Lock()
e, exists := g.edges[key]
if !exists {
e = &NeighborEdge{
NodeA: key.A,
NodeB: key.B,
Observers: make(map[string]bool),
FirstSeen: ts,
LastSeen: ts,
Count: cnt,
}
g.edges[key] = e
g.byNode[key.A] = append(g.byNode[key.A], e)
g.byNode[key.B] = append(g.byNode[key.B], e)
} else {
e.Count += cnt
if ts.After(e.LastSeen) {
e.LastSeen = ts
}
}
g.mu.Unlock()
count++
}
if count > 0 {
g.mu.Lock()
g.builtAt = time.Now()
g.mu.Unlock()
log.Printf("[neighbor] loaded %d edges from neighbor_edges table", count)
}
return g
}
// ─── shared async persistence helper ───────────────────────────────────────────
// persistObsUpdate holds data for a resolved_path SQLite update.
type persistObsUpdate struct {
obsID int
resolvedPath string
}
// persistEdgeUpdate holds data for a neighbor_edges SQLite upsert.
type persistEdgeUpdate struct {
a, b, ts string
}
// asyncPersistResolvedPathsAndEdges writes resolved_path updates and neighbor
// edge upserts to SQLite in a background goroutine. Shared between
// IngestNewFromDB and IngestNewObservations to avoid DRY violation.
func asyncPersistResolvedPathsAndEdges(dbPath string, obsUpdates []persistObsUpdate, edgeUpdates []persistEdgeUpdate, logPrefix string) {
if len(obsUpdates) == 0 && len(edgeUpdates) == 0 {
return
}
// Try-acquire semaphore BEFORE spawning goroutine. If another
// persistence operation is already running, drop this batch —
// data lives in memory and will be backfilled on restart.
select {
case persistSem <- struct{}{}:
// Acquired — spawn goroutine to do the work.
default:
log.Printf("[store] %s skipped: persistence already in progress", logPrefix)
return
}
go func() {
defer func() { <-persistSem }()
rw, err := openRW(dbPath)
if err != nil {
log.Printf("[store] %s rw open error: %v", logPrefix, err)
return
}
defer rw.Close()
if len(obsUpdates) > 0 {
sqlTx, err := rw.Begin()
if err == nil {
stmt, err := sqlTx.Prepare("UPDATE observations SET resolved_path = ? WHERE id = ?")
if err == nil {
var firstErr error
for _, u := range obsUpdates {
if _, err := stmt.Exec(u.resolvedPath, u.obsID); err != nil && firstErr == nil {
firstErr = err
}
}
stmt.Close()
if firstErr != nil {
log.Printf("[store] %s resolved_path error (first): %v", logPrefix, firstErr)
}
} else {
log.Printf("[store] %s resolved_path prepare error: %v", logPrefix, err)
}
sqlTx.Commit()
}
}
if len(edgeUpdates) > 0 {
sqlTx, err := rw.Begin()
if err == nil {
stmt, err := sqlTx.Prepare(`INSERT INTO neighbor_edges (node_a, node_b, count, last_seen)
VALUES (?, ?, 1, ?)
ON CONFLICT(node_a, node_b) DO UPDATE SET
count = count + 1, last_seen = MAX(last_seen, excluded.last_seen)`)
if err == nil {
var firstErr error
for _, e := range edgeUpdates {
if _, err := stmt.Exec(e.a, e.b, e.ts); err != nil && firstErr == nil {
firstErr = err
}
}
stmt.Close()
if firstErr != nil {
log.Printf("[store] %s edge error (first): %v", logPrefix, firstErr)
}
} else {
log.Printf("[store] %s edge prepare error: %v", logPrefix, err)
}
sqlTx.Commit()
}
}
}()
}
// neighborEdgesTableExists checks if the neighbor_edges table has any data.
func neighborEdgesTableExists(conn *sql.DB) bool {
var cnt int
err := conn.QueryRow("SELECT COUNT(*) FROM neighbor_edges").Scan(&cnt)
if err != nil {
return false // table doesn't exist
}
return cnt > 0
}
// buildAndPersistEdges scans all packets in the store, extracts edges per
// ADVERT/non-ADVERT rules, and persists them to SQLite.
func buildAndPersistEdges(store *PacketStore, rw *sql.DB) int {
store.mu.RLock()
packets := make([]*StoreTx, len(store.packets))
copy(packets, store.packets)
store.mu.RUnlock()
_, pm := store.getCachedNodesAndPM()
tx, err := rw.Begin()
if err != nil {
log.Printf("[neighbor] begin tx error: %v", err)
return 0
}
defer tx.Rollback()
stmt, err := tx.Prepare(`INSERT INTO neighbor_edges (node_a, node_b, count, last_seen)
VALUES (?, ?, 1, ?)
ON CONFLICT(node_a, node_b) DO UPDATE SET
count = count + 1, last_seen = MAX(last_seen, excluded.last_seen)`)
if err != nil {
log.Printf("[neighbor] prepare stmt error: %v", err)
return 0
}
defer stmt.Close()
edgeCount := 0
var firstErr error
for _, pkt := range packets {
for _, obs := range pkt.Observations {
for _, ec := range extractEdgesFromObs(obs, pkt, pm) {
if _, err := stmt.Exec(ec.A, ec.B, ec.Timestamp); err != nil && firstErr == nil {
firstErr = err
}
edgeCount++
}
}
}
if firstErr != nil {
log.Printf("[neighbor] edge exec error (first): %v", firstErr)
}
if err := tx.Commit(); err != nil {
log.Printf("[neighbor] commit error: %v", err)
return 0
}
return edgeCount
}
// ─── resolved_path column ──────────────────────────────────────────────────────
// ensureResolvedPathColumn adds the resolved_path column to observations if missing.
func ensureResolvedPathColumn(dbPath string) error {
rw, err := openRW(dbPath)
if err != nil {
return err
}
defer rw.Close()
// Check if column already exists
rows, err := rw.Query("PRAGMA table_info(observations)")
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
var cid int
var colName string
var colType sql.NullString
var notNull, pk int
var dflt sql.NullString
if rows.Scan(&cid, &colName, &colType, &notNull, &dflt, &pk) == nil && colName == "resolved_path" {
return nil // already exists
}
}
_, err = rw.Exec("ALTER TABLE observations ADD COLUMN resolved_path TEXT")
if err != nil {
return fmt.Errorf("add resolved_path column: %w", err)
}
log.Println("[store] Added resolved_path column to observations")
return nil
}
// resolvePathForObs resolves hop prefixes to full pubkeys for an observation.
// Returns nil if path is empty.
func resolvePathForObs(pathJSON, observerID string, tx *StoreTx, pm *prefixMap, graph *NeighborGraph) []*string {
hops := parsePathJSON(pathJSON)
if len(hops) == 0 {
return nil
}
// Build context pubkeys: observer + originator (if known)
contextPKs := make([]string, 0, 3)
if observerID != "" {
contextPKs = append(contextPKs, strings.ToLower(observerID))
}
fromNode := extractFromNode(tx)
if fromNode != "" {
contextPKs = append(contextPKs, strings.ToLower(fromNode))
}
resolved := make([]*string, len(hops))
for i, hop := range hops {
// Add adjacent hops as context for disambiguation
ctx := make([]string, len(contextPKs), len(contextPKs)+2)
copy(ctx, contextPKs)
// Add previously resolved hops as context
if i > 0 && resolved[i-1] != nil {
ctx = append(ctx, *resolved[i-1])
}
node, _, _ := pm.resolveWithContext(hop, ctx, graph)
if node != nil {
pk := strings.ToLower(node.PublicKey)
resolved[i] = &pk
}
}
return resolved
}
// marshalResolvedPath converts []*string to JSON for storage.
func marshalResolvedPath(rp []*string) string {
if len(rp) == 0 {
return ""
}
b, err := json.Marshal(rp)
if err != nil {
return ""
}
return string(b)
}
// unmarshalResolvedPath parses a resolved_path JSON string.
func unmarshalResolvedPath(s string) []*string {
if s == "" {
return nil
}
var result []*string
if json.Unmarshal([]byte(s), &result) != nil {
return nil
}
return result
}
// backfillResolvedPathsAsync processes observations with NULL resolved_path in
// chunks, yielding between batches so HTTP handlers remain responsive. It sets
// store.backfillComplete when finished and re-picks best observations for any
// transmissions affected by newly resolved paths.
func backfillResolvedPathsAsync(store *PacketStore, dbPath string, chunkSize int, yieldDuration time.Duration, backfillHours int) {
defer func() {
if r := recover(); r != nil {
log.Printf("[store] backfillResolvedPathsAsync panic recovered: %v", r)
}
}()
// Collect ALL pending obs refs upfront in one pass under a single RLock (fix A).
type obsRef struct {
obsID int
pathJSON string
observerID string
txJSON string
payloadType *int
txHash string // to re-pick best obs
}
cutoff := time.Now().UTC().Add(-time.Duration(backfillHours) * time.Hour)
store.mu.RLock()
pm := store.nodePM
var allPending []obsRef
for _, tx := range store.packets {
// Skip transmissions older than the backfill window.
if tx.FirstSeen != "" {
if ts, err := time.Parse(time.RFC3339Nano, tx.FirstSeen); err == nil && ts.Before(cutoff) {
continue
}
// Also try the common SQLite format
if ts, err := time.Parse("2006-01-02 15:04:05", tx.FirstSeen); err == nil && ts.Before(cutoff) {
continue
}
}
for _, obs := range tx.Observations {
// Check if this observation has been resolved: look up in the index.
// If the tx has no reverse-map entries AND path is non-empty, it needs backfill.
hasRP := false
if _, ok := store.resolvedPubkeyReverse[tx.ID]; ok {
hasRP = true
}
if !hasRP && obs.PathJSON != "" && obs.PathJSON != "[]" {
allPending = append(allPending, obsRef{
obsID: obs.ID,
pathJSON: obs.PathJSON,
observerID: obs.ObserverID,
txJSON: tx.DecodedJSON,
payloadType: tx.PayloadType,
txHash: tx.Hash,
})
}
}
}
store.mu.RUnlock()
totalPending := len(allPending)
if totalPending == 0 || pm == nil {
store.backfillComplete.Store(true)
log.Printf("[store] async resolved_path backfill: nothing to do")
return
}
store.backfillTotal.Store(int64(totalPending))
store.backfillProcessed.Store(0)
log.Printf("[store] async resolved_path backfill starting: %d observations", totalPending)
// Open RW connection once before the chunk loop (fix B).
var rw *sql.DB
if dbPath != "" {
var err error
rw, err = openRW(dbPath)
if err != nil {
log.Printf("[store] async backfill: open rw error: %v", err)
}
}
defer func() {
if rw != nil {
rw.Close()
}
}()
totalProcessed := 0
for totalProcessed < totalPending {
end := totalProcessed + chunkSize
if end > totalPending {
end = totalPending
}
chunk := allPending[totalProcessed:end]
// Re-read graph under RLock at the start of each chunk so we pick up
// a freshly-built graph once the background build goroutine completes,
// instead of using the potentially-empty graph captured at cold start.
store.mu.RLock()
graph := store.graph
store.mu.RUnlock()
// Resolve paths outside any lock.
type resolved struct {
obsID int
rp []*string
rpJSON string
txHash string
}
var results []resolved
for _, ref := range chunk {
fakeTx := &StoreTx{DecodedJSON: ref.txJSON, PayloadType: ref.payloadType}
rp := resolvePathForObs(ref.pathJSON, ref.observerID, fakeTx, pm, graph)
if len(rp) > 0 {
rpJSON := marshalResolvedPath(rp)
if rpJSON != "" {
results = append(results, resolved{ref.obsID, rp, rpJSON, ref.txHash})
}
}
}
// Persist to SQLite using the shared connection.
if len(results) > 0 && rw != nil {
sqlTx, err := rw.Begin()
if err != nil {
log.Printf("[store] async backfill: begin tx error: %v", err)
} else {
stmt, err := sqlTx.Prepare("UPDATE observations SET resolved_path = ? WHERE id = ?")
if err != nil {
log.Printf("[store] async backfill: prepare error: %v", err)
sqlTx.Rollback()
} else {
var execErr error
for _, r := range results {
if _, e := stmt.Exec(r.rpJSON, r.obsID); e != nil && execErr == nil {
execErr = e
}
}
if execErr != nil {
log.Printf("[store] async backfill: exec error (first): %v", execErr)
}
stmt.Close()
if err := sqlTx.Commit(); err != nil {
log.Printf("[store] async backfill: commit error: %v", err)
}
}
}
// Update in-memory state: update resolved pubkey index, re-pick best observation,
// and invalidate LRU cache entries for backfilled observations (#800).
//
// Lock ordering: always take s.mu BEFORE lruMu. The read path
// (fetchResolvedPathForObs) takes lruMu independently of s.mu,
// so we must NOT hold s.mu while taking lruMu. Instead, collect
// obsIDs to invalidate under s.mu, release it, then take lruMu.
store.mu.Lock()
affectedSet := make(map[string]bool)
lruInvalidate := make([]int, 0, len(results))
for _, r := range results {
// Remove old index entries for this tx, then re-add with new pubkeys
if !affectedSet[r.txHash] {
affectedSet[r.txHash] = true
if tx, ok := store.byHash[r.txHash]; ok {
store.removeFromResolvedPubkeyIndex(tx.ID)
}
}
// Add new resolved pubkeys to index
if tx, ok := store.byHash[r.txHash]; ok {
pks := extractResolvedPubkeys(r.rp)
store.addToResolvedPubkeyIndex(tx.ID, pks)
// Update byNode for relay nodes
for _, pk := range pks {
store.addToByNode(tx, pk)
}
// Update byPathHop resolved-key entries
hopsSeen := make(map[string]bool)
for _, hop := range txGetParsedPath(tx) {
hopsSeen[strings.ToLower(hop)] = true
}
for _, pk := range pks {
if !hopsSeen[pk] {
hopsSeen[pk] = true
store.byPathHop[pk] = append(store.byPathHop[pk], tx)
}
}
}
lruInvalidate = append(lruInvalidate, r.obsID)
}
// Re-pick best observation for affected transmissions
for txHash := range affectedSet {
if tx, ok := store.byHash[txHash]; ok {
pickBestObservation(tx)
}
}
store.mu.Unlock()
// Invalidate LRU entries AFTER releasing s.mu to maintain lock
// ordering (lruMu must never be taken while s.mu is held).
store.lruMu.Lock()
for _, obsID := range lruInvalidate {
store.lruDelete(obsID)
}
store.lruMu.Unlock()
}
totalProcessed += len(chunk)
store.backfillProcessed.Store(int64(totalProcessed))
pct := float64(totalProcessed) / float64(totalPending) * 100
log.Printf("[store] backfill progress: %d/%d observations (%.1f%%)", totalProcessed, totalPending, pct)
time.Sleep(yieldDuration)
}
store.backfillComplete.Store(true)
log.Printf("[store] async resolved_path backfill complete: %d observations processed", totalProcessed)
}
// ─── Shared helpers ────────────────────────────────────────────────────────────
// edgeCandidate represents an extracted edge to be persisted.
type edgeCandidate struct {
A, B, Timestamp string
}
// extractEdgesFromObs extracts neighbor edge candidates from a single observation.
// For ADVERTs: originator↔path[0] (if unambiguous). For ALL types: observer↔path[last] (if unambiguous).
// Also handles zero-hop ADVERTs (originator↔observer direct link).
func extractEdgesFromObs(obs *StoreObs, tx *StoreTx, pm *prefixMap) []edgeCandidate {
isAdvert := tx.PayloadType != nil && *tx.PayloadType == PayloadADVERT
fromNode := extractFromNode(tx)
path := parsePathJSON(obs.PathJSON)
observerPK := strings.ToLower(obs.ObserverID)
ts := obs.Timestamp
var edges []edgeCandidate
if len(path) == 0 {
if isAdvert && fromNode != "" {
fromLower := strings.ToLower(fromNode)
if fromLower != observerPK {
a, b := fromLower, observerPK
if a > b {
a, b = b, a
}
edges = append(edges, edgeCandidate{a, b, ts})
}
}
return edges
}
// Edge 1: originator ↔ path[0] — ADVERTs only (resolve prefix to full pubkey)
if isAdvert && fromNode != "" && pm != nil {
firstHop := strings.ToLower(path[0])
fromLower := strings.ToLower(fromNode)
candidates := pm.m[firstHop]
if len(candidates) == 1 {
resolved := strings.ToLower(candidates[0].PublicKey)
if resolved != fromLower {
a, b := fromLower, resolved
if a > b {
a, b = b, a
}
edges = append(edges, edgeCandidate{a, b, ts})
}
}
}
// Edge 2: observer ↔ path[last] — ALL packet types
if pm != nil {
lastHop := strings.ToLower(path[len(path)-1])
candidates := pm.m[lastHop]
if len(candidates) == 1 {
resolved := strings.ToLower(candidates[0].PublicKey)
if resolved != observerPK {
a, b := observerPK, resolved
if a > b {
a, b = b, a
}
edges = append(edges, edgeCandidate{a, b, ts})
}
}
}
return edges
}
// openRW opens a read-write SQLite connection (same pattern as PruneOldPackets).
func openRW(dbPath string) (*sql.DB, error) {
dsn := fmt.Sprintf("file:%s?_journal_mode=WAL", dbPath)
rw, err := sql.Open("sqlite", dsn)
if err != nil {
return nil, err
}
rw.SetMaxOpenConns(1)
// DSN _busy_timeout may not be honored by all drivers; set via PRAGMA
// to guarantee SQLite retries for up to 5s before returning SQLITE_BUSY.
if _, err := rw.Exec("PRAGMA busy_timeout = 5000"); err != nil {
rw.Close()
return nil, fmt.Errorf("set busy_timeout: %w", err)
}
return rw, nil
}
// PruneNeighborEdges removes edges older than maxAgeDays from both SQLite and
// the in-memory graph. Uses openRW internally because the shared database.conn
// is opened with mode=ro and DELETEs would silently fail.
func PruneNeighborEdges(dbPath string, graph *NeighborGraph, maxAgeDays int) (int, error) {
cutoff := time.Now().UTC().Add(-time.Duration(maxAgeDays) * 24 * time.Hour)
// 1. Prune from SQLite using a read-write connection
var dbPruned int64
rw, err := openRW(dbPath)
if err != nil {
return 0, fmt.Errorf("prune neighbor_edges: open rw: %w", err)
}
defer rw.Close()
res, err := rw.Exec("DELETE FROM neighbor_edges WHERE last_seen < ?", cutoff.Format(time.RFC3339))
if err != nil {
return 0, fmt.Errorf("prune neighbor_edges: %w", err)
}
dbPruned, _ = res.RowsAffected()
// 2. Prune from in-memory graph
memPruned := 0
if graph != nil {
memPruned = graph.PruneOlderThan(cutoff)
}
if dbPruned > 0 || memPruned > 0 {
log.Printf("[neighbor-prune] removed %d DB rows, %d in-memory edges older than %d days", dbPruned, memPruned, maxAgeDays)
}
return int(dbPruned), nil
}
+540
View File
@@ -0,0 +1,540 @@
package main
import (
"database/sql"
"encoding/json"
"path/filepath"
"strings"
"testing"
"time"
_ "modernc.org/sqlite"
)
// createTestDBWithSchema creates a temp SQLite DB with the standard schema + resolved_path column.
func createTestDBWithSchema(t *testing.T) (*DB, string) {
t.Helper()
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
// Create tables
conn.Exec(`CREATE TABLE transmissions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
raw_hex TEXT, hash TEXT UNIQUE, first_seen TEXT,
route_type INTEGER, payload_type INTEGER, payload_version INTEGER,
decoded_json TEXT, channel_hash TEXT DEFAULT NULL
)`)
conn.Exec(`CREATE TABLE observers (
id TEXT PRIMARY KEY, name TEXT, iata TEXT
)`)
conn.Exec(`CREATE TABLE observations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
transmission_id INTEGER NOT NULL REFERENCES transmissions(id),
observer_id TEXT, observer_name TEXT, direction TEXT,
snr REAL, rssi REAL, score INTEGER,
path_json TEXT, timestamp TEXT,
resolved_path TEXT, raw_hex TEXT
)`)
conn.Exec(`CREATE TABLE nodes (
public_key TEXT PRIMARY KEY, name TEXT, role TEXT,
lat REAL, lon REAL, last_seen TEXT, first_seen TEXT,
advert_count INTEGER DEFAULT 0
)`)
conn.Close()
db, err := OpenDB(dbPath)
if err != nil {
t.Fatal(err)
}
return db, dbPath
}
func TestResolvePathForObs(t *testing.T) {
// Build a prefix map with known nodes
nodes := []nodeInfo{
{PublicKey: "aabbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb", Name: "Node-AA"},
{PublicKey: "bbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb11", Name: "Node-BB"},
}
pm := buildPrefixMap(nodes)
graph := NewNeighborGraph()
tx := &StoreTx{
DecodedJSON: `{"pubKey": "originator1234567890"}`,
PayloadType: intPtr(4),
}
// Unambiguous prefixes should resolve
rp := resolvePathForObs(`["aa","bb"]`, "observer1", tx, pm, graph)
if len(rp) != 2 {
t.Fatalf("expected 2 resolved hops, got %d", len(rp))
}
if rp[0] == nil || !strings.HasPrefix(*rp[0], "aabbcc") {
t.Errorf("expected first hop to resolve to Node-AA, got %v", rp[0])
}
if rp[1] == nil || !strings.HasPrefix(*rp[1], "bbccdd") {
t.Errorf("expected second hop to resolve to Node-BB, got %v", rp[1])
}
}
func TestResolvePathForObs_EmptyPath(t *testing.T) {
pm := buildPrefixMap(nil)
rp := resolvePathForObs(`[]`, "", &StoreTx{}, pm, nil)
if rp != nil {
t.Errorf("expected nil for empty path, got %v", rp)
}
rp = resolvePathForObs("", "", &StoreTx{}, pm, nil)
if rp != nil {
t.Errorf("expected nil for empty string, got %v", rp)
}
}
func TestResolvePathForObs_Unresolvable(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aabbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb", Name: "Node-AA"},
}
pm := buildPrefixMap(nodes)
// "zz" prefix doesn't match any node
rp := resolvePathForObs(`["zz"]`, "", &StoreTx{}, pm, nil)
if len(rp) != 1 {
t.Fatalf("expected 1 hop, got %d", len(rp))
}
if rp[0] != nil {
t.Errorf("expected nil for unresolvable hop, got %v", *rp[0])
}
}
func TestMarshalUnmarshalResolvedPath(t *testing.T) {
pk1 := "aabbccdd"
var rp []*string
rp = append(rp, &pk1, nil)
j := marshalResolvedPath(rp)
if j == "" {
t.Fatal("expected non-empty JSON")
}
parsed := unmarshalResolvedPath(j)
if len(parsed) != 2 {
t.Fatalf("expected 2 elements, got %d", len(parsed))
}
if parsed[0] == nil || *parsed[0] != "aabbccdd" {
t.Errorf("first element wrong: %v", parsed[0])
}
if parsed[1] != nil {
t.Errorf("second element should be nil, got %v", *parsed[1])
}
}
func TestMarshalResolvedPath_Empty(t *testing.T) {
if marshalResolvedPath(nil) != "" {
t.Error("expected empty for nil")
}
if marshalResolvedPath([]*string{}) != "" {
t.Error("expected empty for empty slice")
}
}
func TestUnmarshalResolvedPath_Invalid(t *testing.T) {
if unmarshalResolvedPath("") != nil {
t.Error("expected nil for empty string")
}
if unmarshalResolvedPath("not json") != nil {
t.Error("expected nil for invalid JSON")
}
}
func TestEnsureNeighborEdgesTable(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
// Create initial DB
conn, _ := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
conn.Exec("CREATE TABLE test (id INTEGER PRIMARY KEY)")
conn.Close()
if err := ensureNeighborEdgesTable(dbPath); err != nil {
t.Fatal(err)
}
// Verify table exists
conn, _ = sql.Open("sqlite", "file:"+dbPath+"?mode=ro")
defer conn.Close()
var cnt int
if err := conn.QueryRow("SELECT COUNT(*) FROM neighbor_edges").Scan(&cnt); err != nil {
t.Fatalf("neighbor_edges table not created: %v", err)
}
}
func TestLoadNeighborEdgesFromDB(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, _ := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
conn.Exec(`CREATE TABLE neighbor_edges (
node_a TEXT NOT NULL, node_b TEXT NOT NULL,
count INTEGER DEFAULT 1, last_seen TEXT,
PRIMARY KEY (node_a, node_b)
)`)
conn.Exec("INSERT INTO neighbor_edges VALUES ('aaa', 'bbb', 5, '2024-01-01T00:00:00Z')")
conn.Exec("INSERT INTO neighbor_edges VALUES ('ccc', 'ddd', 3, '2024-01-02T00:00:00Z')")
g := loadNeighborEdgesFromDB(conn)
conn.Close()
// Should have 2 edges
edges := g.AllEdges()
if len(edges) != 2 {
t.Errorf("expected 2 edges, got %d", len(edges))
}
// Check neighbors
n := g.Neighbors("aaa")
if len(n) != 1 {
t.Errorf("expected 1 neighbor for aaa, got %d", len(n))
}
}
func TestStoreObsResolvedPathInBroadcast(t *testing.T) {
// After #800 refactor, resolved_path is no longer stored on StoreTx/StoreObs structs.
// Broadcast maps carry resolved_path from the decode-window, not from struct fields.
// This test verifies pickBestObservation no longer sets ResolvedPath on tx.
obs := &StoreObs{
ID: 1,
ObserverID: "obs1",
ObserverName: "Observer 1",
PathJSON: `["aa"]`,
Timestamp: "2024-01-01T00:00:00Z",
}
tx := &StoreTx{
ID: 1,
Hash: "abc123",
Observations: []*StoreObs{obs},
}
pickBestObservation(tx)
// tx should NOT have a ResolvedPath field anymore (compile-time guard)
// Verify the best observation's fields are propagated correctly
if tx.ObserverID != "obs1" {
t.Errorf("expected ObserverID=obs1, got %s", tx.ObserverID)
}
}
func TestResolvedPathInTxToMap(t *testing.T) {
// After #800, txToMap no longer includes resolved_path from the struct.
// resolved_path is only available via on-demand SQL fetch (txToMapWithRP).
tx := &StoreTx{
ID: 1,
Hash: "abc123",
PathJSON: `["aa"]`,
obsKeys: make(map[string]bool),
}
m := txToMap(tx)
if _, ok := m["resolved_path"]; ok {
t.Error("resolved_path should not be in txToMap output (removed in #800)")
}
}
func TestResolvedPathOmittedWhenNil(t *testing.T) {
tx := &StoreTx{
ID: 1,
Hash: "abc123",
obsKeys: make(map[string]bool),
}
m := txToMap(tx)
if _, ok := m["resolved_path"]; ok {
t.Error("resolved_path should not be in map when nil")
}
}
func TestEnsureResolvedPathColumn(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, _ := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
conn.Exec(`CREATE TABLE observations (
id INTEGER PRIMARY KEY, transmission_id INTEGER,
observer_id TEXT, path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
conn.Close()
if err := ensureResolvedPathColumn(dbPath); err != nil {
t.Fatal(err)
}
// Verify column exists
conn, _ = sql.Open("sqlite", "file:"+dbPath+"?mode=ro")
defer conn.Close()
rows, _ := conn.Query("PRAGMA table_info(observations)")
found := false
for rows.Next() {
var cid int
var colName string
var colType sql.NullString
var notNull, pk int
var dflt sql.NullString
rows.Scan(&cid, &colName, &colType, &notNull, &dflt, &pk)
if colName == "resolved_path" {
found = true
}
}
rows.Close()
if !found {
t.Error("resolved_path column not added")
}
// Running again should be idempotent
if err := ensureResolvedPathColumn(dbPath); err != nil {
t.Fatal("second call should be idempotent:", err)
}
}
func TestDBDetectsResolvedPathColumn(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
// Create DB without resolved_path
conn, _ := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
conn.Exec(`CREATE TABLE observations (id INTEGER PRIMARY KEY, observer_idx INTEGER)`)
conn.Exec(`CREATE TABLE transmissions (id INTEGER PRIMARY KEY)`)
conn.Close()
db, err := OpenDB(dbPath)
if err != nil {
t.Fatal(err)
}
if db.hasResolvedPath {
t.Error("should not detect resolved_path when column missing")
}
db.Close()
// Add resolved_path column
conn, _ = sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
conn.Exec("ALTER TABLE observations ADD COLUMN resolved_path TEXT")
conn.Close()
db, err = OpenDB(dbPath)
if err != nil {
t.Fatal(err)
}
if !db.hasResolvedPath {
t.Error("should detect resolved_path when column exists")
}
db.Close()
}
func TestLoadWithResolvedPath(t *testing.T) {
db, dbPath := createTestDBWithSchema(t)
defer db.Close()
// Insert test data
rw, _ := openRW(dbPath)
rw.Exec(`INSERT INTO transmissions (id, hash, first_seen, payload_type, decoded_json)
VALUES (1, 'hash1', '2024-01-01T00:00:00Z', 4, '{"pubKey":"origpk"}')`)
rw.Exec(`INSERT INTO observations (id, transmission_id, observer_id, observer_name, path_json, timestamp, resolved_path)
VALUES (1, 1, 'obs1', 'Observer1', '["aa"]', '2024-01-01T00:00:00Z', '["aabbccdd"]')`)
rw.Close()
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatal(err)
}
if len(store.packets) != 1 {
t.Fatalf("expected 1 packet, got %d", len(store.packets))
}
tx := store.packets[0]
if len(tx.Observations) != 1 {
t.Fatalf("expected 1 observation, got %d", len(tx.Observations))
}
// After #800, ResolvedPath is not stored on StoreObs struct.
// Instead, resolved pubkeys are in the membership index.
_ = tx.Observations[0] // obs exists
h := resolvedPubkeyHash("aabbccdd")
if len(store.resolvedPubkeyIndex[h]) != 1 {
t.Fatal("expected resolved pubkey to be indexed")
}
}
func TestResolvedPathInAPIResponse(t *testing.T) {
// After #800, TransmissionResp no longer has ResolvedPath field.
// resolved_path is included dynamically in map-based API responses.
resp := TransmissionResp{
ID: 1,
Hash: "test",
}
data, err := json.Marshal(resp)
if err != nil {
t.Fatal(err)
}
var m map[string]interface{}
json.Unmarshal(data, &m)
// resolved_path should NOT be in the marshaled JSON
if _, ok := m["resolved_path"]; ok {
t.Error("resolved_path should not be in TransmissionResp JSON (#800)")
}
}
func TestResolvedPathOmittedWhenEmpty(t *testing.T) {
resp := TransmissionResp{
ID: 1,
Hash: "test",
}
data, _ := json.Marshal(resp)
var m map[string]interface{}
json.Unmarshal(data, &m)
if _, ok := m["resolved_path"]; ok {
t.Error("resolved_path should be omitted when nil")
}
}
func TestExtractEdgesFromObs_AdvertNoPath(t *testing.T) {
tx := &StoreTx{
DecodedJSON: `{"pubKey":"aaaa1111"}`,
PayloadType: intPtr(4),
}
obs := &StoreObs{
ObserverID: "bbbb2222",
PathJSON: "",
Timestamp: "2024-01-01T00:00:00Z",
}
edges := extractEdgesFromObs(obs, tx, nil)
if len(edges) != 1 {
t.Fatalf("expected 1 edge for zero-hop advert, got %d", len(edges))
}
// Canonical ordering: aaaa < bbbb
if edges[0].A != "aaaa1111" || edges[0].B != "bbbb2222" {
t.Errorf("unexpected edge: %+v", edges[0])
}
}
func TestExtractEdgesFromObs_NonAdvertNoPath(t *testing.T) {
tx := &StoreTx{PayloadType: intPtr(1)}
obs := &StoreObs{ObserverID: "obs1", PathJSON: ""}
edges := extractEdgesFromObs(obs, tx, nil)
if len(edges) != 0 {
t.Errorf("expected 0 edges for non-advert without path, got %d", len(edges))
}
}
func TestExtractEdgesFromObs_WithPath(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aabbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb", Name: "Node-AA"},
{PublicKey: "ffgghhii1234567890aabbccddee1234567890aabbccddee1234567890aabb11", Name: "Node-FF"},
}
pm := buildPrefixMap(nodes)
tx := &StoreTx{
DecodedJSON: `{"pubKey":"originator00"}`,
PayloadType: intPtr(4),
}
obs := &StoreObs{
ObserverID: "observer00",
PathJSON: `["aa","ff"]`,
Timestamp: "2024-01-01T00:00:00Z",
}
edges := extractEdgesFromObs(obs, tx, pm)
// Should get: originator↔aa (advert), observer↔ff (last hop)
if len(edges) != 2 {
t.Fatalf("expected 2 edges, got %d", len(edges))
}
}
func TestExtractEdgesFromObs_SameNodeNoEdge(t *testing.T) {
tx := &StoreTx{
DecodedJSON: `{"pubKey":"same1234"}`,
PayloadType: intPtr(4),
}
obs := &StoreObs{
ObserverID: "same1234",
PathJSON: "",
Timestamp: "2024-01-01T00:00:00Z",
}
edges := extractEdgesFromObs(obs, tx, nil)
if len(edges) != 0 {
t.Errorf("expected 0 edges when originator == observer, got %d", len(edges))
}
}
func TestPersistSemaphoreTryAcquireSkipsBatch(t *testing.T) {
// Verify that persistSem is a buffered channel of size 1.
if cap(persistSem) != 1 {
t.Errorf("persistSem capacity = %d, want 1", cap(persistSem))
}
// Acquire the semaphore to simulate an in-progress persistence.
persistSem <- struct{}{}
// asyncPersistResolvedPathsAndEdges should skip (not block, not
// spawn a goroutine) when the semaphore is already held.
done := make(chan struct{})
go func() {
asyncPersistResolvedPathsAndEdges(
"/nonexistent/path.db",
[]persistObsUpdate{{obsID: 1, resolvedPath: "x"}},
nil,
"test",
)
close(done)
}()
// If the function blocks on the semaphore instead of skipping,
// this select will hit the timeout.
select {
case <-done:
// Expected: returned immediately because semaphore was busy.
case <-time.After(500 * time.Millisecond):
<-persistSem
t.Fatal("asyncPersistResolvedPathsAndEdges blocked instead of skipping when semaphore was held")
}
<-persistSem // release
}
func TestOpenRW_BusyTimeout(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
// Create the DB file first
db, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
db.Exec("CREATE TABLE dummy (id INTEGER)")
db.Close()
// Open via openRW and verify busy_timeout is set
rw, err := openRW(dbPath)
if err != nil {
t.Fatalf("openRW failed: %v", err)
}
defer rw.Close()
var timeout int
if err := rw.QueryRow("PRAGMA busy_timeout").Scan(&timeout); err != nil {
t.Fatalf("query busy_timeout: %v", err)
}
if timeout != 5000 {
t.Errorf("expected busy_timeout=5000, got %d", timeout)
}
}
+311
View File
@@ -0,0 +1,311 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/gorilla/mux"
)
func TestConfigIsBlacklisted(t *testing.T) {
cfg := &Config{
NodeBlacklist: []string{"AA", "BB", "cc"},
}
tests := []struct {
pubkey string
want bool
}{
{"AA", true},
{"aa", true}, // case-insensitive
{"BB", true},
{"CC", true}, // lowercase "cc" matches uppercase
{"DD", false},
{"", false},
{"AAB", false},
}
for _, tt := range tests {
got := cfg.IsBlacklisted(tt.pubkey)
if got != tt.want {
t.Errorf("IsBlacklisted(%q) = %v, want %v", tt.pubkey, got, tt.want)
}
}
}
func TestConfigIsBlacklistedEmpty(t *testing.T) {
cfg := &Config{}
if cfg.IsBlacklisted("anything") {
t.Error("empty blacklist should not match anything")
}
if cfg.IsBlacklisted("") {
t.Error("empty blacklist should not match empty string")
}
}
func TestConfigBlacklistWhitespace(t *testing.T) {
cfg := &Config{
NodeBlacklist: []string{" AA ", "BB"},
}
if !cfg.IsBlacklisted("AA") {
t.Error("trimmed key should match")
}
if !cfg.IsBlacklisted(" AA ") {
t.Error("whitespace-padded key should match after trimming")
}
}
func TestConfigBlacklistEmptyEntries(t *testing.T) {
cfg := &Config{
NodeBlacklist: []string{"", " ", "AA"},
}
if !cfg.IsBlacklisted("AA") {
t.Error("non-empty entry should match")
}
if cfg.IsBlacklisted("") {
t.Error("empty blacklist entry should not match empty pubkey")
}
}
func TestBlacklistFiltersHandleNodes(t *testing.T) {
db := setupTestDB(t)
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role, last_seen) VALUES ('goodnode', 'GoodNode', 'companion', datetime('now'))")
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role, last_seen) VALUES ('badnode', 'BadNode', 'companion', datetime('now'))")
cfg := &Config{
NodeBlacklist: []string{"badnode"},
}
srv := NewServer(db, cfg, NewHub())
req := httptest.NewRequest("GET", "/api/nodes?limit=50", nil)
w := httptest.NewRecorder()
srv.RegisterRoutes(setupTestRouter(srv))
srv.router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp NodeListResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("failed to parse response: %v", err)
}
for _, node := range resp.Nodes {
if pk, _ := node["public_key"].(string); pk == "badnode" {
t.Error("blacklisted node should not appear in nodes list")
}
}
if resp.Total == 0 {
t.Error("expected at least one non-blacklisted node")
}
}
func TestBlacklistFiltersNodeDetail(t *testing.T) {
db := setupTestDB(t)
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role, last_seen) VALUES ('badnode', 'BadNode', 'companion', datetime('now'))")
cfg := &Config{
NodeBlacklist: []string{"badnode"},
}
srv := NewServer(db, cfg, NewHub())
req := httptest.NewRequest("GET", "/api/nodes/badnode", nil)
w := httptest.NewRecorder()
srv.RegisterRoutes(setupTestRouter(srv))
srv.router.ServeHTTP(w, req)
if w.Code != http.StatusNotFound {
t.Errorf("expected 404 for blacklisted node, got %d", w.Code)
}
}
func TestBlacklistFiltersNodeSearch(t *testing.T) {
db := setupTestDB(t)
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role, last_seen) VALUES ('badnode', 'TrollNode', 'companion', datetime('now'))")
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role, last_seen) VALUES ('goodnode', 'GoodNode', 'companion', datetime('now'))")
cfg := &Config{
NodeBlacklist: []string{"badnode"},
}
srv := NewServer(db, cfg, NewHub())
req := httptest.NewRequest("GET", "/api/nodes/search?q=Troll", nil)
w := httptest.NewRecorder()
srv.RegisterRoutes(setupTestRouter(srv))
srv.router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
var resp NodeSearchResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("failed to parse response: %v", err)
}
for _, node := range resp.Nodes {
if pk, _ := node["public_key"].(string); pk == "badnode" {
t.Error("blacklisted node should not appear in search results")
}
}
}
func TestNoBlacklistPassesAll(t *testing.T) {
db := setupTestDB(t)
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role, last_seen) VALUES ('somenode', 'SomeNode', 'companion', datetime('now'))")
cfg := &Config{}
srv := NewServer(db, cfg, NewHub())
req := httptest.NewRequest("GET", "/api/nodes?limit=50", nil)
w := httptest.NewRecorder()
srv.RegisterRoutes(setupTestRouter(srv))
srv.router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
var resp NodeListResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("failed to parse response: %v", err)
}
if resp.Total == 0 {
t.Error("without blacklist, node should appear")
}
}
// setupTestRouter creates a mux.Router and registers server routes.
func setupTestRouter(srv *Server) *mux.Router {
r := mux.NewRouter()
srv.RegisterRoutes(r)
srv.router = r
return r
}
func TestBlacklistFiltersNeighborGraph(t *testing.T) {
cfg := &Config{
NodeBlacklist: []string{"badnode"},
}
db := setupTestDB(t)
srv := NewServer(db, cfg, NewHub())
srv.RegisterRoutes(setupTestRouter(srv))
req := httptest.NewRequest("GET", "/api/analytics/neighbor-graph", nil)
w := httptest.NewRecorder()
srv.router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp map[string]interface{}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("failed to parse response: %v", err)
}
// Check edges don't contain blacklisted node
if edges, ok := resp["edges"].([]interface{}); ok {
for _, e := range edges {
if edge, ok := e.(map[string]interface{}); ok {
if src, _ := edge["source"].(string); src == "badnode" {
t.Error("blacklisted node should not appear as edge source in neighbor graph")
}
if tgt, _ := edge["target"].(string); tgt == "badnode" {
t.Error("blacklisted node should not appear as edge target in neighbor graph")
}
}
}
}
// Check nodes list doesn't contain blacklisted node
if nodes, ok := resp["nodes"].([]interface{}); ok {
for _, n := range nodes {
if node, ok := n.(map[string]interface{}); ok {
if pk, _ := node["pubkey"].(string); pk == "badnode" {
t.Error("blacklisted node should not appear in neighbor graph nodes")
}
}
}
}
}
func TestBlacklistFiltersResolveHops(t *testing.T) {
db := setupTestDB(t)
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role, last_seen) VALUES ('badnode', 'BadNode', 'companion', datetime('now'))")
cfg := &Config{
NodeBlacklist: []string{"badnode"},
}
srv := NewServer(db, cfg, NewHub())
srv.RegisterRoutes(setupTestRouter(srv))
req := httptest.NewRequest("GET", "/api/resolve-hops?hops=badnode", nil)
w := httptest.NewRecorder()
srv.router.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp ResolveHopsResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("failed to parse response: %v", err)
}
if hr, ok := resp.Resolved["badnode"]; ok {
for _, c := range hr.Candidates {
if c.Pubkey == "badnode" {
t.Error("blacklisted node should not appear as resolve-hops candidate")
}
}
}
}
func TestBlacklistFiltersSubpathDetail(t *testing.T) {
cfg := &Config{
NodeBlacklist: []string{"badnode"},
}
db := setupTestDB(t)
srv := NewServer(db, cfg, NewHub())
srv.RegisterRoutes(setupTestRouter(srv))
req := httptest.NewRequest("GET", "/api/analytics/subpath-detail?hops=badnode,othernode", nil)
w := httptest.NewRecorder()
srv.router.ServeHTTP(w, req)
if w.Code != http.StatusNotFound {
t.Errorf("expected 404 for subpath-detail with blacklisted hop, got %d", w.Code)
}
}
func TestBlacklistConcurrentIsBlacklisted(t *testing.T) {
cfg := &Config{
NodeBlacklist: []string{"AA", "BB", "CC"},
}
errc := make(chan error, 100)
for i := 0; i < 100; i++ {
go func() {
for j := 0; j < 100; j++ {
cfg.IsBlacklisted("AA")
cfg.IsBlacklisted("BB")
cfg.IsBlacklisted("DD")
}
}()
}
// If sync.Once is wrong, this would panic or race.
// We can't run the race detector on ARM, but at least verify no panics.
done := false
for !done {
select {
case <-errc:
t.Error("concurrent IsBlacklisted panicked")
default:
done = true
}
}
}
+134
View File
@@ -0,0 +1,134 @@
package main
import (
"fmt"
"testing"
)
// TestObsDedupCorrectness verifies that the map-based dedup produces correct
// results: no duplicate observations (same observerID + pathJSON) on a single
// transmission.
func TestObsDedupCorrectness(t *testing.T) {
tx := &StoreTx{
ID: 1,
Hash: "abc123",
obsKeys: make(map[string]bool),
}
// Add 5 unique observations
for i := 0; i < 5; i++ {
obsID := fmt.Sprintf("obs-%d", i)
pathJSON := fmt.Sprintf(`["path-%d"]`, i)
dk := obsID + "|" + pathJSON
if tx.obsKeys[dk] {
t.Fatalf("observation %d should not be a duplicate", i)
}
tx.Observations = append(tx.Observations, &StoreObs{
ID: i,
ObserverID: obsID,
PathJSON: pathJSON,
})
tx.obsKeys[dk] = true
tx.ObservationCount++
}
if tx.ObservationCount != 5 {
t.Fatalf("expected 5 observations, got %d", tx.ObservationCount)
}
// Try to add duplicates of each — all should be rejected
for i := 0; i < 5; i++ {
obsID := fmt.Sprintf("obs-%d", i)
pathJSON := fmt.Sprintf(`["path-%d"]`, i)
dk := obsID + "|" + pathJSON
if !tx.obsKeys[dk] {
t.Fatalf("observation %d should be detected as duplicate", i)
}
}
// Same observer, different path — should NOT be a duplicate
dk := "obs-0" + "|" + `["different-path"]`
if tx.obsKeys[dk] {
t.Fatal("different path should not be a duplicate")
}
// Different observer, same path — should NOT be a duplicate
dk = "obs-new" + "|" + `["path-0"]`
if tx.obsKeys[dk] {
t.Fatal("different observer should not be a duplicate")
}
}
// TestObsDedupNilMapSafety ensures obsKeys lazy init works for pre-existing
// transmissions that may not have the map initialized.
func TestObsDedupNilMapSafety(t *testing.T) {
tx := &StoreTx{ID: 1, Hash: "abc"}
// obsKeys is nil — the lazy init pattern used in IngestNewFromDB/IngestNewObservations
if tx.obsKeys == nil {
tx.obsKeys = make(map[string]bool)
}
dk := "obs1|path1"
if tx.obsKeys[dk] {
t.Fatal("should not be duplicate on empty map")
}
tx.obsKeys[dk] = true
if !tx.obsKeys[dk] {
t.Fatal("should be duplicate after insert")
}
}
// BenchmarkObsDedupMap benchmarks the map-based O(1) dedup approach.
func BenchmarkObsDedupMap(b *testing.B) {
for _, obsCount := range []int{10, 50, 100, 500} {
b.Run(fmt.Sprintf("obs=%d", obsCount), func(b *testing.B) {
// Pre-populate a tx with obsCount observations
tx := &StoreTx{
ID: 1,
obsKeys: make(map[string]bool),
}
for i := 0; i < obsCount; i++ {
obsID := fmt.Sprintf("obs-%d", i)
pathJSON := fmt.Sprintf(`["hop-%d"]`, i)
dk := obsID + "|" + pathJSON
tx.Observations = append(tx.Observations, &StoreObs{
ObserverID: obsID,
PathJSON: pathJSON,
})
tx.obsKeys[dk] = true
}
// Benchmark: check dedup for a new observation (not duplicate)
newDK := "new-obs|new-path"
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = tx.obsKeys[newDK]
}
})
}
}
// BenchmarkObsDedupLinear benchmarks the old O(n) linear scan for comparison.
func BenchmarkObsDedupLinear(b *testing.B) {
for _, obsCount := range []int{10, 50, 100, 500} {
b.Run(fmt.Sprintf("obs=%d", obsCount), func(b *testing.B) {
tx := &StoreTx{ID: 1}
for i := 0; i < obsCount; i++ {
tx.Observations = append(tx.Observations, &StoreObs{
ObserverID: fmt.Sprintf("obs-%d", i),
PathJSON: fmt.Sprintf(`["hop-%d"]`, i),
})
}
newObsID := "new-obs"
newPath := "new-path"
b.ResetTimer()
for i := 0; i < b.N; i++ {
for _, existing := range tx.Observations {
if existing.ObserverID == newObsID && existing.PathJSON == newPath {
break
}
}
}
})
}
}
+359
View File
@@ -0,0 +1,359 @@
package main
import (
"encoding/json"
"fmt"
"net/http"
"sort"
"strings"
"github.com/gorilla/mux"
)
// routeMeta holds metadata for a single API route.
type routeMeta struct {
Summary string `json:"summary"`
Description string `json:"description,omitempty"`
Tag string `json:"tag"`
Auth bool `json:"auth,omitempty"`
QueryParams []paramMeta `json:"queryParams,omitempty"`
}
type paramMeta struct {
Name string `json:"name"`
Description string `json:"description"`
Required bool `json:"required,omitempty"`
Type string `json:"type"` // "string", "integer", "boolean"
}
// routeDescriptions returns metadata for all known API routes.
// Key format: "METHOD /path/pattern"
func routeDescriptions() map[string]routeMeta {
return map[string]routeMeta{
// Config
"GET /api/config/cache": {Summary: "Get cache configuration", Tag: "config"},
"GET /api/config/client": {Summary: "Get client configuration", Tag: "config"},
"GET /api/config/regions": {Summary: "Get configured regions", Tag: "config"},
"GET /api/config/theme": {Summary: "Get theme configuration", Description: "Returns color maps, CSS variables, and theme defaults.", Tag: "config"},
"GET /api/config/map": {Summary: "Get map configuration", Tag: "config"},
"GET /api/config/geo-filter": {Summary: "Get geo-filter configuration", Tag: "config"},
// Admin / system
"GET /api/health": {Summary: "Health check", Description: "Returns server health, uptime, and memory stats.", Tag: "admin"},
"GET /api/stats": {Summary: "Network statistics", Description: "Returns aggregate stats (node counts, packet counts, observer counts). Cached for 10s.", Tag: "admin"},
"GET /api/perf": {Summary: "Performance statistics", Description: "Returns per-endpoint request timing and slow query log.", Tag: "admin"},
"POST /api/perf/reset": {Summary: "Reset performance stats", Tag: "admin", Auth: true},
"POST /api/admin/prune": {Summary: "Prune old data", Description: "Deletes packets and nodes older than the configured retention period.", Tag: "admin", Auth: true},
"GET /api/debug/affinity": {Summary: "Debug neighbor affinity scores", Tag: "admin", Auth: true},
// Packets
"GET /api/packets": {Summary: "List packets", Description: "Returns decoded packets with filtering, sorting, and pagination.", Tag: "packets",
QueryParams: []paramMeta{
{Name: "limit", Description: "Max packets to return", Type: "integer"},
{Name: "offset", Description: "Pagination offset", Type: "integer"},
{Name: "sort", Description: "Sort field", Type: "string"},
{Name: "order", Description: "Sort order (asc/desc)", Type: "string"},
{Name: "type", Description: "Filter by packet type", Type: "string"},
{Name: "observer", Description: "Filter by observer ID", Type: "string"},
{Name: "timeRange", Description: "Time range filter (e.g. 1h, 24h, 7d)", Type: "string"},
{Name: "search", Description: "Full-text search", Type: "string"},
{Name: "groupByHash", Description: "Group duplicate packets by hash", Type: "boolean"},
}},
"POST /api/packets": {Summary: "Ingest a packet", Description: "Submit a raw packet for decoding and storage.", Tag: "packets", Auth: true},
"GET /api/packets/{id}": {Summary: "Get packet detail", Tag: "packets"},
"GET /api/packets/timestamps": {Summary: "Get packet timestamp ranges", Tag: "packets"},
"POST /api/packets/observations": {Summary: "Batch submit observations", Description: "Submit multiple observer sightings for existing packets.", Tag: "packets"},
// Decode
"POST /api/decode": {Summary: "Decode a raw packet", Description: "Decodes a hex-encoded packet without storing it.", Tag: "packets"},
// Nodes
"GET /api/nodes": {Summary: "List nodes", Description: "Returns all known mesh nodes with status and metadata.", Tag: "nodes",
QueryParams: []paramMeta{
{Name: "role", Description: "Filter by node role", Type: "string"},
{Name: "status", Description: "Filter by status (active/stale/offline)", Type: "string"},
}},
"GET /api/nodes/search": {Summary: "Search nodes", Description: "Search nodes by name or public key prefix.", Tag: "nodes", QueryParams: []paramMeta{{Name: "q", Description: "Search query", Type: "string", Required: true}}},
"GET /api/nodes/bulk-health": {Summary: "Bulk node health", Description: "Returns health status for all nodes in one call.", Tag: "nodes"},
"GET /api/nodes/network-status": {Summary: "Network status summary", Description: "Returns counts of active, stale, and offline nodes.", Tag: "nodes"},
"GET /api/nodes/{pubkey}": {Summary: "Get node detail", Description: "Returns full detail for a single node by public key.", Tag: "nodes"},
"GET /api/nodes/{pubkey}/health": {Summary: "Get node health", Tag: "nodes"},
"GET /api/nodes/{pubkey}/paths": {Summary: "Get node routing paths", Tag: "nodes"},
"GET /api/nodes/{pubkey}/analytics": {Summary: "Get node analytics", Description: "Per-node packet counts, timing, and RF stats.", Tag: "nodes"},
"GET /api/nodes/{pubkey}/neighbors": {Summary: "Get node neighbors", Description: "Returns neighbor nodes with affinity scores.", Tag: "nodes"},
// Analytics
"GET /api/analytics/rf": {Summary: "RF analytics", Description: "SNR/RSSI distributions and statistics.", Tag: "analytics"},
"GET /api/analytics/topology": {Summary: "Network topology", Description: "Hop-count distribution and route analysis.", Tag: "analytics"},
"GET /api/analytics/channels": {Summary: "Channel analytics", Description: "Message counts and activity per channel.", Tag: "analytics"},
"GET /api/analytics/distance": {Summary: "Distance analytics", Description: "Geographic distance calculations between nodes.", Tag: "analytics"},
"GET /api/analytics/hash-sizes": {Summary: "Hash size analysis", Description: "Distribution of hash prefix sizes across the network.", Tag: "analytics"},
"GET /api/analytics/hash-collisions": {Summary: "Hash collision detection", Description: "Identifies nodes sharing hash prefixes.", Tag: "analytics"},
"GET /api/analytics/subpaths": {Summary: "Subpath analysis", Description: "Common routing subpaths through the mesh.", Tag: "analytics"},
"GET /api/analytics/subpaths-bulk": {Summary: "Bulk subpath analysis", Tag: "analytics"},
"GET /api/analytics/subpath-detail": {Summary: "Subpath detail", Tag: "analytics"},
"GET /api/analytics/neighbor-graph": {Summary: "Neighbor graph", Description: "Full neighbor affinity graph for visualization.", Tag: "analytics"},
// Channels
"GET /api/channels": {Summary: "List channels", Description: "Returns known mesh channels with message counts.", Tag: "channels"},
"GET /api/channels/{hash}/messages": {Summary: "Get channel messages", Description: "Returns messages for a specific channel.", Tag: "channels"},
// Observers
"GET /api/observers": {Summary: "List observers", Description: "Returns all known packet observers/gateways.", Tag: "observers"},
"GET /api/observers/{id}": {Summary: "Get observer detail", Tag: "observers"},
"GET /api/observers/{id}/metrics": {Summary: "Get observer metrics", Description: "Packet rates, uptime, and performance metrics.", Tag: "observers"},
"GET /api/observers/{id}/analytics": {Summary: "Get observer analytics", Tag: "observers"},
"GET /api/observers/metrics/summary": {Summary: "Observer metrics summary", Description: "Aggregate metrics across all observers.", Tag: "observers"},
// Misc
"GET /api/resolve-hops": {Summary: "Resolve hop path", Description: "Resolves hash prefixes in a hop path to node names. Returns affinity scores and best candidates.", Tag: "nodes", QueryParams: []paramMeta{{Name: "hops", Description: "Comma-separated hop hash prefixes", Type: "string", Required: true}}},
"GET /api/traces/{hash}": {Summary: "Get packet traces", Description: "Returns all observer sightings for a packet hash.", Tag: "packets"},
"GET /api/iata-coords": {Summary: "Get IATA airport coordinates", Description: "Returns lat/lon for known airport codes (used for observer positioning).", Tag: "config"},
"GET /api/audio-lab/buckets": {Summary: "Audio lab frequency buckets", Description: "Returns frequency bucket data for audio analysis.", Tag: "analytics"},
}
}
// buildOpenAPISpec constructs an OpenAPI 3.0 spec by walking the mux router.
func buildOpenAPISpec(router *mux.Router, version string) map[string]interface{} {
descriptions := routeDescriptions()
// Collect routes from the router
type routeInfo struct {
path string
method string
authReq bool
}
var routes []routeInfo
router.Walk(func(route *mux.Route, router *mux.Router, ancestors []*mux.Route) error {
path, err := route.GetPathTemplate()
if err != nil {
return nil
}
if !strings.HasPrefix(path, "/api/") {
return nil
}
// Skip the spec/docs endpoints themselves
if path == "/api/spec" || path == "/api/docs" {
return nil
}
methods, err := route.GetMethods()
if err != nil {
return nil
}
for _, m := range methods {
routes = append(routes, routeInfo{path: path, method: m})
}
return nil
})
// Sort routes for deterministic output
sort.Slice(routes, func(i, j int) bool {
if routes[i].path != routes[j].path {
return routes[i].path < routes[j].path
}
return routes[i].method < routes[j].method
})
// Build paths object
paths := make(map[string]interface{})
tagSet := make(map[string]bool)
for _, ri := range routes {
key := ri.method + " " + ri.path
meta, hasMeta := descriptions[key]
// Convert mux path params {name} to OpenAPI {name} (same format, convenient)
openAPIPath := ri.path
// Build operation
op := map[string]interface{}{
"summary": func() string {
if hasMeta {
return meta.Summary
}
return ri.path
}(),
"responses": map[string]interface{}{
"200": map[string]interface{}{
"description": "Success",
"content": map[string]interface{}{
"application/json": map[string]interface{}{
"schema": map[string]interface{}{"type": "object"},
},
},
},
},
}
if hasMeta {
if meta.Description != "" {
op["description"] = meta.Description
}
if meta.Tag != "" {
op["tags"] = []string{meta.Tag}
tagSet[meta.Tag] = true
}
if meta.Auth {
op["security"] = []map[string]interface{}{
{"ApiKeyAuth": []string{}},
}
}
// Add query parameters
if len(meta.QueryParams) > 0 {
params := make([]interface{}, 0, len(meta.QueryParams))
for _, qp := range meta.QueryParams {
p := map[string]interface{}{
"name": qp.Name,
"in": "query",
"required": qp.Required,
"schema": map[string]interface{}{"type": qp.Type},
}
if qp.Description != "" {
p["description"] = qp.Description
}
params = append(params, p)
}
op["parameters"] = params
}
}
// Extract path parameters from {name} patterns
pathParams := extractPathParams(openAPIPath)
if len(pathParams) > 0 {
existing, _ := op["parameters"].([]interface{})
for _, pp := range pathParams {
existing = append(existing, map[string]interface{}{
"name": pp,
"in": "path",
"required": true,
"schema": map[string]interface{}{"type": "string"},
})
}
op["parameters"] = existing
}
// Add to paths
methodLower := strings.ToLower(ri.method)
if _, ok := paths[openAPIPath]; !ok {
paths[openAPIPath] = make(map[string]interface{})
}
paths[openAPIPath].(map[string]interface{})[methodLower] = op
}
// Build tags array (sorted)
tagOrder := []string{"admin", "analytics", "channels", "config", "nodes", "observers", "packets"}
tagDescriptions := map[string]string{
"admin": "Server administration and diagnostics",
"analytics": "Network analytics and statistics",
"channels": "Mesh channel operations",
"config": "Server configuration",
"nodes": "Mesh node operations",
"observers": "Packet observer/gateway operations",
"packets": "Packet capture and decoding",
}
var tags []interface{}
for _, t := range tagOrder {
if tagSet[t] {
tags = append(tags, map[string]interface{}{
"name": t,
"description": tagDescriptions[t],
})
}
}
spec := map[string]interface{}{
"openapi": "3.0.3",
"info": map[string]interface{}{
"title": "CoreScope API",
"description": "MeshCore network analyzer — packet capture, node tracking, and mesh analytics.",
"version": version,
"license": map[string]interface{}{
"name": "MIT",
},
},
"paths": paths,
"tags": tags,
"components": map[string]interface{}{
"securitySchemes": map[string]interface{}{
"ApiKeyAuth": map[string]interface{}{
"type": "apiKey",
"in": "header",
"name": "X-API-Key",
},
},
},
}
return spec
}
// extractPathParams returns parameter names from a mux-style path like /api/nodes/{pubkey}.
func extractPathParams(path string) []string {
var params []string
for {
start := strings.Index(path, "{")
if start == -1 {
break
}
end := strings.Index(path[start:], "}")
if end == -1 {
break
}
params = append(params, path[start+1:start+end])
path = path[start+end+1:]
}
return params
}
// handleOpenAPISpec serves the OpenAPI 3.0 spec as JSON.
// The router is injected via RegisterRoutes storing it on the Server.
func (s *Server) handleOpenAPISpec(w http.ResponseWriter, r *http.Request) {
spec := buildOpenAPISpec(s.router, s.version)
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Access-Control-Allow-Origin", "*")
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
if err := enc.Encode(spec); err != nil {
http.Error(w, fmt.Sprintf("failed to encode spec: %v", err), http.StatusInternalServerError)
}
}
// handleSwaggerUI serves a minimal Swagger UI page.
func (s *Server) handleSwaggerUI(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
fmt.Fprint(w, swaggerUIHTML)
}
const swaggerUIHTML = `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>CoreScope API Swagger UI</title>
<link rel="stylesheet" href="https://unpkg.com/swagger-ui-dist@5/swagger-ui.css">
<style>
html { box-sizing: border-box; overflow-y: scroll; }
*, *:before, *:after { box-sizing: inherit; }
body { margin: 0; background: #fafafa; }
.topbar { display: none; }
</style>
</head>
<body>
<div id="swagger-ui"></div>
<script src="https://unpkg.com/swagger-ui-dist@5/swagger-ui-bundle.js"></script>
<script>
SwaggerUIBundle({
url: '/api/spec',
dom_id: '#swagger-ui',
deepLinking: true,
presets: [
SwaggerUIBundle.presets.apis,
SwaggerUIBundle.SwaggerUIStandalonePreset
],
layout: 'BaseLayout'
});
</script>
</body>
</html>`
+142
View File
@@ -0,0 +1,142 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestOpenAPISpecEndpoint(t *testing.T) {
_, r := setupTestServer(t)
req := httptest.NewRequest("GET", "/api/spec", nil)
w := httptest.NewRecorder()
r.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
ct := w.Header().Get("Content-Type")
if ct != "application/json; charset=utf-8" {
t.Errorf("unexpected content-type: %s", ct)
}
var spec map[string]interface{}
if err := json.Unmarshal(w.Body.Bytes(), &spec); err != nil {
t.Fatalf("invalid JSON: %v", err)
}
// Check required OpenAPI fields
if spec["openapi"] != "3.0.3" {
t.Errorf("expected openapi 3.0.3, got %v", spec["openapi"])
}
info, ok := spec["info"].(map[string]interface{})
if !ok {
t.Fatal("missing info object")
}
if info["title"] != "CoreScope API" {
t.Errorf("unexpected title: %v", info["title"])
}
paths, ok := spec["paths"].(map[string]interface{})
if !ok {
t.Fatal("missing paths object")
}
// Should have at least 20 paths
if len(paths) < 20 {
t.Errorf("expected at least 20 paths, got %d", len(paths))
}
// Check a known path exists
if _, ok := paths["/api/nodes"]; !ok {
t.Error("missing /api/nodes path")
}
if _, ok := paths["/api/packets"]; !ok {
t.Error("missing /api/packets path")
}
// Check tags exist
tags, ok := spec["tags"].([]interface{})
if !ok || len(tags) == 0 {
t.Error("missing or empty tags")
}
// Check security schemes
components, ok := spec["components"].(map[string]interface{})
if !ok {
t.Fatal("missing components")
}
schemes, ok := components["securitySchemes"].(map[string]interface{})
if !ok {
t.Fatal("missing securitySchemes")
}
if _, ok := schemes["ApiKeyAuth"]; !ok {
t.Error("missing ApiKeyAuth security scheme")
}
// Spec should NOT contain /api/spec or /api/docs (self-referencing)
if _, ok := paths["/api/spec"]; ok {
t.Error("/api/spec should not appear in the spec")
}
if _, ok := paths["/api/docs"]; ok {
t.Error("/api/docs should not appear in the spec")
}
}
func TestSwaggerUIEndpoint(t *testing.T) {
_, r := setupTestServer(t)
req := httptest.NewRequest("GET", "/api/docs", nil)
w := httptest.NewRecorder()
r.ServeHTTP(w, req)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d", w.Code)
}
ct := w.Header().Get("Content-Type")
if ct != "text/html; charset=utf-8" {
t.Errorf("unexpected content-type: %s", ct)
}
body := w.Body.String()
if len(body) < 100 {
t.Error("response too short for Swagger UI HTML")
}
if !strings.Contains(body, "swagger-ui") {
t.Error("response doesn't contain swagger-ui reference")
}
if !strings.Contains(body, "/api/spec") {
t.Error("response doesn't point to /api/spec")
}
}
func TestExtractPathParams(t *testing.T) {
tests := []struct {
path string
expect []string
}{
{"/api/nodes", nil},
{"/api/nodes/{pubkey}", []string{"pubkey"}},
{"/api/channels/{hash}/messages", []string{"hash"}},
}
for _, tt := range tests {
got := extractPathParams(tt.path)
if len(got) != len(tt.expect) {
t.Errorf("extractPathParams(%q) = %v, want %v", tt.path, got, tt.expect)
continue
}
for i := range got {
if got[i] != tt.expect[i] {
t.Errorf("extractPathParams(%q)[%d] = %q, want %q", tt.path, i, got[i], tt.expect[i])
}
}
}
}
+309
View File
@@ -0,0 +1,309 @@
package main
import (
"encoding/json"
"net/http/httptest"
"testing"
"time"
)
// ─── resolveWithContext unit tests ─────────────────────────────────────────────
func TestResolveWithContext_UniquePrefix(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1b2c3d4", Name: "Node-A", HasGPS: true, Lat: 1, Lon: 2},
})
ni, confidence, _ := pm.resolveWithContext("a1b2c3d4", nil, nil)
if ni == nil || ni.Name != "Node-A" {
t.Fatal("expected Node-A")
}
if confidence != "unique_prefix" {
t.Fatalf("expected unique_prefix, got %s", confidence)
}
}
func TestResolveWithContext_NoMatch(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1b2c3d4", Name: "Node-A"},
})
ni, confidence, _ := pm.resolveWithContext("ff", nil, nil)
if ni != nil {
t.Fatal("expected nil")
}
if confidence != "no_match" {
t.Fatalf("expected no_match, got %s", confidence)
}
}
func TestResolveWithContext_AffinityWins(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "Node-A1"},
{PublicKey: "a1bbbbbb", Name: "Node-A2"},
})
graph := NewNeighborGraph()
for i := 0; i < 100; i++ {
graph.upsertEdge("c0c0c0c0", "a1aaaaaa", "a1", "obs1", nil, time.Now())
}
ni, confidence, score := pm.resolveWithContext("a1", []string{"c0c0c0c0"}, graph)
if ni == nil || ni.Name != "Node-A1" {
t.Fatalf("expected Node-A1, got %v", ni)
}
if confidence != "neighbor_affinity" {
t.Fatalf("expected neighbor_affinity, got %s", confidence)
}
if score <= 0 {
t.Fatalf("expected positive score, got %f", score)
}
}
func TestResolveWithContext_AffinityTooClose_FallsToGeo(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "Node-A1", HasGPS: true, Lat: 10, Lon: 20},
{PublicKey: "a1bbbbbb", Name: "Node-A2", HasGPS: true, Lat: 11, Lon: 21},
{PublicKey: "c0c0c0c0", Name: "Ctx", HasGPS: true, Lat: 10.1, Lon: 20.1},
})
graph := NewNeighborGraph()
for i := 0; i < 50; i++ {
graph.upsertEdge("c0c0c0c0", "a1aaaaaa", "a1", "obs1", nil, time.Now())
graph.upsertEdge("c0c0c0c0", "a1bbbbbb", "a1", "obs1", nil, time.Now())
}
ni, confidence, _ := pm.resolveWithContext("a1", []string{"c0c0c0c0"}, graph)
if ni == nil {
t.Fatal("expected a result")
}
if confidence != "geo_proximity" {
t.Fatalf("expected geo_proximity, got %s", confidence)
}
if ni.Name != "Node-A1" {
t.Fatalf("expected Node-A1 (closer to context), got %s", ni.Name)
}
}
func TestResolveWithContext_GPSPreference(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "NoGPS"},
{PublicKey: "a1bbbbbb", Name: "HasGPS", HasGPS: true, Lat: 1, Lon: 2},
})
ni, confidence, _ := pm.resolveWithContext("a1", nil, nil)
if ni == nil || ni.Name != "HasGPS" {
t.Fatalf("expected HasGPS, got %v", ni)
}
if confidence != "gps_preference" {
t.Fatalf("expected gps_preference, got %s", confidence)
}
}
func TestResolveWithContext_FirstMatchFallback(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "First"},
{PublicKey: "a1bbbbbb", Name: "Second"},
})
ni, confidence, _ := pm.resolveWithContext("a1", nil, nil)
if ni == nil || ni.Name != "First" {
t.Fatalf("expected First, got %v", ni)
}
if confidence != "first_match" {
t.Fatalf("expected first_match, got %s", confidence)
}
}
func TestResolveWithContext_NilGraphFallsToGPS(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "NoGPS"},
{PublicKey: "a1bbbbbb", Name: "HasGPS", HasGPS: true, Lat: 1, Lon: 2},
})
ni, confidence, _ := pm.resolveWithContext("a1", []string{"someone"}, nil)
if ni == nil || ni.Name != "HasGPS" {
t.Fatalf("expected HasGPS, got %v", ni)
}
if confidence != "gps_preference" {
t.Fatalf("expected gps_preference, got %s", confidence)
}
}
func TestResolveWithContext_BackwardCompatResolve(t *testing.T) {
// Verify original resolve() still works unchanged
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "NoGPS"},
{PublicKey: "a1bbbbbb", Name: "HasGPS", HasGPS: true, Lat: 1, Lon: 2},
})
ni := pm.resolve("a1")
if ni == nil || ni.Name != "HasGPS" {
t.Fatalf("expected HasGPS from resolve(), got %v", ni)
}
}
// ─── geoDistApprox ─────────────────────────────────────────────────────────────
func TestGeoDistApprox_SamePoint(t *testing.T) {
d := geoDistApprox(37.0, -122.0, 37.0, -122.0)
if d != 0 {
t.Fatalf("expected 0, got %f", d)
}
}
func TestGeoDistApprox_Ordering(t *testing.T) {
d1 := geoDistApprox(37.0, -122.0, 37.01, -122.01)
d2 := geoDistApprox(37.0, -122.0, 38.0, -121.0)
if d1 >= d2 {
t.Fatal("closer point should have smaller distance")
}
}
// ─── handleResolveHops enhanced response (API tests) ───────────────────────────
func TestResolveHopsAPI_UniquePrefix(t *testing.T) {
srv, router := setupTestServer(t)
_ = srv
// Insert a unique node
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"ff11223344", "UniqueNode", 37.0, -122.0)
srv.store.InvalidateNodeCache()
req := httptest.NewRequest("GET", "/api/resolve-hops?hops=ff11223344", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
var result ResolveHopsResponse
if err := json.Unmarshal(rr.Body.Bytes(), &result); err != nil {
t.Fatalf("bad JSON: %v", err)
}
hr, ok := result.Resolved["ff11223344"]
if !ok {
t.Fatal("expected hop in resolved map")
}
if hr.Confidence != "unique_prefix" {
t.Fatalf("expected unique_prefix, got %s", hr.Confidence)
}
}
func TestResolveHopsAPI_AmbiguousNoContext(t *testing.T) {
srv, router := setupTestServer(t)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"ee1aaaaaaa", "Node-E1", 37.0, -122.0)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"ee1bbbbbbb", "Node-E2", 38.0, -121.0)
srv.store.InvalidateNodeCache()
req := httptest.NewRequest("GET", "/api/resolve-hops?hops=ee1", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
var result ResolveHopsResponse
json.Unmarshal(rr.Body.Bytes(), &result)
hr := result.Resolved["ee1"]
if hr == nil {
t.Fatal("expected hop in resolved map")
}
// With both candidates having GPS and no affinity context, the resolver
// picks the GPS-preferred candidate → confidence is "gps_preference".
if hr.Confidence != "gps_preference" {
t.Fatalf("expected gps_preference, got %s", hr.Confidence)
}
if len(hr.Candidates) != 2 {
t.Fatalf("expected 2 candidates, got %d", len(hr.Candidates))
}
for _, c := range hr.Candidates {
if c.AffinityScore != nil {
t.Fatal("expected nil affinity score without context")
}
}
}
func TestResolveHopsAPI_WithAffinityContext(t *testing.T) {
srv, router := setupTestServer(t)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"dd1aaaaaaa", "Node-D1", 37.0, -122.0)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"dd1bbbbbbb", "Node-D2", 38.0, -121.0)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"c0c0c0c0c0", "Context", 37.1, -122.1)
// Invalidate node cache so the PM includes newly inserted nodes.
srv.store.cacheMu.Lock()
srv.store.nodeCacheTime = time.Time{}
srv.store.cacheMu.Unlock()
// Build graph with strong affinity
graph := NewNeighborGraph()
for i := 0; i < 100; i++ {
graph.upsertEdge("c0c0c0c0c0", "dd1aaaaaaa", "dd1", "obs1", nil, time.Now())
}
graph.builtAt = time.Now()
srv.neighborMu.Lock()
srv.neighborGraph = graph
srv.neighborMu.Unlock()
req := httptest.NewRequest("GET", "/api/resolve-hops?hops=dd1&from_node=c0c0c0c0c0", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
var result ResolveHopsResponse
json.Unmarshal(rr.Body.Bytes(), &result)
hr := result.Resolved["dd1"]
if hr == nil {
t.Fatal("expected hop in resolved map")
}
if hr.Confidence != "neighbor_affinity" {
t.Fatalf("expected neighbor_affinity, got %s", hr.Confidence)
}
if hr.BestCandidate == nil || *hr.BestCandidate != "dd1aaaaaaa" {
t.Fatalf("expected bestCandidate dd1aaaaaaa, got %v", hr.BestCandidate)
}
// Verify affinity scores present
hasScore := false
for _, c := range hr.Candidates {
if c.AffinityScore != nil && *c.AffinityScore > 0 {
hasScore = true
}
}
if !hasScore {
t.Fatal("expected at least one candidate with affinity score")
}
}
func TestResolveHopsAPI_ResponseShape(t *testing.T) {
srv, router := setupTestServer(t)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"bb1aaaaaaa", "Node-B1", 37.0, -122.0)
req := httptest.NewRequest("GET", "/api/resolve-hops?hops=bb1a", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
var raw map[string]json.RawMessage
json.Unmarshal(rr.Body.Bytes(), &raw)
if _, ok := raw["resolved"]; !ok {
t.Fatal("missing 'resolved' key")
}
var resolved map[string]map[string]interface{}
json.Unmarshal(raw["resolved"], &resolved)
for _, hr := range resolved {
if _, ok := hr["confidence"]; !ok {
t.Error("missing 'confidence' field in HopResolution")
}
if _, ok := hr["candidates"]; !ok {
t.Error("missing 'candidates' field")
}
}
}
// ─── Helpers used only in this test file ───────────────────────────────────────
+475
View File
@@ -0,0 +1,475 @@
package main
// Lock ordering contract (MUST be followed everywhere):
//
// s.mu → s.lruMu (s.mu is the outer lock, lruMu is the inner lock)
//
// • Never acquire s.lruMu while holding s.mu.
// • fetchResolvedPathForObs takes lruMu independently — callers under s.mu
// must NOT call it directly; instead collect IDs under s.mu, release, then
// do LRU ops under lruMu separately.
// • The backfill path (backfillResolvedPathsAsync) follows this by collecting
// obsIDs to invalidate under s.mu, releasing it, then taking lruMu.
import (
"database/sql"
"hash/fnv"
"log"
"strings"
)
// resolvedPubkeyHash computes a fast 64-bit hash for membership index keying.
// Uses FNV-1a from stdlib — good distribution, no external dependency.
func resolvedPubkeyHash(pk string) uint64 {
h := fnv.New64a()
h.Write([]byte(strings.ToLower(pk)))
return h.Sum64()
}
// addToResolvedPubkeyIndex adds a txID under each resolved pubkey hash.
// Deduplicates both within a single call AND across calls — won't add the
// same (hash, txID) pair twice even when called multiple times for the same tx.
// Must be called under s.mu write lock.
func (s *PacketStore) addToResolvedPubkeyIndex(txID int, resolvedPubkeys []string) {
if !s.useResolvedPathIndex {
return
}
seen := make(map[uint64]bool, len(resolvedPubkeys))
for _, pk := range resolvedPubkeys {
if pk == "" {
continue
}
h := resolvedPubkeyHash(pk)
if seen[h] {
continue
}
seen[h] = true
// Cross-call dedup: check if (h, txID) already exists in forward index.
existing := s.resolvedPubkeyIndex[h]
alreadyPresent := false
for _, id := range existing {
if id == txID {
alreadyPresent = true
break
}
}
if alreadyPresent {
continue
}
s.resolvedPubkeyIndex[h] = append(existing, txID)
s.resolvedPubkeyReverse[txID] = append(s.resolvedPubkeyReverse[txID], h)
}
}
// removeFromResolvedPubkeyIndex removes all index entries for a txID using the reverse map.
// Must be called under s.mu write lock.
func (s *PacketStore) removeFromResolvedPubkeyIndex(txID int) {
if !s.useResolvedPathIndex {
return
}
hashes := s.resolvedPubkeyReverse[txID]
for _, h := range hashes {
list := s.resolvedPubkeyIndex[h]
// Remove ALL occurrences of txID (not just the first) to prevent orphans.
filtered := list[:0]
for _, id := range list {
if id != txID {
filtered = append(filtered, id)
}
}
if len(filtered) == 0 {
delete(s.resolvedPubkeyIndex, h)
} else {
s.resolvedPubkeyIndex[h] = filtered
}
}
delete(s.resolvedPubkeyReverse, txID)
}
// extractResolvedPubkeys extracts all non-nil, non-empty pubkeys from a resolved path.
func extractResolvedPubkeys(rp []*string) []string {
if len(rp) == 0 {
return nil
}
result := make([]string, 0, len(rp))
for _, p := range rp {
if p != nil && *p != "" {
result = append(result, *p)
}
}
return result
}
// mergeResolvedPubkeys collects unique non-empty pubkeys from multiple resolved paths.
func mergeResolvedPubkeys(paths ...[]*string) []string {
seen := make(map[string]bool)
var result []string
for _, rp := range paths {
for _, p := range rp {
if p != nil && *p != "" && !seen[*p] {
seen[*p] = true
result = append(result, *p)
}
}
}
return result
}
// nodeInResolvedPathViaIndex checks whether a transmission is associated with
// a target pubkey using the membership index + collision-safety SQL check.
// Must be called under s.mu RLock at minimum.
func (s *PacketStore) nodeInResolvedPathViaIndex(tx *StoreTx, targetPK string) bool {
if !s.useResolvedPathIndex {
// Flag off: can't disambiguate, keep candidate (conservative)
return true
}
// If this tx has no indexed pubkeys at all, we can't disambiguate —
// keep the candidate (same as old behavior for NULL resolved_path).
if _, hasReverse := s.resolvedPubkeyReverse[tx.ID]; !hasReverse {
return true
}
h := resolvedPubkeyHash(targetPK)
txIDs := s.resolvedPubkeyIndex[h]
// Check if this tx's ID is in the candidate list
for _, id := range txIDs {
if id == tx.ID {
// Found in index. Collision-safety: verify with SQL.
if s.db != nil && s.db.conn != nil {
return s.confirmResolvedPathContains(tx.ID, targetPK)
}
return true // no DB, trust the index
}
}
return false
}
// confirmResolvedPathContains verifies an exact pubkey match in resolved_path
// via SQL. This is the collision-safety fallback for the membership index.
func (s *PacketStore) confirmResolvedPathContains(txID int, pubkey string) bool {
if s.db == nil || s.db.conn == nil {
return true
}
// Use INSTR with surrounding quotes for exact match — avoids LIKE escape issues.
// resolved_path format: ["pubkey1","pubkey2",...]
needle := `"` + strings.ToLower(pubkey) + `"`
var count int
err := s.db.conn.QueryRow(
`SELECT COUNT(*) FROM observations WHERE transmission_id = ? AND INSTR(LOWER(resolved_path), ?) > 0`,
txID, needle,
).Scan(&count)
if err != nil {
return true // on error, keep the candidate
}
return count > 0
}
// fetchResolvedPathsForTx fetches resolved_path from SQLite for all observations
// of a transmission. Used for on-demand API responses and eviction cleanup.
func (s *PacketStore) fetchResolvedPathsForTx(txID int) map[int][]*string {
if s.db == nil || s.db.conn == nil {
return nil
}
rows, err := s.db.conn.Query(
`SELECT id, resolved_path FROM observations WHERE transmission_id = ? AND resolved_path IS NOT NULL`,
txID,
)
if err != nil {
return nil
}
defer rows.Close()
result := make(map[int][]*string)
for rows.Next() {
var obsID int
var rpJSON sql.NullString
if err := rows.Scan(&obsID, &rpJSON); err != nil {
continue
}
if rpJSON.Valid && rpJSON.String != "" {
result[obsID] = unmarshalResolvedPath(rpJSON.String)
}
}
return result
}
// fetchResolvedPathForObs fetches resolved_path for a single observation,
// using the LRU cache.
func (s *PacketStore) fetchResolvedPathForObs(obsID int) []*string {
if s.db == nil || s.db.conn == nil {
return nil
}
// Check LRU cache first
s.lruMu.RLock()
if s.apiResolvedPathLRU != nil {
if entry, ok := s.apiResolvedPathLRU[obsID]; ok {
s.lruMu.RUnlock()
return entry
}
}
s.lruMu.RUnlock()
var rpJSON sql.NullString
err := s.db.conn.QueryRow(
`SELECT resolved_path FROM observations WHERE id = ?`, obsID,
).Scan(&rpJSON)
if err != nil || !rpJSON.Valid {
return nil
}
rp := unmarshalResolvedPath(rpJSON.String)
// Store in LRU
s.lruMu.Lock()
s.lruPut(obsID, rp)
s.lruMu.Unlock()
return rp
}
// fetchResolvedPathForTxBest returns the best observation's resolved_path for a tx.
//
// "Best" = the longest path_json among observations that actually have a stored
// resolved_path. Earlier versions picked the longest-path obs unconditionally
// and queried SQL for that single ID — if the longest-path obs had NULL
// resolved_path while a shorter sibling had one, the call returned nil and
// callers (e.g. /api/nodes/{pk}/health.recentPackets) lost the field. Fixes
// #810 by checking all observations and falling back to the longest sibling
// that has a stored path.
func (s *PacketStore) fetchResolvedPathForTxBest(tx *StoreTx) []*string {
if tx == nil || len(tx.Observations) == 0 {
return nil
}
// Fast path: try the longest-path obs first via the LRU/SQL helper.
longest := tx.Observations[0]
longestLen := pathLen(longest.PathJSON)
for _, obs := range tx.Observations[1:] {
if l := pathLen(obs.PathJSON); l > longestLen {
longest = obs
longestLen = l
}
}
if rp := s.fetchResolvedPathForObs(longest.ID); rp != nil {
return rp
}
// Fallback: longest-path obs has no stored resolved_path. Query all
// observations for this tx and pick the one with the longest path_json
// that actually has a stored resolved_path.
rpMap := s.fetchResolvedPathsForTx(tx.ID)
if len(rpMap) == 0 {
return nil
}
var bestRP []*string
bestObsID := 0
bestLen := -1
for _, obs := range tx.Observations {
rp, ok := rpMap[obs.ID]
if !ok || rp == nil {
continue
}
if l := pathLen(obs.PathJSON); l > bestLen {
bestLen = l
bestRP = rp
bestObsID = obs.ID
}
}
// Populate LRU so repeat lookups for this tx don't re-issue the multi-row
// SQL fallback (e.g. dashboard polling /api/nodes/{pk}/health).
if bestRP != nil && bestObsID != 0 {
s.lruMu.Lock()
s.lruPut(bestObsID, bestRP)
s.lruMu.Unlock()
}
return bestRP
}
// --- Simple LRU cache for resolved paths ---
const lruMaxSize = 10000
// lruPut adds an entry. Must be called under s.lruMu write lock.
func (s *PacketStore) lruPut(obsID int, rp []*string) {
if s.apiResolvedPathLRU == nil {
return
}
if _, exists := s.apiResolvedPathLRU[obsID]; exists {
return
}
// Compact lruOrder if stale entries exceed 50% of capacity.
// This prevents effective capacity degradation after bulk deletions.
if len(s.lruOrder) >= lruMaxSize && len(s.apiResolvedPathLRU) < lruMaxSize/2 {
compacted := make([]int, 0, len(s.apiResolvedPathLRU))
for _, id := range s.lruOrder {
if _, ok := s.apiResolvedPathLRU[id]; ok {
compacted = append(compacted, id)
}
}
s.lruOrder = compacted
}
if len(s.lruOrder) >= lruMaxSize {
// Evict oldest, skipping stale entries
for len(s.lruOrder) > 0 {
evictID := s.lruOrder[0]
s.lruOrder = s.lruOrder[1:]
if _, ok := s.apiResolvedPathLRU[evictID]; ok {
delete(s.apiResolvedPathLRU, evictID)
break
}
// stale entry — skip and continue
}
}
s.apiResolvedPathLRU[obsID] = rp
s.lruOrder = append(s.lruOrder, obsID)
}
// lruDelete removes an entry. Must be called under s.lruMu write lock.
func (s *PacketStore) lruDelete(obsID int) {
if s.apiResolvedPathLRU == nil {
return
}
delete(s.apiResolvedPathLRU, obsID)
// Don't scan lruOrder — eviction handles stale entries naturally.
}
// resolvedPubkeysForEvictionBatch fetches resolved pubkeys for multiple txIDs
// from SQL in a single batched query. Returns a map from txID to unique pubkeys.
// MUST be called WITHOUT holding s.mu — this is the whole point of the batch approach.
// Chunks queries to stay under SQLite's 500-parameter limit.
func (s *PacketStore) resolvedPubkeysForEvictionBatch(txIDs []int) map[int][]string {
result := make(map[int][]string, len(txIDs))
if len(txIDs) == 0 || s.db == nil || s.db.conn == nil {
return result
}
const chunkSize = 499 // SQLite SQLITE_MAX_VARIABLE_NUMBER default is 999; stay well under
for start := 0; start < len(txIDs); start += chunkSize {
end := start + chunkSize
if end > len(txIDs) {
end = len(txIDs)
}
chunk := txIDs[start:end]
// Build query with placeholders
placeholders := make([]byte, 0, len(chunk)*2)
args := make([]interface{}, len(chunk))
for i, id := range chunk {
if i > 0 {
placeholders = append(placeholders, ',')
}
placeholders = append(placeholders, '?')
args[i] = id
}
query := "SELECT transmission_id, resolved_path FROM observations WHERE transmission_id IN (" +
string(placeholders) + ") AND resolved_path IS NOT NULL"
rows, err := s.db.conn.Query(query, args...)
if err != nil {
continue
}
for rows.Next() {
var txID int
var rpJSON sql.NullString
if err := rows.Scan(&txID, &rpJSON); err != nil {
continue
}
if !rpJSON.Valid || rpJSON.String == "" {
continue
}
rp := unmarshalResolvedPath(rpJSON.String)
for _, p := range rp {
if p != nil && *p != "" {
result[txID] = append(result[txID], *p)
}
}
}
rows.Close()
}
// Deduplicate per-txID
for txID, pks := range result {
seen := make(map[string]bool, len(pks))
deduped := pks[:0]
for _, pk := range pks {
if !seen[pk] {
seen[pk] = true
deduped = append(deduped, pk)
}
}
result[txID] = deduped
}
return result
}
// initResolvedPathIndex initializes the resolved path index data structures.
func (s *PacketStore) initResolvedPathIndex() {
s.resolvedPubkeyIndex = make(map[uint64][]int, 4096)
s.resolvedPubkeyReverse = make(map[int][]uint64, 4096)
s.apiResolvedPathLRU = make(map[int][]*string, lruMaxSize)
s.lruOrder = make([]int, 0, lruMaxSize)
}
// CompactResolvedPubkeyIndex reclaims memory from the resolved pubkey index maps
// after eviction. It removes empty forward-index entries (shouldn't exist if
// removeFromResolvedPubkeyIndex is correct, but defense in depth) and clips
// oversized slice backing arrays where cap > 2*len.
// Must be called under s.mu write lock.
func (s *PacketStore) CompactResolvedPubkeyIndex() {
if !s.useResolvedPathIndex {
return
}
for h, ids := range s.resolvedPubkeyIndex {
if len(ids) == 0 {
delete(s.resolvedPubkeyIndex, h)
continue
}
// Clip oversized backing arrays: if cap > 2*len, reallocate.
if cap(ids) > 2*len(ids)+8 {
clipped := make([]int, len(ids))
copy(clipped, ids)
s.resolvedPubkeyIndex[h] = clipped
}
}
for txID, hashes := range s.resolvedPubkeyReverse {
if len(hashes) == 0 {
delete(s.resolvedPubkeyReverse, txID)
continue
}
if cap(hashes) > 2*len(hashes)+8 {
clipped := make([]uint64, len(hashes))
copy(clipped, hashes)
s.resolvedPubkeyReverse[txID] = clipped
}
}
}
// defaultMaxResolvedPubkeyIndexEntries is the default hard cap for the forward
// index. When exceeded, a warning is logged. No auto-eviction — that's the
// eviction ticker's job.
const defaultMaxResolvedPubkeyIndexEntries = 5_000_000
// CheckResolvedPubkeyIndexSize logs a warning if the resolved pubkey forward
// index exceeds the configured maximum entries. Must be called under s.mu
// read lock at minimum.
func (s *PacketStore) CheckResolvedPubkeyIndexSize() {
if !s.useResolvedPathIndex {
return
}
maxEntries := s.maxResolvedPubkeyIndexEntries
if maxEntries <= 0 {
maxEntries = defaultMaxResolvedPubkeyIndexEntries
}
fwdLen := len(s.resolvedPubkeyIndex)
revLen := len(s.resolvedPubkeyReverse)
if fwdLen > maxEntries || revLen > maxEntries {
log.Printf("[store] WARNING: resolvedPubkeyIndex size exceeds limit — forward=%d reverse=%d limit=%d",
fwdLen, revLen, maxEntries)
}
}
File diff suppressed because it is too large Load Diff
+882 -88
View File
File diff suppressed because it is too large Load Diff
+966 -21
View File
File diff suppressed because it is too large Load Diff
+95
View File
@@ -0,0 +1,95 @@
package main
import (
"encoding/json"
"net/http/httptest"
"strings"
"testing"
)
// TestStatsMemoryFields verifies that /api/stats exposes the new memory
// breakdown introduced for issue #832: storeDataMB, processRSSMB,
// goHeapInuseMB, goSysMB, plus the deprecated trackedMB alias.
//
// We assert presence, type, sign, and ordering invariants — but NOT
// "RSS within X% of true RSS" because that is flaky in CI under cgo,
// containerization, and shared-runner load.
func TestStatsMemoryFields(t *testing.T) {
_, router := setupTestServer(t)
req := httptest.NewRequest("GET", "/api/stats", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d", w.Code)
}
var body map[string]interface{}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatalf("json decode: %v", err)
}
required := []string{"trackedMB", "storeDataMB", "processRSSMB", "goHeapInuseMB", "goSysMB"}
values := make(map[string]float64, len(required))
for _, k := range required {
v, ok := body[k]
if !ok {
t.Fatalf("missing field %q in /api/stats response", k)
}
f, ok := v.(float64)
if !ok {
t.Fatalf("field %q is %T, expected float64", k, v)
}
if f < 0 {
t.Errorf("field %q is negative: %v", k, f)
}
values[k] = f
}
// trackedMB is a deprecated alias for storeDataMB; they must match.
if values["trackedMB"] != values["storeDataMB"] {
t.Errorf("trackedMB (%v) != storeDataMB (%v); they must remain aliased",
values["trackedMB"], values["storeDataMB"])
}
// Ordering invariants. goSys is the runtime's view of total OS memory;
// HeapInuse is a subset of it. storeData is a subset of HeapInuse.
// processRSS may be 0 in environments without /proc — treat 0 as
// "unknown" rather than a failure.
if values["goHeapInuseMB"] > values["goSysMB"]+0.5 {
t.Errorf("invariant violated: goHeapInuseMB (%v) > goSysMB (%v)",
values["goHeapInuseMB"], values["goSysMB"])
}
if values["storeDataMB"] > values["goHeapInuseMB"]+0.5 && values["storeDataMB"] > 0 {
// In the test fixture storeDataMB is typically 0 (no packets in
// store); only enforce the bound when both are nonzero.
t.Errorf("invariant violated: storeDataMB (%v) > goHeapInuseMB (%v)",
values["storeDataMB"], values["goHeapInuseMB"])
}
if values["processRSSMB"] > 0 && values["goSysMB"] > 0 {
// goSys can briefly exceed RSS if pages are reserved-but-not-touched,
// so allow some slack.
if values["goSysMB"] > values["processRSSMB"]*4 {
t.Errorf("suspicious: goSysMB (%v) >> processRSSMB (%v)",
values["goSysMB"], values["processRSSMB"])
}
}
}
// TestStatsMemoryFieldsRawJSON spot-checks that the JSON wire format uses
// the documented camelCase names (no accidental rename through struct tags).
func TestStatsMemoryFieldsRawJSON(t *testing.T) {
_, router := setupTestServer(t)
req := httptest.NewRequest("GET", "/api/stats", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
body := w.Body.String()
for _, key := range []string{
`"trackedMB":`, `"storeDataMB":`,
`"processRSSMB":`, `"goHeapInuseMB":`, `"goSysMB":`,
} {
if !strings.Contains(body, key) {
t.Errorf("missing %s in raw response: %s", key, body)
}
}
}
+2299 -502
View File
File diff suppressed because it is too large Load Diff
+116
View File
@@ -0,0 +1,116 @@
package main
import (
"testing"
)
func f64(v float64) *float64 { return &v }
func TestDedupeTopHopsByPair(t *testing.T) {
hops := []distHopRecord{
{FromPk: "AAA", ToPk: "BBB", FromName: "A", ToName: "B", Dist: 100, Type: "R↔R", SNR: f64(5.0), Hash: "h1", Timestamp: "t1"},
{FromPk: "AAA", ToPk: "BBB", FromName: "A", ToName: "B", Dist: 90, Type: "R↔R", SNR: f64(8.0), Hash: "h2", Timestamp: "t2"},
{FromPk: "BBB", ToPk: "AAA", FromName: "B", ToName: "A", Dist: 80, Type: "R↔R", SNR: f64(3.0), Hash: "h3", Timestamp: "t3"},
{FromPk: "AAA", ToPk: "BBB", FromName: "A", ToName: "B", Dist: 70, Type: "R↔R", SNR: f64(6.0), Hash: "h4", Timestamp: "t4"},
{FromPk: "AAA", ToPk: "BBB", FromName: "A", ToName: "B", Dist: 60, Type: "R↔R", SNR: f64(4.0), Hash: "h5", Timestamp: "t5"},
{FromPk: "CCC", ToPk: "DDD", FromName: "C", ToName: "D", Dist: 50, Type: "C↔R", SNR: f64(7.0), Hash: "h6", Timestamp: "t6"},
}
result := dedupeHopsByPair(hops, 20)
if len(result) != 2 {
t.Fatalf("expected 2 entries, got %d", len(result))
}
// First entry: A↔B pair, max distance = 100, obsCount = 5
ab := result[0]
if ab["dist"].(float64) != 100 {
t.Errorf("expected dist 100, got %v", ab["dist"])
}
if ab["obsCount"].(int) != 5 {
t.Errorf("expected obsCount 5, got %v", ab["obsCount"])
}
if ab["hash"].(string) != "h1" {
t.Errorf("expected hash h1 (from max-dist record), got %v", ab["hash"])
}
if ab["bestSnr"].(float64) != 8.0 {
t.Errorf("expected bestSnr 8.0, got %v", ab["bestSnr"])
}
// medianSnr of [3,4,5,6,8] = 5.0
if ab["medianSnr"].(float64) != 5.0 {
t.Errorf("expected medianSnr 5.0, got %v", ab["medianSnr"])
}
// Second entry: C↔D pair
cd := result[1]
if cd["dist"].(float64) != 50 {
t.Errorf("expected dist 50, got %v", cd["dist"])
}
if cd["obsCount"].(int) != 1 {
t.Errorf("expected obsCount 1, got %v", cd["obsCount"])
}
}
func TestDedupeTopHopsReversePairMerges(t *testing.T) {
hops := []distHopRecord{
{FromPk: "BBB", ToPk: "AAA", FromName: "B", ToName: "A", Dist: 50, Type: "R↔R", Hash: "h1"},
{FromPk: "AAA", ToPk: "BBB", FromName: "A", ToName: "B", Dist: 80, Type: "R↔R", Hash: "h2"},
}
result := dedupeHopsByPair(hops, 20)
if len(result) != 1 {
t.Fatalf("expected 1 entry, got %d", len(result))
}
if result[0]["obsCount"].(int) != 2 {
t.Errorf("expected obsCount 2, got %v", result[0]["obsCount"])
}
if result[0]["dist"].(float64) != 80 {
t.Errorf("expected dist 80, got %v", result[0]["dist"])
}
}
func TestDedupeTopHopsNilSNR(t *testing.T) {
hops := []distHopRecord{
{FromPk: "AAA", ToPk: "BBB", FromName: "A", ToName: "B", Dist: 100, Type: "R↔R", SNR: nil, Hash: "h1"},
{FromPk: "AAA", ToPk: "BBB", FromName: "A", ToName: "B", Dist: 90, Type: "R↔R", SNR: nil, Hash: "h2"},
}
result := dedupeHopsByPair(hops, 20)
if len(result) != 1 {
t.Fatalf("expected 1 entry, got %d", len(result))
}
if result[0]["bestSnr"] != nil {
t.Errorf("expected bestSnr nil, got %v", result[0]["bestSnr"])
}
if result[0]["medianSnr"] != nil {
t.Errorf("expected medianSnr nil, got %v", result[0]["medianSnr"])
}
}
func TestDedupeTopHopsLimit(t *testing.T) {
// Generate 25 unique pairs, verify limit=20 caps output
hops := make([]distHopRecord, 25)
for i := range hops {
hops[i] = distHopRecord{
FromPk: "A", ToPk: string(rune('a' + i)),
Dist: float64(i), Type: "R↔R", Hash: "h",
}
}
result := dedupeHopsByPair(hops, 20)
if len(result) != 20 {
t.Errorf("expected 20 entries, got %d", len(result))
}
}
func TestDedupeTopHopsEvenMedian(t *testing.T) {
// Even count: median = avg of two middle values
hops := []distHopRecord{
{FromPk: "A", ToPk: "B", Dist: 10, Type: "R↔R", SNR: f64(2.0), Hash: "h1"},
{FromPk: "A", ToPk: "B", Dist: 20, Type: "R↔R", SNR: f64(4.0), Hash: "h2"},
{FromPk: "A", ToPk: "B", Dist: 30, Type: "R↔R", SNR: f64(6.0), Hash: "h3"},
{FromPk: "A", ToPk: "B", Dist: 40, Type: "R↔R", SNR: f64(8.0), Hash: "h4"},
}
result := dedupeHopsByPair(hops, 20)
// sorted SNR: [2,4,6,8], median = (4+6)/2 = 5.0
if result[0]["medianSnr"].(float64) != 5.0 {
t.Errorf("expected medianSnr 5.0, got %v", result[0]["medianSnr"])
}
}
+13 -4
View File
@@ -42,14 +42,20 @@
"type": {
"type": "string"
},
"snr": {
"type": "number"
},
"hash": {
"type": "string"
},
"timestamp": {
"type": "string"
},
"bestSnr": {
"type": "number"
},
"medianSnr": {
"type": "number"
},
"obsCount": {
"type": "number"
}
}
}
@@ -916,6 +922,9 @@
},
"estimatedMB": {
"type": "number"
},
"trackedMB": {
"type": "number"
}
}
},
@@ -1577,4 +1586,4 @@
}
}
}
}
}
+126
View File
@@ -0,0 +1,126 @@
package main
import (
"database/sql"
"testing"
"time"
_ "modernc.org/sqlite"
)
func TestTouchNodeLastSeen_UpdatesDB(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
// Insert a node with no last_seen
db.conn.Exec("INSERT INTO nodes (public_key, name, role) VALUES (?, ?, ?)", "abc123", "relay1", "REPEATER")
err := db.TouchNodeLastSeen("abc123", "2026-04-12T04:00:00Z")
if err != nil {
t.Fatalf("TouchNodeLastSeen returned error: %v", err)
}
var lastSeen sql.NullString
db.conn.QueryRow("SELECT last_seen FROM nodes WHERE public_key = ?", "abc123").Scan(&lastSeen)
if !lastSeen.Valid || lastSeen.String != "2026-04-12T04:00:00Z" {
t.Fatalf("expected last_seen=2026-04-12T04:00:00Z, got %v", lastSeen)
}
}
func TestTouchNodeLastSeen_DoesNotGoBackwards(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"abc123", "relay1", "REPEATER", "2026-04-12T05:00:00Z")
// Try to set an older timestamp
err := db.TouchNodeLastSeen("abc123", "2026-04-12T04:00:00Z")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
var lastSeen string
db.conn.QueryRow("SELECT last_seen FROM nodes WHERE public_key = ?", "abc123").Scan(&lastSeen)
if lastSeen != "2026-04-12T05:00:00Z" {
t.Fatalf("last_seen went backwards: got %s", lastSeen)
}
}
func TestTouchNodeLastSeen_NonExistentNode(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
// Should not error for non-existent node
err := db.TouchNodeLastSeen("nonexistent", "2026-04-12T04:00:00Z")
if err != nil {
t.Fatalf("unexpected error for non-existent node: %v", err)
}
}
func TestTouchRelayLastSeen_Debouncing(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role) VALUES (?, ?, ?)", "relay1", "R1", "REPEATER")
s := &PacketStore{
db: db,
lastSeenTouched: make(map[string]time.Time),
}
// After #800, touchRelayLastSeen takes a []string of pubkeys (from decode-window)
pks := []string{"relay1"}
now := time.Now()
s.touchRelayLastSeen(pks, now)
// Verify it was written
var lastSeen sql.NullString
db.conn.QueryRow("SELECT last_seen FROM nodes WHERE public_key = ?", "relay1").Scan(&lastSeen)
if !lastSeen.Valid {
t.Fatal("expected last_seen to be set after first touch")
}
// Reset last_seen to check debounce prevents second write
db.conn.Exec("UPDATE nodes SET last_seen = NULL WHERE public_key = ?", "relay1")
// Call again within 5 minutes — should be debounced (no write)
s.touchRelayLastSeen(pks, now.Add(2*time.Minute))
db.conn.QueryRow("SELECT last_seen FROM nodes WHERE public_key = ?", "relay1").Scan(&lastSeen)
if lastSeen.Valid {
t.Fatal("expected debounce to prevent second write within 5 minutes")
}
// Call after 5 minutes — should write again
s.touchRelayLastSeen(pks, now.Add(6*time.Minute))
db.conn.QueryRow("SELECT last_seen FROM nodes WHERE public_key = ?", "relay1").Scan(&lastSeen)
if !lastSeen.Valid {
t.Fatal("expected write after debounce interval expired")
}
}
func TestTouchRelayLastSeen_SkipsEmptyPubkeys(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
s := &PacketStore{
db: db,
lastSeenTouched: make(map[string]time.Time),
}
// Empty pubkeys — should not panic or error
s.touchRelayLastSeen([]string{}, time.Now())
s.touchRelayLastSeen(nil, time.Now())
}
func TestTouchRelayLastSeen_NilDB(t *testing.T) {
s := &PacketStore{
db: nil,
lastSeenTouched: make(map[string]time.Time),
}
// Should not panic with nil db
s.touchRelayLastSeen([]string{"abc"}, time.Now())
}
+166
View File
@@ -0,0 +1,166 @@
package main
import (
"testing"
"time"
)
// TestEstimateStoreTxBytes_ReasonableValues verifies the estimate function
// returns reasonable values for different packet sizes.
func TestEstimateStoreTxBytes_ReasonableValues(t *testing.T) {
tx := &StoreTx{
Hash: "abcdef1234567890",
RawHex: "deadbeef",
DecodedJSON: `{"type":"GRP_TXT"}`,
PathJSON: `["hop1","hop2","hop3"]`,
parsedPath: []string{"hop1", "hop2", "hop3"},
pathParsed: true,
}
got := estimateStoreTxBytes(tx)
// Should be at least base (384) + maps (200) + indexes + path/subpath costs
if got < 700 {
t.Errorf("estimate too low for 3-hop tx: %d", got)
}
if got > 5000 {
t.Errorf("estimate unreasonably high for 3-hop tx: %d", got)
}
}
// TestEstimateStoreTxBytes_ManyHopsSubpaths verifies that packets with many
// hops estimate significantly more due to O(path²) subpath index entries.
func TestEstimateStoreTxBytes_ManyHopsSubpaths(t *testing.T) {
tx2 := &StoreTx{
Hash: "aabb",
parsedPath: []string{"a", "b"},
pathParsed: true,
}
tx10 := &StoreTx{
Hash: "aabb",
parsedPath: []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j"},
pathParsed: true,
}
est2 := estimateStoreTxBytes(tx2)
est10 := estimateStoreTxBytes(tx10)
// 10 hops → 45 subpath combos × 40 = 1800 bytes just for subpaths
if est10 <= est2 {
t.Errorf("10-hop (%d) should estimate more than 2-hop (%d)", est10, est2)
}
if est10 < est2+1500 {
t.Errorf("10-hop (%d) should estimate at least 1500 more than 2-hop (%d)", est10, est2)
}
}
// TestEstimateStoreObsBytes_AfterRefactor verifies that after #800 refactor,
// observations no longer have ResolvedPath overhead in their estimate.
func TestEstimateStoreObsBytes_AfterRefactor(t *testing.T) {
obs := &StoreObs{
ObserverID: "obs1",
PathJSON: `["a","b"]`,
}
est := estimateStoreObsBytes(obs)
if est <= 0 {
t.Errorf("estimate should be positive, got %d", est)
}
// After #800, all obs estimates should be the same (no RP field variation)
obs2 := &StoreObs{
ObserverID: "obs1",
PathJSON: `["a","b"]`,
}
est2 := estimateStoreObsBytes(obs2)
if est != est2 {
t.Errorf("estimates should be equal after #800 (no RP field), got %d vs %d", est, est2)
}
}
// TestEstimateStoreObsBytes_ManyObservations verifies that 15 observations
// estimate significantly more than 1.
func TestEstimateStoreObsBytes_ManyObservations(t *testing.T) {
est1 := estimateStoreObsBytes(&StoreObs{ObserverID: "a", PathJSON: `["x"]`})
est15 := int64(0)
for i := 0; i < 15; i++ {
est15 += estimateStoreObsBytes(&StoreObs{ObserverID: "a", PathJSON: `["x"]`})
}
if est15 <= est1*10 {
t.Errorf("15 obs total (%d) should be >10x single obs (%d)", est15, est1)
}
}
// TestTrackedBytesMatchesSumAfterInsert verifies that trackedBytes equals the
// sum of individual estimates after inserting packets via makeTestStore.
func TestTrackedBytesMatchesSumAfterInsert(t *testing.T) {
store := makeTestStore(20, time.Now().Add(-2*time.Hour), 5)
// Manually compute trackedBytes as sum of estimates
var expectedSum int64
for _, tx := range store.packets {
expectedSum += estimateStoreTxBytes(tx)
for _, obs := range tx.Observations {
expectedSum += estimateStoreObsBytes(obs)
}
}
if store.trackedBytes != expectedSum {
t.Errorf("trackedBytes=%d, expected sum=%d", store.trackedBytes, expectedSum)
}
}
// TestEvictionTriggersWithImprovedEstimates verifies that eviction triggers
// at the right point with the improved (higher) estimates.
func TestEvictionTriggersWithImprovedEstimates(t *testing.T) {
store := makeTestStore(100, time.Now().Add(-10*time.Hour), 5)
// trackedBytes for 100 packets is small — artificially set maxMemoryMB
// so highWatermark is just below trackedBytes to trigger eviction.
highWatermarkBytes := store.trackedBytes - 1000
if highWatermarkBytes < 1 {
highWatermarkBytes = 1
}
// maxMemoryMB * 1048576 = highWatermark, so maxMemoryMB = ceil(highWatermarkBytes / 1048576)
// But that'll be 0 for small values. Instead, directly set trackedBytes high.
store.trackedBytes = 6 * 1048576 // 6MB
store.maxMemoryMB = 3 // 3MB limit
beforeCount := len(store.packets)
store.RunEviction()
afterCount := len(store.packets)
if afterCount >= beforeCount {
t.Errorf("expected eviction to remove packets: before=%d, after=%d, trackedBytes=%d, maxMB=%d",
beforeCount, afterCount, store.trackedBytes, store.maxMemoryMB)
}
// trackedBytes should have decreased
if store.trackedBytes >= 6*1048576 {
t.Errorf("trackedBytes should have decreased after eviction")
}
}
// BenchmarkEstimateStoreTxBytes verifies the estimate function is fast.
func BenchmarkEstimateStoreTxBytes(b *testing.B) {
tx := &StoreTx{
Hash: "abcdef1234567890",
RawHex: "deadbeefdeadbeef",
DecodedJSON: `{"type":"GRP_TXT","payload":"hello"}`,
PathJSON: `["hop1","hop2","hop3","hop4","hop5"]`,
parsedPath: []string{"hop1", "hop2", "hop3", "hop4", "hop5"},
pathParsed: true,
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
estimateStoreTxBytes(tx)
}
}
// BenchmarkEstimateStoreObsBytes verifies the obs estimate function is fast.
func BenchmarkEstimateStoreObsBytes(b *testing.B) {
obs := &StoreObs{
ObserverID: "observer1234",
PathJSON: `["a","b","c"]`,
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
estimateStoreObsBytes(obs)
}
}
+43 -11
View File
@@ -68,6 +68,26 @@ type StatsResponse struct {
Commit string `json:"commit"`
BuildTime string `json:"buildTime"`
Counts RoleCounts `json:"counts"`
Backfilling bool `json:"backfilling"`
BackfillProgress float64 `json:"backfillProgress"`
SignatureDrops int64 `json:"signatureDrops,omitempty"`
HashMigrationComplete bool `json:"hashMigrationComplete"`
// Memory accounting (issue #832). All values in MB.
//
// StoreDataMB ("trackedMB" historically) is the in-store packet byte
// estimate — useful packet bytes only. Subset of HeapInuse. Used as
// the eviction watermark input. NOT a proxy for RSS; ops dashboards
// should prefer ProcessRSSMB for capacity decisions.
//
// Old field name TrackedMB is retained for backward compatibility
// with pre-v3.6 consumers; it carries the same value as StoreDataMB
// and is deprecated.
TrackedMB float64 `json:"trackedMB"` // deprecated alias for storeDataMB
StoreDataMB float64 `json:"storeDataMB"` // in-store packet bytes (subset of heap)
ProcessRSSMB float64 `json:"processRSSMB"` // process RSS from /proc (Linux) or runtime.Sys fallback
GoHeapInuseMB float64 `json:"goHeapInuseMB"` // runtime.MemStats.HeapInuse
GoSysMB float64 `json:"goSysMB"` // runtime.MemStats.Sys (total Go-managed)
}
// ─── Health ────────────────────────────────────────────────────────────────────
@@ -113,6 +133,7 @@ type WebSocketStatsResp struct {
type HealthPacketStoreStats struct {
Packets int `json:"packets"`
EstimatedMB float64 `json:"estimatedMB"`
TrackedMB float64 `json:"trackedMB"`
}
type SlowQuery struct {
@@ -172,6 +193,8 @@ type PerfPacketStoreStats struct {
SqliteOnly bool `json:"sqliteOnly"`
MaxPackets int `json:"maxPackets"`
EstimatedMB float64 `json:"estimatedMB"`
TrackedMB float64 `json:"trackedMB"`
AvgBytesPerPacket int64 `json:"avgBytesPerPacket"`
MaxMB int `json:"maxMB"`
Indexes PacketStoreIndexes `json:"indexes"`
}
@@ -254,6 +277,9 @@ type ObservationResp struct {
SNR interface{} `json:"snr"`
RSSI interface{} `json:"rssi"`
PathJSON interface{} `json:"path_json"`
ResolvedPath interface{} `json:"resolved_path,omitempty"`
Direction interface{} `json:"direction,omitempty"`
RawHex interface{} `json:"raw_hex,omitempty"`
Timestamp interface{} `json:"timestamp"`
}
@@ -289,7 +315,6 @@ type PacketTimestampsResponse struct {
type PacketDetailResponse struct {
Packet interface{} `json:"packet"`
Path []interface{} `json:"path"`
Breakdown interface{} `json:"breakdown"`
ObservationCount int `json:"observation_count"`
Observations []ObservationResp `json:"observations,omitempty"`
}
@@ -462,6 +487,7 @@ type NodeAnalyticsResponse struct {
PeerInteractions []PeerInteraction `json:"peerInteractions"`
UptimeHeatmap []HeatmapCell `json:"uptimeHeatmap"`
ComputedStats ComputedNodeStats `json:"computedStats"`
ClockSkew *NodeClockSkew `json:"clockSkew,omitempty"`
}
// ─── Analytics — RF ────────────────────────────────────────────────────────────
@@ -654,7 +680,9 @@ type DistanceHop struct {
ToPk string `json:"toPk"`
Dist float64 `json:"dist"`
Type string `json:"type"`
SNR interface{} `json:"snr"`
BestSnr interface{} `json:"bestSnr"`
MedianSnr interface{} `json:"medianSnr"`
ObsCount int `json:"obsCount"`
Hash string `json:"hash"`
Timestamp string `json:"timestamp"`
}
@@ -873,18 +901,21 @@ type TraceResponse struct {
// ─── Resolve Hops ──────────────────────────────────────────────────────────────
type HopCandidate struct {
Name interface{} `json:"name"`
Pubkey string `json:"pubkey"`
Lat interface{} `json:"lat"`
Lon interface{} `json:"lon"`
Name interface{} `json:"name"`
Pubkey string `json:"pubkey"`
Lat interface{} `json:"lat"`
Lon interface{} `json:"lon"`
AffinityScore *float64 `json:"affinityScore"`
}
type HopResolution struct {
Name interface{} `json:"name"`
Pubkey interface{} `json:"pubkey,omitempty"`
Ambiguous *bool `json:"ambiguous,omitempty"`
Candidates []HopCandidate `json:"candidates"`
Conflicts []interface{} `json:"conflicts"`
Name interface{} `json:"name"`
Pubkey interface{} `json:"pubkey,omitempty"`
Ambiguous *bool `json:"ambiguous,omitempty"`
Candidates []HopCandidate `json:"candidates"`
Conflicts []interface{} `json:"conflicts"`
BestCandidate *string `json:"bestCandidate,omitempty"`
Confidence string `json:"confidence,omitempty"`
}
type ResolveHopsResponse struct {
@@ -921,6 +952,7 @@ type ClientConfigResponse struct {
ExternalUrls interface{} `json:"externalUrls"`
PropagationBufferMs float64 `json:"propagationBufferMs"`
Timestamps TimestampConfig `json:"timestamps"`
DebugAffinity bool `json:"debugAffinity,omitempty"`
}
// ─── IATA Coords ───────────────────────────────────────────────────────────────
+88 -21
View File
@@ -1,20 +1,25 @@
{
"port": 3000,
"apiKey": "your-secret-api-key-here",
"nodeBlacklist": [],
"_comment_nodeBlacklist": "Public keys of nodes to hide from all API responses. Use for trolls, offensive names, or nodes reporting false data that operators refuse to fix.",
"retention": {
"nodeDays": 7,
"observerDays": 14,
"packetDays": 30,
"_comment": "nodeDays: nodes not seen in N days are moved to inactive_nodes (default 7). packetDays: transmissions+observations older than N days are deleted daily (0 = disabled)."
"_comment": "nodeDays: nodes not seen in N days moved to inactive_nodes (default 7). observerDays: observers not sending data in N days are removed (-1 = keep forever, default 14). packetDays: transmissions older than N days are deleted (0 = disabled)."
},
"https": {
"cert": "/path/to/cert.pem",
"key": "/path/to/key.pem"
"key": "/path/to/key.pem",
"_comment": "TLS cert/key paths for direct HTTPS. Most deployments use Caddy (included in Docker) for auto-TLS instead."
},
"branding": {
"siteName": "CoreScope",
"tagline": "Real-time MeshCore LoRa mesh network analyzer",
"logoUrl": null,
"faviconUrl": null
"faviconUrl": null,
"_comment": "Customize site name, tagline, logo, and favicon. logoUrl/faviconUrl can be absolute URLs or relative paths."
},
"theme": {
"accent": "#4a9eff",
@@ -23,38 +28,75 @@
"navBg2": "#1a1a2e",
"statusGreen": "#45644c",
"statusYellow": "#b08b2d",
"statusRed": "#b54a4a"
"statusRed": "#b54a4a",
"_comment": "CSS color overrides. Use the in-app Theme Customizer for live preview, then export values here."
},
"nodeColors": {
"repeater": "#dc2626",
"companion": "#2563eb",
"room": "#16a34a",
"sensor": "#d97706",
"observer": "#8b5cf6"
"observer": "#8b5cf6",
"_comment": "Marker/badge colors per node role. Used on map, nodes list, and live feed."
},
"home": {
"heroTitle": "CoreScope",
"heroSubtitle": "Find your nodes to start monitoring them.",
"steps": [
{ "emoji": "📡", "title": "Connect", "description": "Link your node to the mesh" },
{ "emoji": "🔍", "title": "Monitor", "description": "Watch packets flow in real-time" },
{ "emoji": "📊", "title": "Analyze", "description": "Understand your network's health" }
{
"emoji": "\ud83d\udce1",
"title": "Connect",
"description": "Link your node to the mesh"
},
{
"emoji": "\ud83d\udd0d",
"title": "Monitor",
"description": "Watch packets flow in real-time"
},
{
"emoji": "\ud83d\udcca",
"title": "Analyze",
"description": "Understand your network's health"
}
],
"checklist": [
{ "question": "How do I add my node?", "answer": "Search for your node name or paste your public key." },
{ "question": "What regions are covered?", "answer": "Check the map page to see active observers and nodes." }
{
"question": "How do I add my node?",
"answer": "Search for your node name or paste your public key."
},
{
"question": "What regions are covered?",
"answer": "Check the map page to see active observers and nodes."
}
],
"footerLinks": [
{ "label": "📦 Packets", "url": "#/packets" },
{ "label": "🗺️ Network Map", "url": "#/map" },
{ "label": "🔴 Live", "url": "#/live" },
{ "label": "📡 All Nodes", "url": "#/nodes" },
{ "label": "💬 Channels", "url": "#/channels" }
]
{
"label": "\ud83d\udce6 Packets",
"url": "#/packets"
},
{
"label": "\ud83d\uddfa\ufe0f Network Map",
"url": "#/map"
},
{
"label": "\ud83d\udd34 Live",
"url": "#/live"
},
{
"label": "\ud83d\udce1 All Nodes",
"url": "#/nodes"
},
{
"label": "\ud83d\udcac Channels",
"url": "#/channels"
}
],
"_comment": "Customize the landing page hero, onboarding steps, FAQ, and footer links."
},
"mqtt": {
"broker": "mqtt://localhost:1883",
"topic": "meshcore/+/+/packets"
"topic": "meshcore/+/+/packets",
"_comment": "Legacy single-broker config. Prefer mqttSources[] for multiple brokers."
},
"mqttSources": [
{
@@ -86,7 +128,7 @@
}
],
"channelKeys": {
"public": "8b3387e9c5cdea6ac9e5edbaa115cd72"
"Public": "8b3387e9c5cdea6ac9e5edbaa115cd72"
},
"hashChannels": [
"#LongFast",
@@ -114,6 +156,16 @@
],
"zoom": 9
},
"geo_filter": {
"polygon": [
[37.80, -122.52],
[37.80, -121.80],
[37.20, -121.80],
[37.20, -122.52]
],
"bufferKm": 20,
"_comment": "Optional. Restricts ingestion and API responses to nodes within the polygon + bufferKm. Polygon is an array of [lat, lon] pairs (minimum 3). Use tools/geofilter-builder.html to draw a polygon visually. Remove this section to disable filtering. Nodes with no GPS fix are always allowed through."
},
"regions": {
"SJC": "San Jose, US",
"SFO": "San Francisco, US",
@@ -150,11 +202,26 @@
"timezone": "local",
"formatPreset": "iso",
"customFormat": "",
"allowCustomFormat": false
"allowCustomFormat": false,
"_comment": "defaultMode: ago|local|iso. timezone: local|utc. formatPreset: iso|us|eu. customFormat: strftime-style (requires allowCustomFormat: true)."
},
"packetStore": {
"maxMemoryMB": 1024,
"estimatedPacketBytes": 450,
"_comment": "In-memory packet store. maxMemoryMB caps RAM usage. All packets loaded on startup, served from RAM."
}
}
},
"resolvedPath": {
"backfillHours": 24,
"_comment": "How far back (hours) the async backfill scans for observations with NULL resolved_path. Default: 24. Set higher to backfill older data, lower to speed up startup."
},
"neighborGraph": {
"maxAgeDays": 5,
"_comment": "Neighbor edges older than this many days are pruned on startup and daily. Default: 5."
},
"_comment_mqttSources": "Each source connects to an MQTT broker. topics: what to subscribe to. iataFilter: only ingest packets from these regions (optional).",
"_comment_channelKeys": "Hex keys for decrypting channel messages. Key name = channel display name. public channel key is well-known.",
"_comment_hashChannels": "Channel names whose keys are derived via SHA256. Key = SHA256(name)[:16]. Listed here so the ingestor can auto-derive keys.",
"_comment_defaultRegion": "IATA code shown by default in region filters.",
"_comment_mapDefaults": "Initial map center [lat, lon] and zoom level.",
"_comment_regions": "IATA code to display name mapping. Packets are tagged with region codes by MQTT topic structure."
}
+20
View File
@@ -0,0 +1,20 @@
# CoreScope — simple deployment using pre-built image from GHCR
# Usage: docker compose -f docker-compose.example.yml up -d
# Docs: https://github.com/Kpa-clawbot/CoreScope/blob/master/DEPLOY.md
services:
corescope:
image: ghcr.io/kpa-clawbot/corescope:latest
ports:
- "${HTTP_PORT:-80}:80"
volumes:
- ${DATA_DIR:-./data}:/app/data
environment:
- DISABLE_CADDY=${DISABLE_CADDY:-true}
- DISABLE_MOSQUITTO=${DISABLE_MOSQUITTO:-false}
restart: unless-stopped
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:3000/api/stats"]
interval: 30s
timeout: 5s
retries: 3
+3 -6
View File
@@ -15,15 +15,11 @@ services:
restart: unless-stopped
stop_grace_period: 30s
stop_signal: SIGTERM
deploy:
resources:
limits:
memory: 3g
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- "${STAGING_GO_HTTP_PORT:-82}:80"
- "${STAGING_GO_MQTT_PORT:-1885}:1883"
- "${STAGING_GO_HTTP_PORT:-80}:80"
- "${STAGING_GO_MQTT_PORT:-1883}:1883"
- "6060:6060" # pprof server
- "6061:6061" # pprof ingestor
volumes:
@@ -33,6 +29,7 @@ services:
- NODE_ENV=staging
- ENABLE_PPROF=true
- DISABLE_MOSQUITTO=${DISABLE_MOSQUITTO:-false}
- DISABLE_CADDY=${DISABLE_CADDY:-false}
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:3000/api/stats"]
interval: 30s
+1
View File
@@ -29,6 +29,7 @@ services:
environment:
- NODE_ENV=production
- DISABLE_MOSQUITTO=${DISABLE_MOSQUITTO:-false}
- DISABLE_CADDY=${DISABLE_CADDY:-false}
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:3000/api/stats"]
interval: 30s
+15 -1
View File
@@ -14,10 +14,24 @@ if [ -f /app/data/theme.json ]; then
ln -sf /app/data/theme.json /app/theme.json
fi
# Source .env from data volume if present (works with any launch method)
if [ -f /app/data/.env ]; then
set -a
. /app/data/.env
set +a
fi
SUPERVISORD_CONF="/etc/supervisor/conf.d/supervisord.conf"
if [ "${DISABLE_MOSQUITTO:-false}" = "true" ]; then
if [ "${DISABLE_MOSQUITTO:-false}" = "true" ] && [ "${DISABLE_CADDY:-false}" = "true" ]; then
echo "[config] internal MQTT broker disabled (DISABLE_MOSQUITTO=true)"
echo "[config] Caddy reverse proxy disabled (DISABLE_CADDY=true)"
SUPERVISORD_CONF="/etc/supervisor/conf.d/supervisord-no-mosquitto-no-caddy.conf"
elif [ "${DISABLE_MOSQUITTO:-false}" = "true" ]; then
echo "[config] internal MQTT broker disabled (DISABLE_MOSQUITTO=true)"
SUPERVISORD_CONF="/etc/supervisor/conf.d/supervisord-no-mosquitto.conf"
elif [ "${DISABLE_CADDY:-false}" = "true" ]; then
echo "[config] Caddy reverse proxy disabled (DISABLE_CADDY=true)"
SUPERVISORD_CONF="/etc/supervisor/conf.d/supervisord-no-caddy.conf"
fi
exec /usr/bin/supervisord -c "$SUPERVISORD_CONF"
+43
View File
@@ -0,0 +1,43 @@
[supervisord]
nodaemon=true
user=root
logfile=/dev/stdout
logfile_maxbytes=0
pidfile=/var/run/supervisord.pid
[program:mosquitto]
command=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:corescope-ingestor]
command=/app/corescope-ingestor -config /app/config.json
directory=/app
autostart=true
autorestart=true
startretries=10
startsecs=2
stopsignal=TERM
stopwaitsecs=20
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:corescope-server]
command=/app/corescope-server -config-dir /app -db /app/data/meshcore.db -public /app/public -port 3000
directory=/app
autostart=true
autorestart=true
startretries=10
startsecs=2
stopsignal=TERM
stopwaitsecs=20
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
@@ -0,0 +1,34 @@
[supervisord]
nodaemon=true
user=root
logfile=/dev/stdout
logfile_maxbytes=0
pidfile=/var/run/supervisord.pid
[program:corescope-ingestor]
command=/app/corescope-ingestor -config /app/config.json
directory=/app
autostart=true
autorestart=true
startretries=10
startsecs=2
stopsignal=TERM
stopwaitsecs=20
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:corescope-server]
command=/app/corescope-server -config-dir /app -db /app/data/meshcore.db -public /app/public -port 3000
directory=/app
autostart=true
autorestart=true
startretries=10
startsecs=2
stopsignal=TERM
stopwaitsecs=20
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
+47
View File
@@ -0,0 +1,47 @@
# CoreScope v3.4 Release Notes
**The neighbor affinity release.** CoreScope now understands how nodes relate to each other — not just that they exist, but how strongly they're connected. This powers smarter hop resolution, richer node detail pages, and a new graph visualization in analytics.
---
## 🎯 Features
### Neighbor Affinity System (7 milestones)
A complete neighbor relationship engine, from backend graph building to frontend visualization:
- **Affinity graph builder** — computes neighbor relationships and connection strength from packet traffic (#507)
- **Affinity API endpoints** — REST endpoints to query neighbor data (#508)
- **Show Neighbors via affinity API** — the existing Show Neighbors feature now uses real affinity data instead of raw packet heuristics (#512, fixes #484)
- **Affinity-aware hop resolution** — hop resolver uses neighbor affinity to pick better paths (#511)
- **Node detail neighbors section** — dedicated neighbors panel on the node detail page (#510)
- **Affinity debugging tools** — inspect and troubleshoot affinity calculations (#521)
- **Neighbor graph visualization** — interactive neighbor graph in the analytics tab (#513)
### Customizer v2
- Event-driven state management replaces the old imperative approach — cleaner, more predictable theme/config updates (#503)
---
## 🐛 Bug Fixes
- **Stale parsed cache on observation packets** — observation packets now correctly invalidate the JSON parse cache (#505)
- **Null-guard rAF callbacks** — live page no longer crashes when `requestAnimationFrame` callbacks fire after cleanup (#506)
- **Customizer v2 phantom overrides** — fixed phantom config entries, missing defaults, and stale dark mode state (#520)
- **Neighbor affinity empty results** — fixed pubKey field name mismatch causing empty affinity graphs (#524)
- **Home defaults in server theme** — server-side theme config now includes home page defaults (#526)
- **Neighbor UI crash + dark mode** — fixed Show Neighbors crash and improved dark mode contrast (#527)
- **Home page steps + FAQ** — both steps AND FAQ now render correctly on the home page (#529)
---
## ⚡ Performance
- **Cached JSON.parse for packet data** — packet payloads are parsed once and cached, avoiding redundant `JSON.parse` calls on repeated access (#400)
---
## Known Limitations
- **Affinity graph scales with traffic volume** — networks with very low packet rates may show weak or missing neighbor relationships until enough data accumulates
- **Debugging tools are developer-facing** — the affinity debug panel (#521) is functional but not polished for end-user consumption
- **Customizer v2 migration** — custom themes saved under v1 may need to be re-applied after upgrade
+496
View File
@@ -0,0 +1,496 @@
# CoreScope Deployment Guide
Comprehensive guide to deploying and operating CoreScope. For a quick start, see [DEPLOY.md](../DEPLOY.md).
## Table of Contents
- [System Requirements](#system-requirements)
- [Docker Deployment](#docker-deployment)
- [Configuration Reference](#configuration-reference)
- [MQTT Setup](#mqtt-setup)
- [TLS / HTTPS](#tls--https)
- [Monitoring & Health Checks](#monitoring--health-checks)
- [Backup & Restore](#backup--restore)
- [Troubleshooting](#troubleshooting)
---
## System Requirements
| Resource | Minimum | Recommended |
|----------|---------|-------------|
| RAM | 256 MB | 512 MB+ |
| Disk | 500 MB (image + DB) | 2 GB+ for long-term data |
| CPU | 1 core | 2+ cores |
| Architecture | `linux/amd64`, `linux/arm64` | — |
| Docker | 20.10+ | Latest stable |
CoreScope runs well on Raspberry Pi 4/5 (ARM64). The Go server uses ~300 MB RAM for 56K+ packets.
---
## Docker Deployment
### Quick Start (one command)
```bash
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
```
Open `http://localhost` — you'll see an empty dashboard ready to receive packets.
No `config.json` is required. The server starts with sensible defaults:
- HTTP on port 3000 (Caddy proxies port 80 → 3000 internally)
- Internal Mosquitto MQTT broker on port 1883
- Ingestor connects to `mqtt://localhost:1883` automatically
- SQLite database at `/app/data/meshcore.db`
### Full `docker run` Reference (recommended)
The bare `docker run` command is the primary deployment method. One image, documented parameters — run it however you want.
```bash
docker run -d --name corescope \
--restart=unless-stopped \
-p 80:80 -p 443:443 -p 1883:1883 \
-e DISABLE_MOSQUITTO=false \
-e DISABLE_CADDY=false \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:latest
```
#### Parameters
| Parameter | Required | Description |
|-----------|----------|-------------|
| `-p 80:80` | Yes | HTTP web UI |
| `-p 443:443` | No | HTTPS (only if using built-in Caddy with a domain) |
| `-p 1883:1883` | No | MQTT broker (expose if external gateways connect directly) |
| `-v /your/data:/app/data` | Yes | Persistent data: SQLite DB, config.json, theme.json |
| `-v /your/Caddyfile:/etc/caddy/Caddyfile:ro` | No | Custom Caddyfile for HTTPS |
| `-v /your/caddy-data:/data/caddy` | No | Caddy TLS certificate storage |
| `-e DISABLE_MOSQUITTO=true` | No | Skip the internal Mosquitto broker (use your own) |
| `-e DISABLE_CADDY=true` | No | Skip the built-in Caddy reverse proxy |
| `-e MQTT_BROKER=mqtt://host:1883` | No | Override MQTT broker URL |
#### `/app/data/.env` convenience file
Instead of passing `-e` flags, you can drop a `.env` file in your data volume:
```bash
# /your/data/.env
DISABLE_MOSQUITTO=true
DISABLE_CADDY=true
MQTT_BROKER=mqtt://my-broker:1883
```
The entrypoint sources this file before starting services. This works with any launch method (`docker run`, compose, or manage.sh).
### Docker Compose (legacy alternative)
Docker Compose files are maintained for backward compatibility but are no longer the recommended approach.
```bash
curl -sL https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/docker-compose.example.yml \
-o docker-compose.yml
docker compose up -d
```
#### Compose environment variables
| Variable | Default | Description |
|----------|---------|-------------|
| `HTTP_PORT` | `80` | Host port for the web UI |
| `DATA_DIR` | `./data` | Host path for persistent data |
| `DISABLE_MOSQUITTO` | `false` | Set `true` to use an external MQTT broker |
| `DISABLE_CADDY` | `false` | Set `true` to skip the built-in Caddy proxy |
### manage.sh (legacy alternative)
The `manage.sh` wrapper script provides a setup wizard and convenience commands. It uses Docker Compose internally. See [DEPLOY.md](../DEPLOY.md) for usage. New deployments should prefer bare `docker run`.
### Image tags
| Tag | Use case |
|-----|----------|
| `v3.4.1` | Pinned release — recommended for production |
| `v3.4` | Latest patch in the v3.4.x series |
| `v3` | Latest minor+patch in v3.x |
| `latest` | Latest release tag |
| `edge` | Built from master on every push — unstable |
### Updating
```bash
docker compose pull
docker compose up -d
```
For `docker run` users:
```bash
docker pull ghcr.io/kpa-clawbot/corescope:latest
docker stop corescope && docker rm corescope
docker run -d --name corescope ... # same flags as before
```
Data is preserved in the volume — updates are non-destructive.
---
## Configuration Reference
CoreScope uses a layered configuration system (highest priority wins):
1. **Environment variables**`MQTT_BROKER`, `DB_PATH`, etc.
2. **`/app/data/config.json`** — full config file (volume-mounted)
3. **Built-in defaults** — work out of the box with no config
### Environment variable overrides
| Variable | Default | Description |
|----------|---------|-------------|
| `MQTT_BROKER` | `mqtt://localhost:1883` | MQTT broker URL (overrides config file) |
| `MQTT_TOPIC` | `meshcore/#` | MQTT topic subscription pattern |
| `DB_PATH` | `data/meshcore.db` | SQLite database path |
| `DISABLE_MOSQUITTO` | `false` | Skip the internal Mosquitto broker |
| `DISABLE_CADDY` | `false` | Skip the built-in Caddy reverse proxy |
### config.json
For advanced configuration, create a `config.json` and mount it at `/app/data/config.json`:
```bash
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
-v ./config.json:/app/data/config.json:ro \
ghcr.io/kpa-clawbot/corescope:latest
```
See `config.example.json` in the repository for all available options including:
- MQTT sources (multiple brokers)
- Channel encryption keys
- Branding and theming
- Health thresholds
- Region filters
- Retention policies
- Geo-filtering
---
## MQTT Setup
CoreScope receives MeshCore packets via MQTT. The container ships with an internal Mosquitto broker — no setup needed for basic use.
### Internal broker (default)
The built-in Mosquitto broker listens on port 1883 inside the container. Point your MeshCore gateways at it:
```bash
# Expose MQTT port for external gateways
docker run -d --name corescope \
-p 80:80 -p 1883:1883 \
-v corescope-data:/app/data \
ghcr.io/kpa-clawbot/corescope:latest
```
### External broker
To use your own MQTT broker (Mosquitto, EMQX, HiveMQ, etc.):
1. Disable the internal broker:
```bash
-e DISABLE_MOSQUITTO=true
```
2. Point the ingestor at your broker:
```bash
-e MQTT_BROKER=mqtt://your-broker:1883
```
Or via `config.json`:
```json
{
"mqttSources": [
{
"name": "my-broker",
"broker": "mqtt://your-broker:1883",
"username": "user",
"password": "pass",
"topics": ["meshcore/#"]
}
]
}
```
### Multiple brokers
CoreScope can connect to multiple MQTT brokers simultaneously:
```json
{
"mqttSources": [
{
"name": "local",
"broker": "mqtt://localhost:1883",
"topics": ["meshcore/#"]
},
{
"name": "remote",
"broker": "mqtts://remote-broker:8883",
"username": "reader",
"password": "secret",
"topics": ["meshcore/+/+/packets"]
}
]
}
```
### MQTT topic format
MeshCore gateways typically publish to `meshcore/<gateway>/<region>/packets`. The default subscription `meshcore/#` catches all of them.
---
## TLS / HTTPS
### Option 1: External reverse proxy (recommended)
Run CoreScope behind nginx, Traefik, or Cloudflare Tunnel for TLS termination:
```nginx
# nginx example
server {
listen 443 ssl;
server_name corescope.example.com;
ssl_certificate /etc/ssl/certs/corescope.pem;
ssl_certificate_key /etc/ssl/private/corescope.key;
location / {
proxy_pass http://localhost:80;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
}
```
The `Upgrade` and `Connection` headers are required for WebSocket support.
### Option 2: Built-in Caddy (auto-TLS)
The container includes Caddy for automatic Let's Encrypt certificates:
1. Create a Caddyfile:
```
corescope.example.com {
reverse_proxy localhost:3000
}
```
2. Mount it and expose TLS ports:
```bash
docker run -d --name corescope \
-p 80:80 -p 443:443 \
-v corescope-data:/app/data \
-v caddy-certs:/data/caddy \
-v ./Caddyfile:/etc/caddy/Caddyfile:ro \
ghcr.io/kpa-clawbot/corescope:latest
```
Caddy handles certificate issuance and renewal automatically.
---
## API Documentation
CoreScope auto-generates an OpenAPI 3.0 specification from its route definitions. The spec is always in sync with the running server — no manual maintenance required.
### Endpoints
| URL | Description |
|-----|-------------|
| `/api/spec` | OpenAPI 3.0 JSON schema — machine-readable API definition |
| `/api/docs` | Interactive Swagger UI — browse and test all 40+ endpoints |
### Usage
**Browse the API interactively:**
```
http://your-instance/api/docs
```
**Fetch the spec programmatically:**
```bash
curl http://your-instance/api/spec | jq .
```
**For bot/integration developers:** The spec includes all request parameters, response schemas, and example values. Import it into Postman, Insomnia, or any OpenAPI-compatible tool.
### Public instance
The live instance at [analyzer.00id.net](https://analyzer.00id.net) has all API endpoints publicly accessible:
- Spec: [analyzer.00id.net/api/spec](https://analyzer.00id.net/api/spec)
- Docs: [analyzer.00id.net/api/docs](https://analyzer.00id.net/api/docs)
---
## Monitoring & Health Checks
### Docker health check
The container includes a built-in health check that hits `/api/stats`:
```bash
docker inspect --format='{{.State.Health.Status}}' corescope
```
Docker reports `healthy` or `unhealthy` automatically. The check runs every 30 seconds.
### Manual health check
```bash
curl -f http://localhost/api/stats
```
Returns JSON with packet counts, node counts, and version info:
```json
{
"totalPackets": 56234,
"totalNodes": 142,
"totalObservers": 12,
"packetsLastHour": 830,
"packetsLast24h": 19644,
"engine": "go",
"version": "v3.4.1"
}
```
### Log monitoring
```bash
# All logs
docker compose logs -f
# Server only
docker compose logs -f | grep '\[server\]'
# Ingestor only
docker compose logs -f | grep '\[ingestor\]'
```
### Resource monitoring
```bash
docker stats corescope
```
---
## Backup & Restore
### Backup
All persistent data lives in `/app/data`. The critical file is the SQLite database:
```bash
# Copy from the Docker volume
docker cp corescope:/app/data/meshcore.db ./backup-$(date +%Y%m%d).db
# Or if using a bind mount
cp ./data/meshcore.db ./backup-$(date +%Y%m%d).db
```
Optional files to back up:
- `config.json` — custom configuration
- `theme.json` — custom theme/branding
### Restore
```bash
# Stop the container
docker stop corescope
# Replace the database
docker cp ./backup.db corescope:/app/data/meshcore.db
# Restart
docker start corescope
```
### Automated backups
```bash
# cron: daily backup at 3 AM, keep 7 days
0 3 * * * docker cp corescope:/app/data/meshcore.db /backups/corescope-$(date +\%Y\%m\%d).db && find /backups -name "corescope-*.db" -mtime +7 -delete
```
---
## Troubleshooting
### Container starts but dashboard is empty
This is normal on first start with no MQTT sources configured. The dashboard shows data once packets arrive via MQTT. Either:
- Point a MeshCore gateway at the container's MQTT broker (port 1883)
- Configure an external MQTT source in `config.json`
### "no MQTT connections established" in logs
The ingestor couldn't connect to any MQTT broker. Check:
1. Is the internal Mosquitto running? (`DISABLE_MOSQUITTO` should be `false`)
2. Is the external broker reachable? Test with `mosquitto_sub -h broker -t meshcore/#`
3. Are credentials correct in `config.json`?
### WebSocket disconnects / real-time updates stop
If behind a reverse proxy, ensure WebSocket upgrade headers are forwarded:
```nginx
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
```
Also check proxy timeouts — set them to at least 300s for long-lived WebSocket connections.
### High memory usage
The in-memory packet store grows with retained packets. Configure retention limits in `config.json`:
```json
{
"packetStore": {
"retentionHours": 24,
"maxMemoryMB": 512
},
"retention": {
"nodeDays": 7,
"packetDays": 30
}
}
```
### Database locked errors
SQLite doesn't support concurrent writers well. Ensure only one CoreScope instance accesses the database file. If running multiple containers, each needs its own database.
### Container unhealthy
Check logs: `docker compose logs --tail 50`. Common causes:
- Port 3000 already in use inside the container
- Database file permissions (must be writable by the container user)
- Corrupted database — restore from backup
### ARM / Raspberry Pi issues
- Use `linux/arm64` images (Pi 4 and 5). Pi 3 (armv7) is not supported.
- First pull may be slow — the multi-arch manifest selects the right image automatically.
- If memory is tight, set `packetStore.maxMemoryMB` to limit RAM usage.
+403
View File
@@ -0,0 +1,403 @@
# Security Analysis: MeshCore Channel Encryption
## Scope
This analysis covers MeshCore's encryption vulnerabilities in order of practical severity. Section 1 addresses PSK brute-force (the highest-priority practical threat). Sections 29 cover AES-128-ECB structural weaknesses. Section 8 covers TXT_MSG. All claims are derived from firmware source (`BaseChatMesh.cpp`, `Utils.cpp`, `Mesh.cpp`, `MeshCore.h`) unless explicitly marked as conjecture.
## 1. PSK Brute-Force with Timestamp Oracle
### 1.1 The No-KDF Design
MeshCore channel PSKs are base64-decoded directly into AES-128 keys with no key derivation function (from `BaseChatMesh::addChannel()`):
```cpp
int len = decode_base64((unsigned char *) psk_base64, strlen(psk_base64), dest->channel.secret);
```
No PBKDF2, scrypt, argon2, or HKDF is applied. The base64-decoded bytes ARE the AES key. This means:
1. **Human-memorable passphrases have drastically reduced entropy.** If a user types "SecretChannel" as their PSK, the base64-decoded output is ~10 bytes of ASCII-range values. The key space is determined by the passphrase complexity, not by AES-128's theoretical 2^128 key space.
2. **Short passphrases produce short keys.** `decode_base64` maps every 4 base64 characters to 3 bytes. A passphrase shorter than ~22 base64 characters produces fewer than 16 bytes, and the remainder of the 16-byte key buffer depends on whatever was previously in memory (likely zeros from initialization). An 8-character passphrase decodes to only 6 bytes — the effective key space may be as low as 2^48.
3. **No salt.** Identical passphrases on different meshes produce identical keys. A single precomputed dictionary attack works globally against all MeshCore deployments.
### 1.2 Timestamp as Known-Plaintext Oracle
Every GRP_TXT plaintext begins with a structured, largely predictable header:
```
Block 0: [TS₀][TS₁][TS₂][TS₃][0x00][sender_name][: ][message_start...]
```
An attacker who captures a single packet can verify a candidate PSK by:
1. Decrypting block 0 with the candidate key
2. Checking if bytes 03 produce a plausible Unix timestamp (within a reasonable window of the capture time)
3. Checking if byte 4 is 0x00 (TXT_TYPE_PLAIN)
4. Optionally checking if bytes 5+ are valid ASCII (sender name)
The timestamp alone constrains the search: a ±1-hour window around capture time yields ~7,200 valid timestamps out of 2^32 possibilities — a false-positive rate of ~1.7×10^-6. Combined with the type byte and ASCII sender-name check, false positives are effectively zero. **One captured packet is sufficient for definitive key verification.**
### 1.3 Attack Cost Estimates
Hardware assumption: commodity GPU (e.g., RTX 4090) performing ~10 billion AES-128-ECB block encryptions per second. This is conservative — optimized implementations achieve higher throughput.
| Passphrase style | Search space | Time at 10^10 AES/sec |
|---|---|---|
| Single common English word (10K-word list) | ~10^4 | microseconds |
| Single English word (170K full dictionary) | ~1.7×10^5 | microseconds |
| Two concatenated common words | ~10^8 | ~10 milliseconds |
| Three concatenated common words | ~10^12 | ~100 seconds (~2 min) |
| Four random common words (Diceware-style) | ~10^16 | ~10^6 seconds (~12 days) |
| Random 8-char alphanumeric (62^8) | ~2.2×10^14 | ~22,000 seconds (~6 hours) |
| Random 12-char alphanumeric (62^12) | ~3.2×10^21 | ~10^11 seconds (infeasible) |
| Full random 16-byte key (2^128) | ~3.4×10^38 | infeasible |
**Important caveats on search space:**
- Dictionary sizes vary: "common English words" ≈ 3K10K; full dictionary ≈ 170K. Estimates above use 10K for "common" lists.
- Humans do not choose words uniformly. Zipf's law applies — a small fraction of words account for most selections. The effective entropy is **lower** than the uniform assumption, making attacks faster.
- Concatenation without separators creates ambiguity ("therapist" = "therapist" or "the"+"rapist"), but this marginally reduces search space rather than increasing it.
- Multi-channel amortization: an attacker can test each candidate against ALL captured channels simultaneously, paying the AES cost once per candidate.
### 1.4 Attack Properties
- **Offline attack.** No rate limiting, no lockout, no detection. The attacker works entirely on captured ciphertext.
- **Single-packet verification.** One GRP_TXT packet is sufficient. No need to collect multiple messages.
- **No KDF stretching.** Each candidate requires exactly one AES-128 block decryption (16 bytes), not thousands of hash iterations.
- **Global applicability.** No salt means precomputed tables work across all MeshCore deployments using the same passphrase.
- **Side-channel exposure.** Since the PSK IS the key (no KDF), any AES key-schedule side-channel directly reveals the passphrase. PSK reuse across systems (e.g., same passphrase for MeshCore and WiFi) means compromise of one compromises both.
### 1.5 Severity Assessment
**PSK brute-force is the #1 practical threat to MeshCore channel confidentiality.** Unlike ECB frequency analysis (§5), which requires hundreds of captured messages with repeated content, PSK brute-force requires a single captured packet and succeeds whenever users choose human-memorable passphrases — which is the common case for manually-configured channels.
Any channel using a passphrase of 3 or fewer common words, or any alphanumeric string shorter than 12 characters, should be considered **vulnerable to offline brute-force within hours to days** using commodity hardware.
### 1.6 Recommended Mitigations
**Priority 0 (Critical):** Apply a memory-hard KDF (argon2id preferred; scrypt or PBKDF2 with ≥100K iterations as fallback) to derive the AES key from the passphrase. This transforms each candidate test from ~1 nanosecond to ~100 milliseconds, increasing attack cost by a factor of ~10^8.
**Priority 0a:** Add a per-channel salt (random bytes stored alongside the channel config) to prevent precomputed/global attacks.
**Priority 0b:** Document that channel PSKs should be random 16-byte keys (e.g., generated with `openssl rand -base64 22`), not human-memorable passphrases. This is a stopgap until KDF support is added.
## 2. How Encryption Works
### Constants (from `MeshCore.h`)
- `CIPHER_KEY_SIZE = 16` (AES-128)
- `PUB_KEY_SIZE = 32`
- `CIPHER_MAC_SIZE` = HMAC-SHA256 truncated output size
### encrypt() (from `Utils.cpp`)
AES-128-ECB, block-by-block. No IV, no counter, no chaining:
```cpp
aes.setKey(shared_secret, CIPHER_KEY_SIZE); // first 16 bytes of shared_secret
while (src_len >= 16) {
aes.encryptBlock(dp, src); // each 16-byte block independently
dp += 16; src += 16; src_len -= 16;
}
if (src_len > 0) { // partial final block
uint8_t tmp[16];
memset(tmp, 0, 16); // zero-fill
memcpy(tmp, src, src_len); // copy remaining bytes
aes.encryptBlock(dp, tmp);
}
```
### encryptThenMAC() (from `Utils.cpp`)
```cpp
int enc_len = encrypt(shared_secret, dest + CIPHER_MAC_SIZE, src, src_len);
SHA256 sha;
sha.resetHMAC(shared_secret, PUB_KEY_SIZE); // HMAC uses full 32 bytes
sha.update(dest + CIPHER_MAC_SIZE, enc_len);
sha.finalizeHMAC(shared_secret, PUB_KEY_SIZE, dest, CIPHER_MAC_SIZE);
```
**Key reuse flaw:** The same `shared_secret` buffer serves both AES and HMAC. AES uses `shared_secret[0..15]` (first 16 bytes). HMAC uses `shared_secret[0..31]` (full 32 bytes). The AES key is a prefix of the HMAC key. See §7 for implications.
### GRP_TXT Plaintext Construction (from `BaseChatMesh::sendGroupMessage()`)
```cpp
memcpy(temp, &timestamp, 4); // bytes 0-3: Unix timestamp (seconds)
temp[4] = 0; // byte 4: TXT_TYPE_PLAIN
sprintf((char *)&temp[5], "%s: ", sender_name); // bytes 5+: "SenderName: "
char *ep = strchr((char *)&temp[5], 0);
int prefix_len = ep - (char *)&temp[5]; // length of "SenderName: "
memcpy(ep, text, text_len); // message text (no null terminator)
ep[text_len] = 0; // null written AFTER data boundary
// data_len passed to encrypt = 5 + prefix_len + text_len
```
**The null terminator is NOT part of the encrypted data length.** The call to `createGroupDatagram` passes length `5 + prefix_len + text_len`. The null at `ep[text_len]` is written to the buffer but is beyond `data_len`. In the final partial block, `encrypt()` zero-fills with `memset(tmp, 0, 16)` before copying the remaining bytes — so a zero byte appears at the position where the null would be, but this is an artifact of zero-padding, not an explicit null in the plaintext.
On the receiving side, this is confirmed:
```cpp
data[len] = 0; // need to make a C string again, with null terminator
```
The receiver must re-add the null after decryption.
## 3. Block Layout Analysis
### Notation
Let `N` = length of sender name. Then:
- `prefix_len` = N + 2 (for ": " suffix from `sprintf("%s: ", sender_name)`)
- Header overhead = 4 (timestamp) + 1 (type) + prefix_len = N + 7 bytes
- Message text begins at byte offset N + 7
### Block 0
Block 0 = bytes 015 of plaintext:
```
[TS₀][TS₁][TS₂][TS₃][0x00][sender_name: ][...message start...]
```
The first 9 N bytes of message text fit in block 0 (when N < 9). For N ≥ 9, no message text fits in block 0.
### Boundary Condition: Sender Name ≥ 12 Characters
When N ≥ 12, the header overhead (N + 7 ≥ 19) exceeds 16 bytes. The header itself spills into block 1:
**Example: sender name "LongUserName1" (N = 13), message "hi":**
```
Header = 13 + 7 = 20 bytes. Total plaintext = 20 + 2 = 22 bytes.
Block 0 (bytes 0-15): [TS₀][TS₁][TS₂][TS₃][0x00][L][o][n][g][U][s][e][r][N][a][m]
Block 1 (bytes 16-31): [e][1][:][space][h][i][0x00 ×10] ← zero-padded partial block
```
Block 1 here contains the tail of the sender name, the ": " separator, message text, AND zero-padding. For sender names of length 1215, block 1 is a mix of header and message — **it is NOT "pure message text."**
For sender names ≥ 16, blocks 0 and 1 are both pure header, and message text doesn't begin until block 1 or later.
### General Block Content Table
| Sender name length N | Header bytes | Message starts at byte | Block 0 content | Block 1+ content |
|---|---|---|---|---|
| 18 | 815 | 815 | timestamp + header + message start | message text + zero-pad |
| 911 | 1618 | 1618 | timestamp + header (no message) | header tail + message + zero-pad |
| 1215 | 1922 | 1922 | timestamp + partial header | header tail + message + zero-pad |
| ≥16 | ≥23 | ≥23 | timestamp + partial header | header continuation, then message |
### Typical Case (N = 5, e.g. "Alice")
Header = 12 bytes. Message starts at byte 12. Block 0 holds 4 bytes of message text.
```
Message "hello world" (11 chars). Total plaintext = 12 + 11 = 23 bytes.
Block 0 (bytes 0-15): [TS₀][TS₁][TS₂][TS₃][0x00][A][l][i][c][e][:][space][h][e][l][l]
Block 1 (bytes 16-22): [o][space][w][o][r][l][d] → padded to: [o][space][w][o][r][l][d][0×9]
```
Block 1 contains 7 bytes of message text and 9 bytes of zero-padding.
## 4. Attack Surface by Block Position
### Block 0: Accidental Nonce from Timestamp
The 4-byte Unix timestamp in bytes 03 acts as an **accidental nonce** — it was included "mostly as an extra blob to help make packet_hash unique" (per firmware comment), not as a cryptographic countermeasure against ECB determinism. Nevertheless, it has the effect of making block 0's plaintext vary per message.
**Precision on uniqueness:** Block 0 is unique per (sender, timestamp-second) pair, not per message. Two messages from the same sender within the same second, on the same channel, with the same type byte, produce identical block 0 plaintext and therefore identical block 0 ciphertext. At typical mesh chat rates, same-second collisions are rare but not impossible for automated/scripted senders.
**Known-plaintext observation:** Bytes 415 of block 0 are largely predictable per sender (type byte is always 0x00 for plain text; sender name and ": " are static). The timestamp is predictable within a window (Unix seconds). An attacker who knows the sender name and approximate time can compute all 16 plaintext bytes of block 0. However, **AES-128 is resistant to known-plaintext attacks** — knowing plaintext-ciphertext pairs for block 0 does not help recover the key or decrypt other blocks.
### Blocks 1+: Deterministic ECB (for short sender names)
When the sender name is short enough that the header fits in block 0 (N ≤ 8), blocks 1+ contain **only message text and zero-padding.** No timestamp, no nonce, no per-message varying data. Identical message text at the same block offset produces identical ciphertext, always.
When N ≥ 9, block 1 contains header spillover, which includes static sender name bytes — these vary per sender but not per message, so block 1 is still deterministic for a given sender once the header portion is fixed.
**The fundamental ECB property:** For any block beyond the timestamp's reach, `E_K(P) = E_K(P)`. Same plaintext block → same ciphertext block, regardless of when or how many times it's sent.
### Partial Final Block: Strongest Attack Target
The final block of every message is zero-padded by `encrypt()` to 16 bytes. The padding bytes are deterministic and known (always 0x00). For a message whose final block contains `B` bytes of actual content:
- `B` bytes are unknown message text
- `16 - B` bytes are known zeros
When B is small (short final fragment), most of the block is known plaintext. For B = 1, the attacker knows 15 of 16 bytes — only 256 possible plaintext blocks exist. This means:
- **The final block has at most 2^(8B) possible plaintexts** (versus 2^128 for a full unknown block)
- For B ≤ 4, there are ≤ 2^32 possibilities — a small enough space for dictionary attacks given enough ciphertext samples
- The attacker can precompute all possible final-block plaintexts for small B values and match against observed ciphertext blocks
This makes the partial final block a **stronger frequency analysis target** than interior blocks, where all 16 bytes may be unknown text.
## 5. Feasible Attack Scenarios
### 4.1 Block Frequency Analysis on Blocks 1+
**Preconditions (all must hold):**
1. Attacker can observe encrypted GRP_TXT packets (passive radio capture)
2. Messages from the same sender (or senders with identical name lengths — same block alignment)
3. Messages long enough to produce blocks beyond block 0 (text > 9 N chars)
4. Sufficient message volume with repeated content at the same block positions
**Method:**
1. Collect GRP_TXT packets, group by sender hash
2. Decompose encrypted payloads into 16-byte blocks (after stripping HMAC prefix)
3. Discard block 0 (timestamp-varying)
4. Build frequency tables for blocks 1, 2, 3, etc., per sender
5. Match high-frequency ciphertext blocks against expected plaintext distributions
**Practical constraints limiting this attack:**
- LoRa bandwidth severely limits message length. Most mesh chat messages are short — many fit entirely within block 0 (≤ 9 N chars of text), yielding zero analyzable blocks.
- Messages that spill into block 1+ tend to be longer and more varied — fewer repeated patterns.
- The attack requires repeated identical 16-byte-aligned text fragments from the same sender over time.
**Conditions under which this attack succeeds:** Automated or scripted senders transmitting repetitive messages longer than block 0 capacity, on a channel with a static PSK, over an extended collection period. For human-typed conversational messages with typical length and variety, the number of repeated block 1+ patterns is likely too low for meaningful frequency analysis. (This is an empirical claim that depends on actual traffic patterns — no formal bound is established here.)
### 4.2 Partial Final Block Dictionary Attack
**Preconditions:**
1. Attacker knows (or can estimate) the message length modulo 16
2. Final block has few content bytes (B ≤ 4)
**Method:** Enumerate all 2^(8B) candidate plaintexts for the final block. Since AES-ECB is deterministic with a fixed key, the attacker can build a lookup table: if they ever observe a ciphertext block matching one of the candidates in a known-plaintext scenario (e.g., from a leaked or guessed message), they can identify which final-block value corresponds to which ciphertext.
**Limitation:** Without the key, the attacker cannot compute E_K(candidate) directly. The attack requires collecting enough ciphertext final blocks to perform frequency analysis within the reduced plaintext space. With only 256 possibilities (B=1), convergence is fast given sufficient samples.
### 4.3 Cross-Sender Correlation
Senders with identical name lengths produce identical block alignments. Messages from "Alice" (N=5) and "Bobby" (N=5) place message text at the same byte offsets. If both send the same message, their blocks 1+ are identical ciphertext — **but only if they share the same channel PSK** (same AES key). On the same channel, this enables cross-sender frequency analysis within same-name-length groups.
### 4.4 Message Length Leakage
Ciphertext length = ⌈(5 + prefix_len + text_len) / 16⌉ × 16 bytes. This reveals the message text length within a 16-byte window (not 15, because the block count is the observable quantity). Not ECB-specific — any block cipher without constant-length padding leaks this.
### 4.5 Replay Attacks
`encryptThenMAC()` authenticates the ciphertext, but if the mesh doesn't track previously-seen packet MACs, captured packets can be replayed. The embedded timestamp may be checked for staleness — this requires firmware verification beyond the scope of this analysis.
### 4.6 No Forward Secrecy
Channel PSKs are static and shared among all participants. ECDH shared secrets for direct messages are also static (no ephemeral key exchange). Compromise of any key decrypts all past and future traffic encrypted under that key.
## 6. What Known-Plaintext Does NOT Achieve
AES-128 is designed to resist known-plaintext attacks. An attacker who knows the full plaintext and ciphertext of block 0 (or any block) **cannot**:
- Recover the AES key
- Decrypt other blocks encrypted under the same key
- Derive any information about other plaintexts from their ciphertexts
The ECB weakness is **determinism** (identical plaintext → identical ciphertext), not key recovery. The attacks in §5 exploit pattern matching and frequency analysis, not cryptanalysis of AES itself.
## 7. HMAC Key Reuse: Cryptographic Design Flaw
From `encryptThenMAC()`:
- AES key: `shared_secret[0..15]` (CIPHER_KEY_SIZE = 16)
- HMAC key: `shared_secret[0..31]` (PUB_KEY_SIZE = 32)
The AES key is the first half of the HMAC key. Both are derived from the same `shared_secret` — for channels, this is the PSK; for direct messages, the ECDH shared secret.
**Why this matters:**
1. **Violated key separation principle.** Standard practice dictates that encryption and authentication keys must be independent. Using overlapping portions of the same secret means a weakness in one mechanism could leak information relevant to the other.
2. **HMAC key reveals AES key.** If an attacker recovers the 32-byte HMAC key (e.g., through a side-channel attack on the HMAC computation), they automatically obtain the 16-byte AES key as a prefix.
3. **No key derivation function.** The shared_secret is used directly — no HKDF or similar KDF is applied to derive independent subkeys. This is a departure from cryptographic best practice (cf. RFC 5869).
**Practical impact:** In the current threat model (passive radio capture of LoRa packets), this is unlikely to be directly exploitable — HMAC-SHA256 does not leak its key through normal operation. However, it represents a structural weakness that compounds with any future vulnerability in either the AES or HMAC implementation.
## 8. TXT_MSG (Direct Message) Block Layout
Direct messages use a different plaintext structure (from `BaseChatMesh::composeMsgPacket()`):
```cpp
memcpy(temp, &timestamp, 4); // bytes 0-3: timestamp
temp[4] = (attempt & 3); // byte 4: attempt counter (0-3)
memcpy(&temp[5], text, text_len + 1); // bytes 5+: message text
// data_len = 5 + text_len (null terminator copied but not counted in length)
```
**Block layout for TXT_MSG:**
```
Block 0: [TS₀][TS₁][TS₂][TS₃][attempt][text bytes 0-10]
Block 1: [text bytes 11-26] (if message long enough)
```
Key differences from GRP_TXT:
- **No sender name in plaintext** — the sender is identified by the source hash in the unencrypted packet header, not in the encrypted payload.
- **Header is exactly 5 bytes** (4 timestamp + 1 attempt), always. No variable-length field.
- **11 bytes of message text fit in block 0** (vs. 9 N for GRP_TXT).
- **Encrypted with per-pair ECDH shared secret**, not a group PSK. Each sender-recipient pair has a unique key.
**ECB implications for TXT_MSG:**
- Block 0 is still protected by the timestamp accidental nonce.
- Blocks 1+ are deterministic, same as GRP_TXT — identical message text at the same offset produces identical ciphertext.
- However, frequency analysis is harder: each sender-recipient pair uses a different key, so the attacker can only correlate messages within a single pair. The message volume for any given pair is typically much lower than for a group channel.
- The fixed 5-byte header means block alignment is consistent across ALL direct messages (unlike GRP_TXT where alignment varies by sender name length). An attacker who compromises one ECDH key can build block frequency tables, but only for that specific pair.
## 9. Mitigations
### Priority 1: Switch to AES-128-CTR
Replace ECB with CTR mode. Use the existing 4-byte timestamp + a 4-byte per-message counter as the 8-byte nonce (padded to 16 bytes for the CTR block). Each byte of plaintext gets XORed with a unique keystream byte — eliminates all block-level determinism.
**Wire format change:** None if the nonce is derived from header fields already present. If an explicit counter is added, 4 bytes of overhead per message.
### Priority 2: Derive Independent Subkeys
Apply HKDF (or at minimum, two distinct SHA-256 hashes) to the shared_secret to produce independent AES and HMAC keys. This is a minimal code change:
```
aes_key = SHA256(shared_secret || "encrypt")[0..15]
hmac_key = SHA256(shared_secret || "authenticate")
```
### Priority 3: Constant-Length Padding
Pad all messages to a fixed block count (e.g., 4 blocks = 64 bytes) to eliminate length leakage. Expensive on LoRa — should be configurable per channel as a security-vs-bandwidth tradeoff.
### Priority 4: Replay Protection
Track seen packet HMACs within a time window. Reject messages with timestamps older than N minutes.
### Priority 5: Channel Key Rotation
Manual or automated periodic rotation of channel PSKs. Even monthly rotation limits the exposure window.
### Priority 6: Forward Secrecy
Ephemeral ECDH for direct messages. Significant protocol change but prevents retroactive decryption on key compromise.
## 10. Speculative: LLM-Assisted Analysis
> **This section is speculation, not formal analysis.** The claims below are plausible but unvalidated. They do not affect the formal findings in §19.
An LLM could reduce the sample size needed for block frequency analysis:
1. **Context-aware candidate generation:** Given a sender's known patterns (the sender name is recoverable from block 0's predictable prefix), an LLM could generate likely message continuations and predict which plaintext blocks to look for in the frequency tables.
2. **Conversational inference:** Timestamps + sender IDs + partially decoded messages could let an LLM reconstruct probable conversation flow, narrowing the search space for unknown blocks.
3. **Community-specific vocabulary:** Training on public mesh chat logs could yield common phrases and greeting patterns, further reducing the candidate plaintext space.
This does not change the fundamental requirement (blocks 1+ must repeat, or the final block must be in a small enough space for dictionary matching). It potentially reduces the number of captured messages needed for convergence, but no quantitative bound is established.
## 11. Conclusion
MeshCore's encryption has four vulnerabilities, ranked by practical exploitability:
### Vulnerability #1: PSK Brute-Force (Critical)
**No KDF + known-plaintext oracle = offline key recovery from a single packet.** Any channel using a human-memorable passphrase of ≤3 common words or ≤11 alphanumeric characters is recoverable in minutes to hours on commodity GPU hardware. This is the highest-priority threat because it requires minimal attacker capability (one captured packet), succeeds against the most common deployment pattern (human-chosen passphrases), and completely compromises channel confidentiality. See §1.
### Vulnerability #2: ECB Determinism (Medium)
**Blocks beyond the timestamp's reach are deterministic.** Identical plaintext at the same block offset always produces identical ciphertext. For GRP_TXT messages longer than ~9 N characters (where N is sender name length), this enables frequency analysis on blocks 1+. The partial final block, with its known zero-padding, is the strongest individual target. Exploitation requires hundreds of captured messages with repeated content — a higher bar than PSK brute-force. See §4–§5.
### Vulnerability #3: Key Material Reuse (Medium)
**AES and HMAC share the same key material** without a key derivation function. The AES key is a prefix of the HMAC key. This violates key separation and creates a structural dependency between the encryption and authentication mechanisms. See §7.
### Vulnerability #4: No Forward Secrecy (LowMedium)
**No forward secrecy, no key rotation, no replay protection.** These are independent of the above but compound the risk: a single key compromise (whether via brute-force or other means) exposes all past and future traffic encrypted under that key. See §9.
**Summary of recommended mitigations (in priority order):**
1. **(Critical)** Apply a memory-hard KDF (argon2id) to channel PSKs — §1.6
2. **(Critical)** Add per-channel salt — §1.6
3. **(High)** Switch from AES-128-ECB to AES-128-CTR — §9
4. **(High)** Derive independent AES and HMAC subkeys via HKDF — §9
5. **(Medium)** Constant-length padding, replay protection, key rotation — §9
6. **(Low)** Forward secrecy via ephemeral ECDH — §9
The timestamp in block 0 was not designed as a nonce and should not be relied upon as one.
+132
View File
@@ -0,0 +1,132 @@
# Proposal: Terminal/TUI Interface for CoreScope
**Status:** Approved for MVP
**Issue:** TBD
## Problem
CoreScope's web UI requires a browser. Operators managing remote mesh deployments often work over SSH — headless servers, Raspberry Pis, field laptops with spotty connectivity. They need to check mesh health, view packet flow, and diagnose issues without opening a browser.
## Vision
A terminal-based user interface (TUI) that connects to a CoreScope instance's API and renders key views directly in the terminal. Think `htop` for mesh networks.
---
## Expert Review
### Carmack (Performance / Data Flow)
- **bubbletea is fine for this.** The TUI is a thin API consumer — it's not processing 7.3M observations locally. The server does the heavy lifting; the TUI just renders summary data from `/api/observers/metrics/summary` (dozens of rows, not millions). No performance concern here.
- **WebSocket in a TUI — one gotcha:** reconnection. SSH sessions drop, networks flake. The TUI MUST have automatic reconnect with exponential backoff. Don't let a dropped WS kill the whole UI — show a "reconnecting..." status and keep the last-known state visible.
- **Memory footprint:** Should be trivial. The TUI holds at most a few hundred packets in a ring buffer for the live feed + summary stats. Target <20MB RSS. bubbletea itself is lightweight. The danger is unbounded packet accumulation — use a fixed-size ring buffer (e.g., last 1000 packets) for the live feed, not an ever-growing slice.
- **Batch WS messages.** Don't re-render on every single packet. Coalesce WS messages and re-render at most 10fps (every 100ms). Terminal rendering is slow — flooding it with updates causes flicker and CPU burn.
### Torvalds (Simplicity / Scope)
- **The scope is too big for an MVP.** Node detail view, sparklines, SSH server mode, multi-instance, export — delete all of that from M1. You need TWO views to prove this works: fleet dashboard table and live packet feed. That's it.
- **bubbletea vs tview:** bubbletea. Not because Elm-architecture is "clean" — because it's what the Go community actually uses now, the examples are good, and lipgloss makes table rendering trivial. Don't overthink this.
- **Over-engineering risk is HIGH.** The proposal describes 4 views, stretch features, and SSH server mode before a single line of code exists. Build the two-view demo. Ship it. Then decide what's next based on whether anyone actually uses it.
- **Same repo, `cmd/tui/`.** Don't create a separate repo for what's going to be 500 lines of Go initially. It shares the same API types. Keep it together.
- **Kill the "Open Questions" section.** Answer them: Target user = anyone with SSH access. M1 = dashboard + live feed. Same repo. Name = `corescope-tui`. Done. Stop discussing, start building.
### Doshi (Strategy / Prioritization)
- **This is an N (Neutral) feature, not an L.** It doesn't change CoreScope's trajectory — the web UI already works. But it's a solid N: it unlocks a real use case (SSH-only operators) and proves CoreScope's API is a proper platform, not just a web app backend.
- **The MVP that proves the concept:** Can an operator SSH into a Pi, run `corescope-tui --url http://analyzer:3000`, and immediately see fleet health + live packets? If yes, the concept is proven. Everything else (node detail, sparklines, alerting) is M2+.
- **Defer list:** Node detail view, RF sparklines, SSH server mode, multi-instance, export, mouse support, true-color fallback, alerting. ALL of these are M2 or later.
- **Pre-mortem — why would this fail?**
1. Nobody uses it because the web UI is good enough (likely for most users — that's fine, this is for the SSH-only niche)
2. The API doesn't return what the TUI needs in the right shape (validate this FIRST — curl the endpoints before writing any TUI code)
3. Scope creep kills the demo — someone adds "just one more view" and it's never done
- **Opportunity cost:** Low. This is a day of work for the MVP. The API already exists. The risk is spending a week on polish nobody asked for.
---
## MVP Definition (Demo Target)
**Goal:** A working two-view TUI that connects to any CoreScope instance and displays real-time mesh data in a terminal. Buildable in one focused session.
### View 1: Fleet Dashboard (default)
```
┌─ CoreScope TUI ──────────────────────────────────────────┐
│ Connected: analyzer.00id.net | Observers: 35 | ● Live │
├──────────────────────────────────────────────────────────┤
│ Observer │ Nodes │ Pkts/hr │ NF │ Status │
│ GY889 Repeater │ 142 │ 312 │ -112 │ ● active │
│ C0ffee SF │ 89 │ 201 │ -108 │ ● active │
│ ELC-ONNIE-RPT-1 │ 67 │ 156 │ -95 │ ▲ warning │
│ Bar Repeater │ 12 │ 3 │ -76 │ ▼ stale │
└──────────────────────────────────────────────────────────┘
Tab: [Dashboard] [Live Feed] q: quit ?: help
```
- **Data source:** `GET /api/observers/metrics/summary`
- **Refresh:** Poll every 5s (simple, no WS needed for this view)
- **Sort:** By observer name initially. Stretch: column sort with arrow keys.
### View 2: Live Packet Feed
```
┌─ Live Feed ──────────────────────────────────────────────┐
│ 14:32:01 ADVERT GY889 Repeater → 3 hops -112dB │
│ 14:32:02 GRP_TXT #test "hello world" → 5 hops -98dB │
│ 14:32:03 TXT_MSG [encrypted] → 2 hops -105dB │
│ 14:32:04 CHAN #sf "anyone on?" → 8 hops -91dB │
│ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ │
└──────────────────────────────────────────────────────────┘
Tab: [Dashboard] [Live Feed] p: pause q: quit
```
- **Data source:** WebSocket (`/ws`)
- **Buffer:** Ring buffer, last 500 packets max
- **Render:** Coalesce updates, re-render at most 10fps
- **Reconnect:** Auto-reconnect with exponential backoff (1s, 2s, 4s, max 30s)
### What's NOT in MVP
- Node detail view
- RF sparklines
- SSH server mode (`--serve-ssh`)
- Multi-instance support
- Export to CSV/JSON
- Mouse support
- Alerting / terminal bell
- Color theme configuration
- Custom filters (/ to filter)
### Technical Decisions (Resolved)
| Question | Answer |
|---|---|
| Target user | SSH operators, power users, field techs |
| Library | bubbletea + lipgloss |
| Location | `cmd/tui/` in same repo |
| Binary name | `corescope-tui` |
| Min terminal | 256-color, 80x24 |
| State | Stateless — pure API consumer, no local DB |
### Implementation Plan
1. Scaffold `cmd/tui/main.go` — flag parsing (`--url`), bubbletea app init
2. Fleet dashboard model — fetch `/api/observers/metrics/summary`, render table
3. Live feed model — WebSocket connect, ring buffer, packet rendering
4. Tab switching between views
5. Status bar (connection state, help hints)
6. Test against `https://analyzer.00id.net`
---
## Future Milestones (post-MVP, not scheduled)
### M2: Navigation & Detail
- Node detail view (select observer → see its packets/neighbors)
- Keyboard navigation (j/k, Enter, Esc)
- `/` to filter packets
### M3: Visualization
- RF noise floor sparklines (`▁▂▃▅▇█`)
- Health history over time
- Color theme support
### M4: Advanced
- SSH server mode (`--serve-ssh :2222`)
- Multi-instance tabs
- Export current view to stdout (CSV/JSON)
- Desktop notifications on anomalies
+135
View File
@@ -0,0 +1,135 @@
# CoreScope v3.5.0 🚀
The "stop building from source and start analyzing your mesh" release. 95 commits.
---
## 🐳 Pre-built Docker Images
CoreScope now ships as a ready-to-run Docker image on GitHub Container Registry. No cloning, no building, no dependencies — just pull and run.
```bash
docker run -d --name corescope -p 80:80 -p 443:443 -p 1883:1883 \
-v corescope-data:/app/data \
ghcr.io/kpa-clawbot/corescope:v3.5.0
```
**Using HTTPS with a custom domain?** Mount your Caddyfile and certs directory:
```bash
docker run -d --name corescope -p 80:80 -p 443:443 -p 1883:1883 \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:v3.5.0
```
Caddy auto-provisions Let's Encrypt certs. Your Caddyfile just needs:
```
yourdomain.example.com {
reverse_proxy localhost:3000
}
```
That's it. Zero config required — MQTT broker, Caddy HTTPS, and SQLite are built in.
**Already running CoreScope?**
```bash
# 1. Find your running container name
docker ps --format '{{.Names}}'
# 2. Stop and remove it
docker stop <container-name> && docker rm <container-name>
# 3. Pull the pre-built image
docker pull ghcr.io/kpa-clawbot/corescope:v3.5.0
# 4. Run with your existing data directory
docker run -d --name corescope -p 80:80 -p 443:443 -p 1883:1883 \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:v3.5.0
```
Your data volume stays. Nothing to migrate.
Tags: `v3.5.0` (this release) · `latest` (latest tagged release) · `edge` (master tip, for testing). Env: `DISABLE_CADDY=true` / `DISABLE_MOSQUITTO=true` if you bring your own.
---
## ⚡ 83% Faster
35 performance commits. Packets endpoint p50 dropped from 16.7ms → 2.7ms. Server now serves HTTP within 2 minutes on *any* DB size — async background backfill means you're never staring at a loading screen. N+1 API calls killed everywhere. Prefix map memory cut 10x. WebSocket renders batched via rAF.
---
## 🔬 RF Health Dashboard
New Analytics tab. Per-observer noise floor as color-coded columns (green/yellow/red), airtime utilization, error rates, battery levels. Click any observer for the full breakdown. Region-filterable. This is the beginning of making CoreScope more than just a packet viewer.
---
## 🗺️ See Where Traces Actually Go
Send a trace → watch it on the live map. Solid animated line shows how far it got. Dashed ghost shows where it didn't reach. Finally know *where* your trace failed, not just *that* it failed.
---
## 📊 Things That Were Lying To You
- "By Repeaters" was counting companions. Fixed.
- Zero-hop adverts claimed "1 byte hash" when the hash size was unknowable. Fixed.
- "Packets through this node" showed packets through a *different* node with the same prefix. Fixed — now uses the neighbor affinity graph.
- Table sorting on nodes/neighbors/observers silently did nothing. Fixed.
---
## 🔗 Deep Links · 🎨 Channel Colors · 📱 Mobile · 🔑 Security
**Deep links** — every page state goes in the URL. Share a link to a specific node, filter, or analytics tab.
**Channel colors** — click the color dot next to any channel, pick from 8 colors, see it highlighted across the feed. Persists in localStorage.
**Distance units** — km, miles, or auto-detect from locale. Customizer → Display.
**Mobile** — 44px touch targets, ARIA labels, responsive breakpoints.
**Security** — weak API keys rejected at startup. License: GPL v3.
---
## 📡 Full API Documentation
Every endpoint is now documented with an auto-generated OpenAPI 3.0 spec — always in sync with the running server.
- **Interactive Swagger UI:** [analyzer.00id.net/api/docs](https://analyzer.00id.net/api/docs) — browse and test all 40+ endpoints
- **Machine-readable spec:** [analyzer.00id.net/api/spec](https://analyzer.00id.net/api/spec) — import into Postman, Insomnia, or use for bot/integration development
On your own instance: `/api/docs` and `/api/spec`.
---
## 🐛 14 Bugs Squashed
Live map crash, zero-hop hash lies, animation freezes, repeater miscounts, prefix collisions, dead channel picker, invisible buttons, broken sorting, memory leak, and more.
---
## Upgrade
```bash
docker stop <container-name> && docker rm <container-name>
docker pull ghcr.io/kpa-clawbot/corescope:v3.5.0
# HTTP only:
docker run -d --name corescope -p 80:80 -p 1883:1883 \
-v /your/data:/app/data \
ghcr.io/kpa-clawbot/corescope:v3.5.0
# With HTTPS (custom domain):
docker run -d --name corescope -p 80:80 -p 443:443 -p 1883:1883 \
-v /your/data:/app/data \
-v /your/Caddyfile:/etc/caddy/Caddyfile:ro \
-v /your/caddy-data:/data/caddy \
ghcr.io/kpa-clawbot/corescope:v3.5.0
```
First start backfills `resolved_path` in the background. No downtime. No breaking changes.
+148
View File
@@ -0,0 +1,148 @@
# Channel Color Highlighting Spec
**Status:** Proposed
**Issue:** [#271](https://github.com/Kpa-clawbot/CoreScope/issues/271)
**Author:** Stinkmeaner (AI)
**Date:** 2026-04-05
## Problem
When monitoring multiple active hash channels simultaneously on the Live tab, all `GRP_TXT` traffic renders identically — same color, same styling. Users tracking specific channels (e.g. `#wardriving`) cannot visually distinguish their traffic from other channel activity without reading each row's channel field.
## Solution
Allow users to assign custom highlight colors to specific hash channels. Colors propagate across the Live feed, map animations, and timeline. Unassigned channels retain the default `GRP_TXT` styling.
### Data Model
**Storage:** Single `localStorage` key `live-channel-colors`
```json
{
"#wardriving": "#ef4444",
"#meshnet": "#3b82f6"
}
```
- Keyed by resolved channel name (e.g. `#wardriving`) or raw hash prefix if unresolved
- Included in customizer theme export/import for portability
- Maximum ~16 assignments (no hard limit, but UI should discourage excess — see Edge Cases)
### Channel Matching
- Match on the packet's `channel` or `group` field
- Handle both resolved channel names and raw hash prefixes
- Only applies to `GRP_TXT` packet types — other types retain their existing `TYPE_COLORS` styling
### Visual Treatment
**Feed rows (primary):**
- 4px colored left border
- Subtle background tint: channel color at 810% opacity
- Text color unchanged — contrast must remain accessible
**Map animations:**
- Packet arcs use the assigned channel color instead of default `TYPE_COLORS.GRP_TXT`
- Node markers retain role-based coloring (channel color does NOT override node markers)
**Timeline sparkline:**
- Dots/bars colored per channel assignment
- Unassigned channels use default color
**Auto-legend:**
- Generated from active assignments
- Displayed near the feed header
- Color swatch + channel name, compact horizontal layout
### Configuration UI
**Quick assign (primary workflow):**
- Right-click (long-press on mobile) a channel name in the Live feed
- Color picker popover with ~12 preset swatches + custom hex input
- "Clear" button to remove assignment
**Customizer panel (management):**
- New "Channel Colors" section under existing "Packet Type Colors"
- Lists all assigned channels with color swatches
- Add/edit/remove individual assignments
- "Clear All" button
- Synced with theme export/import
### Priority Rules
| Context | Color source |
|---------|-------------|
| Feed row background/border | Channel color (if assigned), else default |
| Feed row text | Always default (no override) |
| Map packet arcs | Channel color (if assigned), else `TYPE_COLORS.GRP_TXT` |
| Map node markers | Always role color (no override) |
| Timeline dots | Channel color (if assigned), else default |
## Edge Cases
- **10+ colors:** At ~10 simultaneous assignments, colors become hard to distinguish. The UI should show a soft warning ("Many colors assigned — consider clearing unused ones") but not block the user.
- **Color conflicts with role/type colors:** Channel color takes priority for feed row highlighting only. Role colors remain authoritative for node markers.
- **Removal:** Clearing a channel color reverts to default styling immediately — no page refresh needed.
- **Non-GRP_TXT packets:** Channel color never applied. These packets have no channel association.
- **Customizer rework (#288):** If the customizer rework lands first, the Channel Colors section should follow the new single-delta-object pattern (`cs-theme-overrides`). If it hasn't landed, use the standalone `live-channel-colors` key and migrate later.
- **Dark/light mode:** Channel colors are mode-independent (same color in both modes). The 810% opacity tint ensures readability in both themes.
## Milestones
### M1: Core model + feed row highlighting
- `localStorage` read/write for `live-channel-colors`
- Feed row rendering: left border + background tint
- Unit tests for storage CRUD and color application logic
### M2: Quick-assign UI
- Right-click / long-press context menu on channel names
- Color picker popover with presets + custom hex
- Clear button
- Playwright E2E test for assign/clear workflow
### M3: Map animation integration
- Packet arc color lookup from channel assignments
- Falls back to `TYPE_COLORS.GRP_TXT` when unassigned
- Visual verification via browser screenshot
### M4: Customizer section + export/import
- "Channel Colors" management panel in customizer
- Include channel colors in theme export JSON
- Import restores channel colors
- Unit tests for export/import round-trip
### M5: Timeline coloring + auto-legend
- Timeline sparkline uses channel colors
- Auto-legend renders near feed header
- Playwright E2E for legend visibility
## Testing
| Level | What | How |
|-------|------|-----|
| Unit | Storage CRUD, color lookup, merge with defaults | `test-frontend-helpers.js` via `vm.createContext` |
| Unit | Export/import round-trip with channel colors | Same |
| E2E | Quick-assign popover, color applied to feed rows | Playwright against localhost |
| E2E | Customizer channel colors section | Playwright |
| E2E | Legend appears when ≥1 channel colored | Playwright |
| Visual | Map arcs colored, dark/light mode readability | Browser screenshot |
## Expert Review Notes
### Tufte (Visualization)
- **Left border + tint is sound.** The 4px border is data-ink (encodes channel identity). The tint at 810% opacity provides grouping without overwhelming the data. This is information encoding, not decoration.
- **Risk at scale:** Beyond ~8 colors, perceptual distinguishability drops sharply. The spec correctly warns but doesn't enforce. Consider using a curated palette of maximally-distinct colors (like ColorBrewer qualitative sets) as the preset swatches rather than a free-form picker.
- **Auto-legend is correct:** Direct labeling on every row would be redundant (channel name already in the row). A compact legend near the feed is the right balance — it teaches the encoding once.
- **No chartjunk introduced.** The visual treatment adds information (channel identity) without decorative excess.
### Torvalds (Code Quality)
- **localStorage is fine** for user preferences with <1KB payloads. No need for IndexedDB or server-side storage.
- **5 milestones is appropriate.** Each is independently shippable and testable. No milestone depends on speculation about future milestones.
- **Watch the customizer coupling.** If #288 lands, the `live-channel-colors` key should merge into `cs-theme-overrides`. Design the read/write functions to abstract the storage key so migration is a one-line change, not a rewrite.
- **Keep the color picker simple.** Don't build a custom color picker — use `<input type="color">` with preset swatch buttons. The browser's native picker is fine.
### Doshi (Product Strategy)
- **This is N (Neutral).** It's a genuine usability improvement for multi-channel monitoring, but it doesn't change CoreScope's trajectory. It won't attract new users or unlock new use cases — it makes existing power users slightly more efficient.
- **Opportunity cost is low.** Each milestone is small (~1-2 hours of work). The total investment is modest.
- **5 milestones is fine** given each is small. Shipping M1+M2 alone delivers 80% of the value. M3M5 are polish. Consider M1+M2 as the MVP gate — if nobody uses channel colors after M2, stop there.
- **Pre-mortem:** This fails if users rarely monitor 2+ channels simultaneously, making the problem theoretical. Validate that multi-channel monitoring is a real workflow before M3.
+568
View File
@@ -0,0 +1,568 @@
# Customizer Rework Spec
## Overview
The current customizer (`public/customize.js`) suffers from fundamental state management issues documented in [issue #284](https://github.com/Kpa-clawbot/CoreScope/issues/284). State is scattered across 7 localStorage keys, CSS updates bypass the data layer, and there's no single source of truth for the effective configuration.
This spec defines a clean rework based on event-driven state management with a single data flow path. The goal: predictable state, minimal storage footprint, portable config format, and zero ambiguity about which values are active and why.
## Design Decisions
These are agreed and final. Do not reinterpret or deviate.
1. **Three state layers:** server defaults (immutable after fetch), user overrides (delta in localStorage), effective config (computed via merge, never stored directly).
2. **Single data flow:** user action → debounce (~300ms) → write delta to localStorage → read back from localStorage → merge with server defaults → apply CSS variables. No shortcuts, no optimistic CSS updates (see Decision #12 for the one exception).
3. **One localStorage key:** `cs-theme-overrides` — replaces the current 7 scattered keys (`meshcore-user-theme`, `meshcore-timestamp-mode`, `meshcore-timestamp-timezone`, `meshcore-timestamp-format`, `meshcore-timestamp-custom-format`, `meshcore-heatmap-opacity`, `meshcore-live-heatmap-opacity`).
4. **Universal format:** same shape as the server's `ThemeResponse` plus additional keys. Works identically for user export, admin `theme.json`, and user import.
5. **User overrides always win** in merge — `merge(serverDefaults, userOverrides)` = effective config.
6. **Override indicator:** shown in customizer panel ONLY when override value differs from current server default.
7. **No silent pruning:** overrides stay in localStorage until the user explicitly resets them (per-field reset or full reset). The delta may contain values that happen to match current server defaults — that's fine. User intent is preserved; nothing silently disappears.
8. **Per-field reset:** remove a single key from the delta → re-merge → re-apply CSS.
9. **Full reset:** `localStorage.removeItem('cs-theme-overrides')` → re-merge (effective = server defaults) → re-apply CSS.
10. **Export = dump delta object as JSON download. Import = validate shape, write to localStorage, trigger re-merge.**
11. **No CSS magic:** CSS variables ONLY update after the localStorage round-trip completes. No optimistic updates (see Decision #12 for the one exception).
12. **Color picker optimistic CSS exception:** For continuous inputs (color pickers, sliders), CSS is updated optimistically during `input` events for visual responsiveness. The localStorage write only happens on `change` event (mouseup/blur). On `change`, the full pipeline runs: write → read → merge → apply (which will match the optimistic state). If the user refreshes mid-drag before `change` fires, the change is lost — this is acceptable. This is the ONLY exception to the localStorage-first rule.
## Dark/Light Mode
The customizer treats light and dark mode as separate override sections:
- **`theme`** stores light mode color overrides.
- **`themeDark`** stores dark mode color overrides.
- When the user changes a color in the customizer, it writes to whichever section matches their current mode: `theme` if light, `themeDark` if dark.
- The dark/light mode toggle preference (`meshcore-theme` localStorage key) is **separate** from the delta object. It is a view preference, not a customization — it is not stored in `cs-theme-overrides`.
- The customizer UI shows color fields for the currently active mode only. Switching modes re-renders the color fields with values from the matching section.
## Presets
The existing preset themes are preserved and flow through the standard pipeline:
**Available presets:** Default, Ocean, Forest, Sunset, Monochrome.
**How presets work:**
- Clicking a preset writes its values to localStorage via the same pipeline as any other change: preset data → `writeOverrides()` → read back → merge → apply CSS.
- Presets are NOT special — they are pre-built delta objects applied through the standard flow.
- Each preset contains both `theme` (light) and `themeDark` (dark) sections, plus any other overrides the preset defines (e.g., `nodeColors`).
- **"Reset to Default"** = clear all overrides (equivalent to full reset: `localStorage.removeItem('cs-theme-overrides')` → re-merge → apply).
**Preset data format:** Same shape as the delta object. Example:
```json
{
"theme": {
"accent": "#0077b6",
"navBg": "#03045e",
"background": "#f0f7fa"
},
"themeDark": {
"accent": "#48cae4",
"navBg": "#03045e",
"background": "#0a1929"
}
}
```
Applying a preset **replaces** the entire delta (it's a `writeOverrides(presetData)`, not a merge onto existing overrides). The user can then further customize individual fields on top.
## Data Model
### Delta Object Format
The user override delta is a sparse object — it only contains fields the user has explicitly changed. The shape mirrors the server's `ThemeResponse` (from `/api/config/theme`) plus additional client-only sections:
```json
{
"branding": {
"siteName": "string — site name override",
"tagline": "string — tagline override",
"logoUrl": "string — custom logo URL",
"faviconUrl": "string — custom favicon URL"
},
"theme": {
"accent": "string — CSS color, light mode accent",
"accentHover": "string — CSS color, light mode accent hover",
"navBg": "string — CSS color, nav background",
"navBg2": "string — CSS color, nav secondary background",
"navText": "string — CSS color, nav text",
"navTextMuted": "string — CSS color, nav muted text",
"background": "string — CSS color, page background",
"text": "string — CSS color, body text",
"textMuted": "string — CSS color, muted text",
"border": "string — CSS color, borders",
"surface1": "string — CSS color, surface level 1",
"surface2": "string — CSS color, surface level 2",
"cardBg": "string — CSS color, card backgrounds",
"contentBg": "string — CSS color, content area background",
"detailBg": "string — CSS color, detail pane background",
"inputBg": "string — CSS color, input backgrounds",
"rowStripe": "string — CSS color, alternating row stripe",
"rowHover": "string — CSS color, row hover highlight",
"selectedBg": "string — CSS color, selected row background",
"statusGreen": "string — CSS color, healthy status",
"statusYellow": "string — CSS color, degraded status",
"statusRed": "string — CSS color, critical status",
"font": "string — CSS font-family for body text",
"mono": "string — CSS font-family for monospace"
},
"themeDark": {
"/* same keys as theme — dark mode overrides */"
},
"nodeColors": {
"repeater": "string — CSS color",
"companion": "string — CSS color",
"room": "string — CSS color",
"sensor": "string — CSS color",
"observer": "string — CSS color"
},
"typeColors": {
"ADVERT": "string — CSS color",
"GRP_TXT": "string — CSS color",
"TXT_MSG": "string — CSS color",
"ACK": "string — CSS color",
"REQUEST": "string — CSS color",
"RESPONSE": "string — CSS color",
"TRACE": "string — CSS color",
"PATH": "string — CSS color",
"ANON_REQ": "string — CSS color"
},
"home": {
"heroTitle": "string",
"heroSubtitle": "string",
"steps": "[array of {emoji, title, description}]",
"checklist": "[array of strings]",
"footerLinks": "[array of {label, url}]"
},
"timestamps": {
"defaultMode": "string — 'ago' | 'absolute'",
"timezone": "string — 'local' | 'utc'",
"formatPreset": "string — 'iso' | 'iso-seconds' | 'locale'",
"customFormat": "string — custom strftime-style format"
},
"heatmapOpacity": "number — 0.0 to 1.0",
"liveHeatmapOpacity": "number — 0.0 to 1.0"
}
```
**Rules:**
- All sections and keys are optional. An empty object `{}` means "no overrides."
- The `timestamps`, `heatmapOpacity`, and `liveHeatmapOpacity` keys are client-only extensions — not part of the server's `ThemeResponse`, but included in the universal format for portability.
### localStorage Key
**Key:** `cs-theme-overrides`
**Value:** JSON string of the delta object above.
**Absent key** = no overrides = effective config equals server defaults.
### Dark/Light Mode Preference
**Key:** `meshcore-theme`
**Value:** `"dark"` or `"light"` (or absent = follow system preference).
**This key is NOT part of the delta object.** It controls which mode is active, not which colors are used. The delta stores overrides for both modes independently in `theme` and `themeDark`.
## Data Flow Diagrams
### Page Load
```
┌─────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Fetch │ │ Read localStorage │ │ Migration check │
│ /api/config/ │ │ cs-theme-overrides│ │ (one-time) │
│ theme │ └────────┬─────────┘ └────────┬────────┘
└──────┬──────┘ │ │
│ │ ┌────────────────────┘
▼ ▼ ▼
serverDefaults userOverrides (possibly migrated)
│ │
▼ ▼
┌──────────────────────────────────────┐
│ computeEffective(server, userOverrides) │
└──────────────┬───────────────────────┘
┌──────────────────────────────────────┐
│ window.SITE_CONFIG = effective │ ← atomic assignment
└──────────────┬───────────────────────┘
┌──────────────────────┐
│ applyCSS(effective) │ ← sets CSS vars on :root for current mode
└──────────────────────┘
┌──────────────────────────────┐
│ dispatch 'theme-changed' │ ← bare signal, no payload
└──────────────────────────────┘
```
### User Change (e.g., picks new accent color)
```
User action (input/click)
debounce(300ms)
setOverride('theme', 'accent', '#ff0000')
├─► readOverrides() ← read current delta from localStorage
│ │
│ ▼
├─► update delta object ← set delta.theme.accent = '#ff0000'
│ │
│ ▼
├─► writeOverrides(delta) ← serialize & write to localStorage
│ │
│ ▼
├─► readOverrides() ← read BACK from localStorage (round-trip)
│ │
│ ▼
├─► computeEffective(server, delta)
│ │
│ ▼
├─► window.SITE_CONFIG = effective ← atomic assignment
│ │
│ ▼
└─► applyCSS(effective) ← CSS vars updated on :root
dispatch 'theme-changed'
```
**Color picker / slider exception:** During continuous `input` events (drag), CSS is updated optimistically (directly setting `--var` on `:root`) without the localStorage round-trip. The full pipeline above only runs on the `change` event (mouseup/blur).
### Per-Field Reset
```
User clicks reset icon on a field
clearOverride('theme', 'accent')
├─► readOverrides()
├─► delete delta.theme.accent
├─► if delta.theme is empty, delete delta.theme
├─► writeOverrides(delta)
├─► readOverrides() ← round-trip
├─► computeEffective(server, delta)
├─► window.SITE_CONFIG = effective
└─► applyCSS(effective)
dispatch 'theme-changed'
```
### Full Reset
```
User clicks "Reset All"
localStorage.removeItem('cs-theme-overrides')
computeEffective(server, {}) ← no overrides = server defaults
window.SITE_CONFIG = effective
applyCSS(effective)
dispatch 'theme-changed'
```
### Export
```
User clicks "Export"
readOverrides()
JSON.stringify(delta, null, 2)
trigger download as .json file
```
### Import
```
User selects .json file
parse JSON
validateShape(parsed) ← check structure, validate values
├─► invalid → show error, abort
▼ valid
writeOverrides(parsed)
readOverrides() ← round-trip
computeEffective(server, delta)
window.SITE_CONFIG = effective
applyCSS(effective)
dispatch 'theme-changed'
```
## Function Signatures
### `readOverrides() → object`
Reads `cs-theme-overrides` from localStorage, parses as JSON. Returns empty object `{}` on missing key, parse error, or non-object value. Never throws.
### `writeOverrides(delta: object) → void`
Serializes `delta` to JSON and writes to `cs-theme-overrides` in localStorage. If `delta` is empty (`{}`), removes the key entirely.
**Validation on write:**
- Color values must match: `#hex` (3, 4, 6, or 8 digit), `rgb()`, `rgba()`, `hsl()`, `hsla()`, or CSS named colors. Invalid color values are rejected (not written) with `console.warn`.
- Numeric values (`heatmapOpacity`, `liveHeatmapOpacity`) must be finite numbers in the range 01. Invalid values are rejected with `console.warn`.
- Timestamp enum values are validated against known options (`defaultMode`: `'ago'`/`'absolute'`; `timezone`: `'local'`/`'utc'`; `formatPreset`: `'iso'`/`'iso-seconds'`/`'locale'`). Invalid values are rejected with `console.warn`.
**Quota error handling:**
- Wrap `localStorage.setItem` in try/catch.
- On `QuotaExceededError`: show a visible warning to the user ("Storage full — changes may not be saved"), log to console.
- Do NOT silently swallow the error.
### `computeEffective(serverConfig: object, userOverrides: object) → object`
Deep merges `userOverrides` onto `serverConfig`. For each section (e.g., `theme`, `nodeColors`), if `userOverrides` has the section, its keys override the corresponding `serverConfig` keys. Top-level non-object keys (e.g., `heatmapOpacity`) are directly overridden.
Returns a new object — neither input is mutated.
**Merge rules:**
- Object sections: shallow merge per section (`Object.assign({}, server.theme, user.theme)`)
- Array sections (e.g., `home.steps`): full replacement (user array wins entirely, no element-level merge)
- Scalar sections (e.g., `heatmapOpacity`): direct replacement
After computing the effective config, writes it to `window.SITE_CONFIG` atomically (single assignment, not piecemeal mutations).
### `applyCSS(effectiveConfig: object) → void`
Maps effective config values to CSS custom properties on `:root`. Behavior:
1. Reads the current mode (light/dark) from the `meshcore-theme` localStorage key, falling back to system preference (`prefers-color-scheme`).
2. Applies the matching section's values: `theme` for light mode, `themeDark` for dark mode.
3. Also applies mode-independent values: node colors as `--node-{role}`, type colors as `--type-{name}`, font families as `--font-body` and `--font-mono`.
4. Does NOT generate dual CSS rule blocks — only the current mode's values are applied to `:root`.
5. On dark/light mode toggle, `applyCSS` is called again to re-apply the correct section.
Updates the `<style>` element (create if absent, reuse if present). Dispatches a `theme-changed` CustomEvent on `window` after applying.
### `theme-changed` Event
- `theme-changed` is a bare `CustomEvent` with no payload (matches current behavior).
- After each merge cycle, the effective config is written to `window.SITE_CONFIG` atomically (single assignment).
- `window.SITE_CONFIG` is the canonical readable source for effective config throughout the app. All existing listeners that read from `SITE_CONFIG` continue to work without changes.
### `setOverride(section: string, key: string, value: any) → void`
Sets a single override. For nested sections (e.g., `section='theme'`, `key='accent'`), sets `delta[section][key] = value`. For top-level scalars (e.g., `section=null`, `key='heatmapOpacity'`), sets `delta[key] = value`.
Follows the full data flow: read → update → write → read-back → merge → apply CSS → dispatch `theme-changed`. Debounced at ~300ms (the debounce wraps the write-through-to-CSS portion).
### `clearOverride(section: string, key: string) → void`
Removes a single key from the delta. If the section becomes empty after removal, removes the section too. Triggers the full data flow (no debounce — resets should feel instant).
### `migrateOldKeys() → object | null`
One-time migration. Checks for any of the 7 legacy localStorage keys. If found:
1. Reads all legacy values
2. Maps them into the new delta format (see Migration Plan)
3. Writes the merged delta to `cs-theme-overrides`
4. Removes all 7 legacy keys
5. Returns the migrated delta
Returns `null` if no legacy keys found.
### `validateShape(obj: any) → { valid: boolean, errors: string[] }`
Validates that an imported object conforms to the expected shape:
- Must be a plain object
- Top-level keys must be from the known set: `branding`, `theme`, `themeDark`, `nodeColors`, `typeColors`, `home`, `timestamps`, `heatmapOpacity`, `liveHeatmapOpacity`
- Section values must be objects (where expected) or correct scalar types
- Color values are validated: must match `#hex` (3, 4, 6, or 8 digit), `rgb()`, `rgba()`, `hsl()`, `hsla()`, or CSS named colors
- Numeric values (`heatmapOpacity`, `liveHeatmapOpacity`) must be finite numbers in range 01
- Timestamp enum values validated against known options
Unknown top-level keys cause a warning but don't fail validation (forward compatibility).
## Migration Plan
On first page load, before the normal init flow:
1. Check if `cs-theme-overrides` already exists → if yes, skip migration.
2. Check if ANY of the 7 legacy keys exist in localStorage.
3. If legacy keys found, build a delta object using the exact mapping below:
### Field-by-Field Migration Mapping
```
meshcore-user-theme (JSON) → parse, map directly:
.branding → delta.branding
.theme → delta.theme
.themeDark → delta.themeDark
.nodeColors → delta.nodeColors
.typeColors → delta.typeColors
.home → delta.home
(any other keys are dropped)
meshcore-timestamp-mode → delta.timestamps.defaultMode
meshcore-timestamp-timezone → delta.timestamps.timezone
meshcore-timestamp-format → delta.timestamps.formatPreset
meshcore-timestamp-custom-format → delta.timestamps.customFormat
meshcore-heatmap-opacity → delta.heatmapOpacity (parseFloat)
meshcore-live-heatmap-opacity → delta.liveHeatmapOpacity (parseFloat)
```
4. Write the assembled delta to `cs-theme-overrides`.
5. Delete all 7 legacy keys.
6. Continue with normal init.
**Edge cases:**
- If `meshcore-user-theme` contains invalid JSON, skip it (log a warning to console).
- If a legacy value is empty string or null, skip that field.
- Migration runs exactly once — the presence of `cs-theme-overrides` (even as `{}`) prevents re-migration.
## `allowCustomFormat` — User Preferences Trump
The server-side `allowCustomFormat` gate is not enforced client-side. If a user imports a delta with a custom format, it's applied regardless. The server controls what formats are available in the UI (whether the custom format input field is shown), but does not block stored preferences.
## Override Indicator UX
In the customizer panel, each field that has an active override (value differs from server default) shows a visual indicator:
- **Indicator:** A small dot or icon (e.g., `●` or a reset arrow `↺`) adjacent to the field label.
- **Color:** Use the accent color to draw attention without being noisy.
- **Behavior:** Clicking the indicator resets that single field (calls `clearOverride`).
- **Tooltip:** "Reset to server default" or "This value differs from the server default."
- **Absence:** Fields matching the server default show no indicator — clean and minimal.
**Section-level indicator:** If any field in a section (e.g., "Theme Colors") is overridden, the tab/section header shows a count badge (e.g., "Theme Colors (3)").
**"Reset All" button:** Always visible at bottom of panel. Confirms before executing (`localStorage.removeItem` + re-merge).
## UX Requirements
### Browser-Local Banner
The customizer panel must display a persistent, always-visible notice:
> **"These settings are saved in your browser only and don't affect other users."**
This is NOT a tooltip, NOT a dismissible popup — it must be always visible in the panel header or footer area. Users must understand at a glance that their changes are local.
### Auto-Save Indicator
Show a persistent status in the customizer panel footer, Google Docs style — subtle but always present:
- **Default state:** "All changes saved" (muted text)
- **During debounce:** "Saving..." (muted text)
- **On quota error:** "⚠️ Storage full — changes may not be saved" (red text, persistent until resolved)
The indicator reflects the actual state of the localStorage write, not just the UI action.
## Server Compatibility
The delta format is intentionally shaped to be a valid subset of the server's `theme.json` admin config file. This means:
- **User export → admin import:** An admin can take a user's exported JSON and drop it into `theme.json` as server defaults. The `timestamps`, `heatmapOpacity`, and `liveHeatmapOpacity` keys are ignored by the current server (it doesn't read them from `theme.json`), but they don't cause errors.
- **Admin config → user import:** A `theme.json` file can be imported as user overrides. Unknown server-only keys are ignored by the client.
- **Round-trip safe:** Export → import produces identical delta (assuming no server default changes between operations).
The server's `ThemeResponse` struct currently returns: `branding`, `theme`, `themeDark`, `nodeColors`, `typeColors`, `home`. The client-only extensions (`timestamps`, `heatmapOpacity`, `liveHeatmapOpacity`) are additive — they extend the format without conflicting.
## Testing Requirements
### Unit Tests (Node.js, no browser required)
1. **`readOverrides`**
- Returns `{}` when key is absent
- Returns `{}` when key contains invalid JSON
- Returns `{}` when key contains a non-object (string, array, number)
- Returns parsed object when key contains valid JSON object
2. **`writeOverrides`**
- Writes serialized JSON to localStorage
- Removes key when delta is empty `{}`
- Round-trips correctly (write → read = identical object)
- Rejects invalid color values with console.warn
- Rejects out-of-range numeric values with console.warn
- Rejects invalid timestamp enum values with console.warn
- Handles QuotaExceededError gracefully (warns user, does not throw)
3. **`computeEffective`**
- Returns server defaults when overrides is `{}`
- Overrides a single key in a section
- Overrides multiple keys across sections
- Does not mutate either input
- Handles missing sections in overrides gracefully
- Array values (e.g., `home.steps`) are fully replaced, not merged
- Top-level scalars (`heatmapOpacity`) are directly replaced
4. **`setOverride` / `clearOverride`**
- Setting a value stores it in the delta
- Clearing a key removes it from delta
- Clearing the last key in a section removes the section
- Full data flow executes (CSS vars updated)
5. **`migrateOldKeys`**
- Migrates all 7 keys correctly using exact field mapping
- Handles partial migration (only some keys present)
- Handles invalid JSON in `meshcore-user-theme`
- Removes all legacy keys after migration
- Skips migration if `cs-theme-overrides` already exists
- Returns null when no legacy keys found
- Drops unknown keys from `meshcore-user-theme`
6. **`validateShape`**
- Accepts valid delta objects
- Accepts empty object
- Rejects non-objects (string, array, null)
- Warns on unknown top-level keys (doesn't reject)
- Validates section types (object vs scalar)
- Rejects invalid color values
- Rejects out-of-range opacity values
- Rejects invalid timestamp enum values
### Browser/E2E Tests (Playwright)
1. **Customizer opens and shows current values** — fields reflect effective config.
2. **Changing a color updates CSS variable** — after debounce, `:root` has new value.
3. **Override indicator appears** when value differs from server default.
4. **Per-field reset** removes override, reverts to server default, indicator disappears.
5. **Full reset** clears all overrides, all fields show server defaults.
6. **Export** downloads a JSON file with current delta.
7. **Import** applies overrides from uploaded JSON file.
8. **Migration** — set legacy keys, reload, verify they're migrated and removed.
9. **Preset application** — clicking a preset applies its colors, fields update.
10. **Dark/light mode toggle** — switching mode re-applies correct section's CSS vars.
11. **Browser-local banner** — verify persistent notice is visible in customizer panel.
12. **Auto-save indicator** — verify status text updates during and after changes.
## What's NOT In Scope
- **Undo/redo stack** — could be added as P2. For v1, per-field reset to server default is the only revert mechanism.
- **Cross-tab synchronization** — two tabs editing simultaneously may clobber each other's changes. Acceptable for v1.
- **Server-side timestamp config** (`allowCustomFormat` gate) — remains server-only, not exposed in the customizer delta. The server controls UI availability but does not block stored preferences (see `allowCustomFormat` section above).
- **Admin import endpoint** — no server API for uploading `theme.json` via the UI. Admins edit the file directly. Future work.
- **Map config overrides** (`mapDefaults.center`, `mapDefaults.zoom`) — separate concern, not part of theme. Future work.
- **Geo-filter config** — server-only. Not in scope.
- **Per-page layout preferences** (column widths, sort orders) — separate from theming. Future work.
+311
View File
@@ -0,0 +1,311 @@
# Deployment Simplification Spec
**Status:** Draft
**Author:** Kpa-clawbot
**Date:** 2026-04-05
## Current State
CoreScope deployment today requires:
1. **Clone the repo** and build from source (`docker compose build`)
2. **Create a config.json** — the example is 100+ lines with MQTT credentials, channel keys, theme colors, regions, cache TTLs, health thresholds, branding, and more. An operator must understand all of this before seeing a single packet.
3. **Set up a Caddyfile** for TLS (separate `caddy-config/` directory, bind-mounted)
4. **Understand the supervisord architecture** — the container runs 4 processes (mosquitto, ingestor, server, caddy) via supervisord. This is opaque to operators.
5. **No pre-built images** — there's no image on Docker Hub or GHCR. Every operator must `git clone` + `docker compose build`.
6. **Updates require rebuilding**`git pull && docker compose build && docker compose up -d`. No `docker compose pull`.
7. **manage.sh is 100+ lines** of bash wrapping `docker compose` with state files, confirmations, and color output. It's helpful for the maintainer but intimidating for new operators.
### What works well
- **Dockerfile is solid** — multi-stage Go build, Alpine runtime, small image
- **Health checks exist**`wget -qO- http://localhost:3000/api/stats`
- **Environment variable overrides** — ports and data dirs are configurable via `.env`
- **Data persistence** — bind mounts for DB (`~/meshcore-data`), named volume for Caddy certs
- **DISABLE_MOSQUITTO flag** — can use external MQTT broker
- **Graceful shutdown**`stop_grace_period: 30s`, SIGTERM handling
### What's painful
| Pain Point | Impact |
|---|---|
| Must build from source | Blocks anyone without Go/Docker buildx knowledge |
| 100-line config.json required | Operator doesn't know what's optional vs required |
| No sensible defaults for MQTT | Can't connect to public mesh without credentials |
| No pre-built multi-arch images | ARM users (Raspberry Pi) must cross-compile |
| No one-line deploy | Minimum 4 steps: clone, configure, build, start |
| Updates = rebuild | Slow, error-prone, requires git |
## Goal
An operator who has never seen the codebase should be able to run CoreScope with:
```bash
docker run -d -p 80:80 -v corescope-data:/app/data ghcr.io/kpa-clawbot/corescope:v3.4.1
```
And see live MeshCore packets from the public mesh within 60 seconds.
## Pre-built Images
Publish to **GHCR** (`ghcr.io/kpa-clawbot/corescope`) on every release tag.
- **Tags:**
- `vX.Y.Z` (e.g., `v3.4.1`) — specific release, pinned, recommended for production
- `vX.Y` (e.g., `v3.4`) — latest patch in a minor series, auto-updates patches only
- `vX` (e.g., `v3`) — latest minor+patch in a major series
- `latest` — latest release tag (NOT latest commit). Only moves on tagged releases, never on random master commits. Still, production deployments should pin to `vX.Y.Z`
- `edge` — built from master on every push. Unstable, for testing only. Clearly labeled as such
- **Architectures:** `linux/amd64`, `linux/arm64` (Raspberry Pi 4/5)
- **Build trigger:** GitHub Actions on `v*` tag push
- **CI workflow:** New job `publish` after existing `deploy`, uses `docker/build-push-action` with QEMU for multi-arch
```yaml
# .github/workflows/publish.yml (simplified)
on:
push:
tags: ['v*']
jobs:
publish:
runs-on: ubuntu-latest
permissions:
packages: write
steps:
- uses: actions/checkout@v5
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/build-push-action@v6
with:
push: true
platforms: linux/amd64,linux/arm64
tags: |
ghcr.io/kpa-clawbot/corescope:v3.4.1
ghcr.io/kpa-clawbot/corescope:${{ github.ref_name }}
build-args: |
APP_VERSION=${{ github.ref_name }}
GIT_COMMIT=${{ github.sha }}
BUILD_TIME=${{ github.event.head_commit.timestamp }}
```
## Configuration
### Hierarchy (highest priority wins)
1. **Environment variables**`CORESCOPE_MQTT_BROKER`, `CORESCOPE_PORT`, etc.
2. **`/app/data/config.json`** — full config file (volume-mounted)
3. **Built-in defaults** — work out of the box
### Environment variables for common settings
| Variable | Default | Description |
|---|---|---|
| `CORESCOPE_MQTT_BROKER` | `mqtt://localhost:1883` | Primary MQTT broker URL |
| `CORESCOPE_MQTT_TOPIC` | `meshcore/+/+/packets` | MQTT topic pattern |
| `CORESCOPE_PORT` | `3000` | HTTP server port (internal) |
| `CORESCOPE_DB_PATH` | `/app/data/meshcore.db` | SQLite database path |
| `CORESCOPE_SITE_NAME` | `CoreScope` | Branding site name |
| `CORESCOPE_DEFAULT_REGION` | (none) | Default region filter |
| `DISABLE_MOSQUITTO` | `false` | Skip internal MQTT broker |
| `DISABLE_CADDY` | `false` | Skip internal Caddy (when behind reverse proxy) |
### Built-in defaults that work out of the box
The Go server and ingestor already have reasonable defaults compiled in. The only missing piece is **a default public MQTT source** so a fresh instance can see packets immediately. Options:
- **Option A:** Ship with the internal Mosquitto broker only (no external sources). Operator sees an empty dashboard and must configure MQTT. Safe but unhelpful.
- **Option B:** Ship with a public read-only MQTT source pre-configured (e.g., `mqtt.meshtastic.org` or equivalent if one exists for MeshCore). Operator sees live data immediately. Better UX.
**Recommendation:** Option A as default (safe), with a documented one-liner to add a public source. The config.example.json already shows how to add `mqttSources`.
## Compose Profiles
A single `docker-compose.yml` with profiles:
```yaml
services:
corescope:
image: ghcr.io/kpa-clawbot/corescope:v3.4.1
profiles: ["", "standard", "full"] # runs in all profiles
ports:
- "${HTTP_PORT:-80}:80"
volumes:
- ${DATA_DIR:-./data}:/app/data
environment:
- DISABLE_MOSQUITTO=${DISABLE_MOSQUITTO:-false}
- DISABLE_CADDY=${DISABLE_CADDY:-false}
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:3000/api/stats"]
interval: 30s
timeout: 5s
retries: 3
restart: unless-stopped
```
**Note:** Since the container already bundles mosquitto + caddy + server + ingestor via supervisord, "profiles" are really just env var toggles:
| Profile | DISABLE_MOSQUITTO | DISABLE_CADDY | Use case |
|---|---|---|---|
| **minimal** | `true` | `true` | External MQTT + external reverse proxy |
| **standard** (default) | `false` | `true` | Internal MQTT, no TLS (behind nginx/traefik) |
| **full** | `false` | `false` | Everything including Caddy auto-TLS |
This avoids splitting into separate compose services. The monolithic container is actually fine for this use case — it's a single-purpose appliance.
## One-Line Deploy
### Simplest (Docker run, no TLS)
```bash
docker run -d --name corescope \
-p 80:80 \
-v corescope-data:/app/data \
-e DISABLE_CADDY=true \
ghcr.io/kpa-clawbot/corescope:v3.4.1
```
### With Docker Compose
```bash
curl -sL https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/docker-compose.simple.yml -o docker-compose.yml
docker compose up -d
```
Where `docker-compose.simple.yml` is a minimal 15-line file shipped in the repo.
## Update Path
```bash
docker compose pull
docker compose up -d
```
Or for `docker run` users:
```bash
docker pull ghcr.io/kpa-clawbot/corescope:v3.4.1
docker stop corescope && docker rm corescope
docker run -d --name corescope ... # same args as before
```
No rebuild. No git pull. No source code needed.
## Data Persistence
| Path | Content | Mount |
|---|---|---|
| `/app/data/meshcore.db` | SQLite database (all packets, nodes) | Required volume |
| `/app/data/config.json` | Custom configuration (optional) | Same volume |
| `/app/data/theme.json` | Custom theme (optional) | Same volume |
| `/data/caddy` | TLS certificates (Caddy-managed) | Named volume (automatic) |
**Backup:** `cp ~/corescope-data/meshcore.db ~/backup/` — it's just a SQLite file.
**Migration:** Existing `~/meshcore-data` directories work unchanged. Just point the volume at the same path.
## TLS/HTTPS
### Option 1: Caddy auto-TLS (built-in)
The container ships Caddy. To enable auto-TLS:
1. Mount a custom Caddyfile:
```bash
docker run -d \
-p 80:80 -p 443:443 \
-v corescope-data:/app/data \
-v caddy-certs:/data/caddy \
-v ./Caddyfile:/etc/caddy/Caddyfile:ro \
ghcr.io/kpa-clawbot/corescope:v3.4.1
```
2. Caddyfile:
```
your-domain.com {
reverse_proxy localhost:3000
}
```
### Option 2: External reverse proxy (recommended for production)
Run with `DISABLE_CADDY=true` and put nginx/traefik/cloudflare in front. This is the standard approach and what most operators already have.
## Health Checks
Already implemented. The container health check hits `/api/stats`:
```bash
# From outside the container
curl -f http://localhost/api/stats
# Response includes packet counts, node counts, uptime
```
Docker will mark the container as `healthy`/`unhealthy` automatically.
## Monitoring
**Future (M5 from RF health spec):** Expose a `/metrics` Prometheus endpoint with:
- `corescope_packets_total` — total packets ingested
- `corescope_nodes_active` — currently active nodes
- `corescope_mqtt_connected` — MQTT connection status
- `corescope_ingestor_lag_seconds` — time since last packet
This is not required for the deployment simplification work but should be designed alongside it.
## Migration from Current Setup
For existing operators using `manage.sh` + build-from-source:
1. **Keep your data directory** — the bind mount path is the same
2. **Keep your config.json** — it goes in the data directory as before
3. **Replace `docker compose build`** with `docker compose pull`
4. **Update docker-compose.yml** — change `build:` to `image: ghcr.io/kpa-clawbot/corescope:v3.4.1`
5. **manage.sh continues to work** — it wraps `docker compose` and will work with pre-built images
**Breaking changes:** None expected. The container interface (ports, volumes, env vars) stays the same.
## Milestones
### M1: Pre-built images (1-2 days)
- [ ] Create `.github/workflows/publish.yml` for multi-arch builds
- [ ] Push a test `v0.x.0` tag and verify image on GHCR
- [ ] Update README with `docker run` quickstart
- [ ] Create `docker-compose.simple.yml` (minimal compose file using pre-built image)
### M2: Environment variable configuration (1 day)
- [ ] Add env var parsing to Go server `config.go` (overlay on config.json)
- [ ] Add env var parsing to Go ingestor
- [ ] Add `DISABLE_CADDY` support to `entrypoint-go.sh`
- [ ] Document all env vars in README
### M3: Sensible defaults (0.5 day)
- [ ] Ensure server starts with zero config (no config.json required)
- [ ] Verify ingestor connects to localhost MQTT by default
- [ ] Test: `docker run` with no config produces a working (empty) dashboard
### M4: Documentation + migration guide (0.5 day)
- [ ] Write operator-facing deployment docs in `docs/deployment.md`
- [ ] Migration guide for existing users
- [ ] One-page quickstart
**Total estimate:** 3-4 days of work.
## Torvalds Review
> "Is this over-engineered?"
The spec is intentionally simple. Key decisions:
1. **No Kubernetes manifests, Helm charts, or Terraform.** Just Docker.
2. **No config management system.** Env vars + optional JSON file.
3. **Keep the monolithic container.** Splitting into 4 separate services (server, ingestor, mosquitto, caddy) would be "proper" microservices but is worse for operators who just want one thing to run. The supervisord approach is fine for an appliance.
4. **No custom CLI tool.** `docker compose` is the interface.
5. **Profiles are just env vars**, not separate compose files or services.
The simplest version is literally just M1: publish the existing image to GHCR. Everything else is polish. An operator can already `docker run` the image — they just can't `docker pull` it because it's not published anywhere.
+141
View File
@@ -0,0 +1,141 @@
# Movable UI Panels — Draggable Panel Positioning
**Status:** Proposed
**Related:** #279 (original request), PR #606 (collapsible panels — immediate fix)
**Date:** 2026-04-05
---
## Problem
The live map page overlays several UI panels on the map viewport: legend, live feed, node detail, and filters. On smaller screens or dense deployments, these panels obscure map content. Users have no control over where panels sit — they're CSS-fixed in corners, and when they collide with each other or with map data, the only option is to close them entirely. Closing a panel means losing access to the data it shows.
PR #606 addresses the immediate pain with collapsible panels and responsive breakpoints. This spec covers the next step: letting users reposition panels to wherever serves their workflow best.
## Solution
Panels become draggable within the map viewport. Users grab a handle, drag to a new position, release. Position persists in `localStorage` per panel ID. That's it.
### What each panel gets
| Affordance | Behavior |
|---|---|
| **Drag handle** | A subtle grip indicator (6-dot grid or `⋮⋮`) in the panel header. Cursor changes to `grab`/`grabbing`. The handle is the ONLY drag target — the panel body remains interactive (scrollable, clickable). |
| **Snap-to-edge** | When released within 20px of a viewport edge, the panel snaps flush to that edge. Prevents panels floating 3px from the side looking broken. |
| **Position persistence** | `localStorage` key per panel: `panel-pos-{id}``{ x, y }` as viewport percentages (not pixels — survives resize). |
| **Z-index on focus** | Clicking or dragging a panel brings it to front. Simple incrementing counter, reset on page load. |
| **Reset button** | Single button (in settings or as a map control) resets ALL panels to default positions. Clears all `panel-pos-*` keys. |
### What we do NOT build
- **Resizable panels.** Drag-to-resize adds complexity for marginal benefit. Panels have natural content-driven sizes.
- **Docking/tiling/splitting.** This is not a window manager. No snap-to-other-panel, no split view, no tiling grid.
- **Panel minimization to a taskbar.** Collapsible (PR #606) is sufficient.
- **Drag on mobile.** Touch-drag conflicts with map pan. Mobile keeps collapsible behavior from PR #606. Draggable is desktop-only (`pointer: fine` media query).
## Design Considerations
### Drag handle affordance
The handle must be visible enough that users discover it, but not so prominent that it becomes visual noise. A 6-dot grip icon (`⋮⋮`) in the panel title bar, styled at 60% opacity, rising to 100% on hover. The cursor change (`grab``grabbing`) provides the primary affordance.
### Snap-to-edge
Panels snap to the nearest edge when released within a 20px threshold. Snap positions: top-left, top-right, bottom-left, bottom-right, or any edge midpoint. This prevents the "floating at 47px from the left" awkwardness without constraining users to a rigid grid.
### Position persistence
Positions stored as viewport percentages: `{ xPct: 0.02, yPct: 0.15 }`. On window resize, panels stay proportionally positioned. If a resize would push a panel off-screen, clamp it to the nearest visible edge.
### Responsive breakpoints
Below the medium breakpoint (defined in PR #606), panels revert to their fixed/collapsible positions. The draggable behavior is a progressive enhancement for viewports wide enough to have meaningful repositioning space. Persisted positions are preserved in `localStorage` but not applied until the viewport is wide enough again.
### Z-index management
A module-level counter starting at 1000. Each panel interaction (click, drag start) sets that panel's z-index to `++counter`. On page load, counter resets to 1000. No panel can exceed z-index 9999 (modal/overlay territory) — if counter approaches that, compact all panel z-indices down.
### Accessibility
- Panels are focusable (`tabindex="0"` on the drag handle).
- Arrow keys reposition the focused panel by 10px per press (Shift+Arrow = 50px).
- `Escape` while dragging cancels and returns to the previous position.
- `Home` key resets the focused panel to its default position.
- Screen readers: `aria-label="Drag handle for {panel name}. Use arrow keys to reposition."` and `role="slider"` with `aria-valuenow` reflecting position.
## Implementation
### Milestones
**M1: Core drag mechanics** (~2 days)
- `DragManager` class: registers panels, handles pointer events, updates positions
- Snap-to-edge logic
- Z-index management
- No persistence yet — positions reset on reload
**M2: Persistence + reset** (~1 day)
- `localStorage` read/write for panel positions
- Reset-to-defaults button
- Viewport-percentage storage with resize clamping
**M3: Responsive + accessibility** (~1 day)
- Disable drag below medium breakpoint
- Keyboard repositioning (arrow keys)
- ARIA attributes
- Screen reader announcements on position change
**M4: Polish + testing** (~1 day)
- Playwright E2E tests: drag, snap, persist, reset, keyboard
- Performance validation: drag must not trigger layout thrash (use `transform: translate()`, not `top/left`)
- Edge case handling (see below)
### Technical approach
- **No library.** Pointer events (`pointerdown`, `pointermove`, `pointerup`) with `setPointerCapture`. ~150 lines of vanilla JS.
- **CSS transforms for positioning.** `transform: translate(Xpx, Ypx)` avoids layout reflow during drag. Only write to `style.transform`, never `top`/`left`.
- **Debounce persistence.** Write to `localStorage` on `pointerup`, not during drag.
- **Single file:** `public/drag-manager.js` — imported by `live.js`, no other dependencies.
## Edge Cases
| Case | Handling |
|---|---|
| Panel dragged partially off-screen | Clamp to viewport bounds on `pointerup` |
| Window resized while panel is near edge | Re-clamp on `resize` (debounced 200ms) |
| Two panels overlap after drag | Allowed — z-index determines which is on top. Users can move them. |
| `localStorage` full or unavailable | Graceful fallback to default positions. No error shown. |
| Panel content changes size after drag | Panel stays at dragged position; content reflows within. If panel grows past viewport edge, clamp. |
| User has old `localStorage` keys from a removed panel | Ignore unknown keys on load. Clean up stale keys on reset. |
| RTL layouts | Snap logic uses physical viewport edges, not logical start/end. Drag is inherently physical. |
## Expert Reviews
### Tufte (Information Design)
- **Draggability is justified** only if it serves data access — and here it does. Panels obscuring map data is a data-visibility problem, not a UI-decoration problem. Letting users clear their sightlines to the data is correct.
- **The drag handle must be minimal.** Six dots at 60% opacity is acceptable. Anything more prominent (colored bars, icons, labels) becomes chartjunk — UI chrome competing with data for attention.
- **Resist feature creep.** Resizable panels, docking zones, panel-to-panel snapping — all increase interface complexity without increasing data throughput. The spec correctly excludes these.
- **Snap-to-edge is good.** It prevents the visual noise of arbitrarily placed rectangles. Panels aligned to edges create clean negative space for the map data.
### Torvalds (Engineering Pragmatism)
- **This is borderline over-engineering.** The real question: do users actually need free-form drag, or would a simpler "pick a corner" toggle (TL/TR/BL/BR) cover 95% of use cases with 20% of the code?
- **The 4-corner toggle would be ~40 lines.** The full drag system is ~150+ lines plus persistence, snap logic, accessibility, resize handling, z-index management, and edge cases. That's a lot of surface area for "I want the legend on the right instead of the left."
- **Recommendation:** Ship the 4-corner toggle first (M0). If users actually request free-form drag after that, build it. YAGNI applies here.
- **If you do build drag:** the spec is sound. Pointer events + transforms + localStorage is the right stack. No library is correct. But test it on Firefox — pointer capture has quirks.
### Doshi (Product/Business)
- **This is an N (Nice-to-have), not an L (Leverage).** It improves UX for power users who spend hours on the live map, but it doesn't unlock new capabilities or new users.
- **Opportunity cost:** 5 developer-days on draggable panels is 5 days not spent on features that expand what CoreScope can do (new analytics, alerting, multi-site support).
- **The collapsible panels (PR #606) likely resolve the P1 pain.** Track whether users still complain about panel placement after #606 ships. If complaints drop to zero, this spec can stay on the shelf.
- **If built:** ship M1+M2 only (3 days). M3 accessibility can come later if adoption warrants it. M4 testing is non-negotiable.
### Feedback incorporated
Based on the reviews, the spec adds a **Milestone 0** recommendation:
**M0: Corner-position toggle** (~0.5 days)
Before building full drag, ship a simpler panel-position toggle: each panel's header gets a small button that cycles through TL → TR → BR → BL placement. Positions persist in `localStorage`. If this satisfies user needs, M1M4 become unnecessary.
**Decision gate:** Ship M0 with PR #606 or shortly after. Monitor feedback for 2 weeks. If users request free-form repositioning, proceed to M1. If corner toggle is sufficient, close this spec as "resolved by M0."
+493
View File
@@ -0,0 +1,493 @@
# Spec: RF Health Dashboard — Observer Radio Metrics
**Status:** Draft v3
**Purpose:** Enable operators to quickly identify RF jammers, deaf receivers, and radio health issues through per-observer time-series charts.
## Prerequisite Gate
**Before building anything, verify that stats messages arrive periodically from observers.**
The ingestor must receive radio stats messages at a predictable interval via MQTT. Confirmed: status messages arrive every ~5 minutes per observer.
**Verification steps (M0):**
1. Connect ≥3 observers to the MQTT bridge
2. Log all incoming stats messages with timestamps for 24h
3. Confirm messages arrive at a regular interval (expected: every few minutes)
4. If stats are NOT periodic, stop — a stats-request mechanism must be added to the MQTT bridge first (separate spec)
5. **Verify `triggerNoiseFloorCalibrate()` firing frequency.** If it fires on every stats cycle, noise floor readings may be artificially consistent (measuring calibration, not environment). If it fires only on boot, the first sample after reboot is unreliable — document which behavior the firmware uses.
Do not proceed to M1 until this gate passes.
## Problem
Operators currently have no visibility into RF environment quality over time. A jammer could be active for hours before anyone notices degraded mesh performance. A deaf receiver silently drops packets with no alert. There's no way to distinguish "the mesh is quiet" from "my observer can't hear anything."
## Solution
A new Analytics tab ("RF Health") showing per-observer time-series charts for noise floor, TX airtime, RX airtime, and receive errors over configurable time windows (1h to 30d, plus custom from/to range). Automated pattern detection (M3+) flags anomalies and suggests diagnoses after operators have used raw charts to provide feedback.
## Data Model
### New table: `observer_metrics`
```sql
CREATE TABLE IF NOT EXISTS observer_metrics (
observer_id TEXT NOT NULL,
timestamp TEXT NOT NULL, -- ISO 8601, rounded to nearest sample interval
noise_floor REAL, -- dBm, from radio stats (nullable — may arrive without airtime)
tx_air_secs INTEGER, -- cumulative TX seconds since boot (nullable)
rx_air_secs INTEGER, -- cumulative RX seconds since boot (nullable)
packets_sent INTEGER, -- cumulative packets sent since boot (nullable)
packets_recv INTEGER, -- cumulative packets received since boot (nullable)
recv_errors INTEGER, -- cumulative CRC/decode failures since boot (nullable)
battery_mv INTEGER, -- battery voltage in millivolts (nullable, for field/solar nodes)
PRIMARY KEY (observer_id, timestamp)
);
```
**Field notes:**
- **`recv_errors`** (CRC failure count) is the strongest single indicator of channel quality. A rising error rate with stable noise floor points to in-band digital interference rather than broadband jamming. This is more diagnostic than packet_count alone.
- **`packets_sent` / `packets_recv`** are tracked separately because the ratio reveals asymmetric link problems (e.g., observer can transmit but not receive, or vice versa). The old `packet_count` field conflated these.
- **`battery_mv`** is nullable and only relevant for field/solar deployments. Low battery causes erratic radio behavior (reduced TX power, missed RX windows) that looks like RF problems but isn't. Charting voltage alongside RF metrics prevents misdiagnosis.
- All cumulative counters (`tx_air_secs`, `rx_air_secs`, `packets_sent`, `packets_recv`, `recv_errors`) reset on reboot — see reboot handling below.
No additional indexes. The composite primary key covers all query patterns (per-observer time-range scans). At 70K rows, a full scan for any fleet-wide time query is fast enough.
### Clock source
**Always use the ingestor's wall clock for timestamps, not observer-reported timestamps.** Observer clocks may be wrong, drifted, or absent (no RTC). Round the ingestor wall clock to the nearest sample interval boundary (e.g., 5-minute marks) for consistent time alignment.
### Noise floor cold start caveat
**The first noise floor sample after a reboot may be unreliable.** The radio's noise floor reading requires settling time and may reflect calibration artifacts rather than the actual RF environment. Mark the first post-reboot sample with a `reboot` flag (see reboot handling) so the frontend can annotate it. Do not use first-post-reboot noise floor samples in baseline/median calculations.
### Sampling strategy
- **Interval:** Every 5 minutes (configurable via config.json `metrics.sampleIntervalSec`, default 300)
- **Source:** MQTT stats messages (`STATS_TYPE_RADIO`)
- **Insertion:** `INSERT OR REPLACE INTO observer_metrics (observer_id, timestamp, ...) VALUES (?, ?, ...)` with timestamp rounded to the nearest interval boundary. No need to track last-insert time per observer — rounding + `INSERT OR REPLACE` is idempotent and naturally deduplicates.
- **Storage:** ~10K rows/day for 35 observers. At configurable retention. Negligible.
- **Retention:** Configurable, configurable, default 30 days. Prune with a single `DELETE FROM observer_metrics WHERE timestamp < datetime('now', '-N days')` on startup and every 24h. Consider `PRAGMA auto_vacuum = INCREMENTAL` for embedded devices.
### Gap detection
If the time between two consecutive samples for an observer exceeds 2× the sample interval (e.g., >10 minutes for a 5-min interval), insert null values in the response to indicate a gap. This prevents charts from drawing misleading interpolation lines across outages.
### Reboot handling
Cumulative counters (`tx_air_secs`, `rx_air_secs`, `packets_sent`, `packets_recv`, `recv_errors`) reset on device reboot. Detect counter resets (current value < previous value) and:
1. Skip the delta computation for that interval (do not produce a negative value)
2. Log a reboot event for the observer with the timestamp
3. Use the current sample as the new baseline for subsequent deltas
4. **Include reboot timestamps in the API response** so the frontend can render them as annotations directly on the chart (see frontend design)
5. **Flag the first post-reboot noise floor sample** as potentially unreliable (cold start — see above)
### Delta computation (server-side)
Cumulative counters are converted to per-interval rates server-side. **Deltas are computed server-side, not in the frontend.** The API returns percentage/rate values directly. This keeps firmware implementation details (cumulative counters, reboot semantics) out of the UI layer, reduces payload size, and centralizes reboot-handling logic.
### Graceful degradation
Not all observers may report all metrics. If fields are absent:
- Store `NULL` for missing columns
- The API returns `null` for unavailable fields
- The frontend shows only the charts for which data exists — missing charts are hidden, not broken
- Status detection uses only available metrics
- `battery_mv` is expected to be absent on mains-powered observers — this is normal, not an error
Partial data is always better than no data. Never error or crash on missing optional fields.
### Required ingestor changes
1. Parse `tx_air_secs`, `rx_air_secs`, `packets_sent`, `packets_recv`, `recv_errors`, and `battery_mv` from MQTT stats messages (same pattern as existing `noise_floor`)
2. On each stats message, round ingestor wall clock to nearest interval, `INSERT OR REPLACE` into `observer_metrics`
3. Handle missing fields gracefully (insert NULLs for absent metrics)
4. Detect counter resets and record reboot events
5. Add new columns to `observers` table for current/latest values
### API endpoints
```
GET /api/observers/{id}/metrics?since=2026-04-04T00:00:00Z&until=2026-04-05T00:00:00Z&resolution=5m
```
**`resolution` query parameter** controls downsampling:
- `5m` (default) — raw samples
- `1h` — hourly aggregates (`GROUP BY strftime('%Y-%m-%dT%H:00:00', timestamp)` with MIN/MAX/AVG)
- `1d` — daily aggregates
Use `1h` resolution for 7d views to avoid shipping 2,016 points per observer. Essential for the fleet comparison view (35 observers × 2,016 = 70K points at raw resolution → 35 × 168 = 5,880 points at 1h resolution).
Returns:
```json
{
"observer_id": "1F445B...",
"observer_name": "GY889 Repeater",
"reboots": ["2026-04-04T03:15:00Z", "2026-04-04T18:22:00Z"],
"metrics": [
{
"timestamp": "2026-04-04T00:00:00Z",
"noise_floor": -112.5,
"tx_airtime_pct": 2.1,
"rx_airtime_pct": 8.3,
"packets_sent": 42,
"packets_recv": 342,
"recv_errors": 3,
"recv_error_rate": 0.87,
"battery_mv": 3720,
"is_reboot_sample": false
}
]
}
```
Notes:
- `tx_airtime_pct` and `rx_airtime_pct` are server-computed deltas as percentages. Null if airtime data unavailable.
- `recv_error_rate` = `recv_errors / (packets_recv + recv_errors)` as a percentage. Null if either field unavailable.
- `packets_sent` and `packets_recv` are per-interval deltas (not cumulative). Null if unavailable.
- `reboots` array contains timestamps of detected reboots within the queried window, for chart annotation.
- `is_reboot_sample` flags first-post-reboot samples where noise floor may be unreliable.
- `battery_mv` is null for mains-powered observers.
```
GET /api/observers/metrics/summary?window=24h
```
**Fleet summary is cached incrementally.** Maintain a rolling summary struct in memory, updated on each new sample insert (35 observers × 1 sample/5min = 7 inserts/min — trivially cheap). The endpoint reads from the cached struct, not from SQLite queries on every request.
Returns:
```json
{
"observers": [
{
"observer_id": "1F445B...",
"observer_name": "GY889 Repeater",
"current_noise_floor": -112.5,
"avg_noise_floor_24h": -114.2,
"max_noise_floor_24h": -95.0,
"tx_airtime_pct_24h": 2.1,
"rx_airtime_pct_24h": 8.3,
"recv_error_rate_24h": 0.87,
"battery_mv": 3720,
"status": "normal"
}
]
}
```
## Frontend Design
### Design Principles
The dashboard exists for one purpose: **let an operator glance at it at 3 AM and know immediately if something is wrong.** Every design decision follows from this. Decoration that doesn't serve comprehension is removed. Data that can be shown is shown — not hidden behind clicks or hovers.
Key rules (per Tufte):
- **Maximize data-ink ratio.** Every pixel must encode data or directly support reading it. Remove anything that doesn't.
- **No chartjunk.** No gradient fills, no 3D effects, no decorative borders, no ornamental chrome.
- **Labels on the data, not in legends.** Direct-label lines, annotate anomalies at the point they occur. The viewer should never look away from the data to understand it.
- **Show data variation, not design variation.** All observer charts use identical scales, formats, and typography. If two charts look different, it's because the data is different.
- **Respect the viewer's intelligence.** Dense, information-rich displays are fine. Oversimplified displays waste screen space and the operator's time.
### Page structure: small multiples grid
```
Analytics → RF Health tab
├── Time range: [1h] [3h] [6h] [12h] [24h] [3d] [7d] [30d] [Custom ▾]
│ ├── Presets: click to quick-set
│ └── Custom: two datetime inputs (from/to) with calendar picker
│ └── URL hash reflects selected range for deep linking
├── Small Multiples Grid (ALL observers, one cell per observer)
│ │
│ │ Each cell contains:
│ │ ┌─────────────────────────────────────────┐
│ │ │ GY889 Repeater -112.5 dBm 3.7V│ ← name, current NF, battery (if field node)
│ │ │ ┈┈┈╲┈┈┈┈┈┈╱┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈│ ← noise floor sparkline (24h)
│ │ │ err: 0.8% TX: 2.1% RX: 8.3% │ ← key rates, inline text
│ │ │ ▲reboot 03:15 │ ← reboot annotation (if any)
│ │ └─────────────────────────────────────────┘
│ │
│ │ Sorted by: worst status first, then highest noise floor
│ │ Grid: 34 columns on desktop, 2 on tablet, 1 on phone
│ │ Click any cell → expand to full detail below
│ │
│ └── Entire grid is visible at once — no pagination, no "show more"
│ (35 observers × ~60px per cell = ~700px — fits on one screen)
├── Expanded Detail (shown below grid when a cell is clicked)
│ │
│ │ Three time-aligned charts, stacked vertically, sharing X-axis:
│ │
│ │ 1. Noise Floor (dBm)
│ │ - SVG line chart, Y-axis inverted (higher dBm = worse = higher on chart)
│ │ - Thin reference lines at -100 dBm and -85 dBm, directly labeled
│ │ (e.g., "100 warning" / "85 critical") — no color bands
│ │ - Gaps (nulls) break the line — no interpolation across outages
│ │ - Reboot markers: vertical hairline at each reboot timestamp,
│ │ labeled "reboot" directly on the chart
│ │ - First-post-reboot sample marked with open circle (unreliable cold start)
│ │ - Direct labels on notable points (min, max, anomalies)
│ │
│ │ 2. Airtime (%) — hidden if no airtime data
│ │ - Two separate SVG lines (NOT stacked area — stacked areas
│ │ make it impossible to read the lower series accurately)
│ │ - TX line and RX line, directly labeled at their endpoints
│ │ ("TX 2.1%" / "RX 8.3%") — no legend box
│ │ - Same X-axis as noise floor chart above
│ │ - Gaps shown as breaks
│ │
│ │ 3. Channel Quality
│ │ - Receive error rate (%) as a line
│ │ - Packets recv as a light step-line for context
│ │ - Directly labeled — no legend
│ │ - High error rate + low packet count = dead channel
│ │ - High error rate + high packet count = interference
│ │
│ │ 4. Battery Voltage (shown only if battery_mv is non-null)
│ │ - Simple line chart, mV scale
│ │ - Directly labeled with current value
│ │ - Useful for correlating RF anomalies with low-battery behavior
│ │
│ │ All four charts share the same X-axis and time range.
│ │ Reboot markers appear as vertical hairlines across ALL charts
│ │ (same event, visible in all contexts — no hunting).
│ │
│ └── Current values shown as text below charts:
│ NF: 112.5 dBm | TX: 2.1% | RX: 8.3% | Err: 0.87% | Batt: 3.72V
│ 24h: avg 114.2 | max 95.0 | 3 reboots
└── Fleet Comparison (M4)
└── Small multiples of noise floor, one per observer, identical Y-scale
└── NOT an overlay chart — overlays become unreadable past 5 lines
└── Use 1h resolution for 7d views
```
### Why small multiples, not expandable accordion
An accordion (expand/collapse per observer) forces the operator to click through each observer sequentially. At 3 AM with 35 observers, that's unacceptable. The small multiples grid shows ALL observers simultaneously — the eye does the comparison, not the mouse. Anomalies pop out visually because they break the pattern of the grid. This is Tufte's core insight: **small multiples leverage the viewer's ability to detect pattern breaks across a consistent visual template.**
### Why no color bands on charts
Color bands (green/yellow/red zones) are decorative — they add ink that doesn't encode data. They also pre-judge what's "good" and "bad," which varies by deployment environment. Instead, use **thin reference lines with direct text labels** at the warning and critical thresholds. The reference lines take up negligible ink, the labels are informational, and the operator's eye naturally compares the data line against them.
### Why not stacked area for airtime
Stacked area charts are a common source of graphical dishonesty. The bottom series (TX) reads correctly against the X-axis, but the top series (RX) reads against the TX boundary — making it impossible to accurately judge RX values without mental subtraction. Two separate lines, directly labeled, are always more honest and more readable.
### Color usage
Color encodes data category, never decoration:
- **Noise floor line:** single muted color (the line IS the data — it doesn't need to be loud)
- **TX / RX lines:** two distinct colors, directly labeled at endpoints (no legend needed)
- **Error rate:** a third distinct color
- **Reboot markers:** gray hairlines (de-emphasized — context, not data)
- **Status text in grid cells:** text color only (not background fill) — red text for critical, amber for warning, default for normal
- No background color fills on cards. No colored borders. No badge backgrounds. Color on text only where it carries meaning.
### Labels and annotations
- **Reference lines** at threshold values, labeled directly ("100 dBm warning")
- **Reboot events** as vertical hairlines across all charts, labeled "reboot" at the top
- **Cold-start samples** marked with open circles and a subtle "?" annotation
- **Current values** as inline text on the sparkline cells and below detail charts
- **No separate legends.** Lines are labeled at their endpoints or directly on the chart.
- **Hover** shows exact timestamp + value — this is the only interactive element, and it reveals precision, not hidden data
### Data density
- The small multiples grid fits 35 observers in ~700px vertical space (one screen on desktop)
- Each cell is information-dense: name + current value + sparkline + rates + reboot count — all visible without clicking
- Detail charts are stacked vertically sharing the X-axis, eliminating redundant time labels
- No wasted whitespace between chart panels — they are a single visual unit
### Information hierarchy (3 AM glance test)
1. **Grid scan (2 seconds):** Are all sparklines flat and similar? Yes → everything's fine. One cell has a spike or red text → that's the problem.
2. **Cell read (3 seconds):** Which observer, what's the current NF, what's the error rate? All visible without clicking.
3. **Detail dive (10 seconds):** Click the cell, see time-series context, see if it correlates with reboots, check battery, check airtime.
An operator never needs to click anything to know if the fleet is healthy. Clicking only provides temporal detail for diagnosis.
### Mobile considerations
- Grid collapses to 1 column on phone (each cell is full-width, still showing sparkline + values)
- Detail charts fill the viewport width, Y-axis labels move above the chart to save horizontal space
- Touch targets: the entire grid cell is tappable (not a small icon)
- Time range selector uses segmented control (large touch targets) for presets, not a dropdown
- Custom range picker: two datetime inputs with calendar popup, positioned below the presets
- Selected range (preset or custom) persists in URL hash: `&range=24h` or `&from=2026-04-04T14:00:00Z&to=2026-04-04T16:00:00Z`
### Chart rendering
**Use SVG, not Canvas.** The existing analytics.js uses SVG for all charts (sparklines, bar charts, histograms). Canvas is only used for the force-directed neighbor graph. Follow the existing SVG patterns — reuse `sparkSvg()` for fleet overview sparklines.
2,016 SVG polyline points per chart is fine. For the fleet comparison view (M4), use hourly downsampling (168 points per observer) to avoid layout jank on mobile.
### Deep linking
```
#/analytics?tab=rf-health
#/analytics?tab=rf-health&observer=1F445B...&range=24h
```
## Pattern Detection (M3+)
**Pattern detection is deferred until after operators have used raw charts (M1M2) and provided feedback on what patterns actually matter.** Do not implement automated diagnosis until real-world usage informs the rules.
### Planned automated diagnosis
The server computes a `status` field per observer based on the last N samples:
| Pattern | Status | Indicator |
|---|---|---|
| NF stable, RX/TX normal, low error rate | `normal` | (no indicator — absence of alarm is the signal) |
| NF spike + RX drop (broadband interference) | `jammer_suspected` | Red text: "Jammer?" |
| NF normal, RX near zero, fleet active (≥5 observers) | `deaf` | Red text: "Deaf receiver" |
| High `recv_errors` rate + stable NF | `digital_interference` | Amber text: "CRC errors high" |
| TX approaching duty cycle warning | `tx_overload` | Amber text: "TX overload" |
| No samples in >15 min | `offline` | Gray text: "Offline" |
| NF gradually increasing over hours | `interference_trend` | Amber text: "Rising interference" |
| Battery voltage below threshold | `low_battery` | Amber text: "Low battery" |
**Jammer detection logic:** A jammer raises the noise floor AND causes RX to drop (the receiver can't hear legitimate signals over the interference). NF spike + RX spike would indicate a legitimate busy channel, not a jammer. The key signal is: NF goes up, RX goes down.
**Digital interference detection (new):** High `recv_errors` with a stable noise floor indicates in-band digital interference (another protocol sharing the frequency, or a malfunctioning node transmitting garbage). This is distinct from broadband jamming, which raises the noise floor. `recv_errors` is the strongest single signal for this.
**Deaf detection:** Requires a minimum fleet size of ≥5 active observers to establish a meaningful fleet median. With fewer observers, skip deaf detection — the sample size is too small for comparison.
### Status priority
When multiple status conditions apply simultaneously, use this priority order (highest first):
1. `offline` — no data trumps everything
2. `jammer_suspected` — active threat
3. `deaf` — hardware failure
4. `digital_interference` — channel quality issue
5. `tx_overload` — regulatory concern
6. `low_battery` — power issue causing RF symptoms
7. `interference_trend` — gradual degradation
8. `normal` — default
### Baseline computation
- **Baseline noise floor:** rolling median of last 24h, **excluding first-post-reboot samples** (cold start unreliable). Computed once on new sample arrival, cached — not recomputed per request.
- **Spike detection:** current sample exceeds an absolute threshold (configurable) AND exceeds baseline + spike delta. Both conditions must be met — a delta-only threshold could false-positive in environments where the absolute NF is already benign (e.g., -115 dBm + 15 dBm = -100 dBm, which is fine).
- **"Others active" check for deaf detection:** compare this observer's RX packet count against the fleet median. If this observer is <10% of fleet median AND fleet has ≥5 active observers, flag as potentially deaf.
- **Error rate baseline:** rolling average of `recv_error_rate` over 24h. Spike above 2× baseline triggers `digital_interference` status.
### Alert thresholds (configurable)
```json
{
"rfHealth": {
"noiseFloorWarning": -100,
"noiseFloorCritical": -85,
"spikeThresholdDb": 15,
"txDutyCycleWarning": 8,
"deafThresholdPct": 10,
"deafMinFleetSize": 5,
"offlineTimeoutSec": 900,
"sampleIntervalSec": 300,
"retentionDays": 30,
"errorRateWarning": 5,
"lowBatteryMv": 3300
}
}
```
Note: No hardcoded duty cycle limit line on charts. Duty cycle regulations vary by jurisdiction (e.g., 1% in EU 868MHz, 10% in some US ISM bands). The warning threshold is configurable but no "regulatory limit" line is drawn on charts.
## Implementation Milestones
### M0: Prerequisite — Verify stats message frequency ✅ PASSED
- **Confirmed 2026-04-05:** Live MQTT capture on staging shows status messages arriving every ~5 minutes per observer
- **Fields confirmed present:** `noise_floor`, `tx_air_secs`, `rx_air_secs`, `recv_errors`, `battery_mv`, `uptime_secs`
- **Fields NOT yet parsed by ingestor:** `tx_air_secs`, `rx_air_secs`, `recv_errors` (noise_floor and battery_mv already parsed)
- **Ingestor timestamps:** Use ingestor wall clock, not observer timestamps (confirmed in design)
- **Verified:** `triggerNoiseFloorCalibrate()` fires every 2 seconds (`NOISE_FLOOR_CALIB_INTERVAL = 2000ms` in `Dispatcher.cpp`). Continuous calibration with 64 RSSI samples per cycle. Noise floor data is always fresh.
- **Gate: PASSED.** Proceed to M1.
### M1: Store metrics + small multiples grid (MVP)
- Create `observer_metrics` table with all columns (migration)
- Ingestor: parse all available fields from stats, `INSERT OR REPLACE` with rounded timestamps
- Handle missing fields gracefully (store NULLs)
- Detect counter resets and record reboot events
- Add `/api/observers/{id}/metrics` endpoint (all available fields)
- Add `/api/observers/metrics/summary` endpoint (cached incrementally)
- Add "RF Health" tab to Analytics
- **Small multiples grid** with sparklines and inline values for all observers
- Per-observer detail view: noise floor line chart with reference lines (not color bands), reboot markers as vertical hairlines, cold-start sample annotation
- Time range selector (1h/3h/6h/12h/24h/3d/7d/30d + custom range picker)
- Deep linking
- Retention pruning
- Tests: sampling, insertion idempotency, retention, API responses, gap handling, reboot detection
### M2: Airtime + channel quality charts
- Server-side delta computation for all cumulative counters with reboot handling and gap detection
- Add `resolution` query param for downsampling (1h, 1d)
- Airtime charts: two separate lines (TX/RX), directly labeled — not stacked area
- Channel quality chart: recv_error_rate line + packets_recv step-line
- Battery voltage chart (shown only when data exists)
- All charts time-aligned, sharing X-axis, reboot markers spanning all charts
- Tests: delta computation, reboot handling, counter reset, gap insertion, downsampling, error rate calculation
#### M2 feedback improvements (post-M2)
- **Auto-scale airtime Y-axis**: clamp to min/max of actual data values (20% headroom, min 1%) instead of fixed 0-100%, matching noise floor chart behavior. Increases data-ink ratio for low-activity nodes.
- **Hover tooltips on all chart data points**: invisible SVG circles with `<title>` elements on every data point across all 4 charts (noise floor, airtime, error rate, battery). Shows exact value + UTC timestamp on hover. Detail-on-demand without cluttering the chart.
### M3: Pattern detection
- Implement after operators have used raw charts (M1M2) and provided feedback
- Jammer detection (NF spike + RX drop)
- Digital interference detection (high recv_errors + stable NF)
- Deaf receiver detection (with ≥5 fleet minimum)
- Low battery detection
- Interference trend detection
- Status text indicators with priority ordering (no emoji badges — text only)
- Baseline computation (rolling median excluding cold-start samples, cached)
- Configurable alert thresholds
- Tests: each pattern, edge cases, status priority
### M4: Fleet comparison + advanced views
- Fleet comparison as **small multiples** (one noise floor chart per observer, identical Y-scale) — not overlay
- Sort/filter fleet by status, noise floor, error rate
- Optional: per-observer historical baseline trend
- Use 1h resolution for 7d views
### M5: Metrics export — Prometheus / Grafana / external systems
- **Prometheus endpoint:** `GET /metrics` exposing observer radio metrics in Prometheus exposition format
- Gauges per observer: `corescope_observer_noise_floor_dbm{observer="...",name="..."}`, `corescope_observer_tx_air_secs_total`, `corescope_observer_rx_air_secs_total`, `corescope_observer_recv_errors_total`, `corescope_observer_battery_mv`, `corescope_observer_uptime_secs`
- Fleet-level: `corescope_observers_total`, `corescope_observers_online`
- Packet counters: `corescope_packets_total`, `corescope_observations_total`
- Standard `process_*` and `go_*` runtime metrics via `promhttp` handler
- **Configurable:** Enable/disable via `config.json` (`metrics.prometheusEnabled: true`, `metrics.prometheusPath: "/metrics"`)
- **Auth:** Optional bearer token or basic auth on the metrics endpoint (prevents public scraping)
- **Labels:** Each observer metric labeled with `observer` (pubkey), `name` (friendly name), `region`
- **Why Prometheus format:** Industry standard, compatible with Grafana, Datadog, Victoria Metrics, Mimir, and any OpenMetrics consumer. Operators who already run monitoring stacks can integrate CoreScope without any custom work.
- **Implementation:** Use Go `prometheus/client_golang` library. Register collectors that read from the in-memory `PacketStore` and `observer_metrics` table. No additional polling — just expose current state on each scrape.
- **Grafana dashboard template:** Ship a JSON dashboard template (`docs/grafana-dashboard.json`) that operators can import for instant RF health visualization in Grafana. Pre-configured panels matching the built-in RF Health tab.
- **OpenTelemetry (future):** If demand exists, add OTLP export alongside Prometheus. Not in M5 scope.
## Design Decisions
1. **Per-observer, not per-device.** Even if two observers share hardware, their RF environments may differ (different antennas, channels). observer_id is already the natural key.
2. **Poll-on-tab-switch, not WebSocket push.** Data changes every 5 minutes. Users check this tab when investigating issues, not for live monitoring. WebSocket push adds complexity for no UX benefit.
3. **SVG charts.** Matches existing analytics.js patterns. Canvas only if fleet comparison proves too slow with SVG.
4. **Server-side deltas.** Keeps firmware details out of the frontend. Single point for reboot/gap handling logic.
5. **Incremental fleet summary cache.** 7 inserts/min is trivially cheap to process. No need to query SQLite on every summary request.
6. **No standalone timestamp index.** The composite PK handles all query patterns. A standalone index wastes write amplification.
7. **Ingestor wall clock for timestamps.** Observer clocks are unreliable. Consistent time source prevents alignment issues.
8. **Small multiples over accordion/cards.** Enables instant visual fleet comparison without clicking. Anomalies break the visual pattern of the grid. (Tufte: "Small multiples are the best design solution for a wide range of problems in data presentation.")
9. **Reference lines, not color bands.** Color bands add non-data ink and pre-judge thresholds. Reference lines are minimal and informational.
10. **Two lines, not stacked area for airtime.** Stacked areas make the upper series unreadable. Two lines with direct labels are always more honest.
11. **Text status indicators, not emoji badges.** Emoji badges are decorative chrome. Plain text with semantic color (red/amber/default) is higher data-ink ratio and more accessible.
12. **Reboot markers as cross-chart annotations.** Reboots affect all metrics simultaneously. Showing them as vertical hairlines across all charts prevents the operator from having to correlate events across separate views.
13. **Separate packets_sent/packets_recv.** The ratio reveals asymmetric link problems invisible in a combined count.
14. **recv_errors as a first-class metric.** CRC failures are the strongest channel quality signal — more diagnostic than noise floor alone for in-band interference.
15. **Exclude cold-start samples from baseline.** First-post-reboot noise floor readings may reflect calibration artifacts, not the RF environment. Including them would bias the baseline.
## Open Questions
1. **Multiple observers on same channel:** If two observers share a channel, their noise floors should correlate. Could be useful for validation but doesn't change the data model.
2. **EMA vs median for baseline:** Exponential moving average is cheaper (no sort) and smoother than median. Consider for M3 implementation — but median is more robust against outliers. Decision deferred to M3.
3. **`triggerNoiseFloorCalibrate()` frequency:** Must be verified in M0. If it fires on every stats cycle, noise floor readings may be artificially smoothed. If only on boot, cold-start caveat applies. This affects how much weight to give noise floor vs. recv_errors for interference detection.
4. **Battery voltage thresholds:** 3.3V is a reasonable default for LiPo cells, but varies by chemistry and regulator. May need per-observer configuration.
+212
View File
@@ -0,0 +1,212 @@
# Startup Performance: Serve HTTP Within 2 Minutes on Any Database Size
## Problem
CoreScope takes 3045 minutes to start on large databases (325K transmissions, 7.3M observations, 1.4GB SQLite). The HTTP server is completely unavailable during this time. Operators cannot restart without 30+ minutes of downtime.
### Where time goes (7.3M observation benchmark)
| Phase | Time | Blocking? |
|---|---|---|
| `Load()` — read SQLite → memory | ~90s | Yes |
| Build subpath index | ~20s | Yes |
| Build distance index | ~15s | Yes |
| Build path-hop index | <1s | Yes |
| Load neighbor edges from SQLite | <1s | Yes |
| **Backfill `resolved_path` for NULL observations** | **2030+ min** | **Yes — the killer** |
| Re-pick best observations | ~10s | Yes |
The backfill calls `resolvePathForObs` for every observation with `resolved_path IS NULL`, then writes results back to SQLite and updates in-memory state. On first run (or after schema migration), this means resolving all 7.3M observations.
### Root cause
`backfillResolvedPaths()` in `neighbor_persist.go` runs synchronously in `main()` before `httpServer.ListenAndServe()`. It:
1. Collects all observations with `ResolvedPath == nil` under a read lock
2. Resolves paths (CPU-bound, ~millions of calls to `resolvePathForObs`)
3. Writes results to SQLite in a single transaction
4. Updates in-memory state under a write lock
Steps 24 block the main goroutine for 2030 minutes.
## Solution: Async Chunked Backfill
### Design
Move `backfillResolvedPaths` out of the startup critical path. Start the HTTP server immediately after loading data and building indexes. Run backfill in a background goroutine with chunked processing that yields between batches.
### Startup sequence (new)
```
1. OpenDB, verify tables (~1s)
2. store.Load() (~90s)
3. ensureNeighborEdgesTable (<1s)
4. ensureResolvedPathColumn (<1s)
5. Load/build neighbor graph (<1s)
6. Build subpath/distance/path-hop indexes (~35s)
7. pickBestObservation (with whatever (~10s)
resolved_path data exists)
8. *** START HTTP SERVER *** — serving at ~2min mark
9. Background: backfillResolvedPaths (20-30 min, non-blocking)
→ chunked, yields between batches
→ updates in-memory + SQLite incrementally
→ re-picks best obs for affected txs
```
Total time to first HTTP response: **~2 minutes** regardless of database size.
### Implementation details
#### 1. Background backfill goroutine
```go
// In main(), after starting HTTP server:
go func() {
backfillResolvedPathsAsync(store, dbPath, 5000, 100*time.Millisecond)
}()
```
The async backfill processes observations in chunks of N (e.g., 5,000):
```go
func backfillResolvedPathsAsync(store *PacketStore, dbPath string, chunkSize int, yieldDuration time.Duration) {
for {
n := backfillResolvedPathsChunk(store, dbPath, chunkSize)
if n == 0 {
break // done
}
log.Printf("[store] backfilled resolved_path for %d observations (async)", n)
time.Sleep(yieldDuration) // yield to HTTP handlers
}
log.Printf("[store] async resolved_path backfill complete")
}
```
Each chunk:
1. Takes a read lock, collects up to `chunkSize` pending observations, releases lock
2. Resolves paths (no lock held — `resolvePathForObs` only reads immutable data)
3. Opens a separate RW SQLite connection, writes results in a transaction
4. Takes a write lock, updates in-memory `obs.ResolvedPath` and re-picks best obs for affected transmissions, releases lock
5. Sleeps briefly to yield CPU/lock time to HTTP handlers
#### 2. Readiness flag and API degraded-mode header
Add a boolean to `PacketStore`:
```go
type PacketStore struct {
// ...
backfillComplete atomic.Bool
}
```
API responses include a header during backfill:
```
X-CoreScope-Status: backfilling
X-CoreScope-Backfill-Remaining: 4523000
```
After backfill completes:
```
X-CoreScope-Status: ready
```
The frontend can read this header and show a subtle banner: *"Resolving hop paths… some paths may show abbreviated pubkeys."*
#### 3. Index rebuilds
The subpath, distance, and path-hop indexes are built during startup from whatever data exists. During backfill, newly resolved paths need to update these indexes incrementally.
Options (in order of preference):
**Option A: Defer index updates to end of backfill.** Indexes work fine with unresolved paths — they just produce slightly less precise results. After backfill completes, rebuild indexes once. Simple, correct, low risk.
**Option B: Incremental index updates per chunk.** After each chunk, update affected index entries. More complex, better real-time accuracy. Only worth it if index accuracy during backfill matters for production use.
**Recommendation: Option A.** The indexes are usable with unresolved paths. A single rebuild at the end (~35s) is cheap compared to the backfill duration. The API works throughout — results just improve after backfill finishes.
#### 4. SQLite contention
The backfill opens a separate RW connection for writes. The main server uses a read-only connection for polling. SQLite WAL mode (already in use) allows concurrent readers and one writer. Contention risk is minimal:
- Write transactions are small (5,000 UPDATEs per chunk, batched in a single tx)
- Read queries from HTTP handlers are unaffected by WAL writes
- The 100ms yield between chunks prevents sustained write pressure
#### 5. Lock contention
The write lock is held only during the in-memory update phase of each chunk (~5,000 pointer assignments + re-picks). This takes microseconds. HTTP handlers acquire read locks for API responses — they will not be blocked for any perceptible duration.
#### 6. Frontend handling
The `hop-resolver.js` module already handles unresolved (prefix) hops gracefully — it shows abbreviated pubkeys. No frontend changes are required for correctness.
Optional enhancement: read the `X-CoreScope-Status` header and show a transient info banner during backfill. This is cosmetic and can be done in a follow-up.
### What about first-run specifically?
On first run with a pre-existing database (e.g., migrating from a version without `resolved_path`), ALL 7.3M observations need backfill. The async approach handles this identically — it just takes longer in the background while HTTP is already serving.
On subsequent restarts, `resolved_path` is already persisted in SQLite and loaded by `store.Load()`. The backfill loop finds zero pending observations and exits immediately.
### What about new observations during backfill?
The poller ingests new packets continuously. New observations written by the ingestor already have `resolved_path` set at ingest time (this is already implemented). The backfill only processes observations with `ResolvedPath == nil`, so there's no conflict with new data.
## Alternatives considered
### Lazy resolution (resolve on API access)
Resolve `resolved_path` only when an observation is accessed via API, cache the result.
**Rejected because:**
- Adds latency to every API call that touches unresolved observations
- Cache invalidation complexity (when does a cached resolution become stale?)
- Doesn't help with index accuracy — indexes still need full data
- The backfill is a one-time cost; lazy resolution makes it a recurring cost
### Progressive loading (recent data first)
Load only the last 24h into memory, start serving, load historical data in background.
**Rejected because:**
- Significantly more complex — all store operations need "is this data loaded yet?" checks
- Memory implications: need to track which time ranges are loaded
- Historical queries return wrong results during loading (not just degraded — wrong)
- The actual bottleneck is backfill, not `Load()`. Even loading all 7.3M observations takes only ~90s.
### Chunked blocking backfill (yield to HTTP between chunks, but keep in main startup)
Process N observations per tick with `runtime.Gosched()` between chunks, but still in `main()` before `ListenAndServe`.
**Rejected because:**
- HTTP still isn't available until all chunks complete
- Adds complexity without solving the core problem
## Carmack Review (Performance)
**The approach is sound.** Moving a 2030 minute blocking operation to a background goroutine is the right call. Some notes:
1. **Chunk size tuning.** 5,000 is a reasonable starting point. Monitor: if write lock contention shows up in pprof (unlikely with microsecond hold times), reduce chunk size. If backfill is too slow, increase it or reduce yield time.
2. **Memory is not a concern.** The observations are already fully loaded in memory by `Load()`. The backfill only mutates the `ResolvedPath` field on existing objects — no additional memory allocation beyond temporary slices for the chunk.
3. **No hidden costs in `resolvePathForObs`.** It reads `nodePM` (a `PrefixMatcher`, immutable after startup) and `graph` (neighbor graph, immutable after startup). No locks needed during resolution. This is embarrassingly parallelizable if needed, but single-goroutine processing with chunking is sufficient.
4. **The index rebuild at the end is O(n) and takes ~35s.** This is a one-time cost after the first backfill. Not worth optimizing further unless the profile shows otherwise.
5. **Risk: `pickBestObservation` during backfill.** API responses may flip their "best" observation as resolved paths become available. This is cosmetically noisy but functionally correct. Document this as expected behavior.
6. **Future optimization if needed:** The backfill loop could be parallelized across multiple goroutines (partition observations by transmission hash). The resolution step is CPU-bound and read-only. This would reduce backfill wall time from 30 min to ~5 min on 8 cores. Not needed for MVP — the goal is HTTP availability, not backfill speed.
## Implementation plan
1. **Refactor `backfillResolvedPaths` into chunked async version** — new function `backfillResolvedPathsAsync` that processes in chunks and yields
2. **Move backfill call in `main.go` to after `ListenAndServe`** — wrap in goroutine
3. **Add `backfillComplete` atomic flag to `PacketStore`** — set after backfill finishes
4. **Add `X-CoreScope-Status` response header** — middleware reads the flag
5. **Rebuild indexes after backfill completes** — single call to rebuild subpath/distance/path-hop
6. **Tests:** unit test for chunked backfill (mock store with N unresolved obs, verify chunks process correctly)
7. **Frontend (follow-up):** optional banner during backfill state
Estimated effort: 12 hours for steps 15, plus tests.
+144
View File
@@ -0,0 +1,144 @@
# Table Sorting Consistency Spec (#620)
## Problem
CoreScope has 20+ data tables. Only 2 are sortable (nodes list, channel activity). Those 2 use incompatible implementations — different property names (`column`/`direction` vs `col`/`dir`), different data attributes (`data-sort` vs `data-sort-col`), different function signatures. The remaining 18+ tables, including the packets table (30K+ rows), have zero sorting.
This violates AGENTS.md DRY rules and frustrates users who can see data but can't reorder it.
## Solution
One shared `TableSort` module. Every data table uses it. Same UX everywhere.
## Shared Utility Design
### Module: `public/table-sort.js`
IIFE pattern (like `channel-colors.js`). No dependencies. No build step.
```js
window.TableSort = (function() {
return { init, sort, destroy };
})();
```
### API
```js
TableSort.init(tableEl, {
defaultColumn: 'last_seen', // initial sort column
defaultDirection: 'desc', // 'asc' or 'desc'
storageKey: 'nodes-sort', // localStorage key (optional)
comparators: { // custom comparators for non-string columns
time: (a, b) => ...,
snr: (a, b) => ...,
},
onSort: (column, direction) => {} // callback after sort completes
});
```
### How It Works
1. Scans `<th>` elements for `data-sort="columnName"` attribute
2. Attaches click handlers — click toggles asc/desc
3. On sort: reads `<td data-value="...">` (raw sortable value) from each row
4. Sorts rows in-place via DOM reorder (no innerHTML rebuild — important for 30K rows)
5. Updates visual indicator and `aria-sort` on active `<th>`
### Visual Indicator
Active column header gets `▲` (ascending) or `▼` (descending) appended as a `<span class="sort-arrow">`. Inactive columns show no arrow. CSS class `.sort-active` on the active `<th>`.
### Built-in Comparators
| Type | Detected From | Behavior |
|------|--------------|----------|
| `numeric` | `data-type="number"` on `<th>` | `Number(a) - Number(b)`, NaN sorts last |
| `text` | default | `localeCompare` |
| `date` | `data-type="date"` | Parse as timestamp, numeric compare |
| `dbm` | `data-type="dbm"` | Strip " dBm" suffix, numeric compare |
Custom comparators in `options.comparators` override built-in types.
### Accessibility
- `aria-sort="ascending"`, `"descending"`, or `"none"` on every sortable `<th>`
- `role="columnheader"` (already implicit for `<th>`)
- `cursor: pointer` and `:hover` style on sortable headers
- Keyboard: sortable headers are focusable, Enter/Space triggers sort
### Performance (Critical for Packets Table)
- Sort via DOM node reorder (`appendChild` loop), not `innerHTML`. Browser batches reflows.
- `data-value` attributes hold raw values — no parsing during sort.
- For 30K rows: expected sort time ~100-200ms (single `Array.sort` + DOM reorder). If >500ms, add a virtual scroll layer in a follow-up — but don't pre-optimize.
- No re-render of row content. Sort only changes order.
## Milestones
### M1: Shared utility + packets table
- Create `public/table-sort.js`
- Unit tests: `test-table-sort.js` (Node.js, jsdom or vm.createContext)
- Integrate with packets table (highest impact — 30K rows, currently unsortable)
- Default sort: time descending
- Columns: all current packets columns (Region, Time, Hash, Size, HB, Type, Observer, Path, Rpt, Details)
- Browser validation: sort 30K rows, verify <500ms
### M2: Nodes list + node detail tables
- Migrate nodes list from custom sort to `TableSort.init()`
- Add sorting to neighbor table (side pane + detail page)
- Add sorting to observer stats table (detail page)
- Remove old `sortState`/`sortArrow` code from `nodes.js`
### M3: Analytics tables
- Hash collisions tables (node table, sizes table, collision prefixes)
- RF statistics table
- Route frequency, co-appearance, topology tables
- Node health tables (top by packets/SNR/observers, recently active)
- Distance tables (by link type, top 20 longest)
- Per-node analytics: peer contacts
### M4: Channels list + observers list + comparison table
- Channel activity table: migrate from custom sort to `TableSort.init()`
- Remove old `_channelSortState` code from `analytics.js`
- Observers list table
- Comparison table (`compare.js`)
### M5: Cleanup
- Remove all old sorting code (both implementations)
- Verify no dead CSS/JS from old sort code
- Final consistency audit: every data table uses `TableSort.init()`
### Out of Scope
- `packets.js` hex breakdown (structural decode, fixed order)
- `audio-lab.js` debug tables (not user-facing)
- Virtual scroll / pagination (separate issue if perf requires it)
## Testing
### Unit Tests (`test-table-sort.js`)
- Numeric sort ascending/descending
- Text sort with localeCompare
- Date sort
- dBm sort (strip suffix)
- Custom comparator override
- NaN/null/undefined sort to end
- Toggle direction on repeated click
- `aria-sort` attribute updates
- localStorage persistence (read + write)
- `data-value` attribute used over text content
### Integration (per milestone)
- Playwright test: click column header, verify row order changes
- Playwright test: click again, verify direction toggles
- Playwright test: visual indicator present on active column
### Performance
- Unit test: sort 30K mock rows in <500ms (assert timing)
- Required per AGENTS.md: perf claims need proof
## Migration Path
Existing sort code in `nodes.js` and `analytics.js` will be replaced, not wrapped. Both current implementations are <100 lines each — replacing is simpler than adapting. The shared utility subsumes all their functionality.
Old localStorage keys (`nodes-sort-*`, channel sort state) should be migrated or cleared on first use of the new utility.
+266
View File
@@ -0,0 +1,266 @@
# Timestamp-Based Packet Filters
**Issue:** #289
**Status:** Draft
**Depends on:** #286 (timestamp display config)
## Summary
Extend the existing filter engine (`packet-filter.js`) with a `time` field type supporting absolute ISO timestamps, relative durations, and range expressions. The filter compiles date expressions to epoch milliseconds at parse time so per-packet evaluation is a single numeric comparison — no date parsing in the hot path.
## Syntax
### Absolute (ISO 8601)
```
time > "2024-01-01T00:00:00Z"
time <= "2024-06-15"
time == "2024-03-01"
```
Quoted strings after `time` are parsed as dates. Partial dates (`"2024-01-01"`) are treated as midnight UTC. All absolute values are interpreted as UTC regardless of the user's display preference.
### Relative
```
time > 2h ago
time > 30m ago
time > 7d ago
```
The lexer recognizes `<number><unit> ago` as a relative time literal. Supported units: `s` (seconds), `m` (minutes), `h` (hours), `d` (days). At compile time, the relative offset is resolved to an absolute epoch ms value (`Date.now() - offset`). This means a compiled filter's relative thresholds are frozen at compile time — recompile to refresh.
### Shorthand
```
time.ago < 30m
time.ago < 2h
```
`time.ago` resolves to `Date.now() - packet.timestamp`. The comparison value is a duration literal (`30m`, `2h`, `7d`). This is syntactic sugar and semantically equivalent to the relative form but reads more naturally for "show me recent packets."
### Range
```
time between "2024-01-01" "2024-01-02"
time between 1h ago 30m ago
```
`between` is a ternary operator: `field between <low> <high>`. Compiles to `low <= field && field <= high`. Both bounds are inclusive.
### Combinable with existing filters
```
type == Advert && time > 1h ago
snr > 5 && time between "2024-01-01" "2024-01-02"
(type == GRP_TXT || type == TXT_MSG) && time.ago < 30m
```
## Grammar Extension
### New token types
| Token | Pattern | Example |
|-------|---------|---------|
| `DURATION` | `/^\d+[smhd]$/` | `30m`, `2h`, `7d` |
| `AGO` | keyword `ago` | `ago` |
| `BETWEEN` | keyword `between` | `between` |
### Lexer changes
1. After reading an identifier that matches `\d+[smhd]`, emit `DURATION` token instead of `FIELD`.
2. Recognize `ago` and `between` as keywords (like `and`/`or`).
### Parser changes
In `parseComparison()`:
1. **Relative time:** If field is `time` and value tokens are `DURATION AGO`, compute `Date.now() - durationToMs(duration)` and store as a numeric epoch ms value in the AST node.
2. **Absolute time:** If field is `time` and value is a `STRING`, attempt `new Date(value).getTime()`. If `NaN`, return parse error. Store epoch ms.
3. **`time.ago` shorthand:** If field is `time.ago`, the value is a `DURATION`. Store the duration in ms. At evaluation, compute `now - packet_ts` and compare against the duration.
4. **`between`:** If operator token is `BETWEEN`, consume two values (same type resolution as above). Emit `{ type: 'between', field, low, high }`.
### AST node shapes
```js
// Absolute/relative (pre-resolved to epoch ms)
{ type: 'comparison', field: 'time', op: '>', value: 1704067200000 }
// time.ago (duration in ms)
{ type: 'comparison', field: 'time.ago', op: '<', value: 1800000 }
// between (both bounds as epoch ms)
{ type: 'between', field: 'time', low: 1704067200000, high: 1704153600000 }
```
## Field Resolution
Add to `resolveField()`:
```js
if (field === 'time') return packet.timestamp; // epoch ms
if (field === 'time.ago') return Date.now() - packet.timestamp;
```
`packet.timestamp` is the packet's capture time in epoch milliseconds. This field already exists in the data model (populated from the DB `created_at` column).
## Time Semantics
- **Filter expressions:** Always UTC. `"2024-01-01"` means `2024-01-01T00:00:00Z`.
- **Display:** Follows the user's timestamp config from #286 (UTC/local/relative).
- **Relative times:** Computed against `Date.now()` at compile time. The compiled filter is a snapshot — if the filter stays active for hours, relative thresholds drift. This is acceptable; filters are typically short-lived or recompiled on interaction.
**No timezone specifiers in the filter syntax.** UTC only. This avoids ambiguity and parsing complexity. Users who think in local time can use the relative syntax (`time > 2h ago`) which is timezone-agnostic.
## Performance
### Compile-time work (once)
- Parse date strings → epoch ms via `new Date().getTime()` (~1μs per date)
- Parse duration strings → ms via multiplication (~0ns, trivial arithmetic)
- Relative `ago``Date.now() - offset` (~0ns)
### Per-packet evaluation (hot path)
- `time` comparison: one numeric read + one numeric compare. Same cost as `snr > 5`.
- `time.ago`: one subtraction + one compare. Two arithmetic ops. **Important:** cache `Date.now()` once per filter pass (e.g., in a closure variable set before iterating packets), not per-packet. 30K `Date.now()` calls are ~1ms but it's a pointless syscall tax.
- `between`: two numeric compares.
**No `Date` objects created per packet. No string parsing per packet. No regex per packet.**
At 30K packets, the time filter adds ~0.1ms total to filter evaluation — dominated by the existing field resolution and AST walk overhead. No measurable regression.
### Implementation note: `between` as sugar
`between` should compile to `{ type: 'and', left: { type: 'comparison', field, op: '>=', value: low }, right: { type: 'comparison', field, op: '<=', value: high } }` — reusing existing comparison evaluation. No new AST node type, no new evaluator branch. The parser desugars it; the evaluator never sees `between`.
### Implementation note: `time.ago` and `Date.now()` caching
The `compile()` function should return a filter that accepts an optional `now` parameter:
```js
var compiled = compile('time.ago < 30m');
var now = Date.now();
packets.filter(function(p) { return compiled.filter(p, now); });
```
If `now` is not passed, `Date.now()` is called once on the first invocation and reused for the entire filter pass. This avoids 30K syscalls and ensures consistent evaluation within a single pass.
## Carmack Review Notes
Reviewed with a performance-first lens (30K+ packets, real-time updates):
1. **✅ No allocations in hot path.** All date parsing happens at compile time. Per-packet evaluation is pure numeric comparison — same cost as existing `snr > 5` filters.
2. **⚠️ `Date.now()` per-packet for `time.ago`.** Fixed above — cache once per filter pass via optional `now` parameter or closure. Without this, 30K packets × `Date.now()` = ~1ms wasted on a monotonic clock syscall that returns the same value.
3. **`between` as sugar, not a new node type.** Desugar in the parser to reuse existing `and` + `comparison` nodes. Zero new code paths in the evaluator = zero new bugs in the evaluator.
4. **✅ Parser complexity is bounded.** Three new token types, one new keyword. The parser remains LL(1) — no backtracking, no ambiguity. `DURATION AGO` is a clear two-token lookahead only when field is `time`.
5. **✅ Memory impact negligible.** Compiled time filters add one or two floats to the AST. At 16 bytes per node, even complex expressions with multiple time clauses are <100 bytes.
6. **⚠️ Compiled filter staleness for relative times.** Spec acknowledges this. Acceptable for a web UI where filters are recompiled on user interaction. If filters persist across long WebSocket sessions, consider recompiling on a timer (every 60s). This is a future concern, not a blocker.
7. **✅ No regex in hot path.** Duration parsing uses a simple char check on the last character + `parseInt`. Cheaper than any regex.
A compiled time filter adds one or two 64-bit float values to the AST. Negligible — roughly 16 bytes per time comparison node.
## URL Integration
Time filters appear in the URL hash query string like any other filter:
```
#/packets?filter=time%20%3E%201h%20ago
#/packets?filter=type%20%3D%3D%20Advert%20%26%26%20time%20%3E%20%222024-01-01%22
```
The filter text is URL-encoded and round-trips through `encodeURIComponent`/`decodeURIComponent`. No special handling needed — the existing filter-in-URL mechanism (#286 or current) works unchanged.
For convenience, a future milestone could add dedicated `timeFrom`/`timeTo` query params that inject into the filter, but this is not required for the initial implementation.
## Wireshark Compatibility
| Wireshark syntax | CoreScope equivalent | Notes |
|------------------|---------------------|-------|
| `frame.time >= "2024-01-01"` | `time >= "2024-01-01"` | We use `time` instead of `frame.time` for brevity. Could alias `frame.time``time` later. |
| `frame.time_relative < 60` | `time.ago < 60s` | Wireshark uses seconds float; we use duration literals |
| `frame.time_delta` | Not supported | Inter-packet delta is a different feature |
We intentionally diverge from Wireshark where their syntax is verbose or requires pcap-specific concepts. CoreScope's filter language prioritizes brevity and readability for a web UI. A `frame.time` alias for `time` can be added trivially in the field resolver if users request it.
## Milestones
### M1: Core time filtering (parser + evaluator)
- Add `DURATION`, `AGO`, `BETWEEN` tokens to lexer
- Extend parser for `time` field special handling
- Add `time` and `time.ago` to `resolveField()`
- Implement `between` AST node evaluation
- Unit tests: absolute, relative, ago, between, combined with existing filters, edge cases (bad dates, invalid units)
- **Test:** filter 30K packets by time in <50ms (assert in test)
### M2: UI integration
- Filter bar autocomplete hints for time syntax
- Help tooltip / cheat sheet update with time examples
- Verify URL round-trip with time filters
- Playwright E2E test: enter time filter, verify packet list updates
### M3: Polish
- `frame.time` alias
- Error messages for common mistakes ("did you mean `time > 1h ago`?")
- Consider dedicated time range picker UI widget (out of scope for this spec)
## Testing
### Unit tests (add to `test-packet-filter.js`)
```js
// Absolute time
c = compile('time > "2024-01-01"');
assert(c.filter({ timestamp: new Date('2024-06-01').getTime() }), 'after 2024-01-01');
assert(!c.filter({ timestamp: new Date('2023-06-01').getTime() }), 'before 2024-01-01');
// Relative time
c = compile('time > 1h ago');
assert(c.filter({ timestamp: Date.now() - 30 * 60000 }), '30m ago passes 1h filter');
assert(!c.filter({ timestamp: Date.now() - 2 * 3600000 }), '2h ago fails 1h filter');
// time.ago shorthand
c = compile('time.ago < 30m');
assert(c.filter({ timestamp: Date.now() - 10 * 60000 }), '10m ago < 30m');
assert(!c.filter({ timestamp: Date.now() - 60 * 60000 }), '60m ago not < 30m');
// between
c = compile('time between "2024-01-01" "2024-01-02"');
assert(c.filter({ timestamp: new Date('2024-01-01T12:00:00Z').getTime() }), 'in range');
assert(!c.filter({ timestamp: new Date('2024-01-03').getTime() }), 'out of range');
// Combined
c = compile('type == Advert && time > 1h ago');
assert(c.filter({ payload_type: 4, timestamp: Date.now() - 1000 }), 'combined pass');
assert(!c.filter({ payload_type: 4, timestamp: Date.now() - 7200000 }), 'combined fail time');
assert(!c.filter({ payload_type: 1, timestamp: Date.now() - 1000 }), 'combined fail type');
// Error cases
c = compile('time > "not-a-date"');
assert(c.error, 'invalid date string');
c = compile('time > 5x ago');
assert(c.error, 'invalid duration unit');
// Performance
var start = Date.now();
c = compile('time > 1h ago && type == Advert');
var packets = [];
for (var i = 0; i < 30000; i++) {
packets.push({ payload_type: i % 5, timestamp: Date.now() - i * 1000 });
}
packets.forEach(function(p) { c.filter(p); });
assert(Date.now() - start < 50, 'filter 30K packets in <50ms');
```
### Playwright tests
- Enter `time > 1h ago` in filter bar → verify packet count decreases
- Enter invalid time filter → verify error message appears
- Reload page with time filter in URL → verify filter is applied
@@ -0,0 +1,674 @@
# Deep Linking P1 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make P1 UI states in nodes, packets, and channels URL-addressable so they survive refresh and can be shared.
**Architecture:** Each page reads URL params from `location.hash.split('?')[1]` on init (router strips query string before passing `routeParam`, so pages must read `location.hash` directly). State changes call `history.replaceState` to keep the URL in sync. localStorage remains the fallback default; URL params override when present.
**Tech Stack:** Vanilla JS (ES5/6), browser History API, URLSearchParams
---
## Files Changed
| File | Changes |
|---|---|
| `public/region-filter.js` | Add `setSelected(codesArray)`, track `_container` for re-render |
| `public/nodes.js` | Read `?tab=`/`?search=` on init; `updateNodesUrl()` on tab/search change; expose `buildNodesQuery` on `window` |
| `public/packets.js` | Read `?timeWindow=`/`?region=` on init; `updatePacketsUrl()` on timeWindow/region change; expose `buildPacketsUrl` on `window` |
| `public/channels.js` | Read `?node=` on init; update URL in `showNodeDetail`/`closeNodeDetail` |
| `test-frontend-helpers.js` | Add unit tests for `buildNodesQuery` and `buildPacketsUrl` |
| `test-e2e-playwright.js` | Add Playwright tests: tab URL persistence, timeWindow URL persistence |
---
## Task 1: Add `setSelected` to RegionFilter
**Files:**
- Modify: `public/region-filter.js`
- [ ] **Step 1: Write the failing unit test**
Add to `test-frontend-helpers.js` before the `// ===== SUMMARY =====` line:
```javascript
// ===== REGION-FILTER.JS: setSelected =====
console.log('\n=== region-filter.js: setSelected ===');
{
const ctx = makeSandbox();
ctx.fetch = () => Promise.resolve({ json: () => Promise.resolve({ 'US-SFO': 'San Jose', 'US-LAX': 'Los Angeles' }) });
loadInCtx(ctx, 'public/region-filter.js');
const RF = ctx.RegionFilter;
RF.init(document.createElement('div'));
test('setSelected sets region codes', async () => {
await RF.init(document.createElement('div'));
RF.setSelected(['US-SFO', 'US-LAX']);
assert.strictEqual(RF.getRegionParam(), 'US-SFO,US-LAX');
});
test('setSelected with null clears selection', async () => {
await RF.init(document.createElement('div'));
RF.setSelected(['US-SFO']);
RF.setSelected(null);
assert.strictEqual(RF.getRegionParam(), '');
});
test('setSelected with empty array clears selection', async () => {
await RF.init(document.createElement('div'));
RF.setSelected(['US-SFO']);
RF.setSelected([]);
assert.strictEqual(RF.getRegionParam(), '');
});
}
```
- [ ] **Step 2: Run test to verify it fails**
```bash
node test-frontend-helpers.js 2>&1 | grep -A2 "setSelected"
```
Expected: `❌ setSelected sets region codes: RF.setSelected is not a function`
- [ ] **Step 3: Add `_container` tracking and `setSelected` to region-filter.js**
In `region-filter.js`, add `var _container = null;` after the existing module-level vars (after line 9 `var _listeners = [];`):
```javascript
var _listeners = [];
var _container = null; // ← add this line
var _loaded = false;
```
In `initFilter`, save the container:
```javascript
async function initFilter(container, opts) {
_container = container; // ← add this line
if (opts && opts.dropdown) container._forceDropdown = true;
await fetchRegions();
render(container);
}
```
Add `setSelected` function before `// Expose globally`:
```javascript
/** Override selected regions (e.g. from URL param). Persists to localStorage and re-renders. */
function setSelected(codesArray) {
_selected = (codesArray && codesArray.length > 0) ? new Set(codesArray) : null;
saveToStorage();
if (_container) render(_container);
}
```
Add `setSelected` to the public API object:
```javascript
window.RegionFilter = {
init: initFilter,
render: render,
getSelected: getSelected,
getRegionParam: getRegionParam,
regionQueryString: regionQueryString,
onChange: onChange,
offChange: offChange,
fetchRegions: fetchRegions,
setSelected: setSelected, // ← add this line
};
```
- [ ] **Step 4: Run test to verify it passes**
```bash
node test-frontend-helpers.js 2>&1 | grep -E "(setSelected|FAIL|passed|failed)"
```
Expected: 3 passing `setSelected` tests, overall pass.
- [ ] **Step 5: Commit**
```bash
git add public/region-filter.js test-frontend-helpers.js
git commit -m "feat: add RegionFilter.setSelected for URL param initialization (#536)"
```
---
## Task 2: nodes.js — tab and search deep linking
**Files:**
- Modify: `public/nodes.js`
- Test: `test-frontend-helpers.js`
- Test: `test-e2e-playwright.js`
- [ ] **Step 1: Write the unit test (add to test-frontend-helpers.js)**
Add before the `// ===== SUMMARY =====` line:
```javascript
// ===== NODES.JS: buildNodesQuery =====
console.log('\n=== nodes.js: buildNodesQuery ===');
{
const ctx = makeSandbox();
loadInCtx(ctx, 'public/roles.js');
loadInCtx(ctx, 'public/app.js');
// Provide required globals for nodes.js IIFE to execute
ctx.registerPage = () => {};
ctx.RegionFilter = { init: () => Promise.resolve(), onChange: () => () => {}, offChange: () => {}, getSelected: () => null, getRegionParam: () => '' };
ctx.onWS = () => {};
ctx.offWS = () => {};
ctx.debouncedOnWS = () => () => {};
ctx.invalidateApiCache = () => {};
ctx.favStar = () => '';
ctx.bindFavStars = () => {};
ctx.getFavorites = () => [];
ctx.isFavorite = () => false;
ctx.connectWS = () => {};
ctx.HopResolver = { init: () => {}, resolve: () => ({}), ready: () => false };
ctx.initTabBar = () => {};
ctx.debounce = (fn) => fn;
ctx.copyToClipboard = () => {};
ctx.api = () => Promise.resolve({});
ctx.escapeHtml = (s) => s;
ctx.timeAgo = () => '';
ctx.formatTimestampWithTooltip = () => '';
ctx.getTimestampMode = () => 'ago';
ctx.CLIENT_TTL = {};
ctx.qrcode = null;
try {
const src = fs.readFileSync('public/nodes.js', 'utf8');
vm.runInContext(src, ctx);
for (const k of Object.keys(ctx.window)) ctx[k] = ctx.window[k];
} catch (e) {
console.log(' ⚠️ nodes.js sandbox load failed:', e.message.slice(0, 120));
}
const buildNodesQuery = ctx.buildNodesQuery;
if (buildNodesQuery) {
test('buildNodesQuery: all tab + no search = empty', () => {
assert.strictEqual(buildNodesQuery('all', ''), '');
});
test('buildNodesQuery: repeater tab only', () => {
assert.strictEqual(buildNodesQuery('repeater', ''), '?tab=repeater');
});
test('buildNodesQuery: search only (all tab)', () => {
assert.strictEqual(buildNodesQuery('all', 'foo'), '?search=foo');
});
test('buildNodesQuery: tab + search combined', () => {
assert.strictEqual(buildNodesQuery('companion', 'bar'), '?tab=companion&search=bar');
});
test('buildNodesQuery: null search treated as empty', () => {
assert.strictEqual(buildNodesQuery('all', null), '');
});
test('buildNodesQuery: sensor tab', () => {
assert.strictEqual(buildNodesQuery('sensor', ''), '?tab=sensor');
});
} else {
console.log(' ⚠️ buildNodesQuery not exposed — skipping');
}
}
```
- [ ] **Step 2: Run test to verify it fails (or skips)**
```bash
node test-frontend-helpers.js 2>&1 | grep -A3 "buildNodesQuery"
```
Expected: `⚠️ buildNodesQuery not exposed — skipping`
- [ ] **Step 3: Add URL param reading and helpers to nodes.js**
**3a.** Add `buildNodesQuery` and `updateNodesUrl` functions inside the nodes.js IIFE, after the `TABS` definition (around line 86, before `function renderNodeTimestampHtml`):
```javascript
function buildNodesQuery(tab, searchStr) {
var parts = [];
if (tab && tab !== 'all') parts.push('tab=' + encodeURIComponent(tab));
if (searchStr) parts.push('search=' + encodeURIComponent(searchStr));
return parts.length ? '?' + parts.join('&') : '';
}
window.buildNodesQuery = buildNodesQuery;
function updateNodesUrl() {
history.replaceState(null, '', '#/nodes' + buildNodesQuery(activeTab, search));
}
```
**3b.** In the list-view branch of `init` (after the `return;` that ends the full-screen block at line 317), add URL param reading before `app.innerHTML`:
```javascript
// Read URL params for list view (router strips query string from routeParam)
const _listUrlParams = new URLSearchParams(location.hash.split('?')[1] || '');
const _urlTab = _listUrlParams.get('tab');
const _urlSearch = _listUrlParams.get('search');
if (_urlTab && TABS.some(function(t) { return t.key === _urlTab; })) activeTab = _urlTab;
if (_urlSearch) search = _urlSearch;
app.innerHTML = `<div class="nodes-page">
```
**3c.** After `app.innerHTML = ...` (after the closing backtick at line ~330), populate the search input:
```javascript
if (search) {
var _si = document.getElementById('nodeSearch');
if (_si) _si.value = search;
}
```
**3d.** In the search input event listener (around line 335), add `updateNodesUrl()`:
```javascript
document.getElementById('nodeSearch').addEventListener('input', debounce(e => {
search = e.target.value;
updateNodesUrl();
loadNodes();
}, 250));
```
**3e.** In the tab click handler inside `renderLeft` (around line 875), add `updateNodesUrl()`:
```javascript
btn.addEventListener('click', () => { activeTab = btn.dataset.tab; updateNodesUrl(); loadNodes(); });
```
- [ ] **Step 4: Run unit tests**
```bash
node test-frontend-helpers.js 2>&1 | grep -E "(buildNodesQuery|✅|❌)" | grep -v "helpers"
```
Expected: 6 passing `buildNodesQuery` tests.
- [ ] **Step 5: Write Playwright test (add to test-e2e-playwright.js)**
Add before the closing `await browser.close()` line:
```javascript
// --- Group: Deep linking (#536) ---
// Test: nodes tab deep link
await test('Nodes tab deep link restores active tab', async () => {
await page.goto(BASE + '#/nodes?tab=repeater', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.node-tab', { timeout: 8000 });
const activeTab = await page.$('.node-tab.active');
assert(activeTab, 'No active tab found');
const tabText = await activeTab.textContent();
assert(tabText.includes('Repeater'), `Expected Repeater tab active, got: ${tabText}`);
const url = page.url();
assert(url.includes('tab=repeater'), `URL should contain tab=repeater, got: ${url}`);
});
// Test: nodes tab click updates URL
await test('Nodes tab click updates URL', async () => {
await page.goto(BASE + '#/nodes', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.node-tab', { timeout: 8000 });
const roomTab = await page.$('.node-tab[data-tab="room"]');
if (roomTab) {
await roomTab.click();
await page.waitForTimeout(300);
const url = page.url();
assert(url.includes('tab=room'), `URL should contain tab=room after click, got: ${url}`);
}
});
```
- [ ] **Step 6: Run full test suite**
```bash
node test-frontend-helpers.js
```
Expected: all tests pass.
- [ ] **Step 7: Commit**
```bash
git add public/nodes.js test-frontend-helpers.js test-e2e-playwright.js
git commit -m "feat: deep link nodes tab and search query (#536)"
```
---
## Task 3: packets.js — timeWindow and region deep linking
**Files:**
- Modify: `public/packets.js`
- Test: `test-frontend-helpers.js`
- Test: `test-e2e-playwright.js`
> Depends on Task 1 (RegionFilter.setSelected).
- [ ] **Step 1: Write the unit test**
Add to `test-frontend-helpers.js` before `// ===== SUMMARY =====`:
```javascript
// ===== PACKETS.JS: buildPacketsUrl =====
console.log('\n=== packets.js: buildPacketsUrl ===');
{
// Test the pure helper function
// (loaded via packets.js after it exposes window.buildPacketsUrl)
const ctx = makeSandbox();
loadInCtx(ctx, 'public/roles.js');
loadInCtx(ctx, 'public/app.js');
ctx.registerPage = () => {};
ctx.RegionFilter = { init: () => Promise.resolve(), onChange: () => () => {}, offChange: () => {}, getSelected: () => null, getRegionParam: () => '', setSelected: () => {} };
ctx.onWS = () => {};
ctx.offWS = () => {};
ctx.debouncedOnWS = () => () => {};
ctx.invalidateApiCache = () => {};
ctx.api = () => Promise.resolve({});
ctx.observerMap = new Map();
ctx.getParsedPath = () => [];
ctx.getParsedDecoded = () => ({});
ctx.clearParsedCache = () => {};
ctx.escapeHtml = (s) => s;
ctx.timeAgo = () => '';
ctx.formatTimestampWithTooltip = () => '';
ctx.getTimestampMode = () => 'ago';
ctx.copyToClipboard = () => {};
ctx.CLIENT_TTL = {};
ctx.debounce = (fn) => fn;
ctx.initTabBar = () => {};
try {
const src = fs.readFileSync('public/packet-helpers.js', 'utf8');
vm.runInContext(src, ctx);
for (const k of Object.keys(ctx.window)) ctx[k] = ctx.window[k];
const src2 = fs.readFileSync('public/packets.js', 'utf8');
vm.runInContext(src2, ctx);
for (const k of Object.keys(ctx.window)) ctx[k] = ctx.window[k];
} catch (e) {
console.log(' ⚠️ packets.js sandbox load failed:', e.message.slice(0, 120));
}
const buildPacketsUrl = ctx.buildPacketsUrl;
if (buildPacketsUrl) {
test('buildPacketsUrl: default (15min, no region) = bare #/packets', () => {
assert.strictEqual(buildPacketsUrl(15, ''), '#/packets');
});
test('buildPacketsUrl: non-default timeWindow', () => {
assert.strictEqual(buildPacketsUrl(60, ''), '#/packets?timeWindow=60');
});
test('buildPacketsUrl: region only', () => {
assert.strictEqual(buildPacketsUrl(15, 'US-SFO'), '#/packets?region=US-SFO');
});
test('buildPacketsUrl: timeWindow + region', () => {
assert.strictEqual(buildPacketsUrl(30, 'US-SFO,US-LAX'), '#/packets?timeWindow=30&region=US-SFO%2CUS-LAX');
});
test('buildPacketsUrl: timeWindow=0 treated as default', () => {
assert.strictEqual(buildPacketsUrl(0, ''), '#/packets');
});
} else {
console.log(' ⚠️ buildPacketsUrl not exposed — skipping');
}
}
```
- [ ] **Step 2: Run to verify it skips**
```bash
node test-frontend-helpers.js 2>&1 | grep -A2 "buildPacketsUrl"
```
Expected: `⚠️ buildPacketsUrl not exposed — skipping`
- [ ] **Step 3: Add helpers and URL param reading to packets.js**
**3a.** Add `buildPacketsUrl` and `updatePacketsUrl` inside the packets.js IIFE, after the existing constants at the top (around line 36, after `let showHexHashes`):
```javascript
function buildPacketsUrl(timeWindowMin, regionParam) {
var parts = [];
if (timeWindowMin && timeWindowMin !== 15) parts.push('timeWindow=' + timeWindowMin);
if (regionParam) parts.push('region=' + encodeURIComponent(regionParam));
return '#/packets' + (parts.length ? '?' + parts.join('&') : '');
}
window.buildPacketsUrl = buildPacketsUrl;
function updatePacketsUrl() {
history.replaceState(null, '', buildPacketsUrl(savedTimeWindowMin, RegionFilter.getRegionParam()));
}
```
**3b.** In the `init` function (around line 263), add URL param reading after the existing `routeParam`/`directObsId` parsing and before `app.innerHTML`:
```javascript
// Read URL params for filter state (router strips query from routeParam; read from location.hash)
var _initUrlParams = new URLSearchParams(location.hash.split('?')[1] || '');
var _urlTimeWindow = Number(_initUrlParams.get('timeWindow'));
if (Number.isFinite(_urlTimeWindow) && _urlTimeWindow > 0) {
savedTimeWindowMin = _urlTimeWindow;
localStorage.setItem('meshcore-time-window', String(_urlTimeWindow));
}
var _urlRegion = _initUrlParams.get('region');
if (_urlRegion) {
RegionFilter.setSelected(_urlRegion.split(',').filter(Boolean));
}
app.innerHTML = `<div class="split-layout detail-collapsed">
```
**3c.** In the time window change handler (around line 865), add `updatePacketsUrl()`:
```javascript
fTimeWindow.addEventListener('change', () => {
savedTimeWindowMin = Number(fTimeWindow.value);
if (!Number.isFinite(savedTimeWindowMin) || savedTimeWindowMin <= 0) savedTimeWindowMin = 15;
localStorage.setItem('meshcore-time-window', fTimeWindow.value);
updatePacketsUrl();
loadPackets();
});
```
**3d.** In the RegionFilter.onChange callback (around line 719), add `updatePacketsUrl()`:
```javascript
RegionFilter.onChange(function() { updatePacketsUrl(); loadPackets(); });
```
- [ ] **Step 4: Run unit tests**
```bash
node test-frontend-helpers.js 2>&1 | grep -E "(buildPacketsUrl|✅|❌)" | grep -v "helpers"
```
Expected: 5 passing `buildPacketsUrl` tests.
- [ ] **Step 5: Write Playwright test (add to test-e2e-playwright.js, inside the deep-linking group)**
```javascript
// Test: packets timeWindow deep link
await test('Packets timeWindow deep link restores dropdown', async () => {
await page.goto(BASE + '#/packets?timeWindow=60', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#fTimeWindow', { timeout: 8000 });
const val = await page.$eval('#fTimeWindow', el => el.value);
assert(val === '60', `Expected timeWindow dropdown = 60, got: ${val}`);
const url = page.url();
assert(url.includes('timeWindow=60'), `URL should still contain timeWindow=60, got: ${url}`);
});
// Test: timeWindow change updates URL
await test('Packets timeWindow change updates URL', async () => {
await page.goto(BASE + '#/packets', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#fTimeWindow', { timeout: 8000 });
await page.selectOption('#fTimeWindow', '30');
await page.waitForTimeout(300);
const url = page.url();
assert(url.includes('timeWindow=30'), `URL should contain timeWindow=30 after change, got: ${url}`);
});
```
- [ ] **Step 6: Run full test suite**
```bash
node test-frontend-helpers.js
```
Expected: all tests pass.
- [ ] **Step 7: Commit**
```bash
git add public/packets.js test-frontend-helpers.js test-e2e-playwright.js
git commit -m "feat: deep link packets timeWindow and region filter (#536)"
```
---
## Task 4: channels.js — node panel deep linking
**Files:**
- Modify: `public/channels.js`
No unit tests needed for this task — the URL manipulation is side-effectful (DOM + History API). Playwright tests cover it.
- [ ] **Step 1: Write the Playwright test (add to test-e2e-playwright.js, inside the deep-linking group)**
```javascript
// Test: channels selected channel survives refresh (already implemented, verify it still works)
await test('Channels channel selection is URL-addressable', async () => {
await page.goto(BASE + '#/channels', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.ch-item', { timeout: 8000 }).catch(() => null);
const firstChannel = await page.$('.ch-item');
if (firstChannel) {
await firstChannel.click();
await page.waitForTimeout(500);
const url = page.url();
assert(url.includes('#/channels/') || url.includes('#/channels'), `URL should reflect channel selection, got: ${url}`);
}
});
```
- [ ] **Step 2: Update `showNodeDetail` to write `?node=` to the URL**
In `channels.js`, in `showNodeDetail` (around line 171), add the URL update right after `selectedNode = name;`:
```javascript
async function showNodeDetail(name) {
_nodePanelTrigger = document.activeElement;
if (_focusTrapCleanup) { _focusTrapCleanup(); _focusTrapCleanup = null; }
const node = await lookupNode(name);
selectedNode = name;
var _chBase = selectedHash ? '#/channels/' + encodeURIComponent(selectedHash) : '#/channels';
history.replaceState(null, '', _chBase + '?node=' + encodeURIComponent(name));
let panel = document.getElementById('chNodePanel');
```
- [ ] **Step 3: Update `closeNodeDetail` to strip `?node=` from the URL**
In `closeNodeDetail` (around line 232), add URL restore right after `selectedNode = null;`:
```javascript
function closeNodeDetail() {
if (_focusTrapCleanup) { _focusTrapCleanup(); _focusTrapCleanup = null; }
const panel = document.getElementById('chNodePanel');
if (panel) panel.classList.remove('open');
selectedNode = null;
var _chRestoreUrl = selectedHash ? '#/channels/' + encodeURIComponent(selectedHash) : '#/channels';
history.replaceState(null, '', _chRestoreUrl);
if (_nodePanelTrigger && typeof _nodePanelTrigger.focus === 'function') {
```
- [ ] **Step 4: Read `?node=` on init and auto-open panel**
In `channels.js` `init` (line 316), add URL param reading at the very top of the function (before `app.innerHTML`):
```javascript
function init(app, routeParam) {
var _initUrlParams = new URLSearchParams(location.hash.split('?')[1] || '');
var _pendingNode = _initUrlParams.get('node');
app.innerHTML = `<div class="ch-layout">
```
Then update the `loadChannels().then(...)` call (around line 350) to auto-open the node panel:
```javascript
loadChannels().then(async function () {
if (routeParam) await selectChannel(routeParam);
if (_pendingNode) showNodeDetail(_pendingNode);
});
```
- [ ] **Step 5: Run full test suite**
```bash
node test-frontend-helpers.js
```
Expected: all tests pass (no channels unit tests, but regression tests still pass).
- [ ] **Step 6: Commit**
```bash
git add public/channels.js
git commit -m "feat: deep link channels node panel via ?node= (#536)"
```
---
## Task 5: Run E2E Playwright tests
- [ ] **Step 1: Start the local server**
```bash
cd cmd/server && go run . &
```
Wait for it to be ready (check `http://localhost:3000`).
- [ ] **Step 2: Run Playwright tests**
```bash
node test-e2e-playwright.js
```
Expected: all tests pass including the new deep-linking group.
- [ ] **Step 3: If any deep-linking test fails, debug**
Common failures:
- Selector `.node-tab.active` not found: check that nodes.js correctly reads `?tab=` from URL before rendering
- `#fTimeWindow` value wrong: check that `savedTimeWindowMin` is overridden before the DOM is built
- URL doesn't update: check `history.replaceState` calls in the change handlers
- [ ] **Step 4: Final commit (if any fixes needed)**
```bash
git add public/nodes.js public/packets.js public/channels.js
git commit -m "fix: deep linking E2E adjustments (#536)"
```
---
## Self-Review
**Spec coverage check:**
- ✅ P1: Nodes role tab → Task 2
- ✅ P1: Packets time window → Task 3
- ✅ P1: Packets region filter → Task 3 (depends on Task 1)
- ✅ P1: Channels selected channel → Already implemented via `#/channels/{hash}` (verified in channels.js init line 351)
- ✅ P1: Channels node panel → Task 4
- ✅ P2+ items → explicitly out of scope per issue
**Architecture note:** The router in `app.js` strips the query string at line 422 (`const route = hash.split('?')[0]`) before computing `basePage` and `routeParam`. Therefore `#/nodes?tab=repeater` gives `routeParam=null` (not `?tab=repeater`). All pages must read URL params from `location.hash` directly, not from `routeParam`. This is the established pattern in `analytics.js` and `nodes.js` (section scroll).
**Placeholder scan:** No TBDs, no "implement later", all code blocks complete. ✅
**Type consistency:**
- `buildNodesQuery(tab, searchStr)` — used consistently in `updateNodesUrl()` and in tests ✅
- `buildPacketsUrl(timeWindowMin, regionParam)` — used consistently in `updatePacketsUrl()` and in tests ✅
- `RegionFilter.setSelected(codesArray)` — defined in Task 1, used in Task 3 ✅
+162
View File
@@ -0,0 +1,162 @@
# v3.4.2 Manual Validation Checklist
**Tester:** _______________
**Staging:** http://20.109.157.39
**Prod:** https://analyzer.00id.net (READ ONLY — do not deploy until staging passes)
**Browser:** Chrome + Firefox + Safari (mobile for responsive items)
**Time estimate:** ~45 minutes
---
## 🔴 HIGH RISK — Test First
### 1. Zero-hop hash size display (#649, #653)
- [ ] Go to Packets page, find a DIRECT advert (route_type=2, 0 hops)
- [ ] Open packet detail — hash size should say "Unknown (zero-hop)" or be hidden, NOT "1 byte"
- [ ] Check "Path Length" field shows `hash_count=0 (direct advert)`
- [ ] Find a FLOOD advert with 0 hops — it SHOULD show hash size (this is different from DIRECT)
### 2. TRACE packet real path (#651, #656)
- [ ] Send a trace from your companion
- [ ] Watch Live map — the animated dot should only travel along completed hops (solid line)
- [ ] Unreached hops should show as dashed/ghosted line at reduced opacity
- [ ] If trace completes fully, entire path should be solid
- [ ] Ghost line should auto-clean after ~10 seconds
### 3. "Paths through this node" accuracy (#655, #658)
- [ ] Go to: http://20.109.157.39/#/nodes/c0dedad4208acb6cbe44b848943fc6d3c5d43cf38a21e48b43826a70862980e4
- [ ] Check "Packets through this node" — packets should actually have this node in their path
- [ ] Compare with a node that shares a 2-char prefix (e.g. C0ffee SF) — they should show DIFFERENT packets
- [ ] Spot-check 3-4 packets: click through, verify path contains the node
### 4. Hash Stats "By Repeaters" (#652, #654)
- [ ] Go to Analytics → Hash Stats
- [ ] "By Repeaters" section should only show repeater-role nodes
- [ ] Compare count in "Multi-Byte Hash Adopters" vs "By Repeaters" — adopters may include companions, repeaters section should not
- [ ] Check that companions/rooms/sensors are excluded from the repeater distribution
### 5. Noise floor column chart (#600, #659)
- [ ] Go to Analytics → RF Health
- [ ] Noise floor chart should show vertical color-coded bars, NOT a line
- [ ] Green bars (< -100 dBm), yellow (-100 to -85), red (≥ -85)
- [ ] Hover over a bar — tooltip should show exact dBm + timestamp
- [ ] Check with only 1 observer selected — chart should still render (division by zero edge case)
- [ ] Reboot markers (if any) should show as vertical dashed lines
### 6. Async backfill on startup
- [ ] SSH to staging: `ssh -i ~/.ssh/id_ed25519 runner@20.109.157.39`
- [ ] `docker restart corescope-staging-go`
- [ ] Within 30 seconds, hit `curl http://localhost:82/api/stats` — should return data (not hang)
- [ ] Check `backfilling` and `backfillProgress` fields in stats response
- [ ] Server should be serving HTTP while backfill runs in background
---
## 🟡 MEDIUM RISK — Features
### 7. Distance unit preference (#621, #646)
- [ ] Go to Customizer → Display tab
- [ ] Change distance unit to "mi" — all distances should show in miles
- [ ] Change to "km" — all distances should show in km
- [ ] Change to "auto" — should use locale (US = miles, EU = km)
- [ ] Check Analytics page distances update after customizer change (no page reload needed)
- [ ] Check Node detail → Neighbors table distances
- [ ] Very small distances (<0.1 mi) should show in feet, not "0.0 mi"
### 8. Panel corner toggle (#608, #657)
- [ ] Go to Live map page
- [ ] Each panel (feed, legend, node detail) should have a small corner-toggle button
- [ ] Click the button — panel should snap to next corner (TL → TR → BR → BL)
- [ ] Refresh page — panel positions should persist (localStorage)
- [ ] Move two panels to same corner — collision avoidance should skip to next free corner
- [ ] On mobile viewport (<768px?) — toggle buttons should be hidden
### 9. Deep linking (#536, #618)
- [ ] Navigate to Nodes page, click a node → URL should update with pubkey hash
- [ ] Copy URL, open in new tab → should land on same node
- [ ] Apply packet filters → URL hash should include filter params
- [ ] Channels page: select a node → URL should reflect selection
- [ ] Analytics tabs: switch tabs → URL should include tab name
- [ ] Share a deep link with someone — they should see the same view
### 10. Sortable tables (#620, #638, #639)
- [ ] Nodes list: click column headers — should sort ascending/descending
- [ ] Sort indicator (arrow) should be visible on active column
- [ ] Node detail → Neighbors table: sortable
- [ ] Node detail → Observers table: sortable
- [ ] Packets table: sortable by column headers
### 11. Channel color highlighting (#271, #607, #611)
- [ ] Go to Channels page
- [ ] Assign a color to a channel using the color picker
- [ ] Feed rows should highlight with that color
- [ ] Change color — should update immediately
- [ ] Refresh — color assignment should persist
### 12. Collapsible panels (#606)
- [ ] Live map: panels should have collapse/expand toggle
- [ ] Collapsed panel should show just the header
- [ ] State should persist across page navigations
### 13. Mobile accessibility (#630, #633)
- [ ] Open staging on phone (or Chrome DevTools mobile emulation)
- [ ] Touch targets should be at least 44×44px
- [ ] Channel color picker should work on mobile
- [ ] No horizontal scroll on any page
- [ ] ARIA labels present on interactive elements (inspect with accessibility tools)
### 14. Map byte-size filter (#565, #568)
- [ ] Go to Map page
- [ ] Find the byte-size filter control
- [ ] Filter by packet size — map should update to show only matching packets
- [ ] Clear filter — all packets should return
### 15. API key security (#532, #628)
- [ ] Try accessing a write endpoint without API key — should be blocked
- [ ] Try with a weak key (e.g., "test", "admin") — should be rejected at startup
- [ ] Check staging logs for API key warning: `docker logs corescope-staging-go 2>&1 | grep -i "apiKey\|api_key\|security"`
### 16. OpenAPI/Swagger (#530, #632)
- [ ] Hit http://20.109.157.39/api/spec — should return valid OpenAPI 3.0 spec
- [ ] Hit http://20.109.157.39/api/docs — should show Swagger UI
- [ ] Try an endpoint from Swagger UI — should work
---
## 🟢 LOW RISK — Verify Quickly
### 17. View Route on Map button
- [ ] Go to any packet detail page
- [ ] Click "View Route on Map" — should navigate to map with route highlighted
### 18. og-image compression
- [ ] Check page source or network tab — og-image.png should be < 300KB (was 1.1MB)
### 19. Prefix Tool
- [ ] Analytics → Prefix Tool tab should load
- [ ] Should show collision data
### 20. License
- [ ] Check repo footer/LICENSE — should be GPL v3
### 21. Docker DISABLE_CADDY
- [ ] (If testable) Set DISABLE_CADDY=true — Caddy should not start
### 22. Region filter on RF Health
- [ ] RF Health tab: change region filter — charts should update
---
## 🏁 Sign-off
| Section | Status | Notes |
|---------|--------|-------|
| High risk (1-6) | ☐ | |
| Medium risk (7-16) | ☐ | |
| Low risk (17-22) | ☐ | |
| **Overall** | ☐ | |
**Tested by:** _______________
**Date:** _______________
**Staging version:** `curl -s http://20.109.157.39/api/stats | jq .version`
**Ready for release:** ☐ Yes / ☐ No — blockers: _______________
+309
View File
@@ -0,0 +1,309 @@
# v3.4.2 Release Test Plan
**Scope:** 90 commits since v3.4.1 (84 files, +14,931 / -1,005)
**Categories:** 19 perf, 19 feat, 18 fix, 15 docs, 3 chore, 1 test, 1 refactor, 1 ci
**Date:** 2026-04-08
---
## A. Automated Tests — Verify All Pass
### Go Backend
```bash
cd cmd/server && go test -race -count=1 ./...
cd cmd/ingestor && go test -race -count=1 ./...
```
**Test files (27 total):**
| File | Tests For |
|------|-----------|
| `cmd/server/decoder_test.go` | Hash size zero-hop, TRACE hopsCompleted, transport direct |
| `cmd/server/backfill_async_test.go` | **NEW** — Async chunked backfill |
| `cmd/server/eviction_test.go` | Memory eviction with runtime heap stats |
| `cmd/server/apikey_security_test.go` | **NEW** — Weak/default API key rejection |
| `cmd/server/openapi_test.go` | **NEW** — OpenAPI spec generation |
| `cmd/server/routes_test.go` | Batch observations endpoint, subpaths-bulk, expand=observations |
| `cmd/server/cache_invalidation_test.go` | cacheTTL config wiring |
| `cmd/server/config_knobs_test.go` | cacheTTLSec helper |
| `cmd/server/helpers_test.go` | constantTimeEqual, IsWeakAPIKey |
| `cmd/server/obs_dedup_test.go` | UniqueObserverCount tracking |
| `cmd/server/neighbor_*.go` (4 files) | Neighbor graph, affinity, persistence |
| `cmd/server/perfstats_race_test.go` | Perf stats concurrency |
| `cmd/server/resolve_context_test.go` | Resolved path filtering |
| `cmd/server/advert_pubkey_test.go` | Advert pubkey tracking |
| `cmd/server/db_test.go` | SQLite operations |
| `cmd/server/config_test.go` | Config loading |
| `cmd/server/coverage_test.go` | Coverage helpers |
| `cmd/server/parity_test.go` | Go/JS decoder parity |
| `cmd/server/websocket_test.go` | WebSocket broadcast |
| `cmd/ingestor/decoder_test.go` | Ingestor decoder (hash size zero-hop) |
| `cmd/ingestor/db_test.go` | Ingestor DB writes |
| `cmd/ingestor/config_test.go` | Ingestor config |
| `cmd/ingestor/main_test.go` | Ingestor entry |
| `cmd/ingestor/coverage_boost_test.go` | Coverage helpers |
### Frontend Unit Tests
```bash
node test-packet-filter.js
node test-aging.js
node test-frontend-helpers.js
node test-table-sort.js # NEW — shared table sort utility
node test-channel-colors.js # NEW — channel color model
node test-panel-corner.js # NEW — panel corner toggle
node test-packets.js # NEW — packets page logic
node test-hop-resolver-affinity.js
node test-customizer-v2.js
node test-live.js
node test-live-dedup.js
```
### E2E / Playwright
```bash
BASE_URL=http://localhost:13581 node test-e2e-playwright.js
```
**Expected:** All existing tests pass + new tests added for sortable tables, deep linking, collapsible panels.
---
## B. Manual Browser Verification
### B1. HIGH RISK — Data Correctness
| # | Feature | Page | What to Check |
|---|---------|------|---------------|
| 1 | Hash size zero-hop | Packets detail | Find a direct (route_type=0) packet → hash_size should show 0, not a bogus computed value |
| 2 | TRACE hopsCompleted | Packets detail / Live map | Find a TRACE packet → verify `hopsCompleted` shows in decoded JSON, live map shows real path length vs intended |
| 3 | Transport direct hash size | Packets detail | Find route_type=RouteTransportDirect packet → hash_size=0 |
| 4 | resolved_path filtering | Node detail → Paths tab | Verify path-hop candidates use resolved_path, no prefix collision false positives |
| 5 | Hash stats repeater filter | Analytics → Hash Issues | "By Repeaters" should only show nodes with repeater role, not companions/sensors |
| 6 | Async chunked backfill | Server startup | Start server with large DB → verify HTTP serves within 2 minutes, `X-CoreScope-Status: backfilling` header present, then transitions to `ready` |
| 7 | Memory eviction (heap stats) | Admin/stats | Verify `/api/stats` shows realistic memory numbers from runtime heap, not the old estimation |
| 8 | Distance/subpath/path-hop indexes | Analytics → Distances, Subpaths | Verify analytics data matches v3.4.1 output (no missing or extra entries) |
| 9 | cacheTTL config wiring | Config | Set `cacheTTL.analyticsHashSizes: 300` in config → verify collision cache respects it |
### B2. MEDIUM RISK — User-Facing Features
| # | Feature | Page | What to Check |
|---|---------|------|---------------|
| 10 | Distance unit preference | Nodes detail, Map | Toggle km/mi/auto in settings → distances update throughout UI |
| 11 | Panel corner toggle | Live page | Click corner toggle → panel moves to opposite corner, persists on reload |
| 12 | Noise floor column chart | Analytics → RF | Verify column chart renders with color-coded thresholds, hover shows values |
| 13 | Deep linking UI states | All pages | Navigate to `#/nodes?tab=neighbors`, `#/packets?observer=X`, `#/channels?node=Y` → correct state loads. Copy URL, open in new tab → same state |
| 14 | Sortable tables | Nodes list, Neighbors, Observers | Click column headers → sort asc/desc, indicator arrow shows, persists correctly |
| 15 | Channel color highlighting | Channels, Live feed | Assign color to channel → feed rows show that color, persists on reload |
| 16 | Mobile accessibility | All pages (phone viewport) | Touch targets ≥44px, ARIA labels present, small viewport doesn't overflow |
| 17 | Collapsible panels | Live map | Collapse/expand panels, medium breakpoint auto-collapses, state persists |
| 18 | Byte-size map filter | Map page | Filter by byte size → markers update correctly |
| 19 | OpenAPI/Swagger | `/api/spec`, `/api/docs` | Spec loads valid JSON, Swagger UI renders and all endpoints are documented |
| 20 | API key rejection | Protected endpoints | Send weak key (e.g. "changeme", "test123") → 403 forbidden |
| 21 | Channel color picker mobile | Channels (phone viewport) | Color picker usable on touch, doesn't overflow |
| 22 | RF Health dashboard | Analytics → RF Health | Observer metrics grid, airtime charts, battery charts, error rate, region filter |
| 23 | Prefix Tool tab | Analytics → Prefix Tool | Renders correctly, collision data consistent with Hash Issues |
| 24 | View Route on Map | Packet detail page | Button works and shows route on map |
### B3. LOWER RISK — Performance (Verify No Regressions)
| # | Feature | Page | What to Check |
|---|---------|------|---------------|
| 25 | Incremental DOM diff | Packets (30K+) | Virtual scroll renders smoothly, no visible flicker |
| 26 | Coalesced WS renders | Live page | Rapid packets don't cause frame drops (rAF coalescing) |
| 27 | Marker reposition on zoom | Map | Zoom/resize → markers move smoothly, no full rebuild flash |
| 28 | Parallel replay fetches | Live → VCR | Replay loads quickly (parallel observation fetches) |
| 29 | Batch observations API | Packets page (sort change) | Changing sort fetches observations in batch (network tab: 1 POST not N GETs) |
| 30 | Client-side network status | Analytics | No separate API call for network status |
| 31 | og-image compression | `/og-image.png` | Verify loads, ~235KB not ~1.1MB |
---
## C. API Regression Tests
Run against a local server with test-fixture DB:
```bash
BASE=http://localhost:13581
# Core endpoints — verify response shape
curl -s "$BASE/api/stats" | jq '.totalPackets, .backfilling, .backfillProgress'
curl -s "$BASE/api/packets?limit=5" | jq '.packets[0] | keys'
curl -s "$BASE/api/packets?limit=5&expand=observations" | jq '.packets[0].observations | length'
curl -s "$BASE/api/nodes?limit=5" | jq '.[0] | keys'
# New endpoints
curl -s -X POST "$BASE/api/packets/observations" \
-H 'Content-Type: application/json' \
-d '{"hashes":["test123"]}' | jq '.results | keys'
curl -s "$BASE/api/analytics/subpaths-bulk?hops=A,B&hops=B,C" | jq 'keys'
curl -s "$BASE/api/observers/metrics/summary" | jq 'type'
curl -s "$BASE/api/spec" | jq '.openapi'
curl -s "$BASE/api/docs" | head -5 # Should return HTML
# Backfill status header
curl -sI "$BASE/api/stats" | grep X-CoreScope-Status
# API key rejection
curl -s -H 'X-API-Key: changeme' "$BASE/api/debug/vars" | jq '.error'
curl -s -H 'X-API-Key: test' "$BASE/api/debug/vars" | jq '.error'
# Existing endpoints — verify not broken
curl -s "$BASE/api/analytics/rf?timeRange=24h" | jq 'keys'
curl -s "$BASE/api/analytics/hash-sizes" | jq 'type'
curl -s "$BASE/api/analytics/distances" | jq 'type'
curl -s "$BASE/api/analytics/subpaths" | jq 'type'
curl -s "$BASE/api/channels" | jq 'type'
curl -s "$BASE/api/config/client" | jq 'keys'
```
### Expected response shape changes from v3.4.1:
- `/api/stats` now includes `backfilling` (bool) and `backfillProgress` (float 0-1)
- `/api/packets` no longer strips observations by default (lazy via `ExpandObservations` flag) — verify `observations` key absent without `expand=observations`
- Decoded packets with route_type=direct now have `hashSize: 0`
- TRACE packets now have `path.hopsCompleted` field
---
## D. Performance Regression Tests
### D1. Server Startup Time
```bash
# Start server with production-size DB (~30K packets)
# Measure time from process start to first successful HTTP response
time curl -s http://localhost:13581/api/stats > /dev/null
# Target: < 2 minutes (async backfill requirement)
```
### D2. Go Benchmarks
```bash
cd cmd/server && go test -bench=. -benchmem -count=3
```
Key benchmarks to compare with v3.4.1 baseline:
- `BenchmarkQueryPackets` — should not regress with new indexes
- `BenchmarkEvictStale` — batch removal from secondary indexes
- `BenchmarkGetStoreStats` — 2 concurrent queries vs 5 sequential
- `BenchmarkIngestNew` — additional index maintenance overhead
### D3. Frontend Performance
- Open Packets page with 30K+ packets → measure initial render time (DevTools Performance tab)
- Scroll rapidly through virtual scroll → should maintain 60fps
- Switch sort column on packets → single batch POST, not N+1 GETs
- Open Analytics page → no redundant API calls in network tab
### D4. Memory Usage
- After loading 30K packets, check `/api/stats` memory figure
- Compare with v3.4.1 baseline (prefix map cap at 8 chars should reduce ~10x)
- Verify eviction triggers at correct memory threshold using runtime heap stats
---
## E. Infrastructure / Deployment Tests
### E1. Docker Build
```bash
docker build -t corescope:test .
docker run --rm -p 13581:13581 corescope:test
# Verify: container starts, HTTP responds, WebSocket connects
```
### E2. GHCR Publish (CI)
- Verify CI publishes to `ghcr.io/kpa-clawbot/corescope`
- Verify tags: `edge` (master), `vX.Y.Z` (release)
### E3. Staging Deploy
```bash
# Verify staging compose works with standard ports
docker compose -f docker-compose.staging.yml up -d
# Check: no 3GB memory limit, standard port binding
```
### E4. DISABLE_CADDY
```bash
docker run --rm -e DISABLE_CADDY=true corescope:test
# Verify: Caddy not started, Go server serves directly
```
### E5. CI Pipeline
- Verify consolidated pipeline: build → publish GHCR → deploy staging
- Verify runs on `meshcore-runner-2`
---
## F. Edge Cases & Integration Tests
### F1. Cross-Feature Interactions
| Scenario | Risk |
|----------|------|
| Deep link to sorted table → sort state matches URL params | Medium |
| Channel color + deep link → color persists in linked URL | Medium |
| Panel corner toggle + collapsible panels → both states persist independently | Low |
| Distance unit pref + neighbor table sort by distance → sort uses correct unit | Medium |
| Noise floor chart + region filter → chart respects filter | Medium |
| Byte-size map filter + channel color highlighting → both active simultaneously | Low |
### F2. Data Correctness Edge Cases
| Scenario | Risk |
|----------|------|
| Zero-hop TRACE packet (should NOT reset hashSize — TRACE exemption) | **High** |
| Packet with all hops having same 2-char prefix → resolved_path filtering prevents false match | **High** |
| Node that switches role (repeater → companion) → hash stats updates | Medium |
| Backfill interrupted mid-chunk (server restart) → resumes or completes on next start | Medium |
| Empty DB startup → no errors, backfill completes instantly | Low |
| DB with 100K+ packets → async backfill doesn't OOM, progress reported | **High** |
### F3. Concurrency / Race Conditions
| Scenario | Risk |
|----------|------|
| Concurrent API requests during backfill → no deadlock (lock ordering documented) | **High** |
| Eviction running while analytics query in progress → no stale pointer panic | **High** |
| Multiple WebSocket clients during high ingest rate → coalesced broadcasts don't drop | Medium |
| `time.NewTicker` cleanup on graceful shutdown (replaced `time.Tick`) | Low |
### F4. API Key Security
| Scenario | Expected |
|----------|----------|
| No API key configured → write endpoints disabled | 403 "write endpoints disabled" |
| Weak key "changeme" → rejected even if configured | 403 "forbidden" |
| Timing-safe comparison → no timing oracle | Constant-time via `crypto/subtle` |
| Empty string key → rejected | 401 "unauthorized" |
### F5. Browser Compatibility
- Test on Chrome, Firefox, Safari (latest)
- Test on iOS Safari, Android Chrome
- Verify touch targets on mobile (44px minimum)
- Verify ARIA labels with screen reader
---
## G. Test Coverage Gaps — Action Items
| Gap | Priority | Action |
|-----|----------|--------|
| No automated test for distance unit preference rendering | Medium | Add Playwright test |
| No automated test for noise floor column chart | Medium | Add Playwright test |
| No automated test for deep link state restoration | **High** | Add Playwright tests for each deep-linkable state |
| No automated test for channel color persistence | Medium | `test-channel-colors.js` covers model; need Playwright for UI |
| No automated test for mobile viewport behavior | Medium | Add Playwright test with mobile viewport |
| No automated test for backfill progress header | Low | Add to `routes_test.go` |
| No automated test for `time.NewTicker` cleanup | Low | Add to graceful shutdown test |
| Observer metrics endpoints not covered in route tests | Medium | Add to `routes_test.go` |
| Subpaths-bulk endpoint needs test | Medium | Add to `routes_test.go` |
| No load test for batch observations endpoint (200 hash limit) | Low | Add boundary test |
---
## H. Release Checklist
- [ ] All Go tests pass with `-race` flag
- [ ] All frontend unit tests pass
- [ ] Playwright E2E tests pass
- [ ] Manual browser verification (Section B) complete
- [ ] API regression tests (Section C) pass
- [ ] Docker build succeeds
- [ ] Staging deploy verified
- [ ] No console errors on any page
- [ ] Performance spot-checks (Section D) — no regressions
- [ ] Coverage badges updated (backend ≥85%, frontend ≥42%)
- [ ] CHANGELOG updated
- [ ] Tag `v3.4.2` created
+88
View File
@@ -0,0 +1,88 @@
# Analytics
The Analytics page provides deep-dive charts and tables about your mesh network. Select a tab to explore different aspects.
[Screenshot: analytics page with tab bar]
## Overview
Summary dashboard with key network metrics at a glance. Quick sparklines and counts across all data dimensions.
## RF / Signal
Radio frequency analysis:
- **SNR distribution** — histogram of signal-to-noise ratios across all packets
- **RSSI distribution** — histogram of received signal strength
- **SNR by observer** — which observers are getting the best signals
- **Signal trends** — how signal quality changes over time
Use this to identify weak links or noisy observers.
## Topology
Network structure analysis:
- **Hop count distribution** — how many relay hops packets typically take
- **Top relay nodes** — which repeaters handle the most traffic
- **Node connectivity** — how well-connected each node is
## Channels
Channel message statistics:
- **Messages per channel** — which channels are most active
- **Channel activity over time** — traffic trends by channel
- **Top senders** — most active nodes per channel
## Hash Stats
Mesh hash size analysis:
- **Hash size distribution** — how many bytes nodes use for addressing
- **Hash sizes by role** — do repeaters use different hash sizes than companions?
## Hash Issues
Potential hash collision detection:
- **Collision pairs** — nodes whose short hash prefixes overlap
- **Risk assessment** — how likely collisions are at current hash sizes
Hash collisions can cause packet misrouting. If you see collisions here, consider increasing hash sizes on affected nodes.
## Route Patterns (Subpaths)
Common routing paths through the mesh:
- **Frequent subpaths** — which relay chains appear most often
- **Path reliability** — how consistently each path is used
- **Path detail** — click a subpath to see every packet that used it
## Nodes
Per-node analytics with sortable metrics across the fleet.
## Distance
Estimated distances between nodes based on GPS coordinates, correlated with signal quality.
## Neighbor Graph
Interactive visualization of which nodes can directly hear each other. Shows the mesh topology as a network graph.
## RF Health
Per-observer signal health over time. Identifies observers with degrading reception.
## Prefix Tool
Test hash prefix lengths to see how many collisions different sizes would produce. Useful for deciding on hash_size settings.
## Region filter
All analytics tabs respect the **region filter** at the top. Select a region to scope the data to observers in that area.
## Deep linking
Each tab is deep-linkable. Share a URL like `#/analytics?tab=collisions` to point someone directly at hash issues.

Some files were not shown because too many files have changed in this diff Show More