Compare commits

..

58 Commits

Author SHA1 Message Date
Kpa-clawbot 4e4de3fe0d Merge branch 'master' into feat/clock-skew-default-detection 2026-04-30 23:48:16 -07:00
efiten e460932668 fix(store): apply retentionHours cutoff in Load() to prevent OOM on cold start (#917)
## Problem

`Load()` loaded all transmissions from the DB regardless of
`retentionHours`, so `buildSubpathIndex()` processed the full DB history
on every startup. On a DB with ~280K paths this produces ~13.5M subpath
index entries, OOM-killing the process before it ever starts listening —
causing a supervisord crash loop with no useful error message.

## Fix

Apply the same `retentionHours` cutoff to `Load()`'s SQL that
`EvictStale()` already uses at runtime. Both conditions
(`retentionHours` window and `maxPackets` cap) are combined with AND so
neither safety limit is bypassed.

Startup now builds indexes only over the retention window, making
startup time and memory proportional to recent activity rather than
total DB history.

## Docs

- `config.example.json`: adds `retentionHours` to the `packetStore`
block with recommended value `168` (7 days) and a warning about `0` on
large DBs
- `docs/user-guide/configuration.md`: documents the field and adds an
explicit OOM warning

## Test plan

- [x] `cd cmd/server && go test ./... -run TestRetentionLoad` — covers
the retention-filtered load: verifies packets outside the window are
excluded, and that `retentionHours: 0` still loads everything
- [x] Deploy on an instance with a large DB (>100K paths) and
`retentionHours: 168` — server reaches "listening" in seconds instead of
OOM-crashing
- [x] Verify `config.example.json` has `retentionHours: 168` in the
`packetStore` block
- [x] Verify `docs/user-guide/configuration.md` documents the field and
warning

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
2026-05-01 06:47:55 +00:00
Kpa-clawbot aeae7813bc fix: enable SQLite incremental auto-vacuum so DB shrinks after retention (#919) (#920)
Closes #919

## Summary

Enables SQLite incremental auto-vacuum so the database file actually
shrinks after retention reaper deletes old data. Previously, `DELETE`
operations freed pages internally but never returned disk space to the
OS.

## Changes

### 1. Auto-vacuum on new databases
- `PRAGMA auto_vacuum = INCREMENTAL` set via DSN pragma before
`journal_mode(WAL)` in the ingestor's `OpenStoreWithInterval`
- Must be set before any tables are created; DSN ordering ensures this

### 2. Post-reaper incremental vacuum
- `PRAGMA incremental_vacuum(N)` runs after every retention reaper cycle
(packets, metrics, observers, neighbor edges)
- N defaults to 1024 pages, configurable via `db.incrementalVacuumPages`
- Noop on `auto_vacuum=NONE` databases (safe before migration)
- Added to both server and ingestor

### 3. Opt-in full VACUUM for existing databases
- Startup check logs a clear warning if `auto_vacuum != INCREMENTAL`
- `db.vacuumOnStartup: true` config triggers one-time `PRAGMA
auto_vacuum = INCREMENTAL; VACUUM`
- Logs start/end time for operator visibility

### 4. Documentation
- `docs/user-guide/configuration.md`: retention section notes that
lowering retention doesn't immediately shrink the DB
- `docs/user-guide/database.md`: new guide covering WAL, auto-vacuum,
migration, manual VACUUM

### 5. Tests
- `TestNewDBHasIncrementalAutoVacuum` — fresh DB gets `auto_vacuum=2`
- `TestExistingDBHasAutoVacuumNone` — old DB stays at `auto_vacuum=0`
- `TestVacuumOnStartupMigratesDB` — full VACUUM sets `auto_vacuum=2`
- `TestIncrementalVacuumReducesFreelist` — DELETE + vacuum shrinks
freelist
- `TestCheckAutoVacuumLogs` — handles both modes without panic
- `TestConfigIncrementalVacuumPages` — config defaults and overrides

## Migration path for existing databases

1. On startup, CoreScope logs: `[db] auto_vacuum=NONE — DB needs
one-time VACUUM...`
2. Set `db.vacuumOnStartup: true` in config.json
3. Restart — VACUUM runs (blocks startup, minutes on large DBs)
4. Remove `vacuumOnStartup` after migration

## Test results

```
ok  github.com/corescope/server    19.448s
ok  github.com/corescope/ingestor  30.682s
```

---------

Co-authored-by: you <you@example.com>
2026-04-30 23:45:00 -07:00
Kpa-clawbot 43f17ed770 ci: update go-server-coverage.json [skip ci] 2026-05-01 06:40:20 +00:00
Kpa-clawbot 9ada3d7e93 ci: update go-ingestor-coverage.json [skip ci] 2026-05-01 06:40:19 +00:00
Kpa-clawbot 7a04462dde ci: update frontend-tests.json [skip ci] 2026-05-01 06:40:18 +00:00
Kpa-clawbot c0f39e298a ci: update frontend-coverage.json [skip ci] 2026-05-01 06:40:17 +00:00
Kpa-clawbot cc2b731c77 ci: update e2e-tests.json [skip ci] 2026-05-01 06:40:16 +00:00
efiten f2689123f3 fix(geobuilder): wrap longitude to [-180,180] to fix southern hemisphere polygons (#925)
## Summary

- Fixes #912 — geofilter-builder generates out-of-range longitudes for
southern hemisphere locations
- Root cause: Leaflet's `latlng.lng` is unbounded; panning from Europe
to Australia produces values like `-210` instead of `150`
- Fix: call `latlng.wrap()` in `latLonPair()` to normalise longitude to
`[-180, 180]` before writing the config JSON

## Details

When the user opens the builder (default view: Europe, `[50.5, 4.4]`)
and pans east to Australia, Leaflet tracks the cumulative pan offset and
returns `lng = 150 - 360 = -210` to keep the path continuous. The
builder was passing that raw value straight into the output JSON,
producing coordinates that fall outside any valid bounding box.

`L.LatLng.wrap()` is Leaflet's built-in normalisation method — collapses
any longitude to `[-180, 180]` with no loss of precision.

## Test plan

- [x] Open the builder, navigate to NSW Australia, place a polygon —
confirm longitudes are `~141`–`154`, not `~-219`–`-206`
- [x] Repeat for a northern hemisphere location (e.g. Belgium) — confirm
output is unchanged
- [x] Paste the generated config into CoreScope — confirm nodes appear
on Maps and Live view

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kpa-clawbot <kpaclawbot@outlook.com>
2026-05-01 06:40:12 +00:00
Kpa-clawbot 308b67ed66 chore: remove tautological/vacuous frontend tests (#938)
## Summary

Remove tautological/vacuous frontend tests identified in the test audit
(2026-04-30).

## Deleted Files

- **`test-anim-perf.js`** — Tests local simulation functions, not actual
`live.js` animation code. Behavioral equivalents exist in
`test-live-anims.js`.
- **`test-channel-add-ux.js`** — Pure HTML/CSS substring grep tests
(`src.includes(...)`) with zero behavioral value. Any refactor breaks
them without indicating a real bug.

## Edited Files

- **`test-aging.js`** — Removed `mockGetStatusInfo` (5 tests) and
`getStatusTooltip` (6 tests) blocks. These re-implement SUT logic inside
the test file and assert on the local copy, not the real code. The
GOLD-rated `getNodeStatus` boundary tests (18 assertions) and the BUG
CHECK section are preserved.

## Why

These tests are unfailable by construction — they pass regardless of
whether the code under test is correct. They inflate test counts without
providing regression protection, and they break on harmless refactors
(false negatives).

## Validation

- `npm run test:unit` passes (all 3 files: 62 + 18 + 601 assertions).
- 9 pre-existing failures in `test-frontend-helpers.js` (hop-resolver
affinity tests, unrelated to this change).
- No harness references (`test-all.sh`, `package.json`) needed updating
— deleted files were not listed there.

Co-authored-by: you <you@example.com>
2026-04-30 23:28:30 -07:00
Kpa-clawbot 54f7f9d35b feat: path-prefix candidate inspector with map view (#944) (#945)
## feat: path-prefix candidate inspector with map view (#944)

Implements the locked spec from #944: a beam-search-based path prefix
inspector that enumerates candidate full-pubkey paths from short hex
prefixes and scores them.

### Server (`cmd/server/path_inspect.go`)

- **`POST /api/paths/inspect`** — accepts 1-64 hex prefixes (1-3 bytes,
uniform length per request)
- Beam search (width 20) over cached `prefixMap` + `NeighborGraph`
- Per-hop scoring: edge weight (35%), GPS plausibility (20%), recency
(15%), prefix selectivity (30%)
- Geometric mean aggregation with 0.05 floor per hop
- Speculative threshold: score < 0.7
- Score cache: 30s TTL, keyed by (prefixes, observer, window)
- Cold-start: synchronous NeighborGraph rebuild with 2s hard timeout →
503 `{retry:true}`
- Body limit: 4096 bytes via `http.MaxBytesReader`
- Zero SQL queries in handler hot path
- Request validation: rejects empty, odd-length, >3 bytes, mixed
lengths, >64 hops

### Frontend (`public/path-inspector.js`)

- New page under Tools route with input field (comma/space separated hex
prefixes)
- Client-side validation with error feedback
- Results table: rank, score (color-coded speculative), path names,
per-hop evidence (collapsed)
- "Show on Map" button calls `drawPacketRoute` (one path at a time,
clears prior)
- Deep link: `#/tools/path-inspector?prefixes=2c,a1,f4`

### Nav reorganization

- `Traces` nav item renamed to `Tools`
- Backward-compat: `#/traces/<hash>` redirects to `#/tools/trace/<hash>`
- Tools sub-routing dispatches to traces or path-inspector

### Store changes

- Added `LastSeen time.Time` to `nodeInfo` struct, populated from
`nodes.last_seen`
- Added `inspectMu` + `inspectCache` fields to `PacketStore`

### Tests

- **Go unit tests** (`path_inspect_test.go`): scoreHop components, beam
width cap, speculative flag, all validation error cases, valid request
integration
- **Frontend tests** (`test-path-inspector.js`): parse
comma/space/mixed, validation (empty, odd, >3 bytes, mixed lengths,
invalid hex, valid)
- Anti-tautology gate verified: removing beam pruning fails width test;
removing validation fails reject tests

### CSS

- `--path-inspector-speculative` variable in both themes (amber, WCAG AA
on both dark/light backgrounds)
- All colors via CSS variables (no hardcoded hex in production code)

Closes #944

---------

Co-authored-by: you <you@example.com>
2026-04-30 23:28:16 -07:00
Kpa-clawbot dbd2726b27 ci: update go-server-coverage.json [skip ci] 2026-05-01 03:56:41 +00:00
Kpa-clawbot d9757626bc ci: update go-ingestor-coverage.json [skip ci] 2026-05-01 03:56:40 +00:00
Kpa-clawbot 472c9f2aa2 ci: update frontend-tests.json [skip ci] 2026-05-01 03:56:39 +00:00
Kpa-clawbot dccfb0a328 ci: update frontend-coverage.json [skip ci] 2026-05-01 03:56:38 +00:00
Kpa-clawbot 086b8b7983 ci: update e2e-tests.json [skip ci] 2026-05-01 03:56:36 +00:00
efiten 9293ff408d fix(customize): skip panel re-render while a text field has focus (#927)
## Summary

- `_debouncedWrite()` was calling `_refreshPanel()` 300ms after every
keystroke
- `_refreshPanel()` sets `container.innerHTML`, destroying the focused
input element
- On mobile, losing the focused input collapses the virtual keyboard
after each keypress

Guard the `_refreshPanel()` call so it is skipped when
`document.activeElement` is inside the panel. The pipeline
(`_runPipeline`) still runs immediately — CSS updates apply. Override
dots update on the next natural re-render (tab switch, dark-mode toggle,
panel reopen).

## Root cause

`customize-v2.js` → `_debouncedWrite()` → `_refreshPanel()` →
`_renderPanel()` → `container.innerHTML = ...`

## Test plan

- [ ] New Playwright E2E test: open Customize, focus a text field, type,
wait 500ms past debounce — asserts input element is still connected to
DOM and focus remains inside panel
- [ ] Manual: open Customize on mobile (or DevTools mobile emulation),
type in Site Name — keyboard must not collapse after each character

Fixes #896
2026-04-30 20:46:59 -07:00
Kpa-clawbot 8c3b2e2248 test(e2e): retry click on table rows when handles detach (#943)
## Problem

E2E test `Node detail loads` intermittently fails with:

> elementHandle.click: Element is not attached to the DOM

(e.g. PR #938 CI run job 73889426640.) Same flake class as #ngStats
hydration race fixed in #940.

## Root cause

```js
const firstRow = await page.$('table tbody tr');
await firstRow.click();
```

Between the `$()` and `.click()`, the nodes table re-renders from a
WebSocket push. The captured handle is detached from the new DOM, click
throws.

## Fix

Switch to a selector-based click with a small retry loop (3 attempts ×
200ms backoff), so a detach mid-attempt re-resolves a fresh element.

Test logic unchanged; just defensive against re-render between query and
click.

Co-authored-by: Kpa-clawbot <bot@example.invalid>
2026-04-30 20:46:03 -07:00
Kpa-clawbot a4b99a98e1 ci: update go-server-coverage.json [skip ci] 2026-05-01 03:11:11 +00:00
Kpa-clawbot e05e3cb2f2 ci: update go-ingestor-coverage.json [skip ci] 2026-05-01 03:11:10 +00:00
Kpa-clawbot 61719c2218 ci: update frontend-tests.json [skip ci] 2026-05-01 03:11:09 +00:00
Kpa-clawbot e73f8996a8 ci: update frontend-coverage.json [skip ci] 2026-05-01 03:11:08 +00:00
Kpa-clawbot 292075fd0d ci: update e2e-tests.json [skip ci] 2026-05-01 03:11:07 +00:00
Kpa-clawbot 6273a8797b test(e2e): wait for #ngStats hydration before counting cards (#940)
## Problem

E2E test "Analytics Neighbor Graph tab renders canvas and stats"
intermittently fails with `Neighbor Graph stats should have >=3 cards,
got 0` (e.g. run 25185836669).

The same suite passes on neighboring runs (master + PR #939) within
minutes. The failure correlates with timing/load, not code change.

## Root cause

`#ngStats` cards render asynchronously after `#ngCanvas` mounts. The
test waits for the canvas, then immediately reads `#ngStats .stat-card`
count. On slower runs the read happens before stats hydrate → 0 cards →
assert fail.

Other Analytics tabs in the same file already use
`page.waitForFunction(...)` to poll for content (e.g. Distance tab on
line 654). Neighbor Graph block was missing the equivalent wait.

## Fix

Add the same defensive wait before counting:

```js
await page.waitForFunction(
  () => document.querySelectorAll('#ngStats .stat-card').length >= 3,
  { timeout: 8000 },
);
```

Test-only change. No frontend code touched. Bounded by 8s timeout
matching other Analytics waits.

Co-authored-by: Kpa-clawbot <bot@example.invalid>
2026-04-30 20:01:35 -07:00
Kpa-clawbot f84142b1d2 fix(packets): hash filter must bypass saved region filter (#939)
## Summary

Direct packet links like `/#/packets?hash=<HASH>` silently returned zero
rows when the user's saved region filter excluded the packet's observer
region. The packet existed and rendered in the side panel (which fetches
without region filter), but the main packet table was empty — leaving
the user with no rows to click and no obvious diagnostic.

## Root cause

`loadPackets()` in `public/packets.js` always added the `region` query
param to `/api/packets`, even when `filters.hash` was set. The
time-window filter is already correctly suppressed when `filters.hash`
is present (see line 619: `if (windowMin > 0 && !filters.hash)`); the
region filter should follow the same rule. A specific hash is an exact
identifier — the user wants THAT packet regardless of where their saved
region selection points.

## Change

Extracted the param-building logic into a pure helper
`buildPacketsParams(...)` so it's testable, then suppressed the `region`
param when `filters.hash` is set.

## Tests

Added 7 unit tests in `test-packets.js` covering:

- hash filter suppresses region (the bug)
- hash filter suppresses region with default windowMin=0
- region applies normally when no hash filter
- empty regionParam doesn't produce spurious `region=` param
- node/observer/channel filters still pass through alongside a hash
- groupByHash=true / false flag handling

Anti-tautology gate verified: reverting the one-line fix (`!filters.hash
&&` → removed) causes 3 of the 7 new tests to fail. The fix is the
smallest change that makes them pass.

`node test-packets.js`: 80 passed, 0 failed.

## Reproduction

1. Set region filter to e.g. `SJC`
2. Open `/#/packets?hash=<HASH_FROM_ANOTHER_REGION>`
3. Before fix: empty table, no diagnostic
4. After fix: packet renders

---------

Co-authored-by: Kpa-clawbot <bot@example.invalid>
2026-04-30 19:51:53 -07:00
Kpa-clawbot 17df9bf06e ci: update go-server-coverage.json [skip ci] 2026-05-01 02:51:21 +00:00
Kpa-clawbot d0a955b72c ci: update go-ingestor-coverage.json [skip ci] 2026-05-01 02:51:20 +00:00
Kpa-clawbot cbf5b8bbd0 ci: update frontend-tests.json [skip ci] 2026-05-01 02:51:19 +00:00
Kpa-clawbot 5a30406392 ci: update frontend-coverage.json [skip ci] 2026-05-01 02:51:18 +00:00
Kpa-clawbot f3ee60ed62 ci: update e2e-tests.json [skip ci] 2026-05-01 02:51:16 +00:00
Kpa-clawbot d81852736d ci: re-enable staging deploy now that VM is back (#932)
Reverts the `if: false` guard from #908.

## Why
- Azure subscription was blocked, staging VM `meshcore-runner-2`
deallocated.
- Subscription unblocked, VM started, runner online, smoke CI [run
#25117292530](https://github.com/Kpa-clawbot/CoreScope/actions/runs/25117292530)
passed.
- Time to resume automatic staging deploys on master pushes.

## Changes
- `deploy` job: `if: false` → `if: github.event_name == 'push'`
(original condition from before #908).
- `publish` job: `needs: [build-and-publish]` → `needs: [deploy]`
(original wiring restored).

## Verify after merge
- Next master push triggers the full chain: go-test → e2e-test →
build-and-publish → deploy → publish.
- `docker ps` on staging VM shows `corescope-staging-go` updated to the
new commit.

Co-authored-by: you <you@example.com>
2026-04-30 19:40:51 -07:00
Kpa-clawbot 5678874128 fix: exclude non-repeater nodes from path-hop resolution (#935) (#936)
Fixes #935

## Problem

`buildPrefixMap()` indexed ALL nodes regardless of role, causing
companions/sensors to appear as repeater hops when their pubkey prefix
collided with a path-hop hash byte.

## Fix

### Server (`cmd/server/store.go`)
- Added `canAppearInPath(role string) bool` — allowlist of roles that
can forward packets (repeater, room_server, room)
- `buildPrefixMap` now skips nodes that fail this check

### Client (`public/hop-resolver.js`)
- Added matching `canAppearInPath(role)` helper
- `init()` now only populates `prefixIdx` for path-eligible nodes
- `pubkeyIdx` remains complete — `resolveFromServer()` still resolves
any node type by full pubkey (for server-confirmed `resolved_path`
arrays)

## Tests

- `cmd/server/prefix_map_role_test.go`: 7 new tests covering role
filtering in prefix map and resolveWithContext
- `test-hop-resolver-affinity.js`: 4 new tests verifying client-side
role filter + pubkeyIdx completeness
- All existing tests updated to include `Role: "repeater"` where needed
- `go test ./cmd/server/...` — PASS
- `node test-hop-resolver-affinity.js` — 16/17 pass (1 pre-existing
centroid failure unrelated to this change)

## Commits

1. `fix: filter prefix map to only repeater/room roles (#935)` — server
implementation
2. `test: prefix map role filter coverage (#935)` — server tests
3. `ui: filter HopResolver prefix index to repeater/room roles (#935)` —
client implementation
4. `test: hop-resolver role filter coverage (#935)` — client tests

---------

Co-authored-by: you <you@example.com>
2026-04-30 09:25:51 -07:00
Kpa-clawbot e857e0b1ce ci: update go-server-coverage.json [skip ci] 2026-04-25 00:34:06 +00:00
Kpa-clawbot 9da7c71cc5 ci: update go-ingestor-coverage.json [skip ci] 2026-04-25 00:34:05 +00:00
Kpa-clawbot 03484ea38d ci: update frontend-tests.json [skip ci] 2026-04-25 00:34:04 +00:00
Kpa-clawbot 27af4098e6 ci: update frontend-coverage.json [skip ci] 2026-04-25 00:34:03 +00:00
Kpa-clawbot 474023b9b7 ci: update e2e-tests.json [skip ci] 2026-04-25 00:34:02 +00:00
Kpa-clawbot f4484adb52 ci: move to GitHub-hosted runners, disable staging deploy (#908)
## Why

The Azure staging VM (`meshcore-vm`) is offline. Self-hosted runners are
unavailable, blocking all CI.

## What changed (per job)

| Job | Change | Revert |
|-----|--------|--------|
| `e2e-test` | `runs-on: [self-hosted, Linux]` → `ubuntu-latest`;
removed self-hosted-specific "Free disk space" step | Change `runs-on`
back to `[self-hosted, Linux]`, restore disk cleanup step |
| `build-and-publish` | `runs-on: [self-hosted, meshcore-runner-2]` →
`ubuntu-latest`; removed "Free disk space" prune step (noop on fresh
GH-hosted runners) | Change `runs-on` back, restore prune step |
| `deploy` | `if: false # disabled` (was `github.event_name == 'push'`);
`runs-on` kept as-is | Change `if:` back to `github.event_name ==
'push'` |
| `publish` | `runs-on: [self-hosted, Linux]` → `ubuntu-latest`; `needs:
[deploy]` → `needs: [build-and-publish]` | Change both back |

## Notes

- `go-test` and `release-artifacts` were already on `ubuntu-latest` —
untouched.
- The `deploy` job is disabled via `if: false` for trivial one-line
revert when the VM returns.
- No new `setup-*` actions were needed — `setup-node`, `setup-go`,
`docker/setup-buildx-action`, and `docker/login-action` were already
present.

Co-authored-by: you <you@example.com>
2026-04-24 17:25:53 -07:00
you 86ca793b60 fix: bump default-epoch uptime cap to 3 years for solar repeater lifetimes 2026-04-25 00:08:09 +00:00
you 4291b387f5 fix: classifySkew defensive absolute value 2026-04-24 23:49:21 +00:00
you 3cd7186563 test: integration tests for epoch-0 and missing-timestamp adverts
TestGetNodeClockSkew_EpochZeroAdvert: verifies advert with timestamp==0
flows through PacketStore and classifies as severity=default, epoch=0.

TestGetNodeClockSkew_MissingTimestamp: verifies advert with no timestamp
field is skipped (extractTimestamp returns -1, filtered by collectSamples).

Review item #4 on PR #907.
2026-04-24 23:42:56 +00:00
you 86a4403136 fix: computeNodeSkew picks chronologically-latest observation
Uses max observedTS instead of last-appended slice element to
determine the most recent skew sample per hash. Consolidates
the latestObsTS and anyCal loop into a single pass.

Review item #3 on PR #907.
2026-04-24 23:42:56 +00:00
you c46a60f78a fix: rename CSS fleet row classes to match severity names
.clock-fleet-row--warning  → .clock-fleet-row--degrading
.clock-fleet-row--critical → .clock-fleet-row--degraded

The JS in analytics.js builds classes as 'clock-fleet-row--<severity>'
so the CSS must match the actual severity strings.

Review item #2 on PR #907.
2026-04-24 23:42:56 +00:00
you d4b1aa40d0 fix: extractTimestamp returns -1 sentinel for missing timestamp
Distinguishes 'no timestamp field' (returns -1) from real epoch-0
(returns 0). Adds jsonNumberOk helper that returns (value, bool).
The collectSamples guard 'advertTS < 0' correctly filters missing
timestamps while allowing epoch-0 through to isDefaultEpoch.

Updates TestExtractTimestamp to verify both cases.

Review item #1 on PR #907.
2026-04-24 23:42:56 +00:00
you d617a55155 polish: remove leftover recentMedianSkewSec comment, add overlapping-epoch test
- Remove stale 'recentMedianSkewSec' reference in nodes.js comment
- Add TestIsDefault_OverlappingWindowsPicksLargest covering epoch
  selection when default ranges overlap
2026-04-24 23:33:49 +00:00
you 2106cc0b8b ui: per-tier explainer line in node clock card 2026-04-24 23:29:26 +00:00
you 0acbac6fde ui: rename clock skew tiers to default/ok/degrading/degraded/wrong 2026-04-24 23:27:16 +00:00
you 2c675f5ab2 test: cover default-detection classifier tiers and edge cases 2026-04-24 23:25:35 +00:00
you 545df2788d feat: replace clock skew classifier with default-detection model 2026-04-24 23:20:12 +00:00
you f872fd90bf docs: clock-skew classifier redesign spec 2026-04-24 23:18:15 +00:00
Kpa-clawbot a47fe26085 fix(channels): allow removing user-added keys for server-known channels (#898)
## Problem
Adding a channel key in the Channels UI for a channel the server already
knows about (e.g. `#public` from rainbow / config) leaves the
localStorage entry **unremovable**:

- `mergeUserChannels` sees the name already exists in the channel list
and skips the user entry.
- The existing channel row is never marked `userAdded:true`.
- The ✕ button (`[data-remove-channel]`) is only rendered for
`userAdded` rows.
- Result: stuck localStorage key, no UI to delete it.

There was also a latent bug in the remove handler — for non-`user:`
rows, it used the raw hash (e.g. `enc_11`) as the
`ChannelDecrypt.removeKey()` argument, but the storage key is the
channel **name**.

## Fix
1. **`mergeUserChannels`**: when a stored key matches an existing
channel by name/hash, mark the existing channel `userAdded=true` so the
✕ renders on it. (No magical/auto deletion of stored keys — the user
explicitly chooses to remove.)
2. **Remove handler**:
- Look up the channel object to get the correct display name for the
localStorage key.
- Keep server-known channels in the list when their ✕ is clicked (only
the user's localStorage entry + cache are cleared, `userAdded` is
unset). The channel still exists upstream.
   - Pure `user:`-prefixed channels are removed from the list as before.

## Repro
1. Open Channels.
2. Add a key for `#public` (or any rainbow-known channel).
3. Reload. Before this PR: row has no ✕, key is stuck. After this PR: ✕
appears, click clears the local key and cache.

## Files
- `public/channels.js` only.

## Notes
- No backend changes.
- No new APIs.
- Behaviour for purely user-added channels (e.g. `user:#somechannel` not
known to the server) is unchanged.

---------

Co-authored-by: you <you@example.com>
2026-04-22 21:41:43 -07:00
Kpa-clawbot abd9c46aa7 fix: side-panel Details button opens full-screen on desktop (#892)
## Symptom
🔍 Details button in the nodes side panel does nothing on click.

## Root cause (4th regression of the same shape)
- Row click → `selectNode()` → `history.replaceState(null, '',
'#/nodes/' + pk)`
- Details button click → `location.hash = '#/nodes/' + pk`
- Hash is already that value → assignment is a no-op → no `hashchange`
event → no router → panel stays open.

## Fix
Mirror the analytics-link branch already inside the panel click handler:
`destroy()` then `init(appEl, pubkey)` directly (which hits the
`directNode` full-screen branch unconditionally). Also `replaceState` to
keep the URL in sync.

## Test
New Playwright E2E: open side panel via row click, click Details, assert
`.node-fullscreen` appears.

## Why this keeps regressing
Every time we tighten the row-click handler to use `replaceState`
(correct — avoids hashchange flicker), the button-click handler that
uses `location.hash` becomes a no-op for the same pubkey. Need to
remember they're coupled. Worth a follow-up to extract a
`navigateToNode(pk)` helper that always works regardless of current hash
state — filing as #890-followup if not already there.

Co-authored-by: you <you@example.com>
2026-04-21 22:37:15 -07:00
Kpa-clawbot 6ca5e86df6 fix: compute hex-dump byte ranges client-side from per-obs raw_hex (#891)
## Symptom
The colored byte strip in the packet detail pane is offset from the
labeled byte breakdown below it. Off by N bytes where N is the
difference between the top-level packet's path length and the displayed
observation's path length.

## Root cause
Server computes `breakdown.ranges` once from the top-level packet's
raw_hex (in `BuildBreakdown`) and ships it in the API response. After
#882 we render each observation's own raw_hex, but we keep using the
top-level breakdown — so a 7-hop top-level packet shipped "Path: bytes
2-8", and when we rendered an 8-hop observation we coloured 7 of the 8
path bytes and bled into the payload.

The labeled rows below (which use `buildFieldTable`) parse the displayed
raw_hex on the client, so they were correct — they just didn't match the
strip above.

## Fix
Port `BuildBreakdown()` to JS as `computeBreakdownRanges()` in `app.js`.
Use it in `renderDetail()` from the actually-rendered (per-obs) raw_hex.

## Test
Manually verified the JS function output matches the Go implementation
for FLOOD/non-transport, transport, ADVERT, and direct-advert (zero
hops) cases.

Closes nothing (caught in post-tag bug bash).

---------

Co-authored-by: you <you@example.com>
2026-04-21 22:17:14 -07:00
Kpa-clawbot 56ec590bc4 fix(#886): derive path_json from raw_hex at ingest (#887)
## Problem

Per-observation `path_json` disagrees with `raw_hex` path section for
TRACE packets.

**Reproducer:** packet `af081a2c41281b1e`, observer `lutin🏡`
- `path_json`: `["67","33","D6","33","67"]` (5 hops — from TRACE
payload)
- `raw_hex` path section: `30 2D 0D 23` (4 bytes — SNR values in header)

## Root Cause

`DecodePacket` correctly parses TRACE packets by replacing `path.Hops`
with hop IDs from the payload's `pathData` field (the actual route).
However, the header path bytes for TRACE packets contain **SNR values**
(one per completed hop), not hop IDs.

`BuildPacketData` used `decoded.Path.Hops` to build `path_json`, which
for TRACE packets contained the payload-derived hops — not the header
path bytes that `raw_hex` stores. This caused `path_json` and `raw_hex`
to describe completely different paths.

## Fix

- Added `DecodePathFromRawHex(rawHex)` — extracts header path hops
directly from raw hex bytes, independent of any TRACE payload
overwriting.
- `BuildPacketData` now calls `DecodePathFromRawHex(msg.Raw)` instead of
using `decoded.Path.Hops`, guaranteeing `path_json` always matches the
`raw_hex` path section.

## Tests (8 new)

**`DecodePathFromRawHex` unit tests:**
- hash_size 1, 2, 3, 4
- zero-hop direct packets
- transport route (4-byte transport codes before path)

**`BuildPacketData` integration tests:**
- TRACE packet: asserts path_json matches raw_hex header path (not
payload hops)
- Non-TRACE packet: asserts path_json matches raw_hex header path

All existing tests continue to pass (`go test ./...` for both ingestor
and server).

Fixes #886

---------

Co-authored-by: you <you@example.com>
2026-04-21 21:13:58 -07:00
Kpa-clawbot 67aa47175f fix: path pill and byte breakdown agree on hop count (#885)
## Problem
On the packet detail pane, the **path pill** (top) and the **byte
breakdown** (bottom) showed different numbers of hops for the same
packet. Example: `46cf35504a21ef0d` rendered as `1 hop` badge followed
by 8 node names in the path pill, while the byte breakdown listed only 1
hop row.

## Root cause
Mixed data sources:
- Path-pill badge used `(raw_hex path_len) & 0x3F` (= firmware truth for
one observer = 1)
- Path-pill names used `path_json.length` (= server-aggregated longest
path across observers = 8)
- Byte breakdown section header used `(raw_hex path_len) & 0x3F` (= 1)
- Byte breakdown rows were sliced from `raw_hex` (= 1 row)
- `renderPath(pathHops, ...)` iterated all `path_json` entries

For group-header view, `packet.path_json` is aggregated across observers
and therefore longer than the raw_hex of any single observer's packet.

## Fix
Both surfaces now render from `pathHops` (= effective observation's
`path_json`). The raw_hex vs path_json mismatch is still logged as a
console.warn for diagnostics, but does not drive the UI.

With per-observation `raw_hex` (#882) shipped, clicking an observation
row already swaps the effective packet so both surfaces stay consistent.

## Testing
- Adds E2E regression `Packet detail path pill and byte breakdown agree
on hop count` that asserts:
  1. `pill badge count == byte breakdown section count`
  2. `rendered hop names ≈ badge count` (within 1 for separators)
  3. `byte breakdown rendered rows == section count`
- Manually reproduced on staging with `46cf35504a21ef0d` (8-name path +
`1 hop` badge before fix).

Related: #881 #882 #866

---------

Co-authored-by: you <you@example.com>
2026-04-21 17:57:06 -07:00
Kpa-clawbot 2b9f305698 fix(#874): hop-resolver affinity picker — score candidates by neighbor-graph edges + geographic centroid (#876)
## Problem

`pickByAffinity` in `hop-resolver.js` picks wrong regional candidates
when 1-byte pubkey prefixes collide. The old implementation only
considers one adjacent hop (forward OR backward pass), leading to
suboptimal picks when both neighbors provide useful context.

Measured on staging: **61.6% of hops have ≥2 same-prefix candidates**,
making collision resolution critical.

## Fix

Replaced the separate forward/backward pass disambiguation with a
**combined iterative resolver** that scores candidates against BOTH prev
and next resolved hops:

1. **Neighbor-graph edge weight** (priority 1): Sum edge scores to prev
+ next pubkeys. Pick max sum.
2. **Geographic centroid** (priority 2): Average lat/lon of prev + next
positions. Pick closest candidate by haversine distance.
3. **Single-anchor geo** (priority 3): When only one neighbor is
resolved, use it directly.
4. **Fallback** (priority 4): First candidate when no context exists.

The iterative approach resolves cascading dependencies — resolving one
ambiguous hop may unlock context for its neighbors.

### Dev-mode trace

Multi-candidate picks now emit: `[hop-resolver] hash=46 candidates=N
scored=[...] chose=<pubkey> method=graph|centroid|fallback`

## Before/After (staging, 1539 packets, 12928 hops)

| Metric | Before | After |
|--------|--------|-------|
| Unreliable hops | 39 (0.3%) | 23 (0.2%) |
| Packets with unreliable | 33 (2.14%) | 17 (1.10%) |

~41% reduction in unreliable hops, ~48% reduction in affected packets.

## Tests

5 new tests in `test-frontend-helpers.js`:
- Graph edge scoring picks correct regional candidate
- Next hop breaks tie when prev has no edges
- Centroid fallback when no graph edges exist
- Centroid uses average of prev+next positions
- Fallback when no context at all

All 595 tests pass. No regressions in `test-packet-filter.js` (62 pass)
or `test-aging.js` (29 pass).

Closes #874

---------

Co-authored-by: you <you@example.com>
2026-04-21 14:03:40 -07:00
Kpa-clawbot a605518d6d fix(#881): per-observation raw_hex — each observer sees different bytes on air (#882)
## Problem

Each MeshCore observer receives a physically distinct over-the-air byte
sequence for the same transmission (different path bytes, flags/hops
remaining). The `observations` table stored only `path_json` per
observer — all observations pointed at one `transmissions.raw_hex`. This
prevented the hex pane from updating when switching observations in the
packet detail view.

## Changes

| Layer | Change |
|-------|--------|
| **Schema** | `ALTER TABLE observations ADD COLUMN raw_hex TEXT`
(nullable). Migration: `observations_raw_hex_v1` |
| **Ingestor** | `stmtInsertObservation` now stores per-observer
`raw_hex` from MQTT payload |
| **View** | `packets_v` uses `COALESCE(o.raw_hex, t.raw_hex)` —
backward compatible with NULL historical rows |
| **Server** | `enrichObs` prefers `obs.RawHex` when non-empty, falls
back to `tx.RawHex` |
| **Frontend** | No changes — `effectivePkt.raw_hex` already flows
through `renderDetail` |

## Tests

- **Ingestor**: `TestPerObservationRawHex` — two MQTT packets for same
hash from different observers → both stored with distinct raw_hex
- **Server**: `TestPerObservationRawHexEnrich` — enrichObs returns
per-obs raw_hex when present, tx fallback when NULL
- **E2E**: Playwright assertion in `test-e2e-playwright.js` for hex pane
update on observation switch

E2E assertion added: `test-e2e-playwright.js:1794`

## Scope

- Historical observations: raw_hex stays NULL, UI falls back to
transmission raw_hex silently
- No backfill, no path_json reconstruction, no frontend changes

Closes #881

---------

Co-authored-by: you <you@example.com>
2026-04-21 13:45:29 -07:00
Kpa-clawbot 0ca559e348 fix(#866): per-observation children in expanded packet groups (#880)
## Problem
When a packet group is expanded in the Packets table, clicking any child
row pointed the side pane at the same aggregate packet — not the clicked
observation. URL would flip between `?obs=<packet_id>` values instead of
real observation ids.

## Root cause
The expand fetch used `/api/packets?hash=X&limit=20`, which returns ONE
aggregate row keyed by packet.id. Every child therefore carried
`data-value=<packet.id>`.

## Fix
Switch the expand fetch to `/api/packets/<hash>`, which includes the
full `observations[]` array. Build `_children` as `{...pkt, ...obs}` so
each child row gets a unique observation id and observation-level fields
(observer, path, timestamp, snr/rssi) override the aggregate.

## Verified live on staging
Tested on multiple packets:
- Click group-header → side pane shows observation 1 of N (first
observer)
- Click child row → pane updates to show THAT observer's details:
observer name, path, timestamp, obs counter (K of N), URL
`?obs=<observation_id>`

## Tests
592 frontend tests pass (no new ones — this is a wiring fix, live
E2E-verified instead).

Closes #866

---------

Co-authored-by: Kpa-clawbot <agent@corescope.local>
Co-authored-by: you <you@example.com>
2026-04-21 13:36:45 -07:00
67 changed files with 5025 additions and 2020 deletions
+1 -1
View File
@@ -1 +1 @@
{"schemaVersion":1,"label":"e2e tests","message":"45 passed","color":"brightgreen"}
{"schemaVersion":1,"label":"e2e tests","message":"83 passed","color":"brightgreen"}
+1 -1
View File
@@ -1 +1 @@
{"schemaVersion":1,"label":"frontend coverage","message":"39.68%","color":"red"}
{"schemaVersion":1,"label":"frontend coverage","message":"37.74%","color":"red"}
+3 -16
View File
@@ -135,7 +135,7 @@ jobs:
e2e-test:
name: "🎭 Playwright E2E Tests"
needs: [go-test]
runs-on: [self-hosted, Linux]
runs-on: ubuntu-latest
defaults:
run:
shell: bash
@@ -145,13 +145,6 @@ jobs:
with:
fetch-depth: 0
- name: Free disk space
run: |
# Prune old runner diagnostic logs (can accumulate 50MB+)
find ~/actions-runner/_diag/ -name '*.log' -mtime +3 -delete 2>/dev/null || true
# Show available disk space
df -h / | tail -1
- name: Set up Node.js 22
uses: actions/setup-node@v5
with:
@@ -252,17 +245,11 @@ jobs:
build-and-publish:
name: "🏗️ Build & Publish Docker Image"
needs: [e2e-test]
runs-on: [self-hosted, meshcore-runner-2]
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v5
- name: Free disk space
run: |
docker system prune -af 2>/dev/null || true
docker builder prune -af 2>/dev/null || true
df -h /
- name: Compute build metadata
id: meta
run: |
@@ -462,7 +449,7 @@ jobs:
name: "📝 Publish Badges & Summary"
if: github.event_name == 'push'
needs: [deploy]
runs-on: [self-hosted, Linux]
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v5
+2
View File
@@ -14,6 +14,7 @@ WORKDIR /build/server
COPY cmd/server/go.mod cmd/server/go.sum ./
COPY internal/geofilter/ ../../internal/geofilter/
COPY internal/sigvalidate/ ../../internal/sigvalidate/
COPY internal/packetpath/ ../../internal/packetpath/
RUN go mod download
COPY cmd/server/ ./
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
@@ -24,6 +25,7 @@ WORKDIR /build/ingestor
COPY cmd/ingestor/go.mod cmd/ingestor/go.sum ./
COPY internal/geofilter/ ../../internal/geofilter/
COPY internal/sigvalidate/ ../../internal/sigvalidate/
COPY internal/packetpath/ ../../internal/packetpath/
RUN go mod download
COPY cmd/ingestor/ ./
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
+15
View File
@@ -41,6 +41,7 @@ type Config struct {
Metrics *MetricsConfig `json:"metrics,omitempty"`
GeoFilter *GeoFilterConfig `json:"geo_filter,omitempty"`
ValidateSignatures *bool `json:"validateSignatures,omitempty"`
DB *DBConfig `json:"db,omitempty"`
}
// GeoFilterConfig is an alias for the shared geofilter.Config type.
@@ -58,6 +59,20 @@ type MetricsConfig struct {
SampleIntervalSec int `json:"sampleIntervalSec"`
}
// DBConfig controls SQLite vacuum and maintenance behavior (#919).
type DBConfig struct {
VacuumOnStartup bool `json:"vacuumOnStartup"` // one-time full VACUUM on startup if auto_vacuum is not INCREMENTAL
IncrementalVacuumPages int `json:"incrementalVacuumPages"` // pages returned to OS per reaper cycle (default 1024)
}
// IncrementalVacuumPages returns the configured pages per vacuum or 1024 default.
func (c *Config) IncrementalVacuumPages() int {
if c.DB != nil && c.DB.IncrementalVacuumPages > 0 {
return c.DB.IncrementalVacuumPages
}
return 1024
}
// ShouldValidateSignatures returns true (default) unless explicitly disabled.
func (c *Config) ShouldValidateSignatures() bool {
if c.ValidateSignatures != nil {
+87 -10
View File
@@ -11,6 +11,7 @@ import (
"sync/atomic"
"time"
"github.com/meshcore-analyzer/packetpath"
_ "modernc.org/sqlite"
)
@@ -58,7 +59,7 @@ func OpenStoreWithInterval(dbPath string, sampleIntervalSec int) (*Store, error)
return nil, fmt.Errorf("creating data dir: %w", err)
}
db, err := sql.Open("sqlite", dbPath+"?_pragma=journal_mode(WAL)&_pragma=foreign_keys(ON)&_pragma=busy_timeout(5000)")
db, err := sql.Open("sqlite", dbPath+"?_pragma=auto_vacuum(INCREMENTAL)&_pragma=journal_mode(WAL)&_pragma=foreign_keys(ON)&_pragma=busy_timeout(5000)")
if err != nil {
return nil, fmt.Errorf("opening db: %w", err)
}
@@ -84,6 +85,9 @@ func OpenStoreWithInterval(dbPath string, sampleIntervalSec int) (*Store, error)
}
func applySchema(db *sql.DB) error {
// auto_vacuum=INCREMENTAL is set via DSN pragma (must be before journal_mode).
// Logging of current mode is handled by CheckAutoVacuum — no duplicate log here.
schema := `
CREATE TABLE IF NOT EXISTS nodes (
public_key TEXT PRIMARY KEY,
@@ -189,7 +193,7 @@ func applySchema(db *sql.DB) error {
db.Exec(`DROP VIEW IF EXISTS packets_v`)
_, vErr := db.Exec(`
CREATE VIEW packets_v AS
SELECT o.id, t.raw_hex,
SELECT o.id, COALESCE(o.raw_hex, t.raw_hex) AS raw_hex,
datetime(o.timestamp, 'unixepoch') AS timestamp,
obs.id AS observer_id, obs.name AS observer_name,
o.direction, o.snr, o.rssi, o.score, t.hash, t.route_type,
@@ -408,6 +412,15 @@ func applySchema(db *sql.DB) error {
log.Println("[migration] dropped_packets table created")
}
// Migration: add raw_hex column to observations (#881)
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'observations_raw_hex_v1'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Adding raw_hex column to observations...")
db.Exec(`ALTER TABLE observations ADD COLUMN raw_hex TEXT`)
db.Exec(`INSERT INTO _migrations (name) VALUES ('observations_raw_hex_v1')`)
log.Println("[migration] observations.raw_hex column added")
}
return nil
}
@@ -433,12 +446,13 @@ func (s *Store) prepareStatements() error {
}
s.stmtInsertObservation, err = s.db.Prepare(`
INSERT INTO observations (transmission_id, observer_idx, direction, snr, rssi, score, path_json, timestamp)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
INSERT INTO observations (transmission_id, observer_idx, direction, snr, rssi, score, path_json, timestamp, raw_hex)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(transmission_id, observer_idx, COALESCE(path_json, '')) DO UPDATE SET
snr = COALESCE(excluded.snr, snr),
rssi = COALESCE(excluded.rssi, rssi),
score = COALESCE(excluded.score, score)
snr = COALESCE(excluded.snr, snr),
rssi = COALESCE(excluded.rssi, rssi),
score = COALESCE(excluded.score, score),
raw_hex = COALESCE(excluded.raw_hex, raw_hex)
`)
if err != nil {
return err
@@ -584,7 +598,7 @@ func (s *Store) InsertTransmission(data *PacketData) (bool, error) {
_, err = s.stmtInsertObservation.Exec(
txID, observerIdx, data.Direction,
data.SNR, data.RSSI, data.Score,
data.PathJSON, epochTs,
data.PathJSON, epochTs, nilIfEmpty(data.RawHex),
)
if err != nil {
s.Stats.WriteErrors.Add(1)
@@ -777,6 +791,58 @@ func (s *Store) PruneOldMetrics(retentionDays int) (int64, error) {
return n, nil
}
// CheckAutoVacuum inspects the current auto_vacuum mode and logs a warning
// if not INCREMENTAL. Performs opt-in full VACUUM if db.vacuumOnStartup is set (#919).
func (s *Store) CheckAutoVacuum(cfg *Config) {
var autoVacuum int
if err := s.db.QueryRow("PRAGMA auto_vacuum").Scan(&autoVacuum); err != nil {
log.Printf("[db] warning: could not read auto_vacuum: %v", err)
return
}
if autoVacuum == 2 {
log.Printf("[db] auto_vacuum=INCREMENTAL")
return
}
modes := map[int]string{0: "NONE", 1: "FULL", 2: "INCREMENTAL"}
mode := modes[autoVacuum]
if mode == "" {
mode = fmt.Sprintf("UNKNOWN(%d)", autoVacuum)
}
log.Printf("[db] auto_vacuum=%s — DB needs one-time VACUUM to enable incremental auto-vacuum. "+
"Set db.vacuumOnStartup: true in config to migrate (will block startup for several minutes on large DBs). "+
"See https://github.com/Kpa-clawbot/CoreScope/issues/919", mode)
if cfg.DB != nil && cfg.DB.VacuumOnStartup {
// WARNING: Full VACUUM creates a temporary copy of the entire DB file.
// Requires ~2× the DB file size in free disk space or it will fail.
log.Printf("[db] vacuumOnStartup=true — starting one-time full VACUUM (ensure 2x DB size free disk space)...")
start := time.Now()
if _, err := s.db.Exec("PRAGMA auto_vacuum = INCREMENTAL"); err != nil {
log.Printf("[db] VACUUM failed: could not set auto_vacuum: %v", err)
return
}
if _, err := s.db.Exec("VACUUM"); err != nil {
log.Printf("[db] VACUUM failed: %v", err)
return
}
elapsed := time.Since(start)
log.Printf("[db] VACUUM complete in %v — auto_vacuum is now INCREMENTAL", elapsed.Round(time.Millisecond))
}
}
// RunIncrementalVacuum returns free pages to the OS (#919).
// Safe to call on auto_vacuum=NONE databases (noop).
func (s *Store) RunIncrementalVacuum(pages int) {
if _, err := s.db.Exec(fmt.Sprintf("PRAGMA incremental_vacuum(%d)", pages)); err != nil {
log.Printf("[vacuum] incremental_vacuum error: %v", err)
}
}
// Checkpoint forces a WAL checkpoint to release the WAL lock file,
// preventing lock contention with a new process starting up.
func (s *Store) Checkpoint() {
@@ -931,11 +997,22 @@ type MQTTPacketMessage struct {
}
// BuildPacketData constructs a PacketData from a decoded packet and MQTT message.
// path_json is derived directly from raw_hex header bytes (not decoded.Path.Hops)
// to guarantee the stored path always matches the raw bytes. This matters for
// TRACE packets where decoded.Path.Hops is overwritten with payload hops (#886).
func BuildPacketData(msg *MQTTPacketMessage, decoded *DecodedPacket, observerID, region string) *PacketData {
now := time.Now().UTC().Format(time.RFC3339)
pathJSON := "[]"
if len(decoded.Path.Hops) > 0 {
b, _ := json.Marshal(decoded.Path.Hops)
// For TRACE packets, path_json must be the payload-decoded route hops
// (decoded.Path.Hops), NOT the raw_hex header bytes which are SNR values.
// For all other packet types, derive path from raw_hex (#886).
if !packetpath.PathBytesAreHops(byte(decoded.Header.PayloadType)) {
if len(decoded.Path.Hops) > 0 {
b, _ := json.Marshal(decoded.Path.Hops)
pathJSON = string(b)
}
} else if hops, err := packetpath.DecodePathFromRawHex(msg.Raw); err == nil && len(hops) > 0 {
b, _ := json.Marshal(hops)
pathJSON = string(b)
}
+155
View File
@@ -2,6 +2,7 @@ package main
import (
"database/sql"
"encoding/json"
"fmt"
"os"
"path/filepath"
@@ -10,6 +11,8 @@ import (
"sync/atomic"
"testing"
"time"
"github.com/meshcore-analyzer/packetpath"
)
func tempDBPath(t *testing.T) string {
@@ -1968,3 +1971,155 @@ func TestInsertObservationSNRFillIn(t *testing.T) {
t.Errorf("RSSI overwritten by null arrival: got %v, want %v", rssi3, rssi)
}
}
// TestPerObservationRawHex verifies that two MQTT packets for the same hash
// from different observers store distinct raw_hex per observation (#881).
func TestPerObservationRawHex(t *testing.T) {
store, err := OpenStore(tempDBPath(t))
if err != nil {
t.Fatal(err)
}
defer store.Close()
// Register two observers
store.UpsertObserver("obs-A", "Observer A", "", nil)
store.UpsertObserver("obs-B", "Observer B", "", nil)
hash := "abc123def456"
rawA := "c0ffee01"
rawB := "c0ffee0201aa"
dir := "RX"
// First observation from observer A
pdA := &PacketData{
RawHex: rawA,
Hash: hash,
Timestamp: "2026-04-21T10:00:00Z",
ObserverID: "obs-A",
Direction: &dir,
PathJSON: "[]",
}
isNew, err := store.InsertTransmission(pdA)
if err != nil {
t.Fatalf("insert A: %v", err)
}
if !isNew {
t.Fatal("expected new transmission")
}
// Second observation from observer B (same hash, different raw bytes)
pdB := &PacketData{
RawHex: rawB,
Hash: hash,
Timestamp: "2026-04-21T10:00:01Z",
ObserverID: "obs-B",
Direction: &dir,
PathJSON: `["aabb"]`,
}
isNew2, err := store.InsertTransmission(pdB)
if err != nil {
t.Fatalf("insert B: %v", err)
}
if isNew2 {
t.Fatal("expected duplicate transmission")
}
// Query observations and verify per-observation raw_hex
rows, err := store.db.Query(`
SELECT o.raw_hex, obs.id
FROM observations o
LEFT JOIN observers obs ON obs.rowid = o.observer_idx
ORDER BY o.id ASC
`)
if err != nil {
t.Fatalf("query: %v", err)
}
defer rows.Close()
type obsResult struct {
rawHex string
observerID string
}
var results []obsResult
for rows.Next() {
var rh, oid sql.NullString
if err := rows.Scan(&rh, &oid); err != nil {
t.Fatal(err)
}
results = append(results, obsResult{
rawHex: rh.String,
observerID: oid.String,
})
}
if len(results) != 2 {
t.Fatalf("expected 2 observations, got %d", len(results))
}
if results[0].rawHex != rawA {
t.Errorf("obs A raw_hex: got %q, want %q", results[0].rawHex, rawA)
}
if results[1].rawHex != rawB {
t.Errorf("obs B raw_hex: got %q, want %q", results[1].rawHex, rawB)
}
if results[0].rawHex == results[1].rawHex {
t.Error("both observations have same raw_hex — should differ")
}
}
// TestBuildPacketData_TraceUsesPayloadHops verifies that TRACE packets use
// payload-decoded route hops in path_json (NOT the raw_hex header SNR bytes).
// Issue #886 / #887.
func TestBuildPacketData_TraceUsesPayloadHops(t *testing.T) {
// TRACE packet: header path has SNR bytes [30,2D,0D,23], but decoded.Path.Hops
// is overwritten to payload hops [67,33,D6,33,67].
rawHex := "2604302D0D2359FEE7B100000000006733D63367"
decoded, err := DecodePacket(rawHex, nil, false)
if err != nil {
t.Fatal(err)
}
// decoded.Path.Hops should be the TRACE-replaced hops (payload hops)
if len(decoded.Path.Hops) != 5 {
t.Fatalf("expected 5 decoded hops, got %d", len(decoded.Path.Hops))
}
msg := &MQTTPacketMessage{Raw: rawHex}
pd := BuildPacketData(msg, decoded, "test-obs", "TST")
// For TRACE: path_json MUST be the payload-decoded route hops, NOT the SNR bytes
expectedPathJSON := `["67","33","D6","33","67"]`
if pd.PathJSON != expectedPathJSON {
t.Errorf("path_json = %s, want %s (TRACE must use payload hops)", pd.PathJSON, expectedPathJSON)
}
// Verify that DecodePathFromRawHex returns the SNR bytes (header path) which differ
headerHops, herr := packetpath.DecodePathFromRawHex(rawHex)
if herr != nil {
t.Fatal(herr)
}
headerJSON, _ := json.Marshal(headerHops)
if string(headerJSON) == expectedPathJSON {
t.Error("header path (SNR) should differ from payload hops for TRACE")
}
}
// TestBuildPacketData_NonTracePathJSON verifies non-TRACE packets also derive path from raw_hex.
func TestBuildPacketData_NonTracePathJSON(t *testing.T) {
// A simple ADVERT packet (payload type 0) with 2 hops, hash_size 1
// Header 0x09 = FLOOD(1), ADVERT(2), version 0
// Path byte 0x02 = hash_size 1, hash_count 2
// Path bytes: AA BB
rawHex := "0902AABB" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
decoded, err := DecodePacket(rawHex, nil, false)
if err != nil {
t.Fatal(err)
}
msg := &MQTTPacketMessage{Raw: rawHex}
pd := BuildPacketData(msg, decoded, "obs1", "TST")
expectedPathJSON := `["AA","BB"]`
if pd.PathJSON != expectedPathJSON {
t.Errorf("path_json = %s, want %s", pd.PathJSON, expectedPathJSON)
}
}
+3 -1
View File
@@ -12,6 +12,7 @@ import (
"strings"
"unicode/utf8"
"github.com/meshcore-analyzer/packetpath"
"github.com/meshcore-analyzer/sigvalidate"
)
@@ -192,8 +193,9 @@ func decodePath(pathByte byte, buf []byte, offset int) (Path, int) {
}, totalBytes
}
// isTransportRoute delegates to packetpath.IsTransportRoute.
func isTransportRoute(routeType int) bool {
return routeType == RouteTransportFlood || routeType == RouteTransportDirect
return packetpath.IsTransportRoute(routeType)
}
func decodeEncryptedPayload(typeName string, buf []byte) Payload {
+104
View File
@@ -11,6 +11,7 @@ import (
"strings"
"testing"
"github.com/meshcore-analyzer/packetpath"
"github.com/meshcore-analyzer/sigvalidate"
)
@@ -1822,3 +1823,106 @@ func TestDecodeAdvertWithSignatureValidation(t *testing.T) {
t.Error("SignatureValid should be nil when validation disabled")
}
}
// === Tests for DecodePathFromRawHex (issue #886) ===
func TestDecodePathFromRawHex_HashSize1(t *testing.T) {
// Header byte 0x26 = route_type DIRECT, payload TRACE
// Path byte 0x04 = hash_size 1 (bits 7-6 = 00 → 0+1=1), hash_count 4
// Path bytes: 30 2D 0D 23
raw := "2604302D0D2359FEE7B100000000006733D63367"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
expected := []string{"30", "2D", "0D", "23"}
if len(hops) != len(expected) {
t.Fatalf("got %d hops, want %d", len(hops), len(expected))
}
for i, h := range hops {
if h != expected[i] {
t.Errorf("hop[%d] = %s, want %s", i, h, expected[i])
}
}
}
func TestDecodePathFromRawHex_HashSize2(t *testing.T) {
// Path byte 0x42 = hash_size 2 (bits 7-6 = 01 → 1+1=2), hash_count 2
// Header 0x09 = FLOOD route (rt=1), payload ADVERT (pt=2)
// Path bytes: AABB CCDD (4 bytes = 2 hops * 2 bytes)
raw := "0942AABBCCDD" + "00000000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
expected := []string{"AABB", "CCDD"}
if len(hops) != len(expected) {
t.Fatalf("got %d hops, want %d", len(hops), len(expected))
}
for i, h := range hops {
if h != expected[i] {
t.Errorf("hop[%d] = %s, want %s", i, h, expected[i])
}
}
}
func TestDecodePathFromRawHex_HashSize3(t *testing.T) {
// Path byte 0x81 = hash_size 3 (bits 7-6 = 10 → 2+1=3), hash_count 1
// Header 0x09 = FLOOD route (rt=1), payload ADVERT
raw := "0981AABBCC" + "0000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
if len(hops) != 1 || hops[0] != "AABBCC" {
t.Fatalf("got %v, want [AABBCC]", hops)
}
}
func TestDecodePathFromRawHex_HashSize4(t *testing.T) {
// Path byte 0xC1 = hash_size 4 (bits 7-6 = 11 → 3+1=4), hash_count 1
// Header 0x09 = FLOOD route (rt=1)
raw := "09C1AABBCCDD" + "0000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
if len(hops) != 1 || hops[0] != "AABBCCDD" {
t.Fatalf("got %v, want [AABBCCDD]", hops)
}
}
func TestDecodePathFromRawHex_DirectZeroHops(t *testing.T) {
// Path byte 0x00 = hash_size 1, hash_count 0
// Header 0x0A = DIRECT route (rt=2), payload ADVERT
raw := "0A00" + "0000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
if len(hops) != 0 {
t.Fatalf("got %d hops, want 0", len(hops))
}
}
func TestDecodePathFromRawHex_Transport(t *testing.T) {
// Route type 3 = TRANSPORT_DIRECT → 4 transport code bytes before path byte
// Header 0x27 = route_type 3, payload TRACE
// Transport codes: 1122 3344
// Path byte 0x02 = hash_size 1, hash_count 2
// Path bytes: AA BB
raw := "2711223344" + "02AABB" + "0000000000"
hops, err := packetpath.DecodePathFromRawHex(raw)
if err != nil {
t.Fatal(err)
}
expected := []string{"AA", "BB"}
if len(hops) != len(expected) {
t.Fatalf("got %d hops, want %d", len(hops), len(expected))
}
for i, h := range hops {
if h != expected[i] {
t.Errorf("hop[%d] = %s, want %s", i, h, expected[i])
}
}
}
+4
View File
@@ -13,6 +13,10 @@ replace github.com/meshcore-analyzer/geofilter => ../../internal/geofilter
replace github.com/meshcore-analyzer/sigvalidate => ../../internal/sigvalidate
require github.com/meshcore-analyzer/packetpath v0.0.0
replace github.com/meshcore-analyzer/packetpath => ../../internal/packetpath
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
+9
View File
@@ -57,6 +57,9 @@ func main() {
defer store.Close()
log.Printf("SQLite opened: %s", cfg.DBPath)
// Check auto_vacuum mode and optionally migrate (#919)
store.CheckAutoVacuum(cfg)
// Node retention: move stale nodes to inactive_nodes on startup
nodeDays := cfg.NodeDaysOrDefault()
store.MoveStaleNodes(nodeDays)
@@ -69,12 +72,15 @@ func main() {
metricsDays := cfg.MetricsRetentionDays()
store.PruneOldMetrics(metricsDays)
store.PruneDroppedPackets(metricsDays)
vacuumPages := cfg.IncrementalVacuumPages()
store.RunIncrementalVacuum(vacuumPages)
// Daily ticker for node retention
retentionTicker := time.NewTicker(1 * time.Hour)
go func() {
for range retentionTicker.C {
store.MoveStaleNodes(nodeDays)
store.RunIncrementalVacuum(vacuumPages)
}
}()
@@ -83,8 +89,10 @@ func main() {
go func() {
time.Sleep(90 * time.Second) // stagger after metrics prune
store.RemoveStaleObservers(observerDays)
store.RunIncrementalVacuum(vacuumPages)
for range observerRetentionTicker.C {
store.RemoveStaleObservers(observerDays)
store.RunIncrementalVacuum(vacuumPages)
}
}()
@@ -94,6 +102,7 @@ func main() {
for range metricsRetentionTicker.C {
store.PruneOldMetrics(metricsDays)
store.PruneDroppedPackets(metricsDays)
store.RunIncrementalVacuum(vacuumPages)
}
}()
+88 -2
View File
@@ -127,6 +127,92 @@ func TestBoundedLoad_AscendingOrder(t *testing.T) {
}
}
// loadStoreWithRetention creates a PacketStore with retentionHours set.
func loadStoreWithRetention(t *testing.T, dbPath string, retentionHours float64) *PacketStore {
t.Helper()
db, err := OpenDB(dbPath)
if err != nil {
t.Fatal(err)
}
cfg := &PacketStoreConfig{RetentionHours: retentionHours}
store := NewPacketStore(db, cfg)
if err := store.Load(); err != nil {
t.Fatal(err)
}
return store
}
// createTestDBWithAgedPackets inserts numRecent packets with timestamps within
// the last hour and numOld packets with timestamps 48 hours ago.
func createTestDBWithAgedPackets(t *testing.T, numRecent, numOld int) string {
t.Helper()
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
conn, err := sql.Open("sqlite", dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
defer conn.Close()
execOrFail := func(s string) {
if _, err := conn.Exec(s); err != nil {
t.Fatalf("setup: %v\nSQL: %s", err, s)
}
}
execOrFail(`CREATE TABLE transmissions (id INTEGER PRIMARY KEY, raw_hex TEXT, hash TEXT, first_seen TEXT, route_type INTEGER, payload_type INTEGER, payload_version INTEGER, decoded_json TEXT)`)
execOrFail(`CREATE TABLE observations (id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_id TEXT, observer_name TEXT, direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT, raw_hex TEXT)`)
execOrFail(`CREATE TABLE observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT)`)
execOrFail(`CREATE TABLE nodes (pubkey TEXT PRIMARY KEY, name TEXT, role TEXT, lat REAL, lon REAL, last_seen TEXT, first_seen TEXT, frequency REAL)`)
execOrFail(`CREATE TABLE schema_version (version INTEGER)`)
execOrFail(`INSERT INTO schema_version (version) VALUES (1)`)
execOrFail(`CREATE INDEX idx_tx_first_seen ON transmissions(first_seen)`)
now := time.Now().UTC()
id := 1
// Insert old packets (48 hours ago)
for i := 0; i < numOld; i++ {
ts := now.Add(-48 * time.Hour).Add(time.Duration(i) * time.Second).Format(time.RFC3339)
conn.Exec("INSERT INTO transmissions VALUES (?,?,?,?,0,4,1,?)", id, "aa", fmt.Sprintf("old%d", i), ts, `{}`)
conn.Exec("INSERT INTO observations VALUES (?,?,?,?,?,?,?,?,?,?,?)", id, id, "obs1", "Obs1", "RX", -10.0, -80.0, 5, `[]`, ts, "")
id++
}
// Insert recent packets (within last hour)
for i := 0; i < numRecent; i++ {
ts := now.Add(-30 * time.Minute).Add(time.Duration(i) * time.Second).Format(time.RFC3339)
conn.Exec("INSERT INTO transmissions VALUES (?,?,?,?,0,4,1,?)", id, "bb", fmt.Sprintf("new%d", i), ts, `{}`)
conn.Exec("INSERT INTO observations VALUES (?,?,?,?,?,?,?,?,?,?,?)", id, id, "obs1", "Obs1", "RX", -10.0, -80.0, 5, `[]`, ts, "")
id++
}
return dbPath
}
func TestRetentionLoad_OnlyLoadsRecentPackets(t *testing.T) {
dbPath := createTestDBWithAgedPackets(t, 50, 100)
defer os.RemoveAll(filepath.Dir(dbPath))
// retention = 2 hours — should load only the 50 recent packets, not the 100 old ones
store := loadStoreWithRetention(t, dbPath, 2)
defer store.db.conn.Close()
if len(store.packets) != 50 {
t.Errorf("expected 50 recent packets, got %d (old packets should be excluded by retentionHours)", len(store.packets))
}
}
func TestRetentionLoad_ZeroRetentionLoadsAll(t *testing.T) {
dbPath := createTestDBWithAgedPackets(t, 50, 100)
defer os.RemoveAll(filepath.Dir(dbPath))
// retention = 0 (unlimited) — should load all 150 packets
store := loadStoreWithRetention(t, dbPath, 0)
defer store.db.conn.Close()
if len(store.packets) != 150 {
t.Errorf("expected all 150 packets with retentionHours=0, got %d", len(store.packets))
}
}
func TestEstimateStoreTxBytesTypical(t *testing.T) {
est := estimateStoreTxBytesTypical(10)
if est < 1000 {
@@ -229,7 +315,7 @@ func createTestDBAt(tb testing.TB, dbPath string, numTx int) {
id INTEGER PRIMARY KEY,
transmission_id INTEGER, observer_id TEXT, observer_name TEXT,
direction TEXT, snr REAL, rssi REAL, score INTEGER,
path_json TEXT, timestamp TEXT
path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT)`)
execOrFail(`CREATE TABLE IF NOT EXISTS nodes (
@@ -280,7 +366,7 @@ func createTestDBWithObs(tb testing.TB, dbPath string, numTx int) {
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observations (
id INTEGER PRIMARY KEY, transmission_id INTEGER, observer_id TEXT, observer_name TEXT,
direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT
direction TEXT, snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
execOrFail(`CREATE TABLE IF NOT EXISTS observers (rowid INTEGER PRIMARY KEY, id TEXT, name TEXT)`)
execOrFail(`CREATE TABLE IF NOT EXISTS nodes (
+165 -255
View File
@@ -12,20 +12,28 @@ import (
type SkewSeverity string
const (
SkewOK SkewSeverity = "ok" // < 5 min
SkewWarning SkewSeverity = "warning" // 5 min 1 hour
SkewCritical SkewSeverity = "critical" // 1 hour 30 days
SkewAbsurd SkewSeverity = "absurd" // > 30 days
SkewNoClock SkewSeverity = "no_clock" // > 365 days — uninitialized RTC
SkewBimodalClock SkewSeverity = "bimodal_clock" // mixed good+bad recent samples (flaky RTC)
SkewDefault SkewSeverity = "default" // firmware-default epoch + uptime
SkewOK SkewSeverity = "ok" // |skew| <= 15s
SkewDegrading SkewSeverity = "degrading" // 15s < |skew| <= 60s
SkewDegraded SkewSeverity = "degraded" // 60s < |skew| <= 600s
SkewWrong SkewSeverity = "wrong" // |skew| > 600s and not default
)
// Known firmware default epochs. Nodes with advert_ts in
// [epoch, epoch + maxPlausibleUptimeSec] are classified as "default".
// See docs/clock-skew-redesign.md for provenance of each value.
var defaultEpochs = []int64{0, 1609459200, 1672531200, 1715770351}
// Default thresholds in seconds.
const (
skewThresholdWarnSec = 5 * 60 // 5 minutes
skewThresholdCriticalSec = 60 * 60 // 1 hour
skewThresholdAbsurdSec = 30 * 24 * 3600 // 30 days
skewThresholdNoClockSec = 365 * 24 * 3600 // 365 days — uninitialized RTC
// maxPlausibleUptimeSec caps how far past a default epoch we still
// consider "default + uptime ticking". 730 days ≈ 2 years.
maxPlausibleUptimeSec = 1095 * 86400 // 3 years — covers solar repeater deployment lifetimes at firmware default
// Severity band boundaries (absolute skew in seconds).
skewThresholdOKSec = 15
skewThresholdDegradingSec = 60
skewThresholdDegradedSec = 600
// minDriftSamples is the minimum number of advert transmissions needed
// to compute a meaningful linear drift rate.
@@ -35,54 +43,52 @@ const (
// drift rates (> 1 day/day) indicate insufficient or outlier samples.
maxReasonableDriftPerDay = 86400.0
// recentSkewWindowCount is the number of most-recent advert samples
// used to derive the "current" skew for severity classification (see
// issue #789). The all-time median is poisoned by historical bad
// samples (e.g. a node that was off and then GPS-corrected); severity
// must reflect current health, not lifetime statistics.
recentSkewWindowCount = 5
// recentSkewWindowSec bounds the recent-window in time as well: only
// samples from the last N seconds count as "recent" for severity.
// The effective window is min(recentSkewWindowCount, samples in 1h).
recentSkewWindowSec = 3600
// bimodalSkewThresholdSec is the absolute skew threshold (1 hour)
// above which a sample is considered "bad" — likely firmware emitting
// a nonsense timestamp from an uninitialized RTC, not real drift.
// Chosen to match the warning/critical severity boundary: real clock
// drift rarely exceeds 1 hour, while epoch-0 RTCs produce ~1.7B sec.
bimodalSkewThresholdSec = 3600.0
// maxPlausibleSkewJumpSec is the largest skew change between
// consecutive samples that we treat as physical drift. Anything larger
// (e.g. a GPS sync that jumps the clock by minutes/days) is rejected
// as an outlier when computing drift. Real microcontroller drift is
// fractions of a second per advert; 60s is a generous safety factor.
// consecutive samples that we treat as physical drift.
maxPlausibleSkewJumpSec = 60.0
// theilSenMaxPoints caps the number of points fed to Theil-Sen
// regression (O(n²) in pairs). For nodes with thousands of samples we
// keep the most-recent points, which are also the most relevant for
// current drift.
// regression (O(n²) in pairs).
theilSenMaxPoints = 200
)
// classifySkew maps absolute skew (seconds) to a severity level.
// Float64 comparison is safe: inputs are rounded to 1 decimal via round(),
// and thresholds are integer multiples of 60 — no rounding artifacts.
func classifySkew(absSkewSec float64) SkewSeverity {
// isDefaultEpoch returns true if the raw advert timestamp falls within
// [epoch, epoch + maxPlausibleUptimeSec] for any known firmware default.
// If matched, returns the matched epoch; otherwise returns 0.
func isDefaultEpoch(advertTS int64) (bool, int64) {
// Find the largest epoch <= advertTS (closest match). Since ranges
// overlap, picking the closest avoids attributing a 2023-firmware
// node's timestamp to the 2024 epoch.
bestEpoch := int64(-1)
for _, epoch := range defaultEpochs {
if epoch <= advertTS && epoch > bestEpoch {
bestEpoch = epoch
}
}
if bestEpoch >= 0 && advertTS <= bestEpoch+maxPlausibleUptimeSec {
return true, bestEpoch
}
return false, 0
}
// classifySkew maps a raw advert timestamp and corrected skew (signed)
// to a severity level. Takes math.Abs internally so callers may pass
// signed values. Default detection runs on the raw advert_ts
// (independent of observer calibration).
func classifySkew(advertTS int64, skewSec float64) (SkewSeverity, int64) {
if ok, epoch := isDefaultEpoch(advertTS); ok {
return SkewDefault, epoch
}
abs := math.Abs(skewSec)
switch {
case absSkewSec >= skewThresholdNoClockSec:
return SkewNoClock
case absSkewSec >= skewThresholdAbsurdSec:
return SkewAbsurd
case absSkewSec >= skewThresholdCriticalSec:
return SkewCritical
case absSkewSec >= skewThresholdWarnSec:
return SkewWarning
case abs <= skewThresholdOKSec:
return SkewOK, 0
case abs <= skewThresholdDegradingSec:
return SkewDegrading, 0
case abs <= skewThresholdDegradedSec:
return SkewDegraded, 0
default:
return SkewOK
return SkewWrong, 0
}
}
@@ -90,38 +96,35 @@ func classifySkew(absSkewSec float64) SkewSeverity {
// skewSample is a single raw skew measurement from one advert observation.
type skewSample struct {
advertTS int64 // node's advert Unix timestamp
observedTS int64 // observation Unix timestamp
observerID string // which observer saw this
hash string // transmission hash (for multi-observer grouping)
advertTS int64 // node's advert Unix timestamp
observedTS int64 // observation Unix timestamp
observerID string // which observer saw this
hash string // transmission hash (for multi-observer grouping)
}
// ObserverCalibration holds the computed clock offset for an observer.
type ObserverCalibration struct {
ObserverID string `json:"observerID"`
OffsetSec float64 `json:"offsetSec"` // positive = observer clock ahead
Samples int `json:"samples"` // number of multi-observer packets used
OffsetSec float64 `json:"offsetSec"` // positive = observer clock ahead
Samples int `json:"samples"` // number of multi-observer packets used
}
// NodeClockSkew is the API response for a single node's clock skew data.
type NodeClockSkew struct {
Pubkey string `json:"pubkey"`
MeanSkewSec float64 `json:"meanSkewSec"` // corrected mean skew (positive = node ahead)
MedianSkewSec float64 `json:"medianSkewSec"` // corrected median skew
LastSkewSec float64 `json:"lastSkewSec"` // most recent corrected skew
RecentMedianSkewSec float64 `json:"recentMedianSkewSec"` // median across most-recent samples (drives severity, see #789)
DriftPerDaySec float64 `json:"driftPerDaySec"` // linear drift rate (sec/day)
Severity SkewSeverity `json:"severity"`
SampleCount int `json:"sampleCount"`
Calibrated bool `json:"calibrated"` // true if observer calibration was applied
LastAdvertTS int64 `json:"lastAdvertTS"` // most recent advert timestamp
LastObservedTS int64 `json:"lastObservedTS"` // most recent observation timestamp
Samples []SkewSample `json:"samples,omitempty"` // time-series for sparklines
GoodFraction float64 `json:"goodFraction"` // fraction of recent samples with |skew| <= 1h
RecentBadSampleCount int `json:"recentBadSampleCount"` // count of recent samples with |skew| > 1h
RecentSampleCount int `json:"recentSampleCount"` // total recent samples in window
NodeName string `json:"nodeName,omitempty"` // populated in fleet responses
NodeRole string `json:"nodeRole,omitempty"` // populated in fleet responses
Pubkey string `json:"pubkey"`
MeanSkewSec float64 `json:"meanSkewSec"` // corrected mean skew (positive = node ahead)
MedianSkewSec float64 `json:"medianSkewSec"` // corrected median skew
LastSkewSec float64 `json:"lastSkewSec"` // most recent corrected skew
DriftPerDaySec float64 `json:"driftPerDaySec"` // linear drift rate (sec/day)
Severity SkewSeverity `json:"severity"`
SampleCount int `json:"sampleCount"`
Calibrated bool `json:"calibrated"` // true if observer calibration was applied
LastAdvertTS int64 `json:"lastAdvertTS"` // most recent advert timestamp
LastObservedTS int64 `json:"lastObservedTS"` // most recent observation timestamp
DefaultEpoch *int64 `json:"defaultEpoch,omitempty"` // matched epoch when severity=default
Samples []SkewSample `json:"samples,omitempty"` // time-series for sparklines
NodeName string `json:"nodeName,omitempty"` // populated in fleet responses
NodeRole string `json:"nodeRole,omitempty"` // populated in fleet responses
}
// SkewSample is a single (timestamp, skew) point for sparkline rendering.
@@ -130,28 +133,26 @@ type SkewSample struct {
SkewSec float64 `json:"skew"` // corrected skew in seconds
}
// txSkewResult maps tx hash → per-transmission skew stats. This is an
// intermediate result keyed by hash (not pubkey); the store maps hash → pubkey
// when building the final per-node view.
// txSkewResult maps tx hash → per-transmission skew stats.
type txSkewResult = map[string]*NodeClockSkew
// ── Clock Skew Engine ──────────────────────────────────────────────────────────
// ClockSkewEngine computes and caches clock skew data for nodes and observers.
type ClockSkewEngine struct {
mu sync.RWMutex
observerOffsets map[string]float64 // observerID → calibrated offset (seconds)
observerSamples map[string]int // observerID → number of multi-observer packets used
nodeSkew txSkewResult
lastComputed time.Time
computeInterval time.Duration
mu sync.RWMutex
observerOffsets map[string]float64 // observerID → calibrated offset (seconds)
observerSamples map[string]int // observerID → number of multi-observer packets used
nodeSkew txSkewResult
lastComputed time.Time
computeInterval time.Duration
}
func NewClockSkewEngine() *ClockSkewEngine {
return &ClockSkewEngine{
observerOffsets: make(map[string]float64),
observerOffsets: make(map[string]float64),
observerSamples: make(map[string]int),
nodeSkew: make(txSkewResult),
nodeSkew: make(txSkewResult),
computeInterval: 30 * time.Second,
}
}
@@ -188,7 +189,6 @@ func (e *ClockSkewEngine) Recompute(store *PacketStore) {
// Swap results under brief write lock.
e.mu.Lock()
// Re-check: another goroutine may have computed while we were working.
if time.Since(e.lastComputed) < e.computeInterval {
e.mu.Unlock()
return
@@ -214,13 +214,13 @@ func collectSamples(store *PacketStore) []skewSample {
if decoded == nil {
continue
}
// Extract advert timestamp from decoded JSON.
advertTS := extractTimestamp(decoded)
if advertTS <= 0 {
if advertTS < 0 {
continue
}
// Sanity: skip timestamps before year 2020 or after year 2100.
if advertTS < 1577836800 || advertTS > 4102444800 {
// Allow epoch 0 and above (needed for default-epoch detection).
// Upper bound: year 2100.
if advertTS > 4102444800 {
continue
}
@@ -240,21 +240,43 @@ func collectSamples(store *PacketStore) []skewSample {
return samples
}
// timestampMissing is the sentinel returned by extractTimestamp when no
// timestamp field is present in the decoded advert. Using -1 lets us
// distinguish "field absent" from a real epoch-0 timestamp (ts == 0).
const timestampMissing int64 = -1
// extractTimestamp gets the Unix timestamp from a decoded ADVERT payload.
// Returns timestampMissing (-1) if no timestamp field is found.
func extractTimestamp(decoded map[string]interface{}) int64 {
// Try payload.timestamp first (nested in "payload" key).
if payload, ok := decoded["payload"]; ok {
if pm, ok := payload.(map[string]interface{}); ok {
if ts := jsonNumber(pm, "timestamp"); ts > 0 {
if ts, ok := jsonNumberOk(pm, "timestamp"); ok {
return ts
}
}
}
// Fallback: top-level timestamp.
if ts := jsonNumber(decoded, "timestamp"); ts > 0 {
if ts, ok := jsonNumberOk(decoded, "timestamp"); ok {
return ts
}
return 0
return timestampMissing
}
// jsonNumberOk extracts an int64 from a JSON-parsed map, returning (value, true)
// if the key exists and is numeric, or (0, false) otherwise.
func jsonNumberOk(m map[string]interface{}, key string) (int64, bool) {
v, ok := m[key]
if !ok || v == nil {
return 0, false
}
switch n := v.(type) {
case float64:
return int64(n), true
case int64:
return n, true
case int:
return int64(n), true
}
return 0, false
}
// jsonNumber extracts an int64 from a JSON-parsed map (handles float64 and json.Number).
@@ -281,7 +303,6 @@ func parseISO(s string) int64 {
}
t, err := time.Parse(time.RFC3339, s)
if err != nil {
// Try with fractional seconds.
t, err = time.Parse("2006-01-02T15:04:05.999999999Z07:00", s)
if err != nil {
return 0
@@ -295,19 +316,16 @@ func parseISO(s string) int64 {
// calibrateObservers computes each observer's clock offset using multi-observer
// packets. Returns offset map and sample count map.
func calibrateObservers(samples []skewSample) (map[string]float64, map[string]int) {
// Group observations by packet hash.
byHash := make(map[string][]skewSample)
for _, s := range samples {
byHash[s.hash] = append(byHash[s.hash], s)
}
// For each multi-observer packet, compute per-observer deviation from median.
deviations := make(map[string][]float64) // observerID → list of deviations
deviations := make(map[string][]float64)
for _, group := range byHash {
if len(group) < 2 {
continue // single-observer packet, can't calibrate
continue
}
// Compute median observation timestamp for this packet.
obsTimes := make([]float64, len(group))
for i, s := range group {
obsTimes[i] = float64(s.observedTS)
@@ -319,7 +337,6 @@ func calibrateObservers(samples []skewSample) (map[string]float64, map[string]in
}
}
// Each observer's offset = median of its deviations.
offsets := make(map[string]float64, len(deviations))
counts := make(map[string]int, len(deviations))
for obsID, devs := range deviations {
@@ -333,8 +350,6 @@ func calibrateObservers(samples []skewSample) (map[string]float64, map[string]in
// computeNodeSkew calculates corrected skew statistics for each node.
func computeNodeSkew(samples []skewSample, obsOffsets map[string]float64) txSkewResult {
// Compute corrected skew per sample, grouped by hash (each hash = one
// node's advert transmission). The caller maps hash → pubkey via byNode.
type correctedSample struct {
skew float64
observedTS int64
@@ -349,8 +364,6 @@ func computeNodeSkew(samples []skewSample, obsOffsets map[string]float64) txSkew
rawSkew := float64(s.advertTS - s.observedTS)
corrected := rawSkew
if hasCal {
// Observer offset = obs_ts - median(all_obs_ts). If observer is ahead,
// its obs_ts is inflated, making raw_skew too low. Add offset to correct.
corrected = rawSkew + obsOffset
}
byHash[s.hash] = append(byHash[s.hash], correctedSample{
@@ -361,10 +374,7 @@ func computeNodeSkew(samples []skewSample, obsOffsets map[string]float64) txSkew
hashAdvertTS[s.hash] = s.advertTS
}
// Each hash represents one advert from one node. Compute median corrected
// skew per hash (across multiple observers).
result := make(map[string]*NodeClockSkew) // keyed by hash for now
result := make(map[string]*NodeClockSkew)
for hash, cs := range byHash {
skews := make([]float64, len(cs))
for i, c := range cs {
@@ -373,29 +383,37 @@ func computeNodeSkew(samples []skewSample, obsOffsets map[string]float64) txSkew
medSkew := median(skews)
meanSkew := mean(skews)
// Find latest observation.
var latestObsTS int64
// Pick the skew from the most recent observation (max observedTS),
// not the last-appended sample which may be non-chronological.
var latest correctedSample
var anyCal bool
for _, c := range cs {
if c.observedTS > latestObsTS {
latestObsTS = c.observedTS
if c.observedTS > latest.observedTS {
latest = c
}
if c.calibrated {
anyCal = true
}
}
lastCorrectedSkew := latest.skew
advTS := hashAdvertTS[hash]
severity, matchedEpoch := classifySkew(advTS, lastCorrectedSkew)
absMedian := math.Abs(medSkew)
result[hash] = &NodeClockSkew{
ncs := &NodeClockSkew{
MeanSkewSec: round(meanSkew, 1),
MedianSkewSec: round(medSkew, 1),
LastSkewSec: round(cs[len(cs)-1].skew, 1),
Severity: classifySkew(absMedian),
LastSkewSec: round(lastCorrectedSkew, 1),
Severity: severity,
SampleCount: len(cs),
Calibrated: anyCal,
LastAdvertTS: hashAdvertTS[hash],
LastObservedTS: latestObsTS,
LastAdvertTS: advTS,
LastObservedTS: latest.observedTS,
}
if severity == SkewDefault {
ep := matchedEpoch
ncs.DefaultEpoch = &ep
}
result[hash] = ncs
}
return result
}
@@ -457,124 +475,45 @@ func (s *PacketStore) getNodeClockSkewLocked(pubkey string) *NodeClockSkew {
medSkew := median(allSkews)
meanSkew := mean(allSkews)
// Severity is derived from RECENT samples only (issue #789). The
// all-time median is poisoned by historical bad data — a node that
// was off for hours and then GPS-corrected can have median = -59M sec
// while its current skew is -0.8s. Operators need severity to reflect
// current health, so they trust the dashboard.
//
// Sort tsSkews by time and take the last recentSkewWindowCount samples
// (or all samples within recentSkewWindowSec of the latest, whichever
// gives FEWER samples — we want the more-current view; a chatty node
// can fit dozens of samples in 1h, in which case the count cap wins).
sort.Slice(tsSkews, func(i, j int) bool { return tsSkews[i].ts < tsSkews[j].ts })
// Classify using the most recent advert's raw timestamp and
// the most recent corrected skew. No windowing or median-driven
// severity — per-advert classification per the spec.
severity, matchedEpoch := classifySkew(lastAdvTS, lastSkew)
recentSkew := lastSkew
var recentVals []float64
if n := len(tsSkews); n > 0 {
latestTS := tsSkews[n-1].ts
// Index-based window: last K samples.
startByCount := n - recentSkewWindowCount
if startByCount < 0 {
startByCount = 0
}
// Time-based window: samples newer than latestTS - windowSec.
startByTime := n - 1
for i := n - 1; i >= 0; i-- {
if latestTS-tsSkews[i].ts <= recentSkewWindowSec {
startByTime = i
} else {
break
}
}
// Pick the narrower (larger-index) of the two windows — the most
// current view of the node's clock health.
start := startByCount
if startByTime > start {
start = startByTime
}
recentVals = make([]float64, 0, n-start)
for i := start; i < n; i++ {
recentVals = append(recentVals, tsSkews[i].skew)
}
if len(recentVals) > 0 {
recentSkew = median(recentVals)
}
}
// ── Bimodal detection (#845) ─────────────────────────────────────────
// Split recent samples into "good" (|skew| <= 1h, real clock) and
// "bad" (|skew| > 1h, firmware nonsense from uninitialized RTC).
// Classification order (first match wins):
// no_clock — goodFraction < 0.10 (essentially no real clock)
// bimodal_clock — 0.10 <= goodFraction < 0.80 AND badCount > 0
// ok/warn/etc. — goodFraction >= 0.80 (normal, outliers filtered)
var goodSamples []float64
for _, v := range recentVals {
if math.Abs(v) <= bimodalSkewThresholdSec {
goodSamples = append(goodSamples, v)
}
}
recentSampleCount := len(recentVals)
recentBadCount := recentSampleCount - len(goodSamples)
var goodFraction float64
if recentSampleCount > 0 {
goodFraction = float64(len(goodSamples)) / float64(recentSampleCount)
}
var severity SkewSeverity
if goodFraction < 0.10 {
// Essentially no real clock — classify as no_clock regardless
// of the raw skew magnitude.
severity = SkewNoClock
} else if goodFraction < 0.80 && recentBadCount > 0 {
// Bimodal: use median of GOOD samples as the "real" skew.
severity = SkewBimodalClock
if len(goodSamples) > 0 {
recentSkew = median(goodSamples)
}
} else {
// Normal path: if there are good samples, use their median
// (filters out rare outliers in ≥80% good case).
if len(goodSamples) > 0 && recentBadCount > 0 {
recentSkew = median(goodSamples)
}
severity = classifySkew(math.Abs(recentSkew))
}
// For no_clock / bimodal_clock nodes, skip drift when data is unreliable.
// Drift: display only, not a classifier input.
var drift float64
if severity != SkewNoClock && severity != SkewBimodalClock && len(tsSkews) >= minDriftSamples {
if severity != SkewDefault && len(tsSkews) >= minDriftSamples {
drift = computeDrift(tsSkews)
// Cap physically impossible drift rates.
if math.Abs(drift) > maxReasonableDriftPerDay {
drift = 0
}
}
// Build sparkline samples from tsSkews (already sorted by time above).
// Build sparkline samples.
sort.Slice(tsSkews, func(i, j int) bool { return tsSkews[i].ts < tsSkews[j].ts })
samples := make([]SkewSample, len(tsSkews))
for i, p := range tsSkews {
samples[i] = SkewSample{Timestamp: p.ts, SkewSec: round(p.skew, 1)}
}
return &NodeClockSkew{
Pubkey: pubkey,
MeanSkewSec: round(meanSkew, 1),
MedianSkewSec: round(medSkew, 1),
LastSkewSec: round(lastSkew, 1),
RecentMedianSkewSec: round(recentSkew, 1),
DriftPerDaySec: round(drift, 2),
Severity: severity,
SampleCount: totalSamples,
Calibrated: anyCal,
LastAdvertTS: lastAdvTS,
LastObservedTS: lastObsTS,
Samples: samples,
GoodFraction: round(goodFraction, 2),
RecentBadSampleCount: recentBadCount,
RecentSampleCount: recentSampleCount,
result := &NodeClockSkew{
Pubkey: pubkey,
MeanSkewSec: round(meanSkew, 1),
MedianSkewSec: round(medSkew, 1),
LastSkewSec: round(lastSkew, 1),
DriftPerDaySec: round(drift, 2),
Severity: severity,
SampleCount: totalSamples,
Calibrated: anyCal,
LastAdvertTS: lastAdvTS,
LastObservedTS: lastObsTS,
Samples: samples,
}
if severity == SkewDefault {
ep := matchedEpoch
result.DefaultEpoch = &ep
}
return result
}
// GetFleetClockSkew returns clock skew data for all nodes that have skew data.
@@ -583,7 +522,6 @@ func (s *PacketStore) GetFleetClockSkew() []*NodeClockSkew {
s.mu.RLock()
defer s.mu.RUnlock()
// Build name/role lookup from DB cache (requires s.mu held).
allNodes, _ := s.getCachedNodesAndPM()
nameMap := make(map[string]nodeInfo, len(allNodes))
for _, ni := range allNodes {
@@ -596,12 +534,10 @@ func (s *PacketStore) GetFleetClockSkew() []*NodeClockSkew {
if cs == nil {
continue
}
// Enrich with node name/role.
if ni, ok := nameMap[pubkey]; ok {
cs.NodeName = ni.Name
cs.NodeRole = ni.Role
}
// Omit samples in fleet response (too much data).
cs.Samples = nil
results = append(results, cs)
}
@@ -626,7 +562,6 @@ func (s *PacketStore) GetObserverCalibrations() []ObserverCalibration {
Samples: s.clockSkew.observerSamples[obsID],
})
}
// Sort by absolute offset descending.
sort.Slice(result, func(i, j int) bool {
return math.Abs(result[i].OffsetSec) > math.Abs(result[j].OffsetSec)
})
@@ -667,38 +602,20 @@ type tsSkewPair struct {
}
// computeDrift estimates linear drift in seconds per day from time-ordered
// (timestamp, skew) pairs. Issue #789: a single GPS-correction event (huge
// skew jump in seconds) used to dominate ordinary least squares and produce
// absurd drift like 1.7M sec/day. We now:
//
// 1. Drop pairs whose consecutive skew jump exceeds maxPlausibleSkewJumpSec
// (clock corrections, not physical drift). This protects both OLS-style
// consumers and Theil-Sen.
// 2. Use Theil-Sen regression — the slope is the median of all pairwise
// slopes, naturally robust to remaining outliers (breakdown point ~29%).
//
// For very small samples after filtering we fall back to a simple slope
// between first and last calibrated samples.
// (timestamp, skew) pairs using Theil-Sen regression with outlier filtering.
func computeDrift(pairs []tsSkewPair) float64 {
if len(pairs) < 2 {
return 0
}
// Sort by timestamp.
sort.Slice(pairs, func(i, j int) bool {
return pairs[i].ts < pairs[j].ts
})
// Time span too short? Skip.
spanSec := float64(pairs[len(pairs)-1].ts - pairs[0].ts)
if spanSec < 3600 { // need at least 1 hour of data
if spanSec < 3600 {
return 0
}
// Outlier filter: drop samples where the skew jumps more than
// maxPlausibleSkewJumpSec from the running "stable" baseline.
// We anchor on the first sample, then accept each subsequent point
// that's within the threshold of the most recent accepted point —
// this preserves a slow drift while rejecting correction events.
filtered := make([]tsSkewPair, 0, len(pairs))
filtered = append(filtered, pairs[0])
for i := 1; i < len(pairs); i++ {
@@ -707,30 +624,23 @@ func computeDrift(pairs []tsSkewPair) float64 {
filtered = append(filtered, pairs[i])
}
}
// If the filter killed too much (e.g. unstable node), fall back to the
// raw series so we at least produce *something* — it'll be capped by
// maxReasonableDriftPerDay downstream.
if len(filtered) < 2 || float64(filtered[len(filtered)-1].ts-filtered[0].ts) < 3600 {
filtered = pairs
}
// Cap point count for Theil-Sen (O(n²) on pairs). Keep most-recent.
if len(filtered) > theilSenMaxPoints {
filtered = filtered[len(filtered)-theilSenMaxPoints:]
}
return theilSenSlope(filtered) * 86400 // sec/sec → sec/day
return theilSenSlope(filtered) * 86400
}
// theilSenSlope returns the Theil-Sen estimator: median of all pairwise
// slopes (yj - yi) / (tj - ti) for i < j. Naturally robust to outliers.
// Pairs must be sorted by timestamp ascending.
// theilSenSlope returns the Theil-Sen estimator: median of all pairwise slopes.
func theilSenSlope(pairs []tsSkewPair) float64 {
n := len(pairs)
if n < 2 {
return 0
}
// Pre-allocate: n*(n-1)/2 pairs.
slopes := make([]float64, 0, n*(n-1)/2)
for i := 0; i < n; i++ {
for j := i + 1; j < n; j++ {
File diff suppressed because it is too large Load Diff
+16
View File
@@ -62,6 +62,8 @@ type Config struct {
Retention *RetentionConfig `json:"retention,omitempty"`
DB *DBConfig `json:"db,omitempty"`
PacketStore *PacketStoreConfig `json:"packetStore,omitempty"`
GeoFilter *GeoFilterConfig `json:"geo_filter,omitempty"`
@@ -129,6 +131,20 @@ type RetentionConfig struct {
MetricsDays int `json:"metricsDays"`
}
// DBConfig controls SQLite vacuum and maintenance behavior (#919).
type DBConfig struct {
VacuumOnStartup bool `json:"vacuumOnStartup"` // one-time full VACUUM on startup if auto_vacuum is not INCREMENTAL
IncrementalVacuumPages int `json:"incrementalVacuumPages"` // pages returned to OS per reaper cycle (default 1024)
}
// IncrementalVacuumPages returns the configured pages per vacuum or 1024 default.
func (c *Config) IncrementalVacuumPages() int {
if c.DB != nil && c.DB.IncrementalVacuumPages > 0 {
return c.DB.IncrementalVacuumPages
}
return 1024
}
// MetricsRetentionDays returns configured metrics retention or 30 days default.
func (c *Config) MetricsRetentionDays() int {
if c.Retention != nil && c.Retention.MetricsDays > 0 {
+8 -8
View File
@@ -47,7 +47,7 @@ func setupTestDBv2(t *testing.T) *DB {
id INTEGER PRIMARY KEY AUTOINCREMENT,
transmission_id INTEGER NOT NULL REFERENCES transmissions(id),
observer_id TEXT, observer_name TEXT, direction TEXT,
snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp INTEGER NOT NULL
snr REAL, rssi REAL, score INTEGER, path_json TEXT, timestamp INTEGER NOT NULL, raw_hex TEXT
);
`
if _, err := conn.Exec(schema); err != nil {
@@ -763,9 +763,9 @@ func TestGetChannelsFromStore(t *testing.T) {
func TestPrefixMapResolve(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aabbccdd11223344", Name: "NodeA", HasGPS: true, Lat: 37.5, Lon: -122.0},
{PublicKey: "aabbccdd55667788", Name: "NodeB", HasGPS: false},
{PublicKey: "eeff0011aabbccdd", Name: "NodeC", HasGPS: true, Lat: 38.0, Lon: -121.0},
{Role: "repeater", PublicKey: "aabbccdd11223344", Name: "NodeA", HasGPS: true, Lat: 37.5, Lon: -122.0},
{Role: "repeater", PublicKey: "aabbccdd55667788", Name: "NodeB", HasGPS: false},
{Role: "repeater", PublicKey: "eeff0011aabbccdd", Name: "NodeC", HasGPS: true, Lat: 38.0, Lon: -121.0},
}
pm := buildPrefixMap(nodes)
@@ -805,8 +805,8 @@ func TestPrefixMapResolve(t *testing.T) {
t.Run("multiple candidates no GPS", func(t *testing.T) {
noGPSNodes := []nodeInfo{
{PublicKey: "aa11bb22", Name: "X", HasGPS: false},
{PublicKey: "aa11cc33", Name: "Y", HasGPS: false},
{Role: "repeater", PublicKey: "aa11bb22", Name: "X", HasGPS: false},
{Role: "repeater", PublicKey: "aa11cc33", Name: "Y", HasGPS: false},
}
pm2 := buildPrefixMap(noGPSNodes)
n := pm2.resolve("aa11")
@@ -820,8 +820,8 @@ func TestPrefixMapResolve(t *testing.T) {
func TestPrefixMapCap(t *testing.T) {
// 16-char pubkey — longer than maxPrefixLen
nodes := []nodeInfo{
{PublicKey: "aabbccdd11223344", Name: "LongKey"},
{PublicKey: "eeff0011", Name: "ShortKey"}, // exactly 8 chars
{Role: "repeater", PublicKey: "aabbccdd11223344", Name: "LongKey"},
{Role: "repeater", PublicKey: "eeff0011", Name: "ShortKey"}, // exactly 8 chars
}
pm := buildPrefixMap(nodes)
+4
View File
@@ -20,6 +20,7 @@ type DB struct {
path string // filesystem path to the database file
isV3 bool // v3 schema: observer_idx in observations (vs observer_id in v2)
hasResolvedPath bool // observations table has resolved_path column
hasObsRawHex bool // observations table has raw_hex column (#881)
// Channel list cache (60s TTL) — avoids repeated GROUP BY scans (#762)
channelsCacheMu sync.Mutex
@@ -76,6 +77,9 @@ func (db *DB) detectSchema() {
if colName == "resolved_path" {
db.hasResolvedPath = true
}
if colName == "raw_hex" {
db.hasObsRawHex = true
}
}
}
}
+60 -2
View File
@@ -74,7 +74,8 @@ func setupTestDB(t *testing.T) *DB {
score INTEGER,
path_json TEXT,
timestamp INTEGER NOT NULL,
resolved_path TEXT
resolved_path TEXT,
raw_hex TEXT
);
CREATE TABLE IF NOT EXISTS observer_metrics (
@@ -1134,7 +1135,8 @@ func setupTestDBV2(t *testing.T) *DB {
rssi REAL,
score INTEGER,
path_json TEXT,
timestamp INTEGER NOT NULL
timestamp INTEGER NOT NULL,
raw_hex TEXT
);
`
if _, err := conn.Exec(schema); err != nil {
@@ -1975,3 +1977,59 @@ func TestParseWindowDuration(t *testing.T) {
}
}
}
// TestPerObservationRawHexEnrich verifies enrichObs returns per-observation raw_hex
// when available, falling back to transmission raw_hex when NULL (#881).
func TestPerObservationRawHexEnrich(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
// Insert observers
db.conn.Exec(`INSERT INTO observers (id, name) VALUES ('obs-a', 'Observer A')`)
db.conn.Exec(`INSERT INTO observers (id, name) VALUES ('obs-b', 'Observer B')`)
var rowA, rowB int64
db.conn.QueryRow(`SELECT rowid FROM observers WHERE id='obs-a'`).Scan(&rowA)
db.conn.QueryRow(`SELECT rowid FROM observers WHERE id='obs-b'`).Scan(&rowB)
// Insert transmission with raw_hex
txHex := "deadbeef"
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen) VALUES (?, 'hash1', '2026-04-21T10:00:00Z')`, txHex)
// Insert two observations: A has its own raw_hex, B has NULL (historical)
obsAHex := "c0ffee01"
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp, raw_hex)
VALUES (1, ?, -5.0, -90.0, '[]', 1745236800, ?)`, rowA, obsAHex)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, ?, -3.0, -85.0, '["aabb"]', 1745236801)`, rowB)
store := NewPacketStore(db, nil)
if err := store.Load(); err != nil {
t.Fatalf("store load: %v", err)
}
tx := store.byHash["hash1"]
if tx == nil {
t.Fatal("transmission not loaded")
}
if len(tx.Observations) < 2 {
t.Fatalf("expected 2 observations, got %d", len(tx.Observations))
}
// Check enriched observations
for _, obs := range tx.Observations {
m := store.enrichObs(obs)
rh, _ := m["raw_hex"].(string)
if obs.RawHex != "" {
// Observer A: should get per-observation raw_hex
if rh != obsAHex {
t.Errorf("obs with own raw_hex: got %q, want %q", rh, obsAHex)
}
} else {
// Observer B: should fall back to transmission raw_hex
if rh != txHex {
t.Errorf("obs without raw_hex: got %q, want %q (tx fallback)", rh, txHex)
}
}
}
}
+262
View File
@@ -0,0 +1,262 @@
package main
import (
"database/sql"
"os"
"path/filepath"
"strings"
"testing"
"time"
_ "modernc.org/sqlite"
)
// createFreshIngestorDB creates a SQLite DB using the ingestor's applySchema logic
// (simulated here) with auto_vacuum=INCREMENTAL set before tables.
func createFreshDBWithAutoVacuum(t *testing.T, path string) *sql.DB {
t.Helper()
// auto_vacuum must be set via DSN before journal_mode creates the DB file
db, err := sql.Open("sqlite", path+"?_pragma=auto_vacuum(INCREMENTAL)&_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)")
if err != nil {
t.Fatal(err)
}
db.SetMaxOpenConns(1)
// Create minimal schema
_, err = db.Exec(`
CREATE TABLE transmissions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
raw_hex TEXT NOT NULL,
hash TEXT NOT NULL UNIQUE,
first_seen TEXT NOT NULL,
route_type INTEGER,
payload_type INTEGER,
payload_version INTEGER,
decoded_json TEXT,
created_at TEXT DEFAULT (datetime('now')),
channel_hash TEXT
);
CREATE TABLE observations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
transmission_id INTEGER NOT NULL REFERENCES transmissions(id),
observer_idx INTEGER,
direction TEXT,
snr REAL,
rssi REAL,
score INTEGER,
path_json TEXT,
timestamp INTEGER NOT NULL
);
`)
if err != nil {
t.Fatal(err)
}
return db
}
func TestNewDBHasIncrementalAutoVacuum(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "test.db")
db := createFreshDBWithAutoVacuum(t, path)
defer db.Close()
var autoVacuum int
if err := db.QueryRow("PRAGMA auto_vacuum").Scan(&autoVacuum); err != nil {
t.Fatal(err)
}
if autoVacuum != 2 {
t.Fatalf("expected auto_vacuum=2 (INCREMENTAL), got %d", autoVacuum)
}
}
func TestExistingDBHasAutoVacuumNone(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "test.db")
// Create DB WITHOUT setting auto_vacuum (simulates old DB)
db, err := sql.Open("sqlite", path+"?_pragma=journal_mode(WAL)")
if err != nil {
t.Fatal(err)
}
db.SetMaxOpenConns(1)
_, err = db.Exec("CREATE TABLE dummy (id INTEGER PRIMARY KEY)")
if err != nil {
t.Fatal(err)
}
var autoVacuum int
if err := db.QueryRow("PRAGMA auto_vacuum").Scan(&autoVacuum); err != nil {
t.Fatal(err)
}
db.Close()
if autoVacuum != 0 {
t.Fatalf("expected auto_vacuum=0 (NONE) for old DB, got %d", autoVacuum)
}
}
func TestVacuumOnStartupMigratesDB(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "test.db")
// Create DB without auto_vacuum (old DB)
db, err := sql.Open("sqlite", path+"?_pragma=journal_mode(WAL)")
if err != nil {
t.Fatal(err)
}
db.SetMaxOpenConns(1)
_, err = db.Exec("CREATE TABLE dummy (id INTEGER PRIMARY KEY)")
if err != nil {
t.Fatal(err)
}
var before int
db.QueryRow("PRAGMA auto_vacuum").Scan(&before)
if before != 0 {
t.Fatalf("precondition: expected auto_vacuum=0, got %d", before)
}
db.Close()
// Simulate vacuumOnStartup migration using openRW
rw, err := openRW(path)
if err != nil {
t.Fatal(err)
}
if _, err := rw.Exec("PRAGMA auto_vacuum = INCREMENTAL"); err != nil {
t.Fatal(err)
}
if _, err := rw.Exec("VACUUM"); err != nil {
t.Fatal(err)
}
rw.Close()
// Verify migration
db2, err := sql.Open("sqlite", path+"?mode=ro")
if err != nil {
t.Fatal(err)
}
defer db2.Close()
var after int
if err := db2.QueryRow("PRAGMA auto_vacuum").Scan(&after); err != nil {
t.Fatal(err)
}
if after != 2 {
t.Fatalf("expected auto_vacuum=2 after VACUUM migration, got %d", after)
}
}
func TestIncrementalVacuumReducesFreelist(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "test.db")
db := createFreshDBWithAutoVacuum(t, path)
// Insert a bunch of data
now := time.Now().UTC().Format(time.RFC3339)
for i := 0; i < 500; i++ {
_, err := db.Exec(
"INSERT INTO transmissions (raw_hex, hash, first_seen) VALUES (?, ?, ?)",
strings.Repeat("AA", 200), // ~400 bytes each
"hash_"+string(rune('A'+i%26))+string(rune('0'+i/26)),
now,
)
if err != nil {
t.Fatal(err)
}
}
// Get file size before delete
db.Close()
infoBefore, _ := os.Stat(path)
sizeBefore := infoBefore.Size()
// Reopen and delete all
db, err := sql.Open("sqlite", path+"?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)")
if err != nil {
t.Fatal(err)
}
db.SetMaxOpenConns(1)
defer db.Close()
_, err = db.Exec("DELETE FROM transmissions")
if err != nil {
t.Fatal(err)
}
// Check freelist before vacuum
var freelistBefore int64
db.QueryRow("PRAGMA freelist_count").Scan(&freelistBefore)
if freelistBefore == 0 {
t.Fatal("expected non-zero freelist after DELETE")
}
// Run incremental vacuum
_, err = db.Exec("PRAGMA incremental_vacuum(10000)")
if err != nil {
t.Fatal(err)
}
// Check freelist after vacuum
var freelistAfter int64
db.QueryRow("PRAGMA freelist_count").Scan(&freelistAfter)
if freelistAfter >= freelistBefore {
t.Fatalf("expected freelist to shrink: before=%d after=%d", freelistBefore, freelistAfter)
}
// Checkpoint WAL and check file size shrunk
db.Exec("PRAGMA wal_checkpoint(TRUNCATE)")
db.Close()
infoAfter, _ := os.Stat(path)
sizeAfter := infoAfter.Size()
if sizeAfter >= sizeBefore {
t.Logf("warning: file did not shrink (before=%d after=%d) — may depend on page reuse", sizeBefore, sizeAfter)
}
}
func TestCheckAutoVacuumLogs(t *testing.T) {
// This test verifies checkAutoVacuum doesn't panic on various configs
dir := t.TempDir()
path := filepath.Join(dir, "test.db")
// Create a fresh DB with auto_vacuum=INCREMENTAL
dbConn := createFreshDBWithAutoVacuum(t, path)
db := &DB{conn: dbConn, path: path}
cfg := &Config{}
// Should not panic
checkAutoVacuum(db, cfg, path)
dbConn.Close()
// Create a DB without auto_vacuum
path2 := filepath.Join(dir, "test2.db")
dbConn2, _ := sql.Open("sqlite", path2+"?_pragma=journal_mode(WAL)")
dbConn2.SetMaxOpenConns(1)
dbConn2.Exec("CREATE TABLE dummy (id INTEGER PRIMARY KEY)")
db2 := &DB{conn: dbConn2, path: path2}
// Should log warning but not panic
checkAutoVacuum(db2, cfg, path2)
dbConn2.Close()
}
func TestConfigIncrementalVacuumPages(t *testing.T) {
// Default
cfg := &Config{}
if cfg.IncrementalVacuumPages() != 1024 {
t.Fatalf("expected default 1024, got %d", cfg.IncrementalVacuumPages())
}
// Custom
cfg.DB = &DBConfig{IncrementalVacuumPages: 512}
if cfg.IncrementalVacuumPages() != 512 {
t.Fatalf("expected 512, got %d", cfg.IncrementalVacuumPages())
}
// Zero should return default
cfg.DB.IncrementalVacuumPages = 0
if cfg.IncrementalVacuumPages() != 1024 {
t.Fatalf("expected default 1024 for zero, got %d", cfg.IncrementalVacuumPages())
}
}
+3 -101
View File
@@ -10,6 +10,7 @@ import (
"strings"
"time"
"github.com/meshcore-analyzer/packetpath"
"github.com/meshcore-analyzer/sigvalidate"
)
@@ -164,8 +165,9 @@ func decodePath(pathByte byte, buf []byte, offset int) (Path, int) {
}, totalBytes
}
// isTransportRoute delegates to packetpath.IsTransportRoute.
func isTransportRoute(routeType int) bool {
return routeType == RouteTransportFlood || routeType == RouteTransportDirect
return packetpath.IsTransportRoute(routeType)
}
func decodeEncryptedPayload(typeName string, buf []byte) Payload {
@@ -441,106 +443,6 @@ func DecodePacket(hexString string, validateSignatures bool) (*DecodedPacket, er
}, nil
}
// HexRange represents a labeled byte range for the hex breakdown visualization.
type HexRange struct {
Start int `json:"start"`
End int `json:"end"`
Label string `json:"label"`
}
// Breakdown holds colored byte ranges returned by the packet detail endpoint.
type Breakdown struct {
Ranges []HexRange `json:"ranges"`
}
// BuildBreakdown computes labeled byte ranges for each section of a MeshCore packet.
// The returned ranges are consumed by createColoredHexDump() and buildHexLegend()
// in the frontend (public/app.js).
func BuildBreakdown(hexString string) *Breakdown {
hexString = strings.ReplaceAll(hexString, " ", "")
hexString = strings.ReplaceAll(hexString, "\n", "")
hexString = strings.ReplaceAll(hexString, "\r", "")
buf, err := hex.DecodeString(hexString)
if err != nil || len(buf) < 2 {
return &Breakdown{Ranges: []HexRange{}}
}
var ranges []HexRange
offset := 0
// Byte 0: Header
ranges = append(ranges, HexRange{Start: 0, End: 0, Label: "Header"})
offset = 1
header := decodeHeader(buf[0])
// Bytes 1-4: Transport Codes (TRANSPORT_FLOOD / TRANSPORT_DIRECT only)
if isTransportRoute(header.RouteType) {
if len(buf) < offset+4 {
return &Breakdown{Ranges: ranges}
}
ranges = append(ranges, HexRange{Start: offset, End: offset + 3, Label: "Transport Codes"})
offset += 4
}
if offset >= len(buf) {
return &Breakdown{Ranges: ranges}
}
// Next byte: Path Length (bits 7-6 = hashSize-1, bits 5-0 = hashCount)
ranges = append(ranges, HexRange{Start: offset, End: offset, Label: "Path Length"})
pathByte := buf[offset]
offset++
hashSize := int(pathByte>>6) + 1
hashCount := int(pathByte & 0x3F)
pathBytes := hashSize * hashCount
// Path hops
if hashCount > 0 && offset+pathBytes <= len(buf) {
ranges = append(ranges, HexRange{Start: offset, End: offset + pathBytes - 1, Label: "Path"})
}
offset += pathBytes
if offset >= len(buf) {
return &Breakdown{Ranges: ranges}
}
payloadStart := offset
// Payload — break ADVERT into named sub-fields; everything else is one Payload range
if header.PayloadType == PayloadADVERT && len(buf)-payloadStart >= 100 {
ranges = append(ranges, HexRange{Start: payloadStart, End: payloadStart + 31, Label: "PubKey"})
ranges = append(ranges, HexRange{Start: payloadStart + 32, End: payloadStart + 35, Label: "Timestamp"})
ranges = append(ranges, HexRange{Start: payloadStart + 36, End: payloadStart + 99, Label: "Signature"})
appStart := payloadStart + 100
if appStart < len(buf) {
ranges = append(ranges, HexRange{Start: appStart, End: appStart, Label: "Flags"})
appFlags := buf[appStart]
fOff := appStart + 1
if appFlags&0x10 != 0 && fOff+8 <= len(buf) {
ranges = append(ranges, HexRange{Start: fOff, End: fOff + 3, Label: "Latitude"})
ranges = append(ranges, HexRange{Start: fOff + 4, End: fOff + 7, Label: "Longitude"})
fOff += 8
}
if appFlags&0x20 != 0 && fOff+2 <= len(buf) {
fOff += 2
}
if appFlags&0x40 != 0 && fOff+2 <= len(buf) {
fOff += 2
}
if appFlags&0x80 != 0 && fOff < len(buf) {
ranges = append(ranges, HexRange{Start: fOff, End: len(buf) - 1, Label: "Name"})
}
}
} else {
ranges = append(ranges, HexRange{Start: payloadStart, End: len(buf) - 1, Label: "Payload"})
}
return &Breakdown{Ranges: ranges}
}
// ComputeContentHash computes the SHA-256-based content hash (first 16 hex chars).
// It hashes the payload-type nibble + payload (skipping path bytes) to produce a
// route-independent identifier for the same logical packet. For TRACE packets,
-140
View File
@@ -97,146 +97,6 @@ func TestDecodePacket_FloodHasNoCodes(t *testing.T) {
}
}
func TestBuildBreakdown_InvalidHex(t *testing.T) {
b := BuildBreakdown("not-hex!")
if len(b.Ranges) != 0 {
t.Errorf("expected empty ranges for invalid hex, got %d", len(b.Ranges))
}
}
func TestBuildBreakdown_TooShort(t *testing.T) {
b := BuildBreakdown("11") // 1 byte — no path byte
if len(b.Ranges) != 0 {
t.Errorf("expected empty ranges for too-short packet, got %d", len(b.Ranges))
}
}
func TestBuildBreakdown_FloodNonAdvert(t *testing.T) {
// Header 0x15: route=1/FLOOD, payload=5/GRP_TXT
// PathByte 0x01: 1 hop, 1-byte hash
// PathHop: AA
// Payload: FF0011
b := BuildBreakdown("1501AAFFFF00")
labels := rangeLabels(b.Ranges)
expect := []string{"Header", "Path Length", "Path", "Payload"}
if !equalLabels(labels, expect) {
t.Errorf("expected labels %v, got %v", expect, labels)
}
// Verify byte positions
assertRange(t, b.Ranges, "Header", 0, 0)
assertRange(t, b.Ranges, "Path Length", 1, 1)
assertRange(t, b.Ranges, "Path", 2, 2)
assertRange(t, b.Ranges, "Payload", 3, 5)
}
func TestBuildBreakdown_TransportFlood(t *testing.T) {
// Header 0x14: route=0/TRANSPORT_FLOOD, payload=5/GRP_TXT
// TransportCodes: AABBCCDD (4 bytes)
// PathByte 0x01: 1 hop, 1-byte hash
// PathHop: EE
// Payload: FF00
b := BuildBreakdown("14AABBCCDD01EEFF00")
assertRange(t, b.Ranges, "Header", 0, 0)
assertRange(t, b.Ranges, "Transport Codes", 1, 4)
assertRange(t, b.Ranges, "Path Length", 5, 5)
assertRange(t, b.Ranges, "Path", 6, 6)
assertRange(t, b.Ranges, "Payload", 7, 8)
}
func TestBuildBreakdown_FloodNoHops(t *testing.T) {
// Header 0x15: FLOOD/GRP_TXT; PathByte 0x00: 0 hops; Payload: AABB
b := BuildBreakdown("150000AABB")
assertRange(t, b.Ranges, "Header", 0, 0)
assertRange(t, b.Ranges, "Path Length", 1, 1)
// No Path range since hashCount=0
for _, r := range b.Ranges {
if r.Label == "Path" {
t.Error("expected no Path range for zero-hop packet")
}
}
assertRange(t, b.Ranges, "Payload", 2, 4)
}
func TestBuildBreakdown_AdvertBasic(t *testing.T) {
// Header 0x11: FLOOD/ADVERT
// PathByte 0x01: 1 hop, 1-byte hash
// PathHop: AA
// Payload: 100 bytes (PubKey32 + Timestamp4 + Signature64) + Flags=0x02 (repeater, no extras)
pubkey := repeatHex("AB", 32)
ts := "00000000" // 4 bytes
sig := repeatHex("CD", 64)
flags := "02"
hex := "1101AA" + pubkey + ts + sig + flags
b := BuildBreakdown(hex)
assertRange(t, b.Ranges, "Header", 0, 0)
assertRange(t, b.Ranges, "Path Length", 1, 1)
assertRange(t, b.Ranges, "Path", 2, 2)
assertRange(t, b.Ranges, "PubKey", 3, 34)
assertRange(t, b.Ranges, "Timestamp", 35, 38)
assertRange(t, b.Ranges, "Signature", 39, 102)
assertRange(t, b.Ranges, "Flags", 103, 103)
}
func TestBuildBreakdown_AdvertWithLocation(t *testing.T) {
// flags=0x12: hasLocation bit set
pubkey := repeatHex("00", 32)
ts := "00000000"
sig := repeatHex("00", 64)
flags := "12" // 0x10 = hasLocation
latBytes := "00000000"
lonBytes := "00000000"
hex := "1101AA" + pubkey + ts + sig + flags + latBytes + lonBytes
b := BuildBreakdown(hex)
assertRange(t, b.Ranges, "Latitude", 104, 107)
assertRange(t, b.Ranges, "Longitude", 108, 111)
}
func TestBuildBreakdown_AdvertWithName(t *testing.T) {
// flags=0x82: hasName bit set
pubkey := repeatHex("00", 32)
ts := "00000000"
sig := repeatHex("00", 64)
flags := "82" // 0x80 = hasName
name := "4E6F6465" // "Node" in hex
hex := "1101AA" + pubkey + ts + sig + flags + name
b := BuildBreakdown(hex)
assertRange(t, b.Ranges, "Name", 104, 107)
}
// helpers
func rangeLabels(ranges []HexRange) []string {
out := make([]string, len(ranges))
for i, r := range ranges {
out[i] = r.Label
}
return out
}
func equalLabels(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
func assertRange(t *testing.T, ranges []HexRange, label string, wantStart, wantEnd int) {
t.Helper()
for _, r := range ranges {
if r.Label == label {
if r.Start != wantStart || r.End != wantEnd {
t.Errorf("range %q: want [%d,%d], got [%d,%d]", label, wantStart, wantEnd, r.Start, r.End)
}
return
}
}
t.Errorf("range %q not found in %v", label, rangeLabels(ranges))
}
func TestZeroHopDirectHashSize(t *testing.T) {
// DIRECT (RouteType=2) + REQ (PayloadType=0) → header byte = 0x02
+4
View File
@@ -14,6 +14,10 @@ replace github.com/meshcore-analyzer/geofilter => ../../internal/geofilter
replace github.com/meshcore-analyzer/sigvalidate => ../../internal/sigvalidate
require github.com/meshcore-analyzer/packetpath v0.0.0
replace github.com/meshcore-analyzer/packetpath => ../../internal/packetpath
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
-194
View File
@@ -1,194 +0,0 @@
package main
import (
"database/sql"
"testing"
"time"
"path/filepath"
_ "modernc.org/sqlite"
)
// setupTestDB871 creates a test DB with schema and returns a read-only *DB handle.
func setupTestDB871(t *testing.T) (*DB, *sql.DB) {
t.Helper()
dbPath := filepath.Join(t.TempDir(), "test871.db")
// Open writable connection for setup
rw, err := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
if err != nil {
t.Fatal(err)
}
_, err = rw.Exec(`
CREATE TABLE IF NOT EXISTS nodes (
public_key TEXT PRIMARY KEY,
name TEXT, role TEXT,
lat REAL, lon REAL,
last_seen TEXT, first_seen TEXT,
advert_count INTEGER DEFAULT 0,
battery_mv INTEGER, temperature_c REAL
);
CREATE TABLE IF NOT EXISTS transmissions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
raw_hex TEXT NOT NULL,
hash TEXT NOT NULL UNIQUE,
first_seen TEXT NOT NULL,
route_type INTEGER,
payload_type INTEGER,
payload_version INTEGER,
decoded_json TEXT,
created_at TEXT DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS observers (
rowid INTEGER PRIMARY KEY AUTOINCREMENT,
id TEXT NOT NULL UNIQUE,
name TEXT
);
CREATE TABLE IF NOT EXISTS observations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
transmission_id INTEGER NOT NULL,
observer_id TEXT,
observer_name TEXT,
direction TEXT,
snr REAL, rssi REAL, score INTEGER,
path_json TEXT, timestamp TEXT
);
`)
if err != nil {
t.Fatal(err)
}
// Open read-only handle for the store
db, err := OpenDB(dbPath)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
db.Close()
rw.Close()
})
return db, rw
}
// TestEnrichObsFallbackToDB verifies that enrichObs falls back to the DB when
// the parent transmission has been evicted from memory (#871 root cause).
func TestEnrichObsFallbackToDB(t *testing.T) {
db, rw := setupTestDB871(t)
now := time.Now().UTC().Format(time.RFC3339)
_, err := rw.Exec(
`INSERT INTO transmissions (raw_hex, hash, first_seen, payload_type, decoded_json) VALUES (?, ?, ?, ?, ?)`,
"aabbcc", "abc123", now, 4, `{"pubKey":"pk1"}`,
)
if err != nil {
t.Fatal(err)
}
store := NewPacketStore(db, &PacketStoreConfig{})
// Observation references tx_id=1, but tx is NOT in byTxID (simulates eviction)
obs := &StoreObs{
ID: 1,
TransmissionID: 1,
ObserverID: "obs1",
ObserverName: "Observer1",
Timestamp: now,
}
result := store.enrichObs(obs)
// hash must be present from DB fallback
if result["hash"] == nil {
t.Errorf("enrichObs: hash is nil — DB fallback failed")
}
if h, ok := result["hash"].(string); !ok || h != "abc123" {
t.Errorf("enrichObs: expected hash 'abc123', got %v", result["hash"])
}
if result["payload_type"] == nil {
t.Errorf("enrichObs: payload_type is nil — DB fallback failed")
}
// When tx IS in memory, it should use the in-memory path
pt := 4
store.byTxID[1] = &StoreTx{
ID: 1, Hash: "abc123", FirstSeen: now,
PayloadType: &pt, RawHex: "aabbcc",
}
result2 := store.enrichObs(obs)
if result2["hash"] == nil {
t.Errorf("enrichObs with in-memory tx: hash is nil")
}
}
// TestGetNodeHealthRecentPacketsNoNilFields verifies that GetNodeHealth's
// recentPackets never contains entries with nil hash or timestamp.
func TestGetNodeHealthRecentPacketsNoNilFields(t *testing.T) {
db, rw := setupTestDB871(t)
now := time.Now().UTC().Format(time.RFC3339)
_, err := rw.Exec(
`INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)`,
"pk1", "TestNode", "repeater", now,
)
if err != nil {
t.Fatal(err)
}
store := NewPacketStore(db, &PacketStoreConfig{})
pt := 4
tx := &StoreTx{
ID: 1, Hash: "hash1", FirstSeen: now,
PayloadType: &pt, DecodedJSON: `{"pubKey":"pk1"}`,
obsKeys: make(map[string]bool), observerSet: make(map[string]bool),
}
store.byTxID[1] = tx
store.byHash["hash1"] = tx
store.byNode["pk1"] = []*StoreTx{tx}
store.nodeHashes["pk1"] = map[string]bool{"hash1": true}
result, err := store.GetNodeHealth("pk1")
if err != nil {
t.Fatal(err)
}
if result == nil {
t.Fatal("GetNodeHealth returned nil")
}
packets, ok := result["recentPackets"].([]map[string]interface{})
if !ok {
t.Fatal("recentPackets is not []map[string]interface{}")
}
for i, p := range packets {
if p["hash"] == nil {
t.Errorf("recentPackets[%d] has nil hash", i)
}
if p["timestamp"] == nil {
t.Errorf("recentPackets[%d] has nil timestamp", i)
}
}
}
// TestEnrichObsNilDB verifies enrichObs doesn't panic when db is nil.
func TestEnrichObsNilDB(t *testing.T) {
store := &PacketStore{
byTxID: make(map[int]*StoreTx),
byObsID: make(map[int]*StoreObs),
}
obs := &StoreObs{
ID: 1, TransmissionID: 999,
Timestamp: "2026-01-01T00:00:00Z",
}
// Should not panic
result := store.enrichObs(obs)
if result["hash"] != nil {
t.Errorf("expected nil hash when no DB and no in-memory tx, got %v", result["hash"])
}
}
+16
View File
@@ -148,6 +148,9 @@ func main() {
stats.TotalTransmissions, stats.TotalObservations, stats.TotalNodes, stats.TotalObservers)
}
// Check auto_vacuum mode and optionally migrate (#919)
checkAutoVacuum(database, cfg, resolvedDB)
// In-memory packet store
store := NewPacketStore(database, cfg.PacketStore, cfg.CacheTTL)
if err := store.Load(); err != nil {
@@ -266,6 +269,7 @@ func main() {
defer stopEviction()
// Auto-prune old packets if retention.packetDays is configured
vacuumPages := cfg.IncrementalVacuumPages()
var stopPrune func()
if cfg.Retention != nil && cfg.Retention.PacketDays > 0 {
days := cfg.Retention.PacketDays
@@ -286,6 +290,9 @@ func main() {
log.Printf("[prune] error: %v", err)
} else {
log.Printf("[prune] deleted %d transmissions older than %d days", n, days)
if n > 0 {
runIncrementalVacuum(resolvedDB, vacuumPages)
}
}
for {
select {
@@ -294,6 +301,9 @@ func main() {
log.Printf("[prune] error: %v", err)
} else {
log.Printf("[prune] deleted %d transmissions older than %d days", n, days)
if n > 0 {
runIncrementalVacuum(resolvedDB, vacuumPages)
}
}
case <-pruneDone:
return
@@ -321,10 +331,12 @@ func main() {
}()
time.Sleep(2 * time.Minute) // stagger after packet prune
database.PruneOldMetrics(metricsDays)
runIncrementalVacuum(resolvedDB, vacuumPages)
for {
select {
case <-metricsPruneTicker.C:
database.PruneOldMetrics(metricsDays)
runIncrementalVacuum(resolvedDB, vacuumPages)
case <-metricsPruneDone:
return
}
@@ -354,10 +366,12 @@ func main() {
}()
time.Sleep(3 * time.Minute) // stagger after metrics prune
database.RemoveStaleObservers(observerDays)
runIncrementalVacuum(resolvedDB, vacuumPages)
for {
select {
case <-observerPruneTicker.C:
database.RemoveStaleObservers(observerDays)
runIncrementalVacuum(resolvedDB, vacuumPages)
case <-observerPruneDone:
return
}
@@ -388,6 +402,7 @@ func main() {
g := store.graph
store.mu.RUnlock()
PruneNeighborEdges(dbPath, g, maxAgeDays)
runIncrementalVacuum(resolvedDB, vacuumPages)
for {
select {
case <-edgePruneTicker.C:
@@ -395,6 +410,7 @@ func main() {
g := store.graph
store.mu.RUnlock()
PruneNeighborEdges(dbPath, g, maxAgeDays)
runIncrementalVacuum(resolvedDB, vacuumPages)
case <-edgePruneDone:
return
}
+18 -18
View File
@@ -12,9 +12,9 @@ import (
func TestResolveAmbiguousEdges_GeoProximity(t *testing.T) {
// Node A at lat=45, lon=-122. Candidate B1 at lat=45.1, lon=-122.1 (close).
// Candidate B2 at lat=10, lon=10 (far away). Prefix "b0" matches both.
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB1 := nodeInfo{PublicKey: "b0b1eeee", Name: "CloseNode", HasGPS: true, Lat: 45.1, Lon: -122.1}
nodeB2 := nodeInfo{PublicKey: "b0c2ffff", Name: "FarNode", HasGPS: true, Lat: 10.0, Lon: 10.0}
nodeA := nodeInfo{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB1 := nodeInfo{Role: "repeater", PublicKey: "b0b1eeee", Name: "CloseNode", HasGPS: true, Lat: 45.1, Lon: -122.1}
nodeB2 := nodeInfo{Role: "repeater", PublicKey: "b0c2ffff", Name: "FarNode", HasGPS: true, Lat: 10.0, Lon: 10.0}
pm := buildPrefixMap([]nodeInfo{nodeA, nodeB1, nodeB2})
@@ -62,8 +62,8 @@ func TestResolveAmbiguousEdges_GeoProximity(t *testing.T) {
// Test 2: Ambiguous edge merged with existing resolved edge (count accumulation).
func TestResolveAmbiguousEdges_MergeWithExisting(t *testing.T) {
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{PublicKey: "b0b1eeee", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
nodeA := nodeInfo{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{Role: "repeater", PublicKey: "b0b1eeee", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
pm := buildPrefixMap([]nodeInfo{nodeA, nodeB})
@@ -133,9 +133,9 @@ func TestResolveAmbiguousEdges_MergeWithExisting(t *testing.T) {
// Test 3: Ambiguous edge left as-is when resolution fails.
func TestResolveAmbiguousEdges_FailsNoChange(t *testing.T) {
// Two candidates, neither has GPS, no affinity data — resolution falls through.
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA"}
nodeB1 := nodeInfo{PublicKey: "b0b1eeee", Name: "B1"}
nodeB2 := nodeInfo{PublicKey: "b0c2ffff", Name: "B2"}
nodeA := nodeInfo{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeA"}
nodeB1 := nodeInfo{Role: "repeater", PublicKey: "b0b1eeee", Name: "B1"}
nodeB2 := nodeInfo{Role: "repeater", PublicKey: "b0c2ffff", Name: "B2"}
pm := buildPrefixMap([]nodeInfo{nodeA, nodeB1, nodeB2})
@@ -175,7 +175,7 @@ func TestResolveAmbiguousEdges_FailsNoChange(t *testing.T) {
// Test 3 (corrected): Resolution fails when prefix has no candidates in prefix map.
func TestResolveAmbiguousEdges_NoMatch(t *testing.T) {
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA"}
nodeA := nodeInfo{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeA"}
// pm has no entries matching prefix "zz"
pm := buildPrefixMap([]nodeInfo{nodeA})
@@ -215,8 +215,8 @@ func TestResolveAmbiguousEdges_NoMatch(t *testing.T) {
// Test 6: Phase 1 edge collection unchanged (no regression).
func TestPhase1EdgeCollection_Unchanged(t *testing.T) {
// Build a simple graph and verify non-ambiguous edges are not touched.
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{PublicKey: "bbbb2222", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
nodeA := nodeInfo{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{Role: "repeater", PublicKey: "bbbb2222", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
ts := time.Now().UTC().Format(time.RFC3339)
payloadType := 4
@@ -232,7 +232,7 @@ func TestPhase1EdgeCollection_Unchanged(t *testing.T) {
Observations: obs,
}
store := ngTestStore([]nodeInfo{nodeA, nodeB, {PublicKey: "cccc3333", Name: "Observer"}}, []*StoreTx{tx})
store := ngTestStore([]nodeInfo{nodeA, nodeB, {Role: "repeater", PublicKey: "cccc3333", Name: "Observer"}}, []*StoreTx{tx})
graph := BuildFromStore(store)
edges := graph.Neighbors("aaaa1111")
@@ -255,8 +255,8 @@ func TestPhase1EdgeCollection_Unchanged(t *testing.T) {
// Test 7: Merge preserves higher LastSeen timestamp.
func TestResolveAmbiguousEdges_PreservesHigherLastSeen(t *testing.T) {
nodeA := nodeInfo{PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{PublicKey: "b0b1eeee", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
nodeA := nodeInfo{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{Role: "repeater", PublicKey: "b0b1eeee", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
pm := buildPrefixMap([]nodeInfo{nodeA, nodeB})
graph := NewNeighborGraph()
@@ -307,10 +307,10 @@ func TestResolveAmbiguousEdges_PreservesHigherLastSeen(t *testing.T) {
// Test 5: Integration — node with both 1-byte and 2-byte prefix observations shows single entry.
func TestIntegration_DualPrefixSingleNeighbor(t *testing.T) {
nodeA := nodeInfo{PublicKey: "aaaa1111aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{PublicKey: "b0b1eeeeb0b1eeee", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
nodeB2 := nodeInfo{PublicKey: "b0c2ffffb0c2ffff", Name: "NodeB2", HasGPS: true, Lat: 10.0, Lon: 10.0}
observer := nodeInfo{PublicKey: "cccc3333cccc3333", Name: "Observer"}
nodeA := nodeInfo{Role: "repeater", PublicKey: "aaaa1111aaaa1111", Name: "NodeA", HasGPS: true, Lat: 45.0, Lon: -122.0}
nodeB := nodeInfo{Role: "repeater", PublicKey: "b0b1eeeeb0b1eeee", Name: "NodeB", HasGPS: true, Lat: 45.1, Lon: -122.1}
nodeB2 := nodeInfo{Role: "repeater", PublicKey: "b0c2ffffb0c2ffff", Name: "NodeB2", HasGPS: true, Lat: 10.0, Lon: 10.0}
observer := nodeInfo{Role: "repeater", PublicKey: "cccc3333cccc3333", Name: "Observer"}
ts := time.Now().UTC().Format(time.RFC3339)
pt := 4
+63 -63
View File
@@ -86,9 +86,9 @@ func TestBuildNeighborGraph_EmptyStore(t *testing.T) {
func TestBuildNeighborGraph_AdvertSingleHopPath(t *testing.T) {
// ADVERT from X, path=["R1_prefix"] → edges: X↔R1 and Observer↔R1
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "r1aabbcc", Name: "R1"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa"]`, nowStr, ngFloatPtr(-10)),
@@ -132,10 +132,10 @@ func TestBuildNeighborGraph_AdvertSingleHopPath(t *testing.T) {
func TestBuildNeighborGraph_AdvertMultiHopPath(t *testing.T) {
// ADVERT from X, path=["R1","R2"] → X↔R1 and Observer↔R2
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "r2ddeeff", Name: "R2"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "r1aabbcc", Name: "R1"},
{Role: "repeater", PublicKey: "r2ddeeff", Name: "R2"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa","r2dd"]`, nowStr, nil),
@@ -170,8 +170,8 @@ func TestBuildNeighborGraph_AdvertMultiHopPath(t *testing.T) {
func TestBuildNeighborGraph_AdvertZeroHop(t *testing.T) {
// ADVERT from X, path=[] → X↔Observer direct edge
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `[]`, nowStr, nil),
@@ -195,8 +195,8 @@ func TestBuildNeighborGraph_AdvertZeroHop(t *testing.T) {
func TestBuildNeighborGraph_NonAdvertEmptyPath(t *testing.T) {
// Non-ADVERT, path=[] → no edges
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 2, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `[]`, nowStr, nil),
@@ -212,10 +212,10 @@ func TestBuildNeighborGraph_NonAdvertEmptyPath(t *testing.T) {
func TestBuildNeighborGraph_NonAdvertOnlyObserverEdge(t *testing.T) {
// Non-ADVERT with path=["R1","R2"] → only Observer↔R2, NO originator edge
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "r2ddeeff", Name: "R2"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "r1aabbcc", Name: "R1"},
{Role: "repeater", PublicKey: "r2ddeeff", Name: "R2"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 2, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa","r2dd"]`, nowStr, nil),
@@ -236,9 +236,9 @@ func TestBuildNeighborGraph_NonAdvertOnlyObserverEdge(t *testing.T) {
func TestBuildNeighborGraph_NonAdvertSingleHop(t *testing.T) {
// Non-ADVERT with path=["R1"] → Observer↔R1 only
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "r1aabbcc", Name: "R1"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 2, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa"]`, nowStr, nil),
@@ -259,10 +259,10 @@ func TestBuildNeighborGraph_NonAdvertSingleHop(t *testing.T) {
func TestBuildNeighborGraph_HashCollision(t *testing.T) {
// Two nodes share prefix "a3" → ambiguous edge
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "a3bb1111", Name: "CandidateA"},
{PublicKey: "a3bb2222", Name: "CandidateB"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "a3bb1111", Name: "CandidateA"},
{Role: "repeater", PublicKey: "a3bb2222", Name: "CandidateB"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["a3bb"]`, nowStr, nil),
@@ -308,13 +308,13 @@ func TestBuildNeighborGraph_ConfidenceAutoResolve(t *testing.T) {
// CandidateB has no known neighbors (Jaccard = 0).
// An ambiguous edge X↔prefix "a3" with candidates [A, B] should auto-resolve to A.
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "n1111111", Name: "N1"},
{PublicKey: "n2222222", Name: "N2"},
{PublicKey: "n3333333", Name: "N3"},
{PublicKey: "a3001111", Name: "CandidateA"},
{PublicKey: "a3002222", Name: "CandidateB"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "n1111111", Name: "N1"},
{Role: "repeater", PublicKey: "n2222222", Name: "N2"},
{Role: "repeater", PublicKey: "n3333333", Name: "N3"},
{Role: "repeater", PublicKey: "a3001111", Name: "CandidateA"},
{Role: "repeater", PublicKey: "a3002222", Name: "CandidateB"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
// Create resolved edges: X↔N1, X↔N2, X↔N3, A↔N1, A↔N2, A↔N3
@@ -373,11 +373,11 @@ func TestBuildNeighborGraph_ConfidenceAutoResolve(t *testing.T) {
func TestBuildNeighborGraph_EqualScoresAmbiguous(t *testing.T) {
// Two candidates with identical neighbor sets → should NOT auto-resolve.
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "n1111111", Name: "N1"},
{PublicKey: "a3001111", Name: "CandidateA"},
{PublicKey: "a3002222", Name: "CandidateB"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "n1111111", Name: "N1"},
{Role: "repeater", PublicKey: "a3001111", Name: "CandidateA"},
{Role: "repeater", PublicKey: "a3002222", Name: "CandidateB"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
var txs []*StoreTx
@@ -425,8 +425,8 @@ func TestBuildNeighborGraph_EqualScoresAmbiguous(t *testing.T) {
func TestBuildNeighborGraph_ObserverSelfEdgeGuard(t *testing.T) {
// Observer's own prefix in path → should NOT create self-edge.
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["obs0"]`, nowStr, nil),
@@ -445,8 +445,8 @@ func TestBuildNeighborGraph_ObserverSelfEdgeGuard(t *testing.T) {
func TestBuildNeighborGraph_OrphanPrefix(t *testing.T) {
// Path contains prefix matching zero nodes → edge recorded as unresolved.
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["ff99"]`, nowStr, nil),
@@ -506,9 +506,9 @@ func TestAffinityScore_StaleAndLow(t *testing.T) {
func TestBuildNeighborGraph_CountAccumulation(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "r1aabbcc", Name: "R1"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
var txs []*StoreTx
@@ -535,10 +535,10 @@ func TestBuildNeighborGraph_CountAccumulation(t *testing.T) {
func TestBuildNeighborGraph_MultipleObservers(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Obs1"},
{PublicKey: "obs00002", Name: "Obs2"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "r1aabbcc", Name: "R1"},
{Role: "repeater", PublicKey: "obs00001", Name: "Obs1"},
{Role: "repeater", PublicKey: "obs00002", Name: "Obs2"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
@@ -565,9 +565,9 @@ func TestBuildNeighborGraph_MultipleObservers(t *testing.T) {
func TestBuildNeighborGraph_TimeDecayOldObservations(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "r1aabbcc", Name: "R1"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngFromNodeJSON("aaaa1111"), []*StoreObs{
@@ -592,10 +592,10 @@ func TestBuildNeighborGraph_TimeDecayOldObservations(t *testing.T) {
func TestBuildNeighborGraph_ADVERTOnlyConstraint(t *testing.T) {
// Non-ADVERT: should NOT create originator↔path[0] edge, only observer↔path[last].
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeX"},
{PublicKey: "r1aabbcc", Name: "R1"},
{PublicKey: "r2ddeeff", Name: "R2"},
{PublicKey: "obs00001", Name: "Observer"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeX"},
{Role: "repeater", PublicKey: "r1aabbcc", Name: "R1"},
{Role: "repeater", PublicKey: "r2ddeeff", Name: "R2"},
{Role: "repeater", PublicKey: "obs00001", Name: "Observer"},
}
tx := ngMakeTx(1, 2, ngFromNodeJSON("aaaa1111"), []*StoreObs{
ngMakeObs("obs00001", `["r1aa","r2dd"]`, nowStr, nil),
@@ -631,9 +631,9 @@ func ngPubKeyJSON(pubkey string) string {
func TestBuildNeighborGraph_AdvertPubKeyField(t *testing.T) {
// Real ADVERTs use "pubKey", not "from_node". Verify the builder handles it.
nodes := []nodeInfo{
{PublicKey: "99bf37abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234", Name: "Originator"},
{PublicKey: "r1aabbccdd001122334455667788990011223344556677889900112233445566", Name: "R1"},
{PublicKey: "obs0000100112233445566778899001122334455667788990011223344556677", Name: "Observer"},
{Role: "repeater", PublicKey: "99bf37abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234", Name: "Originator"},
{Role: "repeater", PublicKey: "r1aabbccdd001122334455667788990011223344556677889900112233445566", Name: "R1"},
{Role: "repeater", PublicKey: "obs0000100112233445566778899001122334455667788990011223344556677", Name: "Observer"},
}
tx := ngMakeTx(1, 4, ngPubKeyJSON("99bf37abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234"), []*StoreObs{
ngMakeObs("obs0000100112233445566778899001122334455667788990011223344556677", `["r1"]`, nowStr, ngFloatPtr(-8.5)),
@@ -666,10 +666,10 @@ func TestBuildNeighborGraph_OneByteHashPrefixes(t *testing.T) {
// Real-world scenario: 1-byte hash prefixes with multiple candidates.
// Should create edges (possibly ambiguous) rather than empty graph.
nodes := []nodeInfo{
{PublicKey: "c0dedad400000000000000000000000000000000000000000000000000000001", Name: "NodeC0-1"},
{PublicKey: "c0dedad900000000000000000000000000000000000000000000000000000002", Name: "NodeC0-2"},
{PublicKey: "a3bbccdd00000000000000000000000000000000000000000000000000000003", Name: "Originator"},
{PublicKey: "obs1234500000000000000000000000000000000000000000000000000000004", Name: "Observer"},
{Role: "repeater", PublicKey: "c0dedad400000000000000000000000000000000000000000000000000000001", Name: "NodeC0-1"},
{Role: "repeater", PublicKey: "c0dedad900000000000000000000000000000000000000000000000000000002", Name: "NodeC0-2"},
{Role: "repeater", PublicKey: "a3bbccdd00000000000000000000000000000000000000000000000000000003", Name: "Originator"},
{Role: "repeater", PublicKey: "obs1234500000000000000000000000000000000000000000000000000000004", Name: "Observer"},
}
// ADVERT from Originator with 1-byte path hop "c0"
tx := ngMakeTx(1, 4, ngPubKeyJSON("a3bbccdd00000000000000000000000000000000000000000000000000000003"), []*StoreObs{
@@ -809,10 +809,10 @@ func TestExtractFromNode_UsesCachedParse(t *testing.T) {
func BenchmarkBuildFromStore(b *testing.B) {
// Simulate a dataset with many packets and repeated pubkeys
nodes := []nodeInfo{
{PublicKey: "aaaa1111", Name: "NodeA"},
{PublicKey: "bbbb2222", Name: "NodeB"},
{PublicKey: "cccc3333", Name: "NodeC"},
{PublicKey: "dddd4444", Name: "NodeD"},
{Role: "repeater", PublicKey: "aaaa1111", Name: "NodeA"},
{Role: "repeater", PublicKey: "bbbb2222", Name: "NodeB"},
{Role: "repeater", PublicKey: "cccc3333", Name: "NodeC"},
{Role: "repeater", PublicKey: "dddd4444", Name: "NodeD"},
}
const numPackets = 1000
packets := make([]*StoreTx, 0, numPackets)
+7 -7
View File
@@ -38,7 +38,7 @@ func createTestDBWithSchema(t *testing.T) (*DB, string) {
observer_id TEXT, observer_name TEXT, direction TEXT,
snr REAL, rssi REAL, score INTEGER,
path_json TEXT, timestamp TEXT,
resolved_path TEXT
resolved_path TEXT, raw_hex TEXT
)`)
conn.Exec(`CREATE TABLE nodes (
public_key TEXT PRIMARY KEY, name TEXT, role TEXT,
@@ -58,8 +58,8 @@ func createTestDBWithSchema(t *testing.T) (*DB, string) {
func TestResolvePathForObs(t *testing.T) {
// Build a prefix map with known nodes
nodes := []nodeInfo{
{PublicKey: "aabbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb", Name: "Node-AA"},
{PublicKey: "bbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb11", Name: "Node-BB"},
{Role: "repeater", PublicKey: "aabbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb", Name: "Node-AA"},
{Role: "repeater", PublicKey: "bbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb11", Name: "Node-BB"},
}
pm := buildPrefixMap(nodes)
graph := NewNeighborGraph()
@@ -97,7 +97,7 @@ func TestResolvePathForObs_EmptyPath(t *testing.T) {
func TestResolvePathForObs_Unresolvable(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aabbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb", Name: "Node-AA"},
{Role: "repeater", PublicKey: "aabbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb", Name: "Node-AA"},
}
pm := buildPrefixMap(nodes)
@@ -264,7 +264,7 @@ func TestEnsureResolvedPathColumn(t *testing.T) {
conn, _ := sql.Open("sqlite", "file:"+dbPath+"?_journal_mode=WAL")
conn.Exec(`CREATE TABLE observations (
id INTEGER PRIMARY KEY, transmission_id INTEGER,
observer_id TEXT, path_json TEXT, timestamp TEXT
observer_id TEXT, path_json TEXT, timestamp TEXT, raw_hex TEXT
)`)
conn.Close()
@@ -437,8 +437,8 @@ func TestExtractEdgesFromObs_NonAdvertNoPath(t *testing.T) {
func TestExtractEdgesFromObs_WithPath(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "aabbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb", Name: "Node-AA"},
{PublicKey: "ffgghhii1234567890aabbccddee1234567890aabbccddee1234567890aabb11", Name: "Node-FF"},
{Role: "repeater", PublicKey: "aabbccddee1234567890aabbccddee1234567890aabbccddee1234567890aabb", Name: "Node-AA"},
{Role: "repeater", PublicKey: "ffgghhii1234567890aabbccddee1234567890aabbccddee1234567890aabb11", Name: "Node-FF"},
}
pm := buildPrefixMap(nodes)
+427
View File
@@ -0,0 +1,427 @@
package main
import (
"encoding/hex"
"encoding/json"
"math"
"net/http"
"sort"
"strings"
"time"
)
// ─── Path Inspector ────────────────────────────────────────────────────────────
// POST /api/paths/inspect — beam-search scorer for prefix path candidates.
// Spec: issue #944 §2.12.5.
// pathInspectRequest is the JSON body for the inspect endpoint.
type pathInspectRequest struct {
Prefixes []string `json:"prefixes"`
Context *pathInspectContext `json:"context,omitempty"`
Limit int `json:"limit,omitempty"`
}
type pathInspectContext struct {
ObserverID string `json:"observerId,omitempty"`
Since string `json:"since,omitempty"`
Until string `json:"until,omitempty"`
}
// pathCandidate is one scored candidate path in the response.
type pathCandidate struct {
Path []string `json:"path"`
Names []string `json:"names"`
Score float64 `json:"score"`
Speculative bool `json:"speculative"`
Evidence pathEvidence `json:"evidence"`
}
type pathEvidence struct {
PerHop []hopEvidence `json:"perHop"`
}
type hopEvidence struct {
Prefix string `json:"prefix"`
CandidatesConsidered int `json:"candidatesConsidered"`
Chosen string `json:"chosen"`
EdgeWeight float64 `json:"edgeWeight"`
Alternatives []hopAlternative `json:"alternatives,omitempty"`
}
// hopAlternative shows a candidate that was considered but not chosen for this hop.
type hopAlternative struct {
PublicKey string `json:"publicKey"`
Name string `json:"name"`
Score float64 `json:"score"`
}
type pathInspectResponse struct {
Candidates []pathCandidate `json:"candidates"`
Input map[string]interface{} `json:"input"`
Stats map[string]interface{} `json:"stats"`
}
// beamEntry represents a partial path being extended during beam search.
type beamEntry struct {
pubkeys []string
names []string
evidence []hopEvidence
score float64 // product of per-hop scores (pre-geometric-mean)
}
const (
beamWidth = 20
maxInputHops = 64
maxPrefixBytes = 3
maxRequestItems = 64
geoMaxKm = 50.0
hopScoreFloor = 0.05
speculativeThreshold = 0.7
inspectCacheTTL = 30 * time.Second
inspectBodyLimit = 4096
)
// Weights per spec §2.3.
const (
wEdge = 0.35
wGeo = 0.20
wRecency = 0.15
wSelectivity = 0.30
)
func (s *Server) handlePathInspect(w http.ResponseWriter, r *http.Request) {
// Body limit per spec §2.1.
r.Body = http.MaxBytesReader(w, r.Body, inspectBodyLimit)
var req pathInspectRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, `{"error":"invalid JSON"}`, http.StatusBadRequest)
return
}
// Validate prefixes.
if len(req.Prefixes) == 0 {
http.Error(w, `{"error":"prefixes required"}`, http.StatusBadRequest)
return
}
if len(req.Prefixes) > maxRequestItems {
http.Error(w, `{"error":"too many prefixes (max 64)"}`, http.StatusBadRequest)
return
}
// Normalize + validate each prefix.
prefixByteLen := -1
for i, p := range req.Prefixes {
p = strings.ToLower(strings.TrimSpace(p))
req.Prefixes[i] = p
if len(p) == 0 || len(p)%2 != 0 {
http.Error(w, `{"error":"prefixes must be even-length hex"}`, http.StatusBadRequest)
return
}
if _, err := hex.DecodeString(p); err != nil {
http.Error(w, `{"error":"prefixes must be valid hex"}`, http.StatusBadRequest)
return
}
byteLen := len(p) / 2
if byteLen > maxPrefixBytes {
http.Error(w, `{"error":"prefix exceeds 3 bytes"}`, http.StatusBadRequest)
return
}
if prefixByteLen == -1 {
prefixByteLen = byteLen
} else if byteLen != prefixByteLen {
http.Error(w, `{"error":"mixed prefix lengths not allowed"}`, http.StatusBadRequest)
return
}
}
limit := req.Limit
if limit <= 0 {
limit = 10
}
if limit > 50 {
limit = 50
}
// Check cache.
cacheKey := s.store.inspectCacheKey(req)
s.store.inspectMu.RLock()
if cached, ok := s.store.inspectCache[cacheKey]; ok && time.Now().Before(cached.expiresAt) {
s.store.inspectMu.RUnlock()
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(cached.data)
return
}
s.store.inspectMu.RUnlock()
// Snapshot data under read lock.
nodes, pm := s.store.getCachedNodesAndPM()
// Build pubkey→nodeInfo map for O(1) geo lookup in scorer.
nodeByPK := make(map[string]*nodeInfo, len(nodes))
for i := range nodes {
nodeByPK[strings.ToLower(nodes[i].PublicKey)] = &nodes[i]
}
// Get neighbor graph; handle cold start.
graph := s.store.graph
if graph == nil || graph.IsStale() {
rebuilt := make(chan struct{})
go func() {
s.store.ensureNeighborGraph()
close(rebuilt)
}()
select {
case <-rebuilt:
graph = s.store.graph
case <-time.After(2 * time.Second):
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
json.NewEncoder(w).Encode(map[string]interface{}{"retry": true})
return
}
if graph == nil {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
json.NewEncoder(w).Encode(map[string]interface{}{"retry": true})
return
}
}
now := time.Now()
start := now
// Beam search.
beam := s.store.beamSearch(req.Prefixes, pm, graph, nodeByPK, now)
// Sort by score descending, take top limit.
sortBeam(beam)
if len(beam) > limit {
beam = beam[:limit]
}
// Build response with per-hop alternatives (spec §2.7, M2 fix).
candidates := make([]pathCandidate, 0, len(beam))
for _, entry := range beam {
nHops := len(entry.pubkeys)
var score float64
if nHops > 0 {
score = math.Pow(entry.score, 1.0/float64(nHops))
}
// Populate per-hop alternatives: other candidates at each hop that weren't chosen.
evidence := make([]hopEvidence, len(entry.evidence))
copy(evidence, entry.evidence)
for hi, ev := range evidence {
if hi >= len(req.Prefixes) {
break
}
prefix := req.Prefixes[hi]
allCands := pm.m[prefix]
var alts []hopAlternative
for _, c := range allCands {
if !canAppearInPath(c.Role) || c.PublicKey == ev.Chosen {
continue
}
// Score this alternative in context of the partial path up to this hop.
var partialEntry beamEntry
if hi > 0 {
partialEntry = beamEntry{pubkeys: entry.pubkeys[:hi], names: entry.names[:hi], score: 1.0}
}
altScore := s.store.scoreHop(partialEntry, c, ev.CandidatesConsidered, graph, nodeByPK, now, hi)
alts = append(alts, hopAlternative{PublicKey: c.PublicKey, Name: c.Name, Score: math.Round(altScore*1000) / 1000})
}
// Sort alts by score desc, cap at 5.
sort.Slice(alts, func(i, j int) bool { return alts[i].Score > alts[j].Score })
if len(alts) > 5 {
alts = alts[:5]
}
evidence[hi] = hopEvidence{
Prefix: ev.Prefix,
CandidatesConsidered: ev.CandidatesConsidered,
Chosen: ev.Chosen,
EdgeWeight: ev.EdgeWeight,
Alternatives: alts,
}
}
candidates = append(candidates, pathCandidate{
Path: entry.pubkeys,
Names: entry.names,
Score: math.Round(score*1000) / 1000,
Speculative: score < speculativeThreshold,
Evidence: pathEvidence{PerHop: evidence},
})
}
elapsed := time.Since(start).Milliseconds()
resp := pathInspectResponse{
Candidates: candidates,
Input: map[string]interface{}{
"prefixes": req.Prefixes,
"hops": len(req.Prefixes),
},
Stats: map[string]interface{}{
"beamWidth": beamWidth,
"expansionsRun": len(req.Prefixes) * beamWidth,
"elapsedMs": elapsed,
},
}
// Cache result (and evict stale entries).
s.store.inspectMu.Lock()
if s.store.inspectCache == nil {
s.store.inspectCache = make(map[string]*inspectCachedResult)
}
now2 := time.Now()
for k, v := range s.store.inspectCache {
if now2.After(v.expiresAt) {
delete(s.store.inspectCache, k)
}
}
s.store.inspectCache[cacheKey] = &inspectCachedResult{
data: resp,
expiresAt: now2.Add(inspectCacheTTL),
}
s.store.inspectMu.Unlock()
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
}
type inspectCachedResult struct {
data pathInspectResponse
expiresAt time.Time
}
func (s *PacketStore) inspectCacheKey(req pathInspectRequest) string {
key := strings.Join(req.Prefixes, ",")
if req.Context != nil {
key += "|" + req.Context.ObserverID + "|" + req.Context.Since + "|" + req.Context.Until
}
return key
}
func (s *PacketStore) beamSearch(prefixes []string, pm *prefixMap, graph *NeighborGraph, nodeByPK map[string]*nodeInfo, now time.Time) []beamEntry {
// Start with empty beam.
beam := []beamEntry{{pubkeys: nil, names: nil, evidence: nil, score: 1.0}}
for hopIdx, prefix := range prefixes {
candidates := pm.m[prefix]
// Filter by role at lookup time (spec §2.2 step 2).
var filtered []nodeInfo
for _, c := range candidates {
if canAppearInPath(c.Role) {
filtered = append(filtered, c)
}
}
candidateCount := len(filtered)
if candidateCount == 0 {
// No candidates for this hop — beam dies.
return nil
}
var nextBeam []beamEntry
for _, entry := range beam {
for _, cand := range filtered {
hopScore := s.scoreHop(entry, cand, candidateCount, graph, nodeByPK, now, hopIdx)
if hopScore < hopScoreFloor {
hopScore = hopScoreFloor
}
newEntry := beamEntry{
pubkeys: append(append([]string{}, entry.pubkeys...), cand.PublicKey),
names: append(append([]string{}, entry.names...), cand.Name),
evidence: append(append([]hopEvidence{}, entry.evidence...), hopEvidence{
Prefix: prefix,
CandidatesConsidered: candidateCount,
Chosen: cand.PublicKey,
EdgeWeight: hopScore,
}),
score: entry.score * hopScore,
}
nextBeam = append(nextBeam, newEntry)
}
}
// Prune to beam width.
sortBeam(nextBeam)
if len(nextBeam) > beamWidth {
nextBeam = nextBeam[:beamWidth]
}
beam = nextBeam
}
return beam
}
func (s *PacketStore) scoreHop(entry beamEntry, cand nodeInfo, candidateCount int, graph *NeighborGraph, nodeByPK map[string]*nodeInfo, now time.Time, hopIdx int) float64 {
var edgeScore float64
var geoScore float64 = 1.0
var recencyScore float64 = 1.0
if hopIdx == 0 || len(entry.pubkeys) == 0 {
// First hop: no prior node to compare against.
edgeScore = 1.0
} else {
lastPK := entry.pubkeys[len(entry.pubkeys)-1]
// Single scan over neighbors for both edge weight and recency.
edges := graph.Neighbors(lastPK)
var foundEdge *NeighborEdge
for _, e := range edges {
peer := e.NodeA
if strings.EqualFold(peer, lastPK) {
peer = e.NodeB
}
if strings.EqualFold(peer, cand.PublicKey) {
foundEdge = e
break
}
}
if foundEdge != nil {
edgeScore = foundEdge.Score(now)
hoursSince := now.Sub(foundEdge.LastSeen).Hours()
if hoursSince <= 24 {
recencyScore = 1.0
} else {
recencyScore = math.Max(0.1, 24.0/hoursSince)
}
} else {
edgeScore = 0
recencyScore = 0
}
// Geographic plausibility.
prevNode := nodeByPK[strings.ToLower(lastPK)]
if prevNode != nil && prevNode.HasGPS && cand.HasGPS {
dist := haversineKm(prevNode.Lat, prevNode.Lon, cand.Lat, cand.Lon)
if dist > geoMaxKm {
geoScore = math.Max(0.1, geoMaxKm/dist)
}
}
}
// Prefix selectivity.
selectivityScore := 1.0 / float64(candidateCount)
return wEdge*edgeScore + wGeo*geoScore + wRecency*recencyScore + wSelectivity*selectivityScore
}
func sortBeam(beam []beamEntry) {
sort.Slice(beam, func(i, j int) bool {
return beam[i].score > beam[j].score
})
}
// ensureNeighborGraph triggers a graph rebuild if nil or stale.
func (s *PacketStore) ensureNeighborGraph() {
if s.graph != nil && !s.graph.IsStale() {
return
}
g := BuildFromStore(s)
s.graph = g
}
+308
View File
@@ -0,0 +1,308 @@
package main
import (
"bytes"
"encoding/json"
"fmt"
"math"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
// ─── Unit tests for path inspector (issue #944) ────────────────────────────────
func TestScoreHop_EdgeWeight(t *testing.T) {
store := &PacketStore{}
graph := NewNeighborGraph()
now := time.Now()
// Add an edge between A and B.
graph.mu.Lock()
edge := &NeighborEdge{
NodeA: "aaaa", NodeB: "bbbb",
Count: 50, LastSeen: now.Add(-1 * time.Hour),
Observers: map[string]bool{"obs1": true},
}
key := edgeKey{"aaaa", "bbbb"}
graph.edges[key] = edge
graph.byNode["aaaa"] = append(graph.byNode["aaaa"], edge)
graph.byNode["bbbb"] = append(graph.byNode["bbbb"], edge)
graph.mu.Unlock()
entry := beamEntry{pubkeys: []string{"aaaa"}, names: []string{"NodeA"}}
cand := nodeInfo{PublicKey: "bbbb", Name: "NodeB", Role: "repeater"}
score := store.scoreHop(entry, cand, 2, graph, nil, now, 1)
// With edge present, edgeScore > 0. With 2 candidates, selectivity = 0.5.
// Anti-tautology: if we zero out edge weight constant, score would change.
if score <= 0.05 {
t.Errorf("expected score > floor, got %f", score)
}
// No edge: score should be lower.
candNoEdge := nodeInfo{PublicKey: "cccc", Name: "NodeC", Role: "repeater"}
scoreNoEdge := store.scoreHop(entry, candNoEdge, 2, graph, nil, now, 1)
if scoreNoEdge >= score {
t.Errorf("expected no-edge score (%f) < edge score (%f)", scoreNoEdge, score)
}
}
func TestScoreHop_FirstHop(t *testing.T) {
store := &PacketStore{}
graph := NewNeighborGraph()
now := time.Now()
entry := beamEntry{pubkeys: nil, names: nil}
cand := nodeInfo{PublicKey: "aaaa", Name: "NodeA", Role: "repeater"}
score := store.scoreHop(entry, cand, 3, graph, nil, now, 0)
// First hop: edgeScore=1.0, geoScore=1.0, recencyScore=1.0, selectivity=1/3
// = 0.35*1 + 0.20*1 + 0.15*1 + 0.30*(1/3) = 0.35+0.20+0.15+0.10 = 0.80
expected := 0.35 + 0.20 + 0.15 + 0.30/3.0
if score < expected-0.01 || score > expected+0.01 {
t.Errorf("expected ~%f, got %f", expected, score)
}
}
func TestScoreHop_GeoPlausibility(t *testing.T) {
store := &PacketStore{}
store.nodeCache = []nodeInfo{
{PublicKey: "aaaa", Name: "A", Role: "repeater", Lat: 37.0, Lon: -122.0, HasGPS: true},
{PublicKey: "bbbb", Name: "B", Role: "repeater", Lat: 37.01, Lon: -122.01, HasGPS: true}, // ~1.4km
{PublicKey: "cccc", Name: "C", Role: "repeater", Lat: 40.0, Lon: -120.0, HasGPS: true}, // ~400km
}
store.nodePM = buildPrefixMap(store.nodeCache)
store.nodeCacheTime = time.Now()
graph := NewNeighborGraph()
now := time.Now()
nodeByPK := map[string]*nodeInfo{
"aaaa": &store.nodeCache[0],
"bbbb": &store.nodeCache[1],
"cccc": &store.nodeCache[2],
}
entry := beamEntry{pubkeys: []string{"aaaa"}, names: []string{"A"}}
// Close node should score higher than far node (geo component).
scoreClose := store.scoreHop(entry, store.nodeCache[1], 2, graph, nodeByPK, now, 1)
scoreFar := store.scoreHop(entry, store.nodeCache[2], 2, graph, nodeByPK, now, 1)
if scoreFar >= scoreClose {
t.Errorf("expected far node score (%f) < close node score (%f)", scoreFar, scoreClose)
}
}
func TestBeamSearch_WidthCap(t *testing.T) {
store := &PacketStore{}
graph := NewNeighborGraph()
graph.builtAt = time.Now()
now := time.Now()
// Create 25 nodes that all match prefix "aa".
var nodes []nodeInfo
for i := 0; i < 25; i++ {
// Each node has pubkey starting with "aa" followed by unique hex.
pk := "aa" + strings.Repeat("0", 4) + fmt.Sprintf("%02x", i)
nodes = append(nodes, nodeInfo{PublicKey: pk, Name: pk, Role: "repeater"})
}
pm := buildPrefixMap(nodes)
// Two hops of "aa" — should produce 25*25=625 combos, pruned to 20.
beam := store.beamSearch([]string{"aa", "aa"}, pm, graph, nil, now)
if len(beam) > beamWidth {
t.Errorf("beam exceeded width: got %d, want <= %d", len(beam), beamWidth)
}
// Anti-tautology: without beam pruning, we'd have up to 25*min(25,beamWidth)=500 entries.
// The test verifies pruning is effective.
}
func TestBeamSearch_Speculative(t *testing.T) {
store := &PacketStore{}
graph := NewNeighborGraph()
graph.builtAt = time.Now()
now := time.Now()
// Create nodes with no edges and multiple candidates — should result in low scores (speculative).
nodes := []nodeInfo{
{PublicKey: "aabb", Name: "N1", Role: "repeater"},
{PublicKey: "aabb22", Name: "N1b", Role: "repeater"},
{PublicKey: "ccdd", Name: "N2", Role: "repeater"},
{PublicKey: "ccdd22", Name: "N2b", Role: "repeater"},
{PublicKey: "ccdd33", Name: "N2c", Role: "repeater"},
}
pm := buildPrefixMap(nodes)
beam := store.beamSearch([]string{"aa", "cc"}, pm, graph, nil, now)
if len(beam) == 0 {
t.Fatal("expected at least one result")
}
// Score should be < 0.7 since there's no edge and multiple candidates (speculative).
nHops := len(beam[0].pubkeys)
score := 1.0
if nHops > 0 {
product := beam[0].score
score = pow(product, 1.0/float64(nHops))
}
if score >= speculativeThreshold {
t.Errorf("expected speculative score (< %f), got %f", speculativeThreshold, score)
}
}
func TestHandlePathInspect_EmptyPrefixes(t *testing.T) {
srv := newTestServerForInspect(t)
body := `{"prefixes":[]}`
rr := doInspectRequest(srv, body)
if rr.Code != http.StatusBadRequest {
t.Errorf("expected 400, got %d", rr.Code)
}
}
func TestHandlePathInspect_OddLengthPrefix(t *testing.T) {
srv := newTestServerForInspect(t)
body := `{"prefixes":["abc"]}`
rr := doInspectRequest(srv, body)
if rr.Code != http.StatusBadRequest {
t.Errorf("expected 400 for odd-length prefix, got %d", rr.Code)
}
}
func TestHandlePathInspect_MixedLengths(t *testing.T) {
srv := newTestServerForInspect(t)
body := `{"prefixes":["aa","bbcc"]}`
rr := doInspectRequest(srv, body)
if rr.Code != http.StatusBadRequest {
t.Errorf("expected 400 for mixed lengths, got %d", rr.Code)
}
}
func TestHandlePathInspect_TooLongPrefix(t *testing.T) {
srv := newTestServerForInspect(t)
body := `{"prefixes":["aabbccdd"]}`
rr := doInspectRequest(srv, body)
if rr.Code != http.StatusBadRequest {
t.Errorf("expected 400 for >3-byte prefix, got %d", rr.Code)
}
}
func TestHandlePathInspect_TooManyPrefixes(t *testing.T) {
srv := newTestServerForInspect(t)
prefixes := make([]string, 65)
for i := range prefixes {
prefixes[i] = "aa"
}
b, _ := json.Marshal(map[string]interface{}{"prefixes": prefixes})
rr := doInspectRequest(srv, string(b))
if rr.Code != http.StatusBadRequest {
t.Errorf("expected 400 for >64 prefixes, got %d", rr.Code)
}
}
func TestHandlePathInspect_ValidRequest(t *testing.T) {
srv := newTestServerForInspect(t)
// Seed nodes in the store — multiple candidates per prefix to lower selectivity.
srv.store.nodeCache = []nodeInfo{
{PublicKey: "aabb1234", Name: "NodeA", Role: "repeater", Lat: 37.0, Lon: -122.0, HasGPS: true},
{PublicKey: "aabb5678", Name: "NodeA2", Role: "repeater"},
{PublicKey: "ccdd5678", Name: "NodeB", Role: "repeater", Lat: 37.01, Lon: -122.01, HasGPS: true},
{PublicKey: "ccdd9999", Name: "NodeB2", Role: "repeater"},
{PublicKey: "ccdd1111", Name: "NodeB3", Role: "repeater"},
}
srv.store.nodePM = buildPrefixMap(srv.store.nodeCache)
srv.store.nodeCacheTime = time.Now()
srv.store.graph = NewNeighborGraph()
srv.store.graph.builtAt = time.Now()
body := `{"prefixes":["aa","cc"]}`
rr := doInspectRequest(srv, body)
if rr.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", rr.Code, rr.Body.String())
}
var resp pathInspectResponse
if err := json.Unmarshal(rr.Body.Bytes(), &resp); err != nil {
t.Fatalf("invalid JSON response: %v", err)
}
if len(resp.Candidates) == 0 {
t.Error("expected at least one candidate")
}
if resp.Candidates[0].Speculative != true {
// No edge between nodes, so score should be < 0.7.
t.Error("expected speculative=true for no-edge path")
}
}
// ─── Helpers ──────────────────────────────────────────────────────────────────
func newTestServerForInspect(t *testing.T) *Server {
t.Helper()
store := &PacketStore{
inspectCache: make(map[string]*inspectCachedResult),
}
store.graph = NewNeighborGraph()
store.graph.builtAt = time.Now()
return &Server{store: store}
}
func doInspectRequest(srv *Server, body string) *httptest.ResponseRecorder {
req := httptest.NewRequest("POST", "/api/paths/inspect", bytes.NewBufferString(body))
req.Header.Set("Content-Type", "application/json")
rr := httptest.NewRecorder()
srv.handlePathInspect(rr, req)
return rr
}
func pow(base, exp float64) float64 {
return math.Pow(base, exp)
}
// BenchmarkBeamSearch — performance proof for spec §2.5 (<100ms p99 for ≤64 hops).
// Anti-tautology: removing beam pruning makes this ~625x slower; timing assertion catches it.
func BenchmarkBeamSearch(b *testing.B) {
// Setup: 100 nodes, 10-hop prefix input, realistic neighbor graph.
// Anti-tautology: removing beam pruning makes this ~625x slower.
store := &PacketStore{}
pm := &prefixMap{m: make(map[string][]nodeInfo)}
graph := NewNeighborGraph()
nodes := make([]nodeInfo, 100)
now := time.Now()
for i := 0; i < 100; i++ {
pk := fmt.Sprintf("%064x", i)
prefix := fmt.Sprintf("%02x", i%256)
node := nodeInfo{PublicKey: pk, Name: fmt.Sprintf("Node%d", i), Role: "repeater", Lat: 37.0 + float64(i)*0.01, Lon: -122.0 + float64(i)*0.01}
nodes[i] = node
pm.m[prefix] = append(pm.m[prefix], node)
// Add neighbor edges to create a connected graph.
if i > 0 {
prevPK := fmt.Sprintf("%064x", i-1)
key := makeEdgeKey(prevPK, pk)
edge := &NeighborEdge{NodeA: prevPK, NodeB: pk, LastSeen: now, Count: 10}
graph.edges[key] = edge
graph.byNode[prevPK] = append(graph.byNode[prevPK], edge)
graph.byNode[pk] = append(graph.byNode[pk], edge)
}
}
// 10-hop input using prefixes that map to multiple candidates.
prefixes := make([]string, 10)
for i := 0; i < 10; i++ {
prefixes[i] = fmt.Sprintf("%02x", (i*3)%256)
}
nodeByPK := make(map[string]*nodeInfo)
for idx := range nodes {
nodeByPK[nodes[idx].PublicKey] = &nodes[idx]
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
store.beamSearch(prefixes, pm, graph, nodeByPK, now)
}
}
+212
View File
@@ -0,0 +1,212 @@
package main
import (
"encoding/json"
"testing"
)
func TestCanAppearInPath(t *testing.T) {
cases := []struct {
role string
want bool
}{
{"repeater", true},
{"Repeater", true},
{"REPEATER", true},
{"room_server", true},
{"Room_Server", true},
{"room", true},
{"companion", false},
{"sensor", false},
{"", false},
{"unknown", false},
}
for _, tc := range cases {
if got := canAppearInPath(tc.role); got != tc.want {
t.Errorf("canAppearInPath(%q) = %v, want %v", tc.role, got, tc.want)
}
}
}
func TestBuildPrefixMap_ExcludesCompanions(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "7a1234abcdef", Role: "companion", Name: "MyCompanion"},
}
pm := buildPrefixMap(nodes)
if len(pm.m) != 0 {
t.Fatalf("expected empty prefix map, got %d entries", len(pm.m))
}
}
func TestBuildPrefixMap_ExcludesSensors(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "7a1234abcdef", Role: "sensor", Name: "MySensor"},
}
pm := buildPrefixMap(nodes)
if len(pm.m) != 0 {
t.Fatalf("expected empty prefix map, got %d entries", len(pm.m))
}
}
func TestResolveWithContext_NilWhenOnlyCompanionMatchesPrefix(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "7a1234abcdef", Role: "companion", Name: "MyCompanion"},
}
pm := buildPrefixMap(nodes)
r, _, _ := pm.resolveWithContext("7a", nil, nil)
if r != nil {
t.Fatalf("expected nil, got %+v", r)
}
}
func TestResolveWithContext_NilWhenOnlySensorMatchesPrefix(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "7a1234abcdef", Role: "sensor", Name: "MySensor"},
}
pm := buildPrefixMap(nodes)
r, _, _ := pm.resolveWithContext("7a", nil, nil)
if r != nil {
t.Fatalf("expected nil for sensor-only prefix, got %+v", r)
}
}
func TestResolveWithContext_PrefersRepeaterOverCompanionAtSamePrefix(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "7a1234abcdef", Role: "companion", Name: "MyCompanion"},
{PublicKey: "7a5678901234", Role: "repeater", Name: "MyRepeater"},
}
pm := buildPrefixMap(nodes)
r, _, _ := pm.resolveWithContext("7a", nil, nil)
if r == nil {
t.Fatal("expected non-nil result")
}
if r.Name != "MyRepeater" {
t.Fatalf("expected MyRepeater, got %s", r.Name)
}
}
func TestResolveWithContext_PrefersRoomServerOverCompanionAtSamePrefix(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "ab1234abcdef", Role: "companion", Name: "MyCompanion"},
{PublicKey: "ab5678901234", Role: "room_server", Name: "MyRoom"},
}
pm := buildPrefixMap(nodes)
r, _, _ := pm.resolveWithContext("ab", nil, nil)
if r == nil {
t.Fatal("expected non-nil result")
}
if r.Name != "MyRoom" {
t.Fatalf("expected MyRoom, got %s", r.Name)
}
}
func TestResolve_NilWhenOnlyCompanionMatchesPrefix(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "7a1234abcdef", Role: "companion", Name: "MyCompanion"},
}
pm := buildPrefixMap(nodes)
r := pm.resolve("7a")
if r != nil {
t.Fatalf("expected nil from resolve() for companion-only prefix, got %+v", r)
}
}
func TestResolve_NilWhenOnlySensorMatchesPrefix(t *testing.T) {
nodes := []nodeInfo{
{PublicKey: "7a1234abcdef", Role: "sensor", Name: "MySensor"},
}
pm := buildPrefixMap(nodes)
r := pm.resolve("7a")
if r != nil {
t.Fatalf("expected nil from resolve() for sensor-only prefix, got %+v", r)
}
}
func TestResolveWithContext_PicksRepeaterEvenWhenCompanionHasGPS(t *testing.T) {
// Adversarial: companion has GPS, repeater doesn't. Role filter should
// exclude companion entirely, so repeater wins despite lacking GPS.
nodes := []nodeInfo{
{PublicKey: "7a1234abcdef", Role: "companion", Name: "GPSCompanion", Lat: 37.0, Lon: -122.0, HasGPS: true},
{PublicKey: "7a5678901234", Role: "repeater", Name: "NoGPSRepeater", Lat: 0, Lon: 0, HasGPS: false},
}
pm := buildPrefixMap(nodes)
r, _, _ := pm.resolveWithContext("7a", nil, nil)
if r == nil {
t.Fatal("expected non-nil result")
}
if r.Name != "NoGPSRepeater" {
t.Fatalf("expected NoGPSRepeater (role filter excludes companion), got %s", r.Name)
}
}
func TestComputeDistancesForTx_CompanionNeverInResolvedChain(t *testing.T) {
// Integration test: a path with a prefix matching both a companion and a
// repeater. The resolveHop function (using buildPrefixMap) should only
// return the repeater.
nodes := []nodeInfo{
{PublicKey: "7a1234abcdef", Role: "companion", Name: "BadCompanion", Lat: 37.0, Lon: -122.0, HasGPS: true},
{PublicKey: "7a5678901234", Role: "repeater", Name: "GoodRepeater", Lat: 38.0, Lon: -123.0, HasGPS: true},
{PublicKey: "bb1111111111", Role: "repeater", Name: "OtherRepeater", Lat: 39.0, Lon: -124.0, HasGPS: true},
}
pm := buildPrefixMap(nodes)
nodeByPk := make(map[string]*nodeInfo)
for i := range nodes {
nodeByPk[nodes[i].PublicKey] = &nodes[i]
}
repeaterSet := map[string]bool{
"7a5678901234": true,
"bb1111111111": true,
}
// Build a synthetic StoreTx with a path ["7a", "bb"] and a sender with GPS
senderPK := "cc0000000000"
sender := nodeInfo{PublicKey: senderPK, Role: "repeater", Name: "Sender", Lat: 36.0, Lon: -121.0, HasGPS: true}
nodeByPk[senderPK] = &sender
pathJSON, _ := json.Marshal([]string{"7a", "bb"})
decoded, _ := json.Marshal(map[string]interface{}{"pubKey": senderPK})
tx := &StoreTx{
PathJSON: string(pathJSON),
DecodedJSON: string(decoded),
FirstSeen: "2026-04-30T12:00",
}
resolveHop := func(hop string) *nodeInfo {
return pm.resolve(hop)
}
hops, pathRec := computeDistancesForTx(tx, nodeByPk, repeaterSet, resolveHop)
// Verify BadCompanion's pubkey never appears in hops
badPK := "7a1234abcdef"
for i, h := range hops {
if h.FromPk == badPK || h.ToPk == badPK {
t.Fatalf("hop[%d] contains BadCompanion pubkey: from=%s to=%s", i, h.FromPk, h.ToPk)
}
}
// Verify BadCompanion's pubkey never appears in pathRec
if pathRec == nil {
t.Fatal("expected non-nil path record (3 GPS nodes in chain)")
}
for i, hop := range pathRec.Hops {
if hop.FromPk == badPK || hop.ToPk == badPK {
t.Fatalf("pathRec.Hops[%d] contains BadCompanion pubkey: from=%s to=%s", i, hop.FromPk, hop.ToPk)
}
}
// Verify GoodRepeater IS in the chain (proves the prefix was resolved to the right node)
goodPK := "7a5678901234"
foundGood := false
for _, hop := range pathRec.Hops {
if hop.FromPk == goodPK || hop.ToPk == goodPK {
foundGood = true
break
}
}
if !foundGood {
t.Fatal("expected GoodRepeater (7a5678901234) in pathRec.Hops but not found")
}
}
+29 -29
View File
@@ -11,7 +11,7 @@ import (
func TestResolveWithContext_UniquePrefix(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1b2c3d4", Name: "Node-A", HasGPS: true, Lat: 1, Lon: 2},
{Role: "repeater", PublicKey: "a1b2c3d4", Name: "Node-A", HasGPS: true, Lat: 1, Lon: 2},
})
ni, confidence, _ := pm.resolveWithContext("a1b2c3d4", nil, nil)
if ni == nil || ni.Name != "Node-A" {
@@ -24,7 +24,7 @@ func TestResolveWithContext_UniquePrefix(t *testing.T) {
func TestResolveWithContext_NoMatch(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1b2c3d4", Name: "Node-A"},
{Role: "repeater", PublicKey: "a1b2c3d4", Name: "Node-A"},
})
ni, confidence, _ := pm.resolveWithContext("ff", nil, nil)
if ni != nil {
@@ -37,8 +37,8 @@ func TestResolveWithContext_NoMatch(t *testing.T) {
func TestResolveWithContext_AffinityWins(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "Node-A1"},
{PublicKey: "a1bbbbbb", Name: "Node-A2"},
{Role: "repeater", PublicKey: "a1aaaaaa", Name: "Node-A1"},
{Role: "repeater", PublicKey: "a1bbbbbb", Name: "Node-A2"},
})
graph := NewNeighborGraph()
@@ -60,9 +60,9 @@ func TestResolveWithContext_AffinityWins(t *testing.T) {
func TestResolveWithContext_AffinityTooClose_FallsToGeo(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "Node-A1", HasGPS: true, Lat: 10, Lon: 20},
{PublicKey: "a1bbbbbb", Name: "Node-A2", HasGPS: true, Lat: 11, Lon: 21},
{PublicKey: "c0c0c0c0", Name: "Ctx", HasGPS: true, Lat: 10.1, Lon: 20.1},
{Role: "repeater", PublicKey: "a1aaaaaa", Name: "Node-A1", HasGPS: true, Lat: 10, Lon: 20},
{Role: "repeater", PublicKey: "a1bbbbbb", Name: "Node-A2", HasGPS: true, Lat: 11, Lon: 21},
{Role: "repeater", PublicKey: "c0c0c0c0", Name: "Ctx", HasGPS: true, Lat: 10.1, Lon: 20.1},
})
graph := NewNeighborGraph()
@@ -85,8 +85,8 @@ func TestResolveWithContext_AffinityTooClose_FallsToGeo(t *testing.T) {
func TestResolveWithContext_GPSPreference(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "NoGPS"},
{PublicKey: "a1bbbbbb", Name: "HasGPS", HasGPS: true, Lat: 1, Lon: 2},
{Role: "repeater", PublicKey: "a1aaaaaa", Name: "NoGPS"},
{Role: "repeater", PublicKey: "a1bbbbbb", Name: "HasGPS", HasGPS: true, Lat: 1, Lon: 2},
})
ni, confidence, _ := pm.resolveWithContext("a1", nil, nil)
@@ -100,8 +100,8 @@ func TestResolveWithContext_GPSPreference(t *testing.T) {
func TestResolveWithContext_FirstMatchFallback(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "First"},
{PublicKey: "a1bbbbbb", Name: "Second"},
{Role: "repeater", PublicKey: "a1aaaaaa", Name: "First"},
{Role: "repeater", PublicKey: "a1bbbbbb", Name: "Second"},
})
ni, confidence, _ := pm.resolveWithContext("a1", nil, nil)
@@ -115,8 +115,8 @@ func TestResolveWithContext_FirstMatchFallback(t *testing.T) {
func TestResolveWithContext_NilGraphFallsToGPS(t *testing.T) {
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "NoGPS"},
{PublicKey: "a1bbbbbb", Name: "HasGPS", HasGPS: true, Lat: 1, Lon: 2},
{Role: "repeater", PublicKey: "a1aaaaaa", Name: "NoGPS"},
{Role: "repeater", PublicKey: "a1bbbbbb", Name: "HasGPS", HasGPS: true, Lat: 1, Lon: 2},
})
ni, confidence, _ := pm.resolveWithContext("a1", []string{"someone"}, nil)
@@ -131,8 +131,8 @@ func TestResolveWithContext_NilGraphFallsToGPS(t *testing.T) {
func TestResolveWithContext_BackwardCompatResolve(t *testing.T) {
// Verify original resolve() still works unchanged
pm := buildPrefixMap([]nodeInfo{
{PublicKey: "a1aaaaaa", Name: "NoGPS"},
{PublicKey: "a1bbbbbb", Name: "HasGPS", HasGPS: true, Lat: 1, Lon: 2},
{Role: "repeater", PublicKey: "a1aaaaaa", Name: "NoGPS"},
{Role: "repeater", PublicKey: "a1bbbbbb", Name: "HasGPS", HasGPS: true, Lat: 1, Lon: 2},
})
ni := pm.resolve("a1")
if ni == nil || ni.Name != "HasGPS" {
@@ -164,8 +164,8 @@ func TestResolveHopsAPI_UniquePrefix(t *testing.T) {
_ = srv
// Insert a unique node
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"ff11223344", "UniqueNode", 37.0, -122.0)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon, role) VALUES (?, ?, ?, ?, ?)",
"ff11223344", "UniqueNode", 37.0, -122.0, "repeater")
srv.store.InvalidateNodeCache()
req := httptest.NewRequest("GET", "/api/resolve-hops?hops=ff11223344", nil)
@@ -189,10 +189,10 @@ func TestResolveHopsAPI_UniquePrefix(t *testing.T) {
func TestResolveHopsAPI_AmbiguousNoContext(t *testing.T) {
srv, router := setupTestServer(t)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"ee1aaaaaaa", "Node-E1", 37.0, -122.0)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"ee1bbbbbbb", "Node-E2", 38.0, -121.0)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon, role) VALUES (?, ?, ?, ?, ?)",
"ee1aaaaaaa", "Node-E1", 37.0, -122.0, "repeater")
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon, role) VALUES (?, ?, ?, ?, ?)",
"ee1bbbbbbb", "Node-E2", 38.0, -121.0, "repeater")
srv.store.InvalidateNodeCache()
req := httptest.NewRequest("GET", "/api/resolve-hops?hops=ee1", nil)
@@ -224,12 +224,12 @@ func TestResolveHopsAPI_AmbiguousNoContext(t *testing.T) {
func TestResolveHopsAPI_WithAffinityContext(t *testing.T) {
srv, router := setupTestServer(t)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"dd1aaaaaaa", "Node-D1", 37.0, -122.0)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"dd1bbbbbbb", "Node-D2", 38.0, -121.0)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"c0c0c0c0c0", "Context", 37.1, -122.1)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon, role) VALUES (?, ?, ?, ?, ?)",
"dd1aaaaaaa", "Node-D1", 37.0, -122.0, "repeater")
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon, role) VALUES (?, ?, ?, ?, ?)",
"dd1bbbbbbb", "Node-D2", 38.0, -121.0, "repeater")
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon, role) VALUES (?, ?, ?, ?, ?)",
"c0c0c0c0c0", "Context", 37.1, -122.1, "repeater")
// Invalidate node cache so the PM includes newly inserted nodes.
srv.store.cacheMu.Lock()
@@ -279,8 +279,8 @@ func TestResolveHopsAPI_WithAffinityContext(t *testing.T) {
func TestResolveHopsAPI_ResponseShape(t *testing.T) {
srv, router := setupTestServer(t)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon) VALUES (?, ?, ?, ?)",
"bb1aaaaaaa", "Node-B1", 37.0, -122.0)
srv.db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, lat, lon, role) VALUES (?, ?, ?, ?, ?)",
"bb1aaaaaaa", "Node-B1", 37.0, -122.0, "repeater")
req := httptest.NewRequest("GET", "/api/resolve-hops?hops=bb1a", nil)
rr := httptest.NewRecorder()
+14 -5
View File
@@ -16,6 +16,7 @@ import (
"time"
"github.com/gorilla/mux"
"github.com/meshcore-analyzer/packetpath"
)
// Server holds shared state for route handlers.
@@ -172,6 +173,7 @@ func (s *Server) RegisterRoutes(r *mux.Router) {
r.HandleFunc("/api/observers/{id}", s.handleObserverDetail).Methods("GET")
r.HandleFunc("/api/observers", s.handleObservers).Methods("GET")
r.HandleFunc("/api/traces/{hash}", s.handleTraces).Methods("GET")
r.HandleFunc("/api/paths/inspect", s.handlePathInspect).Methods("POST")
r.HandleFunc("/api/iata-coords", s.handleIATACoords).Methods("GET")
r.HandleFunc("/api/audio-lab/buckets", s.handleAudioLabBuckets).Methods("GET")
@@ -957,11 +959,9 @@ func (s *Server) handlePacketDetail(w http.ResponseWriter, r *http.Request) {
pathHops = []interface{}{}
}
rawHex, _ := packet["raw_hex"].(string)
writeJSON(w, PacketDetailResponse{
Packet: packet,
Path: pathHops,
Breakdown: BuildBreakdown(rawHex),
ObservationCount: observationCount,
Observations: mapSliceToObservations(observations),
})
@@ -1020,8 +1020,17 @@ func (s *Server) handlePostPacket(w http.ResponseWriter, r *http.Request) {
contentHash := ComputeContentHash(hexStr)
pathJSON := "[]"
if len(decoded.Path.Hops) > 0 {
if pj, e := json.Marshal(decoded.Path.Hops); e == nil {
// For TRACE packets, path_json must be the payload-decoded route hops
// (decoded.Path.Hops), NOT the raw_hex header bytes which are SNR values.
// For all other packet types, derive path from raw_hex (#886).
if !packetpath.PathBytesAreHops(byte(decoded.Header.PayloadType)) {
if len(decoded.Path.Hops) > 0 {
if pj, e := json.Marshal(decoded.Path.Hops); e == nil {
pathJSON = string(pj)
}
}
} else if hops, err := packetpath.DecodePathFromRawHex(hexStr); err == nil && len(hops) > 0 {
if pj, e := json.Marshal(hops); e == nil {
pathJSON = string(pj)
}
}
@@ -2083,7 +2092,7 @@ func (s *Server) handleObserverAnalytics(w http.ResponseWriter, r *http.Request)
}
snrBuckets[bucket].Count++
}
if i < 20 && enriched["hash"] != nil {
if i < 20 {
recentPackets = append(recentPackets, enriched)
}
}
+101 -48
View File
@@ -63,6 +63,7 @@ type StoreObs struct {
RSSI *float64
Score *int
PathJSON string
RawHex string
Timestamp string
}
@@ -208,6 +209,10 @@ type PacketStore struct {
// Persisted neighbor graph for hop resolution at ingest time.
graph *NeighborGraph
// Path inspector score cache (issue #944).
inspectMu sync.RWMutex
inspectCache map[string]*inspectCachedResult
// Clock skew detection engine.
clockSkew *ClockSkewEngine
@@ -458,29 +463,42 @@ func (s *PacketStore) Load() error {
if s.db.hasResolvedPath {
rpCol = ",\n\t\t\t\to.resolved_path"
}
obsRawHexCol := ""
if s.db.hasObsRawHex {
obsRawHexCol = ", o.raw_hex"
}
limitClause := ""
// Build WHERE conditions: retention cutoff (mirrors Evict logic) + optional memory-cap limit.
var loadConditions []string
if s.retentionHours > 0 {
cutoff := time.Now().UTC().Add(-time.Duration(s.retentionHours*3600) * time.Second).Format(time.RFC3339)
loadConditions = append(loadConditions, fmt.Sprintf("t.first_seen >= '%s'", cutoff))
}
if maxPackets > 0 {
limitClause = fmt.Sprintf(
"\n\t\t\tWHERE t.id IN (SELECT id FROM transmissions ORDER BY first_seen DESC LIMIT %d)", maxPackets)
loadConditions = append(loadConditions, fmt.Sprintf(
"t.id IN (SELECT id FROM transmissions ORDER BY first_seen DESC LIMIT %d)", maxPackets))
}
filterClause := ""
if len(loadConditions) > 0 {
filterClause = "\n\t\t\tWHERE " + strings.Join(loadConditions, "\n\t\t\t AND ")
}
if s.db.isV3 {
loadSQL = `SELECT t.id, t.raw_hex, t.hash, t.first_seen, t.route_type,
t.payload_type, t.payload_version, t.decoded_json,
o.id, obs.id, obs.name, o.direction,
o.snr, o.rssi, o.score, o.path_json, strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch')` + rpCol + `
o.snr, o.rssi, o.score, o.path_json, strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch')` + obsRawHexCol + rpCol + `
FROM transmissions t
LEFT JOIN observations o ON o.transmission_id = t.id
LEFT JOIN observers obs ON obs.rowid = o.observer_idx` + limitClause + `
LEFT JOIN observers obs ON obs.rowid = o.observer_idx` + filterClause + `
ORDER BY t.first_seen ASC, o.timestamp DESC`
} else {
loadSQL = `SELECT t.id, t.raw_hex, t.hash, t.first_seen, t.route_type,
t.payload_type, t.payload_version, t.decoded_json,
o.id, o.observer_id, o.observer_name, o.direction,
o.snr, o.rssi, o.score, o.path_json, o.timestamp` + rpCol + `
o.snr, o.rssi, o.score, o.path_json, o.timestamp` + obsRawHexCol + rpCol + `
FROM transmissions t
LEFT JOIN observations o ON o.transmission_id = t.id` + limitClause + `
LEFT JOIN observations o ON o.transmission_id = t.id` + filterClause + `
ORDER BY t.first_seen ASC, o.timestamp DESC`
}
@@ -500,12 +518,16 @@ func (s *PacketStore) Load() error {
var observerID, observerName, direction, pathJSON, obsTimestamp sql.NullString
var snr, rssi sql.NullFloat64
var score sql.NullInt64
var obsRawHex sql.NullString
var resolvedPathStr sql.NullString
scanArgs := []interface{}{&txID, &rawHex, &hash, &firstSeen, &routeType, &payloadType,
&payloadVersion, &decodedJSON,
&obsID, &observerID, &observerName, &direction,
&snr, &rssi, &score, &pathJSON, &obsTimestamp}
if s.db.hasObsRawHex {
scanArgs = append(scanArgs, &obsRawHex)
}
if s.db.hasResolvedPath {
scanArgs = append(scanArgs, &resolvedPathStr)
}
@@ -565,6 +587,7 @@ func (s *PacketStore) Load() error {
RSSI: nullFloatPtr(rssi),
Score: nullIntPtr(score),
PathJSON: obsPJ,
RawHex: nullStrVal(obsRawHex),
Timestamp: normalizeTimestamp(nullStrVal(obsTimestamp)),
}
@@ -1384,11 +1407,15 @@ func (s *PacketStore) IngestNewFromDB(sinceID, limit int) ([]map[string]interfac
// New ingests always resolve fresh using the current prefix map and neighbor graph.
// On restart, Load() handles reading persisted resolved_path values. (review item #7)
var querySQL string
obsRHCol := ""
if s.db.hasObsRawHex {
obsRHCol = ", o.raw_hex"
}
if s.db.isV3 {
querySQL = `SELECT t.id, t.raw_hex, t.hash, t.first_seen, t.route_type,
t.payload_type, t.payload_version, t.decoded_json,
o.id, obs.id, obs.name, o.direction,
o.snr, o.rssi, o.score, o.path_json, strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch')
o.snr, o.rssi, o.score, o.path_json, strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch')` + obsRHCol + `
FROM transmissions t
LEFT JOIN observations o ON o.transmission_id = t.id
LEFT JOIN observers obs ON obs.rowid = o.observer_idx
@@ -1398,7 +1425,7 @@ func (s *PacketStore) IngestNewFromDB(sinceID, limit int) ([]map[string]interfac
querySQL = `SELECT t.id, t.raw_hex, t.hash, t.first_seen, t.route_type,
t.payload_type, t.payload_version, t.decoded_json,
o.id, o.observer_id, o.observer_name, o.direction,
o.snr, o.rssi, o.score, o.path_json, o.timestamp
o.snr, o.rssi, o.score, o.path_json, o.timestamp` + obsRHCol + `
FROM transmissions t
LEFT JOIN observations o ON o.transmission_id = t.id
WHERE t.id > ?
@@ -1419,6 +1446,7 @@ func (s *PacketStore) IngestNewFromDB(sinceID, limit int) ([]map[string]interfac
routeType, payloadType *int
obsID *int
observerID, observerName, direction, pathJSON, obsTS string
obsRawHex string
snr, rssi *float64
score *int
}
@@ -1435,11 +1463,16 @@ func (s *PacketStore) IngestNewFromDB(sinceID, limit int) ([]map[string]interfac
var observerID, observerName, direction, pathJSON, obsTimestamp sql.NullString
var snrVal, rssiVal sql.NullFloat64
var scoreVal sql.NullInt64
var obsRawHex sql.NullString
if err := rows.Scan(&txID, &rawHex, &hash, &firstSeen, &routeType, &payloadType,
scanArgs2 := []interface{}{&txID, &rawHex, &hash, &firstSeen, &routeType, &payloadType,
&payloadVersion, &decodedJSON,
&obsIDVal, &observerID, &observerName, &direction,
&snrVal, &rssiVal, &scoreVal, &pathJSON, &obsTimestamp); err != nil {
&snrVal, &rssiVal, &scoreVal, &pathJSON, &obsTimestamp}
if s.db.hasObsRawHex {
scanArgs2 = append(scanArgs2, &obsRawHex)
}
if err := rows.Scan(scanArgs2...); err != nil {
continue
}
@@ -1464,6 +1497,7 @@ func (s *PacketStore) IngestNewFromDB(sinceID, limit int) ([]map[string]interfac
direction: nullStrVal(direction),
pathJSON: nullStrVal(pathJSON),
obsTS: nullStrVal(obsTimestamp),
obsRawHex: nullStrVal(obsRawHex),
snr: nullFloatPtr(snrVal),
rssi: nullFloatPtr(rssiVal),
score: nullIntPtr(scoreVal),
@@ -1564,6 +1598,7 @@ func (s *PacketStore) IngestNewFromDB(sinceID, limit int) ([]map[string]interfac
RSSI: r.rssi,
Score: r.score,
PathJSON: r.pathJSON,
RawHex: r.obsRawHex,
Timestamp: normalizeTimestamp(r.obsTS),
}
@@ -1806,9 +1841,13 @@ func (s *PacketStore) IngestNewObservations(sinceObsID, limit int) []map[string]
}
var querySQL string
obsRHCol2 := ""
if s.db.hasObsRawHex {
obsRHCol2 = ", o.raw_hex"
}
if s.db.isV3 {
querySQL = `SELECT o.id, o.transmission_id, obs.id, obs.name, o.direction,
o.snr, o.rssi, o.score, o.path_json, strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch')
o.snr, o.rssi, o.score, o.path_json, strftime('%Y-%m-%dT%H:%M:%fZ', o.timestamp, 'unixepoch')` + obsRHCol2 + `
FROM observations o
LEFT JOIN observers obs ON obs.rowid = o.observer_idx
WHERE o.id > ?
@@ -1816,7 +1855,7 @@ func (s *PacketStore) IngestNewObservations(sinceObsID, limit int) []map[string]
LIMIT ?`
} else {
querySQL = `SELECT o.id, o.transmission_id, o.observer_id, o.observer_name, o.direction,
o.snr, o.rssi, o.score, o.path_json, o.timestamp
o.snr, o.rssi, o.score, o.path_json, o.timestamp` + obsRHCol2 + `
FROM observations o
WHERE o.id > ?
ORDER BY o.id ASC
@@ -1839,6 +1878,7 @@ func (s *PacketStore) IngestNewObservations(sinceObsID, limit int) []map[string]
snr, rssi *float64
score *int
pathJSON string
rawHex string
timestamp string
}
@@ -1848,9 +1888,14 @@ func (s *PacketStore) IngestNewObservations(sinceObsID, limit int) []map[string]
var observerID, observerName, direction, pathJSON, ts sql.NullString
var snr, rssi sql.NullFloat64
var score sql.NullInt64
var obsRawHex sql.NullString
if err := rows.Scan(&oid, &txID, &observerID, &observerName, &direction,
&snr, &rssi, &score, &pathJSON, &ts); err != nil {
scanArgs3 := []interface{}{&oid, &txID, &observerID, &observerName, &direction,
&snr, &rssi, &score, &pathJSON, &ts}
if s.db.hasObsRawHex {
scanArgs3 = append(scanArgs3, &obsRawHex)
}
if err := rows.Scan(scanArgs3...); err != nil {
continue
}
@@ -1864,6 +1909,7 @@ func (s *PacketStore) IngestNewObservations(sinceObsID, limit int) []map[string]
rssi: nullFloatPtr(rssi),
score: nullIntPtr(score),
pathJSON: nullStrVal(pathJSON),
rawHex: nullStrVal(obsRawHex),
timestamp: nullStrVal(ts),
})
}
@@ -1919,6 +1965,7 @@ func (s *PacketStore) IngestNewObservations(sinceObsID, limit int) []map[string]
RSSI: r.rssi,
Score: r.score,
PathJSON: r.pathJSON,
RawHex: r.rawHex,
Timestamp: normalizeTimestamp(r.timestamp),
}
@@ -2408,35 +2455,15 @@ func (s *PacketStore) enrichObs(obs *StoreObs) map[string]interface{} {
if tx != nil {
m["hash"] = strOrNil(tx.Hash)
m["raw_hex"] = strOrNil(tx.RawHex)
// Prefer per-observation raw_hex; fall back to transmission-level (#881)
if obs.RawHex != "" {
m["raw_hex"] = obs.RawHex
} else {
m["raw_hex"] = strOrNil(tx.RawHex)
}
m["payload_type"] = intPtrOrNil(tx.PayloadType)
m["route_type"] = intPtrOrNil(tx.RouteType)
m["decoded_json"] = strOrNil(tx.DecodedJSON)
} else {
// Parent tx was evicted from memory — fall back to DB lookup so that
// hash/timestamp are always present in the response (root cause of #857).
if s.db != nil {
if row, err := s.db.GetTransmissionByID(obs.TransmissionID); err == nil && row != nil {
if h, ok := row["hash"]; ok {
m["hash"] = h
}
if ts, ok := row["first_seen"]; ok && m["timestamp"] == nil {
m["timestamp"] = ts
}
if rh, ok := row["raw_hex"]; ok {
m["raw_hex"] = rh
}
if pt, ok := row["payload_type"]; ok {
m["payload_type"] = pt
}
if rt, ok := row["route_type"]; ok {
m["route_type"] = rt
}
if dj, ok := row["decoded_json"]; ok {
m["decoded_json"] = dj
}
}
}
}
return m
@@ -4503,12 +4530,19 @@ type nodeInfo struct {
Lat float64
Lon float64
HasGPS bool
LastSeen time.Time
}
func (s *PacketStore) getAllNodes() []nodeInfo {
rows, err := s.db.conn.Query("SELECT public_key, name, role, lat, lon FROM nodes")
// Try with last_seen first; fall back to without if column doesn't exist.
rows, err := s.db.conn.Query("SELECT public_key, name, role, lat, lon, last_seen FROM nodes")
hasLastSeen := true
if err != nil {
return nil
rows, err = s.db.conn.Query("SELECT public_key, name, role, lat, lon FROM nodes")
hasLastSeen = false
if err != nil {
return nil
}
}
defer rows.Close()
var nodes []nodeInfo
@@ -4516,13 +4550,25 @@ func (s *PacketStore) getAllNodes() []nodeInfo {
var pk string
var name, role sql.NullString
var lat, lon sql.NullFloat64
rows.Scan(&pk, &name, &role, &lat, &lon)
var lastSeen sql.NullString
if hasLastSeen {
rows.Scan(&pk, &name, &role, &lat, &lon, &lastSeen)
} else {
rows.Scan(&pk, &name, &role, &lat, &lon)
}
n := nodeInfo{PublicKey: pk, Name: nullStrVal(name), Role: nullStrVal(role)}
if lat.Valid && lon.Valid {
n.Lat = lat.Float64
n.Lon = lon.Float64
n.HasGPS = !(n.Lat == 0 && n.Lon == 0)
}
if hasLastSeen && lastSeen.Valid && lastSeen.String != "" {
if t, err := time.Parse(time.RFC3339, lastSeen.String); err == nil {
n.LastSeen = t
} else if t, err := time.Parse("2006-01-02 15:04:05", lastSeen.String); err == nil {
n.LastSeen = t
}
}
nodes = append(nodes, n)
}
return nodes
@@ -4537,9 +4583,20 @@ type prefixMap struct {
// entries to ~7×N (+ 1 full-key entry per node for exact-match lookups).
const maxPrefixLen = 8
// canAppearInPath returns true if the node's role allows it to appear as a
// path hop. Only repeaters, room servers, and rooms can forward packets;
// companions and sensors originate but never relay.
func canAppearInPath(role string) bool {
r := strings.ToLower(role)
return strings.Contains(r, "repeater") || strings.Contains(r, "room_server") || r == "room"
}
func buildPrefixMap(nodes []nodeInfo) *prefixMap {
pm := &prefixMap{m: make(map[string][]nodeInfo, len(nodes)*(maxPrefixLen+1))}
for _, n := range nodes {
if !canAppearInPath(n.Role) {
continue
}
pk := strings.ToLower(n.PublicKey)
maxLen := maxPrefixLen
if maxLen > len(pk) {
@@ -6605,10 +6662,6 @@ func (s *PacketStore) GetNodeHealth(pubkey string) (map[string]interface{}, erro
for i := len(packets) - 1; i >= len(packets)-recentLimit; i-- {
p := s.txToMapWithRP(packets[i])
delete(p, "observations")
// Defensive: skip packets missing hash or timestamp (belt-and-suspenders for #857)
if p["hash"] == nil || p["timestamp"] == nil {
continue
}
recentPackets = append(recentPackets, p)
}
-1
View File
@@ -315,7 +315,6 @@ type PacketTimestampsResponse struct {
type PacketDetailResponse struct {
Packet interface{} `json:"packet"`
Path []interface{} `json:"path"`
Breakdown *Breakdown `json:"breakdown"`
ObservationCount int `json:"observation_count"`
Observations []ObservationResp `json:"observations,omitempty"`
}
+84
View File
@@ -0,0 +1,84 @@
package main
import (
"fmt"
"log"
"time"
)
// checkAutoVacuum inspects the current auto_vacuum mode and logs a warning
// if it's not INCREMENTAL. Optionally performs a one-time full VACUUM if
// the operator has set db.vacuumOnStartup: true in config (#919).
func checkAutoVacuum(db *DB, cfg *Config, dbPath string) {
var autoVacuum int
if err := db.conn.QueryRow("PRAGMA auto_vacuum").Scan(&autoVacuum); err != nil {
log.Printf("[db] warning: could not read auto_vacuum: %v", err)
return
}
if autoVacuum == 2 {
log.Printf("[db] auto_vacuum=INCREMENTAL")
return
}
modes := map[int]string{0: "NONE", 1: "FULL", 2: "INCREMENTAL"}
mode := modes[autoVacuum]
if mode == "" {
mode = fmt.Sprintf("UNKNOWN(%d)", autoVacuum)
}
log.Printf("[db] auto_vacuum=%s — DB needs one-time VACUUM to enable incremental auto-vacuum. "+
"Set db.vacuumOnStartup: true in config to migrate (will block startup for several minutes on large DBs). "+
"See https://github.com/Kpa-clawbot/CoreScope/issues/919", mode)
if cfg.DB != nil && cfg.DB.VacuumOnStartup {
// WARNING: Full VACUUM creates a temporary copy of the entire DB file.
// Requires ~2× the DB file size in free disk space or it will fail.
log.Printf("[db] vacuumOnStartup=true — starting one-time full VACUUM (ensure 2x DB size free disk space)...")
start := time.Now()
rw, err := openRW(dbPath)
if err != nil {
log.Printf("[db] VACUUM failed: could not open RW connection: %v", err)
return
}
defer rw.Close()
if _, err := rw.Exec("PRAGMA auto_vacuum = INCREMENTAL"); err != nil {
log.Printf("[db] VACUUM failed: could not set auto_vacuum: %v", err)
return
}
if _, err := rw.Exec("VACUUM"); err != nil {
log.Printf("[db] VACUUM failed: %v", err)
return
}
elapsed := time.Since(start)
log.Printf("[db] VACUUM complete in %v — auto_vacuum is now INCREMENTAL", elapsed.Round(time.Millisecond))
// Re-check
var newMode int
if err := db.conn.QueryRow("PRAGMA auto_vacuum").Scan(&newMode); err == nil {
if newMode == 2 {
log.Printf("[db] auto_vacuum=INCREMENTAL (confirmed after VACUUM)")
} else {
log.Printf("[db] warning: auto_vacuum=%d after VACUUM — expected 2", newMode)
}
}
}
}
// runIncrementalVacuum runs PRAGMA incremental_vacuum(N) on a read-write
// connection. Safe to call on auto_vacuum=NONE databases (noop).
func runIncrementalVacuum(dbPath string, pages int) {
rw, err := openRW(dbPath)
if err != nil {
log.Printf("[vacuum] could not open RW connection: %v", err)
return
}
defer rw.Close()
if _, err := rw.Exec(fmt.Sprintf("PRAGMA incremental_vacuum(%d)", pages)); err != nil {
log.Printf("[vacuum] incremental_vacuum error: %v", err)
}
}
+7 -1
View File
@@ -9,6 +9,11 @@
"packetDays": 30,
"_comment": "nodeDays: nodes not seen in N days moved to inactive_nodes (default 7). observerDays: observers not sending data in N days are removed (-1 = keep forever, default 14). packetDays: transmissions older than N days are deleted (0 = disabled)."
},
"db": {
"vacuumOnStartup": false,
"incrementalVacuumPages": 1024,
"_comment": "vacuumOnStartup: run one-time full VACUUM to enable incremental auto-vacuum on existing DBs (blocks startup for minutes on large DBs; requires 2x DB file size in free disk space). incrementalVacuumPages: free pages returned to OS after each retention reaper cycle (default 1024). See #919."
},
"https": {
"cert": "/path/to/cert.pem",
"key": "/path/to/key.pem",
@@ -208,7 +213,8 @@
"packetStore": {
"maxMemoryMB": 1024,
"estimatedPacketBytes": 450,
"_comment": "In-memory packet store. maxMemoryMB caps RAM usage. All packets loaded on startup, served from RAM."
"retentionHours": 168,
"_comment": "In-memory packet store. maxMemoryMB caps RAM usage. retentionHours: only packets younger than this are loaded on startup and kept in memory (0 = unlimited, not recommended for large DBs — causes OOM on cold start). 168 = 7 days. Must be ≤ retention.packetDays * 24."
},
"resolvedPath": {
"backfillHours": 24,
+241
View File
@@ -0,0 +1,241 @@
# Clock Skew Classifier — Redesign
**Status:** spec, pre-implementation
**Supersedes:** parts of #690 / #789 / #845 / PR #894
**Date drafted:** 2026-04-24
## Problem
The current classifier (`cmd/server/clock_skew.go`) uses windowed medians, hysteresis, "good fraction" floors, and a 365-day `no_clock` threshold. It produces:
- False `no_clock` flags on nodes whose clocks are working today but had garbage timestamps in recent samples.
- Symmetric severity bands that conflate "clock at firmware default" with "operator set the clock wrong by a year" — completely different operator actions required.
- Compounding over-engineering as each operator complaint added a new tier or window.
The actual physical reality of these devices is much simpler than the classifier assumes.
## Hardware reality
Most MeshCore nodes have **no auto-updating RTC**. There are two hardware paths:
1. **Volatile RTC nodes** (`firmware/src/helpers/ArduinoHelpers.h:11``VolatileRTCClock`):
- On boot, `base_time` is hardcoded to a firmware-build constant (currently `1715770351` = 2024-05-15 20:52:31 UTC).
- `getCurrentTime()` returns `base_time + millis()/1000`.
- On reboot the value snaps back to the constant.
- User must manually sync via companion app (`set time` CLI invokes `setCurrentTime(...)`) to set a real wall-clock time, which then ticks until the next reboot.
2. **Hardware-RTC nodes** (`firmware/src/helpers/AutoDiscoverRTCClock.cpp` — DS3231 / RV3028 / PCF8563):
- Real-time chip with battery backup. Holds the time across reboots.
- Behaves correctly once set; no default-snap behavior.
The `set time RESET` CLI command (`firmware/src/helpers/CommonCLI.cpp:215`) explicitly calls `setCurrentTime(1715770351)` regardless of hardware — so even hardware-RTC nodes can be deliberately reset to the default epoch.
**Therefore every node is in exactly one of these states:**
| State | Description |
|---|---|
| **Default / never set** | RTC is at a firmware-default epoch + ticking up since the last boot. |
| **Set, drifting normally** | RTC was synced; small skew accumulating at ~0.8s/day per #789 reports. |
| **Set, drifted past tolerance** | Like above but skew has grown beyond what's useful. |
| **Wrong** | Operator-set incorrect time, or genuine RTC malfunction not matching any known default. |
There is no "bimodal RTC bug" — what looked bimodal in #845 is just a sequence of `defaulted → user sync → reboot → defaulted again`. The "bad" timestamps are not noise; they're a constant (the default epoch + a small uptime).
## Production data analysis (2026-04-24)
### 00id.net (this deployment, 416 nodes, commit `abd9c46`)
`lastSkewSec` (advert_ts observed_ts) distribution:
| Bucket | Count | Pct |
|---|---:|---:|
| OK ≤15s | 90 | 22% |
| Degrading ≤60s | 93 | 22% |
| Degraded ≤10m | 13 | 3% |
| off ≤1d | 5 | 1% |
| off ≤1y | 110 | 26% |
| absurd >1y | 105 | 25% |
Per-node `lastAdvertTS` raw timestamp distribution shows a sharp default cluster:
```
+0 days count=19 samples=114969 ← exactly at 1715770351 (just rebooted)
+1d count=9 samples=24766
+2d count=7 samples=58101
+3d count=2 samples=360
... ← decay through ~110 days
+113d count=2 samples=53776
```
103 of 416 nodes (25%) have `lastAdvertTS` between `1715770351` and `1715770351 + 1095 days`, consistent with the volatile-RTC-default-ticking-up pattern.
A second cluster of 5 nodes has `lastAdvertTS = 1672531542 ≈ 1672531200 + 5min` = **2023-01-01 00:00:00 UTC** + small uptime. This is a *different* firmware-default epoch from an older firmware version.
### Cascadia (analyzer.cascadiamesh.org, 433 nodes in 5000-packet sample, commit `111b03c` v3.5.0)
ADVERT timestamp by year-month:
```
1970-01 1 ← epoch zero (ESP32 native fallback OR ancient firmware)
2021-01 1 ← possible third default epoch
2023-01 2 ← old firmware default (matches 00id)
2024-05 60 ← current VolatileRTCClock + days uptime
2024-06 39 ← same default + weeks uptime
2024-07 21
2024-08 10
2024-09 2
2024-10 1
2024-11 2 ← decays out as fewer nodes have multi-month uptime since reboot
2025-10 1 ← pre-current-now miscellany
2025-11 2
2026-03 4
2026-04 285 ← currently set clocks (this is "now-ish")
2027-04 1 ← operator set wrong by ~1 year (typo?)
2067-12 1 ← operator set wildly wrong / corrupted RTC
```
Confirms the model: ~67% of nodes have a current clock, ~32% are at known firmware defaults at varying uptime offsets, ~3 outliers represent genuine misconfigurations.
## Known firmware default epochs
These are the values discovered in production data so far:
| Epoch (unix) | UTC | Source |
|---:|---|---|
| `0` | 1970-01-01 | Likely ESP32 boot when no RTC initialization runs (`time(NULL)` returns 0). |
| `1609459200` | 2021-01-01 | Speculation — single-sample evidence, validate as more data arrives. |
| `1672531200` | 2023-01-01 | Older firmware `VolatileRTCClock::base_time` value. |
| `1715770351` | 2024-05-15 20:52:31 | **Current** `VolatileRTCClock` constructor + `set time RESET` CLI. |
Treat the table as data, not fixed code. New firmware versions will introduce new defaults; expect to add to the list over time.
## Reconciliation with #690 — the four timestamps
#690 lists three timestamps; in practice there are four signals worth distinguishing:
| Signal | Source | Used for |
|---|---|---|
| `advert_ts` | Inside MeshCore packet, set by sending node | Per-node classification (THE signal). |
| `mqtt_envelope_ts` | Set by observer when it forwards via MQTT | Observer-side calibration only — *not* a direct node-skew signal because observer clock can itself be wrong. |
| `corescope_received_ts` | Wall clock when CoreScope ingested the message | Reference "now"; calibration cross-check. |
| `same_packet_across_observers` | Multiple observers seeing the same hash | Phase 2 calibration (triangulation). |
**Inputs flow:**
1. **Phase 2 (existing, kept):** for each packet hash seen by ≥2 observers, compute each observer's deviation from the per-packet median observed_ts → `observerOffset`. This is the triangulation #690 calls for ("Same packet observed by more than one (ideally 3+) observers gives good indication if one observer is off"). Observer offsets are the calibration table.
2. **Per-advert correction (existing, kept):** `correctedSkew = (advert_ts - observed_ts) + observerOffset[observer_id]`. If no calibration exists for an observer, fall back to raw skew with `calibrated: false`.
3. **Default detection (new):** runs on RAW `advert_ts`, not corrected. The firmware default is a fixed wall-clock value; observer offsets are seconds-to-minutes scale and cannot move `advert_ts` from 2024 to 2026. Default check is independent of calibration.
4. **Severity classification (new):** if `is_default(advert_ts)``default`; else classify by `|correctedSkew|` band.
This keeps everything #690 asks for (observer detection, bias subtraction, triangulation), and adds the firmware-default cluster as a new pre-empting tier.
## UI: explain WHY (#690 requirement)
The classifier alone doesn't satisfy #690's "present on the UI why clock skew is obvious or suspected." The evidence panel from PR #906 (per-hash observer breakdown showing raw vs corrected skew per observer) is the WHY.
For each per-node clock card the UI must show:
- **Tier badge** (default / ok / degrading / degraded / wrong) + magnitude.
- **Plain-English reason line**: e.g. "Last advert at 2024-05-15 + 3.2 days uptime — matches firmware default (volatile RTC, not yet user-set)" or "Last advert 12s vs wall clock — within OK tolerance."
- **Calibration footnote**: "Skew corrected using observer X offset +1.7s (computed from 412 multi-observer packets)" or "Single-observer measurement, no calibration available."
- **Evidence accordion** (PR #906 shape, retained): for the most recent N hashes, each observer's raw vs corrected skew + the observer's offset.
For the per-observer page (also from PR #906): show the observer's offset, the multi-observer sample count, and a tier badge using the same scale (treating `|observerOffset|` as the skew).
## Proposed classifier
Per-advert classification, no windowing:
```python
DEFAULT_EPOCHS = [0, 1609459200, 1672531200, 1715770351]
MAX_PLAUSIBLE_UPTIME_SEC = 1095 * 86400 # 3 years
def is_default(ts):
return any(d <= ts <= d + MAX_PLAUSIBLE_UPTIME_SEC for d in DEFAULT_EPOCHS)
def classify(advert_ts, corrected_skew_sec):
if is_default(advert_ts):
return "default" # gray
abs_skew = abs(corrected_skew_sec)
if abs_skew <= 15: return "ok" # green
if abs_skew <= 60: return "degrading" # yellow
if abs_skew <= 600: return "degraded" # orange
return "wrong" # red
```
`corrected_skew_sec` is the observer-bias-subtracted skew per Phase 2 calibration. Default detection is independent of calibration (runs on raw `advert_ts`).
Per-node state = classification of the node's most-recent advert (per hash, picking the most recent observation across all observers). No medians, no good-fraction, no hysteresis.
## Severity tier definitions
| Tier | Condition | Color | UI label | Meaning |
|---|---|---|---|---|
| `default` | Advert ts within `[default, default + 3y]` of any known epoch | Gray | "Default" | Volatile RTC at firmware boot constant; never set or rebooted and not re-synced. |
| `ok` | abs(skew) ≤ 15s | Green | "OK" | Working clock. |
| `degrading` | 15s < abs(skew) ≤ 60s | Yellow | "Degrading" | Real but accumulating drift. |
| `degraded` | 60s < abs(skew) ≤ 600s | Orange | "Degraded" | Off by minutes — needs re-sync. |
| `wrong` | abs(skew) > 600s and not `default` | Red | "Wrong" | Operator-set error or RTC malfunction. |
## What this kills
- The 365-day `no_clock` threshold and the entire `recentSkewWindow{Count,Sec}` machinery.
- The hysteresis / `goodFraction` / `longTermGoodFraction` logic from PR #894.
- The proposed `bimodal_clock` tier from #845 — the pattern is not bimodal, it's defaulted vs set.
- All Theil-Sen drift calculations as classifier inputs (drift remains a derived display value).
## What this preserves
- **Phase 2 observer calibration** (`calibrateObservers()`) — kept verbatim. It's what powers the "subtract observer bias" requirement from #690 and provides the triangulation evidence the UI needs.
- **Drift display** (computed but not classifying).
- **PR #906 evidence UI** — orthogonal to the classifier; it is in fact the implementation of #690's "explain WHY" requirement. Only label strings change to match the new tier names.
- **`/api/observers/clock-skew`** — unchanged shape.
## API impact
`/api/nodes/{pubkey}/clock-skew` response changes:
- `severity` enum: `default | ok | degrading | degraded | wrong` (no more `no_clock | severe | warn | absurd`).
- New field `defaultEpoch` (int, optional): if `severity == "default"`, the matched epoch.
- Drop fields: `recentMedianSkewSec`, `goodFraction`, `recentBadSampleCount`, `longTermGoodFraction`.
- Keep: `lastSkewSec`, `medianSkewSec`, `meanSkewSec`, `driftPerDaySec`, `sampleCount`, `calibrated`, `lastAdvertTS`, `lastObservedTS`, `nodeName`, `nodeRole`.
`/api/nodes/clock-skew` (fleet) shape unchanged except severity enum values.
## UI impact
- New CSS classes `skew-badge--default`, `skew-badge--degrading`, `skew-badge--degraded`, `skew-badge--wrong`. Drop `--no_clock`, `--severe`, `--warn`, `--absurd`, `--bimodal_clock`.
- Tooltip text updated per tier.
- "Default" badge tooltip should explain the clock is at firmware default plus uptime since boot, and the operator hasn't set it yet (or hasn't re-set it since the last reboot).
## Migration
Single PR replaces the classifier in `clock_skew.go` and updates the frontend badges/labels. No database schema change, no data migration — all per-call computation.
## Open issues to close
- **#789** (median hides corrected clocks) — resolved by per-advert classification.
- **#845** (bimodal_clock tier) — replaced by `default` tier; the pattern that motivated it is correctly captured.
- **PR #894** — close without merging; this design supersedes Option C entirely.
- **#690** UI completion (PR #906) — keeps moving in parallel; only label updates needed.
## Validation plan
1. Hand-run the classifier against a snapshot of `/api/nodes/clock-skew` from 00id and cascadia. Confirm:
- All 103 00id "absurd" nodes reclassify as `default`.
- All 5 cascadia 2023-01 nodes reclassify as `default`.
- The 2027 / 2067 cascadia outliers reclassify as `wrong`.
- The 285 cascadia 2026-04 nodes reclassify as `ok` (or `degrading` if drift exceeds 15s).
2. Add per-tier unit tests in `cmd/server/clock_skew_test.go`.
3. Add a regression test for each known default epoch (synthesize advert at `default + 0s`, `default + 1d`, `default + 3y - 1s` → all classify as `default`).
4. Edge cases:
- `advert_ts == 0` → matches default epoch 0.
- `advert_ts == 1715770351 + 731 days` → no longer matches (uptime cap exceeded) — should fall through to time-based classification, likely `wrong`.
- Future timestamps beyond `now + 600s``wrong`.
## Out of scope (follow-ups)
- Per-firmware-version known-default lookup (when `firmware_version` field becomes reliable on adverts).
- Reboot-count / flakiness indicator ("this node has hit default N times in last 30d").
- Auto-discovery of new default epochs from clustering analysis (could detect a 4th default emerging in the wild).
- Filtering defaulted-clock adverts out of time-windowed analytics queries (separate spec — affects path attribution).
+19
View File
@@ -98,6 +98,22 @@ How long (in hours) before a node is marked degraded or silent:
| `retention.nodeDays` | `7` | Nodes not seen in N days move to inactive |
| `retention.packetDays` | `30` | Packets older than N days are deleted daily |
> **Note:** Lowering retention does **not** immediately shrink the database file.
> SQLite marks deleted pages as free but does not return them to the filesystem
> unless [incremental auto-vacuum](database.md) is enabled. New databases created
> after v0.x.x have auto-vacuum enabled automatically. Existing databases require
> a one-time migration — see the [Database](database.md) guide.
## Database
| Field | Default | Description |
|-------|---------|-------------|
| `db.vacuumOnStartup` | `false` | Run a one-time full `VACUUM` on startup to enable incremental auto-vacuum (blocks for minutes on large DBs) |
| `db.incrementalVacuumPages` | `1024` | Free pages returned to the OS after each retention reaper cycle |
See [Database](database.md) for details on SQLite auto-vacuum, WAL, and manual maintenance.
See [#919](https://github.com/Kpa-clawbot/CoreScope/issues/919) for background.
## Channel decryption
| Field | Description |
@@ -150,6 +166,9 @@ Lower values = fresher data but more server load.
|-------|---------|-------------|
| `packetStore.maxMemoryMB` | `1024` | Maximum RAM for in-memory packet store |
| `packetStore.estimatedPacketBytes` | `450` | Estimated bytes per packet (for memory budgeting) |
| `packetStore.retentionHours` | `0` | Only load packets younger than N hours on startup and keep them in memory. **Set this on any instance with a large DB.** `0` = unlimited (loads full DB history — causes OOM on cold start when the DB has hundreds of thousands of paths). Recommended: same as `retention.packetDays × 24` (e.g. `168` for 7 days). |
> **Warning:** Leaving `retentionHours` at `0` on a large database will cause the server to OOM-kill itself on every cold start. The full packet history is loaded into the subpath index at startup; a DB with ~280K paths produces ~13M index entries before the process is killed.
## Timestamps
+82
View File
@@ -0,0 +1,82 @@
# Database
CoreScope uses SQLite in WAL (Write-Ahead Log) mode for both the server
(read-only) and ingestor (read-write).
## WAL mode
WAL mode allows concurrent reads while writes happen. It is set automatically
at connection time via `PRAGMA journal_mode=WAL`. No operator action needed.
The WAL file (`meshcore.db-wal`) grows during writes and is checkpointed
(merged back into the main DB) periodically and at clean shutdown.
## Auto-vacuum
By default, SQLite does not shrink the database file after `DELETE` operations.
Deleted pages are marked free and reused by future writes, but the file size
on disk stays the same. This is surprising when lowering retention settings.
### New databases
Databases created after this feature was added automatically have
`PRAGMA auto_vacuum = INCREMENTAL`. After each retention reaper cycle,
CoreScope runs `PRAGMA incremental_vacuum(N)` to return free pages to the OS.
### Existing databases
The `auto_vacuum` mode is stored in the database header and can only be changed
by rewriting the entire file with `VACUUM`. CoreScope will **not** do this
automatically — on large databases (5+ GB seen in the wild) it takes minutes
and holds an exclusive lock.
**To migrate an existing database:**
1. At startup, CoreScope logs a warning:
```
[db] auto_vacuum=NONE — DB needs one-time VACUUM to enable incremental auto-vacuum.
```
2. **Ensure at least 2× the database file size in free disk space.** Full VACUUM
creates a temporary copy of the entire file — on a near-full disk it will fail.
3. Set `db.vacuumOnStartup: true` in your `config.json`:
```json
{
"db": {
"vacuumOnStartup": true
}
}
```
4. Restart CoreScope. The one-time `VACUUM` will run and block startup.
5. After migration, remove or set `vacuumOnStartup: false` — it's not needed again.
### Configuration
| Field | Default | Description |
|-------|---------|-------------|
| `db.vacuumOnStartup` | `false` | One-time full VACUUM to enable incremental auto-vacuum |
| `db.incrementalVacuumPages` | `1024` | Pages returned to OS per reaper cycle |
## Manual VACUUM
You can also run a manual vacuum from the SQLite CLI:
```bash
sqlite3 data/meshcore.db "PRAGMA auto_vacuum = INCREMENTAL; VACUUM;"
```
This is equivalent to `vacuumOnStartup: true` but can be done offline.
> ⚠️ Full VACUUM requires **2× the database file size** in free disk space (it
> creates a temporary copy). Check with `ls -lh data/meshcore.db` before running.
## Checking current mode
```bash
sqlite3 data/meshcore.db "PRAGMA auto_vacuum;"
```
- `0` = NONE (default for old databases)
- `1` = FULL (automatic, but slower writes)
- `2` = INCREMENTAL (recommended — CoreScope triggers vacuum after deletes)
See [#919](https://github.com/Kpa-clawbot/CoreScope/issues/919) for background on this feature.
+3
View File
@@ -0,0 +1,3 @@
module github.com/meshcore-analyzer/packetpath
go 1.22
+76
View File
@@ -0,0 +1,76 @@
// Package packetpath provides shared helpers for extracting path hops from
// raw MeshCore packet hex bytes.
package packetpath
import (
"encoding/hex"
"fmt"
"strings"
)
// DecodePathFromRawHex extracts the header path hops directly from raw hex bytes.
// This is the authoritative path that matches what's in raw_hex, as opposed to
// decoded.Path.Hops which may be overwritten for TRACE packets (issue #886).
//
// WARNING: This function returns the literal header path bytes regardless of
// payload type. For TRACE packets these bytes are SNR values, NOT hop hashes.
// Callers that may receive TRACE packets MUST check PathBytesAreHops(payloadType)
// first, or use the safer DecodeHopsForPayload wrapper.
func DecodePathFromRawHex(rawHex string) ([]string, error) {
buf, err := hex.DecodeString(rawHex)
if err != nil || len(buf) < 2 {
return nil, fmt.Errorf("invalid or too-short hex")
}
headerByte := buf[0]
offset := 1
if IsTransportRoute(int(headerByte & 0x03)) {
if len(buf) < offset+4 {
return nil, fmt.Errorf("too short for transport codes")
}
offset += 4
}
if offset >= len(buf) {
return nil, fmt.Errorf("too short for path byte")
}
pathByte := buf[offset]
offset++
hashSize := int(pathByte>>6) + 1
hashCount := int(pathByte & 0x3F)
hops := make([]string, 0, hashCount)
for i := 0; i < hashCount; i++ {
start := offset + i*hashSize
end := start + hashSize
if end > len(buf) {
break
}
hops = append(hops, strings.ToUpper(hex.EncodeToString(buf[start:end])))
}
return hops, nil
}
// DecodeHopsForPayload returns the header path hops only when the payload type's
// header bytes are actually route hops (i.e. PathBytesAreHops(payloadType) is true).
// For TRACE packets it returns (nil, ErrPayloadHasNoHeaderHops) so the caller is
// forced to source hops from the decoded payload instead.
//
// Prefer this over DecodePathFromRawHex when the payload type is known.
func DecodeHopsForPayload(rawHex string, payloadType byte) ([]string, error) {
if !PathBytesAreHops(payloadType) {
return nil, ErrPayloadHasNoHeaderHops
}
return DecodePathFromRawHex(rawHex)
}
// ErrPayloadHasNoHeaderHops is returned by DecodeHopsForPayload when the
// payload type repurposes the raw_hex header path bytes (e.g. TRACE → SNR values).
var ErrPayloadHasNoHeaderHops = errPayloadHasNoHeaderHops{}
type errPayloadHasNoHeaderHops struct{}
func (errPayloadHasNoHeaderHops) Error() string {
return "payload type repurposes header path bytes; source hops from decoded payload"
}
+150
View File
@@ -0,0 +1,150 @@
package packetpath
import (
"encoding/hex"
"encoding/json"
"strings"
"testing"
)
func TestDecodePathFromRawHex_Basic(t *testing.T) {
// Build a simple FLOOD packet (route_type=1) with 2 hops of hashSize=1
// header: route_type=1, payload_type=2 (TXT_MSG), version=0 → 0b00_0010_01 = 0x09
// path byte: hashSize=1 (bits 7-6 = 0), hashCount=2 (bits 5-0 = 2) → 0x02
// hops: AB, CD
// payload: some bytes
raw := "0902ABCD" + "DEADBEEF"
hops, err := DecodePathFromRawHex(raw)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(hops) != 2 || hops[0] != "AB" || hops[1] != "CD" {
t.Fatalf("expected [AB, CD], got %v", hops)
}
}
func TestDecodePathFromRawHex_ZeroHops(t *testing.T) {
// DIRECT route (type=2), no hops → 0b00_0010_10 = 0x0A
// path byte: 0x00 (0 hops)
raw := "0A00" + "DEADBEEF"
hops, err := DecodePathFromRawHex(raw)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(hops) != 0 {
t.Fatalf("expected 0 hops, got %v", hops)
}
}
func TestDecodePathFromRawHex_TransportRoute(t *testing.T) {
// TRANSPORT_FLOOD (route_type=0), payload_type=5 (GRP_TXT), version=0
// header: 0b00_0101_00 = 0x14
// transport codes: 4 bytes
// path byte: hashSize=1, hashCount=1 → 0x01
// hop: FF
raw := "14" + "00112233" + "01" + "FF" + "DEAD"
hops, err := DecodePathFromRawHex(raw)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(hops) != 1 || hops[0] != "FF" {
t.Fatalf("expected [FF], got %v", hops)
}
}
// buildTracePacket creates a TRACE packet hex string where header path bytes are
// SNR values, and payload contains the actual route hops.
func buildTracePacket() (rawHex string, headerPathHops []string, payloadHops []string) {
// DIRECT route (type=2), TRACE payload (type=9), version=0
// header byte: 0b00_1001_10 = 0x26
headerByte := byte(0x26)
// Header path: 2 SNR bytes (hashSize=1, hashCount=2) → path byte = 0x02
// SNR values: 0x1A (26 dB), 0x0F (15 dB)
pathByte := byte(0x02)
snrBytes := []byte{0x1A, 0x0F}
// TRACE payload: tag(4) + authCode(4) + flags(1) + path hops
tag := []byte{0x01, 0x00, 0x00, 0x00}
authCode := []byte{0x02, 0x00, 0x00, 0x00}
// flags: path_sz=0 (1 byte hops), other bits=0 → 0x00
flags := byte(0x00)
// Payload hops: AA, BB, CC (the actual route)
payloadPathBytes := []byte{0xAA, 0xBB, 0xCC}
var buf []byte
buf = append(buf, headerByte, pathByte)
buf = append(buf, snrBytes...)
buf = append(buf, tag...)
buf = append(buf, authCode...)
buf = append(buf, flags)
buf = append(buf, payloadPathBytes...)
rawHex = strings.ToUpper(hex.EncodeToString(buf))
headerPathHops = []string{"1A", "0F"} // SNR values — NOT route hops
payloadHops = []string{"AA", "BB", "CC"} // actual route hops from payload
return
}
func TestDecodePathFromRawHex_TraceReturnsSNR(t *testing.T) {
rawHex, expectedSNR, _ := buildTracePacket()
hops, err := DecodePathFromRawHex(rawHex)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// DecodePathFromRawHex always returns header path bytes — for TRACE these are SNR values
if len(hops) != len(expectedSNR) {
t.Fatalf("expected %d hops (SNR), got %d: %v", len(expectedSNR), len(hops), hops)
}
for i, h := range hops {
if h != expectedSNR[i] {
t.Errorf("hop[%d]: expected %s, got %s", i, expectedSNR[i], h)
}
}
}
func TestTracePathJSON_UsesPayloadHops(t *testing.T) {
// This test validates the TRACE vs non-TRACE logic that callers should implement:
// For TRACE: path_json = decoded.Path.Hops (payload-decoded route hops)
// For non-TRACE: path_json = DecodePathFromRawHex(raw_hex)
rawHex, snrHops, payloadHops := buildTracePacket()
// DecodePathFromRawHex returns SNR bytes for TRACE
headerHops, _ := DecodePathFromRawHex(rawHex)
headerJSON, _ := json.Marshal(headerHops)
// payload hops (what decoded.Path.Hops would return after TRACE decoding)
payloadJSON, _ := json.Marshal(payloadHops)
// They must differ — SNR != route hops
if string(headerJSON) == string(payloadJSON) {
t.Fatalf("SNR hops and payload hops should differ for TRACE; both are %s", headerJSON)
}
// For TRACE, path_json should be payloadHops, not headerHops
_ = snrHops // snrHops == headerHops — used for documentation
t.Logf("TRACE: header path (SNR) = %s, payload path (route) = %s", headerJSON, payloadJSON)
}
func TestDecodeHopsForPayload_NonTrace(t *testing.T) {
// header 0x01, path_len 0x02, hops 0xAA 0xBB, then payload bytes
raw := "0102AABB00"
hops, err := DecodeHopsForPayload(raw, 0x05) // GRP_TXT — header path bytes ARE hops
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(hops) != 2 || hops[0] != "AA" || hops[1] != "BB" {
t.Errorf("expected [AA BB], got %v", hops)
}
}
func TestDecodeHopsForPayload_TraceReturnsError(t *testing.T) {
raw := "010205F00100"
hops, err := DecodeHopsForPayload(raw, PayloadTRACE)
if err != ErrPayloadHasNoHeaderHops {
t.Errorf("expected ErrPayloadHasNoHeaderHops, got %v", err)
}
if hops != nil {
t.Errorf("expected nil hops for TRACE, got %v", hops)
}
}
+24
View File
@@ -0,0 +1,24 @@
package packetpath
// Route type constants (header bits 1-0).
const (
RouteTransportFlood = 0
RouteFlood = 1
RouteDirect = 2
RouteTransportDirect = 3
)
// PayloadTRACE is the payload type constant for TRACE packets.
const PayloadTRACE = 0x09
// IsTransportRoute returns true for TRANSPORT_FLOOD (0) and TRANSPORT_DIRECT (3).
func IsTransportRoute(routeType int) bool {
return routeType == RouteTransportFlood || routeType == RouteTransportDirect
}
// PathBytesAreHops returns true when the raw_hex header path bytes represent
// route hop hashes (the normal case). Returns false for packet types where
// header path bytes are repurposed (e.g. TRACE uses them for SNR values).
func PathBytesAreHops(payloadType byte) bool {
return payloadType != PayloadTRACE
}
+31
View File
@@ -0,0 +1,31 @@
package packetpath
import "testing"
func TestIsTransportRoute(t *testing.T) {
if !IsTransportRoute(RouteTransportFlood) {
t.Error("RouteTransportFlood should be transport")
}
if !IsTransportRoute(RouteTransportDirect) {
t.Error("RouteTransportDirect should be transport")
}
if IsTransportRoute(RouteFlood) {
t.Error("RouteFlood should not be transport")
}
if IsTransportRoute(RouteDirect) {
t.Error("RouteDirect should not be transport")
}
}
func TestPathBytesAreHops(t *testing.T) {
if PathBytesAreHops(PayloadTRACE) {
t.Error("PathBytesAreHops(PayloadTRACE) should be false")
}
// All other known payload types should return true.
otherTypes := []byte{0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F}
for _, pt := range otherTypes {
if !PathBytesAreHops(pt) {
t.Errorf("PathBytesAreHops(0x%02X) should be true", pt)
}
}
}
+5 -5
View File
@@ -3495,12 +3495,12 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
});
// Summary
var counts = { ok: 0, warning: 0, critical: 0, absurd: 0 };
var counts = { ok: 0, degrading: 0, degraded: 0, wrong: 0, default: 0 };
data.forEach(function(n) { if (counts[n.severity] !== undefined) counts[n.severity]++; });
// Filter buttons (also serve as summary — no separate stats pills needed)
var filterColors = { ok: 'var(--status-green)', warning: 'var(--status-yellow)', critical: 'var(--status-orange)', absurd: 'var(--status-purple)', no_clock: 'var(--text-muted)' };
var filters = ['all', 'ok', 'warning', 'critical', 'absurd', 'no_clock'];
var filterColors = { ok: 'var(--status-green)', degrading: 'var(--status-yellow)', degraded: 'var(--status-orange)', wrong: 'var(--status-red)', default: 'var(--text-muted)' };
var filters = ['all', 'ok', 'degrading', 'degraded', 'wrong', 'default'];
var filterHtml = '<div style="margin-bottom:10px">' + filters.map(function(f) {
var dot = f !== 'all' ? '<span style="display:inline-block;width:8px;height:8px;border-radius:50%;background:' + filterColors[f] + ';margin-right:4px;vertical-align:middle"></span>' : '';
return '<button class="clock-filter-btn' + (activeFilter === f ? ' active' : '') + '" data-filter="' + f + '">' +
@@ -3513,8 +3513,8 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
var rowClass = 'clock-fleet-row--' + (n.severity || 'ok');
var lastAdv = n.lastObservedTS ? new Date(n.lastObservedTS * 1000).toISOString().replace('T', ' ').replace(/\.\d+Z/, ' UTC') : '—';
var skewVal = window.currentSkewValue(n);
var skewText = n.severity === 'no_clock' ? 'No Clock' : formatSkew(skewVal);
var driftText = n.severity === 'no_clock' || !n.driftPerDaySec ? '' : formatDrift(n.driftPerDaySec);
var skewText = n.severity === 'default' ? 'Default' : formatSkew(skewVal);
var driftText = n.severity === 'default' || !n.driftPerDaySec ? '' : formatDrift(n.driftPerDaySec);
return '<tr class="' + rowClass + '" data-pubkey="' + esc(n.pubkey) + '" style="cursor:pointer">' +
'<td><strong>' + esc(n.nodeName || n.pubkey.slice(0, 12)) + '</strong></td>' +
'<td style="font-family:var(--mono,monospace)">' + skewText + '</td>' +
+105 -1
View File
@@ -14,6 +14,71 @@ function isTransportRoute(rt) { return rt === 0 || rt === 3; }
function getPathLenOffset(routeType) { return isTransportRoute(routeType) ? 5 : 1; }
function transportBadge(rt) { return isTransportRoute(rt) ? ' <span class="badge badge-transport" title="' + routeTypeName(rt) + '">T</span>' : ''; }
/**
* Compute breakdown byte ranges from raw_hex on the client.
* Mirrors cmd/server/decoder.go BuildBreakdown(). Used so per-observation raw_hex
* (which can differ in path length from the top-level packet) gets accurate
* highlighted byte ranges, instead of using the server-supplied breakdown
* computed once from the top-level raw_hex.
*/
function computeBreakdownRanges(hexString, routeType, payloadType) {
if (!hexString) return [];
const clean = hexString.replace(/\s+/g, '');
const bytes = clean.length / 2;
if (bytes < 2) return [];
const ranges = [];
// Header
ranges.push({ start: 0, end: 0, label: 'Header' });
let offset = 1;
if (isTransportRoute(routeType)) {
if (bytes < offset + 4) return ranges;
ranges.push({ start: offset, end: offset + 3, label: 'Transport Codes' });
offset += 4;
}
if (offset >= bytes) return ranges;
// Path Length byte
ranges.push({ start: offset, end: offset, label: 'Path Length' });
const pathByte = parseInt(clean.slice(offset * 2, offset * 2 + 2), 16);
offset += 1;
if (isNaN(pathByte)) return ranges;
const hashSize = (pathByte >> 6) + 1;
const hashCount = pathByte & 0x3F;
const pathBytes = hashSize * hashCount;
if (hashCount > 0 && offset + pathBytes <= bytes) {
ranges.push({ start: offset, end: offset + pathBytes - 1, label: 'Path' });
}
offset += pathBytes;
if (offset >= bytes) return ranges;
const payloadStart = offset;
// ADVERT (payload_type 4) gets sub-fields when full record present
if (payloadType === 4 && bytes - payloadStart >= 100) {
ranges.push({ start: payloadStart, end: payloadStart + 31, label: 'PubKey' });
ranges.push({ start: payloadStart + 32, end: payloadStart + 35, label: 'Timestamp' });
ranges.push({ start: payloadStart + 36, end: payloadStart + 99, label: 'Signature' });
const appStart = payloadStart + 100;
if (appStart < bytes) {
ranges.push({ start: appStart, end: appStart, label: 'Flags' });
const appFlags = parseInt(clean.slice(appStart * 2, appStart * 2 + 2), 16);
let fOff = appStart + 1;
if (!isNaN(appFlags)) {
if ((appFlags & 0x10) && fOff + 8 <= bytes) {
ranges.push({ start: fOff, end: fOff + 3, label: 'Latitude' });
ranges.push({ start: fOff + 4, end: fOff + 7, label: 'Longitude' });
fOff += 8;
}
if ((appFlags & 0x20) && fOff + 2 <= bytes) fOff += 2;
if ((appFlags & 0x40) && fOff + 2 <= bytes) fOff += 2;
if ((appFlags & 0x80) && fOff < bytes) {
ranges.push({ start: fOff, end: bytes - 1, label: 'Name' });
}
}
}
} else {
ranges.push({ start: payloadStart, end: bytes - 1, label: 'Payload' });
}
return ranges;
}
// --- Utilities ---
const _apiPerf = { calls: 0, totalMs: 0, log: [], cacheHits: 0 };
const _apiCache = new Map();
@@ -440,6 +505,21 @@ const pages = {};
function registerPage(name, mod) { pages[name] = mod; }
// Tools landing page — shows sub-menu with Trace and Path Inspector (spec §2.8, M1 fix).
registerPage('tools-landing', {
init: function (container) {
container.innerHTML =
'<div class="tools-landing">' +
'<h2>Tools</h2>' +
'<div class="tools-menu">' +
'<a href="#/tools/path-inspector" class="tools-card"><h3>🔍 Path Inspector</h3><p>Resolve prefix paths to candidate full-pubkey routes with confidence scoring.</p></a>' +
'<a href="#/tools/trace/" class="tools-card"><h3>📡 Trace Viewer</h3><p>View detailed packet traces by hash.</p></a>' +
'</div>' +
'</div>';
},
destroy: function () {}
});
let currentPage = null;
function closeNav() {
@@ -460,6 +540,12 @@ function closeMoreMenu() {
function navigate() {
closeNav();
// Backward-compat redirect: #/traces/<hash> → #/tools/trace/<hash> (issue #944).
if (location.hash.startsWith('#/traces/')) {
location.hash = location.hash.replace('#/traces/', '#/tools/trace/');
return;
}
const hash = location.hash.replace('#/', '') || 'packets';
const route = hash.split('?')[0];
@@ -487,9 +573,27 @@ function navigate() {
basePage = 'observer-detail';
}
// Tools sub-routing (issue #944): tools/trace/<hash>, tools/path-inspector
if (basePage === 'tools') {
if (routeParam && routeParam.startsWith('trace/')) {
basePage = 'traces';
routeParam = routeParam.substring(6); // strip "trace/"
} else if (routeParam === 'path-inspector' || (routeParam && routeParam.startsWith('path-inspector'))) {
basePage = 'path-inspector';
routeParam = null;
} else if (!routeParam) {
// Default tools landing shows menu with both entries.
basePage = 'tools-landing';
}
}
// Also support old #/traces (no sub-path) → traces page.
if (basePage === 'traces' && !routeParam) {
basePage = 'traces';
}
// Update nav active state
document.querySelectorAll('.nav-link[data-route]').forEach(el => {
el.classList.toggle('active', el.dataset.route === basePage);
el.classList.toggle('active', el.dataset.route === basePage || (el.dataset.route === 'tools' && (basePage === 'traces' || basePage === 'path-inspector' || basePage === 'tools-landing')));
});
// Update "More" button to show active state if a low-priority page is selected
var moreBtn = document.getElementById('navMoreBtn');
+44 -17
View File
@@ -393,17 +393,25 @@
}
}
// Merge user-stored keys into the channel list
// Merge user-stored keys into the channel list.
// If a stored key matches a server-known channel, mark that channel as
// userAdded so the ✕ button appears — otherwise the user has no way to
// remove a key they added but that the server already knows about.
function mergeUserChannels() {
var keys = ChannelDecrypt.getStoredKeys();
var names = Object.keys(keys);
for (var i = 0; i < names.length; i++) {
var name = names[i];
// Check if channel already exists by name
var exists = channels.some(function (ch) {
return ch.name === name || ch.hash === name || ch.hash === ('user:' + name);
});
if (!exists) {
var matched = false;
for (var j = 0; j < channels.length; j++) {
var ch = channels[j];
if (ch.name === name || ch.hash === name || ch.hash === ('user:' + name)) {
ch.userAdded = true;
matched = true;
break;
}
}
if (!matched) {
channels.push({
hash: 'user:' + name,
name: name,
@@ -749,19 +757,38 @@
e.stopPropagation();
var channelHash = removeBtn.getAttribute('data-remove-channel');
if (!channelHash) return;
var chName = channelHash.startsWith('user:') ? channelHash.substring(5) : channelHash;
// The localStorage key is the channel name. For user:-prefixed entries
// strip the prefix; for server-known channels look up the channel
// object so we use its display name (the hash itself isn't the key).
var ch = channels.find(function (c) { return c.hash === channelHash; });
var chName = channelHash.startsWith('user:')
? channelHash.substring(5)
: (ch && ch.name) || channelHash;
if (!confirm('Remove channel "' + chName + '"? This will clear saved keys and cached messages.')) return;
ChannelDecrypt.removeKey(chName);
// Remove from channels array
channels = channels.filter(function (c) { return c.hash !== channelHash; });
if (selectedHash === channelHash) {
selectedHash = null;
messages = [];
history.replaceState(null, '', '#/channels');
var msgEl2 = document.getElementById('chMessages');
if (msgEl2) msgEl2.innerHTML = '<div class="ch-empty">Choose a channel from the sidebar to view messages</div>';
var header2 = document.getElementById('chHeader');
if (header2) header2.querySelector('.ch-header-text').textContent = 'Select a channel';
if (channelHash.startsWith('user:')) {
// Pure user-added channel — drop from the list entirely.
channels = channels.filter(function (c) { return c.hash !== channelHash; });
if (selectedHash === channelHash) {
selectedHash = null;
messages = [];
history.replaceState(null, '', '#/channels');
var msgEl2 = document.getElementById('chMessages');
if (msgEl2) msgEl2.innerHTML = '<div class="ch-empty">Choose a channel from the sidebar to view messages</div>';
var header2 = document.getElementById('chHeader');
if (header2) header2.querySelector('.ch-header-text').textContent = 'Select a channel';
}
} else if (ch) {
// Server-known channel: keep the row, just unmark as user-added so
// the ✕ disappears until they re-add a key.
ch.userAdded = false;
// If this was the selected channel, clear decrypted messages since
// the key is gone — they can't be re-decrypted without re-adding it.
if (selectedHash === channelHash) {
messages = [];
var msgEl2 = document.getElementById('chMessages');
if (msgEl2) msgEl2.innerHTML = '<div class="ch-empty">Key removed — add a key to decrypt messages</div>';
}
}
renderChannelList();
return;
+5 -1
View File
@@ -629,7 +629,11 @@
}
writeOverrides(delta);
_runPipeline();
_refreshPanel();
// Skip re-render while the user is typing inside the panel — setting
// innerHTML would destroy the focused input and collapse the mobile keyboard.
if (!(_panelEl && _panelEl.contains(document.activeElement))) {
_refreshPanel();
}
}, 300);
}
+2 -1
View File
@@ -87,7 +87,8 @@ let polygon = null;
let closingLine = null;
function latLonPair(latlng) {
return [parseFloat(latlng.lat.toFixed(6)), parseFloat(latlng.lng.toFixed(6))];
const w = latlng.wrap();
return [parseFloat(w.lat.toFixed(6)), parseFloat(w.lng.toFixed(6))];
}
function render() {
+138 -68
View File
@@ -7,6 +7,14 @@ window.HopResolver = (function() {
const MAX_HOP_DIST = 1.8; // ~200km in degrees
const REGION_RADIUS_KM = 300;
// Only repeaters and room servers can appear as path hops per protocol.
// Companions/sensors originate but never relay packets.
function canAppearInPath(role) {
if (!role) return false;
var r = String(role).toLowerCase();
return r.indexOf('repeater') >= 0 || r.indexOf('room_server') >= 0 || r === 'room';
}
let prefixIdx = {}; // lowercase hex prefix → [node, ...]
let pubkeyIdx = {}; // full lowercase pubkey → node (O(1) lookup)
let nodesList = [];
@@ -40,7 +48,11 @@ window.HopResolver = (function() {
for (const n of nodesList) {
if (!n.public_key) continue;
const pk = n.public_key.toLowerCase();
// pubkeyIdx includes ALL nodes — used by resolveFromServer for
// server-confirmed full-pubkey lookups (any node type).
pubkeyIdx[pk] = n;
// prefixIdx only includes nodes that can appear as path hops.
if (!canAppearInPath(n.role)) continue;
for (let len = 1; len <= 3; len++) {
const p = pk.slice(0, len * 2);
if (!prefixIdx[p]) prefixIdx[p] = [];
@@ -72,33 +84,89 @@ window.HopResolver = (function() {
}
/**
* Pick the best candidate using affinity first, then geo-distance fallback.
* Pick the best candidate by scoring against BOTH prev and next resolved hops.
*
* Strategy (in priority order):
* 1. Neighbor-graph edge weight: sum of edge scores to prevPubkey + nextPubkey. Pick max.
* 2. Geographic centroid: if no candidate has graph edges, compute centroid of
* prev+next positions and pick closest candidate by haversine distance.
* 3. Single-anchor geo fallback: if only one neighbor is resolved, use it as anchor.
* 4. Original heuristic: first candidate (when no context at all).
*
* @param {Array} candidates - candidates with lat/lon/pubkey/name
* @param {string|null} adjacentPubkey - pubkey of the previously/next resolved hop
* @param {Object|null} anchor - {lat, lon} for geo fallback
* @param {number|null} fallbackLat - fallback anchor lat (e.g. observer)
* @param {number|null} fallbackLon - fallback anchor lon
* @param {string|null} prevPubkey - pubkey of previous resolved hop
* @param {string|null} nextPubkey - pubkey of next resolved hop
* @param {Object|null} prevPos - {lat, lon} of previous resolved hop or origin
* @param {Object|null} nextPos - {lat, lon} of next resolved hop or observer
* @returns {Object} best candidate
*/
function pickByAffinity(candidates, adjacentPubkey, anchor, fallbackLat, fallbackLon) {
// If we have affinity data and an adjacent hop, prefer neighbors
if (adjacentPubkey && Object.keys(affinityMap).length > 0) {
const withAffinity = candidates
.map(c => ({ ...c, affinity: getAffinity(adjacentPubkey, c.pubkey) }))
.filter(c => c.affinity > 0);
if (withAffinity.length > 0) {
withAffinity.sort((a, b) => b.affinity - a.affinity);
return withAffinity[0];
function pickByAffinity(candidates, prevPubkey, nextPubkey, prevPos, nextPos) {
const hasGraph = Object.keys(affinityMap).length > 0;
const hasAdj = prevPubkey || nextPubkey;
// Strategy 1: neighbor-graph edge weights (sum of prev + next)
if (hasGraph && hasAdj) {
const scored = candidates.map(function(c) {
let s = 0;
if (prevPubkey) s += getAffinity(prevPubkey, c.pubkey);
if (nextPubkey) s += getAffinity(nextPubkey, c.pubkey);
return { candidate: c, edgeScore: s };
});
const withEdges = scored.filter(function(s) { return s.edgeScore > 0; });
if (withEdges.length > 0) {
withEdges.sort(function(a, b) { return b.edgeScore - a.edgeScore; });
_traceMultiCandidate(candidates, scored, withEdges[0].candidate, 'graph');
return withEdges[0].candidate;
}
}
// Fallback: geo-distance sort (existing behavior)
const effectiveAnchor = anchor || (fallbackLat != null ? { lat: fallbackLat, lon: fallbackLon } : null);
if (effectiveAnchor) {
candidates.sort((a, b) => dist(a.lat, a.lon, effectiveAnchor.lat, effectiveAnchor.lon) - dist(b.lat, b.lon, effectiveAnchor.lat, effectiveAnchor.lon));
// Strategy 2/3: geographic — centroid of prev+next, or single anchor
let anchorLat = null, anchorLon = null, anchorCount = 0;
if (prevPos && prevPos.lat != null && prevPos.lon != null) {
anchorLat = (anchorLat || 0) + prevPos.lat;
anchorLon = (anchorLon || 0) + prevPos.lon;
anchorCount++;
}
if (nextPos && nextPos.lat != null && nextPos.lon != null) {
anchorLat = (anchorLat || 0) + nextPos.lat;
anchorLon = (anchorLon || 0) + nextPos.lon;
anchorCount++;
}
if (anchorCount > 0) {
anchorLat /= anchorCount;
anchorLon /= anchorCount;
const geoScored = candidates.map(function(c) {
const d = (c.lat != null && c.lon != null && !(c.lat === 0 && c.lon === 0))
? haversineKm(c.lat, c.lon, anchorLat, anchorLon) : 999999;
return { candidate: c, distKm: d };
});
geoScored.sort(function(a, b) { return a.distKm - b.distKm; });
_traceMultiCandidate(candidates, geoScored, geoScored[0].candidate, 'centroid');
return geoScored[0].candidate;
}
// Strategy 4: no context — return first candidate
_traceMultiCandidate(candidates, null, candidates[0], 'fallback');
return candidates[0];
}
/** Dev-mode console trace for multi-candidate picks */
function _traceMultiCandidate(candidates, scored, chosen, method) {
if (typeof console === 'undefined' || !console.debug) return;
if (candidates.length < 2) return;
try {
const prefix = candidates[0].pubkey ? candidates[0].pubkey.slice(0, 2) : '??';
const scoreSummary = scored ? scored.map(function(s) {
const pk = (s.candidate || s).pubkey || '?';
const val = s.edgeScore != null ? s.edgeScore : (s.distKm != null ? s.distKm + 'km' : '?');
return pk.slice(0, 8) + ':' + val;
}) : [];
console.debug('[hop-resolver] hash=' + prefix + ' candidates=' + candidates.length +
' scored=[' + scoreSummary.join(',') + '] chose=' + (chosen.pubkey || '?').slice(0, 8) +
' method=' + method);
} catch(e) { /* trace is best-effort */ }
}
/**
* Resolve an array of hex hop prefixes to node info.
* Returns a map: { hop: {name, pubkey, lat, lon, ambiguous, unreliable} }
@@ -169,52 +237,54 @@ window.HopResolver = (function() {
}
}
// Forward pass
let lastPos = (originLat != null && originLon != null) ? { lat: originLat, lon: originLon } : null;
let lastResolvedPubkey = null;
for (let i = 0; i < hops.length; i++) {
const hop = hops[i];
if (hopPositions[hop]) {
lastPos = hopPositions[hop];
lastResolvedPubkey = resolved[hop] ? resolved[hop].pubkey : null;
continue;
// Combined disambiguation: resolve ambiguous hops using both neighbors.
// We iterate until no more hops can be resolved (handles cascading dependencies).
const originPos = (originLat != null && originLon != null) ? { lat: originLat, lon: originLon } : null;
const observerPos = (observerLat != null && observerLon != null) ? { lat: observerLat, lon: observerLon } : null;
let changed = true;
let maxIter = hops.length + 1; // prevent infinite loops
while (changed && maxIter-- > 0) {
changed = false;
for (let i = 0; i < hops.length; i++) {
const hop = hops[i];
if (hopPositions[hop]) continue; // already resolved
const r = resolved[hop];
if (!r || !r.ambiguous) continue;
const withLoc = r.candidates.filter(c => c.lat != null && c.lon != null && !(c.lat === 0 && c.lon === 0));
if (!withLoc.length) continue;
// Find prev resolved neighbor
let prevPubkey = null, prevPos = null;
for (let j = i - 1; j >= 0; j--) {
if (hopPositions[hops[j]]) {
prevPos = hopPositions[hops[j]];
prevPubkey = resolved[hops[j]] ? resolved[hops[j]].pubkey : null;
break;
}
}
if (!prevPos && originPos) prevPos = originPos;
// Find next resolved neighbor
let nextPubkey = null, nextPos = null;
for (let j = i + 1; j < hops.length; j++) {
if (hopPositions[hops[j]]) {
nextPos = hopPositions[hops[j]];
nextPubkey = resolved[hops[j]] ? resolved[hops[j]].pubkey : null;
break;
}
}
if (!nextPos && observerPos) nextPos = observerPos;
// Skip if we have zero context (wait for a later iteration or neighbor resolution)
if (!prevPubkey && !nextPubkey && !prevPos && !nextPos) continue;
const picked = pickByAffinity(withLoc, prevPubkey, nextPubkey, prevPos, nextPos);
r.name = picked.name;
r.pubkey = picked.pubkey;
hopPositions[hop] = { lat: picked.lat, lon: picked.lon };
changed = true;
}
const r = resolved[hop];
if (!r || !r.ambiguous) continue;
const withLoc = r.candidates.filter(c => c.lat && c.lon && !(c.lat === 0 && c.lon === 0));
if (!withLoc.length) continue;
// Affinity-aware: prefer candidates that are neighbors of the previous hop
const picked = pickByAffinity(withLoc, lastResolvedPubkey, lastPos, i === hops.length - 1 ? observerLat : null, i === hops.length - 1 ? observerLon : null);
r.name = picked.name;
r.pubkey = picked.pubkey;
hopPositions[hop] = { lat: picked.lat, lon: picked.lon };
lastPos = hopPositions[hop];
lastResolvedPubkey = picked.pubkey;
}
// Backward pass
let nextPos = (observerLat != null && observerLon != null) ? { lat: observerLat, lon: observerLon } : null;
let nextResolvedPubkey = null;
for (let i = hops.length - 1; i >= 0; i--) {
const hop = hops[i];
if (hopPositions[hop]) {
nextPos = hopPositions[hop];
nextResolvedPubkey = resolved[hop] ? resolved[hop].pubkey : null;
continue;
}
const r = resolved[hop];
if (!r || !r.ambiguous) continue;
const withLoc = r.candidates.filter(c => c.lat && c.lon && !(c.lat === 0 && c.lon === 0));
if (!withLoc.length || !nextPos) continue;
// Affinity-aware: prefer candidates that are neighbors of the next hop
const picked = pickByAffinity(withLoc, nextResolvedPubkey, nextPos, null, null);
r.name = picked.name;
r.pubkey = picked.pubkey;
hopPositions[hop] = { lat: picked.lat, lon: picked.lon };
nextPos = hopPositions[hop];
nextResolvedPubkey = picked.pubkey;
}
// Sanity check: drop hops impossibly far from neighbors
@@ -276,13 +346,13 @@ window.HopResolver = (function() {
*/
function resolveFromServer(hops, resolvedPath) {
if (!hops || !resolvedPath || hops.length !== resolvedPath.length) return {};
var result = {};
for (var i = 0; i < hops.length; i++) {
var hop = hops[i];
var pubkey = resolvedPath[i];
const result = {};
for (let i = 0; i < hops.length; i++) {
const hop = hops[i];
const pubkey = resolvedPath[i];
if (!pubkey) continue; // null = unresolved, leave for client-side fallback
// O(1) lookup via pubkeyIdx built during init()
var node = pubkeyIdx[pubkey.toLowerCase()] || null;
const node = pubkeyIdx[pubkey.toLowerCase()] || null;
result[hop] = {
name: node ? node.name : pubkey.slice(0, 8),
pubkey: pubkey,
+2 -1
View File
@@ -50,7 +50,7 @@
<a href="#/live" class="nav-link" data-route="live" data-priority="high">🔴 Live</a>
<a href="#/channels" class="nav-link" data-route="channels">Channels</a>
<a href="#/nodes" class="nav-link" data-route="nodes" data-priority="high">Nodes</a>
<a href="#/traces" class="nav-link" data-route="traces">Traces</a>
<a href="#/tools" class="nav-link" data-route="tools">Tools</a>
<a href="#/observers" class="nav-link" data-route="observers">Observers</a>
<a href="#/analytics" class="nav-link" data-route="analytics">Analytics</a>
<a href="#/perf" class="nav-link" data-route="perf">⚡ Perf</a>
@@ -105,6 +105,7 @@
<script src="table-sort.js?v=__BUST__"></script>
<script src="nodes.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="traces.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="path-inspector.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="analytics.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="audio.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
<script src="audio-v1-constellation.js?v=__BUST__" onerror="console.error('Failed to load:', this.src)"></script>
+144 -2
View File
@@ -102,8 +102,21 @@
async function init(container) {
container.innerHTML = `
<div id="map-wrap" style="position:relative;width:100%;height:100%;">
<div id="leaflet-map" style="width:100%;height:100%;"></div>
<div id="map-wrap" style="position:relative;width:100%;height:100%;display:flex;">
<div id="leaflet-map" style="flex:1 1 0%;height:100%;"></div>
<div class="map-side-pane" id="mapSidePane">
<div class="pane-toggle" id="mapPaneToggle" title="Path Inspector"></div>
<div class="pane-content">
<h3 style="margin:0 0 8px 0;font-size:14px;">Path Inspector</h3>
<p style="font-size:11px;color:var(--text-muted);margin:0 0 8px 0;">Hex prefixes (1-3 bytes), comma or space separated.</p>
<div style="display:flex;gap:4px;margin-bottom:8px;">
<input type="text" id="mapPiInput" class="input" placeholder="2C,A1,F4" style="flex:1;">
<button id="mapPiSubmit" class="btn btn-primary btn-sm">Go</button>
</div>
<div id="mapPiError" class="path-inspector-error"></div>
<div id="mapPiResults"></div>
</div>
</div>
<button class="map-controls-toggle" id="mapControlsToggle" aria-label="Toggle map controls" aria-expanded="true"></button>
<div class="map-controls" id="mapControls" role="region" aria-label="Map controls">
<h3>🗺 Map Controls</h3>
@@ -553,6 +566,19 @@
}
}
// Check for pending path inspector route (cross-page navigation from Path Inspector).
if (window._pendingPathInspectorRoute) {
var pending = window._pendingPathInspectorRoute;
delete window._pendingPathInspectorRoute;
if (pending.path && pending.path.length > 0) {
if (window.routeLayer) window.routeLayer.clearLayers();
drawPacketRoute(pending.path.slice(1), pending.path[0]);
}
}
// Wire up map side pane (Path Inspector embedded - spec §2.7).
initMapSidePane();
// Don't fitBounds on initial load — respect the Bay Area default or saved view
// Only fitBounds on subsequent data refreshes if user hasn't manually panned
} catch (e) {
@@ -981,6 +1007,122 @@
map.fitBounds(bounds, { padding: [50, 50], maxZoom: 14 });
}
// === Map Side Pane — Path Inspector (spec §2.7) ===
function initMapSidePane() {
var pane = document.getElementById('mapSidePane');
var toggle = document.getElementById('mapPaneToggle');
var input = document.getElementById('mapPiInput');
var btn = document.getElementById('mapPiSubmit');
if (!pane || !toggle) return;
toggle.addEventListener('click', function () {
pane.classList.toggle('expanded');
toggle.textContent = pane.classList.contains('expanded') ? '▶' : '◀';
// Invalidate map size after transition.
setTimeout(function () { if (map) map.invalidateSize(); }, 220);
});
if (btn && input) {
btn.addEventListener('click', function () { mapPiSubmit(input.value); });
input.addEventListener('keydown', function (e) {
if (e.key === 'Enter') mapPiSubmit(input.value);
});
}
// Auto-open if URL has prefixes param while on map.
var params = new URLSearchParams(location.hash.split('?')[1] || '');
var prefixParam = params.get('prefixes');
if (prefixParam && input) {
pane.classList.add('expanded');
toggle.textContent = '▶';
input.value = prefixParam;
setTimeout(function () { if (map) map.invalidateSize(); }, 220);
mapPiSubmit(prefixParam);
}
}
function mapPiSubmit(raw) {
var errDiv = document.getElementById('mapPiError');
var resultsDiv = document.getElementById('mapPiResults');
if (!errDiv || !resultsDiv) return;
errDiv.textContent = '';
resultsDiv.innerHTML = '';
// Reuse PathInspector validation if available.
var prefixes = raw.trim().split(/[\s,]+/).filter(function (s) { return s.length > 0; }).map(function (s) { return s.toLowerCase(); });
var err = (window.PathInspector && window.PathInspector.validatePrefixes) ? window.PathInspector.validatePrefixes(prefixes) : null;
if (!err && prefixes.length === 0) err = 'Enter at least one prefix.';
if (err) { errDiv.textContent = err; return; }
resultsDiv.innerHTML = '<p style="font-size:12px;">Loading...</p>';
fetch('/api/paths/inspect', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ prefixes: prefixes })
})
.then(function (r) {
if (r.status === 503) return r.json().then(function () { throw new Error('Service warming up, retry shortly.'); });
if (!r.ok) return r.json().then(function (d) { throw new Error(d.error || 'Request failed'); });
return r.json();
})
.then(function (data) { renderMapPiResults(data, resultsDiv); })
.catch(function (e) { resultsDiv.innerHTML = ''; errDiv.textContent = e.message; });
}
function renderMapPiResults(data, div) {
if (!data.candidates || data.candidates.length === 0) {
div.innerHTML = '<p style="font-size:12px;color:var(--text-muted);">No candidates found.</p>';
return;
}
var html = '<table class="path-inspector-table" style="font-size:11px;width:100%;"><thead><tr><th>#</th><th>Score</th><th>Path</th><th></th></tr></thead><tbody>';
for (var i = 0; i < data.candidates.length; i++) {
var c = data.candidates[i];
var rowClass = c.speculative ? 'speculative-row' : '';
html += '<tr class="' + rowClass + '">';
html += '<td>' + (i + 1) + '</td>';
html += '<td class="' + (c.speculative ? 'speculative-warning' : '') + '">' + c.score.toFixed(2) + (c.speculative ? ' ⚠' : '') + '</td>';
html += '<td title="' + safeEsc(c.names.join(' → ')) + '">' + safeEsc(c.names.slice(0, 3).join('→')) + (c.names.length > 3 ? '…' : '') + '</td>';
html += '<td><button class="btn btn-sm" data-idx="' + i + '" title="Show on Map">📍</button></td>';
html += '</tr>';
// Per-hop evidence (collapsed).
html += '<tr class="evidence-row collapsed" data-evidence="' + i + '"><td colspan="4"><div class="evidence-detail" style="font-size:10px;">';
if (c.evidence && c.evidence.perHop) {
for (var j = 0; j < c.evidence.perHop.length; j++) {
var h = c.evidence.perHop[j];
html += '<div>Hop ' + (j+1) + ': ' + h.prefix + ' (×' + h.candidatesConsidered + ') w=' + h.edgeWeight.toFixed(2);
if (h.alternatives && h.alternatives.length > 0) {
html += ' <span style="color:var(--text-muted);">[+' + h.alternatives.length + ' alt]</span>';
}
html += '</div>';
}
}
html += '</div></td></tr>';
}
html += '</tbody></table>';
div.innerHTML = html;
// Wire buttons.
div.querySelectorAll('button[data-idx]').forEach(function (btn) {
btn.addEventListener('click', function () {
var idx = parseInt(btn.dataset.idx);
var cand = data.candidates[idx];
if (routeLayer) routeLayer.clearLayers();
drawPacketRoute(cand.path.slice(1), cand.path[0]);
});
});
// Expand evidence on row click.
div.querySelectorAll('.path-inspector-table tbody tr:not(.evidence-row)').forEach(function (row) {
row.style.cursor = 'pointer';
row.addEventListener('click', function (e) {
if (e.target.tagName === 'BUTTON') return;
var b = row.querySelector('button[data-idx]');
if (!b) return;
var ev = div.querySelector('tr[data-evidence="' + b.dataset.idx + '"]');
if (ev) ev.classList.toggle('collapsed');
});
});
}
function destroy() {
if (wsHandler) offWS(wsHandler);
wsHandler = null;
+38 -12
View File
@@ -808,7 +808,7 @@
let _themeRefreshHandler = null;
let _allNodes = null; // cached full node list
let _fleetSkew = null; // cached clock skew map: pubkey → {severity, recentMedianSkewSec, medianSkewSec, ...}
let _fleetSkew = null; // cached clock skew map: pubkey → {severity, medianSkewSec, ...}
/**
* Fetch per-node clock skew and render into the given container element.
@@ -824,14 +824,28 @@
var driftHtml = cs.driftPerDaySec ? '<div style="font-size:12px;color:var(--text-muted);margin-top:2px">Drift: ' + formatDrift(cs.driftPerDaySec) + '</div>' : '';
var sparkHtml = renderSkewSparkline(cs.samples, 200, 32);
var skewVal = window.currentSkewValue(cs);
var skewDisplay = cs.severity === 'no_clock'
? '<span style="font-size:18px;font-weight:700;color:var(--text-muted)">No Clock</span>'
var skewDisplay = cs.severity === 'default'
? '<span style="font-size:18px;font-weight:700;color:var(--text-muted)">Default</span>'
: '<span style="font-size:18px;font-weight:700;font-family:var(--mono)">' + formatSkew(skewVal) + '</span>';
var bimodalWarning = '';
if (cs.severity === 'bimodal_clock') {
var totalRecent = cs.recentSampleCount || 0;
bimodalWarning = '<div style="font-size:12px;color:var(--status-amber-text);margin-top:4px">⚠️ ' + (cs.recentBadSampleCount || '?') + ' of last ' + (totalRecent || '?') + ' adverts had nonsense timestamps (likely RTC reset)</div>';
// Per-tier explainer line (plain English reason).
var explainer = '';
var absSkew = Math.abs(cs.lastSkewSec || 0);
var skewStr = Math.round(absSkew) + 's';
if (cs.severity === 'default') {
var isoAdv = cs.lastAdvertTS ? new Date(cs.lastAdvertTS * 1000).toISOString() : '?';
explainer = 'Last advert at ' + isoAdv + ' — matches firmware default (volatile RTC, not user-set since boot)';
} else if (cs.severity === 'ok') {
explainer = 'Last advert ' + skewStr + ' vs wall clock — within OK tolerance (≤15s)';
} else if (cs.severity === 'degrading') {
explainer = 'Last advert ' + skewStr + ' vs wall clock — drift accumulating (≤60s)';
} else if (cs.severity === 'degraded') {
explainer = 'Last advert ' + skewStr + ' vs wall clock — significantly off (≤10m)';
} else if (cs.severity === 'wrong') {
explainer = 'Last advert ' + skewStr + ' vs wall clock — clock incorrect (operator-set or RTC failure)';
}
var explainerHtml = explainer ? '<div style="font-size:12px;color:var(--text-muted);margin-top:4px">' + explainer + '</div>' : '';
container.innerHTML =
'<h4 style="margin:0 0 6px">⏰ Clock Skew</h4>' +
'<div style="display:flex;align-items:center;gap:12px;flex-wrap:wrap">' +
@@ -839,9 +853,9 @@
renderSkewBadge(cs.severity, skewVal, cs) +
(cs.calibrated ? ' <span style="font-size:10px;color:var(--text-muted)" title="Observer-calibrated">✓ calibrated</span>' : '') +
'</div>' +
explainerHtml +
driftHtml +
(sparkHtml ? '<div class="skew-sparkline-wrap" style="margin-top:8px">' + sparkHtml + '<div style="font-size:10px;color:var(--text-muted)">Skew over time (' + (cs.samples || []).length + ' samples)</div></div>' : '') +
bimodalWarning;
(sparkHtml ? '<div class="skew-sparkline-wrap" style="margin-top:8px">' + sparkHtml + '<div style="font-size:10px;color:var(--text-muted)">Skew over time (' + (cs.samples || []).length + ' samples)</div></div>' : '');
} catch (e) {
// Non-fatal — section stays hidden
}
@@ -1144,6 +1158,19 @@
makeColumnsResizable('#nodesTable', 'meshcore-nodes-col-widths');
}
/**
* Navigate to the full-screen node view for `pubkey` from anywhere within
* the nodes module. Single source of navigation truth works regardless
* of current hash state (hash assignment alone is a no-op when the hash
* is already the target).
*/
function navigateToNode(pubkey) {
destroy();
var appEl = document.getElementById('app');
history.replaceState(null, '', '#/nodes/' + encodeURIComponent(pubkey));
init(appEl, pubkey);
}
async function selectNode(pubkey) {
// On mobile, navigate to full-screen node view
if (window.innerWidth <= 640) {
@@ -1307,12 +1334,11 @@
} catch {}
}
// #856: Wire "Details" button to navigate to full-screen node view
// Wire "Details" button via the unified navigateToNode helper
var detailBtn = panel.querySelector('.node-detail-btn');
if (detailBtn) {
detailBtn.addEventListener('click', function() {
var pk = detailBtn.getAttribute('data-pubkey');
location.hash = '#/nodes/' + pk;
navigateToNode(decodeURIComponent(detailBtn.getAttribute('data-pubkey')));
});
}
+95 -38
View File
@@ -389,7 +389,7 @@
expandedHashes.add(h);
const obsPacket = {...data.packet, observer_id: obs.observer_id, observer_name: obs.observer_name, snr: obs.snr, rssi: obs.rssi, path_json: obs.path_json, resolved_path: obs.resolved_path, direction: obs.direction, timestamp: obs.timestamp, first_seen: obs.timestamp};
clearParsedCache(obsPacket);
selectPacket(obs.id, h, {packet: obsPacket, breakdown: data.breakdown, observations: data.observations}, obs.id);
selectPacket(obs.id, h, {packet: obsPacket, observations: data.observations}, obs.id);
} else {
selectPacket(data.packet.id, h, data);
}
@@ -468,6 +468,9 @@
// Check if new packets pass current filters
const filtered = newPkts.filter(p => {
// When user pinned a hash, accept ONLY that exact packet — bypass all
// other filters (window/region/type/observer/node).
if (filters.hash) return p.hash === filters.hash;
// Respect time window filter — drop packets outside the selected window
const windowMin = savedTimeWindowMin;
if (windowMin > 0) {
@@ -477,7 +480,6 @@
}
if (filters.type) { const types = filters.type.split(',').map(Number); if (!types.includes(p.payload_type)) return false; }
if (filters.observer) { const obsSet = new Set(filters.observer.split(',')); if (!obsSet.has(p.observer_id) && !(p._children && p._children.some(c => obsSet.has(String(c.observer_id))))) return false; }
if (filters.hash && p.hash !== filters.hash) return false;
if (RegionFilter.getRegionParam()) {
const selectedRegions = RegionFilter.getRegionParam().split(',');
const obs = observerMap.get(p.observer_id);
@@ -519,7 +521,7 @@
if (p.decoded_json) existing.decoded_json = p.decoded_json;
// Update expanded children if this group is expanded
if (expandedHashes.has(h) && existing._children) {
existing._children.unshift(p);
existing._children.unshift(clearParsedCache({...p, _isObservation: true}));
if (existing._children.length > 200) existing._children.length = 200;
sortGroupChildren(existing);
// Invalidate row counts — child count changed, so virtual scroll
@@ -610,27 +612,52 @@
} catch {}
}
async function loadPackets() {
try {
const params = new URLSearchParams();
const selectedWindow = Number(document.getElementById('fTimeWindow')?.value);
const windowMin = Number.isFinite(selectedWindow) ? selectedWindow : savedTimeWindowMin;
if (windowMin > 0 && !filters.hash) {
const since = new Date(Date.now() - windowMin * 60000).toISOString();
params.set('since', since);
}
params.set('limit', String(PACKET_LIMIT));
const regionParam = RegionFilter.getRegionParam();
if (regionParam) params.set('region', regionParam);
if (filters.hash) params.set('hash', filters.hash);
if (filters.node) params.set('node', filters.node);
if (filters.observer) params.set('observer', filters.observer);
if (filters.channel) params.set('channel', filters.channel);
// Build URLSearchParams for /api/packets given UI state. Pure function for
// testability — returns the params object the next call to /api/packets
// would use. The hash filter is an exact identifier: when present it
// suppresses ALL other filters (region, time window, observer, node,
// channel). The user is asking for THAT packet regardless of saved
// selections.
function buildPacketsParams({ filters, regionParam, windowMin, groupByHash, limit }) {
const params = new URLSearchParams();
if (filters.hash) {
params.set('hash', filters.hash);
params.set('limit', String(limit));
if (groupByHash) {
params.set('groupByHash', 'true');
} else {
params.set('expand', 'observations');
}
return params;
}
if (windowMin > 0) {
const since = new Date(Date.now() - windowMin * 60000).toISOString();
params.set('since', since);
}
params.set('limit', String(limit));
if (regionParam) params.set('region', regionParam);
if (filters.node) params.set('node', filters.node);
if (filters.observer) params.set('observer', filters.observer);
if (filters.channel) params.set('channel', filters.channel);
if (groupByHash) {
params.set('groupByHash', 'true');
} else {
params.set('expand', 'observations');
}
return params;
}
async function loadPackets() {
try {
const selectedWindow = Number(document.getElementById('fTimeWindow')?.value);
const windowMin = Number.isFinite(selectedWindow) ? selectedWindow : savedTimeWindowMin;
const params = buildPacketsParams({
filters,
regionParam: RegionFilter.getRegionParam(),
windowMin,
groupByHash,
limit: PACKET_LIMIT,
});
const data = await api('/packets?' + params.toString());
packets = data.packets || [];
@@ -683,10 +710,14 @@
// Restore expanded group children (parallel fetch, Map lookup)
if (groupByHash && expandedHashes.size > 0) {
const expandedArr = [...expandedHashes];
// Fetch the full packet detail (which includes per-observation rows) for each expanded hash.
// Previously this used `/packets?hash=X&limit=20` which returned ONE aggregate row, causing
// every "child" row in the table to carry the parent packet.id instead of unique observation
// ids — so clicking any child pointed the side pane at the same aggregate. See #866.
const results = await Promise.all(expandedArr.map(hash => {
const group = hashIndex.get(hash);
if (!group) return { hash, group: null, data: null };
return api(`/packets?hash=${hash}&limit=20`)
return api(`/packets/${hash}`)
.then(data => ({ hash, group, data }))
.catch(() => ({ hash, group, data: null }));
}));
@@ -694,7 +725,15 @@
if (!group) {
expandedHashes.delete(hash);
} else if (data) {
group._children = data.packets || [];
const pkt = data.packet || group;
// Build per-observation children. Spread (pkt, obs) so obs-level fields
// (id, observer_id/name, path_json, snr/rssi, timestamp, raw_hex) override
// the aggregate. Each child's `id` is the observation id (unique per observer).
const obs = data.observations || [];
group._children = obs.length
? obs.map(o => clearParsedCache({...pkt, ...o, _isObservation: true}))
: [pkt];
group._fetchedData = { packet: pkt, observations: obs };
sortGroupChildren(group);
}
}
@@ -1248,7 +1287,7 @@
const parentData = group._fetchedData;
const obsPacket = parentData ? {...parentData.packet, observer_id: child.observer_id, observer_name: child.observer_name, snr: child.snr, rssi: child.rssi, path_json: child.path_json, resolved_path: child.resolved_path, direction: child.direction, timestamp: child.timestamp, first_seen: child.timestamp} : child;
if (parentData) { clearParsedCache(obsPacket); }
selectPacket(child.id, parentHash, {packet: obsPacket, breakdown: parentData?.breakdown, observations: parentData?.observations}, child.id);
selectPacket(child.id, parentHash, {packet: obsPacket, observations: parentData?.observations}, child.id);
}
}
else if (action === 'select-hash') pktSelectHash(value);
@@ -1635,7 +1674,14 @@
// Filter to claimed/favorited nodes — pure client-side filter (no server round-trip)
let displayPackets = packets;
if (filters.myNodes) {
// When loading a specific packet by hash, bypass ALL client-side filters
// (myNodes, type, observer, packet-filter-expression). The user is asking
// for THAT exact packet — saved type/observer/expression filters must not
// hide it. Hash filter is the exact identifier; nothing else applies.
const hashOnly = !!filters.hash;
if (!hashOnly && filters.myNodes) {
const myNodes = JSON.parse(localStorage.getItem('meshcore-my-nodes') || '[]');
const myKeys = myNodes.map(n => n.pubkey).filter(Boolean);
const favs = getFavorites();
@@ -1651,11 +1697,11 @@
}
// Client-side type/observer filtering
if (filters.type) {
if (!hashOnly && filters.type) {
const types = filters.type.split(',').map(Number);
displayPackets = displayPackets.filter(p => types.includes(p.payload_type));
}
if (filters.observer) {
if (!hashOnly && filters.observer) {
const obsIds = new Set(filters.observer.split(','));
displayPackets = displayPackets.filter(p => {
if (obsIds.has(p.observer_id)) return true;
@@ -1666,7 +1712,7 @@
// Packet Filter Language
const pfCount = document.getElementById('packetFilterCount');
if (filters._packetFilter) {
if (!hashOnly && filters._packetFilter) {
const beforeCount = displayPackets.length;
displayPackets = displayPackets.filter(filters._packetFilter);
if (pfCount) {
@@ -1806,8 +1852,6 @@
async function renderDetail(panel, data, chosenObsId) {
const pkt = data.packet;
const breakdown = data.breakdown || {};
const ranges = breakdown.ranges || [];
const observations = data.observations || [];
// Per-observation rendering (issue #849):
@@ -1828,6 +1872,15 @@
const decoded = getParsedDecoded(effectivePkt) || {};
const pathHops = getParsedPath(effectivePkt) || [];
// Compute breakdown ranges from the actually-rendered raw_hex (per-observation).
// Single source of truth — derived from the same bytes we display, so a
// post-#882 per-obs raw_hex with a different path length than the top-level
// packet's raw_hex still gets accurate byte highlights.
const obsRawHexForRanges = effectivePkt.raw_hex || pkt.raw_hex || '';
const ranges = obsRawHexForRanges
? computeBreakdownRanges(obsRawHexForRanges, pkt.route_type, pkt.payload_type)
: [];
// Cross-check: hop count from raw_hex path_len byte vs path_json length
const obsRawHex = effectivePkt.raw_hex || pkt.raw_hex || '';
let rawHopCount = null;
@@ -1838,7 +1891,7 @@
if (!isNaN(plByte)) rawHopCount = plByte & 0x3F;
}
if (rawHopCount != null && pathHops.length !== rawHopCount) {
console.warn(`[CoreScope] Hop count inconsistency for packet ${pkt.hash}: path_json has ${pathHops.length} hops but raw_hex path_len has ${rawHopCount}. Trusting raw_hex.`);
console.warn(`[CoreScope] Hop count inconsistency for packet ${pkt.hash}: path_json has ${pathHops.length} hops but raw_hex path_len has ${rawHopCount}. UI shows path_json.`);
}
// Resolve sender GPS — from packet directly, or from known node in DB
@@ -1975,8 +2028,10 @@
? `<div class="anomaly-banner" style="background:var(--warning, #f0ad4e); color:#000; padding:8px 12px; border-radius:4px; margin-bottom:8px; font-weight:600;">⚠️ Anomaly: ${escapeHtml(decoded.anomaly)}</div>`
: '';
// Hop count display: trust raw_hex (firmware truth) over path_json
const displayHopCount = rawHopCount != null ? rawHopCount : pathHops.length;
// Hop count display: use pathHops length (= effective observation's path_json).
// The raw_hex/path_json mismatch warning is logged above for diagnostics; the UI
// must stay self-consistent — top pill names and byte breakdown rows must agree.
const displayHopCount = pathHops.length;
const obsIndicator = currentObs && observations.length > 1
? `<span style="font-size:0.8em;color:var(--text-muted);margin-left:6px">(observation ${observations.indexOf(currentObs) + 1} of ${observations.length})</span>`
: '';
@@ -2181,18 +2236,19 @@
rows += fieldRow(off, 'Path Length', '0x' + (buf.slice(off * 2, off * 2 + 2) || '??'), hashCountVal === 0 ? `hash_count=0 (direct advert)` : `hash_size=${hashSizeVal} byte${hashSizeVal !== 1 ? 's' : ''}, hash_count=${hashCountVal}`);
off += 1;
// Path — derive hop count from path_len byte (firmware truth), not aggregated _parsedPath
// Path — render hops from path_json (what this observation reported).
// Byte offsets advance by hashSize * pathHops.length to match.
const hashSize = isNaN(pathByte0) ? 1 : ((pathByte0 >> 6) + 1);
if (typeof hashCountVal === 'number' && hashCountVal > 0) {
rows += sectionRow('Path (' + hashCountVal + ' hops)', 'section-path');
for (let i = 0; i < hashCountVal; i++) {
if (pathHops.length > 0) {
rows += sectionRow('Path (' + pathHops.length + ' hops)', 'section-path');
for (let i = 0; i < pathHops.length; i++) {
const hopOff = off + i * hashSize;
const hex = buf.slice(hopOff * 2, (hopOff + hashSize) * 2).toUpperCase();
const hex = String(pathHops[i] || '').toUpperCase();
const hopHtml = HopDisplay.renderHop(hex, hopNameCache[hex]);
const label = `Hop ${i}${hopHtml}`;
rows += fieldRow(hopOff, label, hex, '');
}
off += hashSize * hashCountVal;
off += hashSize * pathHops.length;
}
// Payload
@@ -2466,7 +2522,7 @@
renderTableRows();
return;
}
// Single fetch — gets packet + observations + path + breakdown
// Single fetch — gets packet + observations + path
try {
const data = await api(`/packets/${hash}`);
const pkt = data.packet;
@@ -2541,6 +2597,7 @@
buildGroupRowHtml,
buildFlatRowHtml,
_calcVisibleRange,
buildPacketsParams,
};
}
+202
View File
@@ -0,0 +1,202 @@
// Path Inspector — prefix candidate scoring with map overlay (issue #944).
// IIFE; exports window.PathInspector for testability.
(function () {
'use strict';
var container = null;
var currentResults = null;
function init(app) {
container = app;
var params = new URLSearchParams(location.hash.split('?')[1] || '');
var prefixParam = params.get('prefixes') || '';
container.innerHTML =
'<div class="path-inspector-page">' +
'<h2>Path Inspector</h2>' +
'<p class="help-text">Enter comma or space-separated hex prefixes (1-3 bytes each, e.g. <code>2C,A1,F4</code> or <code>2C A1 F4</code>).</p>' +
'<div class="path-inspector-input-row">' +
'<input type="text" id="path-inspector-input" class="input" placeholder="2C,A1,F4 or 2C A1 F4" value="' + escapeAttr(prefixParam) + '">' +
'<button id="path-inspector-submit" class="btn btn-primary">Inspect</button>' +
'</div>' +
'<div id="path-inspector-error" class="path-inspector-error"></div>' +
'<div id="path-inspector-results"></div>' +
'</div>';
var input = document.getElementById('path-inspector-input');
var btn = document.getElementById('path-inspector-submit');
btn.addEventListener('click', function () { submit(input.value); });
input.addEventListener('keydown', function (e) {
if (e.key === 'Enter') submit(input.value);
});
// Auto-run if prefixes in URL.
if (prefixParam) submit(prefixParam);
}
function destroy() {
container = null;
currentResults = null;
}
function parsePrefixes(raw) {
// Accept comma or space separated.
var parts = raw.trim().split(/[\s,]+/).filter(function (s) { return s.length > 0; });
return parts.map(function (p) { return p.toLowerCase(); });
}
function validatePrefixes(prefixes) {
if (prefixes.length === 0) return 'Enter at least one prefix.';
if (prefixes.length > 64) return 'Too many prefixes (max 64).';
var hexRe = /^[0-9a-f]+$/;
var byteLen = -1;
for (var i = 0; i < prefixes.length; i++) {
var p = prefixes[i];
if (!hexRe.test(p)) return 'Invalid hex: ' + p;
if (p.length % 2 !== 0) return 'Odd-length prefix: ' + p;
var bl = p.length / 2;
if (bl > 3) return 'Prefix too long (max 3 bytes): ' + p;
if (byteLen === -1) byteLen = bl;
else if (bl !== byteLen) return 'Mixed prefix lengths not allowed.';
}
return null;
}
function submit(raw) {
var errDiv = document.getElementById('path-inspector-error');
var resultsDiv = document.getElementById('path-inspector-results');
errDiv.textContent = '';
resultsDiv.innerHTML = '';
var prefixes = parsePrefixes(raw);
var err = validatePrefixes(prefixes);
if (err) {
errDiv.textContent = err;
return;
}
// Update URL.
var base = '#/tools/path-inspector';
if (location.hash.indexOf(base) === 0) {
history.replaceState(null, '', base + '?prefixes=' + prefixes.join(','));
}
resultsDiv.innerHTML = '<p>Loading...</p>';
fetch('/api/paths/inspect', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ prefixes: prefixes })
})
.then(function (r) {
if (r.status === 503) return r.json().then(function (d) { throw new Error('Service warming up, retry in a few seconds.'); });
if (!r.ok) return r.json().then(function (d) { throw new Error(d.error || 'Request failed'); });
return r.json();
})
.then(function (data) {
currentResults = data;
renderResults(data, resultsDiv);
})
.catch(function (e) {
resultsDiv.innerHTML = '';
errDiv.textContent = e.message;
});
}
function renderResults(data, div) {
if (!data.candidates || data.candidates.length === 0) {
div.innerHTML = '<p class="no-results">No candidates found. The prefixes may not match any known path-eligible nodes.</p>';
return;
}
var html = '<table class="path-inspector-table"><thead><tr>' +
'<th>#</th><th>Score</th><th>Path</th><th>Action</th>' +
'</tr></thead><tbody>';
for (var i = 0; i < data.candidates.length; i++) {
var c = data.candidates[i];
var rowClass = c.speculative ? 'speculative-row' : '';
html += '<tr class="' + rowClass + '">';
html += '<td>' + (i + 1) + '</td>';
html += '<td class="' + (c.speculative ? 'speculative-warning' : '') + '">' +
c.score.toFixed(3) +
(c.speculative ? ' <span class="speculative-badge" title="Low evidence; may be wrong">⚠</span>' : '') +
'</td>';
html += '<td>' + escapeHtml(c.names.join(' → ')) + '</td>';
html += '<td><button class="btn btn-sm" data-idx="' + i + '">Show on Map</button></td>';
html += '</tr>';
// Per-hop evidence (collapsed).
html += '<tr class="evidence-row collapsed" data-evidence="' + i + '"><td colspan="4"><div class="evidence-detail">';
for (var j = 0; j < c.evidence.perHop.length; j++) {
var h = c.evidence.perHop[j];
html += '<div class="hop-evidence">Hop ' + (j + 1) + ': prefix=' + h.prefix +
', candidates=' + h.candidatesConsidered +
', edge=' + h.edgeWeight.toFixed(3);
if (h.alternatives && h.alternatives.length > 0) {
html += '<div class="hop-alternatives" style="margin-left:12px;font-size:12px;color:var(--text-muted);">';
for (var k = 0; k < h.alternatives.length; k++) {
var alt = h.alternatives[k];
html += '<div>↳ ' + escapeHtml(alt.name || alt.publicKey.substring(0, 8)) + ' (score=' + alt.score.toFixed(3) + ')</div>';
}
html += '</div>';
}
html += '</div>';
}
html += '</div></td></tr>';
}
html += '</tbody></table>';
html += '<div class="path-inspector-stats">Beam width: ' + data.stats.beamWidth +
' | Expansions: ' + data.stats.expansionsRun +
' | Elapsed: ' + data.stats.elapsedMs + 'ms</div>';
div.innerHTML = html;
// Wire up Show on Map buttons.
div.querySelectorAll('button[data-idx]').forEach(function (btn) {
btn.addEventListener('click', function () {
var idx = parseInt(btn.dataset.idx);
showOnMap(data.candidates[idx]);
});
});
// Wire up row expand for evidence.
div.querySelectorAll('.path-inspector-table tbody tr:not(.evidence-row)').forEach(function (row) {
row.style.cursor = 'pointer';
row.addEventListener('click', function (e) {
if (e.target.tagName === 'BUTTON') return;
var idx = row.querySelector('button[data-idx]');
if (!idx) return;
var evidenceRow = div.querySelector('tr[data-evidence="' + idx.dataset.idx + '"]');
if (evidenceRow) evidenceRow.classList.toggle('collapsed');
});
});
}
function showOnMap(candidate) {
// Store pending route for map init to pick up.
window._pendingPathInspectorRoute = candidate;
// Switch to map page if not there; map init will draw the route.
if (location.hash.indexOf('#/map') !== 0) {
location.hash = '#/map';
} else {
// Already on map — draw directly.
delete window._pendingPathInspectorRoute;
if (window.routeLayer) window.routeLayer.clearLayers();
var hops = candidate.path.slice(1);
var origin = candidate.path[0] || null;
if (window.drawPacketRoute) window.drawPacketRoute(hops, origin);
}
}
function escapeAttr(s) {
return s.replace(/&/g, '&amp;').replace(/"/g, '&quot;').replace(/</g, '&lt;');
}
function escapeHtml(s) {
return s.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');
}
window.PathInspector = { init: init, destroy: destroy, parsePrefixes: parsePrefixes, validatePrefixes: validatePrefixes };
if (typeof registerPage === 'function') registerPage('path-inspector', { init: init, destroy: destroy });
})();
+11 -19
View File
@@ -397,17 +397,16 @@
// #690 — Clock Skew shared helpers
var SKEW_SEVERITY_COLORS = {
default: 'var(--text-muted)',
ok: 'var(--status-green)',
warning: 'var(--status-yellow)',
critical: 'var(--status-orange)',
absurd: 'var(--status-purple)',
bimodal_clock: 'var(--status-amber)',
no_clock: 'var(--text-muted)'
degrading: 'var(--status-yellow)',
degraded: 'var(--status-orange)',
wrong: 'var(--status-red)'
};
var SKEW_SEVERITY_LABELS = {
ok: 'OK', warning: 'Warning', critical: 'Critical', absurd: 'Absurd', bimodal_clock: 'Bimodal', no_clock: 'No Clock'
default: 'Default', ok: 'OK', degrading: 'Degrading', degraded: 'Degraded', wrong: 'Wrong'
};
var SKEW_SEVERITY_ORDER = { no_clock: 0, bimodal_clock: 1, absurd: 2, critical: 3, warning: 4, ok: 5 };
var SKEW_SEVERITY_ORDER = { default: 0, wrong: 1, degraded: 2, degrading: 3, ok: 4 };
window.SKEW_SEVERITY_COLORS = SKEW_SEVERITY_COLORS;
window.SKEW_SEVERITY_LABELS = SKEW_SEVERITY_LABELS;
@@ -430,26 +429,19 @@
return (secPerDay >= 0 ? '+' : '') + secPerDay.toFixed(1) + ' s/day';
};
/** Pick the skew value that drives current-health UI: prefer the
* recent-window median (#789, current health) over the all-time median
* (poisoned by historical bad samples). Falls back gracefully if the
* field isn't present (older API responses). */
/** Pick the skew value that drives current-health UI. Uses lastSkewSec
* (most recent corrected skew) when available, falls back to medianSkewSec. */
window.currentSkewValue = function(cs) {
if (!cs) return null;
return cs.recentMedianSkewSec != null ? cs.recentMedianSkewSec : cs.medianSkewSec;
return cs.lastSkewSec != null ? cs.lastSkewSec : cs.medianSkewSec;
};
/** Render a clock skew badge HTML */
window.renderSkewBadge = function(severity, skewSec, cs) {
if (!severity) return '';
var cls = 'skew-badge skew-badge--' + severity;
if (severity === 'no_clock') {
return '<span class="' + cls + '" title="Uninitialized RTC — no valid clock">🚫 No Clock</span>';
}
if (severity === 'bimodal_clock' && cs) {
var badPct = cs.goodFraction != null ? Math.round((1 - cs.goodFraction) * 100) : '?';
var label = '⏰ ' + window.formatSkew(skewSec);
return '<span class="' + cls + '" title="Clock skew: ' + window.formatSkew(skewSec) + ' (bimodal: ' + badPct + '% of recent adverts have nonsense timestamps)">' + label + '</span>';
if (severity === 'default') {
return '<span class="' + cls + '" title="Firmware default clock — volatile RTC not yet user-set since boot">⏰ Default</span>';
}
var label = severity === 'ok' ? '⏰' : '⏰ ' + window.formatSkew(skewSec);
return '<span class="' + cls + '" title="Clock skew: ' + window.formatSkew(skewSec) + ' (' + (SKEW_SEVERITY_LABELS[severity] || severity) + ')">' + label + '</span>';
+44 -9
View File
@@ -16,6 +16,7 @@
--status-amber: #f59e0b;
--status-amber-light: #fef3c7;
--status-amber-text: #92400e;
--path-inspector-speculative: #d97706;
--role-observer: #8b5cf6;
--accent-hover: #6db3ff;
--text: #1a1a2e;
@@ -52,6 +53,7 @@
--status-amber: #f59e0b;
--status-amber-light: #422006;
--status-amber-text: #fcd34d;
--path-inspector-speculative: #f59e0b;
--surface-0: #0f0f23;
--surface-1: #1a1a2e;
--surface-2: #232340;
@@ -2291,22 +2293,55 @@ th.sort-active { color: var(--accent, #60a5fa); }
/* #690 — Clock Skew badges & fleet table */
.skew-badge { display: inline-block; font-size: 10px; padding: 1px 5px; border-radius: 3px; margin-left: 4px; font-weight: 600; white-space: nowrap; }
.skew-badge--default { background: var(--text-muted); color: #fff; }
.skew-badge--ok { background: var(--status-green); color: #fff; }
.skew-badge--warning { background: var(--status-yellow); color: #000; }
.skew-badge--critical { background: var(--status-orange); color: #fff; }
.skew-badge--absurd { background: var(--status-purple); color: #fff; }
.skew-badge--no_clock { background: var(--text-muted); color: #fff; }
.skew-badge--bimodal_clock { background: var(--status-amber-light); color: var(--status-amber-text); border: 1px solid var(--status-amber); }
.skew-badge--degrading { background: var(--status-yellow); color: #000; }
.skew-badge--degraded { background: var(--status-orange); color: #fff; }
.skew-badge--wrong { background: var(--status-red); color: #fff; }
.skew-detail-section { padding: 10px 16px; margin-bottom: 8px; }
.skew-sparkline-wrap { margin-top: 6px; }
.skew-sparkline-wrap svg { display: block; }
.clock-fleet-row--warning { background: color-mix(in srgb, var(--status-yellow) 10%, transparent); }
.clock-fleet-row--critical { background: color-mix(in srgb, var(--status-orange) 10%, transparent); }
.clock-fleet-row--absurd { background: color-mix(in srgb, var(--status-purple) 10%, transparent); }
.clock-fleet-row--no_clock { background: color-mix(in srgb, var(--text-muted) 10%, transparent); }
.clock-fleet-row--degrading { background: color-mix(in srgb, var(--status-yellow) 10%, transparent); }
.clock-fleet-row--degraded { background: color-mix(in srgb, var(--status-orange) 10%, transparent); }
.clock-fleet-row--wrong { background: color-mix(in srgb, var(--status-red) 10%, transparent); }
.clock-fleet-row--default { background: color-mix(in srgb, var(--text-muted) 10%, transparent); }
.clock-filter-btn { font-size: 12px; padding: 3px 8px; border: 1px solid var(--border); border-radius: 4px; background: var(--card-bg, #fff); color: var(--text); cursor: pointer; margin-right: 4px; }
.clock-filter-btn.active { background: var(--accent); color: #fff; border-color: var(--accent); }
/* === Path Inspector (issue #944) === */
.path-inspector-page { padding: 16px; max-width: 900px; margin: 0 auto; }
.path-inspector-input-row { display: flex; gap: 8px; margin-bottom: 12px; }
.path-inspector-input-row .input { flex: 1; }
.path-inspector-error { color: var(--status-red, #ef4444); font-size: 13px; margin-bottom: 8px; }
.path-inspector-table { width: 100%; border-collapse: collapse; font-size: 13px; }
.path-inspector-table th,
.path-inspector-table td { padding: 6px 10px; border-bottom: 1px solid var(--border); text-align: left; }
.path-inspector-table th { background: var(--card-bg); font-weight: 600; }
.speculative-warning { color: var(--path-inspector-speculative, #d97706); font-weight: 600; }
.speculative-badge { cursor: help; }
.speculative-row { background: color-mix(in srgb, var(--path-inspector-speculative, #d97706) 8%, transparent); }
.evidence-row { font-size: 12px; color: var(--text-muted); }
.evidence-row.collapsed { display: none; }
.evidence-detail { padding: 4px 10px; }
.hop-evidence { margin: 2px 0; }
.path-inspector-stats { margin-top: 12px; font-size: 12px; color: var(--text-muted); }
.no-results { color: var(--text-muted); font-style: italic; }
/* Map side pane for path inspector */
.map-side-pane { flex: 0 0 32px; overflow: hidden; transition: flex-basis 0.2s; border-left: 1px solid var(--border); background: var(--card-bg); }
.map-side-pane.expanded { flex: 0 0 320px; overflow-y: auto; padding: 12px; }
.map-side-pane .pane-toggle { cursor: pointer; padding: 8px; font-size: 14px; text-align: center; }
.map-side-pane .pane-content { display: none; }
.map-side-pane.expanded .pane-content { display: block; }
/* Tools landing page */
.tools-landing { padding: 24px; max-width: 600px; }
.tools-menu { display: flex; flex-direction: column; gap: 12px; margin-top: 16px; }
.tools-card { display: block; padding: 16px; border-radius: 8px; border: 1px solid var(--border); background: var(--card-bg); color: var(--text); text-decoration: none; transition: border-color 0.2s; }
.tools-card:hover { border-color: var(--primary); }
.tools-card h3 { margin: 0 0 4px 0; font-size: 16px; }
.tools-card p { margin: 0; font-size: 13px; color: var(--text-muted); }
-111
View File
@@ -59,118 +59,7 @@ test('null lastSeenMs → stale', () => assert.strictEqual(getNodeStatus('repeat
test('undefined lastSeenMs → stale', () => assert.strictEqual(getNodeStatus('repeater', undefined), 'stale'));
test('0 lastSeenMs → stale', () => assert.strictEqual(getNodeStatus('repeater', 0), 'stale'));
// === getStatusInfo tests (inline since nodes.js has too many DOM deps) ===
console.log('\n=== getStatusInfo (logic validation) ===');
// Simulate getStatusInfo logic
function mockGetStatusInfo(n) {
const ROLE_COLORS = ctx.window.ROLE_COLORS;
const role = (n.role || '').toLowerCase();
const roleColor = ROLE_COLORS[n.role] || '#6b7280';
const lastHeardTime = n._lastHeard || n.last_heard || n.last_seen;
const lastHeardMs = lastHeardTime ? new Date(lastHeardTime).getTime() : 0;
const status = getNodeStatus(role, lastHeardMs);
const statusLabel = status === 'active' ? '🟢 Active' : '⚪ Stale';
const isInfra = role === 'repeater' || role === 'room';
let explanation = '';
if (status === 'active') {
explanation = 'Last heard recently';
} else {
const reason = isInfra
? 'repeaters typically advertise every 12-24h'
: 'companions only advertise when user initiates, this may be normal';
explanation = 'Not heard — ' + reason;
}
return { status, statusLabel, roleColor, explanation, role };
}
test('active repeater → 🟢 Active, red color', () => {
const info = mockGetStatusInfo({ role: 'repeater', last_seen: new Date(now - 1*h).toISOString() });
assert.strictEqual(info.status, 'active');
assert.strictEqual(info.statusLabel, '🟢 Active');
assert.strictEqual(info.roleColor, '#dc2626');
});
test('stale companion → ⚪ Stale, explanation mentions "this may be normal"', () => {
const info = mockGetStatusInfo({ role: 'companion', last_seen: new Date(now - 25*h).toISOString() });
assert.strictEqual(info.status, 'stale');
assert.strictEqual(info.statusLabel, '⚪ Stale');
assert(info.explanation.includes('this may be normal'), 'should mention "this may be normal"');
});
test('missing last_seen → stale', () => {
const info = mockGetStatusInfo({ role: 'repeater' });
assert.strictEqual(info.status, 'stale');
});
test('missing role → defaults to empty string, uses node threshold', () => {
const info = mockGetStatusInfo({ last_seen: new Date(now - 25*h).toISOString() });
assert.strictEqual(info.status, 'stale');
assert.strictEqual(info.roleColor, '#6b7280');
});
test('prefers last_heard over last_seen', () => {
// last_seen is stale, but last_heard is recent
const info = mockGetStatusInfo({
role: 'companion',
last_seen: new Date(now - 48*h).toISOString(),
last_heard: new Date(now - 1*h).toISOString()
});
assert.strictEqual(info.status, 'active');
});
// === getStatusTooltip tests ===
console.log('\n=== getStatusTooltip ===');
// Load from nodes.js by extracting the function
// Since nodes.js is complex, I'll re-implement the tooltip function for testing
function getStatusTooltip(role, status) {
const isInfra = role === 'repeater' || role === 'room';
const threshold = isInfra ? '72h' : '24h';
if (status === 'active') {
return 'Active — heard within the last ' + threshold + '.' + (isInfra ? ' Repeaters typically advertise every 12-24h.' : '');
}
if (role === 'companion') {
return 'Stale — not heard for over ' + threshold + '. Companions only advertise when the user initiates — this may be normal.';
}
if (role === 'sensor') {
return 'Stale — not heard for over ' + threshold + '. This sensor may be offline.';
}
return 'Stale — not heard for over ' + threshold + '. This ' + role + ' may be offline or out of range.';
}
test('active repeater mentions "72h" and "advertise every 12-24h"', () => {
const tip = getStatusTooltip('repeater', 'active');
assert(tip.includes('72h'), 'should mention 72h');
assert(tip.includes('advertise every 12-24h'), 'should mention advertise frequency');
});
test('active companion mentions "24h"', () => {
const tip = getStatusTooltip('companion', 'active');
assert(tip.includes('24h'), 'should mention 24h');
});
test('stale companion mentions "24h" and "user initiates"', () => {
const tip = getStatusTooltip('companion', 'stale');
assert(tip.includes('24h'), 'should mention 24h');
assert(tip.includes('user initiates'), 'should mention user initiates');
});
test('stale repeater mentions "offline or out of range"', () => {
const tip = getStatusTooltip('repeater', 'stale');
assert(tip.includes('offline or out of range'), 'should mention offline or out of range');
});
test('stale sensor mentions "sensor may be offline"', () => {
const tip = getStatusTooltip('sensor', 'stale');
assert(tip.includes('sensor may be offline'));
});
test('stale room uses 72h threshold', () => {
const tip = getStatusTooltip('room', 'stale');
assert(tip.includes('72h'));
});
// === Bug check: renderRows uses last_seen instead of last_heard || last_seen ===
console.log('\n=== BUG CHECK ===');
-123
View File
@@ -1,123 +0,0 @@
/**
* test-anim-perf.js Performance benchmark for animation timer management
*
* Demonstrates that the rAF + concurrency-cap approach keeps active animation
* count bounded, whereas the old setInterval approach accumulated without limit.
*
* Run: node test-anim-perf.js
*/
'use strict';
let passed = 0, failed = 0;
function assert(cond, msg) {
if (cond) { console.log(`${msg}`); passed++; }
else { console.log(`${msg}`); failed++; }
}
// ---------------------------------------------------------------------------
// Simulate OLD behaviour: setInterval-based, no concurrency cap
// ---------------------------------------------------------------------------
function simulateOldModel(packetsPerSec, hopsPerPacket, durationSec) {
// Each hop spawns 3 intervals (pulse 26ms, line 33ms, fade 52ms).
// Pulse lasts ~2s, line ~0.66s, fade ~0.8s+0.4s ≈ 1.2s
// At any moment, timers from the last ~2s of packets are still alive.
const intervalLifetimes = [2.0, 0.66, 1.2]; // seconds each interval lives
let maxConcurrent = 0;
// Walk through time in 0.1s steps
const dt = 0.1;
const spawns = []; // {time, lifetime}
for (let t = 0; t < durationSec; t += dt) {
// Spawn timers for packets arriving in this window
const pktsInWindow = packetsPerSec * dt;
for (let p = 0; p < pktsInWindow; p++) {
for (let h = 0; h < hopsPerPacket; h++) {
for (const lt of intervalLifetimes) {
spawns.push({ time: t, lifetime: lt });
}
}
}
// Count alive timers
const alive = spawns.filter(s => t < s.time + s.lifetime).length;
if (alive > maxConcurrent) maxConcurrent = alive;
}
return maxConcurrent;
}
// ---------------------------------------------------------------------------
// Simulate NEW behaviour: rAF + MAX_CONCURRENT_ANIMS cap
// ---------------------------------------------------------------------------
function simulateNewModel(packetsPerSec, hopsPerPacket, durationSec) {
const MAX_CONCURRENT_ANIMS = 20;
let activeAnims = 0;
let maxConcurrent = 0;
const anims = []; // {endTime}
const dt = 0.1;
for (let t = 0; t < durationSec; t += dt) {
// Expire finished animations
while (anims.length && anims[0].endTime <= t) {
anims.shift();
activeAnims--;
}
// Try to start new animations
const pktsInWindow = packetsPerSec * dt;
for (let p = 0; p < pktsInWindow; p++) {
if (activeAnims >= MAX_CONCURRENT_ANIMS) break; // cap reached — drop
activeAnims++;
// rAF animation lifetime: longest is pulse ~2s
anims.push({ endTime: t + 2.0 });
}
// Sort by endTime so expiry works
anims.sort((a, b) => a.endTime - b.endTime);
if (activeAnims > maxConcurrent) maxConcurrent = activeAnims;
}
return maxConcurrent;
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
console.log('\n=== Animation timer accumulation: old vs new ===');
// Scenario: 5 pkts/sec, 3 hops each, 30 seconds
const oldPeak30s = simulateOldModel(5, 3, 30);
const newPeak30s = simulateNewModel(5, 3, 30);
console.log(` Old model (30s @ 5pkt/s×3hops): peak ${oldPeak30s} concurrent timers`);
console.log(` New model (30s @ 5pkt/s×3hops): peak ${newPeak30s} concurrent animations`);
assert(oldPeak30s > 100, `old model accumulates >100 timers (got ${oldPeak30s})`);
assert(newPeak30s <= 20, `new model stays ≤20 (got ${newPeak30s})`);
// Scenario: 5 minutes sustained
const oldPeak5m = simulateOldModel(5, 3, 300);
const newPeak5m = simulateNewModel(5, 3, 300);
console.log(` Old model (5min @ 5pkt/s×3hops): peak ${oldPeak5m} concurrent timers`);
console.log(` New model (5min @ 5pkt/s×3hops): peak ${newPeak5m} concurrent animations`);
assert(oldPeak5m > 100, `old model at 5min still unbounded (got ${oldPeak5m})`);
assert(newPeak5m <= 20, `new model at 5min still ≤20 (got ${newPeak5m})`);
// Scenario: burst — 20 pkts/sec for 10s
const oldBurst = simulateOldModel(20, 3, 10);
const newBurst = simulateNewModel(20, 3, 10);
console.log(` Old model (burst 20pkt/s×3hops, 10s): peak ${oldBurst} concurrent timers`);
console.log(` New model (burst 20pkt/s×3hops, 10s): peak ${newBurst} concurrent animations`);
assert(oldBurst > 200, `old model under burst >200 timers (got ${oldBurst})`);
assert(newBurst <= 20, `new model under burst stays ≤20 (got ${newBurst})`);
console.log('\n=== drawAnimatedLine frame-drop catch-up ===');
// Read the source and verify catch-up logic exists
const fs = require('fs');
const src = fs.readFileSync(__dirname + '/public/live.js', 'utf8');
// Extract the animateLine function body
const lineMatch = src.match(/function animateLine\(now\)\s*\{[\s\S]*?requestAnimationFrame\(animateLine\)/);
assert(lineMatch && /Math\.min\(Math\.floor\(elapsed\s*\/\s*33\)/.test(lineMatch[0]),
'drawAnimatedLine catches up on frame drops (multi-tick per frame)');
const fadeMatch = src.match(/function animateFade\(now\)\s*\{[\s\S]*?requestAnimationFrame\(animateFade\)/);
assert(fadeMatch && /Math\.min\(Math\.floor\(fadeElapsed\s*\/\s*52\)/.test(fadeMatch[0]),
'animateFade catches up on frame drops (multi-tick per frame)');
console.log(`\n${passed} passed, ${failed} failed\n`);
process.exit(failed ? 1 : 0);
-64
View File
@@ -1,64 +0,0 @@
/**
* Tests for #759 Add channel UX: button, hint, status feedback.
* Validates the HTML structure rendered by channels.js init.
*/
'use strict';
const fs = require('fs');
let passed = 0;
let failed = 0;
function assert(cond, msg) {
if (cond) { passed++; console.log(' ✓ ' + msg); }
else { failed++; console.error(' ✗ ' + msg); }
}
function assertIncludes(html, substr, msg) {
assert(html.includes(substr), msg);
}
// Read the channels.js source to extract the HTML template
const src = fs.readFileSync(__dirname + '/public/channels.js', 'utf8');
// Extract the sidebar HTML from the template literal
const htmlMatch = src.match(/app\.innerHTML\s*=\s*`([\s\S]*?)`;/);
const html = htmlMatch ? htmlMatch[1] : '';
console.log('Test: Add channel UX (#759)');
// 1. Button renders in the form
assertIncludes(html, 'class="ch-add-btn"', 'Add button has ch-add-btn class');
assertIncludes(html, 'type="submit"', 'Button is type=submit');
assertIncludes(html, '>+</button>', 'Button shows + text');
// 2. Form has proper structure
assertIncludes(html, 'class="ch-add-form"', 'Form has ch-add-form class');
assertIncludes(html, 'class="ch-add-row"', 'Row wrapper present');
assert(!html.includes('class="ch-add-label"'), 'Label removed (redundant with hint)');
// 3. Hint text present
assertIncludes(html, 'class="ch-add-hint"', 'Hint div present');
assertIncludes(html, 'e.g. #LongFast or 32-char hex key', 'Hint text correct');
// 4. Status div present
assertIncludes(html, 'id="chAddStatus"', 'Status div has correct id');
assertIncludes(html, 'class="ch-add-status"', 'Status div has correct class');
assertIncludes(html, 'style="display:none"', 'Status div hidden by default');
// 5. showAddStatus function exists in source
assert(src.includes('function showAddStatus('), 'showAddStatus function defined');
assert(src.includes("'success'"), 'Success status type referenced');
assert(src.includes("'error'"), 'Error status type referenced');
// 6. CSS classes exist
const css = fs.readFileSync(__dirname + '/public/style.css', 'utf8');
assert(css.includes('.ch-add-form'), 'CSS: .ch-add-form defined');
assert(css.includes('.ch-add-btn'), 'CSS: .ch-add-btn defined');
assert(css.includes('.ch-add-hint'), 'CSS: .ch-add-hint defined');
assert(css.includes('.ch-add-status'), 'CSS: .ch-add-status defined');
assert(css.includes('.ch-add-row'), 'CSS: .ch-add-row defined');
// .ch-add-label CSS kept for backward compat but label removed from HTML
console.log('\n' + passed + ' passed, ' + failed + ' failed');
process.exit(failed > 0 ? 1 : 0);
+378 -14
View File
@@ -15,6 +15,11 @@ async function test(name, fn) {
results.push({ name, pass: true });
console.log(` \u2705 ${name}`);
} catch (err) {
if (err.skip) {
results.push({ name, pass: true, skipped: true });
console.log(`${name}: ${err.message}`);
return;
}
results.push({ name, pass: false, error: err.message });
console.log(` \u274c ${name}: ${err.message}`);
console.log(`\nFail-fast: stopping after first failure.`);
@@ -219,10 +224,7 @@ async function run() {
// Test 5: Node detail loads (reuses nodes page from test 2)
await test('Node detail loads', async () => {
await page.waitForSelector('table tbody tr');
// Click first row
const firstRow = await page.$('table tbody tr');
assert(firstRow, 'No node rows found');
await firstRow.click();
await page.click('table tbody tr');
// Wait for detail pane to appear
await page.waitForSelector('.node-detail');
const html = await page.content();
@@ -235,17 +237,14 @@ async function run() {
await test('Node side panel Details link navigates', async () => {
await page.goto(`${BASE}/#/nodes`, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('table tbody tr');
// Click first row to open side panel
const firstRow = await page.$('table tbody tr');
assert(firstRow, 'No node rows found');
await firstRow.click();
await page.click('table tbody tr');
await page.waitForSelector('.node-detail');
// Find the Details link in the side panel
const detailsLink = await page.$('#nodesRight a.btn-primary[href^="#/nodes/"]');
assert(detailsLink, 'Details link not found in side panel');
const href = await detailsLink.getAttribute('href');
await page.waitForSelector('#nodesRight a.btn-primary[href^="#/nodes/"]');
const href = await page.$eval('#nodesRight a.btn-primary[href^="#/nodes/"]', el => el.getAttribute('href'));
assert(href, 'Details link not found in side panel');
// Click the Details link — this should navigate to the full detail page
await detailsLink.click();
await page.click('#nodesRight a.btn-primary[href^="#/nodes/"]');
// Wait for navigation — the full detail page has sections like neighbors/packets
await page.waitForFunction((expectedHash) => {
return location.hash === expectedHash;
@@ -658,6 +657,8 @@ async function run() {
await page.waitForSelector('#ngCanvas', { timeout: 8000 });
const hasCanvas = await page.$('#ngCanvas');
assert(hasCanvas, 'Neighbor Graph tab should have a canvas element');
// Stats are populated after the async API call — wait for at least one card before counting
await page.waitForSelector('#ngStats .stat-card', { timeout: 8000 });
const hasStats = await page.$$eval('#ngStats .stat-card', els => els.length);
assert(hasStats >= 3, `Neighbor Graph stats should have >=3 cards, got ${hasStats}`);
// Verify filters exist
@@ -1353,6 +1354,38 @@ async function run() {
await page.evaluate(() => localStorage.removeItem('cs-theme-overrides'));
});
await test('Customizer v2: typing in text field does not collapse focus (re-render guard)', async () => {
await page.goto(BASE, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('nav, .navbar, .nav, [class*="nav"]');
await page.waitForFunction(() => window._customizerV2 && window._customizerV2.initDone, { timeout: 5000 });
const toggleSel = '#customizeToggle, button[title*="ustom" i], [class*="customize"]';
const btn = await page.$(toggleSel);
if (!btn) { console.log(' ⏭️ Customizer toggle not found'); return; }
await btn.click();
await page.waitForSelector('.cust-overlay', { timeout: 5000 });
const result = await page.evaluate(() => {
const input = document.querySelector('.cust-overlay input[type="text"][data-cv2-field]');
if (!input) return { skipped: true };
input.focus();
input.value = 'test';
input.dispatchEvent(new Event('input', { bubbles: true }));
const inputRef = input;
return new Promise(resolve => {
setTimeout(() => {
const panel = document.querySelector('.cust-overlay');
resolve({
inputConnected: inputRef.isConnected,
focusInPanel: panel ? panel.contains(document.activeElement) : false,
});
}, 500);
});
});
if (result.skipped) { console.log(' ⏭️ No text input with data-cv2-field found in panel'); return; }
assert(result.inputConnected, 'Input element should remain connected to DOM after debounce fires');
assert(result.focusInPanel, 'Focus should remain inside panel after debounce — re-render must not run while typing');
await page.evaluate(() => localStorage.removeItem('cs-theme-overrides'));
});
await test('Show Neighbors populates neighborPubkeys from affinity API', async () => {
const testPubkey = 'aabbccdd11223344556677889900aabbccddeeff00112233445566778899001122';
@@ -1778,12 +1811,343 @@ async function run() {
}
});
// Test: Expanded group children have unique observation ids (#866)
await test('Expanded group children update detail pane per-observation', async () => {
await page.goto(`${BASE}/#/packets`, { waitUntil: 'domcontentloaded' });
// Ensure grouped mode and wide time window
await page.evaluate(() => {
localStorage.setItem('meshcore-time-window', '525600');
localStorage.setItem('meshcore-groupbyhash', 'true');
});
await page.reload({ waitUntil: 'load' });
await page.waitForSelector('table tbody tr', { timeout: 15000 });
// Find a group row with observation_count > 1 (has expand button)
const expandBtn = await page.$('table tbody tr .expand-btn, table tbody tr [data-expand]');
if (!expandBtn) {
console.log(' ️ No expandable groups found — skipping child assertion');
return;
}
// Click expand and wait for the /packets/<hash> detail API call
const [detailResp] = await Promise.all([
page.waitForResponse(resp => {
const u = new URL(resp.url(), BASE);
// Match /api/packets/<hash> but not /api/packets?... or /api/packets/observations
return /\/api\/packets\/[A-Fa-f0-9]+$/.test(u.pathname) && resp.status() === 200;
}, { timeout: 15000 }),
expandBtn.click(),
]);
assert(detailResp, 'Expected /api/packets/<hash> response on expand');
// Wait for child rows to appear
await page.waitForSelector('table tbody tr.child-row, table tbody tr[class*="child"]', { timeout: 5000 });
const childRows = await page.$$('table tbody tr.child-row, table tbody tr[class*="child"]');
if (childRows.length < 2) {
console.log(' ️ Group has < 2 children — skipping per-observation assertion');
return;
}
// Click first child row
await childRows[0].click();
await page.waitForFunction(() => {
const panel = document.getElementById('pktRight');
return panel && !panel.classList.contains('empty') && panel.textContent.trim().length > 0;
}, { timeout: 10000 });
const content1 = await page.$eval('#pktRight', el => el.textContent.trim());
const url1 = page.url();
// Click second child row
await childRows[1].click();
await page.waitForTimeout(500);
const content2 = await page.$eval('#pktRight', el => el.textContent.trim());
const url2 = page.url();
// URL should contain ?obs= with a real observation id
assert(url1.includes('obs=') || url2.includes('obs='), `URL should contain obs= parameter, got: ${url1}`);
// The two children should show different detail pane content (different observers)
// At minimum, the URL obs= values should differ
if (url1.includes('obs=') && url2.includes('obs=')) {
const obs1 = new URL(url1).hash.match(/obs=(\d+)/)?.[1];
const obs2 = new URL(url2).hash.match(/obs=(\d+)/)?.[1];
if (obs1 && obs2) {
assert(obs1 !== obs2, `Two children should have different obs ids, both got obs=${obs1}`);
}
}
// Verify obs id is NOT the aggregate packet id (the bug from #866)
const obsMatch = url2.match(/obs=(\d+)/);
if (obsMatch) {
const detailJson = await detailResp.json().catch(() => null);
if (detailJson?.packet?.id) {
const aggId = String(detailJson.packet.id);
// At least one child obs id should differ from the aggregate packet id
const obs1 = url1.match(/obs=(\d+)/)?.[1];
const obs2 = url2.match(/obs=(\d+)/)?.[1];
const allSameAsAgg = obs1 === aggId && obs2 === aggId;
assert(!allSameAsAgg, `Child obs ids should not all equal aggregate packet.id (${aggId})`);
}
}
});
// Test: per-observation raw_hex — hex pane updates when switching observations (#881)
await test('Packet detail hex pane updates per observation', async () => {
await page.goto(BASE + '#/packets', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('table tbody tr', { timeout: 15000 });
await page.waitForTimeout(500);
// Try clicking packet rows to find one with multiple observations
const rows = await page.$$('table tbody tr[data-action]');
let obsRows = [];
for (let i = 0; i < Math.min(rows.length, 10); i++) {
await rows[i].click({ timeout: 3000 }).catch(() => null);
await page.waitForTimeout(600);
obsRows = await page.$$('.detail-obs-row');
if (obsRows.length >= 2) break;
}
if (obsRows.length < 2) {
console.log(' ⏭ Skipped: no packet with ≥2 observations found in first 10 rows');
return;
}
// Click first observation, capture hex dump
await obsRows[0].click({ timeout: 5000 });
await page.waitForTimeout(500);
const hex1 = await page.$eval('.hex-dump', el => el.textContent).catch(() => '');
// Click second observation, capture hex dump
await obsRows[1].click({ timeout: 5000 });
await page.waitForTimeout(500);
const hex2 = await page.$eval('.hex-dump', el => el.textContent).catch(() => '');
// If both have content and differ, the feature works
if (hex1 && hex2 && hex1 !== hex2) {
console.log(' ✓ Hex pane content differs between observations');
} else if (hex1 && hex2 && hex1 === hex2) {
console.log(' ⏭ Hex same for both observations (likely historical NULL raw_hex — OK)');
} else {
console.log(' ⏭ Could not capture hex content from both observations');
}
});
// Test: path pill (top) and byte breakdown (bottom) agree on hop count
// Regression for visual mismatch where badge said "1 hop" but path text listed N names
await test('Packet detail path pill and byte breakdown agree on hop count', async () => {
await page.goto(BASE + '#/packets', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('table tbody tr', { timeout: 15000 });
await page.waitForTimeout(500);
// Click rows until we find one whose detail pane renders a multi-hop path
const rows = await page.$$('table tbody tr[data-action]');
let found = false;
for (let i = 0; i < Math.min(rows.length, 15); i++) {
await rows[i].click({ timeout: 3000 }).catch(() => null);
await page.waitForTimeout(500);
const result = await page.evaluate(() => {
// Path pill: <dt>Path</dt><dd><span class="badge ...">N hops</span> ...names...</dd>
const dts = document.querySelectorAll('dl.detail-meta dt');
let pillBadgeCount = null;
let pillNameCount = null;
for (const dt of dts) {
if (dt.textContent.trim() === 'Path') {
const dd = dt.nextElementSibling;
if (!dd) break;
const badge = dd.querySelector('.badge');
if (badge) {
const m = badge.textContent.match(/(\d+)\s*hop/);
if (m) pillBadgeCount = parseInt(m[1], 10);
}
// Count rendered hop links/spans (HopDisplay.renderHop output)
const hops = dd.querySelectorAll('.hop-link, [data-hop-link], .hop-named, .hop-anonymous');
pillNameCount = hops.length;
break;
}
}
// Byte breakdown: section row "Path (N hops)" + N "Hop X — ..." rows
let breakdownSectionCount = null;
let breakdownRowCount = 0;
const fieldTable = document.querySelector('table.field-table');
if (fieldTable) {
for (const tr of fieldTable.querySelectorAll('tr')) {
const txt = tr.textContent.trim();
const sec = txt.match(/^Path\s*\((\d+)\s*hops?\)/);
if (sec) breakdownSectionCount = parseInt(sec[1], 10);
if (/^\s*\d+\s*Hop\s+\d+\s*—/.test(txt) || /^Hop\s+\d+\s*—/.test(txt.replace(/^\d+/, '').trim())) {
breakdownRowCount++;
}
}
}
return { pillBadgeCount, pillNameCount, breakdownSectionCount, breakdownRowCount };
});
if (result.pillBadgeCount && result.pillBadgeCount > 0 && result.breakdownSectionCount != null) {
found = true;
// Top badge count must equal bottom section count
assert(result.pillBadgeCount === result.breakdownSectionCount,
`Path pill badge says ${result.pillBadgeCount} hops but byte breakdown says ${result.breakdownSectionCount} hops`);
// Number of rendered hop names in pill should also match (within 1, since renderPath may add separators)
if (result.pillNameCount != null && result.pillNameCount > 0) {
assert(Math.abs(result.pillNameCount - result.pillBadgeCount) <= 1,
`Path pill badge ${result.pillBadgeCount} but rendered ${result.pillNameCount} hop names`);
}
// And breakdown rendered rows should match its own section count
assert(result.breakdownRowCount > 0,
'breakdown rows selector matched nothing — selector or DOM changed');
assert(result.breakdownRowCount === result.breakdownSectionCount,
`Byte breakdown section says ${result.breakdownSectionCount} hops but rendered ${result.breakdownRowCount} hop rows`);
console.log(` ✓ Path pill (${result.pillBadgeCount}) and byte breakdown (${result.breakdownSectionCount}) agree`);
break;
}
}
if (!found) {
if (process.env.E2E_REQUIRE_PATH_TEST === '1') {
throw new Error('BLOCKED — no multi-hop packet found in first 15 rows (E2E_REQUIRE_PATH_TEST=1 requires it)');
}
const skipErr = new Error('SKIP: No multi-hop packet with byte breakdown found in first 15 rows — needs fixture');
skipErr.skip = true;
throw skipErr;
}
});
// Test: hex-strip color spans match the labeled byte rows (per-obs raw_hex).
// Regression #891: server-supplied breakdown was computed once from top-level
// raw_hex, so per-observation rendering had off-by-N highlights vs the labels.
await test('Packet detail hex strip Path range matches hop row count', async () => {
await page.goto(BASE + '#/packets', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('table tbody tr', { timeout: 15000 });
await page.waitForTimeout(500);
const rows = await page.$$('table tbody tr[data-action]');
let checked = 0;
for (let i = 0; i < Math.min(rows.length, 25) && checked < 3; i++) {
await rows[i].click({ timeout: 3000 }).catch(() => null);
await page.waitForTimeout(400);
const result = await page.evaluate(() => {
const dump = document.querySelector('.hex-dump');
const fieldTable = document.querySelector('table.field-table');
if (!dump || !fieldTable) return null;
const pathSpan = dump.querySelector('span.hex-byte.hex-path');
const pathBytes = pathSpan ? pathSpan.textContent.trim().split(/\s+/).filter(Boolean).length : 0;
const hopRows = [];
for (const tr of fieldTable.querySelectorAll('tr')) {
const cells = [...tr.cells].map(c => c.textContent.trim());
if (cells.length >= 2 && /^Hop\s+\d+/.test(cells[1])) hopRows.push(cells[2]);
}
return { pathBytes, hopRows };
});
if (!result || (result.pathBytes === 0 && result.hopRows.length === 0)) continue;
checked++;
// Either both zero, or the count of bytes inside hex-path == hop rows.
// (For multi-byte hash sizes this is bytes-per-hop * hops; for hash_size=1 it's just hops.)
// The simpler invariant: if there are hop rows, hex-path span must exist and have at least
// as many bytes as there are hops (== exactly hops * hash_size).
assert(result.hopRows.length > 0,
`row ${i}: hex-path span has ${result.pathBytes} bytes but no hop rows in the labeled table`);
assert(result.pathBytes >= result.hopRows.length,
`row ${i}: hex-path has ${result.pathBytes} bytes but ${result.hopRows.length} hop rows — strip and labels disagree`);
assert(result.pathBytes % result.hopRows.length === 0,
`row ${i}: hex-path has ${result.pathBytes} bytes but ${result.hopRows.length} hop rows — bytes/hops not divisible (hash_size violated)`);
console.log(` ✓ row ${i}: hex-path ${result.pathBytes} bytes / ${result.hopRows.length} hop rows (hash_size=${result.pathBytes / result.hopRows.length})`);
}
if (checked === 0) {
const skipErr = new Error('SKIP: no packet with rendered hex strip + hop rows found in first 25 rows');
skipErr.skip = true;
throw skipErr;
}
});
// Test: clicking a different observation row re-renders strip + breakdown consistently.
// Regression: observations of the same packet hash have different raw_hex (#882),
// so picking a different obs must recompute the byte ranges, not reuse the old ones.
await test('Packet detail switches consistently across observations', async () => {
await page.goto(BASE + '#/packets?groupByHash=1', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('table tbody tr', { timeout: 15000 });
await page.waitForTimeout(500);
let opened = false;
const groupRows = await page.$$('table tbody tr[data-action]');
for (let i = 0; i < Math.min(groupRows.length, 10); i++) {
await groupRows[i].click({ timeout: 3000 }).catch(() => null);
await page.waitForTimeout(400);
const obsCount = await page.evaluate(() => {
return document.querySelectorAll('table.observations-table tbody tr, .obs-row').length;
});
if (obsCount >= 2) { opened = true; break; }
}
if (!opened) {
const skipErr = new Error('SKIP: no multi-observation packet found in first 10 group rows');
skipErr.skip = true;
throw skipErr;
}
async function snapshot() {
return page.evaluate(() => {
const dump = document.querySelector('.hex-dump');
const fieldTable = document.querySelector('table.field-table');
if (!dump || !fieldTable) return null;
const pathSpan = dump.querySelector('span.hex-byte.hex-path');
const pathBytes = pathSpan ? pathSpan.textContent.trim().split(/\s+/).filter(Boolean).length : 0;
const hopRows = [];
for (const tr of fieldTable.querySelectorAll('tr')) {
const cells = [...tr.cells].map(c => c.textContent.trim());
if (cells.length >= 2 && /^Hop\s+\d+/.test(cells[1])) hopRows.push(cells[2]);
}
const rawHexParts = [...dump.querySelectorAll('span.hex-byte')].map(s => s.textContent.trim());
return { pathBytes, hopCount: hopRows.length, rawHexJoined: rawHexParts.join('|') };
});
}
const snapA = await snapshot();
assert(snapA, 'first snapshot must have hex dump + field table');
assert(snapA.hopCount === 0 || snapA.pathBytes >= snapA.hopCount,
`obs A inconsistent: hex-path ${snapA.pathBytes} bytes vs ${snapA.hopCount} hop rows`);
const switched = await page.evaluate(() => {
const obsRows = [...document.querySelectorAll('table.observations-table tbody tr, .obs-row')];
if (obsRows.length < 2) return false;
obsRows[1].click();
return true;
});
assert(switched, 'should click second observation row');
await page.waitForTimeout(500);
const snapB = await snapshot();
assert(snapB, 'second snapshot must have hex dump + field table');
assert(snapB.hopCount === 0 || snapB.pathBytes >= snapB.hopCount,
`obs B inconsistent: hex-path ${snapB.pathBytes} bytes vs ${snapB.hopCount} hop rows`);
console.log(` ✓ obs A: ${snapA.pathBytes} path bytes / ${snapA.hopCount} hops; obs B: ${snapB.pathBytes} / ${snapB.hopCount}`);
});
// Test: clicking the 🔍 Details button in the nodes side panel navigates to
// the full-screen node detail view. Regression: hash already === target,
// so location.hash assignment was a no-op and the panel stayed open.
await test('Nodes side panel Details button opens full-screen view', async () => {
await page.goto(BASE + '#/nodes', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('table tbody tr[data-action]', { timeout: 15000 });
await page.waitForTimeout(500);
// Open side panel
await page.click('table tbody tr[data-action]');
await page.waitForSelector('#nodesRight .node-detail-btn', { timeout: 5000 });
// Click Details
await page.click('#nodesRight .node-detail-btn');
// Wait for full-screen view to appear
await page.waitForSelector('.node-fullscreen', { timeout: 5000 });
const isFullScreen = await page.evaluate(() => !!document.querySelector('.node-fullscreen'));
assert(isFullScreen, 'Details button should open full-screen node view');
});
await browser.close();
// Summary
const passed = results.filter(r => r.pass).length;
const skipped = results.filter(r => r.skipped).length;
const passed = results.filter(r => r.pass && !r.skipped).length;
const failed = results.filter(r => !r.pass).length;
console.log(`\n${passed}/${results.length} tests passed${failed ? `, ${failed} failed` : ''}`);
console.log(`\n${passed}/${results.length} tests passed${skipped ? `, ${skipped} skipped` : ''}${failed ? `, ${failed} failed` : ''}`);
process.exit(failed > 0 ? 1 : 0);
}
+222 -26
View File
@@ -690,6 +690,88 @@ console.log('\n=== haversineKm (hop-resolver.js) ===');
});
}
// ===== pickByAffinity — neighbor-graph + centroid scoring (#874) =====
console.log('\n=== pickByAffinity neighbor-graph scoring (#874) ===');
{
const ctx = makeSandbox();
ctx.IATA_COORDS_GEO = {};
loadInCtx(ctx, 'public/hop-resolver.js');
const HR = ctx.window.HopResolver;
// Two nodes sharing prefix "ab", hundreds of km apart.
// NodeSF is near San Francisco, NodeDEN is near Denver.
const nodeSF = { public_key: 'ab11111111111111', name: 'NodeSF', lat: 37.7, lon: -122.4 };
const nodeDEN = { public_key: 'ab22222222222222', name: 'NodeDEN', lat: 39.7, lon: -104.9 };
// A known neighbor of NodeSF (in the graph)
const nodeNeighbor = { public_key: 'cc33333333333333', name: 'SFNeighbor', lat: 37.8, lon: -122.3 };
// Another known node near Denver
const nodeDenNeighbor = { public_key: 'dd44444444444444', name: 'DENNeighbor', lat: 39.8, lon: -105.0 };
test('#874: graph edge scoring picks correct regional candidate (SF)', () => {
HR.init([nodeSF, nodeDEN, nodeNeighbor, nodeDenNeighbor]);
HR.setAffinity({ edges: [
{ source: 'cc33333333333333', target: 'ab11111111111111', weight: 5 },
{ source: 'dd44444444444444', target: 'ab22222222222222', weight: 5 },
]});
// Path: SFNeighbor → [ab??] → DENNeighbor
// With graph edges, ab11 (NodeSF) has edge to SFNeighbor, ab22 (NodeDEN) has edge to DENNeighbor
// Prev=SFNeighbor, Next=DENNeighbor → both have score 5, but SFNeighbor edge only to ab11
const result = HR.resolve(['cc', 'ab', 'dd'],
null, null, null, null);
assert.strictEqual(result['ab'].name, 'NodeSF',
'Should pick NodeSF because it has a graph edge to prev hop SFNeighbor');
});
test('#874: graph edge scoring — next hop breaks tie', () => {
HR.init([nodeSF, nodeDEN, nodeNeighbor, nodeDenNeighbor]);
HR.setAffinity({ edges: [
{ source: 'dd44444444444444', target: 'ab22222222222222', weight: 8 },
// No edge from SFNeighbor to either ab node
]});
// Path: SFNeighbor → [ab??] → DENNeighbor
// Only ab22 (NodeDEN) has edge to DENNeighbor (next hop)
const result = HR.resolve(['cc', 'ab', 'dd'],
null, null, null, null);
assert.strictEqual(result['ab'].name, 'NodeDEN',
'Should pick NodeDEN because it has graph edge to next hop DENNeighbor');
});
test('#874: centroid fallback when no graph edges exist', () => {
HR.init([nodeSF, nodeDEN, nodeNeighbor]);
HR.setAffinity({ edges: [] }); // no edges at all
// Path: SFNeighbor → [ab??]
// SFNeighbor is at (37.8, -122.3), centroid is just that point
// NodeSF (37.7, -122.4) is ~14km away, NodeDEN (39.7, -104.9) is ~1500km away
const result = HR.resolve(['cc', 'ab'],
null, null, null, null);
assert.strictEqual(result['ab'].name, 'NodeSF',
'Should pick NodeSF via centroid proximity to SFNeighbor');
});
test('#874: centroid uses average of prev+next positions', () => {
// Prev near SF, next near Denver → centroid is midpoint (~Nevada)
// NodeDEN is closer to Nevada midpoint than NodeSF
const nodeMid = { public_key: 'ee55555555555555', name: 'MidNode', lat: 38.5, lon: -114.0 };
HR.init([nodeSF, nodeDEN, nodeNeighbor, nodeDenNeighbor, nodeMid]);
HR.setAffinity({ edges: [] });
// Path: SFNeighbor → [ab??] → DENNeighbor
// centroid = avg(37.8,-122.3, 39.8,-105.0) = (38.8, -113.65) — closer to Denver
const result = HR.resolve(['cc', 'ab', 'dd'],
null, null, null, null);
assert.strictEqual(result['ab'].name, 'NodeDEN',
'Should pick NodeDEN because centroid of SF+Denver neighbors is closer to Denver');
});
test('#874: fallback when no context at all', () => {
HR.init([nodeSF, nodeDEN]);
HR.setAffinity({ edges: [] });
// Single ambiguous hop, no origin/observer, no neighbors
const result = HR.resolve(['ab'], null, null, null, null);
assert.ok(result['ab'].ambiguous || result['ab'].name != null,
'Should resolve to some candidate without crashing');
});
}
// ===== SNR/RSSI Number casting =====
{
// These test the pattern used in observer-detail.js, home.js, traces.js, live.js
@@ -1722,6 +1804,128 @@ console.log('\n=== app.js: formatEngineBadge ===');
});
}
// ===== APP.JS: computeBreakdownRanges =====
console.log('\n=== app.js: computeBreakdownRanges ===');
{
const ctx = makeSandbox();
loadInCtx(ctx, 'public/roles.js');
loadInCtx(ctx, 'public/app.js');
const computeBreakdownRanges = ctx.computeBreakdownRanges;
function findRange(ranges, label) {
return ranges.find(r => r.label === label);
}
test('returns [] for empty hex', () => {
assert.deepEqual(computeBreakdownRanges('', 1, 5), []);
});
test('returns [] for too-short hex (< 2 bytes)', () => {
assert.deepEqual(computeBreakdownRanges('15', 1, 5), []);
});
test('FLOOD non-transport: 4-hop hash_size=1', () => {
// header=15, plb=04 → hash_size=1, hash_count=4
// bytes: 15 04 90 FA F9 10 6E 01 D9
const r = computeBreakdownRanges('150490FAF910 6E01D9'.replace(/\s/g,''), 1, 5);
assert.deepEqual(findRange(r, 'Header'), { start: 0, end: 0, label: 'Header' });
assert.deepEqual(findRange(r, 'Path Length'), { start: 1, end: 1, label: 'Path Length' });
assert.deepEqual(findRange(r, 'Path'), { start: 2, end: 5, label: 'Path' });
assert.deepEqual(findRange(r, 'Payload'), { start: 6, end: 8, label: 'Payload' });
assert.strictEqual(findRange(r, 'Transport Codes'), undefined);
});
test('FLOOD non-transport: 7-hop hash_size=1', () => {
// header=15, plb=07
const hex = '15077f6d7d1cadeca33988fd95e0851ebf01ea12e1879e';
const r = computeBreakdownRanges(hex, 1, 5);
assert.deepEqual(findRange(r, 'Path'), { start: 2, end: 8, label: 'Path' });
const payload = findRange(r, 'Payload');
assert.strictEqual(payload.start, 9, 'payload starts after the 7 path bytes');
});
test('FLOOD non-transport: 8-hop hash_size=1', () => {
const hex = '1508' + '11223344556677AA' + 'BBCCDD';
const r = computeBreakdownRanges(hex, 1, 5);
assert.deepEqual(findRange(r, 'Path'), { start: 2, end: 9, label: 'Path' });
assert.deepEqual(findRange(r, 'Payload'), { start: 10, end: 12, label: 'Payload' });
});
test('Direct advert: 0-hop, no Path range', () => {
// plb=00 → 0 hops; expect Path Length but NO Path range
const r = computeBreakdownRanges('1100AABBCCDD', 1, 4);
assert.deepEqual(findRange(r, 'Path Length'), { start: 1, end: 1, label: 'Path Length' });
assert.strictEqual(findRange(r, 'Path'), undefined);
});
test('Transport route shifts path-length offset by 4', () => {
// route_type=0 (TRANSPORT_FLOOD): bytes 1..4 are Transport Codes
// header=14, transport=AABBCCDD, plb=02, hops=11 22, payload=99
const hex = '14AABBCCDD021122' + '99';
const r = computeBreakdownRanges(hex, 0, 5);
assert.deepEqual(findRange(r, 'Transport Codes'), { start: 1, end: 4, label: 'Transport Codes' });
assert.deepEqual(findRange(r, 'Path Length'), { start: 5, end: 5, label: 'Path Length' });
assert.deepEqual(findRange(r, 'Path'), { start: 6, end: 7, label: 'Path' });
assert.deepEqual(findRange(r, 'Payload'), { start: 8, end: 8, label: 'Payload' });
});
test('hash_size=2 (plb top bits=01): 4 hops × 2 bytes', () => {
// plb = 01 0001 00 = 0x44 → hash_size=2, hash_count=4 → 8 path bytes
const hex = '15' + '44' + 'AABB' + 'CCDD' + 'EEFF' + '1122' + '9988';
const r = computeBreakdownRanges(hex, 1, 5);
assert.deepEqual(findRange(r, 'Path'), { start: 2, end: 9, label: 'Path' });
assert.deepEqual(findRange(r, 'Payload'), { start: 10, end: 11, label: 'Payload' });
});
test('hash_size=3 (plb top bits=10): 2 hops × 3 bytes', () => {
// plb = 10 0000 10 = 0x82 → hash_size=3, hash_count=2 → 6 path bytes
const hex = '15' + '82' + 'AABBCC' + 'DDEEFF' + '99';
const r = computeBreakdownRanges(hex, 1, 5);
assert.deepEqual(findRange(r, 'Path'), { start: 2, end: 7, label: 'Path' });
assert.deepEqual(findRange(r, 'Payload'), { start: 8, end: 8, label: 'Payload' });
});
test('hash_size=4 (plb top bits=11): 2 hops × 4 bytes', () => {
// plb = 11 0000 10 = 0xC2 → hash_size=4, hash_count=2 → 8 path bytes
const hex = '15' + 'C2' + 'AABBCCDD' + 'EEFF1122' + '99887766';
const r = computeBreakdownRanges(hex, 1, 5);
assert.deepEqual(findRange(r, 'Path'), { start: 2, end: 9, label: 'Path' });
assert.deepEqual(findRange(r, 'Payload'), { start: 10, end: 13, label: 'Payload' });
});
test('truncated path: not enough bytes → no Path range', () => {
// plb=04 says 4 hops but only 2 bytes remain
const hex = '1504AABB';
const r = computeBreakdownRanges(hex, 1, 5);
assert.strictEqual(findRange(r, 'Path'), undefined);
});
test('ADVERT (payload_type=4) with full record: PubKey/Timestamp/Signature/Flags', () => {
// header=11, plb=00 (direct advert)
// payload: 32 bytes pubkey + 4 bytes ts + 64 bytes sig + 1 byte flags
const pubkey = 'AB'.repeat(32);
const ts = '11223344';
const sig = 'CD'.repeat(64);
const flags = '00';
const hex = '1100' + pubkey + ts + sig + flags;
const r = computeBreakdownRanges(hex, 1, 4);
assert.deepEqual(findRange(r, 'PubKey'), { start: 2, end: 33, label: 'PubKey' });
assert.deepEqual(findRange(r, 'Timestamp'), { start: 34, end: 37, label: 'Timestamp' });
assert.deepEqual(findRange(r, 'Signature'), { start: 38, end: 101, label: 'Signature' });
assert.deepEqual(findRange(r, 'Flags'), { start: 102, end: 102, label: 'Flags' });
});
test('NaN-safe: malformed path-length byte produces no Path range', () => {
// hex with non-hex char in plb position would parseInt-fail → bail
// Use a 1-byte payload that makes pathByte parseInt produce NaN-ish via X
// (parseInt of 'XY' is NaN). Since fs reads only hex chars, simulate via short hex.
// Easier: empty string already returns []; 1-byte returns []. Both covered above.
// Use plb=FF (hash_size=4, hash_count=63) too long for input → no Path
const r = computeBreakdownRanges('15FF' + 'AA', 1, 5);
assert.strictEqual(findRange(r, 'Path'), undefined);
});
}
// ===== APP.JS: isTransportRoute + transportBadge =====
console.log('\n=== app.js: isTransportRoute + transportBadge ===');
{
@@ -5462,40 +5666,33 @@ console.log('\n=== packets.js: buildFieldTable hop count from path_len (#844) ==
loadInCtx(ftCtx, 'public/packets.js');
const { buildFieldTable } = ftCtx.window._packetsTestAPI;
test('#844: byte breakdown uses path_len hop count, not aggregated _parsedPath', () => {
test('#885: byte breakdown uses pathHops length (single source of truth)', () => {
// After #885 the byte breakdown agrees with the path pill: both render
// from the per-observation path_json. raw_hex is the underlying bytes
// for that same observation, so consistency is by construction.
// path_len = 0x42 → hash_size=2, hash_count=2
// raw_hex: header(11) + path_len(42) + hop0(41B1) + hop1(27D7) + pubkey(32 bytes)...
const pubkey = 'C0DEDAD4'.padEnd(64, '0'); // 32 bytes = 64 hex chars
const raw = '1142' + '41B1' + '27D7' + pubkey + '00000000' + '0'.repeat(128);
const pkt = { raw_hex: raw, route_type: 1, payload_type: 0 };
// Pass aggregated pathHops with 7 hops (mismatched)
const pathHops = ['41B1', '5EB0', '1000', '2DD2', '52F8', '9535', '762B'];
// Per-obs path_json IS the source of truth — pass the 2 hops that match raw_hex.
const pathHops = ['41B1', '27D7'];
const html = buildFieldTable(pkt, {}, pathHops, {});
// Section header should say "2 hops", not "7 hops"
assert.ok(html.includes('Path (2 hops)'), 'Should show "Path (2 hops)" from path_len, got: ' +
(html.match(/Path \(\d+ hops\)/)?.[0] || 'no match'));
assert.ok(!html.includes('Path (7 hops)'), 'Should NOT show 7 hops from aggregated path');
// Should contain hop values from raw_hex
assert.ok(html.includes('Path (2 hops)'), 'Should show "Path (2 hops)"');
assert.ok(html.includes('41B1'), 'Should show hop 0 = 41B1');
assert.ok(html.includes('27D7'), 'Should show hop 1 = 27D7');
// Should NOT contain hops from aggregated path that aren't in raw_hex
assert.ok(!html.includes('5EB0'), 'Should NOT show aggregated hop 5EB0');
assert.ok(!html.includes('9535'), 'Should NOT show aggregated hop 9535');
});
test('#844: pubkey offset correct after 2-hop path (not after 7-hop)', () => {
test('#885: pubkey offset advances by hashSize * pathHops.length', () => {
const pubkey = 'C0DEDAD4'.padEnd(64, '0');
const raw = '1142' + '41B1' + '27D7' + pubkey + '00000000' + '0'.repeat(128);
const pkt = { raw_hex: raw, route_type: 1, payload_type: 0 };
const html = buildFieldTable(pkt, { type: 'ADVERT', pubKey: pubkey }, ['41B1','5EB0','1000','2DD2','52F8','9535','762B'], {});
const html = buildFieldTable(pkt, { type: 'ADVERT', pubKey: pubkey }, ['41B1', '27D7'], {});
// Public Key should be at offset 6 (1 header + 1 path_len + 2*2 hops = 6)
// Not at offset 16 (1 + 1 + 2*7 = 16)
assert.ok(html.includes('>6<') || html.includes('"6"'),
'Public Key should be at offset 6, not 16');
'Public Key should be at offset 6');
});
test('#844: hashCountVal=0 (direct advert) skips Path section', () => {
@@ -5707,12 +5904,11 @@ console.log('\n=== channel-decrypt.js: key derivation, MAC, parsing, storage ===
assert.strictEqual(ctx.window.renderSkewBadge(null, 0), '');
});
test('renderSkewBadge renders bimodal_clock badge with tooltip (#845)', () => {
var cs = { goodFraction: 0.6, recentBadSampleCount: 4, recentSampleCount: 10 };
var html = ctx.window.renderSkewBadge('bimodal_clock', -5, cs);
assert.ok(html.includes('skew-badge--bimodal_clock'), 'should contain bimodal_clock class');
assert.ok(html.includes('bimodal'), 'tooltip should mention bimodal');
assert.ok(html.includes('40%'), 'tooltip should show bad percentage');
test('renderSkewBadge renders default badge with tooltip', () => {
var cs = {};
var html = ctx.window.renderSkewBadge('default', 0, cs);
assert.ok(html.includes('skew-badge--default'), 'should contain default class');
assert.ok(html.toLowerCase().includes('firmware default'), 'tooltip should mention firmware default');
assert.ok(html.includes('⏰'), 'should contain clock emoji');
});
@@ -5736,9 +5932,9 @@ console.log('\n=== channel-decrypt.js: key derivation, MAC, parsing, storage ===
test('SKEW_SEVERITY_ORDER sorts worst first', () => {
var order = ctx.window.SKEW_SEVERITY_ORDER;
assert.ok(order.absurd < order.critical, 'absurd should sort before critical');
assert.ok(order.critical < order.warning, 'critical should sort before warning');
assert.ok(order.warning < order.ok, 'warning should sort before ok');
assert.ok(order.wrong < order.degraded, 'wrong should sort before degraded');
assert.ok(order.degraded < order.degrading, 'degraded should sort before degrading');
assert.ok(order.degrading < order.ok, 'degrading should sort before ok');
});
}
+57 -4
View File
@@ -22,9 +22,9 @@ function assert(condition, msg) {
// ── Test nodes ──
// Two nodes share the same 1-byte prefix "ab"
const nodeA = { public_key: 'ab1111', name: 'NodeA', lat: 37.0, lon: -122.0 };
const nodeB = { public_key: 'ab2222', name: 'NodeB', lat: 38.0, lon: -123.0 };
const nodeC = { public_key: 'cd3333', name: 'NodeC', lat: 37.5, lon: -122.5 };
const nodeA = { public_key: 'ab1111', name: 'NodeA', role: 'repeater', lat: 37.0, lon: -122.0 };
const nodeB = { public_key: 'ab2222', name: 'NodeB', role: 'repeater', lat: 38.0, lon: -123.0 };
const nodeC = { public_key: 'cd3333', name: 'NodeC', role: 'repeater', lat: 37.5, lon: -122.5 };
console.log('\n=== HopResolver Affinity Tests ===\n');
@@ -88,12 +88,65 @@ assert(result5['ab'].name === 'NodeB', 'Should pick NodeB (highest affinity 0.9)
// Test 6: Unambiguous hops are not affected by affinity
console.log('\nTest 6: Unambiguous hops unaffected by affinity');
const nodeD = { public_key: 'ee4444', name: 'NodeD', lat: 36.0, lon: -121.0 };
const nodeD = { public_key: 'ee4444', name: 'NodeD', role: 'repeater', lat: 36.0, lon: -121.0 };
HopResolver.init([nodeA, nodeB, nodeC, nodeD]);
HopResolver.setAffinity({ edges: [] });
const result6 = HopResolver.resolve(['ee44'], null, null, null, null, null);
assert(result6['ee44'].name === 'NodeD', 'Unique prefix resolves directly — got: ' + result6['ee44'].name);
assert(!result6['ee44'].ambiguous, 'Should not be marked ambiguous');
// Test 7: lat=0 / lon=0 candidates are NOT excluded (equator/prime-meridian bug fix)
console.log('\nTest 7: lat=0 / lon=0 candidates are included in geo scoring');
const nodeEquator = { public_key: 'ab5555', name: 'EquatorNode', role: 'repeater', lat: 0, lon: 10 };
const nodeFar = { public_key: 'ab6666', name: 'FarNode', role: 'repeater', lat: 60, lon: 60 };
const anchorNearEq = { public_key: 'cd7777', name: 'AnchorEq', role: 'repeater', lat: 1, lon: 11 };
HopResolver.init([nodeEquator, nodeFar, anchorNearEq]);
HopResolver.setAffinity({});
// Anchor near equator — EquatorNode (0,10) should be geo-closest
const result7 = HopResolver.resolve(['cd77', 'ab'], 1.0, 11.0, null, null, null);
assert(result7['ab'].name === 'EquatorNode',
'lat=0 candidate should be included and win by geo — got: ' + result7['ab'].name);
// Test 8: lon=0 candidate is also included
console.log('\nTest 8: lon=0 candidate is included in geo scoring');
const nodePrime = { public_key: 'ab8888', name: 'PrimeMeridian', role: 'repeater', lat: 10, lon: 0 };
const anchorNearPM = { public_key: 'cd9999', name: 'AnchorPM', role: 'repeater', lat: 11, lon: 1 };
HopResolver.init([nodePrime, nodeFar, anchorNearPM]);
HopResolver.setAffinity({});
const result8 = HopResolver.resolve(['cd99', 'ab'], 11.0, 1.0, null, null, null);
assert(result8['ab'].name === 'PrimeMeridian',
'lon=0 candidate should be included and win by geo — got: ' + result8['ab'].name);
// ── Role filter tests (#935) ──
console.log('\nTest: Role filter — companions excluded from prefixIdx');
const companion = { public_key: 'ab9999', name: 'Companion1', role: 'companion', lat: 37.0, lon: -122.0 };
const sensor = { public_key: 'ab7777', name: 'Sensor1', role: 'sensor', lat: 37.0, lon: -122.0 };
const repeater = { public_key: 'ab1234', name: 'Repeater1', role: 'repeater', lat: 37.0, lon: -122.0 };
const roomSrv = { public_key: 'ff1234', name: 'RoomSrv1', role: 'room_server', lat: 37.0, lon: -122.0 };
HopResolver.init([companion, sensor, repeater, roomSrv]);
HopResolver.setAffinity({});
// Prefix 'ab' should only resolve to repeater (companion/sensor excluded)
const r1 = HopResolver.resolve(['ab12'], 0, 0, null, null, null);
assert(r1['ab12'] && r1['ab12'].name === 'Repeater1',
'prefix ab12 resolves to Repeater1 not companion — got: ' + (r1['ab12'] && r1['ab12'].name));
// Prefix 'ff' should resolve to room_server
const r2 = HopResolver.resolve(['ff12'], 0, 0, null, null, null);
assert(r2['ff12'] && r2['ff12'].name === 'RoomSrv1',
'prefix ff12 resolves to RoomSrv1 — got: ' + (r2['ff12'] && r2['ff12'].name));
// Prefix that only matches companion should return nothing
const r3 = HopResolver.resolve(['ab99'], 0, 0, null, null, null);
assert(!r3['ab99'] || !r3['ab99'].name,
'prefix ab99 (companion only) resolves to nothing — got: ' + (r3['ab99'] && r3['ab99'].name));
// pubkeyIdx should still have companion (full pubkey lookup)
console.log('\nTest: pubkeyIdx still includes all roles');
const fromServer = HopResolver.resolveFromServer(['ab99'], [companion.public_key]);
assert(fromServer['ab99'] && fromServer['ab99'].name === 'Companion1',
'resolveFromServer finds companion by full pubkey — got: ' + (fromServer['ab99'] && fromServer['ab99'].name));
console.log('\n' + (passed + failed) + ' tests, ' + passed + ' passed, ' + failed + ' failed\n');
process.exit(failed > 0 ? 1 : 0);
+114
View File
@@ -844,6 +844,120 @@ console.log('\n=== packets.js: _invalidateRowCounts / _refreshRowCountsIfDirty (
});
}
console.log('\n=== packets.js: buildPacketsParams ===');
{
const ctx = loadPacketsSandbox();
const api = ctx._packetsTestAPI;
assert(typeof api.buildPacketsParams === 'function', 'buildPacketsParams must be exported');
test('hash filter suppresses region — direct hash links work regardless of saved region', () => {
// This is the bug from URL https://analyzer.../#/packets?hash=178525e9f693aa7e
// when the user's saved RegionFilter excludes the packet's observer region.
// The hash is an exact identifier; ALL other filters must be ignored.
const p = api.buildPacketsParams({
filters: { hash: 'abc123' },
regionParam: 'SJC,SFO,OAK,MRY',
windowMin: 60,
groupByHash: false,
limit: 200,
});
assert.strictEqual(p.get('hash'), 'abc123');
assert.strictEqual(p.get('region'), null, 'region must NOT be set when hash is present');
assert.strictEqual(p.get('since'), null, 'since must NOT be set when hash is present');
});
test('hash filter suppresses ALL other filters — observer, node, channel too', () => {
const p = api.buildPacketsParams({
filters: { hash: 'h', node: 'n', observer: 'o', channel: 'c' },
regionParam: 'SJC',
windowMin: 60,
groupByHash: false,
limit: 200,
});
assert.strictEqual(p.get('hash'), 'h');
assert.strictEqual(p.get('node'), null);
assert.strictEqual(p.get('observer'), null);
assert.strictEqual(p.get('channel'), null);
assert.strictEqual(p.get('region'), null);
assert.strictEqual(p.get('since'), null);
});
test('hash filter suppresses region with default windowMin=0', () => {
const p = api.buildPacketsParams({
filters: { hash: 'deadbeef' },
regionParam: 'COA',
windowMin: 0,
groupByHash: false,
limit: 50,
});
assert.strictEqual(p.get('hash'), 'deadbeef');
assert.strictEqual(p.get('region'), null);
});
test('region applied normally when hash filter is absent', () => {
const p = api.buildPacketsParams({
filters: {},
regionParam: 'SJC,SFO',
windowMin: 60,
groupByHash: false,
limit: 200,
});
assert.strictEqual(p.get('region'), 'SJC,SFO', 'region must apply when no hash');
assert.strictEqual(p.get('hash'), null);
assert(p.get('since'), 'since must apply when no hash and windowMin>0');
});
test('observer/node/channel pass through normally when no hash', () => {
const p = api.buildPacketsParams({
filters: { observer: 'obs1', node: 'node1', channel: '#test' },
regionParam: '',
windowMin: 0,
groupByHash: false,
limit: 50,
});
assert.strictEqual(p.get('observer'), 'obs1');
assert.strictEqual(p.get('node'), 'node1');
assert.strictEqual(p.get('channel'), '#test');
});
test('region absent when regionParam empty — no spurious empty region= param', () => {
const p = api.buildPacketsParams({
filters: {},
regionParam: '',
windowMin: 0,
groupByHash: false,
limit: 50,
});
assert.strictEqual(p.get('region'), null);
});
test('groupByHash=true with hash sets groupByHash and omits expand', () => {
const p = api.buildPacketsParams({
filters: { hash: 'h' }, regionParam: '', windowMin: 0, groupByHash: true, limit: 50,
});
assert.strictEqual(p.get('groupByHash'), 'true');
assert.strictEqual(p.get('expand'), null);
assert.strictEqual(p.get('hash'), 'h');
});
test('groupByHash=false with hash sets expand=observations', () => {
const p = api.buildPacketsParams({
filters: { hash: 'h' }, regionParam: '', windowMin: 0, groupByHash: false, limit: 50,
});
assert.strictEqual(p.get('expand'), 'observations');
assert.strictEqual(p.get('groupByHash'), null);
assert.strictEqual(p.get('hash'), 'h');
});
test('groupByHash=false without hash sets expand=observations', () => {
const p = api.buildPacketsParams({
filters: {}, regionParam: '', windowMin: 0, groupByHash: false, limit: 50,
});
assert.strictEqual(p.get('expand'), 'observations');
assert.strictEqual(p.get('groupByHash'), null);
});
}
// ===== SUMMARY =====
console.log(`\n${'='.repeat(40)}`);
console.log(`packets.js tests: ${passed} passed, ${failed} failed`);
+87
View File
@@ -0,0 +1,87 @@
// E2E tests for Path Inspector (spec §5 — Playwright).
// Run: npx playwright test test-path-inspector-e2e.js
// Requires: running server on BASE_URL (default http://localhost:3000).
'use strict';
const { test, expect } = require('@playwright/test');
const BASE_URL = process.env.BASE_URL || 'http://localhost:3000';
test.describe('Path Inspector — Map Side Pane (spec §2.7)', () => {
test('side pane present and collapsed by default', async ({ page }) => {
await page.goto(`${BASE_URL}/#/map`);
const pane = page.locator('#mapSidePane');
await expect(pane).toBeVisible();
await expect(pane).not.toHaveClass(/expanded/);
});
test('click toggle expands the pane', async ({ page }) => {
await page.goto(`${BASE_URL}/#/map`);
await page.click('#mapPaneToggle');
const pane = page.locator('#mapSidePane');
await expect(pane).toHaveClass(/expanded/);
});
test('submit valid prefixes renders candidates within 1s', async ({ page }) => {
await page.goto(`${BASE_URL}/#/map`);
await page.click('#mapPaneToggle');
await page.fill('#mapPiInput', '2c,a1,f4');
await page.click('#mapPiSubmit');
// Wait for results or error (both indicate API round-trip complete).
await expect(page.locator('#mapPiResults table, #mapPiResults .no-results, #mapPiError')).toBeVisible({ timeout: 1000 });
});
test('Show on Map button draws polyline on map', async ({ page }) => {
await page.goto(`${BASE_URL}/#/map`);
await page.click('#mapPaneToggle');
await page.fill('#mapPiInput', '2c,a1');
await page.click('#mapPiSubmit');
// Wait for results.
const btn = page.locator('#mapPiResults button[data-idx="0"]');
await btn.waitFor({ timeout: 2000 });
await btn.click();
// Check that route layer has SVG polyline paths drawn.
const svg = page.locator('#leaflet-map .leaflet-overlay-pane svg path');
await expect(svg.first()).toBeVisible({ timeout: 2000 });
});
test('switching candidate clears prior polyline', async ({ page }) => {
await page.goto(`${BASE_URL}/#/map`);
await page.click('#mapPaneToggle');
await page.fill('#mapPiInput', '2c,a1');
await page.click('#mapPiSubmit');
const btn0 = page.locator('#mapPiResults button[data-idx="0"]');
await btn0.waitFor({ timeout: 2000 });
await btn0.click();
// Click second candidate if available.
const btn1 = page.locator('#mapPiResults button[data-idx="1"]');
if (await btn1.isVisible()) {
await btn1.click();
// Prior route should be cleared — only one polyline group visible.
}
});
});
test.describe('Path Inspector — Standalone Page', () => {
test('deep link auto-fills and runs', async ({ page }) => {
await page.goto(`${BASE_URL}/#/tools/path-inspector?prefixes=2c,a1,f4`);
const input = page.locator('#path-inspector-input');
await expect(input).toHaveValue('2c,a1,f4');
// Should auto-submit and show results or error.
await expect(page.locator('#path-inspector-results table, #path-inspector-results .no-results, #path-inspector-error')).toBeVisible({ timeout: 2000 });
});
test('old #/traces/<hash> redirects to #/tools/trace/<hash>', async ({ page }) => {
await page.goto(`${BASE_URL}/#/traces/abc123`);
await page.waitForTimeout(500);
expect(page.url()).toContain('#/tools/trace/abc123');
});
});
test.describe('Path Inspector — Tools Landing (spec §2.8)', () => {
test('Tools nav shows landing with both entries', async ({ page }) => {
await page.goto(`${BASE_URL}/#/tools`);
await expect(page.locator('.tools-landing')).toBeVisible();
await expect(page.locator('a[href="#/tools/path-inspector"]')).toBeVisible();
await expect(page.locator('a[href*="#/tools/trace"]')).toBeVisible();
});
});
+106
View File
@@ -0,0 +1,106 @@
// test-path-inspector.js — vm.createContext sandbox tests for path-inspector.js
'use strict';
const vm = require('vm');
const fs = require('fs');
const assert = require('assert');
const src = fs.readFileSync(__dirname + '/public/path-inspector.js', 'utf8');
function createSandbox() {
const sandbox = {
window: {},
document: {
getElementById: () => ({ textContent: '', innerHTML: '', addEventListener: () => {}, querySelectorAll: () => [] }),
querySelectorAll: () => []
},
location: { hash: '#/tools/path-inspector' },
history: { replaceState: () => {} },
fetch: () => Promise.resolve({ ok: true, json: () => Promise.resolve({ candidates: [] }) }),
URLSearchParams: URLSearchParams,
registerPage: function () {},
escapeHtml: s => s,
console: console
};
sandbox.self = sandbox;
sandbox.globalThis = sandbox;
const ctx = vm.createContext(sandbox);
vm.runInContext(src, ctx);
return sandbox;
}
// Test: parsePrefixes accepts comma-separated.
(function testParseComma() {
const sb = createSandbox();
const result = sb.window.PathInspector.parsePrefixes('2C,A1,F4');
assert.strictEqual(JSON.stringify(result), JSON.stringify(['2c', 'a1', 'f4']));
console.log('✓ parsePrefixes comma-separated');
})();
// Test: parsePrefixes accepts space-separated.
(function testParseSpace() {
const sb = createSandbox();
const result = sb.window.PathInspector.parsePrefixes('2C A1 F4');
assert.strictEqual(JSON.stringify(result), JSON.stringify(['2c', 'a1', 'f4']));
console.log('✓ parsePrefixes space-separated');
})();
// Test: parsePrefixes accepts mixed.
(function testParseMixed() {
const sb = createSandbox();
const result = sb.window.PathInspector.parsePrefixes(' 2C, A1 F4 ');
assert.strictEqual(JSON.stringify(result), JSON.stringify(['2c', 'a1', 'f4']));
console.log('✓ parsePrefixes mixed separators');
})();
// Test: validatePrefixes rejects empty.
(function testValidateEmpty() {
const sb = createSandbox();
const err = sb.window.PathInspector.validatePrefixes([]);
assert.ok(err !== null, 'should reject empty');
console.log('✓ validatePrefixes rejects empty');
})();
// Test: validatePrefixes rejects odd-length.
(function testValidateOdd() {
const sb = createSandbox();
const err = sb.window.PathInspector.validatePrefixes(['abc']);
assert.ok(err !== null && err.includes('Odd'), 'should reject odd-length');
console.log('✓ validatePrefixes rejects odd-length');
})();
// Test: validatePrefixes rejects >3 bytes.
(function testValidateTooLong() {
const sb = createSandbox();
const err = sb.window.PathInspector.validatePrefixes(['aabbccdd']);
assert.ok(err !== null && err.includes('too long'), 'should reject >3 bytes');
console.log('✓ validatePrefixes rejects >3 bytes');
})();
// Test: validatePrefixes rejects mixed lengths.
(function testValidateMixed() {
const sb = createSandbox();
const err = sb.window.PathInspector.validatePrefixes(['aa', 'bbcc']);
assert.ok(err !== null && err.includes('Mixed'), 'should reject mixed');
console.log('✓ validatePrefixes rejects mixed lengths');
})();
// Test: validatePrefixes accepts valid input.
(function testValidateValid() {
const sb = createSandbox();
const err = sb.window.PathInspector.validatePrefixes(['2c', 'a1', 'f4']);
assert.strictEqual(err, null);
console.log('✓ validatePrefixes accepts valid');
})();
// Test: validatePrefixes rejects invalid hex.
(function testValidateInvalidHex() {
const sb = createSandbox();
const err = sb.window.PathInspector.validatePrefixes(['zz']);
assert.ok(err !== null && err.includes('Invalid hex'), 'should reject invalid hex');
console.log('✓ validatePrefixes rejects invalid hex');
})();
// Anti-tautology: if validation were removed (always return null), the odd-length test would fail.
// Mental revert: validatePrefixes = () => null; → testValidateOdd would fail because err would be null.
console.log('\nAll path-inspector tests passed!');
+2 -1
View File
@@ -72,7 +72,8 @@ let polygon = null;
let closingLine = null;
function latLonPair(latlng) {
return [parseFloat(latlng.lat.toFixed(6)), parseFloat(latlng.lng.toFixed(6))];
const w = latlng.wrap();
return [parseFloat(w.lat.toFixed(6)), parseFloat(w.lng.toFixed(6))];
}
function render() {