Compare commits

...
Author SHA1 Message Date
you e837a9d416 fix: Details button opens full-screen node detail on desktop (#778)
The previous fix (#779) used hash navigation for the Details button,
which only re-selected the node in the side panel on desktop. Now the
Details button calls showFullScreenNode() to render the same full-screen
detail view that mobile users see, regardless of screen width.

- Extract showFullScreenNode() from mobile-only init code
- Reuse it for both mobile init and desktop Details button click
- Analytics links still navigate via hash as before
- Update Playwright test to verify full-screen view renders
2026-04-17 07:41:14 +00:00
dfe383cc51 fix: node detail panel Details/Analytics links don't navigate (#779)
Fixes #778

## Problem

The Details and Analytics links in the node side panel don't navigate
when clicked. This is a regression from #739 (desktop node deep
linking).

**Root cause:** When a node is selected, `selectNode()` uses
`history.replaceState()` to set the URL to `#/nodes/{pubkey}`. The
Details link has `href="#/nodes/{pubkey}"` — the same hash. Clicking an
anchor with the same hash as the current URL doesn't fire the
`hashchange` event, so the SPA router never triggers navigation.

## Fix

Added a click handler on the `nodesRight` panel that intercepts clicks
on `.btn-primary` navigation links:

1. `e.preventDefault()` to stop the default anchor behavior
2. If the current hash already matches the target, temporarily clear it
via `replaceState`
3. Set `location.hash` to the target, which fires `hashchange` and
triggers the SPA router

This handles both the Details link (`#/nodes/{pubkey}`) and the
Analytics link (`#/nodes/{pubkey}/analytics`).

## Testing

- All frontend helper tests pass (552/552)
- All packet filter tests pass (62/62)
- All aging tests pass (29/29)
- Go server tests pass

---------

Co-authored-by: you <you@example.com>
2026-04-16 23:21:05 -07:00
you fa348efe2a fix: force-remove staging container before deploy — handles both compose and docker-run containers
The deploy step used only 'docker compose down' which can't remove
containers created via 'docker run'. Now explicitly stops+removes
the named container first, then runs compose down as cleanup.

Permanent fix for the recurring CI deploy failure.
2026-04-17 05:08:32 +00:00
a9a18ff051 fix: neighbor graph slider persists to localStorage, default 0.7 (#776)
## Summary

The neighbor graph min score slider didn't persist its value to
localStorage, resetting to 0.10 on every page load. This was a poor
default for most use cases.

## Changes

- **Default changed from 0.10 to 0.70** — more useful starting point
that filters out low-confidence edges
- **localStorage persistence** — slider value saved on change, restored
on page load
- **3 new tests** in `test-frontend-helpers.js` verifying default value,
load behavior, and save behavior

## Testing

- `node test-frontend-helpers.js` — 547 passed, 0 failed
- `node test-packet-filter.js` — 62 passed, 0 failed
- `node test-aging.js` — 29 passed, 0 failed

---------

Co-authored-by: you <you@example.com>
2026-04-16 22:04:51 -07:00
ceea136e97 feat: observer graph representation (M1+M2) (#774)
## Summary

Fixes #753 — Milestones M1 and M2: Observer nodes in the neighbor graph
are now correctly labeled, colored, and filterable.

### M1: Label + color observers

**Backend** (`cmd/server/neighbor_api.go`):
- `buildNodeInfoMap()` now queries the `observers` table after building
from `nodes`
- Observer-only pubkeys (not already in the map as repeaters etc.) get
`role: "observer"` and their name from the observers table
- Observer-repeaters keep their repeater role (not overwritten)

**Frontend**:
- CSS variable `--role-observer: #8b5cf6` added to `:root`
- `ROLE_COLORS.observer` was already defined in `roles.js`

### M2: Observer filter checkbox (default unchecked)

**Frontend** (`public/analytics.js`):
- Observer checkbox added to the role filter section, **unchecked by
default**
- Observers create hub-and-spoke patterns (one observer can have 100+
edges) that drown out the actual repeater topology — hiding them by
default keeps the graph clean
- Fixed `applyNGFilters()` which previously always showed observers
regardless of checkbox state

### Tests

- Backend: `TestBuildNodeInfoMap_ObserverEnrichment` — verifies
observer-only pubkeys get name+role from observers table, and
observer-repeaters keep their repeater role
- All existing Go tests pass
- All frontend helper tests pass (544/544)

---------

Co-authored-by: you <you@example.com>
2026-04-16 21:35:14 -07:00
you 99dc4f805a fix: E2E neighbor test — use hash evaluation instead of page.goto for reliable SPA navigation
page.goto with hash-only change may not reliably trigger hashchange in
Playwright, causing the mobile full-screen node view to never render.
Use page.evaluate to set location.hash directly, which guarantees the
SPA router fires. Also increase timeout from 10s to 15s for CI margin.
2026-04-17 03:45:16 +00:00
ba7cd0fba7 fix: clock skew sanity checks — filter epoch-0, cap drift, min samples (#769)
Nodes with dead RTCs show -690d skew and -3 billion s/day drift. Fix:

1. **No Clock severity**: |skew| > 365d → `no_clock`, skip drift
2. **Drift cap**: |drift| > 86400 s/day → nil (physically impossible)
3. **Min samples**: < 5 samples → no drift regression
4. **Frontend**: 'No Clock' badge, '–' for unreliable drift

Fixes the crazy stats on the Clock Health fleet view.

---------

Co-authored-by: you <you@example.com>
2026-04-16 08:10:47 -07:00
6a648dea11 fix: multi-byte adopters — all node types, role column, advert precedence (#754) (#767)
## Fix: Multi-Byte Adopters Table — Three Bugs (#754)

### Bug 1: Companions in "Unknown"
`computeMultiByteCapability()` was repeater-only. Extended to classify
**all node types** (companions, rooms, sensors). A companion advertising
with 2-byte hash is now correctly "Confirmed".

### Bug 2: No Role Column
Added a **Role** column to the merged Multi-Byte Hash Adopters table,
color-coded using `ROLE_COLORS` from `roles.js`. Users can now
distinguish repeaters from companions without clicking through to node
detail.

### Bug 3: Data Source Disagreement
When adopter data (from `computeAnalyticsHashSizes`) shows `hashSize >=
2` but capability only found path evidence ("Suspected"), the
advert-based adopter data now takes precedence → "Confirmed". The
adopter hash sizes are passed into `computeMultiByteCapability()` as an
additional confirmed evidence source.

### Changes
- `cmd/server/store.go`: Extended capability to all node types, accept
adopter hash sizes, prioritize advert evidence
- `public/analytics.js`: Added Role column with color-coded badges
- `cmd/server/multibyte_capability_test.go`: 3 new tests (companion
confirmed, role populated, adopter precedence)

### Tests
- All 10 multi-byte capability tests pass
- All 544 frontend helper tests pass
- All 62 packet filter tests pass
- All 29 aging tests pass

---------

Co-authored-by: you <you@example.com>
2026-04-16 00:51:38 -07:00
29157742eb feat: show collision details in Hash Usage Matrix for all hash sizes (#758)
## Summary

Shows which prefixes are colliding in the Hash Usage Matrix, making the
"PREFIX COLLISIONS: N" count actionable.

Fixes #757

## Changes

### Frontend (`public/analytics.js`)
- **Clickable collision count**: When collisions > 0, the stat card is
clickable and scrolls to the collision details section. Shows a `▼`
indicator.
- **3-byte collision table**: The collision risk section and
`renderCollisionsFromServer` now render for all hash sizes including
3-byte (was previously hidden/skipped for 3-byte).
- **Helpful hint**: 3-byte panel now says "See collision details below"
when collisions exist.

### Backend (`cmd/server/collision_details_test.go`)
- Test that collision details include correct prefix and node
name/pubkey pairs
- Test that collision details are empty when no collisions exist

### Frontend Tests (`test-frontend-helpers.js`)
- Test clickable stat card renders `onclick` and `cursor:pointer` when
collisions > 0
- Test non-clickable card when collisions = 0
- Test collision table renders correct node links (`#/nodes/{pubkey}`)
- Test no-collision message renders correctly

## What was already there

The backend already returned full collision details (prefix, nodes with
pubkeys/names/coords, distance classification) in the `hash-collisions`
API. The frontend already had `renderCollisionsFromServer` rendering a
rich table with node links. The gap was:
1. The 3-byte tab hid the collision risk section entirely
2. No visual affordance to navigate from the stat count to the details

## Perf justification

No new computation — collision data was already computed and returned by
the API. The only change is rendering it for 3-byte (same as
1-byte/2-byte). The collision list is already limited by the backend
sort+slice pattern.

---------

Co-authored-by: you <you@example.com>
2026-04-16 00:18:25 -07:00
ed19a19473 fix: correct field table offsets for transport routes (#766)
## Summary

Fixes #765 — packet detail field table showed wrong byte offsets for
transport routes.

## Problem

`buildFieldTable()` hardcoded `path_length` at byte 1 for ALL packet
types. For `TRANSPORT_FLOOD` (route_type=0) and `TRANSPORT_DIRECT`
(route_type=3), transport codes occupy bytes 1-4, pushing `path_length`
to byte 5.

This caused:
- Wrong offset numbers in the field table for transport packets
- Transport codes displayed AFTER path length (wrong byte order)
- `Advertised Hash Size` row referenced wrong byte

## Fix

- Use dynamic `offset` tracking that accounts for transport codes
- Render transport code rows before path length (matching actual wire
format)
- Store `pathLenOffset` for correct reference in ADVERT payload section
- Reuse already-parsed `pathByte0` for hash size calculation in path
section

## Tests

Added 4 regression tests in `test-frontend-helpers.js`:
- FLOOD (route_type=1): path_length at byte 1, no transport codes
- TRANSPORT_FLOOD (route_type=0): transport codes at bytes 1-4,
path_length at byte 5
- TRANSPORT_DIRECT (route_type=3): same offsets as TRANSPORT_FLOOD
- Field table row order matches byte layout for transport routes

All existing tests pass (538 frontend helpers, 62 packet filter, 29
aging).

Co-authored-by: you <you@example.com>
2026-04-16 00:15:56 -07:00
copelajeandGitHub d27a7a653e fix case on channel key so Public decode/display works right (#761)
Simple change. Before this change Public wasn't showing up in the
channels display due to the case issue.
2026-04-16 00:14:47 -07:00
0e286d85fd fix: channel query performance — add channel_hash column, SQL-level filtering (#762) (#763)
## Problem
Channel API endpoints scan entire DB — 2.4s for channel list, 30s for
messages.

## Fix
- Added `channel_hash` column to transmissions (populated on ingest,
backfilled on startup)
- `GetChannels()` rewrites to GROUP BY channel_hash (one row per channel
vs scanning every packet)
- `GetChannelMessages()` filters by channel_hash at SQL level with
proper LIMIT/OFFSET
- 60s cache for channel list
- Index: `idx_tx_channel_hash` for fast lookups

Expected: 2.4s → <100ms for list, 30s → <500ms for messages.

Fixes #762

---------

Co-authored-by: you <you@example.com>
2026-04-16 00:09:36 -07:00
bffcbdaa0b feat: add channel UX — visible button, hint, status feedback (#760)
## Fixes #759

The "Add Channel" input was a bare text field with no visible submit
button and no feedback — users didn't know how to submit or whether it
worked.

### Changes

**`public/channels.js`**
- Replaced bare `<input>` with structured form: label, input + button
row, hint text, status div
- Added `showAddStatus()` helper for visual feedback during/after
channel add
- Status messages: loading → success (with decrypted message count) /
warning (no messages) / error
- Auto-hide status after 5 seconds
- Fallback click handler on the `+` button for browsers that don't fire
form submit

**`public/style.css`**
- `.ch-add-form` — form container
- `.ch-add-label` — bold 13px label
- `.ch-add-row` — flex row for input + button
- `.ch-add-btn` — 32×32 accent-colored submit button
- `.ch-add-hint` — muted helper text
- `.ch-add-status` — feedback with success/warn/error/loading variants

**`test-channel-add-ux.js`** — 20 tests validating HTML structure, CSS
classes, and feedback logic

### Before / After

**Before:** Bare input field, no button, no hint, no feedback  
**After:** Labeled section with visible `+` button, format hint, and
status messages showing decryption results

---------

Co-authored-by: you <you@example.com>
2026-04-15 18:54:35 -07:00
3bdf72b4cf feat: clock skew UI — node badges, detail sparkline, fleet analytics (#690 M2+M3) (#752)
## Summary

Frontend visualizations for clock skew detection.

Implements #690 M2 and M3. Does NOT close #690 — M4+M5 remain.

### M2: Node badges + detail sparkline
- Severity badges ( green/yellow/orange/red) on node list next to each
node
- Node detail: Clock Skew section with current value, severity, drift
rate
- Inline SVG sparkline showing skew history, color-coded by severity
zones

### M3: Fleet analytics view
- 'Clock Health' section on Analytics page
- Sortable table: Name | Skew | Severity | Drift | Last Advert
- Filter buttons by severity (OK/Warning/Critical/Absurd)
- Summary stats: X nodes OK, Y warning, Z critical
- Color-coded rows

### Changes
- `public/nodes.js` — badge rendering + detail section
- `public/analytics.js` — fleet clock health view
- `public/roles.js` — severity color helpers
- `public/style.css` — badge + sparkline + fleet table styles
- `cmd/server/clock_skew.go` — added fleet summary endpoint
- `cmd/server/routes.go` — wired fleet endpoint
- `test-frontend-helpers.js` — 11 new tests

---------

Co-authored-by: you <you@example.com>
2026-04-15 15:25:50 -07:00
401fd070f8 fix: improve trackedBytes accuracy for memory estimation (#751)
## Problem

Fixes #743 — High memory usage / OOM with relatively small dataset.

`trackedBytes` severely undercounted actual per-packet memory because it
only tracked base struct sizes and string field lengths, missing major
allocations:

| Structure | Untracked Cost | Scale Impact |
|-----------|---------------|--------------|
| `spTxIndex` (O(path²) subpath entries) | 40 bytes × path combos |
50-150MB |
| `ResolvedPath` on observations | 24 bytes × elements | ~25MB |
| Per-tx maps (`obsKeys`, `observerSet`) | 200 bytes/tx flat | ~11MB |
| `byPathHop` index entries | 50 bytes/hop | 20-40MB |

This caused eviction to trigger too late (or not at all), leading to
OOM.

## Fix

Expanded `estimateStoreTxBytes` and `estimateStoreObsBytes` to account
for:

- **Per-tx maps**: +200 bytes flat for `obsKeys` + `observerSet` map
headers
- **Path hop index**: +50 bytes per hop in `byPathHop`
- **Subpath index**: +40 bytes × `hops*(hops-1)/2` combinations for
`spTxIndex`
- **Resolved paths**: +24 bytes per `ResolvedPath` element on
observations

Updated the existing `TestEstimateStoreTxBytes` to match new formula.
All existing eviction tests continue to pass — the eviction logic itself
is unchanged.

Also exposed `avgBytesPerPacket` in the perf API (`/api/perf`) so
operators can monitor per-packet memory costs.

## Performance

Benchmark confirms negligible overhead (called on every insert):

```
BenchmarkEstimateStoreTxBytes    159M ops    7.5 ns/op    0 B/op    0 allocs
BenchmarkEstimateStoreObsBytes   1B ops      1.0 ns/op    0 B/op    0 allocs
```

## Tests

- 6 new tests in `tracked_bytes_test.go`:
  - Reasonable value ranges for different packet sizes
  - 10-hop packets estimate significantly more than 2-hop (subpath cost)
  - Observations with `ResolvedPath` estimate more than without
  - 15 observations estimate >10x a single observation
- `trackedBytes` matches sum of individual estimates after batch insert
  - Eviction triggers correctly with improved estimates
- 2 benchmarks confirming sub-10ns estimate cost
- Updated existing `TestEstimateStoreTxBytes` for new formula
- Full test suite passes

---------

Co-authored-by: you <you@example.com>
2026-04-15 07:53:32 -07:00
1b315bf6d0 feat: PSK channels, channel removal, message caching (#725 M3+M4+M5) (#750)
## Summary

Implements milestones M3, M4, and M5 from #725 — all client-side, zero
server changes.

### M3: PSK channel support

The channel input field now accepts both `#channelname` (hashtag
derivation) and raw 32-char hex keys (PSK). Auto-detection: if input
starts with `#`, derive key via SHA-256; otherwise validate as hex and
store directly. Same decrypt pipeline — `ChannelDecrypt.decrypt()` takes
key bytes regardless of source.

Input placeholder updated to: `#LongFast or paste hex key`

### M4: Channel removal

User-added channels now show a ✕ button on hover. Click → confirm dialog
→ removes:
- Key from localStorage (`ChannelDecrypt.removeKey()`)
- Cached messages from localStorage
(`ChannelDecrypt.clearChannelCache()`)
- Channel entry from sidebar

If the removed channel was selected, the view resets to the empty state.

### M5: localStorage message caching with delta fetch

After client-side decryption, results are cached in localStorage keyed
by channel name:

```
{ messages: [...], lastTimestamp: "...", count: N, ts: Date.now() }
```

On subsequent visits:
1. **Instant render** — cached messages displayed immediately via
`onCacheHit` callback
2. **Delta fetch** — only packets newer than `lastTimestamp` are fetched
and decrypted
3. **Merge** — new messages merged with cache, deduplicated by
`packetHash`
4. **Cache invalidation** — if total candidate count changes, full
re-decrypt triggered
5. **Size limit** — max 1000 messages cached per channel (most recent
kept)

### Performance

- Delta fetch avoids re-decrypting the full history on every page load
- Cache-first rendering provides instant UI response
- `deduplicateAndMerge()` uses a hash set for O(n) dedup
- 1000-message cap prevents localStorage quota issues

### Tests (24 new)

- M3: hex key detection (valid/invalid patterns)
- M3: key derivation round-trip, channel hash computation
- M3: PSK key storage and retrieval
- M4: channel removal clears both key and cache
- M5: cache size limit enforcement (1200 → 1000 stored)
- M5: cache stores count and lastTimestamp
- M5: clearChannelCache works independently
- All existing tests pass (523 frontend helpers, 62 packet filter)

### Files changed

| File | Change |
|------|--------|
| `public/channel-decrypt.js` | `removeKey()` now clears cache;
`clearChannelCache()`; `setCache()` with count + size limit |
| `public/channels.js` | Extracted `decryptCandidates()`,
`deduplicateAndMerge()`; delta fetch logic; remove button handler;
cache-first rendering |
| `public/style.css` | `.ch-remove-btn` styles (hover-reveal ✕) |
| `test-channel-decrypt-m345.js` | 24 new tests |

Implements #725

Co-authored-by: you <you@example.com>
2026-04-14 23:23:02 -07:00
a815e70975 feat: Clock skew detection — backend computation (M1) (#746)
## Summary

Implements **Milestone 1** of #690 — backend clock skew computation for
nodes and observers.

## What's New

### Clock Skew Engine (`clock_skew.go`)

**Phase 1 — Raw Skew Calculation:**
For every ADVERT observation: `raw_skew = advert_timestamp -
observation_timestamp`

**Phase 2 — Observer Calibration:**
Same packet seen by multiple observers → compute each observer's clock
offset as the median deviation from the per-packet median observation
timestamp. This identifies observers with their own clock drift.

**Phase 3 — Corrected Node Skew:**
`corrected_skew = raw_skew + observer_offset` — compensates for observer
clock error.

**Phase 4 — Trend Analysis:**
Linear regression over time-ordered skew samples estimates drift rate in
seconds/day. Detects crystal drift vs stable offset vs sudden jumps.

### Severity Classification

| Level | Threshold | Meaning |
|-------|-----------|---------|
|  OK | < 5 min | Normal |
| ⚠️ Warning | 5 min – 1 hour | Clock drifting |
| 🔴 Critical | 1 hour – 30 days | Likely no time source |
| 🟣 Absurd | > 30 days | Firmware default or epoch 0 |

### New API Endpoints

- `GET /api/nodes/{pubkey}/clock-skew` — per-node skew data (mean,
median, last, drift, severity)
- `GET /api/observers/clock-skew` — observer calibration offsets
- Clock skew also included in `GET /api/nodes/{pubkey}/analytics`
response as `clockSkew` field

### Performance

- 30-second compute cache avoids reprocessing on every request
- Operates on in-memory `byPayloadType[ADVERT]` index — no DB queries
- O(n) in total ADVERT observations, O(m log m) for median calculations

## Tests

15 unit tests covering:
- Severity classification at all thresholds
- Median/mean math helpers
- ISO timestamp parsing
- Timestamp extraction from decoded JSON (nested and top-level)
- Observer calibration with single and multi-observer scenarios
- Observer offset correction direction (verified the sign is
`+obsOffset`)
- Drift estimation: stable, linear, insufficient data, short time span
- JSON number extraction edge cases

## What's NOT in This PR

- No UI changes (M2–M4)
- No customizer integration (M5)
- Thresholds are hardcoded constants (will be configurable in M5)

Implements #690 M1.

---------

Co-authored-by: you <you@example.com>
2026-04-14 23:22:35 -07:00
aa84ce1e6a fix: correct hash_size detection for transport routes and zero-hop adverts (#747)
## Summary

Fixes #744
Fixes #722

Three bugs in hash_size computation caused zero-hop adverts to
incorrectly report `hash_size=1`, masking nodes that actually use
multi-byte hashes.

## Bugs Fixed

### 1. Wrong path byte offset for transport routes
(`computeNodeHashSizeInfo`)

Transport routes (types 0 and 3) have 4 transport code bytes before the
path byte. The code read the path byte from offset 1 (byte index
`RawHex[2:4]`) for all route types. For transport routes, the correct
offset is 5 (`RawHex[10:12]`).

### 2. Missing RouteTransportDirect skip (`computeNodeHashSizeInfo`)

Zero-hop adverts from `RouteDirect` (type 2) were correctly skipped, but
`RouteTransportDirect` (type 3) zero-hop adverts were not. Both have
locally-generated path bytes with unreliable hash_size bits.

### 3. Zero-hop adverts not skipped in analytics
(`computeAnalyticsHashSizes`)

`computeAnalyticsHashSizes()` unconditionally overwrote a node's
`hashSize` with whatever the latest advert reported. A zero-hop direct
advert with `hash_size=1` could overwrite a previously-correct
`hash_size=2` from a multi-hop flood advert.

Fix: skip hash_size update for zero-hop direct/transport-direct adverts
while still counting the packet and updating `lastSeen`.

## Tests Added

- `TestHashSizeTransportRoutePathByteOffset` — verifies transport routes
read path byte at offset 5, regular flood reads at offset 1
- `TestHashSizeTransportDirectZeroHopSkipped` — verifies both
RouteDirect and RouteTransportDirect zero-hop adverts are skipped
- `TestAnalyticsHashSizesZeroHopSkip` — verifies analytics hash_size is
not overwritten by zero-hop adverts
- Fixed 3 existing tests (`FlipFlop`, `Dominant`, `LatestWins`) that
used route_type 0 (TransportFlood) header bytes without proper transport
code padding

## Complexity

All changes are O(1) per packet — no new loops or data structures. The
additional offset computation and zero-hop check are constant-time
operations within the existing packet scan loop.

Co-authored-by: you <you@example.com>
2026-04-14 23:04:26 -07:00
2aea01f10c fix: merge repeater+observer into single map marker (#745)
## Problem

When a node acts as both a repeater and an observer (same public key —
common with powered repeaters running MQTT clients), the map shows two
separate markers at the same location: a repeater rectangle and an
observer star. This causes visual clutter and both markers get pushed
out from the real location by the deconfliction algorithm.

## Solution

Detect combined repeater+observer nodes by matching node `public_key`
against observer `id`. When matched:

- **Label mode (hash labels on):** The repeater label gets a gold ★
appended inside the rectangle
- **Icon mode (hash labels off):** The repeater diamond gets a small
star overlay in the top-right corner of the SVG
- **Popup:** Shows both REPEATER and OBSERVER badges
- **Observer markers:** Skipped when the observer is already represented
as a combined node marker
- **Legend count:** Observer count excludes combined nodes (shows
standalone observers only)

## Performance

- Observer lookup uses a `Map` keyed by lowercase pubkey — O(1) per node
check
- Legend count uses a `Set` of node pubkeys — O(n+m) instead of O(n×m)
- No additional API calls; uses existing `observers` array already
fetched

## Testing

- All 523 frontend helper tests pass
- All 62 packet filter tests pass
- Visual: combined nodes show as single marker with star indicator

Fixes #719

---------

Co-authored-by: you <you@example.com>
2026-04-14 22:47:28 -07:00
b7c2cb070c docs: geofilter manual + config.example.json entry (#734)
## Summary

- Add missing `geo_filter` block to `config.example.json` with polygon
example, `bufferKm`, and inline `_comment`
- Add `docs/user-guide/geofilter.md`: full operator guide covering
config schema, GeoFilter Builder workflow, and prune script as one-time
migration tool
- Add Geographic filtering section to `docs/user-guide/configuration.md`
with link to the full guide

Closes #669 (M1: documentation)

## Test plan

- [x] `config.example.json` parses cleanly (no JSON errors)
- [x] `docs/user-guide/geofilter.md` renders correctly in GitHub preview
- [x] Link from `configuration.md` to `geofilter.md` resolves

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:43:19 -07:00
1de80a9eaf feat: serve geofilter builder from app, link from customizer (#735)
## Summary

Part of #669 — M2: Link the builder from the app.

- **`public/geofilter-builder.html`** — the existing
`tools/geofilter-builder.html` is now served by the static file server
at `/geofilter-builder.html`. Additions vs the original: a `← CoreScope`
back-link in the header, inline code comments explaining the output
format, and a help bar below the output panel with paste instructions
and a link to the documentation.
- **`public/customize-v2.js`** — adds a "Tools" section at the bottom of
the Export tab with a `🗺️ GeoFilter Builder →` link and a one-line
description.
- **`docs/user-guide/customization.md`** — documents the new GeoFilter
Builder entry in the Export tab.

> **Note:** `tools/geofilter-builder.html` is kept as-is for
local/offline use. The `public/` copy is what the server serves.

> **Depends on:** #734 (M1 docs) for `docs/user-guide/geofilter.md` —
the link in the help bar references that file. Can be merged
independently; the link still works once M1 lands.

## Test plan

- [x] Open the app, go to Customizer → Export tab — "Tools" section
appears with GeoFilter Builder link
- [x] Click the link — opens `/geofilter-builder.html` in a new tab
- [x] Builder loads the Leaflet map, draw 3+ points — JSON output
appears
- [x] Copy button works, output is valid `{ "geo_filter": { ... } }`
JSON
- [x] `← CoreScope` back-link navigates to `/`
- [x] Help bar shows paste instructions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:42:27 -07:00
e6ace95059 fix: desktop node click updates URL hash, deep link opens split panel (#676) (#739)
## Problem

Clicking a node on desktop opened the side panel but never updated the
URL hash, making nodes non-shareable/bookmarkable on desktop. Loading
`#/nodes/{pubkey}` directly on desktop also incorrectly showed the
full-screen mobile view.

## Changes

- `selectNode()` on desktop: adds `history.replaceState(null, '',
'#/nodes/' + pubkey)` so the URL updates on every click
- `init()`: full-screen path is now gated to `window.innerWidth <= 640`
(mobile only); desktop with a `routeParam` falls through to the split
panel and calls `selectNode()` to pre-select the node
- Deselect (Escape / close button): also calls `history.replaceState`
back to `#/nodes`

## Test plan

- [x] Desktop: click a node → URL updates to `#/nodes/{pubkey}`, split
panel opens
- [x] Desktop: copy URL, open in new tab → split panel opens with that
node selected (not full-screen)
- [x] Desktop: press Escape → URL reverts to `#/nodes`
- [x] Mobile (≤640px): clicking a node still navigates to full-screen
view

Closes #676

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:38:06 -07:00
f605d4ce7e fix: serialize filter params in URL hash for deep linking (#682) (#740)
## Problem

Applying packet filters (hash, node, observer, Wireshark expression) did
not update the URL hash, so filtered views could not be shared or
bookmarked.

## Changes

**`buildPacketsQuery()`** — extended to include:
- `hash=` from `filters.hash`
- `node=` from `filters.node`  
- `observer=` from `filters.observer`
- `filter=` from `filters._filterExpr` (Wireshark expression string)

**`updatePacketsUrl()`** — now called on every filter change:
- hash input (debounced)
- observer multi-select change
- node autocomplete select and clear
- Wireshark filter input (on valid expression or clear)

**URL restore on load** — `getHashParams()` now reads `hash`, `node`,
`observer`, `filter` and restores them into `filters` before the DOM is
built. Input fields pick up values from `filters` as before. Wireshark
expression is also recompiled and `filter-active` class applied.

## Test plan

- [ ] Type in hash filter → URL updates with `&hash=...`
- [ ] Copy URL, open in new tab → hash filter is pre-filled
- [ ] Select an observer → URL updates with `&observer=...`
- [ ] Select a node filter → URL updates with `&node=...`
- [ ] Type `type=ADVERT` in Wireshark filter → URL updates with
`&filter=type%3DADVERT`
- [ ] Load that URL → filter expression restored and active

Closes #682

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 22:37:33 -07:00
84f03f4f41 fix: hide undecryptable channel messages by default (#727) (#728)
## Problem

Channels page shows 53K 'Unknown' messages — undecryptable GRP_TXT
packets with no content. Pure noise.

## Fix

- Backend: channels API filters out undecrypted messages by default
- `?includeEncrypted=true` param to include them
- Frontend: 'Show encrypted' toggle in channels sidebar
- Unknown channels grayed out with '(no key)' label
- Toggle persists in localStorage

Fixes #727

---------

Co-authored-by: you <you@example.com>
2026-04-13 19:40:20 +00:00
8158631d02 feat: client-side channel decryption — add custom channels in browser (#725 M2) (#733)
## Summary

Pure client-side channel decryption. Users can add custom hashtag
channels or PSK channels directly in the browser. **The server never
sees the keys.**

Implements #725 M2 (revised). Does NOT close #725.

## How it works

1. User types `#channelname` or pastes a hex PSK in the channels sidebar
2. Browser derives key (`SHA256("#name")[:16]`) using Web Crypto API
3. Key stored in **localStorage** — never sent to the server
4. Browser fetches encrypted GRP_TXT packets via existing API
5. Browser decrypts client-side: AES-128-ECB + HMAC-SHA256 MAC
verification
6. Decrypted messages cached in localStorage
7. Progressive rendering — newest messages first, chunk-based

## Security

- Keys never leave the browser
- No new API endpoints
- No server-side changes whatsoever
- Channel interest partially observable via hash-based API requests
(documented, acceptable tradeoff)

## Changes

- `public/channels.js` — client-side decrypt module + UI integration
(+307 lines)
- `public/index.html` — no new script (inline in channels.js IIFE)
- `public/style.css` — add-channel input styling

---------

Co-authored-by: you <you@example.com>
2026-04-13 12:28:41 -07:00
14367488e2 fix: TRACE path_json uses path_sz from flags byte, not header hash_size (#732)
## Summary

TRACE packets encode their route hash size in the flags byte (`flags &
0x03`), not the header path byte. The decoder was using `path.HashSize`
from the header, which could be wrong or zero for direct-route TRACEs,
producing incorrect hop counts in `path_json`.

## Protocol Note

Per firmware, TRACE packets are **always direct-routed** (route_type 2 =
DIRECT, or 3 = TRANSPORT_DIRECT). FLOOD-routed TRACEs (route_type 1) are
anomalous — firmware explicitly rejects TRACE via flood. The decoder
handles these gracefully without crashing.

## Changes

**`cmd/server/decoder.go` and `cmd/ingestor/decoder.go`:**
- Read `pathSz` from TRACE flags byte: `(traceFlags & 0x03) + 1`
(0→1byte, 1→2byte, 2→3byte)
- Use `pathSz` instead of `path.HashSize` for splitting TRACE payload
path data into hops
- Update `path.HashSize` to reflect the actual TRACE path size
- Added `HopsCompleted` field to ingestor `Path` struct for parity with
server
- Updated comments to clarify TRACE is always direct-routed per firmware

**`cmd/server/decoder_test.go` — 5 new tests:**
- `TraceFlags1_TwoBytePathSz`: flags=1 → 2-byte hashes via DIRECT route
- `TraceFlags2_ThreeBytePathSz`: flags=2 → 3-byte hashes via DIRECT
route
- `TracePathSzUnevenPayload`: payload not evenly divisible by path_sz
- `TraceTransportDirect`: route_type=3 with transport codes + TRACE path
parsing
- `TraceFloodRouteGraceful`: anomalous FLOOD+TRACE handled without crash

All existing TRACE tests (flags=0, 1-byte hashes) continue to pass.

Fixes #731

---------

Co-authored-by: you <you@example.com>
2026-04-13 08:20:09 -07:00
71be54f085 feat: DB-backed channel messages for full history (#725 M1) (#726)
## Summary

Switches channel API endpoints to query SQLite instead of the in-memory
packet store, giving users access to the full message history.

Implements #725 (M1 only — DB-backed channel messages). Does NOT close
#725 — M2-M5 (custom channels, PSK, persistence, retroactive decryption)
remain.

## Problem

Channel endpoints (`/api/channels`, `/api/channels/{hash}/messages`)
preferred the in-memory packet store when available. The store is
bounded by `packetStore.maxMemoryMB` — typically showing only recent
messages. The SQLite database has the complete history (weeks/months of
channel messages) but was only used as a fallback when the store was nil
(never in production).

## Fix

Reversed the preference order: DB first, in-memory store fallback.
Region filtering added to the DB path.

Co-authored-by: you <you@example.com>
2026-04-12 23:22:52 -07:00
46 changed files with 5136 additions and 247 deletions
+4 -1
View File
@@ -387,7 +387,10 @@ jobs:
- name: Deploy staging
run: |
# Stop old container and release memory
# Force-remove the staging container regardless of how it was created
# (compose-managed OR manually created via docker run)
docker stop corescope-staging-go 2>/dev/null || true
docker rm -f corescope-staging-go 2>/dev/null || true
docker compose -f "$STAGING_COMPOSE_FILE" -p corescope-staging down --timeout 30 2>/dev/null || true
# Wait for container to be fully gone and OS to reclaim memory (3GB limit)
+46 -4
View File
@@ -345,6 +345,28 @@ func applySchema(db *sql.DB) error {
log.Println("[migration] packets_sent/packets_recv columns added")
}
// Migration: add channel_hash column for fast channel queries (#762)
row = db.QueryRow("SELECT 1 FROM _migrations WHERE name = 'channel_hash_v1'")
if row.Scan(&migDone) != nil {
log.Println("[migration] Adding channel_hash column to transmissions...")
db.Exec(`ALTER TABLE transmissions ADD COLUMN channel_hash TEXT DEFAULT NULL`)
db.Exec(`CREATE INDEX IF NOT EXISTS idx_tx_channel_hash ON transmissions(channel_hash) WHERE payload_type = 5`)
// Backfill: extract channel name for decrypted (CHAN) packets
res, err := db.Exec(`UPDATE transmissions SET channel_hash = json_extract(decoded_json, '$.channel') WHERE payload_type = 5 AND channel_hash IS NULL AND json_extract(decoded_json, '$.type') = 'CHAN'`)
if err == nil {
n, _ := res.RowsAffected()
log.Printf("[migration] Backfilled channel_hash for %d CHAN packets", n)
}
// Backfill: extract channelHashHex for encrypted (GRP_TXT) packets, prefixed with 'enc_'
res, err = db.Exec(`UPDATE transmissions SET channel_hash = 'enc_' || json_extract(decoded_json, '$.channelHashHex') WHERE payload_type = 5 AND channel_hash IS NULL AND json_extract(decoded_json, '$.type') = 'GRP_TXT'`)
if err == nil {
n, _ := res.RowsAffected()
log.Printf("[migration] Backfilled channel_hash for %d GRP_TXT packets", n)
}
db.Exec(`INSERT INTO _migrations (name) VALUES ('channel_hash_v1')`)
log.Println("[migration] channel_hash column added and backfilled")
}
return nil
}
@@ -357,8 +379,8 @@ func (s *Store) prepareStatements() error {
}
s.stmtInsertTransmission, err = s.db.Prepare(`
INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, payload_version, decoded_json)
VALUES (?, ?, ?, ?, ?, ?, ?)
INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, payload_version, decoded_json, channel_hash)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`)
if err != nil {
return err
@@ -481,7 +503,7 @@ func (s *Store) InsertTransmission(data *PacketData) (bool, error) {
result, err := s.stmtInsertTransmission.Exec(
data.RawHex, hash, now,
data.RouteType, data.PayloadType, data.PayloadVersion,
data.DecodedJSON,
data.DecodedJSON, nilIfEmpty(data.ChannelHash),
)
if err != nil {
s.Stats.WriteErrors.Add(1)
@@ -773,6 +795,15 @@ type PacketData struct {
PayloadVersion int
PathJSON string
DecodedJSON string
ChannelHash string // grouping key for channel queries (#762)
}
// nilIfEmpty returns nil for empty strings (for nullable DB columns).
func nilIfEmpty(s string) interface{} {
if s == "" {
return nil
}
return s
}
// MQTTPacketMessage is the JSON payload from an MQTT raw packet message.
@@ -794,7 +825,7 @@ func BuildPacketData(msg *MQTTPacketMessage, decoded *DecodedPacket, observerID,
pathJSON = string(b)
}
return &PacketData{
pd := &PacketData{
RawHex: msg.Raw,
Timestamp: now,
ObserverID: observerID,
@@ -810,4 +841,15 @@ func BuildPacketData(msg *MQTTPacketMessage, decoded *DecodedPacket, observerID,
PathJSON: pathJSON,
DecodedJSON: PayloadJSON(&decoded.Payload),
}
// Populate channel_hash for fast channel queries (#762)
if decoded.Header.PayloadType == PayloadGRP_TXT {
if decoded.Payload.Type == "CHAN" && decoded.Payload.Channel != "" {
pd.ChannelHash = decoded.Payload.Channel
} else if decoded.Payload.Type == "GRP_TXT" && decoded.Payload.ChannelHashHex != "" {
pd.ChannelHash = "enc_" + decoded.Payload.ChannelHashHex
}
}
return pd
}
+30 -9
View File
@@ -80,9 +80,10 @@ type TransportCodes struct {
// Path holds decoded path/hop information.
type Path struct {
HashSize int `json:"hashSize"`
HashCount int `json:"hashCount"`
Hops []string `json:"hops"`
HashSize int `json:"hashSize"`
HashCount int `json:"hashCount"`
Hops []string `json:"hops"`
HopsCompleted *int `json:"hopsCompleted,omitempty"`
}
// AdvertFlags holds decoded advert flag bits.
@@ -143,6 +144,7 @@ type DecodedPacket struct {
Path Path `json:"path"`
Payload Payload `json:"payload"`
Raw string `json:"raw"`
Anomaly string `json:"anomaly,omitempty"`
}
func decodeHeader(b byte) Header {
@@ -586,17 +588,35 @@ func DecodePacket(hexString string, channelKeys map[string]string, validateSigna
payload := decodePayload(header.PayloadType, payloadBuf, channelKeys, validateSignatures)
// TRACE packets store hop IDs in the payload (buf[9:]) rather than the header
// path field. The header path byte still encodes hashSize in bits 6-7, which
// we use to split the payload path data into individual hop prefixes.
// path field. Firmware always sends TRACE as DIRECT (route_type 2 or 3);
// FLOOD-routed TRACEs are anomalous but handled gracefully (parsed, but
// flagged). The TRACE flags byte (payload offset 8) encodes path_sz in
// bits 0-1 as a power-of-two exponent: hash_bytes = 1 << path_sz.
// NOT the header path byte's hash_size bits. The header path contains SNR
// bytes — one per hop that actually forwarded.
// We expose hopsCompleted (count of SNR bytes) so consumers can distinguish
// how far the trace got vs the full intended route.
var anomaly string
if header.PayloadType == PayloadTRACE && payload.PathData != "" {
// Flag anomalous routing — firmware only sends TRACE as DIRECT
if header.RouteType != RouteDirect && header.RouteType != RouteTransportDirect {
anomaly = "TRACE packet with non-DIRECT routing (expected DIRECT or TRANSPORT_DIRECT)"
}
// The header path hops count represents SNR entries = completed hops
hopsCompleted := path.HashCount
pathBytes, err := hex.DecodeString(payload.PathData)
if err == nil && path.HashSize > 0 {
hops := make([]string, 0, len(pathBytes)/path.HashSize)
for i := 0; i+path.HashSize <= len(pathBytes); i += path.HashSize {
hops = append(hops, strings.ToUpper(hex.EncodeToString(pathBytes[i:i+path.HashSize])))
if err == nil && payload.TraceFlags != nil {
// path_sz from flags byte is a power-of-two exponent per firmware:
// hash_bytes = 1 << (flags & 0x03)
pathSz := 1 << (*payload.TraceFlags & 0x03)
hops := make([]string, 0, len(pathBytes)/pathSz)
for i := 0; i+pathSz <= len(pathBytes); i += pathSz {
hops = append(hops, strings.ToUpper(hex.EncodeToString(pathBytes[i:i+pathSz])))
}
path.Hops = hops
path.HashCount = len(hops)
path.HashSize = pathSz
path.HopsCompleted = &hopsCompleted
}
}
@@ -616,6 +636,7 @@ func DecodePacket(hexString string, channelKeys map[string]string, validateSigna
Path: path,
Payload: payload,
Raw: strings.ToUpper(hexString),
Anomaly: anomaly,
}, nil
}
+1
View File
@@ -440,6 +440,7 @@ func handleMessage(store *Store, tag string, source MQTTSource, m mqtt.Message,
PayloadType: 5, // GRP_TXT
PathJSON: "[]",
DecodedJSON: string(decodedJSON),
ChannelHash: channelName, // fast channel queries (#762)
}
if _, err := store.InsertTransmission(pktData); err != nil {
+580
View File
@@ -0,0 +1,580 @@
package main
import (
"math"
"sort"
"sync"
"time"
)
// ── Clock Skew Severity ────────────────────────────────────────────────────────
type SkewSeverity string
const (
SkewOK SkewSeverity = "ok" // < 5 min
SkewWarning SkewSeverity = "warning" // 5 min 1 hour
SkewCritical SkewSeverity = "critical" // 1 hour 30 days
SkewAbsurd SkewSeverity = "absurd" // > 30 days
SkewNoClock SkewSeverity = "no_clock" // > 365 days — uninitialized RTC
)
// Default thresholds in seconds.
const (
skewThresholdWarnSec = 5 * 60 // 5 minutes
skewThresholdCriticalSec = 60 * 60 // 1 hour
skewThresholdAbsurdSec = 30 * 24 * 3600 // 30 days
skewThresholdNoClockSec = 365 * 24 * 3600 // 365 days — uninitialized RTC
// minDriftSamples is the minimum number of advert transmissions needed
// to compute a meaningful linear drift rate.
minDriftSamples = 5
// maxReasonableDriftPerDay caps drift display. Physically impossible
// drift rates (> 1 day/day) indicate insufficient or outlier samples.
maxReasonableDriftPerDay = 86400.0
)
// classifySkew maps absolute skew (seconds) to a severity level.
// Float64 comparison is safe: inputs are rounded to 1 decimal via round(),
// and thresholds are integer multiples of 60 — no rounding artifacts.
func classifySkew(absSkewSec float64) SkewSeverity {
switch {
case absSkewSec >= skewThresholdNoClockSec:
return SkewNoClock
case absSkewSec >= skewThresholdAbsurdSec:
return SkewAbsurd
case absSkewSec >= skewThresholdCriticalSec:
return SkewCritical
case absSkewSec >= skewThresholdWarnSec:
return SkewWarning
default:
return SkewOK
}
}
// ── Data Types ─────────────────────────────────────────────────────────────────
// skewSample is a single raw skew measurement from one advert observation.
type skewSample struct {
advertTS int64 // node's advert Unix timestamp
observedTS int64 // observation Unix timestamp
observerID string // which observer saw this
hash string // transmission hash (for multi-observer grouping)
}
// ObserverCalibration holds the computed clock offset for an observer.
type ObserverCalibration struct {
ObserverID string `json:"observerID"`
OffsetSec float64 `json:"offsetSec"` // positive = observer clock ahead
Samples int `json:"samples"` // number of multi-observer packets used
}
// NodeClockSkew is the API response for a single node's clock skew data.
type NodeClockSkew struct {
Pubkey string `json:"pubkey"`
MeanSkewSec float64 `json:"meanSkewSec"` // corrected mean skew (positive = node ahead)
MedianSkewSec float64 `json:"medianSkewSec"` // corrected median skew
LastSkewSec float64 `json:"lastSkewSec"` // most recent corrected skew
DriftPerDaySec float64 `json:"driftPerDaySec"` // linear drift rate (sec/day)
Severity SkewSeverity `json:"severity"`
SampleCount int `json:"sampleCount"`
Calibrated bool `json:"calibrated"` // true if observer calibration was applied
LastAdvertTS int64 `json:"lastAdvertTS"` // most recent advert timestamp
LastObservedTS int64 `json:"lastObservedTS"` // most recent observation timestamp
Samples []SkewSample `json:"samples,omitempty"` // time-series for sparklines
NodeName string `json:"nodeName,omitempty"` // populated in fleet responses
NodeRole string `json:"nodeRole,omitempty"` // populated in fleet responses
}
// SkewSample is a single (timestamp, skew) point for sparkline rendering.
type SkewSample struct {
Timestamp int64 `json:"ts"` // Unix epoch of observation
SkewSec float64 `json:"skew"` // corrected skew in seconds
}
// txSkewResult maps tx hash → per-transmission skew stats. This is an
// intermediate result keyed by hash (not pubkey); the store maps hash → pubkey
// when building the final per-node view.
type txSkewResult = map[string]*NodeClockSkew
// ── Clock Skew Engine ──────────────────────────────────────────────────────────
// ClockSkewEngine computes and caches clock skew data for nodes and observers.
type ClockSkewEngine struct {
mu sync.RWMutex
observerOffsets map[string]float64 // observerID → calibrated offset (seconds)
observerSamples map[string]int // observerID → number of multi-observer packets used
nodeSkew txSkewResult
lastComputed time.Time
computeInterval time.Duration
}
func NewClockSkewEngine() *ClockSkewEngine {
return &ClockSkewEngine{
observerOffsets: make(map[string]float64),
observerSamples: make(map[string]int),
nodeSkew: make(txSkewResult),
computeInterval: 30 * time.Second,
}
}
// Recompute recalculates all clock skew data from the packet store.
// Called periodically or on demand. Holds store RLock externally.
// Uses read-copy-update: heavy computation runs outside the write lock,
// then results are swapped in under a brief lock.
func (e *ClockSkewEngine) Recompute(store *PacketStore) {
// Fast path: check under read lock if recompute is needed.
e.mu.RLock()
fresh := time.Since(e.lastComputed) < e.computeInterval
e.mu.RUnlock()
if fresh {
return
}
// Phase 1: Collect skew samples from ADVERT packets (store RLock held by caller).
samples := collectSamples(store)
// Phase 23: Compute outside the write lock.
var newOffsets map[string]float64
var newSamples map[string]int
var newNodeSkew txSkewResult
if len(samples) > 0 {
newOffsets, newSamples = calibrateObservers(samples)
newNodeSkew = computeNodeSkew(samples, newOffsets)
} else {
newOffsets = make(map[string]float64)
newSamples = make(map[string]int)
newNodeSkew = make(txSkewResult)
}
// Swap results under brief write lock.
e.mu.Lock()
// Re-check: another goroutine may have computed while we were working.
if time.Since(e.lastComputed) < e.computeInterval {
e.mu.Unlock()
return
}
e.observerOffsets = newOffsets
e.observerSamples = newSamples
e.nodeSkew = newNodeSkew
e.lastComputed = time.Now()
e.mu.Unlock()
}
// collectSamples extracts skew samples from ADVERT packets in the store.
// Must be called with store.mu held (at least RLock).
func collectSamples(store *PacketStore) []skewSample {
adverts := store.byPayloadType[PayloadADVERT]
if len(adverts) == 0 {
return nil
}
samples := make([]skewSample, 0, len(adverts)*2)
for _, tx := range adverts {
decoded := tx.ParsedDecoded()
if decoded == nil {
continue
}
// Extract advert timestamp from decoded JSON.
advertTS := extractTimestamp(decoded)
if advertTS <= 0 {
continue
}
// Sanity: skip timestamps before year 2020 or after year 2100.
if advertTS < 1577836800 || advertTS > 4102444800 {
continue
}
for _, obs := range tx.Observations {
obsTS := parseISO(obs.Timestamp)
if obsTS <= 0 {
continue
}
samples = append(samples, skewSample{
advertTS: advertTS,
observedTS: obsTS,
observerID: obs.ObserverID,
hash: tx.Hash,
})
}
}
return samples
}
// extractTimestamp gets the Unix timestamp from a decoded ADVERT payload.
func extractTimestamp(decoded map[string]interface{}) int64 {
// Try payload.timestamp first (nested in "payload" key).
if payload, ok := decoded["payload"]; ok {
if pm, ok := payload.(map[string]interface{}); ok {
if ts := jsonNumber(pm, "timestamp"); ts > 0 {
return ts
}
}
}
// Fallback: top-level timestamp.
if ts := jsonNumber(decoded, "timestamp"); ts > 0 {
return ts
}
return 0
}
// jsonNumber extracts an int64 from a JSON-parsed map (handles float64 and json.Number).
func jsonNumber(m map[string]interface{}, key string) int64 {
v, ok := m[key]
if !ok || v == nil {
return 0
}
switch n := v.(type) {
case float64:
return int64(n)
case int64:
return n
case int:
return int64(n)
}
return 0
}
// parseISO parses an ISO 8601 timestamp string to Unix seconds.
func parseISO(s string) int64 {
if s == "" {
return 0
}
t, err := time.Parse(time.RFC3339, s)
if err != nil {
// Try with fractional seconds.
t, err = time.Parse("2006-01-02T15:04:05.999999999Z07:00", s)
if err != nil {
return 0
}
}
return t.Unix()
}
// ── Phase 2: Observer Calibration ──────────────────────────────────────────────
// calibrateObservers computes each observer's clock offset using multi-observer
// packets. Returns offset map and sample count map.
func calibrateObservers(samples []skewSample) (map[string]float64, map[string]int) {
// Group observations by packet hash.
byHash := make(map[string][]skewSample)
for _, s := range samples {
byHash[s.hash] = append(byHash[s.hash], s)
}
// For each multi-observer packet, compute per-observer deviation from median.
deviations := make(map[string][]float64) // observerID → list of deviations
for _, group := range byHash {
if len(group) < 2 {
continue // single-observer packet, can't calibrate
}
// Compute median observation timestamp for this packet.
obsTimes := make([]float64, len(group))
for i, s := range group {
obsTimes[i] = float64(s.observedTS)
}
medianObs := median(obsTimes)
for _, s := range group {
dev := float64(s.observedTS) - medianObs
deviations[s.observerID] = append(deviations[s.observerID], dev)
}
}
// Each observer's offset = median of its deviations.
offsets := make(map[string]float64, len(deviations))
counts := make(map[string]int, len(deviations))
for obsID, devs := range deviations {
offsets[obsID] = median(devs)
counts[obsID] = len(devs)
}
return offsets, counts
}
// ── Phase 3: Per-Node Skew ─────────────────────────────────────────────────────
// computeNodeSkew calculates corrected skew statistics for each node.
func computeNodeSkew(samples []skewSample, obsOffsets map[string]float64) txSkewResult {
// Compute corrected skew per sample, grouped by hash (each hash = one
// node's advert transmission). The caller maps hash → pubkey via byNode.
type correctedSample struct {
skew float64
observedTS int64
calibrated bool
}
byHash := make(map[string][]correctedSample)
hashAdvertTS := make(map[string]int64)
for _, s := range samples {
obsOffset, hasCal := obsOffsets[s.observerID]
rawSkew := float64(s.advertTS - s.observedTS)
corrected := rawSkew
if hasCal {
// Observer offset = obs_ts - median(all_obs_ts). If observer is ahead,
// its obs_ts is inflated, making raw_skew too low. Add offset to correct.
corrected = rawSkew + obsOffset
}
byHash[s.hash] = append(byHash[s.hash], correctedSample{
skew: corrected,
observedTS: s.observedTS,
calibrated: hasCal,
})
hashAdvertTS[s.hash] = s.advertTS
}
// Each hash represents one advert from one node. Compute median corrected
// skew per hash (across multiple observers).
result := make(map[string]*NodeClockSkew) // keyed by hash for now
for hash, cs := range byHash {
skews := make([]float64, len(cs))
for i, c := range cs {
skews[i] = c.skew
}
medSkew := median(skews)
meanSkew := mean(skews)
// Find latest observation.
var latestObsTS int64
var anyCal bool
for _, c := range cs {
if c.observedTS > latestObsTS {
latestObsTS = c.observedTS
}
if c.calibrated {
anyCal = true
}
}
absMedian := math.Abs(medSkew)
result[hash] = &NodeClockSkew{
MeanSkewSec: round(meanSkew, 1),
MedianSkewSec: round(medSkew, 1),
LastSkewSec: round(cs[len(cs)-1].skew, 1),
Severity: classifySkew(absMedian),
SampleCount: len(cs),
Calibrated: anyCal,
LastAdvertTS: hashAdvertTS[hash],
LastObservedTS: latestObsTS,
}
}
return result
}
// ── Integration with PacketStore ───────────────────────────────────────────────
// GetNodeClockSkew returns the clock skew data for a specific node (acquires RLock).
func (s *PacketStore) GetNodeClockSkew(pubkey string) *NodeClockSkew {
s.mu.RLock()
defer s.mu.RUnlock()
return s.getNodeClockSkewLocked(pubkey)
}
// getNodeClockSkewLocked returns clock skew for a node.
// Must be called with s.mu held (at least RLock).
func (s *PacketStore) getNodeClockSkewLocked(pubkey string) *NodeClockSkew {
s.clockSkew.Recompute(s)
txs := s.byNode[pubkey]
if len(txs) == 0 {
return nil
}
s.clockSkew.mu.RLock()
defer s.clockSkew.mu.RUnlock()
var allSkews []float64
var lastSkew float64
var lastObsTS, lastAdvTS int64
var totalSamples int
var anyCal bool
var tsSkews []tsSkewPair
for _, tx := range txs {
if tx.PayloadType == nil || *tx.PayloadType != PayloadADVERT {
continue
}
cs, ok := s.clockSkew.nodeSkew[tx.Hash]
if !ok {
continue
}
allSkews = append(allSkews, cs.MedianSkewSec)
totalSamples += cs.SampleCount
if cs.Calibrated {
anyCal = true
}
if cs.LastObservedTS > lastObsTS {
lastObsTS = cs.LastObservedTS
lastSkew = cs.LastSkewSec
lastAdvTS = cs.LastAdvertTS
}
tsSkews = append(tsSkews, tsSkewPair{ts: cs.LastObservedTS, skew: cs.MedianSkewSec})
}
if len(allSkews) == 0 {
return nil
}
medSkew := median(allSkews)
meanSkew := mean(allSkews)
absMedian := math.Abs(medSkew)
severity := classifySkew(absMedian)
// For no_clock nodes (uninitialized RTC), skip drift — data is meaningless.
var drift float64
if severity != SkewNoClock && len(tsSkews) >= minDriftSamples {
drift = computeDrift(tsSkews)
// Cap physically impossible drift rates.
if math.Abs(drift) > maxReasonableDriftPerDay {
drift = 0
}
}
// Build sparkline samples from tsSkews (sorted by time).
sort.Slice(tsSkews, func(i, j int) bool { return tsSkews[i].ts < tsSkews[j].ts })
samples := make([]SkewSample, len(tsSkews))
for i, p := range tsSkews {
samples[i] = SkewSample{Timestamp: p.ts, SkewSec: round(p.skew, 1)}
}
return &NodeClockSkew{
Pubkey: pubkey,
MeanSkewSec: round(meanSkew, 1),
MedianSkewSec: round(medSkew, 1),
LastSkewSec: round(lastSkew, 1),
DriftPerDaySec: round(drift, 2),
Severity: severity,
SampleCount: totalSamples,
Calibrated: anyCal,
LastAdvertTS: lastAdvTS,
LastObservedTS: lastObsTS,
Samples: samples,
}
}
// GetFleetClockSkew returns clock skew data for all nodes that have skew data.
// Must NOT be called with s.mu held.
func (s *PacketStore) GetFleetClockSkew() []*NodeClockSkew {
s.mu.RLock()
defer s.mu.RUnlock()
// Build name/role lookup from DB cache (requires s.mu held).
allNodes, _ := s.getCachedNodesAndPM()
nameMap := make(map[string]nodeInfo, len(allNodes))
for _, ni := range allNodes {
nameMap[ni.PublicKey] = ni
}
var results []*NodeClockSkew
for pubkey := range s.byNode {
cs := s.getNodeClockSkewLocked(pubkey)
if cs == nil {
continue
}
// Enrich with node name/role.
if ni, ok := nameMap[pubkey]; ok {
cs.NodeName = ni.Name
cs.NodeRole = ni.Role
}
// Omit samples in fleet response (too much data).
cs.Samples = nil
results = append(results, cs)
}
return results
}
// GetObserverCalibrations returns the current observer clock offsets.
func (s *PacketStore) GetObserverCalibrations() []ObserverCalibration {
s.mu.RLock()
defer s.mu.RUnlock()
s.clockSkew.Recompute(s)
s.clockSkew.mu.RLock()
defer s.clockSkew.mu.RUnlock()
result := make([]ObserverCalibration, 0, len(s.clockSkew.observerOffsets))
for obsID, offset := range s.clockSkew.observerOffsets {
result = append(result, ObserverCalibration{
ObserverID: obsID,
OffsetSec: round(offset, 1),
Samples: s.clockSkew.observerSamples[obsID],
})
}
// Sort by absolute offset descending.
sort.Slice(result, func(i, j int) bool {
return math.Abs(result[i].OffsetSec) > math.Abs(result[j].OffsetSec)
})
return result
}
// ── Math Helpers ───────────────────────────────────────────────────────────────
func median(vals []float64) float64 {
if len(vals) == 0 {
return 0
}
sorted := make([]float64, len(vals))
copy(sorted, vals)
sort.Float64s(sorted)
n := len(sorted)
if n%2 == 0 {
return (sorted[n/2-1] + sorted[n/2]) / 2
}
return sorted[n/2]
}
func mean(vals []float64) float64 {
if len(vals) == 0 {
return 0
}
sum := 0.0
for _, v := range vals {
sum += v
}
return sum / float64(len(vals))
}
// tsSkewPair is a (timestamp, skew) pair for drift estimation.
type tsSkewPair struct {
ts int64
skew float64
}
// computeDrift estimates linear drift in seconds per day from time-ordered
// (timestamp, skew) pairs using simple linear regression.
func computeDrift(pairs []tsSkewPair) float64 {
if len(pairs) < 2 {
return 0
}
// Sort by timestamp.
sort.Slice(pairs, func(i, j int) bool {
return pairs[i].ts < pairs[j].ts
})
// Time span too short? Skip.
spanSec := float64(pairs[len(pairs)-1].ts - pairs[0].ts)
if spanSec < 3600 { // need at least 1 hour of data
return 0
}
// Simple linear regression: skew = a + b*t
n := float64(len(pairs))
var sumX, sumY, sumXY, sumX2 float64
for _, p := range pairs {
x := float64(p.ts - pairs[0].ts) // normalize to avoid large numbers
y := p.skew
sumX += x
sumY += y
sumXY += x * y
sumX2 += x * x
}
denom := n*sumX2 - sumX*sumX
if denom == 0 {
return 0
}
slope := (n*sumXY - sumX*sumY) / denom // seconds of drift per second
return slope * 86400 // convert to seconds per day
}
+546
View File
@@ -0,0 +1,546 @@
package main
import (
"fmt"
"math"
"testing"
"time"
)
// ── classifySkew ───────────────────────────────────────────────────────────────
func TestClassifySkew(t *testing.T) {
tests := []struct {
absSkew float64
expected SkewSeverity
}{
{0, SkewOK},
{60, SkewOK}, // 1 min
{299, SkewOK}, // just under 5 min
{300, SkewWarning}, // exactly 5 min
{1800, SkewWarning}, // 30 min
{3599, SkewWarning}, // just under 1 hour
{3600, SkewCritical}, // exactly 1 hour
{86400, SkewCritical}, // 1 day
{2592000 - 1, SkewCritical}, // just under 30 days
{2592000, SkewAbsurd}, // exactly 30 days
{86400 * 365 - 1, SkewAbsurd}, // just under 365 days
{86400 * 365, SkewNoClock}, // exactly 365 days
{86400 * 365 * 10, SkewNoClock}, // 10 years (epoch-0 style)
}
for _, tc := range tests {
got := classifySkew(tc.absSkew)
if got != tc.expected {
t.Errorf("classifySkew(%v) = %v, want %v", tc.absSkew, got, tc.expected)
}
}
}
// ── median ─────────────────────────────────────────────────────────────────────
func TestMedian(t *testing.T) {
tests := []struct {
vals []float64
expected float64
}{
{nil, 0},
{[]float64{}, 0},
{[]float64{5}, 5},
{[]float64{1, 3}, 2},
{[]float64{3, 1, 2}, 2},
{[]float64{4, 1, 3, 2}, 2.5},
{[]float64{-10, 0, 10}, 0},
}
for _, tc := range tests {
got := median(tc.vals)
if got != tc.expected {
t.Errorf("median(%v) = %v, want %v", tc.vals, got, tc.expected)
}
}
}
func TestMean(t *testing.T) {
tests := []struct {
vals []float64
expected float64
}{
{nil, 0},
{[]float64{10}, 10},
{[]float64{2, 4, 6}, 4},
}
for _, tc := range tests {
got := mean(tc.vals)
if got != tc.expected {
t.Errorf("mean(%v) = %v, want %v", tc.vals, got, tc.expected)
}
}
}
// ── parseISO ───────────────────────────────────────────────────────────────────
func TestParseISO(t *testing.T) {
tests := []struct {
input string
expected int64
}{
{"", 0},
{"garbage", 0},
{"2026-04-15T12:00:00Z", 1776254400},
{"2026-04-15T12:00:00+00:00", 1776254400},
}
for _, tc := range tests {
got := parseISO(tc.input)
if got != tc.expected {
t.Errorf("parseISO(%q) = %v, want %v", tc.input, got, tc.expected)
}
}
}
// ── extractTimestamp ────────────────────────────────────────────────────────────
func TestExtractTimestamp(t *testing.T) {
// Nested payload.timestamp
decoded := map[string]interface{}{
"payload": map[string]interface{}{
"timestamp": float64(1776340800),
},
}
got := extractTimestamp(decoded)
if got != 1776340800 {
t.Errorf("extractTimestamp (nested) = %v, want 1776340800", got)
}
// Top-level timestamp
decoded2 := map[string]interface{}{
"timestamp": float64(1776340900),
}
got2 := extractTimestamp(decoded2)
if got2 != 1776340900 {
t.Errorf("extractTimestamp (top-level) = %v, want 1776340900", got2)
}
// No timestamp
decoded3 := map[string]interface{}{"foo": "bar"}
got3 := extractTimestamp(decoded3)
if got3 != 0 {
t.Errorf("extractTimestamp (missing) = %v, want 0", got3)
}
}
// ── calibrateObservers ─────────────────────────────────────────────────────────
func TestCalibrateObservers_SingleObserver(t *testing.T) {
// Single-observer packets can't calibrate — should return empty.
samples := []skewSample{
{advertTS: 1000, observedTS: 1000, observerID: "obs1", hash: "h1"},
{advertTS: 2000, observedTS: 2000, observerID: "obs1", hash: "h2"},
}
offsets, _ := calibrateObservers(samples)
if len(offsets) != 0 {
t.Errorf("expected no offsets for single-observer, got %v", offsets)
}
}
func TestCalibrateObservers_MultiObserver(t *testing.T) {
// Packet h1 seen by 3 observers: obs1 at t=100, obs2 at t=110, obs3 at t=100.
// Median observation = 100. obs1=0, obs2=+10, obs3=0
// Packet h2 seen by 3 observers: obs1 at t=200, obs2 at t=210, obs3 at t=200.
// Median observation = 200. obs1=0, obs2=+10, obs3=0
samples := []skewSample{
{advertTS: 100, observedTS: 100, observerID: "obs1", hash: "h1"},
{advertTS: 100, observedTS: 110, observerID: "obs2", hash: "h1"},
{advertTS: 100, observedTS: 100, observerID: "obs3", hash: "h1"},
{advertTS: 200, observedTS: 200, observerID: "obs1", hash: "h2"},
{advertTS: 200, observedTS: 210, observerID: "obs2", hash: "h2"},
{advertTS: 200, observedTS: 200, observerID: "obs3", hash: "h2"},
}
offsets, _ := calibrateObservers(samples)
if offsets["obs1"] != 0 {
t.Errorf("obs1 offset = %v, want 0", offsets["obs1"])
}
if offsets["obs2"] != 10 {
t.Errorf("obs2 offset = %v, want 10", offsets["obs2"])
}
if offsets["obs3"] != 0 {
t.Errorf("obs3 offset = %v, want 0", offsets["obs3"])
}
}
// ── computeNodeSkew ────────────────────────────────────────────────────────────
func TestComputeNodeSkew_BasicCorrection(t *testing.T) {
// Validates observer offset correction direction.
//
// Setup: node is 60s ahead, obs1 accurate, obs2 is 10s ahead.
// With 2 observers, median obs_ts = 1005.
// obs1 offset = 1000 - 1005 = -5
// obs2 offset = 1010 - 1005 = +5
// Correction: corrected = raw_skew + obsOffset
// obs1: raw=60, corrected = 60 + (-5) = 55
// obs2: raw=50, corrected = 50 + 5 = 55
// Both converge to 55 (not exact 60 because with only 2 observers,
// the median can't fully distinguish which observer is drifted).
samples := []skewSample{
// Same packet seen by accurate obs1 and obs2 (+10s ahead)
{advertTS: 1060, observedTS: 1000, observerID: "obs1", hash: "h1"},
{advertTS: 1060, observedTS: 1010, observerID: "obs2", hash: "h1"},
}
offsets, _ := calibrateObservers(samples)
// median obs = 1005, obs1 offset = -5, obs2 offset = +5
// So the median approach finds obs2 is +5 ahead (relative to median)
// Now compute node skew with those offsets:
nodeSkew := computeNodeSkew(samples, offsets)
cs, ok := nodeSkew["h1"]
if !ok {
t.Fatal("expected skew data for hash h1")
}
// With only 2 observers, median obs_ts = 1005.
// obs1 offset = 1000-1005 = -5, obs2 offset = 1010-1005 = +5
// raw from obs1 = 60, corrected = 60 + (-5) = 55
// raw from obs2 = 50, corrected = 50 + 5 = 55
// median = 55
if cs.MedianSkewSec != 55 {
t.Errorf("median skew = %v, want 55", cs.MedianSkewSec)
}
}
func TestComputeNodeSkew_ThreeObservers(t *testing.T) {
// Node is exactly 60s ahead. obs1 accurate, obs2 accurate, obs3 +30s ahead.
// advertTS = 1060, real time = 1000
samples := []skewSample{
{advertTS: 1060, observedTS: 1000, observerID: "obs1", hash: "h1"},
{advertTS: 1060, observedTS: 1000, observerID: "obs2", hash: "h1"},
{advertTS: 1060, observedTS: 1030, observerID: "obs3", hash: "h1"},
}
offsets, _ := calibrateObservers(samples)
// median obs_ts = 1000. obs1=0, obs2=0, obs3=+30
if offsets["obs3"] != 30 {
t.Errorf("obs3 offset = %v, want 30", offsets["obs3"])
}
nodeSkew := computeNodeSkew(samples, offsets)
cs := nodeSkew["h1"]
if cs == nil {
t.Fatal("expected skew data for h1")
}
// raw from obs1 = 60, corrected = 60 + 0 = 60
// raw from obs2 = 60, corrected = 60 + 0 = 60
// raw from obs3 = 30, corrected = 30 + 30 = 60
// All three converge to 60.
if cs.MedianSkewSec != 60 {
t.Errorf("median skew = %v, want 60 (node is 60s ahead)", cs.MedianSkewSec)
}
}
// ── computeDrift ───────────────────────────────────────────────────────────────
func TestComputeDrift_Stable(t *testing.T) {
// Constant skew = no drift.
pairs := []tsSkewPair{
{ts: 0, skew: 60},
{ts: 7200, skew: 60},
{ts: 14400, skew: 60},
}
drift := computeDrift(pairs)
if drift != 0 {
t.Errorf("drift = %v, want 0 for stable skew", drift)
}
}
func TestComputeDrift_LinearDrift(t *testing.T) {
// 1 second drift per hour = 24 sec/day.
pairs := []tsSkewPair{
{ts: 0, skew: 0},
{ts: 3600, skew: 1},
{ts: 7200, skew: 2},
}
drift := computeDrift(pairs)
expected := 24.0
if math.Abs(drift-expected) > 0.1 {
t.Errorf("drift = %v, want ~%v", drift, expected)
}
}
func TestComputeDrift_TooFewSamples(t *testing.T) {
pairs := []tsSkewPair{{ts: 0, skew: 10}}
if computeDrift(pairs) != 0 {
t.Error("expected 0 drift for single sample")
}
}
func TestComputeDrift_TooShortSpan(t *testing.T) {
// Less than 1 hour apart.
pairs := []tsSkewPair{
{ts: 0, skew: 0},
{ts: 1800, skew: 10},
}
if computeDrift(pairs) != 0 {
t.Error("expected 0 drift for short time span")
}
}
// ── jsonNumber ─────────────────────────────────────────────────────────────────
func TestJsonNumber(t *testing.T) {
m := map[string]interface{}{
"a": float64(42),
"b": int64(99),
"c": "not a number",
"d": nil,
}
if jsonNumber(m, "a") != 42 {
t.Error("float64 case failed")
}
if jsonNumber(m, "b") != 99 {
t.Error("int64 case failed")
}
if jsonNumber(m, "c") != 0 {
t.Error("string case should return 0")
}
if jsonNumber(m, "d") != 0 {
t.Error("nil case should return 0")
}
if jsonNumber(m, "missing") != 0 {
t.Error("missing key should return 0")
}
}
// ── Integration: GetNodeClockSkew via PacketStore ──────────────────────────────
func TestGetNodeClockSkew_Integration(t *testing.T) {
ps := NewPacketStore(nil, nil)
// Simulate two ADVERT transmissions for the same node, seen by 2 observers each.
// Node "AABB" has clock 120s ahead.
pt := 4 // ADVERT
tx1 := &StoreTx{
Hash: "hash1",
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":1700002320}}`, // obs=1700002200, node ahead by 120s
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: "2023-11-14T22:50:00Z"}, // 1700002200
{ObserverID: "obs2", Timestamp: "2023-11-14T22:50:00Z"}, // 1700002200
},
}
tx2 := &StoreTx{
Hash: "hash2",
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":1700005920}}`, // obs=1700005800, node ahead by 120s
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: "2023-11-14T23:50:00Z"}, // 1700005800
{ObserverID: "obs2", Timestamp: "2023-11-14T23:50:00Z"}, // 1700005800
},
}
ps.mu.Lock()
ps.byNode["AABB"] = []*StoreTx{tx1, tx2}
ps.byPayloadType[4] = []*StoreTx{tx1, tx2}
// Force recompute by setting interval to 0.
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("AABB")
if result == nil {
t.Fatal("expected clock skew result for node AABB")
}
if result.Pubkey != "AABB" {
t.Errorf("pubkey = %q, want AABB", result.Pubkey)
}
// Both transmissions show 120s skew, so median should be 120.
if result.MedianSkewSec != 120 {
t.Errorf("median skew = %v, want 120", result.MedianSkewSec)
}
if result.SampleCount < 2 {
t.Errorf("sample count = %v, want >= 2", result.SampleCount)
}
if result.Severity != SkewOK {
t.Errorf("severity = %v, want ok (120s < 5min)", result.Severity)
}
// Drift should be ~0 since skew is constant.
if math.Abs(result.DriftPerDaySec) > 1 {
t.Errorf("drift = %v, want ~0 for constant skew", result.DriftPerDaySec)
}
}
func TestGetNodeClockSkew_NoData(t *testing.T) {
ps := NewPacketStore(nil, nil)
result := ps.GetNodeClockSkew("nonexistent")
if result != nil {
t.Error("expected nil for nonexistent node")
}
}
// ── Sanity check tests (#XXX — clock skew crazy stats) ────────────────────────
func TestGetNodeClockSkew_NoClock_EpochZero(t *testing.T) {
// Node with epoch-0 timestamp produces huge skew → no_clock severity, drift=0.
ps := NewPacketStore(nil, nil)
pt := 4 // ADVERT
// Epoch-ish advert: advertTS near start of 2020, observed in 2023 → |skew| > 365 days
var txs []*StoreTx
baseObs := int64(1700000000) // ~Nov 2023
for i := 0; i < 6; i++ {
obsTS := baseObs + int64(i)*7200
tx := &StoreTx{
Hash: "epoch-h" + string(rune('0'+i)),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":1577836800}}`, // Jan 1 2020 — valid but way off
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["EPOCH"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("EPOCH")
if result == nil {
t.Fatal("expected clock skew result for epoch-0 node")
}
if result.Severity != SkewNoClock {
t.Errorf("severity = %v, want no_clock", result.Severity)
}
if result.DriftPerDaySec != 0 {
t.Errorf("drift = %v, want 0 for no_clock node", result.DriftPerDaySec)
}
}
func TestGetNodeClockSkew_TooFewSamplesForDrift(t *testing.T) {
// Node with only 2 advert samples → drift should not be computed.
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 2; i++ {
obsTS := baseObs + int64(i)*7200
advTS := obsTS + 120 // 120s ahead
tx := &StoreTx{
Hash: "few-h" + string(rune('0'+i)),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["FEWSAMP"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("FEWSAMP")
if result == nil {
t.Fatal("expected clock skew result")
}
if result.DriftPerDaySec != 0 {
t.Errorf("drift = %v, want 0 for 2-sample node (minimum is %d)", result.DriftPerDaySec, minDriftSamples)
}
}
func TestGetNodeClockSkew_AbsurdDriftCapped(t *testing.T) {
// Node with wildly varying skew producing |drift| > 86400 s/day → drift capped to 0.
ps := NewPacketStore(nil, nil)
pt := 4
// Create 6 samples with extreme skew variation to produce absurd drift.
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 6; i++ {
obsTS := baseObs + int64(i)*3600
// Alternate between huge positive and negative skew offsets
skewOffset := int64(50000 * (1 - 2*(i%2))) // +50000 or -50000
advTS := obsTS + skewOffset
tx := &StoreTx{
Hash: "wild-h" + string(rune('0'+i)),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["WILD"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("WILD")
if result == nil {
t.Fatal("expected clock skew result")
}
if math.Abs(result.DriftPerDaySec) > maxReasonableDriftPerDay {
t.Errorf("drift = %v, should be capped (|drift| > %v)", result.DriftPerDaySec, maxReasonableDriftPerDay)
}
}
func TestGetNodeClockSkew_NormalNodeWithDrift(t *testing.T) {
// Normal node with 6 samples and consistent linear drift → drift computed correctly.
ps := NewPacketStore(nil, nil)
pt := 4
baseObs := int64(1700000000)
var txs []*StoreTx
for i := 0; i < 6; i++ {
obsTS := baseObs + int64(i)*7200 // every 2 hours
// Drift: 1 sec/hour = 24 sec/day
advTS := obsTS + 120 + int64(i) // skew grows by 1s per sample (2h apart)
tx := &StoreTx{
Hash: "norm-h" + string(rune('0'+i)),
PayloadType: &pt,
DecodedJSON: `{"payload":{"timestamp":` + formatInt64(advTS) + `}}`,
Observations: []*StoreObs{
{ObserverID: "obs1", Timestamp: time.Unix(obsTS, 0).UTC().Format(time.RFC3339)},
},
}
txs = append(txs, tx)
}
ps.mu.Lock()
ps.byNode["NORMAL"] = txs
for _, tx := range txs {
ps.byPayloadType[4] = append(ps.byPayloadType[4], tx)
}
ps.clockSkew.computeInterval = 0
ps.mu.Unlock()
result := ps.GetNodeClockSkew("NORMAL")
if result == nil {
t.Fatal("expected clock skew result")
}
if result.Severity != SkewOK {
t.Errorf("severity = %v, want ok", result.Severity)
}
// 1s per 7200s = 12 s/day
if result.DriftPerDaySec == 0 {
t.Error("expected non-zero drift for linearly drifting node")
}
if math.Abs(result.DriftPerDaySec) > maxReasonableDriftPerDay {
t.Errorf("drift = %v, should be reasonable", result.DriftPerDaySec)
}
}
// formatInt64 is a test helper to format int64 as string for JSON embedding.
func formatInt64(n int64) string {
return fmt.Sprintf("%d", n)
}
+131
View File
@@ -0,0 +1,131 @@
package main
import (
"testing"
"time"
)
// TestCollisionDetailsIncludeNodePairs verifies that collision details contain
// the correct prefix and matching node pairs (#757).
func TestCollisionDetailsIncludeNodePairs(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
// Insert two repeater nodes with the same 3-byte prefix "AABB11"
db.conn.Exec(`INSERT INTO nodes (public_key, name, role) VALUES ('aabb11ccdd001122', 'Node Alpha', 'repeater')`)
db.conn.Exec(`INSERT INTO nodes (public_key, name, role) VALUES ('aabb11eeff334455', 'Node Beta', 'repeater')`)
// Add advert transmissions with hash_size=3 path bytes (0x80 = bits 10 → size 3)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('0180aabb11ccdd', 'col_hash_01', ?, 1, 4, '{"pubKey":"aabb11ccdd001122","name":"Node Alpha","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 10.0, -91, '["aabb11"]', ?)`, recentEpoch)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('0180aabb11eeff', 'col_hash_02', ?, 1, 4, '{"pubKey":"aabb11eeff334455","name":"Node Beta","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (2, 1, 9.0, -93, '["aabb11"]', ?)`, recentEpoch)
store := NewPacketStore(db, nil)
store.Load()
result := store.GetAnalyticsHashCollisions("")
bySize, ok := result["by_size"].(map[string]interface{})
if !ok {
t.Fatal("expected by_size map")
}
size3, ok := bySize["3"].(map[string]interface{})
if !ok {
t.Fatal("expected by_size[3] map")
}
collisions, ok := size3["collisions"].([]collisionEntry)
if !ok {
t.Fatalf("expected collisions as []collisionEntry, got %T", size3["collisions"])
}
// Find our collision
var found *collisionEntry
for i := range collisions {
if collisions[i].Prefix == "AABB11" {
found = &collisions[i]
break
}
}
if found == nil {
t.Fatal("expected collision with prefix AABB11")
}
if found.Appearances != 2 {
t.Errorf("expected 2 appearances, got %d", found.Appearances)
}
if len(found.Nodes) != 2 {
t.Fatalf("expected 2 nodes in collision, got %d", len(found.Nodes))
}
// Verify node pairs
pubkeys := map[string]bool{}
names := map[string]bool{}
for _, n := range found.Nodes {
pubkeys[n.PublicKey] = true
names[n.Name] = true
}
if !pubkeys["aabb11ccdd001122"] {
t.Error("expected node aabb11ccdd001122 in collision")
}
if !pubkeys["aabb11eeff334455"] {
t.Error("expected node aabb11eeff334455 in collision")
}
if !names["Node Alpha"] {
t.Error("expected Node Alpha in collision")
}
if !names["Node Beta"] {
t.Error("expected Node Beta in collision")
}
}
// TestCollisionDetailsEmptyWhenNoCollisions verifies that collision details are
// empty when there are no collisions (#757).
func TestCollisionDetailsEmptyWhenNoCollisions(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
// Insert one repeater node with 3-byte hash
db.conn.Exec(`INSERT INTO nodes (public_key, name, role) VALUES ('aabb11ccdd001122', 'Solo Node', 'repeater')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('0180aabb11ccdd', 'solo_hash_01', ?, 1, 4, '{"pubKey":"aabb11ccdd001122","name":"Solo Node","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 10.0, -91, '["aabb11"]', ?)`, recentEpoch)
store := NewPacketStore(db, nil)
store.Load()
result := store.GetAnalyticsHashCollisions("")
bySize, ok := result["by_size"].(map[string]interface{})
if !ok {
t.Fatal("expected by_size map")
}
size3, ok := bySize["3"].(map[string]interface{})
if !ok {
t.Fatal("expected by_size[3] map")
}
collisions, ok := size3["collisions"].([]collisionEntry)
if !ok {
t.Fatalf("expected collisions as []collisionEntry, got %T", size3["collisions"])
}
if len(collisions) != 0 {
t.Errorf("expected 0 collisions, got %d", len(collisions))
}
}
+184 -1
View File
@@ -41,7 +41,7 @@ func setupTestDBv2(t *testing.T) *DB {
id INTEGER PRIMARY KEY AUTOINCREMENT, raw_hex TEXT NOT NULL,
hash TEXT NOT NULL UNIQUE, first_seen TEXT NOT NULL,
route_type INTEGER, payload_type INTEGER, payload_version INTEGER,
decoded_json TEXT, created_at TEXT DEFAULT (datetime('now'))
decoded_json TEXT, channel_hash TEXT DEFAULT NULL, created_at TEXT DEFAULT (datetime('now'))
);
CREATE TABLE observations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -3217,6 +3217,189 @@ func TestGetNodeHashSizeInfoEdgeCases(t *testing.T) {
}
}
// TestHashSizeTransportRoutePathByteOffset verifies that transport routes (0, 3)
// read the path byte from offset 5 (after 4 transport code bytes), not offset 1.
// Regression test for #744 / #722.
func TestHashSizeTransportRoutePathByteOffset(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
db.conn.Exec(`INSERT INTO observers (id, name, iata, last_seen, first_seen, packet_count)
VALUES ('obs1', 'Obs', 'SJC', ?, '2026-01-01T00:00:00Z', 10)`, recent)
// Route type 0 (TRANSPORT_FLOOD): header=0x04 (payload_type=1, route_type=0)
// 4 transport bytes + path byte at offset 5.
// Path byte 0x80 → hash_size bits = 10 → size 3
// If bug is present, code reads byte 1 (0xAA) → hash_size bits = 10 → size 3 (coincidence)
// Use path byte 0x40 (hash_size=2) and transport byte 0x01 at offset 1 (hash_size=1 if misread)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('100102030440aabb', 'tf_offset', ?, 0, 4, '{"pubKey":"aaaa000000000001","name":"TF-Node","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 10.0, -90, '[]', ?)`, recentEpoch)
// Route type 3 (TRANSPORT_DIRECT): header=0x13 (payload_type=4, route_type=3)
// 4 transport bytes + path byte at offset 5.
// Path byte 0xC1 → hash_size bits = 11 → size 4, hop_count = 1 (not zero-hop)
// Byte 1 = 0x05 → hash_size bits = 00 → size 1 if misread
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('1305060708C1bbcc', 'td_offset', ?, 3, 4, '{"pubKey":"aaaa000000000002","name":"TD-Node","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (2, 1, 10.0, -90, '[]', ?)`, recentEpoch)
// Route type 1 (FLOOD): header=0x11 (payload_type=4, route_type=1)
// Path byte at offset 1. Path byte 0x80 → hash_size bits = 10 → size 3
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('1180aabbccdd', 'flood_offset', ?, 1, 4, '{"pubKey":"aaaa000000000003","name":"Flood-Node","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (3, 1, 10.0, -90, '[]', ?)`, recentEpoch)
store := NewPacketStore(db, nil)
store.Load()
info := store.GetNodeHashSizeInfo()
// Transport flood node: path byte 0x40 → hash_size = 2
if ni, ok := info["aaaa000000000001"]; !ok {
t.Error("transport flood node missing from hash size info")
} else if ni.HashSize != 2 {
t.Errorf("transport flood node: want HashSize=2 (from path byte at offset 5), got %d", ni.HashSize)
}
// Transport direct node: path byte 0xC1 → hash_size = 4
if ni, ok := info["aaaa000000000002"]; !ok {
t.Error("transport direct node missing from hash size info")
} else if ni.HashSize != 4 {
t.Errorf("transport direct node: want HashSize=4 (from path byte at offset 5), got %d", ni.HashSize)
}
// Regular flood node: path byte 0x80 → hash_size = 3
if ni, ok := info["aaaa000000000003"]; !ok {
t.Error("regular flood node missing from hash size info")
} else if ni.HashSize != 3 {
t.Errorf("regular flood node: want HashSize=3 (from path byte at offset 1), got %d", ni.HashSize)
}
}
// TestHashSizeTransportDirectZeroHopSkipped verifies that RouteTransportDirect
// zero-hop adverts are skipped (same as RouteDirect). Regression test for #744.
func TestHashSizeTransportDirectZeroHopSkipped(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
db.conn.Exec(`INSERT INTO observers (id, name, iata, last_seen, first_seen, packet_count)
VALUES ('obs1', 'Obs', 'SJC', ?, '2026-01-01T00:00:00Z', 10)`, recent)
// RouteDirect (2) zero-hop: path byte 0x40 → hop_count=0, hash_size bits=01
// Should be skipped (existing behavior)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('1240aabbccdd', 'direct_zh', ?, 2, 4, '{"pubKey":"bbbb000000000001","name":"Direct-ZH","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 10.0, -90, '[]', ?)`, recentEpoch)
// RouteTransportDirect (3) zero-hop: 4 transport bytes + path byte 0x40 → hop_count=0
// Should ALSO be skipped (this was the missing case)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('130102030440aabb', 'tdirect_zh', ?, 3, 4, '{"pubKey":"bbbb000000000002","name":"TDirect-ZH","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (2, 1, 10.0, -90, '[]', ?)`, recentEpoch)
// RouteDirect (2) non-zero-hop: path byte 0x41 → hop_count=1
// Should NOT be skipped
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('1241aabbccdd', 'direct_1h', ?, 2, 4, '{"pubKey":"bbbb000000000003","name":"Direct-1H","type":"ADVERT"}')`, recent)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (3, 1, 10.0, -90, '[]', ?)`, recentEpoch)
store := NewPacketStore(db, nil)
store.Load()
info := store.GetNodeHashSizeInfo()
// RouteDirect zero-hop should be absent
if _, ok := info["bbbb000000000001"]; ok {
t.Error("RouteDirect zero-hop advert should be skipped")
}
// RouteTransportDirect zero-hop should also be absent
if _, ok := info["bbbb000000000002"]; ok {
t.Error("RouteTransportDirect zero-hop advert should be skipped")
}
// RouteDirect non-zero-hop should be present with hash_size=2
if ni, ok := info["bbbb000000000003"]; !ok {
t.Error("RouteDirect non-zero-hop should be in hash size info")
} else if ni.HashSize != 2 {
t.Errorf("RouteDirect non-zero-hop: want HashSize=2, got %d", ni.HashSize)
}
}
// TestAnalyticsHashSizesZeroHopSkip verifies that computeAnalyticsHashSizes
// does not overwrite a node's hash_size with a zero-hop advert's unreliable value.
// Regression test for #744.
func TestAnalyticsHashSizesZeroHopSkip(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
db.conn.Exec(`INSERT INTO observers (id, name, iata, last_seen, first_seen, packet_count)
VALUES ('obs1', 'Obs', 'SJC', ?, '2026-01-01T00:00:00Z', 10)`, recent)
pk := "cccc000000000001"
db.conn.Exec(`INSERT INTO nodes (public_key, name, role) VALUES (?, 'ZH-Analytics', 'repeater')`, pk)
decoded := `{"pubKey":"` + pk + `","name":"ZH-Analytics","type":"ADVERT"}`
// First: a flood advert with hashSize=2 (reliable, multi-hop)
// header 0x11 = route_type 1 (flood), payload_type 4
// pathByte 0x41 = hashSize bits 01 → size 2, hop_count 1
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('1141aabbccdd', 'az_flood', ?, 1, 4, ?)`, recent, decoded)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 10.0, -90, '["aabb"]', ?)`, recentEpoch)
// Second: a direct zero-hop advert with pathByte=0x00 → would give hashSize=1
// header 0x12 = route_type 2 (direct), payload_type 4
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('1200aabbccdd', 'az_direct', ?, 2, 4, ?)`, recent, decoded)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (2, 1, 10.0, -90, '[]', ?)`, recentEpoch)
store := NewPacketStore(db, nil)
store.Load()
result := store.GetAnalyticsHashSizes("")
// The node should appear in multiByteNodes (hashSize=2 from the flood advert)
// If the zero-hop bug is present, hashSize would be 1 and the node would NOT
// appear in multiByteNodes.
multiByteNodes, ok := result["multiByteNodes"].([]map[string]interface{})
if !ok {
t.Fatal("expected multiByteNodes slice in analytics hash sizes")
}
found := false
for _, n := range multiByteNodes {
if n["pubkey"] == pk {
found = true
if hs, ok := n["hashSize"].(int); ok && hs != 2 {
t.Errorf("expected hashSize=2 from flood advert, got %d", hs)
}
}
}
if !found {
t.Error("node should appear in multiByteNodes with hashSize=2; zero-hop advert should not overwrite to 1")
}
}
func TestHandleResolveHopsEdgeCases(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
+218 -74
View File
@@ -8,6 +8,7 @@ import (
"math"
"os"
"strings"
"sync"
"time"
_ "modernc.org/sqlite"
@@ -19,6 +20,12 @@ type DB struct {
path string // filesystem path to the database file
isV3 bool // v3 schema: observer_idx in observations (vs observer_id in v2)
hasResolvedPath bool // observations table has resolved_path column
// Channel list cache (60s TTL) — avoids repeated GROUP BY scans (#762)
channelsCacheMu sync.Mutex
channelsCacheKey string
channelsCacheRes []map[string]interface{}
channelsCacheExp time.Time
}
// OpenDB opens a read-only SQLite connection with WAL mode.
@@ -1153,69 +1160,219 @@ func (db *DB) GetTraces(hash string) ([]map[string]interface{}, error) {
// Queries transmissions directly (not a VIEW) to avoid observation-level
// duplicates that could cause stale lastMessage when an older message has
// a later re-observation timestamp.
func (db *DB) GetChannels() ([]map[string]interface{}, error) {
rows, err := db.conn.Query(`SELECT decoded_json, first_seen FROM transmissions WHERE payload_type = 5 ORDER BY first_seen ASC`)
func (db *DB) GetChannels(region ...string) ([]map[string]interface{}, error) {
regionParam := ""
if len(region) > 0 {
regionParam = region[0]
}
// Check cache (60s TTL)
db.channelsCacheMu.Lock()
if db.channelsCacheRes != nil && db.channelsCacheKey == regionParam && time.Now().Before(db.channelsCacheExp) {
res := db.channelsCacheRes
db.channelsCacheMu.Unlock()
return res, nil
}
db.channelsCacheMu.Unlock()
regionCodes := normalizeRegionCodes(regionParam)
var querySQL string
args := make([]interface{}, 0, len(regionCodes))
if len(regionCodes) > 0 {
placeholders := make([]string, len(regionCodes))
for i, code := range regionCodes {
placeholders[i] = "?"
args = append(args, code)
}
regionPlaceholder := strings.Join(placeholders, ",")
if db.isV3 {
querySQL = fmt.Sprintf(`SELECT t.channel_hash,
COUNT(*) AS msg_count,
MAX(t.first_seen) AS last_activity,
(SELECT t2.decoded_json FROM transmissions t2
WHERE t2.channel_hash = t.channel_hash AND t2.payload_type = 5
ORDER BY t2.first_seen DESC LIMIT 1) AS sample_json
FROM transmissions t
JOIN observations o ON o.transmission_id = t.id
LEFT JOIN observers obs ON obs.rowid = o.observer_idx
WHERE t.payload_type = 5
AND t.channel_hash IS NOT NULL
AND t.channel_hash NOT LIKE 'enc_%%'
AND obs.rowid IS NOT NULL AND UPPER(TRIM(obs.iata)) IN (%s)
GROUP BY t.channel_hash
ORDER BY last_activity DESC`, regionPlaceholder)
} else {
querySQL = fmt.Sprintf(`SELECT t.channel_hash,
COUNT(*) AS msg_count,
MAX(t.first_seen) AS last_activity,
(SELECT t2.decoded_json FROM transmissions t2
WHERE t2.channel_hash = t.channel_hash AND t2.payload_type = 5
ORDER BY t2.first_seen DESC LIMIT 1) AS sample_json
FROM transmissions t
JOIN observations o ON o.transmission_id = t.id
WHERE t.payload_type = 5
AND t.channel_hash IS NOT NULL
AND t.channel_hash NOT LIKE 'enc_%%'
AND EXISTS (
SELECT 1 FROM observers obs
WHERE obs.id = o.observer_id
AND UPPER(TRIM(obs.iata)) IN (%s)
)
GROUP BY t.channel_hash
ORDER BY last_activity DESC`, regionPlaceholder)
}
} else {
querySQL = `SELECT channel_hash,
COUNT(*) AS msg_count,
MAX(first_seen) AS last_activity,
(SELECT t2.decoded_json FROM transmissions t2
WHERE t2.channel_hash = t.channel_hash AND t2.payload_type = 5
ORDER BY t2.first_seen DESC LIMIT 1) AS sample_json
FROM transmissions t
WHERE payload_type = 5
AND channel_hash IS NOT NULL
AND channel_hash NOT LIKE 'enc_%%'
GROUP BY channel_hash
ORDER BY last_activity DESC`
}
rows, err := db.conn.Query(querySQL, args...)
if err != nil {
return nil, err
}
defer rows.Close()
channelMap := map[string]map[string]interface{}{}
channels := make([]map[string]interface{}, 0)
for rows.Next() {
var dj, fs sql.NullString
rows.Scan(&dj, &fs)
if !dj.Valid {
var chHash, lastActivity, sampleJSON sql.NullString
var msgCount int
if err := rows.Scan(&chHash, &msgCount, &lastActivity, &sampleJSON); err != nil {
continue
}
var decoded map[string]interface{}
if json.Unmarshal([]byte(dj.String), &decoded) != nil {
continue
}
dtype, _ := decoded["type"].(string)
if dtype != "CHAN" {
continue
}
// Filter out garbage-decrypted channel names/messages (pre-#197 data still in DB)
chanStr, _ := decoded["channel"].(string)
textStr, _ := decoded["text"].(string)
if hasGarbageChars(chanStr) || hasGarbageChars(textStr) {
continue
}
channelName, _ := decoded["channel"].(string)
channelName := nullStr(chHash)
if channelName == "" {
channelName = "unknown"
continue
}
key := channelName
ch, exists := channelMap[key]
if !exists {
ch = map[string]interface{}{
"hash": key, "name": channelName,
"lastMessage": nil, "lastSender": nil,
"messageCount": 0, "lastActivity": nullStr(fs),
}
channelMap[key] = ch
}
ch["messageCount"] = ch["messageCount"].(int) + 1
if fs.Valid {
ch["lastActivity"] = fs.String
}
if text, ok := decoded["text"].(string); ok && text != "" {
idx := strings.Index(text, ": ")
if idx > 0 {
ch["lastMessage"] = text[idx+2:]
} else {
ch["lastMessage"] = text
}
if sender, ok := decoded["sender"].(string); ok {
ch["lastSender"] = sender
var lastMessage, lastSender interface{}
if sampleJSON.Valid {
var decoded map[string]interface{}
if json.Unmarshal([]byte(sampleJSON.String), &decoded) == nil {
if text, ok := decoded["text"].(string); ok && text != "" {
idx := strings.Index(text, ": ")
if idx > 0 {
lastMessage = text[idx+2:]
} else {
lastMessage = text
}
if sender, ok := decoded["sender"].(string); ok {
lastSender = sender
}
}
}
}
channels = append(channels, map[string]interface{}{
"hash": channelName, "name": channelName,
"lastMessage": lastMessage, "lastSender": lastSender,
"messageCount": msgCount, "lastActivity": nullStr(lastActivity),
})
}
channels := make([]map[string]interface{}, 0, len(channelMap))
for _, ch := range channelMap {
channels = append(channels, ch)
// Store in cache (60s TTL)
db.channelsCacheMu.Lock()
db.channelsCacheRes = channels
db.channelsCacheKey = regionParam
db.channelsCacheExp = time.Now().Add(60 * time.Second)
db.channelsCacheMu.Unlock()
return channels, nil
}
// GetEncryptedChannels returns channels where all messages are undecryptable (no key).
// Uses channel_hash column (prefixed with 'enc_') for fast grouped queries.
func (db *DB) GetEncryptedChannels(region ...string) ([]map[string]interface{}, error) {
regionParam := ""
if len(region) > 0 {
regionParam = region[0]
}
regionCodes := normalizeRegionCodes(regionParam)
var querySQL string
args := make([]interface{}, 0, len(regionCodes))
if len(regionCodes) > 0 {
placeholders := make([]string, len(regionCodes))
for i, code := range regionCodes {
placeholders[i] = "?"
args = append(args, code)
}
regionPlaceholder := strings.Join(placeholders, ",")
if db.isV3 {
querySQL = fmt.Sprintf(`SELECT t.channel_hash,
COUNT(*) AS msg_count,
MAX(t.first_seen) AS last_activity
FROM transmissions t
JOIN observations o ON o.transmission_id = t.id
LEFT JOIN observers obs ON obs.rowid = o.observer_idx
WHERE t.payload_type = 5
AND t.channel_hash LIKE 'enc_%%'
AND obs.rowid IS NOT NULL AND UPPER(TRIM(obs.iata)) IN (%s)
GROUP BY t.channel_hash
ORDER BY last_activity DESC`, regionPlaceholder)
} else {
querySQL = fmt.Sprintf(`SELECT t.channel_hash,
COUNT(*) AS msg_count,
MAX(t.first_seen) AS last_activity
FROM transmissions t
JOIN observations o ON o.transmission_id = t.id
WHERE t.payload_type = 5
AND t.channel_hash LIKE 'enc_%%'
AND EXISTS (
SELECT 1 FROM observers obs
WHERE obs.id = o.observer_id
AND UPPER(TRIM(obs.iata)) IN (%s)
)
GROUP BY t.channel_hash
ORDER BY last_activity DESC`, regionPlaceholder)
}
} else {
querySQL = `SELECT channel_hash,
COUNT(*) AS msg_count,
MAX(first_seen) AS last_activity
FROM transmissions
WHERE payload_type = 5
AND channel_hash LIKE 'enc_%%'
GROUP BY channel_hash
ORDER BY last_activity DESC`
}
rows, err := db.conn.Query(querySQL, args...)
if err != nil {
return nil, err
}
defer rows.Close()
channels := make([]map[string]interface{}, 0)
for rows.Next() {
var chHash, lastActivity sql.NullString
var msgCount int
if err := rows.Scan(&chHash, &msgCount, &lastActivity); err != nil {
continue
}
fullHash := nullStrVal(chHash) // e.g. "enc_3A"
hexPart := strings.TrimPrefix(fullHash, "enc_")
channels = append(channels, map[string]interface{}{
"hash": fullHash,
"name": "Encrypted (0x" + hexPart + ")",
"lastMessage": nil,
"lastSender": nil,
"messageCount": msgCount,
"lastActivity": nullStr(lastActivity),
"encrypted": true,
})
}
return channels, nil
}
@@ -1244,15 +1401,16 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int, region .
regionPlaceholders = strings.Join(placeholders, ",")
}
// Fetch messages with channel_hash filter (pagination applied in Go after dedup)
var querySQL string
args := make([]interface{}, 0, len(regionArgs))
args := []interface{}{channelHash}
if db.isV3 {
querySQL = `SELECT o.id, t.hash, t.decoded_json, t.first_seen,
obs.id, obs.name, o.snr, o.path_json
FROM observations o
JOIN transmissions t ON t.id = o.transmission_id
LEFT JOIN observers obs ON obs.rowid = o.observer_idx
WHERE t.payload_type = 5`
WHERE t.channel_hash = ? AND t.payload_type = 5`
if len(regionCodes) > 0 {
querySQL += fmt.Sprintf(" AND obs.rowid IS NOT NULL AND UPPER(TRIM(obs.iata)) IN (%s)", regionPlaceholders)
args = append(args, regionArgs...)
@@ -1264,14 +1422,11 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int, region .
o.observer_id, o.observer_name, o.snr, o.path_json
FROM observations o
JOIN transmissions t ON t.id = o.transmission_id
WHERE t.payload_type = 5`
WHERE t.channel_hash = ? AND t.payload_type = 5`
if len(regionCodes) > 0 {
querySQL += fmt.Sprintf(` AND EXISTS (
SELECT 1
FROM observers obs
WHERE obs.id = o.observer_id
AND UPPER(TRIM(obs.iata)) IN (%s)
)`, regionPlaceholders)
SELECT 1 FROM observers obs WHERE obs.id = o.observer_id
AND UPPER(TRIM(obs.iata)) IN (%s))`, regionPlaceholders)
args = append(args, regionArgs...)
}
querySQL += `
@@ -1303,17 +1458,6 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int, region .
if json.Unmarshal([]byte(dj.String), &decoded) != nil {
continue
}
dtype, _ := decoded["type"].(string)
if dtype != "CHAN" {
continue
}
ch, _ := decoded["channel"].(string)
if ch == "" {
ch = "unknown"
}
if ch != channelHash {
continue
}
text, _ := decoded["text"].(string)
sender, _ := decoded["sender"].(string)
@@ -1373,18 +1517,18 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int, region .
}
}
total := len(msgOrder)
// Return latest messages (tail)
start := total - limit - offset
// Return latest messages (tail) with pagination
msgTotal := len(msgOrder)
start := msgTotal - limit - offset
if start < 0 {
start = 0
}
end := total - offset
end := msgTotal - offset
if end < 0 {
end = 0
}
if end > total {
end = total
if end > msgTotal {
end = msgTotal
}
messages := make([]map[string]interface{}, 0)
@@ -1395,7 +1539,7 @@ func (db *DB) GetChannelMessages(channelHash string, limit, offset int, region .
messages = append(messages, m.Data)
}
return messages, total, nil
return messages, msgTotal, nil
}
+81 -24
View File
@@ -60,6 +60,7 @@ func setupTestDB(t *testing.T) *DB {
payload_type INTEGER,
payload_version INTEGER,
decoded_json TEXT,
channel_hash TEXT DEFAULT NULL,
created_at TEXT DEFAULT (datetime('now'))
);
@@ -124,10 +125,10 @@ func seedTestData(t *testing.T, db *DB) {
VALUES ('1122334455667788', 'TestRoom', 'room', 37.4, -121.9, ?, '2026-01-01T00:00:00Z', 5)`, twoDaysAgo)
// Seed transmissions
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('AABB', 'abc123def4567890', ?, 1, 4, '{"pubKey":"aabbccdd11223344","name":"TestRepeater","type":"ADVERT","timestamp":1700000000,"timestampISO":"2023-11-14T22:13:20.000Z","signature":"abcdef","flags":{"isRepeater":true},"lat":37.5,"lon":-122.0}')`, recent)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('CCDD', '1234567890abcdef', ?, 1, 5, '{"type":"CHAN","channel":"#test","text":"Hello: World","sender":"TestUser"}')`, yesterday)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AABB', 'abc123def4567890', ?, 1, 4, '{"pubKey":"aabbccdd11223344","name":"TestRepeater","type":"ADVERT","timestamp":1700000000,"timestampISO":"2023-11-14T22:13:20.000Z","signature":"abcdef","flags":{"isRepeater":true},"lat":37.5,"lon":-122.0}', '#test')`, recent)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('CCDD', '1234567890abcdef', ?, 1, 5, '{"type":"CHAN","channel":"#test","text":"Hello: World","sender":"TestUser"}', '#test')`, yesterday)
// Second ADVERT for same node with different hash_size (raw_hex byte 0x1F → hs=1 vs 0xBB → hs=3)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('AA1F', 'def456abc1230099', ?, 1, 4, '{"pubKey":"aabbccdd11223344","name":"TestRepeater","type":"ADVERT","timestamp":1700000100,"timestampISO":"2023-11-14T22:14:40.000Z","signature":"fedcba","flags":{"isRepeater":true},"lat":37.5,"lon":-122.0}')`, yesterday)
@@ -735,12 +736,12 @@ func TestGetChannelMessagesRegionFiltering(t *testing.T) {
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer One', 'SJC')`)
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs2', 'Observer Two', ' sfo ')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'chanregion0001', ?, 1, 5,
'{"type":"CHAN","channel":"#region","text":"SjcUser: One","sender":"SjcUser"}')`, ts1)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
'{"type":"CHAN","channel":"#region","text":"SjcUser: One","sender":"SjcUser"}', '#region')`, ts1)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'chanregion0002', ?, 1, 5,
'{"type":"CHAN","channel":"#region","text":"SfoUser: Two","sender":"SfoUser"}')`, ts2)
'{"type":"CHAN","channel":"#region","text":"SfoUser: Two","sender":"SfoUser"}', '#region')`, ts2)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 10.0, -90, '[]', ?)`, epoch1)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
@@ -1119,6 +1120,7 @@ func setupTestDBV2(t *testing.T) *DB {
payload_type INTEGER,
payload_version INTEGER,
decoded_json TEXT,
channel_hash TEXT DEFAULT NULL,
created_at TEXT DEFAULT (datetime('now'))
);
@@ -1202,12 +1204,12 @@ func TestGetChannelMessagesDedup(t *testing.T) {
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs2', 'Observer Two', 'SFO')`)
// Insert two transmissions with same hash to test dedup
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'chanmsg00000001', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#general","text":"User1: Hello","sender":"User1"}')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
'{"type":"CHAN","channel":"#general","text":"User1: Hello","sender":"User1"}', '#general')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'chanmsg00000002', '2026-01-15T10:01:00Z', 1, 5,
'{"type":"CHAN","channel":"#general","text":"User2: World","sender":"User2"}')`)
'{"type":"CHAN","channel":"#general","text":"User2: World","sender":"User2"}', '#general')`)
// Observations: first msg seen by two observers (dedup), second by one
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
@@ -1251,9 +1253,9 @@ func TestGetChannelMessagesNoSender(t *testing.T) {
defer db.Close()
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer One', 'SJC')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('CC', 'chanmsg00000003', '2026-01-15T10:02:00Z', 1, 5,
'{"type":"CHAN","channel":"#noname","text":"plain text no colon"}')`)
'{"type":"CHAN","channel":"#noname","text":"plain text no colon"}', '#noname')`)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, 1, 12.0, -90, null, 1736935300)`)
@@ -1356,9 +1358,9 @@ func TestGetChannelMessagesObserverFallback(t *testing.T) {
defer db.Close()
// Observer with ID but no name entry (observer_idx won't match)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'chanmsg00000004', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#obs","text":"Sender: Test","sender":"Sender"}')`)
'{"type":"CHAN","channel":"#obs","text":"Sender: Test","sender":"Sender"}', '#obs')`)
// Observation without observer (observer_idx = NULL)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES (1, NULL, 12.0, -90, null, 1736935200)`)
@@ -1380,12 +1382,12 @@ func TestGetChannelsMultiple(t *testing.T) {
defer db.Close()
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer', 'SJC')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'chan1hash', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#alpha","text":"Alice: Hello","sender":"Alice"}')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
'{"type":"CHAN","channel":"#alpha","text":"Alice: Hello","sender":"Alice"}', '#alpha')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'chan2hash', '2026-01-15T10:01:00Z', 1, 5,
'{"type":"CHAN","channel":"#beta","text":"Bob: World","sender":"Bob"}')`)
'{"type":"CHAN","channel":"#beta","text":"Bob: World","sender":"Bob"}', '#beta')`)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
VALUES ('CC', 'chan3hash', '2026-01-15T10:02:00Z', 1, 5,
'{"type":"CHAN","channel":"","text":"No channel"}')`)
@@ -1468,13 +1470,13 @@ func TestGetChannelsStaleMessage(t *testing.T) {
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs2', 'Observer2', 'SFO')`)
// Older message (first_seen T1)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'oldhash1', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#test","text":"Alice: Old message","sender":"Alice"}')`)
'{"type":"CHAN","channel":"#test","text":"Alice: Old message","sender":"Alice"}', '#test')`)
// Newer message (first_seen T2 > T1)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'newhash2', '2026-01-15T10:05:00Z', 1, 5,
'{"type":"CHAN","channel":"#test","text":"Bob: New message","sender":"Bob"}')`)
'{"type":"CHAN","channel":"#test","text":"Bob: New message","sender":"Bob"}', '#test')`)
// Observations: older message re-observed AFTER newer message (stale scenario)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, timestamp)
@@ -1504,6 +1506,61 @@ func TestGetChannelsStaleMessage(t *testing.T) {
}
}
func TestGetChannelsRegionFiltering(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs1', 'Observer1', 'SJC')`)
db.conn.Exec(`INSERT INTO observers (id, name, iata) VALUES ('obs2', 'Observer2', 'SFO')`)
// Channel message seen only in SJC
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('AA', 'hash1', '2026-01-15T10:00:00Z', 1, 5,
'{"type":"CHAN","channel":"#sjc-only","text":"Alice: Hello SJC","sender":"Alice"}', '#sjc-only')`)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, timestamp)
VALUES (1, 1, 12.0, -90, 1736935200)`)
// Channel message seen only in SFO
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('BB', 'hash2', '2026-01-15T10:05:00Z', 1, 5,
'{"type":"CHAN","channel":"#sfo-only","text":"Bob: Hello SFO","sender":"Bob"}', '#sfo-only')`)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, timestamp)
VALUES (2, 2, 14.0, -88, 1736935500)`)
// No region filter — both channels
all, err := db.GetChannels()
if err != nil {
t.Fatal(err)
}
if len(all) != 2 {
t.Fatalf("expected 2 channels without region filter, got %d", len(all))
}
// Filter SJC — only #sjc-only
sjc, err := db.GetChannels("SJC")
if err != nil {
t.Fatal(err)
}
if len(sjc) != 1 {
t.Fatalf("expected 1 channel for SJC, got %d", len(sjc))
}
if sjc[0]["name"] != "#sjc-only" {
t.Errorf("expected channel '#sjc-only', got %q", sjc[0]["name"])
}
// Filter SFO — only #sfo-only
sfo, err := db.GetChannels("SFO")
if err != nil {
t.Fatal(err)
}
if len(sfo) != 1 {
t.Fatalf("expected 1 channel for SFO, got %d", len(sfo))
}
if sfo[0]["name"] != "#sfo-only" {
t.Errorf("expected channel '#sfo-only', got %q", sfo[0]["name"])
}
}
func TestNodeTelemetryFields(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
+21 -7
View File
@@ -116,6 +116,7 @@ type DecodedPacket struct {
Path Path `json:"path"`
Payload Payload `json:"payload"`
Raw string `json:"raw"`
Anomaly string `json:"anomaly,omitempty"`
}
func decodeHeader(b byte) Header {
@@ -388,22 +389,34 @@ func DecodePacket(hexString string, validateSignatures bool) (*DecodedPacket, er
payload := decodePayload(header.PayloadType, payloadBuf, validateSignatures)
// TRACE packets store hop IDs in the payload (buf[9:]) rather than the header
// path field. The header path byte still encodes hashSize in bits 6-7, which
// we use to split the payload path data into individual hop prefixes.
// The header path contains SNR bytes — one per hop that actually forwarded.
// path field. Firmware always sends TRACE as DIRECT (route_type 2 or 3);
// FLOOD-routed TRACEs are anomalous but handled gracefully (parsed, but
// flagged). The TRACE flags byte (payload offset 8) encodes path_sz in
// bits 0-1 as a power-of-two exponent: hash_bytes = 1 << path_sz.
// NOT the header path byte's hash_size bits. The header path contains SNR
// bytes — one per hop that actually forwarded.
// We expose hopsCompleted (count of SNR bytes) so consumers can distinguish
// how far the trace got vs the full intended route.
var anomaly string
if header.PayloadType == PayloadTRACE && payload.PathData != "" {
// Flag anomalous routing — firmware only sends TRACE as DIRECT
if header.RouteType != RouteDirect && header.RouteType != RouteTransportDirect {
anomaly = "TRACE packet with non-DIRECT routing (expected DIRECT or TRANSPORT_DIRECT)"
}
// The header path hops count represents SNR entries = completed hops
hopsCompleted := path.HashCount
pathBytes, err := hex.DecodeString(payload.PathData)
if err == nil && path.HashSize > 0 {
hops := make([]string, 0, len(pathBytes)/path.HashSize)
for i := 0; i+path.HashSize <= len(pathBytes); i += path.HashSize {
hops = append(hops, strings.ToUpper(hex.EncodeToString(pathBytes[i:i+path.HashSize])))
if err == nil && payload.TraceFlags != nil {
// path_sz from flags byte is a power-of-two exponent per firmware:
// hash_bytes = 1 << (flags & 0x03)
pathSz := 1 << (*payload.TraceFlags & 0x03)
hops := make([]string, 0, len(pathBytes)/pathSz)
for i := 0; i+pathSz <= len(pathBytes); i += pathSz {
hops = append(hops, strings.ToUpper(hex.EncodeToString(pathBytes[i:i+pathSz])))
}
path.Hops = hops
path.HashCount = len(hops)
path.HashSize = pathSz
path.HopsCompleted = &hopsCompleted
}
}
@@ -424,6 +437,7 @@ func DecodePacket(hexString string, validateSignatures bool) (*DecodedPacket, er
Path: path,
Payload: payload,
Raw: strings.ToUpper(hexString),
Anomaly: anomaly,
}, nil
}
+122
View File
@@ -357,6 +357,10 @@ func TestDecodePacket_TraceHopsCompleted(t *testing.T) {
if *pkt.Path.HopsCompleted != 2 {
t.Errorf("expected HopsCompleted=2, got %d", *pkt.Path.HopsCompleted)
}
// FLOOD routing for TRACE is anomalous
if pkt.Anomaly == "" {
t.Error("expected anomaly flag for FLOOD-routed TRACE")
}
}
func TestDecodePacket_TraceNoSNR(t *testing.T) {
@@ -407,6 +411,124 @@ func TestDecodePacket_TraceFullyCompleted(t *testing.T) {
}
}
func TestDecodePacket_TraceFlags1_TwoBytePathSz(t *testing.T) {
// TRACE with flags=1 → path_sz = 1 << (1 & 0x03) = 2-byte hashes
// Firmware always sends TRACE as DIRECT (route_type=2), so header byte =
// (0<<6)|(9<<2)|2 = 0x26. path_length 0x00 = 0 SNR bytes.
hex := "2600" + // header (DIRECT+TRACE) + path_length (0 SNR)
"01000000" + // tag
"02000000" + // authCode
"01" + // flags = 1 → path_sz = 2
"AABBCCDD" // 4 bytes = 2 hops of 2-byte each
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if len(pkt.Path.Hops) != 2 {
t.Errorf("expected 2 hops (2-byte path_sz), got %d: %v", len(pkt.Path.Hops), pkt.Path.Hops)
}
if pkt.Path.HashSize != 2 {
t.Errorf("expected HashSize=2, got %d", pkt.Path.HashSize)
}
if pkt.Anomaly != "" {
t.Errorf("expected no anomaly for DIRECT TRACE, got %q", pkt.Anomaly)
}
}
func TestDecodePacket_TraceFlags2_FourBytePathSz(t *testing.T) {
// TRACE with flags=2 → path_sz = 1 << (2 & 0x03) = 4-byte hashes
// DIRECT route_type (0x26)
hex := "2600" + // header (DIRECT+TRACE) + path_length (0 SNR)
"01000000" + // tag
"02000000" + // authCode
"02" + // flags = 2 → path_sz = 4
"AABBCCDD11223344" // 8 bytes = 2 hops of 4-byte each
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if len(pkt.Path.Hops) != 2 {
t.Errorf("expected 2 hops (4-byte path_sz), got %d: %v", len(pkt.Path.Hops), pkt.Path.Hops)
}
if pkt.Path.HashSize != 4 {
t.Errorf("expected HashSize=4, got %d", pkt.Path.HashSize)
}
}
func TestDecodePacket_TracePathSzUnevenPayload(t *testing.T) {
// TRACE with flags=1 → path_sz=2, but 5 bytes of path data (not evenly divisible)
// Should produce 2 hops (4 bytes) and ignore the trailing byte
hex := "2600" + // header (DIRECT+TRACE) + path_length (0 SNR)
"01000000" + // tag
"02000000" + // authCode
"01" + // flags = 1 → path_sz = 2
"AABBCCDDEE" // 5 bytes → 2 hops, 1 byte remainder ignored
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if len(pkt.Path.Hops) != 2 {
t.Errorf("expected 2 hops (trailing byte ignored), got %d: %v", len(pkt.Path.Hops), pkt.Path.Hops)
}
}
func TestDecodePacket_TraceTransportDirect(t *testing.T) {
// TRACE via TRANSPORT_DIRECT (route_type=3) — includes 4 transport code bytes
// header: (0<<6)|(9<<2)|3 = 0x27
hex := "27" + // header (TRANSPORT_DIRECT+TRACE)
"AABB" + "CCDD" + // transport codes (2+2 bytes)
"02" + // path_length: hash_count=2 SNR bytes
"EEFF" + // 2 SNR bytes
"01000000" + // tag
"02000000" + // authCode
"00" + // flags = 0 → path_sz = 1
"112233" // 3 hops (1-byte each)
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("DecodePacket error: %v", err)
}
if pkt.TransportCodes == nil {
t.Fatal("expected transport codes for TRANSPORT_DIRECT")
}
if pkt.TransportCodes.Code1 != "AABB" {
t.Errorf("expected Code1=AABB, got %s", pkt.TransportCodes.Code1)
}
if len(pkt.Path.Hops) != 3 {
t.Errorf("expected 3 hops, got %d: %v", len(pkt.Path.Hops), pkt.Path.Hops)
}
if pkt.Path.HopsCompleted == nil || *pkt.Path.HopsCompleted != 2 {
t.Errorf("expected HopsCompleted=2, got %v", pkt.Path.HopsCompleted)
}
if pkt.Anomaly != "" {
t.Errorf("expected no anomaly for TRANSPORT_DIRECT TRACE, got %q", pkt.Anomaly)
}
}
func TestDecodePacket_TraceFloodRouteAnomaly(t *testing.T) {
// TRACE via FLOOD (route_type=1) — anomalous per firmware (firmware only
// sends TRACE as DIRECT). Should still parse but flag the anomaly.
hex := "2500" + // header (FLOOD+TRACE) + path_length (0 SNR)
"01000000" + // tag
"02000000" + // authCode
"01" + // flags = 1 → path_sz = 2
"AABBCCDD" // 4 bytes = 2 hops of 2-byte each
pkt, err := DecodePacket(hex, false)
if err != nil {
t.Fatalf("should not crash on anomalous FLOOD+TRACE: %v", err)
}
if len(pkt.Path.Hops) != 2 {
t.Errorf("expected 2 hops even for anomalous FLOOD route, got %d", len(pkt.Path.Hops))
}
if pkt.Anomaly == "" {
t.Error("expected anomaly flag for FLOOD-routed TRACE, got empty string")
}
}
func TestDecodeAdvertSignatureValidation(t *testing.T) {
pub, priv, err := ed25519.GenerateKey(nil)
if err != nil {
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"encoding/json"
"net/http/httptest"
"testing"
"time"
)
// seedEncryptedChannelData adds undecryptable GRP_TXT packets to the test DB.
func seedEncryptedChannelData(t *testing.T, db *DB) {
t.Helper()
now := time.Now().UTC()
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
recentEpoch := now.Add(-1 * time.Hour).Unix()
// Two encrypted GRP_TXT packets on channel hash "A1B2"
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('EE01', 'enc_hash_001', ?, 1, 5, '{"type":"GRP_TXT","channelHashHex":"A1B2","decryptionStatus":"no_key"}', 'enc_A1B2')`, recent)
db.conn.Exec(`INSERT INTO transmissions (raw_hex, hash, first_seen, route_type, payload_type, decoded_json, channel_hash)
VALUES ('EE02', 'enc_hash_002', ?, 1, 5, '{"type":"GRP_TXT","channelHashHex":"A1B2","decryptionStatus":"no_key"}', 'enc_A1B2')`, recent)
// Observations for both
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES ((SELECT id FROM transmissions WHERE hash='enc_hash_001'), 1, 10.0, -90, '[]', ?)`, recentEpoch)
db.conn.Exec(`INSERT INTO observations (transmission_id, observer_idx, snr, rssi, path_json, timestamp)
VALUES ((SELECT id FROM transmissions WHERE hash='enc_hash_002'), 1, 10.0, -90, '[]', ?)`, recentEpoch)
}
func TestGetEncryptedChannels(t *testing.T) {
db := setupTestDB(t)
defer db.Close()
seedTestData(t, db)
seedEncryptedChannelData(t, db)
channels, err := db.GetEncryptedChannels()
if err != nil {
t.Fatal(err)
}
if len(channels) != 1 {
t.Fatalf("expected 1 encrypted channel, got %d", len(channels))
}
ch := channels[0]
if ch["hash"] != "enc_A1B2" {
t.Errorf("expected hash enc_A1B2, got %v", ch["hash"])
}
if ch["encrypted"] != true {
t.Errorf("expected encrypted=true, got %v", ch["encrypted"])
}
if ch["messageCount"] != 2 {
t.Errorf("expected messageCount=2, got %v", ch["messageCount"])
}
}
func TestChannelsAPIExcludesEncrypted(t *testing.T) {
_, router := setupTestServer(t)
// Seed encrypted data into the server's DB
// setupTestServer uses seedTestData which has no encrypted packets,
// so default /api/channels should NOT include encrypted channels.
req := httptest.NewRequest("GET", "/api/channels", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d", w.Code)
}
var body map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &body)
channels := body["channels"].([]interface{})
for _, ch := range channels {
m := ch.(map[string]interface{})
if enc, ok := m["encrypted"]; ok && enc == true {
t.Errorf("default /api/channels should not include encrypted channels, found: %v", m["hash"])
}
}
}
func TestChannelsAPIIncludesEncryptedWithParam(t *testing.T) {
srv, router := setupTestServer(t)
// Add encrypted data to the server's DB
seedEncryptedChannelData(t, srv.db)
// Reload store so in-memory also has the data
store := NewPacketStore(srv.db, nil)
if err := store.Load(); err != nil {
t.Fatalf("store.Load: %v", err)
}
srv.store = store
req := httptest.NewRequest("GET", "/api/channels?includeEncrypted=true", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d", w.Code)
}
var body map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &body)
channels := body["channels"].([]interface{})
foundEncrypted := false
for _, ch := range channels {
m := ch.(map[string]interface{})
if enc, ok := m["encrypted"]; ok && enc == true {
foundEncrypted = true
break
}
}
if !foundEncrypted {
t.Error("expected encrypted channels with includeEncrypted=true, found none")
}
}
func TestChannelMessagesExcludesEncrypted(t *testing.T) {
srv, router := setupTestServer(t)
seedEncryptedChannelData(t, srv.db)
store := NewPacketStore(srv.db, nil)
if err := store.Load(); err != nil {
t.Fatalf("store.Load: %v", err)
}
srv.store = store
// Request messages for the encrypted channel — should return empty
req := httptest.NewRequest("GET", "/api/channels/enc_A1B2/messages", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("expected 200, got %d", w.Code)
}
var body map[string]interface{}
json.Unmarshal(w.Body.Bytes(), &body)
messages, ok := body["messages"].([]interface{})
if !ok {
// messages might be null/missing — that's fine, means no messages
return
}
// Encrypted messages should not be returned as readable messages
for _, msg := range messages {
m := msg.(map[string]interface{})
if text, ok := m["text"].(string); ok && text != "" {
t.Errorf("encrypted channel should not return readable messages, got text: %s", text)
}
}
}
+9 -3
View File
@@ -541,13 +541,19 @@ func TestEstimateStoreTxBytes(t *testing.T) {
PathJSON: `["aa","bb"]`,
}
est := estimateStoreTxBytes(tx)
// Verify the function returns a reasonable value matching our manual calculation
// Manual calculation: base + string lengths + index entries + perTxMaps + path hops + subpaths
hops := int64(len(txGetParsedPath(tx)))
manualCalc := int64(storeTxBaseBytes) + int64(len(tx.RawHex)+len(tx.Hash)+len(tx.DecodedJSON)+len(tx.PathJSON)) + int64(numIndexesPerTx*indexEntryBytes)
manualCalc += perTxMapsBytes
manualCalc += hops * perPathHopBytes
if hops > 1 {
manualCalc += (hops * (hops - 1) / 2) * perSubpathEntryBytes
}
if est != manualCalc {
t.Fatalf("estimateStoreTxBytes = %d, want %d (manual calc)", est, manualCalc)
}
if est < 600 || est > 800 {
t.Fatalf("estimateStoreTxBytes = %d, expected in range [600, 800]", est)
if est < 600 || est > 1200 {
t.Fatalf("estimateStoreTxBytes = %d, expected in range [600, 1200]", est)
}
}
+121 -7
View File
@@ -85,7 +85,7 @@ func TestMultiByteCapability_Confirmed(t *testing.T) {
store := NewPacketStore(db, nil)
addTestPacket(store, makeTestAdvert("aabbccdd11223344", 2))
caps := store.computeMultiByteCapability()
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
@@ -123,7 +123,7 @@ func TestMultiByteCapability_Suspected(t *testing.T) {
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability()
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
@@ -152,7 +152,7 @@ func TestMultiByteCapability_Unknown(t *testing.T) {
// Advert with 1-byte hash only
addTestPacket(store, makeTestAdvert("aabbccdd11223344", 1))
caps := store.computeMultiByteCapability()
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
@@ -194,7 +194,7 @@ func TestMultiByteCapability_PrefixCollision(t *testing.T) {
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability()
caps := store.computeMultiByteCapability(nil)
if len(caps) != 2 {
t.Fatalf("expected 2 entries, got %d", len(caps))
}
@@ -237,7 +237,7 @@ func TestMultiByteCapability_TraceExcluded(t *testing.T) {
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability()
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
@@ -269,7 +269,7 @@ func TestMultiByteCapability_NonTraceStillSuspected(t *testing.T) {
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability()
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
@@ -304,7 +304,7 @@ func TestMultiByteCapability_ConfirmedUnaffectedByTraceExclusion(t *testing.T) {
}
addTestPacket(store, pkt)
caps := store.computeMultiByteCapability()
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
@@ -312,3 +312,117 @@ func TestMultiByteCapability_ConfirmedUnaffectedByTraceExclusion(t *testing.T) {
t.Errorf("expected confirmed (unaffected by TRACE), got %s", caps[0].Status)
}
}
// TestMultiByteCapability_CompanionConfirmed tests that a companion with
// multi-byte advert is classified as "confirmed", not "unknown" (Bug 1, #754).
func TestMultiByteCapability_CompanionConfirmed(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "CompA", "companion", "2026-04-11T00:00:00Z")
store := NewPacketStore(db, nil)
addTestPacket(store, makeTestAdvert("aabbccdd11223344", 2))
caps := store.computeMultiByteCapability(nil)
if len(caps) != 1 {
t.Fatalf("expected 1 entry, got %d", len(caps))
}
if caps[0].Status != "confirmed" {
t.Errorf("expected confirmed for companion, got %s", caps[0].Status)
}
if caps[0].Role != "companion" {
t.Errorf("expected role companion, got %s", caps[0].Role)
}
if caps[0].Evidence != "advert" {
t.Errorf("expected advert evidence, got %s", caps[0].Evidence)
}
}
// TestMultiByteCapability_RoleColumnPopulated tests that the Role field is
// populated for all node types (Bug 2, #754).
func TestMultiByteCapability_RoleColumnPopulated(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabb000000000001", "Rep1", "repeater", "2026-04-11T00:00:00Z")
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"ccdd000000000002", "Comp1", "companion", "2026-04-11T00:00:00Z")
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"eeff000000000003", "Room1", "room_server", "2026-04-11T00:00:00Z")
store := NewPacketStore(db, nil)
addTestPacket(store, makeTestAdvert("aabb000000000001", 2))
addTestPacket(store, makeTestAdvert("ccdd000000000002", 2))
addTestPacket(store, makeTestAdvert("eeff000000000003", 1))
caps := store.computeMultiByteCapability(nil)
if len(caps) != 3 {
t.Fatalf("expected 3 entries, got %d", len(caps))
}
roleByName := map[string]string{}
for _, c := range caps {
roleByName[c.Name] = c.Role
}
if roleByName["Rep1"] != "repeater" {
t.Errorf("Rep1 role: expected repeater, got %s", roleByName["Rep1"])
}
if roleByName["Comp1"] != "companion" {
t.Errorf("Comp1 role: expected companion, got %s", roleByName["Comp1"])
}
if roleByName["Room1"] != "room_server" {
t.Errorf("Room1 role: expected room_server, got %s", roleByName["Room1"])
}
}
// TestMultiByteCapability_AdopterEvidenceTakesPrecedence tests that when
// adopter data shows hashSize >= 2 but path evidence says "suspected",
// the node is upgraded to "confirmed" (Bug 3, #754).
func TestMultiByteCapability_AdopterEvidenceTakesPrecedence(t *testing.T) {
db := setupCapabilityTestDB(t)
defer db.conn.Close()
db.conn.Exec("INSERT INTO nodes (public_key, name, role, last_seen) VALUES (?, ?, ?, ?)",
"aabbccdd11223344", "RepAdopter", "repeater", "2026-04-11T00:00:00Z")
store := NewPacketStore(db, nil)
// Only a path-based packet (no advert) — would normally be "suspected"
pathByte := buildPathByte(2, 1)
rawHex := "01" + pathByte + "aabb"
pt := 1
pkt := &StoreTx{
RawHex: rawHex,
PayloadType: &pt,
PathJSON: `["aabb"]`,
FirstSeen: "2026-04-10T00:00:00.000Z",
}
addTestPacket(store, pkt)
// Without adopter data: should be suspected
caps := store.computeMultiByteCapability(nil)
capByName := map[string]MultiByteCapEntry{}
for _, c := range caps {
capByName[c.Name] = c
}
if capByName["RepAdopter"].Status != "suspected" {
t.Errorf("without adopter data: expected suspected, got %s", capByName["RepAdopter"].Status)
}
// With adopter data showing hashSize 2: should be confirmed
adopterHS := map[string]int{"aabbccdd11223344": 2}
caps = store.computeMultiByteCapability(adopterHS)
capByName = map[string]MultiByteCapEntry{}
for _, c := range caps {
capByName[c.Name] = c
}
if capByName["RepAdopter"].Status != "confirmed" {
t.Errorf("with adopter data: expected confirmed, got %s", capByName["RepAdopter"].Status)
}
if capByName["RepAdopter"].Evidence != "advert" {
t.Errorf("with adopter data: expected advert evidence, got %s", capByName["RepAdopter"].Evidence)
}
}
+20
View File
@@ -373,6 +373,26 @@ func (s *Server) buildNodeInfoMap() map[string]nodeInfo {
for _, n := range nodes {
m[strings.ToLower(n.PublicKey)] = n
}
// Enrich observer-only nodes: if an observer pubkey isn't already in the
// map (i.e. it's not also a repeater/companion), add it with role "observer".
if s.db != nil {
rows, err := s.db.conn.Query("SELECT id, name FROM observers")
if err == nil {
defer rows.Close()
for rows.Next() {
var id, name string
if rows.Scan(&id, &name) != nil {
continue
}
key := strings.ToLower(id)
if _, exists := m[key]; !exists {
m[key] = nodeInfo{PublicKey: id, Name: name, Role: "observer"}
}
}
}
}
return m
}
+68
View File
@@ -1,6 +1,7 @@
package main
import (
"database/sql"
"encoding/json"
"net/http"
"net/http/httptest"
@@ -8,6 +9,7 @@ import (
"time"
"github.com/gorilla/mux"
_ "modernc.org/sqlite"
)
// ─── Helpers ───────────────────────────────────────────────────────────────────
@@ -457,3 +459,69 @@ func TestNeighborGraphAPI_ResponseShape(t *testing.T) {
}
}
}
// ─── Tests: buildNodeInfoMap observer enrichment (#753) ────────────────────────
func TestBuildNodeInfoMap_ObserverEnrichment(t *testing.T) {
// Create a temp SQLite DB with nodes and observers tables.
tmpDir := t.TempDir()
dbPath := tmpDir + "/test.db"
conn, err := sql.Open("sqlite", dbPath)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
// Create tables
for _, stmt := range []string{
"CREATE TABLE nodes (public_key TEXT, name TEXT, role TEXT, lat REAL, lon REAL)",
"CREATE TABLE observers (id TEXT, name TEXT)",
"INSERT INTO nodes VALUES ('AAAA1111', 'Repeater-1', 'repeater', 0, 0)",
"INSERT INTO observers VALUES ('BBBB2222', 'Observer-Alpha')",
"INSERT INTO observers VALUES ('AAAA1111', 'Obs-also-repeater')",
} {
if _, err := conn.Exec(stmt); err != nil {
t.Fatalf("exec %q: %v", stmt, err)
}
}
conn.Close()
// Open via our DB wrapper
db, err := OpenDB(dbPath)
if err != nil {
t.Fatal(err)
}
defer db.conn.Close()
// Build a PacketStore with this DB (minimal — just need getCachedNodesAndPM)
store := NewPacketStore(db, nil)
store.Load()
srv := &Server{
db: db,
store: store,
perfStats: NewPerfStats(),
}
m := srv.buildNodeInfoMap()
// AAAA1111 should be from nodes table (repeater), NOT overwritten by observer
if info, ok := m["aaaa1111"]; !ok {
t.Error("expected aaaa1111 in map")
} else if info.Role != "repeater" {
t.Errorf("expected role=repeater for aaaa1111, got %q", info.Role)
}
// BBBB2222 should be enriched from observers table
if info, ok := m["bbbb2222"]; !ok {
t.Error("expected bbbb2222 in map (observer-only node)")
} else {
if info.Role != "observer" {
t.Errorf("expected role=observer for bbbb2222, got %q", info.Role)
}
if info.Name != "Observer-Alpha" {
t.Errorf("expected name=Observer-Alpha for bbbb2222, got %q", info.Name)
}
}
}
+1 -1
View File
@@ -166,7 +166,7 @@ func BuildFromStoreWithLog(store *PacketStore, enableLog bool) *NeighborGraph {
// Phase 1: Extract edges from every transmission + observation.
for _, tx := range packets {
isAdvert := tx.PayloadType != nil && *tx.PayloadType == 4
isAdvert := tx.PayloadType != nil && *tx.PayloadType == PayloadADVERT
fromNode := extractFromNode(tx)
// Pre-compute lowered originator once per tx (not per observation).
fromLower := ""
+1 -1
View File
@@ -525,7 +525,7 @@ type edgeCandidate struct {
// For ADVERTs: originator↔path[0] (if unambiguous). For ALL types: observer↔path[last] (if unambiguous).
// Also handles zero-hop ADVERTs (originator↔observer direct link).
func extractEdgesFromObs(obs *StoreObs, tx *StoreTx, pm *prefixMap) []edgeCandidate {
isAdvert := tx.PayloadType != nil && *tx.PayloadType == 4
isAdvert := tx.PayloadType != nil && *tx.PayloadType == PayloadADVERT
fromNode := extractFromNode(tx)
path := parsePathJSON(obs.PathJSON)
observerPK := strings.ToLower(obs.ObserverID)
+1 -1
View File
@@ -27,7 +27,7 @@ func createTestDBWithSchema(t *testing.T) (*DB, string) {
id INTEGER PRIMARY KEY AUTOINCREMENT,
raw_hex TEXT, hash TEXT UNIQUE, first_seen TEXT,
route_type INTEGER, payload_type INTEGER, payload_version INTEGER,
decoded_json TEXT
decoded_json TEXT, channel_hash TEXT DEFAULT NULL
)`)
conn.Exec(`CREATE TABLE observers (
id TEXT PRIMARY KEY, name TEXT, iata TEXT
+68 -13
View File
@@ -142,6 +142,9 @@ func (s *Server) RegisterRoutes(r *mux.Router) {
r.HandleFunc("/api/nodes/{pubkey}/health", s.handleNodeHealth).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/paths", s.handleNodePaths).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/analytics", s.handleNodeAnalytics).Methods("GET")
r.HandleFunc("/api/nodes/clock-skew", s.handleFleetClockSkew).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/clock-skew", s.handleNodeClockSkew).Methods("GET")
r.HandleFunc("/api/observers/clock-skew", s.handleObserverClockSkew).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}/neighbors", s.handleNodeNeighbors).Methods("GET")
r.HandleFunc("/api/nodes/{pubkey}", s.handleNodeDetail).Methods("GET")
r.HandleFunc("/api/nodes", s.handleNodes).Methods("GET")
@@ -1315,6 +1318,36 @@ func (s *Server) handleNodeAnalytics(w http.ResponseWriter, r *http.Request) {
writeError(w, 404, "Not found")
}
func (s *Server) handleNodeClockSkew(w http.ResponseWriter, r *http.Request) {
pubkey := mux.Vars(r)["pubkey"]
if s.store == nil {
writeError(w, 404, "Not found")
return
}
result := s.store.GetNodeClockSkew(pubkey)
if result == nil {
writeError(w, 404, "No clock skew data for this node")
return
}
writeJSON(w, result)
}
func (s *Server) handleObserverClockSkew(w http.ResponseWriter, r *http.Request) {
if s.store == nil {
writeJSON(w, []ObserverCalibration{})
return
}
writeJSON(w, s.store.GetObserverCalibrations())
}
func (s *Server) handleFleetClockSkew(w http.ResponseWriter, r *http.Request) {
if s.store == nil {
writeJSON(w, []*NodeClockSkew{})
return
}
writeJSON(w, s.store.GetFleetClockSkew())
}
// --- Analytics Handlers ---
func (s *Server) handleAnalyticsRF(w http.ResponseWriter, r *http.Request) {
@@ -1655,18 +1688,35 @@ func (s *Server) handleResolveHops(w http.ResponseWriter, r *http.Request) {
}
func (s *Server) handleChannels(w http.ResponseWriter, r *http.Request) {
if s.store != nil {
region := r.URL.Query().Get("region")
channels := s.store.GetChannels(region)
region := r.URL.Query().Get("region")
includeEncrypted := r.URL.Query().Get("includeEncrypted") == "true"
// Prefer DB for full history (in-memory store has limited retention)
if s.db != nil {
channels, err := s.db.GetChannels(region)
if err != nil {
writeError(w, 500, err.Error())
return
}
if includeEncrypted {
encrypted, err := s.db.GetEncryptedChannels(region)
if err != nil {
log.Printf("WARN GetEncryptedChannels: %v", err)
} else {
channels = append(channels, encrypted...)
}
}
writeJSON(w, ChannelListResponse{Channels: channels})
return
}
channels, err := s.db.GetChannels()
if err != nil {
writeError(w, 500, err.Error())
if s.store != nil {
channels := s.store.GetChannels(region)
if includeEncrypted {
channels = append(channels, s.store.GetEncryptedChannels(region)...)
}
writeJSON(w, ChannelListResponse{Channels: channels})
return
}
writeJSON(w, ChannelListResponse{Channels: channels})
writeJSON(w, ChannelListResponse{Channels: []map[string]interface{}{}})
}
func (s *Server) handleChannelMessages(w http.ResponseWriter, r *http.Request) {
@@ -1674,17 +1724,22 @@ func (s *Server) handleChannelMessages(w http.ResponseWriter, r *http.Request) {
limit := queryInt(r, "limit", 100)
offset := queryInt(r, "offset", 0)
region := r.URL.Query().Get("region")
// Prefer DB for full history (in-memory store has limited retention)
if s.db != nil {
messages, total, err := s.db.GetChannelMessages(hash, limit, offset, region)
if err != nil {
writeError(w, 500, err.Error())
return
}
writeJSON(w, ChannelMessagesResponse{Messages: messages, Total: total})
return
}
if s.store != nil {
messages, total := s.store.GetChannelMessages(hash, limit, offset, region)
writeJSON(w, ChannelMessagesResponse{Messages: messages, Total: total})
return
}
messages, total, err := s.db.GetChannelMessages(hash, limit, offset, region)
if err != nil {
writeError(w, 500, err.Error())
return
}
writeJSON(w, ChannelMessagesResponse{Messages: messages, Total: total})
writeJSON(w, ChannelMessagesResponse{Messages: []map[string]interface{}{}, Total: 0})
}
func (s *Server) handleObservers(w http.ResponseWriter, r *http.Request) {
+6 -6
View File
@@ -2219,8 +2219,8 @@ pk := "abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890"
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role) VALUES (?, 'TestNode', 'repeater')", pk)
decoded := `{"name":"TestNode","pubKey":"` + pk + `"}`
raw1 := "04" + "00" + "aabb"
raw2 := "04" + "40" + "aabb"
raw1 := "11" + "01" + "aabb"
raw2 := "11" + "41" + "aabb"
payloadType := 4
for i := 0; i < 3; i++ {
@@ -2267,8 +2267,8 @@ pk := "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role) VALUES (?, 'Repeater2B', 'repeater')", pk)
decoded := `{"name":"Repeater2B","pubKey":"` + pk + `"}`
raw1byte := "04" + "00" + "aabb" // pathByte=0x00 → hashSize=1 (direct send, no hops)
raw2byte := "04" + "40" + "aabb" // pathByte=0x40 → hashSize=2
raw1byte := "11" + "01" + "aabb" // FLOOD, pathByte=0x01 → hashSize=1
raw2byte := "11" + "41" + "aabb" // FLOOD, pathByte=0x41 → hashSize=2
payloadType := 4
// 1 packet with hashSize=1, 4 packets with hashSize=2 (latest is 2-byte)
@@ -2310,8 +2310,8 @@ func TestGetNodeHashSizeInfoLatestWins(t *testing.T) {
db.conn.Exec("INSERT OR IGNORE INTO nodes (public_key, name, role) VALUES (?, 'LatestWins', 'repeater')", pk)
decoded := `{"name":"LatestWins","pubKey":"` + pk + `"}`
raw1byte := "04" + "00" + "aabb" // pathByte=0x00 → hashSize=1
raw2byte := "04" + "40" + "aabb" // pathByte=0x40 → hashSize=2
raw1byte := "11" + "01" + "aabb" // FLOOD, pathByte=0x01 → hashSize=1
raw2byte := "11" + "41" + "aabb" // FLOOD, pathByte=0x41 → hashSize=2
payloadType := 4
// 4 historical 1-byte adverts, then 1 recent 2-byte advert (latest).
+188 -25
View File
@@ -200,6 +200,9 @@ type PacketStore struct {
// Persisted neighbor graph for hop resolution at ingest time.
graph *NeighborGraph
// Clock skew detection engine.
clockSkew *ClockSkewEngine
// Async backfill state: set after backfillResolvedPathsAsync completes.
backfillComplete atomic.Bool
// Progress tracking for async backfill (total pending and processed so far).
@@ -304,6 +307,7 @@ func NewPacketStore(db *DB, cfg *PacketStoreConfig, cacheTTLs ...map[string]inte
spTxIndex: make(map[string][]*StoreTx, 4096),
advertPubkeys: make(map[string]int),
lastSeenTouched: make(map[string]time.Time),
clockSkew: NewClockSkewEngine(),
}
if cfg != nil {
ps.retentionHours = cfg.RetentionHours
@@ -640,7 +644,7 @@ func (s *PacketStore) touchRelayLastSeen(tx *StoreTx, now time.Time) {
// trackAdvertPubkey increments the advertPubkeys refcount for ADVERT packets.
// Must be called under s.mu write lock.
func (s *PacketStore) trackAdvertPubkey(tx *StoreTx) {
if tx.PayloadType == nil || *tx.PayloadType != 4 || tx.DecodedJSON == "" {
if tx.PayloadType == nil || *tx.PayloadType != PayloadADVERT || tx.DecodedJSON == "" {
return
}
d := tx.ParsedDecoded()
@@ -661,7 +665,7 @@ func (s *PacketStore) trackAdvertPubkey(tx *StoreTx) {
// untrackAdvertPubkey decrements the advertPubkeys refcount for ADVERT packets.
// Must be called under s.mu write lock.
func (s *PacketStore) untrackAdvertPubkey(tx *StoreTx) {
if tx.PayloadType == nil || *tx.PayloadType != 4 || tx.DecodedJSON == "" {
if tx.PayloadType == nil || *tx.PayloadType != PayloadADVERT || tx.DecodedJSON == "" {
return
}
var d map[string]interface{}
@@ -1086,6 +1090,11 @@ func (s *PacketStore) GetPerfStoreStatsTyped() PerfPacketStoreStats {
estimatedMB := math.Round(s.estimatedMemoryMB()*10) / 10
trackedMB := math.Round(s.trackedMemoryMB()*10) / 10
var avgBytesPerPacket int64
if totalLoaded > 0 {
avgBytesPerPacket = s.trackedBytes / int64(totalLoaded)
}
return PerfPacketStoreStats{
TotalLoaded: totalLoaded,
TotalObservations: totalObs,
@@ -1097,6 +1106,7 @@ func (s *PacketStore) GetPerfStoreStatsTyped() PerfPacketStoreStats {
MaxPackets: 2386092,
EstimatedMB: estimatedMB,
TrackedMB: trackedMB,
AvgBytesPerPacket: avgBytesPerPacket,
MaxMB: s.maxMemoryMB,
Indexes: PacketStoreIndexes{
ByHash: hashIdx,
@@ -2596,27 +2606,68 @@ func (s *PacketStore) buildDistanceIndex() {
// These estimate the in-memory cost of StoreTx and StoreObs structs including
// map/index overhead. They don't need to be exact — just proportional to actual
// usage and independent of GC state.
//
// Issue #743: Previous estimates missed major per-packet allocations:
// - spTxIndex: O(path²) entries per tx (50-150MB at scale)
// - ResolvedPath on observations (~25MB at scale)
// - Per-tx maps: obsKeys, observerSet (~11MB at scale)
// - byPathHop index entries (20-40MB at scale)
const (
storeTxBaseBytes = 384 // StoreTx struct fields + map headers + sync.Once + string headers
storeObsBaseBytes = 192 // StoreObs struct fields + string headers
indexEntryBytes = 48 // average cost of one index map entry (key + pointer + bucket overhead)
numIndexesPerTx = 5 // byHash, byTxID, byNode, byPayloadType, nodeHashes entries
numIndexesPerObs = 2 // byObsID, byObserver entries
// Per-tx map overhead (obsKeys + observerSet): map header + initial buckets
perTxMapsBytes = 200
// Per path hop: byPathHop index entry (pointer + map bucket)
perPathHopBytes = 50
// Per subpath entry in spTxIndex: string key + slice append + pointer
perSubpathEntryBytes = 40
// Per resolved path element on an observation
perResolvedPathElemBytes = 24 // *string pointer + string header + avg pubkey length
)
// estimateStoreTxBytes returns the estimated memory cost of a StoreTx (excluding observations).
// Includes per-tx maps (obsKeys, observerSet), byPathHop entries, and spTxIndex subpath entries.
func estimateStoreTxBytes(tx *StoreTx) int64 {
base := int64(storeTxBaseBytes)
base += int64(len(tx.RawHex) + len(tx.Hash) + len(tx.DecodedJSON) + len(tx.PathJSON))
base += int64(numIndexesPerTx * indexEntryBytes)
// Per-tx maps: obsKeys + observerSet
base += perTxMapsBytes
// Path-dependent costs
hops := int64(len(txGetParsedPath(tx)))
base += hops * perPathHopBytes
// spTxIndex: O(path²) subpath combinations
if hops > 1 {
subpaths := hops * (hops - 1) / 2
base += subpaths * perSubpathEntryBytes
}
return base
}
// estimateStoreObsBytes returns the estimated memory cost of a StoreObs.
// Includes ResolvedPath slice overhead.
func estimateStoreObsBytes(obs *StoreObs) int64 {
base := int64(storeObsBaseBytes)
base += int64(len(obs.PathJSON) + len(obs.ObserverID))
base += int64(numIndexesPerObs * indexEntryBytes)
// ResolvedPath: slice header + per-element pointer/string
if obs.ResolvedPath != nil {
base += 24 // slice header
base += int64(len(obs.ResolvedPath)) * perResolvedPathElemBytes
}
return base
}
@@ -3155,6 +3206,84 @@ func (s *PacketStore) GetChannels(region string) []map[string]interface{} {
return channels
}
// GetEncryptedChannels returns undecryptable GRP_TXT channels from in-memory packets.
func (s *PacketStore) GetEncryptedChannels(region string) []map[string]interface{} {
s.mu.RLock()
var regionObs map[string]bool
if region != "" {
regionObs = s.resolveRegionObservers(region)
}
grpTxts := s.byPayloadType[5]
type encInfo struct {
hash string
messageCount int
lastActivity string
}
type grpDec struct {
Type string `json:"type"`
ChannelHash interface{} `json:"channelHash"`
ChannelHashHex string `json:"channelHashHex"`
DecryptionStatus string `json:"decryptionStatus"`
}
channelMap := map[string]*encInfo{}
for _, tx := range grpTxts {
if regionObs != nil {
match := false
for _, obs := range tx.Observations {
if regionObs[obs.ObserverID] {
match = true
break
}
}
if !match {
continue
}
}
var decoded grpDec
if json.Unmarshal([]byte(tx.DecodedJSON), &decoded) != nil {
continue
}
if decoded.Type != "GRP_TXT" || decoded.DecryptionStatus != "no_key" {
continue
}
chHash := decoded.ChannelHashHex
if chHash == "" {
if num, ok := decoded.ChannelHash.(float64); ok {
chHash = fmt.Sprintf("%02X", int(num))
}
}
if chHash == "" {
chHash = "?"
}
ch := channelMap[chHash]
if ch == nil {
ch = &encInfo{hash: chHash, lastActivity: tx.FirstSeen}
channelMap[chHash] = ch
}
ch.messageCount++
if tx.FirstSeen >= ch.lastActivity {
ch.lastActivity = tx.FirstSeen
}
}
s.mu.RUnlock()
channels := make([]map[string]interface{}, 0, len(channelMap))
for _, ch := range channelMap {
channels = append(channels, map[string]interface{}{
"hash": "enc_" + ch.hash,
"name": "Encrypted (0x" + ch.hash + ")",
"lastMessage": nil,
"lastSender": nil,
"messageCount": ch.messageCount,
"lastActivity": ch.lastActivity,
"encrypted": true,
})
}
return channels
}
// GetChannelMessages returns deduplicated messages for a channel from in-memory packets.
func (s *PacketStore) GetChannelMessages(channelHash string, limit, offset int, region ...string) ([]map[string]interface{}, int) {
s.mu.RLock()
@@ -4976,7 +5105,18 @@ func (s *PacketStore) GetAnalyticsHashSizes(region string) map[string]interface{
// Add multi-byte capability data (only for unfiltered/global view)
if region == "" {
result["multiByteCapability"] = s.computeMultiByteCapability()
// Pass adopter hash sizes so capability can cross-reference
adopterHS := make(map[string]int)
if mbNodes, ok := result["multiByteNodes"].([]map[string]interface{}); ok {
for _, n := range mbNodes {
pk, _ := n["pubkey"].(string)
hs, _ := n["hashSize"].(int)
if pk != "" && hs >= 2 {
adopterHS[pk] = hs
}
}
}
result["multiByteCapability"] = s.computeMultiByteCapability(adopterHS)
}
s.cacheMu.Lock()
@@ -5056,7 +5196,7 @@ func (s *PacketStore) computeAnalyticsHashSizes(region string) map[string]interf
// Track originator from advert packets (including zero-hop adverts,
// keyed by pubKey so same-name nodes don't merge).
if tx.PayloadType != nil && *tx.PayloadType == 4 && tx.DecodedJSON != "" {
if tx.PayloadType != nil && *tx.PayloadType == PayloadADVERT && tx.DecodedJSON != "" {
var d map[string]interface{}
if json.Unmarshal([]byte(tx.DecodedJSON), &d) == nil {
pk := ""
@@ -5077,16 +5217,26 @@ func (s *PacketStore) computeAnalyticsHashSizes(region string) map[string]interf
name = pk
}
}
// Skip zero-hop direct adverts for hash_size — the
// path byte is locally generated and unreliable.
// Still count the packet and update lastSeen.
isZeroHop := (routeType == uint64(RouteDirect) || routeType == uint64(RouteTransportDirect)) && (actualPathByte&0x3F) == 0
if byNode[pk] == nil {
role := nodeRoleByPK[pk] // empty if unknown
initHS := hashSize
if isZeroHop {
initHS = 0
}
byNode[pk] = map[string]interface{}{
"hashSize": hashSize, "packets": 0,
"hashSize": initHS, "packets": 0,
"lastSeen": tx.FirstSeen, "name": name,
"role": role,
}
}
byNode[pk]["packets"] = byNode[pk]["packets"].(int) + 1
byNode[pk]["hashSize"] = hashSize
if !isZeroHop {
byNode[pk]["hashSize"] = hashSize
}
byNode[pk]["lastSeen"] = tx.FirstSeen
}
}
@@ -5591,13 +5741,22 @@ func (s *PacketStore) computeNodeHashSizeInfo() map[string]*hashSizeNodeInfo {
continue
}
routeType := int(header & 0x03)
pathByte, err := strconv.ParseUint(tx.RawHex[2:4], 16, 8)
// Transport routes (0, 3) have 4 transport code bytes before the path
// byte, so the path byte is at offset 5 instead of 1.
pbOffset := 1
if routeType == RouteTransportFlood || routeType == RouteTransportDirect {
pbOffset = 5
}
if len(tx.RawHex) < (pbOffset+1)*2 {
continue
}
pathByte, err := strconv.ParseUint(tx.RawHex[pbOffset*2:pbOffset*2+2], 16, 8)
if err != nil {
continue
}
// DIRECT zero-hop adverts use path byte 0x00 locally and can misreport
// multibyte repeater hash mode as 1-byte.
if routeType == RouteDirect && (pathByte&0x3F) == 0 {
// Direct zero-hop adverts (route types 2 and 3) use path byte 0x00
// locally and can misreport multibyte hash mode as 1-byte.
if (routeType == RouteDirect || routeType == RouteTransportDirect) && (pathByte&0x3F) == 0 {
continue
}
hs := int((pathByte>>6)&0x3) + 1
@@ -5670,7 +5829,7 @@ func EnrichNodeWithHashSize(node map[string]interface{}, info *hashSizeNodeInfo)
// --- Multi-Byte Capability Inference ---
// MultiByteCapEntry represents a repeater's inferred multi-byte capability.
// MultiByteCapEntry represents a node's inferred multi-byte capability.
type MultiByteCapEntry struct {
PublicKey string `json:"pubkey"`
Name string `json:"name"`
@@ -5682,7 +5841,7 @@ type MultiByteCapEntry struct {
}
// computeMultiByteCapability determines multi-byte capability for each
// repeater using two methods:
// node (repeaters, companions, rooms, sensors) using two methods:
//
// 1. Confirmed: the node has advertised with hash_size >= 2 (from advert
// path byte). This is 100% reliable because the full public key is
@@ -5699,7 +5858,7 @@ type MultiByteCapEntry struct {
// with default (1-byte) settings.
//
// Caller must hold NO locks — this method acquires mu.RLock internally.
func (s *PacketStore) computeMultiByteCapability() []MultiByteCapEntry {
func (s *PacketStore) computeMultiByteCapability(adopterHashSizes map[string]int) []MultiByteCapEntry {
// Get hash size info from adverts (has its own locking)
hashInfo := s.GetNodeHashSizeInfo()
@@ -5734,24 +5893,21 @@ func (s *PacketStore) computeMultiByteCapability() []MultiByteCapEntry {
pubkey string
prefix string
}
repeaterPrefixes := make(map[string][]prefixEntry) // prefix → entries
for pk, n := range nodeByPK {
if !strings.Contains(strings.ToLower(n.Role), "repeater") {
continue
}
nodePrefixes := make(map[string][]prefixEntry) // prefix → entries
for pk := range nodeByPK {
// Generate 1-byte, 2-byte, 3-byte prefixes
pkLower := strings.ToLower(pk)
for byteLen := 1; byteLen <= 3; byteLen++ {
hexLen := byteLen * 2
if len(pkLower) >= hexLen {
pfx := pkLower[:hexLen]
repeaterPrefixes[pfx] = append(repeaterPrefixes[pfx], prefixEntry{pk, pfx})
nodePrefixes[pfx] = append(nodePrefixes[pfx], prefixEntry{pk, pfx})
}
}
}
suspected := make(map[string]int) // pubkey → max hash size from path appearances
for pfx, entries := range repeaterPrefixes {
for pfx, entries := range nodePrefixes {
txList := s.byPathHop[pfx]
for _, tx := range txList {
if tx.RawHex == "" || len(tx.RawHex) < 4 {
@@ -5797,9 +5953,9 @@ func (s *PacketStore) computeMultiByteCapability() []MultiByteCapEntry {
}
s.mu.RUnlock()
// Build result for all repeaters — fetch last_seen from DB
// Build result for all nodes — fetch last_seen from DB
dbLastSeen := make(map[string]string)
rows, err := s.db.conn.Query("SELECT public_key, last_seen FROM nodes WHERE role LIKE '%repeater%'")
rows, err := s.db.conn.Query("SELECT public_key, last_seen FROM nodes")
if err == nil {
defer rows.Close()
for rows.Next() {
@@ -5814,9 +5970,6 @@ func (s *PacketStore) computeMultiByteCapability() []MultiByteCapEntry {
var result []MultiByteCapEntry
for pk, n := range nodeByPK {
if !strings.Contains(strings.ToLower(n.Role), "repeater") {
continue
}
entry := MultiByteCapEntry{
PublicKey: pk,
Name: n.Name,
@@ -5829,6 +5982,12 @@ func (s *PacketStore) computeMultiByteCapability() []MultiByteCapEntry {
entry.Status = "confirmed"
entry.Evidence = "advert"
entry.MaxHashSize = maxHS
} else if maxHS, ok := adopterHashSizes[pk]; ok && maxHS >= 2 {
// Adopter data (from computeAnalyticsHashSizes) shows hash_size >= 2
// from advert analysis — this is advert-based evidence, so confirmed.
entry.Status = "confirmed"
entry.Evidence = "advert"
entry.MaxHashSize = maxHS
} else if maxHS, ok := suspected[pk]; ok {
entry.Status = "suspected"
entry.Evidence = "path"
@@ -6524,6 +6683,9 @@ func (s *PacketStore) GetNodeAnalytics(pubkey string, days int) (*NodeAnalyticsR
relayPct = round(float64(relayedCount)*100.0/float64(totalWithPath), 1)
}
// Compute clock skew (already under RLock).
clockSkew := s.getNodeClockSkewLocked(pubkey)
return &NodeAnalyticsResponse{
Node: node,
TimeRange: TimeRangeResp{From: fromISO, To: toISO, Days: days},
@@ -6547,6 +6709,7 @@ func (s *PacketStore) GetNodeAnalytics(pubkey string, days int) (*NodeAnalyticsR
UniquePeers: len(peerSlice),
AvgPacketsPerDay: avgPacketsPerDay,
},
ClockSkew: clockSkew,
}, nil
}
+168
View File
@@ -0,0 +1,168 @@
package main
import (
"testing"
"time"
)
// TestEstimateStoreTxBytes_ReasonableValues verifies the estimate function
// returns reasonable values for different packet sizes.
func TestEstimateStoreTxBytes_ReasonableValues(t *testing.T) {
tx := &StoreTx{
Hash: "abcdef1234567890",
RawHex: "deadbeef",
DecodedJSON: `{"type":"GRP_TXT"}`,
PathJSON: `["hop1","hop2","hop3"]`,
parsedPath: []string{"hop1", "hop2", "hop3"},
pathParsed: true,
}
got := estimateStoreTxBytes(tx)
// Should be at least base (384) + maps (200) + indexes + path/subpath costs
if got < 700 {
t.Errorf("estimate too low for 3-hop tx: %d", got)
}
if got > 5000 {
t.Errorf("estimate unreasonably high for 3-hop tx: %d", got)
}
}
// TestEstimateStoreTxBytes_ManyHopsSubpaths verifies that packets with many
// hops estimate significantly more due to O(path²) subpath index entries.
func TestEstimateStoreTxBytes_ManyHopsSubpaths(t *testing.T) {
tx2 := &StoreTx{
Hash: "aabb",
parsedPath: []string{"a", "b"},
pathParsed: true,
}
tx10 := &StoreTx{
Hash: "aabb",
parsedPath: []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j"},
pathParsed: true,
}
est2 := estimateStoreTxBytes(tx2)
est10 := estimateStoreTxBytes(tx10)
// 10 hops → 45 subpath combos × 40 = 1800 bytes just for subpaths
if est10 <= est2 {
t.Errorf("10-hop (%d) should estimate more than 2-hop (%d)", est10, est2)
}
if est10 < est2+1500 {
t.Errorf("10-hop (%d) should estimate at least 1500 more than 2-hop (%d)", est10, est2)
}
}
// TestEstimateStoreObsBytes_WithResolvedPath verifies that observations with
// ResolvedPath estimate more than those without.
func TestEstimateStoreObsBytes_WithResolvedPath(t *testing.T) {
s1, s2, s3 := "node1", "node2", "node3"
obsNoRP := &StoreObs{
ObserverID: "obs1",
PathJSON: `["a","b"]`,
}
obsWithRP := &StoreObs{
ObserverID: "obs1",
PathJSON: `["a","b"]`,
ResolvedPath: []*string{&s1, &s2, &s3},
}
estNo := estimateStoreObsBytes(obsNoRP)
estWith := estimateStoreObsBytes(obsWithRP)
if estWith <= estNo {
t.Errorf("obs with ResolvedPath (%d) should estimate more than without (%d)", estWith, estNo)
}
}
// TestEstimateStoreObsBytes_ManyObservations verifies that 15 observations
// estimate significantly more than 1.
func TestEstimateStoreObsBytes_ManyObservations(t *testing.T) {
est1 := estimateStoreObsBytes(&StoreObs{ObserverID: "a", PathJSON: `["x"]`})
est15 := int64(0)
for i := 0; i < 15; i++ {
est15 += estimateStoreObsBytes(&StoreObs{ObserverID: "a", PathJSON: `["x"]`})
}
if est15 <= est1*10 {
t.Errorf("15 obs total (%d) should be >10x single obs (%d)", est15, est1)
}
}
// TestTrackedBytesMatchesSumAfterInsert verifies that trackedBytes equals the
// sum of individual estimates after inserting packets via makeTestStore.
func TestTrackedBytesMatchesSumAfterInsert(t *testing.T) {
store := makeTestStore(20, time.Now().Add(-2*time.Hour), 5)
// Manually compute trackedBytes as sum of estimates
var expectedSum int64
for _, tx := range store.packets {
expectedSum += estimateStoreTxBytes(tx)
for _, obs := range tx.Observations {
expectedSum += estimateStoreObsBytes(obs)
}
}
if store.trackedBytes != expectedSum {
t.Errorf("trackedBytes=%d, expected sum=%d", store.trackedBytes, expectedSum)
}
}
// TestEvictionTriggersWithImprovedEstimates verifies that eviction triggers
// at the right point with the improved (higher) estimates.
func TestEvictionTriggersWithImprovedEstimates(t *testing.T) {
store := makeTestStore(100, time.Now().Add(-10*time.Hour), 5)
// trackedBytes for 100 packets is small — artificially set maxMemoryMB
// so highWatermark is just below trackedBytes to trigger eviction.
highWatermarkBytes := store.trackedBytes - 1000
if highWatermarkBytes < 1 {
highWatermarkBytes = 1
}
// maxMemoryMB * 1048576 = highWatermark, so maxMemoryMB = ceil(highWatermarkBytes / 1048576)
// But that'll be 0 for small values. Instead, directly set trackedBytes high.
store.trackedBytes = 6 * 1048576 // 6MB
store.maxMemoryMB = 3 // 3MB limit
beforeCount := len(store.packets)
store.RunEviction()
afterCount := len(store.packets)
if afterCount >= beforeCount {
t.Errorf("expected eviction to remove packets: before=%d, after=%d, trackedBytes=%d, maxMB=%d",
beforeCount, afterCount, store.trackedBytes, store.maxMemoryMB)
}
// trackedBytes should have decreased
if store.trackedBytes >= 6*1048576 {
t.Errorf("trackedBytes should have decreased after eviction")
}
}
// BenchmarkEstimateStoreTxBytes verifies the estimate function is fast.
func BenchmarkEstimateStoreTxBytes(b *testing.B) {
tx := &StoreTx{
Hash: "abcdef1234567890",
RawHex: "deadbeefdeadbeef",
DecodedJSON: `{"type":"GRP_TXT","payload":"hello"}`,
PathJSON: `["hop1","hop2","hop3","hop4","hop5"]`,
parsedPath: []string{"hop1", "hop2", "hop3", "hop4", "hop5"},
pathParsed: true,
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
estimateStoreTxBytes(tx)
}
}
// BenchmarkEstimateStoreObsBytes verifies the obs estimate function is fast.
func BenchmarkEstimateStoreObsBytes(b *testing.B) {
s := "resolvedNodePubkey123456"
obs := &StoreObs{
ObserverID: "observer1234",
PathJSON: `["a","b","c"]`,
ResolvedPath: []*string{&s, &s, &s},
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
estimateStoreObsBytes(obs)
}
}
+2
View File
@@ -176,6 +176,7 @@ type PerfPacketStoreStats struct {
MaxPackets int `json:"maxPackets"`
EstimatedMB float64 `json:"estimatedMB"`
TrackedMB float64 `json:"trackedMB"`
AvgBytesPerPacket int64 `json:"avgBytesPerPacket"`
MaxMB int `json:"maxMB"`
Indexes PacketStoreIndexes `json:"indexes"`
}
@@ -468,6 +469,7 @@ type NodeAnalyticsResponse struct {
PeerInteractions []PeerInteraction `json:"peerInteractions"`
UptimeHeatmap []HeatmapCell `json:"uptimeHeatmap"`
ComputedStats ComputedNodeStats `json:"computedStats"`
ClockSkew *NodeClockSkew `json:"clockSkew,omitempty"`
}
// ─── Analytics — RF ────────────────────────────────────────────────────────────
+11 -1
View File
@@ -125,7 +125,7 @@
}
],
"channelKeys": {
"public": "8b3387e9c5cdea6ac9e5edbaa115cd72"
"Public": "8b3387e9c5cdea6ac9e5edbaa115cd72"
},
"hashChannels": [
"#LongFast",
@@ -153,6 +153,16 @@
],
"zoom": 9
},
"geo_filter": {
"polygon": [
[37.80, -122.52],
[37.80, -121.80],
[37.20, -121.80],
[37.20, -122.52]
],
"bufferKm": 20,
"_comment": "Optional. Restricts ingestion and API responses to nodes within the polygon + bufferKm. Polygon is an array of [lat, lon] pairs (minimum 3). Use tools/geofilter-builder.html to draw a polygon visually. Remove this section to disable filtering. Nodes with no GPS fix are always allowed through."
},
"regions": {
"SJC": "San Jose, US",
"SFO": "San Francisco, US",
+13
View File
@@ -176,6 +176,19 @@ Lower values = fresher data but more server load.
Provide cert and key paths to enable HTTPS.
## Geographic filtering
```json
"geo_filter": {
"polygon": [[51.55, 3.80], [51.55, 5.90], [50.65, 5.90], [50.65, 3.80]],
"bufferKm": 20
}
```
Restricts ingestion and API responses to nodes within the polygon plus a buffer margin. Remove the block to disable filtering. Nodes with no GPS fix always pass through.
See [Geographic Filtering](geofilter.md) for the full guide including the visual polygon builder and the prune script for cleaning up historical data.
## Home page
The `home` section customizes the onboarding experience. See `config.example.json` for the full structure including `steps`, `checklist`, and `footerLinks`.
+6
View File
@@ -66,6 +66,12 @@ Click **Import JSON** and paste a previously exported theme. The customizer load
Click **Reset to Defaults** to restore all settings to the built-in defaults.
## GeoFilter Builder
The Export tab includes a **GeoFilter Builder →** link. Click it to open a Leaflet map where you can draw a polygon boundary for your deployment area. The tool generates a `geo_filter` block you can paste directly into `config.json`.
See [Geographic Filtering](geofilter.md) for full details on what geo filtering does and how to configure it.
## How it works
The customizer writes CSS custom properties (variables) to override the defaults. Exported JSON maps directly to the `theme`, `nodeColors`, `branding`, and `home` sections of [config.json](configuration.md).
+114
View File
@@ -0,0 +1,114 @@
# Geographic Filtering
CoreScope supports geographic filtering to restrict which nodes are ingested and returned in API responses. This is useful for public-facing deployments that should only show activity in a specific region.
## How it works
Geographic filtering operates at two levels:
- **Ingest time** — ADVERT packets carrying GPS coordinates are rejected by the ingestor if the node falls outside the configured area. The node never reaches the database.
- **API responses** — Nodes already in the database are filtered from the `/api/nodes` response if they fall outside the area. This covers nodes ingested before the filter was configured.
Nodes with no GPS fix (`lat=0, lon=0` or missing coordinates) always pass the filter regardless of configuration.
## Configuration
Add a `geo_filter` block to `config.json`:
```json
"geo_filter": {
"polygon": [
[51.55, 3.80],
[51.55, 5.90],
[50.65, 5.90],
[50.65, 3.80]
],
"bufferKm": 20
}
```
| Field | Type | Description |
|-------|------|-------------|
| `polygon` | `[[lat, lon], ...]` | Array of at least 3 coordinate pairs defining the boundary |
| `bufferKm` | number | Extra distance (km) around the polygon edge that is also accepted. `0` = exact boundary |
Both the server and the ingestor read `geo_filter` from `config.json`. Restart both after changing this section.
To disable filtering entirely, remove the `geo_filter` block.
### Legacy bounding box
An older bounding box format is also supported as a fallback when no `polygon` is present:
```json
"geo_filter": {
"latMin": 50.65,
"latMax": 51.55,
"lonMin": 3.80,
"lonMax": 5.90
}
```
Prefer the polygon format — it supports irregular shapes and the `bufferKm` margin.
## API endpoint
The current geo filter configuration is exposed at:
```
GET /api/config/geo-filter
```
The frontend reads this endpoint to display the active filter. No authentication is required (the endpoint returns config, not private data).
## GeoFilter Builder
The simplest way to create a polygon is the included visual builder:
**File:** `tools/geofilter-builder.html`
Open it directly in a browser — it runs entirely client-side, no server required:
```bash
# From the project root
open tools/geofilter-builder.html # macOS
xdg-open tools/geofilter-builder.html # Linux
start tools/geofilter-builder.html # Windows
```
**Workflow:**
1. The map opens centered on Belgium by default. Navigate to your region.
2. Click on the map to add polygon vertices. Each click adds a numbered point.
3. Add at least 3 points to form a closed polygon.
4. Adjust **Buffer km** (default 20) to add a margin around the polygon edge.
5. The generated JSON block appears at the bottom of the page — copy it directly into `config.json`.
6. Use **↩ Undo** to remove the last point, **✕ Clear** to start over.
The output is a complete `{ "geo_filter": { ... } }` block ready to paste into `config.json`.
## Cleaning up historical nodes
The ingestor prevents new out-of-bounds nodes from being ingested, but it does not retroactively remove nodes that were stored before the filter was configured. For that, use the prune script.
**File:** `scripts/prune-nodes-outside-geo-filter.py`
```bash
# Dry run — shows what would be deleted without making any changes
python3 scripts/prune-nodes-outside-geo-filter.py --dry-run
# Default paths: /app/data/meshcore.db and /app/config.json
python3 scripts/prune-nodes-outside-geo-filter.py
# Custom paths
python3 scripts/prune-nodes-outside-geo-filter.py /path/to/meshcore.db \
--config /path/to/config.json
# In Docker — run inside the container
docker exec -it meshcore-analyzer \
python3 /app/scripts/prune-nodes-outside-geo-filter.py --dry-run
```
The script reads `geo_filter.polygon` and `geo_filter.bufferKm` from config, lists the nodes that fall outside, then asks for `yes` confirmation before deleting. Nodes without coordinates are always kept.
This is a **one-time migration tool** — run it once after first configuring `geo_filter` to clean up pre-filter data. The ingestor handles all subsequent filtering automatically at ingest time.
+136 -9
View File
@@ -87,6 +87,7 @@
<button class="tab-btn" data-tab="distance">Distance</button>
<button class="tab-btn" data-tab="neighbor-graph">Neighbor Graph</button>
<button class="tab-btn" data-tab="rf-health">RF Health</button>
<button class="tab-btn" data-tab="clock-health">Clock Health</button>
<button class="tab-btn" data-tab="prefix-tool">Prefix Tool</button>
</div>
</div>
@@ -181,6 +182,7 @@
case 'distance': await renderDistanceTab(el); break;
case 'neighbor-graph': await renderNeighborGraphTab(el); break;
case 'rf-health': await renderRFHealthTab(el); break;
case 'clock-health': await renderClockHealthTab(el); break;
case 'prefix-tool': await renderPrefixTool(el); break;
}
// Auto-apply column resizing to all analytics tables
@@ -998,6 +1000,7 @@
return (filtered.length ? '<table class="analytics-table" id="mbAdoptersTable" style="margin-top:12px">' +
'<thead><tr>' +
'<th scope="col" data-sort="name">Node</th>' +
'<th scope="col" data-sort="role">Role</th>' +
'<th scope="col" data-sort="status">Status</th>' +
'<th scope="col" data-sort="hashSize">Hash Size</th>' +
'<th scope="col" data-sort="packets">Adverts</th>' +
@@ -1005,8 +1008,10 @@
'</tr></thead>' +
'<tbody>' +
filtered.map(function(r) {
var roleColor = (window.ROLE_COLORS || {})[r.role] || '#6b7280';
return '<tr class="clickable-row" data-action="navigate" data-value="#/nodes/' + encodeURIComponent(r.pubkey) + '" tabindex="0" role="row">' +
'<td><strong>' + esc(r.name) + '</strong></td>' +
'<td><span class="badge" style="background:' + roleColor + '20;color:' + roleColor + '">' + esc(r.role || 'unknown') + '</span></td>' +
'<td><span style="color:' + (statusColor[r.status] || statusColor.unknown) + '">' +
(statusIcon[r.status] || '❓') + ' ' + (statusLabel[r.status] || 'Unknown') + '</span></td>' +
'<td><span class="badge badge-hash-' + r.hashSize + '">' + r.hashSize + '-byte</span></td>' +
@@ -1190,10 +1195,10 @@
else matrixDesc.textContent = '3-byte prefix space is too large to visualize as a matrix — collision table is shown below.';
}
renderHashMatrixFromServer(cData.by_size[String(bytes)], bytes);
// Hide collision risk card for 3-byte — stats are shown in the matrix panel
// Show collision risk section for all byte sizes
const riskCard = document.getElementById('collisionRiskSection');
if (riskCard) riskCard.style.display = bytes === 3 ? 'none' : '';
if (bytes !== 3) renderCollisionsFromServer(cData.by_size[String(bytes)], bytes);
if (riskCard) riskCard.style.display = '';
renderCollisionsFromServer(cData.by_size[String(bytes)], bytes);
}
// Wire up selector
@@ -1285,9 +1290,9 @@
<div class="analytics-stat-value" style="font-size:16px">${pctStr}%</div>
<div style="font-size:10px;color:var(--text-muted);margin-top:2px">${usedCount > 256 ? usedCount + ' of ' : 'of '}${spaceLabel} possible</div>
</div>
<div class="analytics-stat-card" style="flex:1;min-width:110px;border-color:${collisionCount > 0 ? 'var(--status-red)' : 'var(--border)'}">
<div class="analytics-stat-card" style="flex:1;min-width:110px;border-color:${collisionCount > 0 ? 'var(--status-red)' : 'var(--border)'}${collisionCount > 0 ? ';cursor:pointer' : ''}" ${collisionCount > 0 ? 'onclick="document.getElementById(\'collisionRiskSection\')?.scrollIntoView({behavior:\'smooth\',block:\'start\'})"' : ''} ${collisionCount > 0 ? 'title="Click to see collision details"' : ''}>
<div class="analytics-stat-label">Prefix collisions</div>
<div class="analytics-stat-value" style="color:${collisionCount > 0 ? 'var(--status-red)' : 'var(--status-green)'}">${collisionCount}</div>
<div class="analytics-stat-value" style="color:${collisionCount > 0 ? 'var(--status-red)' : 'var(--status-green)'}">${collisionCount}${collisionCount > 0 ? ' <span style="font-size:11px;opacity:0.7">▼</span>' : ''}</div>
</div>
</div>`;
}
@@ -1362,7 +1367,7 @@
// 3-byte: show a summary panel instead of a matrix
if (bytes === 3) {
el.innerHTML = hashStatCardsHtml(totalNodes, stats.using_this_size || 0, '3-byte', 16777216, stats.unique_prefixes || 0, stats.collision_count || 0) +
`<p class="text-muted" style="margin:0;font-size:0.8em">The 3-byte prefix space (16.7M values) is too large to visualize as a grid.</p>` +
`<p class="text-muted" style="margin:0;font-size:0.8em">The 3-byte prefix space (16.7M values) is too large to visualize as a grid.${(stats.collision_count || 0) > 0 ? ' See collision details below.' : ''}</p>` +
`<p class="text-muted" style="margin:8px 0 0;font-size:0.8em">️ This tab only counts collisions among repeaters configured for this hash size. The <a href="#/analytics?tab=prefix-tool" style="color:var(--accent)">Prefix Tool</a> checks all repeaters regardless of configured hash size.</p>`;
return;
}
@@ -1995,6 +2000,8 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
window._analyticsRfNFColumnChart = rfNFColumnChart;
window._analyticsRenderMultiByteCapability = renderMultiByteCapability;
window._analyticsRenderMultiByteAdopters = renderMultiByteAdopters;
window._analyticsHashStatCardsHtml = hashStatCardsHtml;
window._analyticsRenderCollisionsFromServer = renderCollisionsFromServer;
}
// ─── Neighbor Graph Tab ─────────────────────────────────────────────────────
@@ -2009,8 +2016,8 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
<label style="font-size:13px">Roles:
<span id="ngRoleChecks" style="margin-left:4px"></span>
</label>
<label style="font-size:13px">Min Score: <input type="range" id="ngMinScore" min="0" max="100" value="10" style="width:100px;vertical-align:middle">
<span id="ngMinScoreVal">0.10</span>
<label style="font-size:13px">Min Score: <input type="range" id="ngMinScore" min="0" max="100" value="70" style="width:100px;vertical-align:middle">
<span id="ngMinScoreVal">0.70</span>
</label>
<label style="font-size:13px">Confidence:
<select id="ngConfidence" style="font-size:12px;padding:2px 4px">
@@ -2038,6 +2045,11 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
const color = (window.ROLE_COLORS || {})[r] || '#888';
rcEl.innerHTML += `<label style="font-size:12px;margin-right:8px"><input type="checkbox" data-role="${r}" checked> <span style="color:${esc(color)}">${esc(r)}</span></label>`;
});
// Observer checkbox — unchecked by default (observers create hub-and-spoke noise)
{
const color = (window.ROLE_COLORS || {}).observer || '#8b5cf6';
rcEl.innerHTML += `<label style="font-size:12px;margin-right:8px"><input type="checkbox" data-role="observer"> <span style="color:${esc(color)}">observer</span></label>`;
}
// Load data
const rqs = RegionFilter.regionQueryString();
@@ -2055,8 +2067,17 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
startGraphRenderer();
// Filter listeners
// Restore saved min score from localStorage
var savedScore = localStorage.getItem('ng-min-score');
if (savedScore !== null) {
document.getElementById('ngMinScore').value = savedScore;
document.getElementById('ngMinScoreVal').textContent = (savedScore / 100).toFixed(2);
applyNGFilters();
}
document.getElementById('ngMinScore').addEventListener('input', function() {
document.getElementById('ngMinScoreVal').textContent = (this.value / 100).toFixed(2);
localStorage.setItem('ng-min-score', this.value);
applyNGFilters();
});
document.getElementById('ngConfidence').addEventListener('change', applyNGFilters);
@@ -2095,7 +2116,7 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
// Filter nodes by role
const visibleNodes = _ngState.allNodes.filter(n => {
const role = (n.role || 'unknown').toLowerCase();
return checkedRoles.has(role) || role === 'unknown' || role === 'observer';
return checkedRoles.has(role) || role === 'unknown';
});
const visiblePKs = new Set(visibleNodes.map(n => n.pubkey));
@@ -3402,5 +3423,111 @@ function destroy() { _analyticsData = {}; _channelData = null; if (_ngState && _
return svg;
}
// #690 — Clock Health fleet view (M3)
async function renderClockHealthTab(el) {
el.innerHTML = '<div class="text-center text-muted" style="padding:40px">Loading clock health data…</div>';
try {
var data = await (await fetch('/api/nodes/clock-skew')).json();
if (!Array.isArray(data) || !data.length) {
el.innerHTML = '<div class="text-center text-muted" style="padding:40px">No clock skew data available. Nodes need recent adverts for clock analysis.</div>';
return;
}
// State
var activeFilter = 'all';
var sortKey = 'severity';
var sortDir = 'asc'; // severity worst-first
function render() {
// Filter
var filtered = activeFilter === 'all' ? data : data.filter(function(n) { return n.severity === activeFilter; });
// Sort
filtered = filtered.slice().sort(function(a, b) {
var v;
if (sortKey === 'severity') {
v = (SKEW_SEVERITY_ORDER[a.severity] || 9) - (SKEW_SEVERITY_ORDER[b.severity] || 9);
} else if (sortKey === 'skew') {
v = Math.abs(b.medianSkewSec || 0) - Math.abs(a.medianSkewSec || 0);
} else if (sortKey === 'name') {
v = (a.nodeName || '').localeCompare(b.nodeName || '');
} else if (sortKey === 'drift') {
v = Math.abs(b.driftPerDaySec || 0) - Math.abs(a.driftPerDaySec || 0);
}
return sortDir === 'desc' ? -v : v;
});
// Summary
var counts = { ok: 0, warning: 0, critical: 0, absurd: 0 };
data.forEach(function(n) { if (counts[n.severity] !== undefined) counts[n.severity]++; });
// Filter buttons (also serve as summary — no separate stats pills needed)
var filterColors = { ok: 'var(--status-green)', warning: 'var(--status-yellow)', critical: 'var(--status-orange)', absurd: 'var(--status-purple)', no_clock: 'var(--text-muted)' };
var filters = ['all', 'ok', 'warning', 'critical', 'absurd', 'no_clock'];
var filterHtml = '<div style="margin-bottom:10px">' + filters.map(function(f) {
var dot = f !== 'all' ? '<span style="display:inline-block;width:8px;height:8px;border-radius:50%;background:' + filterColors[f] + ';margin-right:4px;vertical-align:middle"></span>' : '';
return '<button class="clock-filter-btn' + (activeFilter === f ? ' active' : '') + '" data-filter="' + f + '">' +
dot + (f === 'all' ? 'All (' + data.length + ')' : (SKEW_SEVERITY_LABELS[f] || f) + ' (' + (counts[f] || 0) + ')') +
'</button>';
}).join('') + '</div>';
// Table
var rowsHtml = filtered.map(function(n) {
var rowClass = 'clock-fleet-row--' + (n.severity || 'ok');
var lastAdv = n.lastObservedTS ? new Date(n.lastObservedTS * 1000).toISOString().replace('T', ' ').replace(/\.\d+Z/, ' UTC') : '—';
var skewText = n.severity === 'no_clock' ? 'No Clock' : formatSkew(n.medianSkewSec);
var driftText = n.severity === 'no_clock' || !n.driftPerDaySec ? '' : formatDrift(n.driftPerDaySec);
return '<tr class="' + rowClass + '" data-pubkey="' + esc(n.pubkey) + '" style="cursor:pointer">' +
'<td><strong>' + esc(n.nodeName || n.pubkey.slice(0, 12)) + '</strong></td>' +
'<td style="font-family:var(--mono,monospace)">' + skewText + '</td>' +
'<td>' + renderSkewBadge(n.severity, n.medianSkewSec) + '</td>' +
'<td style="font-family:var(--mono,monospace)">' + driftText + '</td>' +
'<td style="font-size:11px">' + lastAdv + '</td>' +
'</tr>';
}).join('');
el.innerHTML = '<h3 style="margin:0 0 10px">⏰ Clock Health</h3>' +
filterHtml +
'<table class="data-table analytics-table" id="clock-health-table">' +
'<thead><tr>' +
'<th data-sort-col="name" style="cursor:pointer">Name</th>' +
'<th data-sort-col="skew" style="cursor:pointer">Skew</th>' +
'<th data-sort-col="severity" style="cursor:pointer">Severity</th>' +
'<th data-sort-col="drift" style="cursor:pointer">Drift Rate</th>' +
'<th>Last Advert</th>' +
'</tr></thead><tbody>' + rowsHtml + '</tbody></table>';
// Bind filter clicks
el.querySelectorAll('.clock-filter-btn').forEach(function(btn) {
btn.addEventListener('click', function() {
activeFilter = btn.dataset.filter;
render();
});
});
// Bind header sort clicks
el.querySelectorAll('[data-sort-col]').forEach(function(th) {
th.addEventListener('click', function() {
var col = th.dataset.sortCol;
if (sortKey === col) { sortDir = sortDir === 'asc' ? 'desc' : 'asc'; }
else { sortKey = col; sortDir = 'asc'; }
render();
});
});
// Bind row clicks → navigate to node
el.querySelectorAll('tr[data-pubkey]').forEach(function(tr) {
tr.addEventListener('click', function() {
location.hash = '#/nodes/' + encodeURIComponent(tr.dataset.pubkey);
});
});
}
render();
} catch (err) {
el.innerHTML = '<div class="text-center" style="color:var(--status-red);padding:40px">Failed to load clock health data: ' + esc(String(err)) + '</div>';
}
}
registerPage('analytics', { init, destroy });
})();
+295
View File
@@ -0,0 +1,295 @@
/**
* Client-side MeshCore channel decryption module.
*
* Implements the same crypto as internal/channel/channel.go:
* - Key derivation: SHA-256("#channelname")[:16]
* - Channel hash: SHA-256(key)[0]
* - MAC: HMAC-SHA256 with 32-byte secret (key + 16 zero bytes), truncated to 2 bytes
* - Encryption: AES-128-ECB (block-by-block)
* - Plaintext: timestamp(4 LE) + flags(1) + "sender: message\0"
*
* Keys NEVER leave the browser. No fetch/XHR/network calls in this module.
*/
/* eslint-disable no-var */
window.ChannelDecrypt = (function () {
'use strict';
var STORAGE_KEY = 'corescope_channel_keys';
var CACHE_KEY = 'corescope_channel_cache';
// ---- Hex utilities ----
function bytesToHex(bytes) {
var hex = '';
for (var i = 0; i < bytes.length; i++) {
hex += (bytes[i] < 16 ? '0' : '') + bytes[i].toString(16);
}
return hex;
}
function hexToBytes(hex) {
var bytes = new Uint8Array(hex.length / 2);
for (var i = 0; i < hex.length; i += 2) {
bytes[i / 2] = parseInt(hex.substring(i, i + 2), 16);
}
return bytes;
}
// ---- Key derivation ----
/**
* Derive AES-128 key from channel name: SHA-256("#channelname")[:16].
* @param {string} channelName - e.g. "#LongFast"
* @returns {Promise<Uint8Array>} 16-byte key
*/
async function deriveKey(channelName) {
var enc = new TextEncoder();
var hash = await crypto.subtle.digest('SHA-256', enc.encode(channelName));
return new Uint8Array(hash).slice(0, 16);
}
/**
* Compute the 1-byte channel hash: SHA-256(key)[0].
* @param {Uint8Array} key - 16-byte key
* @returns {Promise<number>} single byte (0-255)
*/
async function computeChannelHash(key) {
var hash = await crypto.subtle.digest('SHA-256', key);
return new Uint8Array(hash)[0];
}
// ---- AES-128-ECB via Web Crypto (CBC with zero IV, block-by-block) ----
/**
* Decrypt AES-128-ECB by decrypting each 16-byte block independently
* using AES-CBC with a zero IV (equivalent to ECB for single blocks).
* @param {Uint8Array} key - 16-byte AES key
* @param {Uint8Array} ciphertext - must be multiple of 16 bytes
* @returns {Promise<Uint8Array>} plaintext
*/
async function decryptECB(key, ciphertext) {
if (ciphertext.length === 0 || ciphertext.length % 16 !== 0) {
return null;
}
var cryptoKey = await crypto.subtle.importKey(
'raw', key, { name: 'AES-CBC' }, false, ['decrypt']
);
var zeroIV = new Uint8Array(16);
var plaintext = new Uint8Array(ciphertext.length);
for (var i = 0; i < ciphertext.length; i += 16) {
var block = ciphertext.slice(i, i + 16);
// Append a dummy block (16 bytes of 0x10 = PKCS7 padding for empty next block)
// so Web Crypto doesn't complain about padding
var padded = new Uint8Array(32);
padded.set(block, 0);
// Second block is PKCS7 padding: 16 bytes of 0x10
for (var j = 16; j < 32; j++) padded[j] = 16;
var decrypted = await crypto.subtle.decrypt(
{ name: 'AES-CBC', iv: zeroIV }, cryptoKey, padded
);
var decBytes = new Uint8Array(decrypted);
plaintext.set(decBytes.slice(0, 16), i);
}
return plaintext;
}
// ---- MAC verification ----
/**
* Verify HMAC-SHA256 MAC (first 2 bytes) using 32-byte secret (key + 16 zero bytes).
* @param {Uint8Array} key - 16-byte AES key
* @param {Uint8Array} ciphertext - encrypted data
* @param {string} macHex - 4-char hex string (2 bytes)
* @returns {Promise<boolean>}
*/
async function verifyMAC(key, ciphertext, macHex) {
// Build 32-byte channel secret: key + 16 zero bytes
var secret = new Uint8Array(32);
secret.set(key, 0);
// remaining 16 bytes are already 0
var cryptoKey = await crypto.subtle.importKey(
'raw', secret, { name: 'HMAC', hash: 'SHA-256' }, false, ['sign']
);
var sig = await crypto.subtle.sign('HMAC', cryptoKey, ciphertext);
var sigBytes = new Uint8Array(sig);
var macBytes = hexToBytes(macHex);
return sigBytes[0] === macBytes[0] && sigBytes[1] === macBytes[1];
}
// ---- Plaintext parsing ----
/**
* Parse decrypted plaintext: timestamp(4 LE) + flags(1) + "sender: message\0..."
* @param {Uint8Array} plaintext
* @returns {{ timestamp: number, flags: number, sender: string, message: string } | null}
*/
function parsePlaintext(plaintext) {
if (!plaintext || plaintext.length < 5) return null;
var timestamp = plaintext[0] | (plaintext[1] << 8) | (plaintext[2] << 16) | ((plaintext[3] << 24) >>> 0);
var flags = plaintext[4];
// Extract text up to first null byte
var textBytes = plaintext.slice(5);
var nullIdx = -1;
for (var i = 0; i < textBytes.length; i++) {
if (textBytes[i] === 0) { nullIdx = i; break; }
}
var text = new TextDecoder().decode(nullIdx >= 0 ? textBytes.slice(0, nullIdx) : textBytes);
// Count non-printable characters
var nonPrintable = 0;
for (var c = 0; c < text.length; c++) {
var code = text.charCodeAt(c);
if (code < 32 && code !== 10 && code !== 13 && code !== 9) nonPrintable++;
}
if (nonPrintable > 2) return null;
// Parse "sender: message" format
var colonIdx = text.indexOf(': ');
if (colonIdx > 0 && colonIdx < 50) {
var potentialSender = text.substring(0, colonIdx);
if (potentialSender.indexOf(':') < 0 && potentialSender.indexOf('[') < 0 && potentialSender.indexOf(']') < 0) {
return { timestamp: timestamp, flags: flags, sender: potentialSender, message: text.substring(colonIdx + 2) };
}
}
return { timestamp: timestamp, flags: flags, sender: '', message: text };
}
// ---- Full decrypt pipeline ----
/**
* Verify MAC, decrypt, and parse a single packet.
* @param {Uint8Array} keyBytes - 16-byte key
* @param {string} macHex - 4-char hex MAC
* @param {string} encryptedHex - hex-encoded ciphertext
* @returns {Promise<{ sender: string, message: string, timestamp: number } | null>}
*/
async function decrypt(keyBytes, macHex, encryptedHex) {
var ciphertext = hexToBytes(encryptedHex);
if (ciphertext.length === 0 || ciphertext.length % 16 !== 0) return null;
var macOk = await verifyMAC(keyBytes, ciphertext, macHex);
if (!macOk) return null;
var plaintext = await decryptECB(keyBytes, ciphertext);
if (!plaintext) return null;
return parsePlaintext(plaintext);
}
// Alias used by channels.js
var decryptPacket = decrypt;
// ---- Key storage (localStorage) ----
function saveKey(channelName, keyHex) {
var keys = getKeys();
keys[channelName] = keyHex;
try { localStorage.setItem(STORAGE_KEY, JSON.stringify(keys)); } catch (e) { /* quota */ }
}
// Alias used by channels.js
var storeKey = saveKey;
function getKeys() {
try {
var raw = localStorage.getItem(STORAGE_KEY);
return raw ? JSON.parse(raw) : {};
} catch (e) { return {}; }
}
// Alias used by channels.js
var getStoredKeys = getKeys;
function removeKey(channelName) {
var keys = getKeys();
delete keys[channelName];
try { localStorage.setItem(STORAGE_KEY, JSON.stringify(keys)); } catch (e) { /* quota */ }
// Also clear cached messages for this channel
clearChannelCache(channelName);
}
/** Remove cached messages for a specific channel (by name or hash). */
function clearChannelCache(channelKey) {
try {
var cache = JSON.parse(localStorage.getItem(CACHE_KEY) || '{}');
delete cache[channelKey];
localStorage.setItem(CACHE_KEY, JSON.stringify(cache));
} catch (e) { /* quota */ }
}
// ---- Message cache (localStorage) ----
function cacheMessages(channelHash, messages) {
try {
var cache = JSON.parse(localStorage.getItem(CACHE_KEY) || '{}');
cache[channelHash] = { messages: messages, ts: Date.now() };
localStorage.setItem(CACHE_KEY, JSON.stringify(cache));
} catch (e) { /* quota */ }
}
function getCachedMessages(channelHash) {
try {
var cache = JSON.parse(localStorage.getItem(CACHE_KEY) || '{}');
var entry = cache[channelHash];
return entry ? entry.messages : null;
} catch (e) { return null; }
}
// Cache with lastTimestamp and count (used by channels.js via getCache/setCache)
var MAX_CACHED_MESSAGES = 1000;
function setCache(key, messages, lastTimestamp, totalCount) {
try {
// Enforce cache size limit: only keep most recent MAX_CACHED_MESSAGES
var toStore = messages;
if (messages.length > MAX_CACHED_MESSAGES) {
toStore = messages.slice(messages.length - MAX_CACHED_MESSAGES);
}
var cache = JSON.parse(localStorage.getItem(CACHE_KEY) || '{}');
cache[key] = {
messages: toStore,
lastTimestamp: lastTimestamp,
count: totalCount || toStore.length,
ts: Date.now()
};
localStorage.setItem(CACHE_KEY, JSON.stringify(cache));
} catch (e) { /* quota */ }
}
function getCache(key) {
try {
var cache = JSON.parse(localStorage.getItem(CACHE_KEY) || '{}');
return cache[key] || null;
} catch (e) { return null; }
}
return {
deriveKey: deriveKey,
decrypt: decrypt,
decryptPacket: decryptPacket,
decryptECB: decryptECB,
verifyMAC: verifyMAC,
parsePlaintext: parsePlaintext,
computeChannelHash: computeChannelHash,
bytesToHex: bytesToHex,
hexToBytes: hexToBytes,
saveKey: saveKey,
storeKey: storeKey,
getKeys: getKeys,
getStoredKeys: getStoredKeys,
removeKey: removeKey,
clearChannelCache: clearChannelCache,
cacheMessages: cacheMessages,
getCachedMessages: getCachedMessages,
setCache: setCache,
getCache: getCache
};
})();
+454 -11
View File
@@ -318,6 +318,290 @@
let regionChangeHandler = null;
// --- Client-side channel decryption (#725 M2) ---
// Check if input is a valid hex string (32 hex chars = 16 bytes)
function isHexKey(val) {
return /^[0-9a-fA-F]{32}$/.test(val);
}
// Show status message in the add-channel form (#759)
var statusTimer = null;
function showAddStatus(msg, type) {
var el = document.getElementById('chAddStatus');
if (!el) return;
el.textContent = msg;
el.className = 'ch-add-status ch-add-status--' + (type || 'info');
el.style.display = '';
clearTimeout(statusTimer);
if (type !== 'loading') {
statusTimer = setTimeout(function () { el.style.display = 'none'; }, 5000);
}
}
// Add a user channel by name (#channelname) or hex key
async function addUserChannel(val) {
var displayName = val.startsWith('#') ? val : (isHexKey(val) ? val.substring(0, 8) + '…' : '#' + val);
showAddStatus('Decrypting ' + displayName + ' messages…', 'loading');
var channelName, keyHex;
try {
if (val.startsWith('#')) {
channelName = val;
var keyBytes = await ChannelDecrypt.deriveKey(channelName);
keyHex = ChannelDecrypt.bytesToHex(keyBytes);
} else if (isHexKey(val)) {
keyHex = val.toLowerCase();
channelName = 'psk:' + keyHex.substring(0, 8);
} else {
// Try with # prefix if user forgot
channelName = '#' + val;
var keyBytes2 = await ChannelDecrypt.deriveKey(channelName);
keyHex = ChannelDecrypt.bytesToHex(keyBytes2);
}
ChannelDecrypt.storeKey(channelName, keyHex);
// Compute channel hash byte to find matching encrypted channels
var keyBytes3 = ChannelDecrypt.hexToBytes(keyHex);
var hashByte = await ChannelDecrypt.computeChannelHash(keyBytes3);
// Add to sidebar or merge with existing encrypted channel
mergeUserChannels();
renderChannelList();
// Auto-select and start decrypting
var targetHash = 'user:' + channelName;
// Check if there's an existing encrypted channel with this hash byte
var existingEncrypted = channels.find(function (ch) {
return ch.encrypted && String(ch.hash) === String(hashByte);
});
if (existingEncrypted) {
targetHash = existingEncrypted.hash;
}
await selectChannel(targetHash, { userKey: keyHex, channelHashByte: hashByte, channelName: channelName });
// Show success feedback (#759)
var msgCount = document.querySelectorAll('#chMessages .ch-msg').length;
var userDisplay = channelName.startsWith('psk:') ? 'Custom channel (' + channelName.substring(4) + ')' : channelName;
if (msgCount > 0) {
showAddStatus('Added ' + userDisplay + ' — ' + msgCount + ' messages decrypted', 'success');
} else {
showAddStatus('No messages found for ' + userDisplay, 'warn');
}
} catch (err) {
showAddStatus('Failed to decrypt', 'error');
}
}
// Merge user-stored keys into the channel list
function mergeUserChannels() {
var keys = ChannelDecrypt.getStoredKeys();
var names = Object.keys(keys);
for (var i = 0; i < names.length; i++) {
var name = names[i];
// Check if channel already exists by name
var exists = channels.some(function (ch) {
return ch.name === name || ch.hash === name || ch.hash === ('user:' + name);
});
if (!exists) {
channels.push({
hash: 'user:' + name,
name: name,
messageCount: 0,
lastActivityMs: 0,
lastSender: '',
lastMessage: 'Encrypted — click to decrypt',
encrypted: true,
userAdded: true
});
}
}
}
// Fetch and decrypt GRP_TXT packets client-side (M5: delta fetch + cache)
async function fetchAndDecryptChannel(keyHex, channelHashByte, channelName, opts) {
opts = opts || {};
var keyBytes = ChannelDecrypt.hexToBytes(keyHex);
// M5: Check cache first — serve cached messages immediately
var cacheKey = channelName || String(channelHashByte);
var cached = ChannelDecrypt.getCache(cacheKey);
var cachedMsgs = cached ? cached.messages : [];
var lastTs = cached ? cached.lastTimestamp : '';
var cachedCount = cached ? (cached.count || 0) : 0;
// If we have cached messages and caller wants instant render, return them first
if (cachedMsgs.length > 0 && !opts.forceFullDecrypt) {
// Signal caller to render cache immediately, then do delta fetch
if (opts.onCacheHit) opts.onCacheHit(cachedMsgs);
}
// Fetch packets from API — get all payload_type=5 (GRP_TXT/CHAN)
var rp = RegionFilter.getRegionParam();
var qs = rp ? '&region=' + encodeURIComponent(rp) : '';
var data;
try {
data = await api('/packets?limit=1000&payloadType=5' + qs, { ttl: 10000 });
} catch (e) {
return { messages: cachedMsgs, error: 'Failed to fetch packets: ' + e.message, fromCache: cachedMsgs.length > 0 };
}
var packets = data.packets || [];
// Filter for GRP_TXT (encrypted) packets matching our channel hash byte
var candidates = [];
for (var i = 0; i < packets.length; i++) {
var p = packets[i];
var dj;
try { dj = typeof p.decoded_json === 'string' ? JSON.parse(p.decoded_json) : p.decoded_json; }
catch (e) { continue; }
if (!dj) continue;
if (dj.type === 'CHAN' && dj.channel === channelName) {
candidates.push({ type: 'already_decrypted', decoded: dj, packet: p });
} else if (dj.type === 'GRP_TXT' && dj.encryptedData && dj.mac) {
if (dj.channelHash === channelHashByte) {
candidates.push({ type: 'encrypted', decoded: dj, packet: p });
}
}
}
// M5: Cache invalidation — if total candidate count changed, re-decrypt everything
var totalCandidates = candidates.length;
var needFullDecrypt = (totalCandidates !== cachedCount) || opts.forceFullDecrypt;
// M5: Delta fetch — only decrypt packets newer than lastTs
if (!needFullDecrypt && cachedMsgs.length > 0 && lastTs) {
// Filter candidates to only those newer than cached lastTimestamp
var newCandidates = candidates.filter(function (c) {
var ts = c.packet.first_seen || c.packet.timestamp || '';
return ts > lastTs;
});
if (newCandidates.length === 0) {
// Nothing new — return cache as-is
return { messages: cachedMsgs, fromCache: true };
}
// Decrypt only new candidates
var newDecrypted = await decryptCandidates(keyBytes, newCandidates);
if (newDecrypted.wrongKey) {
return { messages: cachedMsgs, wrongKey: true };
}
// Merge: cached + new, deduplicate by packetHash, sort chronologically
var merged = deduplicateAndMerge(cachedMsgs, newDecrypted.messages);
var newLastTs = merged.length ? merged[merged.length - 1].timestamp : lastTs;
ChannelDecrypt.setCache(cacheKey, merged, newLastTs, totalCandidates);
return { messages: merged, deltaCount: newDecrypted.messages.length };
}
if (candidates.length === 0) {
return { messages: cachedMsgs, empty: true };
}
// Full decrypt
var result = await decryptCandidates(keyBytes, candidates);
if (result.wrongKey) {
return { messages: result.messages, wrongKey: true };
}
var decrypted = result.messages;
// Sort chronologically (oldest first)
decrypted.sort(function (a, b) {
var ta = a.timestamp || '';
var tb = b.timestamp || '';
return ta.localeCompare(tb);
});
// M5: Cache results
var newLastTimestamp = decrypted.length ? decrypted[decrypted.length - 1].timestamp : '';
ChannelDecrypt.setCache(cacheKey, decrypted, newLastTimestamp, totalCandidates);
return { messages: decrypted };
}
/** Decrypt an array of candidate packets. Returns { messages, wrongKey }. */
async function decryptCandidates(keyBytes, candidates) {
// Sort newest first for progressive rendering
candidates.sort(function (a, b) {
var ta = a.packet.first_seen || a.packet.timestamp || '';
var tb = b.packet.first_seen || b.packet.timestamp || '';
return tb.localeCompare(ta);
});
var decrypted = [];
var macFailCount = 0;
var macCheckCount = 0;
for (var j = 0; j < candidates.length; j++) {
var c = candidates[j];
if (c.type === 'already_decrypted') {
var d = c.decoded;
var sender = d.sender || 'Unknown';
var text = d.text || '';
var ci = text.indexOf(': ');
if (ci > 0 && ci < 50 && text.substring(0, ci) === sender) {
text = text.substring(ci + 2);
}
decrypted.push({
sender: sender, text: text,
timestamp: c.packet.first_seen || c.packet.timestamp,
sender_timestamp: d.sender_timestamp || null,
packetHash: c.packet.hash, packetId: c.packet.id,
hops: d.path_len || 0, snr: c.packet.snr || null,
observers: c.packet.observer_name ? [c.packet.observer_name] : [],
repeats: 1
});
continue;
}
macCheckCount++;
var result = await ChannelDecrypt.decryptPacket(keyBytes, c.decoded.mac, c.decoded.encryptedData);
if (result) {
macFailCount = 0;
decrypted.push({
sender: result.sender, text: result.message,
timestamp: c.packet.first_seen || c.packet.timestamp,
sender_timestamp: result.timestamp || null,
packetHash: c.packet.hash, packetId: c.packet.id,
hops: 0, snr: c.packet.snr || null,
observers: c.packet.observer_name ? [c.packet.observer_name] : [],
repeats: 1
});
} else {
macFailCount++;
if (macCheckCount >= 10 && macFailCount >= macCheckCount) {
return { messages: decrypted, wrongKey: true };
}
}
}
return { messages: decrypted, wrongKey: false };
}
/** Merge cached and new messages, deduplicate by packetHash, sort chronologically. */
function deduplicateAndMerge(cached, newMsgs) {
var seen = {};
var merged = [];
// Add cached first
for (var i = 0; i < cached.length; i++) {
var key = cached[i].packetHash || ('idx:' + i);
if (!seen[key]) { seen[key] = true; merged.push(cached[i]); }
}
// Add new
for (var j = 0; j < newMsgs.length; j++) {
var key2 = newMsgs[j].packetHash || ('new:' + j);
if (!seen[key2]) { seen[key2] = true; merged.push(newMsgs[j]); }
}
merged.sort(function (a, b) {
var ta = a.timestamp || '';
var tb = b.timestamp || '';
return ta.localeCompare(tb);
});
return merged;
}
function init(app, routeParam) {
var _initUrlParams = getHashParams();
var _pendingNode = _initUrlParams.get('node');
@@ -326,6 +610,21 @@
<div class="ch-sidebar" aria-label="Channel list">
<div class="ch-sidebar-header">
<div class="ch-sidebar-title"><span class="ch-icon">💬</span> Channels</div>
<label class="ch-encrypted-toggle" title="Show encrypted channels (no key configured)">
<input type="checkbox" id="chShowEncrypted"> <span class="ch-toggle-label">🔒 No key</span>
</label>
</div>
<div class="ch-key-input-wrap" style="padding:4px 8px">
<form id="chKeyForm" autocomplete="off" class="ch-add-form">
<div class="ch-add-row">
<input type="text" id="chKeyInput" class="ch-key-input"
placeholder="#channelname"
aria-label="Channel name or hex key" spellcheck="false">
<button type="submit" class="ch-add-btn" title="Add channel">+</button>
</div>
<div class="ch-add-hint">e.g. #LongFast or 32-char hex key decrypted in your browser.</div>
<div id="chAddStatus" class="ch-add-status" style="display:none"></div>
</form>
</div>
<div id="chRegionFilter" class="region-filter-container" style="padding:0 8px"></div>
<div class="ch-channel-list" id="chList" role="listbox" aria-label="Channels">
@@ -347,6 +646,17 @@
</div>`;
RegionFilter.init(document.getElementById('chRegionFilter'));
// Encrypted channels toggle (#727)
var showEncryptedCb = document.getElementById('chShowEncrypted');
var showEncrypted = localStorage.getItem('channels-show-encrypted') === 'true';
showEncryptedCb.checked = showEncrypted;
showEncryptedCb.addEventListener('change', function () {
showEncrypted = showEncryptedCb.checked;
localStorage.setItem('channels-show-encrypted', showEncrypted ? 'true' : 'false');
loadChannels(true);
});
regionChangeHandler = RegionFilter.onChange(function () {
loadChannels(true).then(async function () {
if (!selectedHash) return;
@@ -354,8 +664,38 @@
});
});
// Channel key input handler (#725 M2, improved UX #759)
var chKeyForm = document.getElementById('chKeyForm');
if (chKeyForm) {
var submitHandler = async function (e) {
e.preventDefault();
var input = document.getElementById('chKeyInput');
var val = (input.value || '').trim();
if (!val) return;
input.value = '';
await addUserChannel(val);
};
chKeyForm.addEventListener('submit', submitHandler);
var chKeyInput = document.getElementById('chKeyInput');
if (chKeyInput) {
chKeyInput.addEventListener('focus', function () {
var st = document.getElementById('chAddStatus');
if (st) { st.style.display = 'none'; clearTimeout(statusTimer); statusTimer = null; }
});
}
}
// Auto-enable encrypted toggle if deep-linking to an encrypted channel
if (routeParam && routeParam.startsWith('enc_') && !showEncrypted) {
showEncrypted = true;
showEncryptedCb.checked = true;
localStorage.setItem('channels-show-encrypted', 'true');
}
loadObserverRegions();
loadChannels().then(async function () {
// Also load user-added encrypted channels into the sidebar
mergeUserChannels();
if (routeParam) await selectChannel(routeParam);
if (_pendingNode && _pendingNode.length < 200) await showNodeDetail(_pendingNode);
});
@@ -403,6 +743,29 @@
// Event delegation for channel selection (touch-friendly)
document.getElementById('chList').addEventListener('click', (e) => {
// M4: Remove channel button
const removeBtn = e.target.closest('[data-remove-channel]');
if (removeBtn) {
e.stopPropagation();
var channelHash = removeBtn.getAttribute('data-remove-channel');
if (!channelHash) return;
var chName = channelHash.startsWith('user:') ? channelHash.substring(5) : channelHash;
if (!confirm('Remove channel "' + chName + '"? This will clear saved keys and cached messages.')) return;
ChannelDecrypt.removeKey(chName);
// Remove from channels array
channels = channels.filter(function (c) { return c.hash !== channelHash; });
if (selectedHash === channelHash) {
selectedHash = null;
messages = [];
history.replaceState(null, '', '#/channels');
var msgEl2 = document.getElementById('chMessages');
if (msgEl2) msgEl2.innerHTML = '<div class="ch-empty">Choose a channel from the sidebar to view messages</div>';
var header2 = document.getElementById('chHeader');
if (header2) header2.querySelector('.ch-header-text').textContent = 'Select a channel';
}
renderChannelList();
return;
}
// Color dot click — open picker, don't select channel
const dot = e.target.closest('.ch-color-dot');
if (dot && window.ChannelColorPicker) {
@@ -652,7 +1015,11 @@
async function loadChannels(silent) {
try {
const rp = RegionFilter.getRegionParam();
const qs = rp ? '?region=' + encodeURIComponent(rp) : '';
var showEnc = localStorage.getItem('channels-show-encrypted') === 'true';
var params = [];
if (rp) params.push('region=' + encodeURIComponent(rp));
if (showEnc) params.push('includeEncrypted=true');
const qs = params.length ? '?' + params.join('&') : '';
const data = await api('/channels' + qs, { ttl: CLIENT_TTL.channels });
channels = (data.channels || []).map(ch => {
ch.lastActivityMs = ch.lastActivity ? new Date(ch.lastActivity).getTime() : 0;
@@ -679,27 +1046,33 @@
});
el.innerHTML = sorted.map(ch => {
const name = ch.name || `Channel ${formatHashHex(ch.hash)}`;
const color = getChannelColor(ch.hash);
const isEncrypted = ch.encrypted === true;
const name = isEncrypted ? (ch.name || 'Unknown') : (ch.name || `Channel ${formatHashHex(ch.hash)}`);
const color = isEncrypted ? 'var(--text-muted, #6b7280)' : getChannelColor(ch.hash);
const time = ch.lastActivityMs ? formatSecondsAgo(Math.floor((Date.now() - ch.lastActivityMs) / 1000)) : '';
const preview = ch.lastSender && ch.lastMessage
? `${ch.lastSender}: ${truncate(ch.lastMessage, 28)}`
: `${ch.messageCount} messages`;
const preview = isEncrypted
? `${ch.messageCount} encrypted messages (no key configured)`
: ch.lastSender && ch.lastMessage
? `${ch.lastSender}: ${truncate(ch.lastMessage, 28)}`
: `${ch.messageCount} messages`;
const sel = selectedHash === ch.hash ? ' selected' : '';
const abbr = name.startsWith('#') ? name.slice(0, 3) : name.slice(0, 2).toUpperCase();
const encClass = isEncrypted ? ' ch-encrypted' : '';
const abbr = isEncrypted ? '🔒' : (name.startsWith('#') ? name.slice(0, 3) : name.slice(0, 2).toUpperCase());
// Channel color dot for color picker (#674)
const chColor = window.ChannelColors ? window.ChannelColors.get(ch.hash) : null;
const dotStyle = chColor ? ` style="background:${chColor}"` : '';
// Left border for assigned color
const borderStyle = chColor ? ` style="border-left:3px solid ${chColor}"` : '';
// M4: Remove button for user-added channels
const removeBtn = ch.userAdded ? ' <button class="ch-remove-btn" data-remove-channel="' + escapeHtml(ch.hash) + '" title="Remove channel" aria-label="Remove ' + escapeHtml(name) + '">✕</button>' : '';
return `<button class="ch-item${sel}" data-hash="${ch.hash}"${borderStyle} type="button" role="option" aria-selected="${selectedHash === ch.hash ? 'true' : 'false'}" aria-label="${escapeHtml(name)}">
<div class="ch-badge" style="background:${color}" aria-hidden="true">${escapeHtml(abbr)}</div>
return `<button class="ch-item${sel}${encClass}" data-hash="${ch.hash}"${borderStyle} type="button" role="option" aria-selected="${selectedHash === ch.hash ? 'true' : 'false'}" aria-label="${escapeHtml(name)}"${isEncrypted ? ' data-encrypted="true"' : ''}>
<div class="ch-badge" style="background:${color}" aria-hidden="true">${isEncrypted ? '🔒' : escapeHtml(abbr)}</div>
<div class="ch-item-body">
<div class="ch-item-top">
<span class="ch-item-name">${escapeHtml(name)}</span>
<span class="ch-color-dot" data-channel="${escapeHtml(ch.hash)}"${dotStyle} title="Change channel color" aria-label="Change color for ${escapeHtml(name)}"></span>
<span class="ch-item-time" data-channel-hash="${ch.hash}">${time}</span>
<span class="ch-item-time" data-channel-hash="${ch.hash}">${time}</span>${removeBtn}
</div>
<div class="ch-item-preview">${escapeHtml(preview)}</div>
</div>
@@ -707,7 +1080,7 @@
}).join('');
}
async function selectChannel(hash) {
async function selectChannel(hash, decryptOpts) {
const rp = RegionFilter.getRegionParam() || '';
const request = beginMessageRequest(hash, rp);
selectedHash = hash;
@@ -722,6 +1095,73 @@
document.querySelector('.ch-layout')?.classList.add('ch-show-main');
const msgEl = document.getElementById('chMessages');
// Shared helper: fetch, decrypt, and render messages for a channel key (M5: cache-first)
async function decryptAndRender(keyHex, channelHashByte, channelName) {
msgEl.innerHTML = '<div class="ch-loading">Decrypting messages…</div>';
var result = await fetchAndDecryptChannel(keyHex, channelHashByte, channelName, {
onCacheHit: function (cachedMsgs) {
// M5: Render cached messages immediately while delta fetch runs
messages = cachedMsgs;
if (messages.length > 0) {
header.querySelector('.ch-header-text').textContent = name + ' — ' + messages.length + ' messages (cached)';
renderMessages();
scrollToBottom();
}
}
});
if (isStaleMessageRequest(request)) return true;
if (result.wrongKey) {
msgEl.innerHTML = '<div class="ch-empty ch-wrong-key">🔒 Key does not match — no messages could be decrypted</div>';
return true;
}
if (result.error) {
msgEl.innerHTML = '<div class="ch-empty">' + escapeHtml(result.error) + '</div>';
return true;
}
messages = result.messages || [];
if (messages.length === 0) {
msgEl.innerHTML = '<div class="ch-empty">No encrypted messages found for this channel</div>';
} else {
header.querySelector('.ch-header-text').textContent = `${name}${messages.length} messages (decrypted)`;
renderMessages();
scrollToBottom();
}
return true;
}
// Client-side decryption path (#725 M2)
if (decryptOpts && decryptOpts.userKey) {
await decryptAndRender(decryptOpts.userKey, decryptOpts.channelHashByte, decryptOpts.channelName);
return;
}
// Check if this is a user-added channel that needs decryption
var storedKeys = typeof ChannelDecrypt !== 'undefined' ? ChannelDecrypt.getStoredKeys() : {};
if (hash.startsWith('user:')) {
var chName = hash.substring(5);
if (storedKeys[chName]) {
var keyHex = storedKeys[chName];
var keyBytes = ChannelDecrypt.hexToBytes(keyHex);
var hashByte = await ChannelDecrypt.computeChannelHash(keyBytes);
await decryptAndRender(keyHex, hashByte, chName);
return;
}
}
// Also check if an encrypted channel hash matches a stored key
if (ch && ch.encrypted) {
for (var kn in storedKeys) {
var kh = storedKeys[kn];
var kb = ChannelDecrypt.hexToBytes(kh);
var hb = await ChannelDecrypt.computeChannelHash(kb);
if (String(hb) === String(hash) || String(ch.hash) === String(hb)) {
await decryptAndRender(kh, hb, kn);
return;
}
}
}
msgEl.innerHTML = '<div class="ch-loading">Loading messages…</div>';
try {
@@ -743,6 +1183,9 @@
async function refreshMessages(opts) {
if (!selectedHash) return;
// Skip refresh for encrypted channels — no messages to fetch
var selCh = channels.find(function (c) { return c.hash === selectedHash; });
if (selCh && selCh.encrypted) return;
opts = opts || {};
const msgEl = document.getElementById('chMessages');
if (!msgEl) return;
+4
View File
@@ -1173,6 +1173,10 @@
'<details style="margin-top:12px"><summary style="font-size:12px;font-weight:600;cursor:pointer;color:var(--text-muted)">Raw JSON</summary>' +
'<textarea id="cv2ExportJson" style="width:100%;min-height:200px;font-family:var(--mono);font-size:12px;background:var(--surface-1);border:1px solid var(--border);border-radius:6px;padding:12px;color:var(--text);resize:vertical;box-sizing:border-box;margin-top:8px">' + esc(json) + '</textarea>' +
'</details>' +
'<p class="cust-section-title" style="margin-top:20px">Tools</p>' +
'<p style="font-size:12px;color:var(--text-muted);margin-bottom:10px">Server-side configuration helpers.</p>' +
'<a href="/geofilter-builder.html" target="_blank" style="display:inline-block;padding:7px 14px;background:var(--surface-1);border:1px solid var(--border);border-radius:6px;color:var(--accent);font-size:13px;text-decoration:none;font-weight:500">🗺️ GeoFilter Builder →</a>' +
'<p style="font-size:11px;color:var(--text-muted);margin-top:6px">Draw a polygon on the map to generate a <code style="font-family:var(--mono)">geo_filter</code> block for <code style="font-family:var(--mono)">config.json</code>.</p>' +
'</div>';
}
+170
View File
@@ -0,0 +1,170 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>GeoFilter Builder — CoreScope</title>
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css"/>
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
<style>
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: system-ui, sans-serif; background: #1a1a2e; color: #e0e0e0; height: 100vh; display: flex; flex-direction: column; }
header { padding: 12px 16px; background: #0f0f23; border-bottom: 1px solid #333; display: flex; align-items: center; gap: 16px; flex-wrap: wrap; }
header h1 { font-size: 1rem; font-weight: 600; color: #4a9eff; white-space: nowrap; }
.controls { display: flex; gap: 8px; flex-wrap: wrap; }
button { padding: 6px 14px; border: none; border-radius: 6px; cursor: pointer; font-size: 0.85rem; font-weight: 500; }
#btnUndo { background: #333; color: #ccc; }
#btnClear { background: #5a2020; color: #ffaaaa; }
#btnUndo:hover { background: #444; }
#btnClear:hover { background: #7a2020; }
.hint { font-size: 0.8rem; color: #888; margin-left: auto; }
#map { flex: 1; }
#output-panel { background: #0f0f23; border-top: 1px solid #333; padding: 12px 16px; display: flex; gap: 12px; align-items: flex-start; }
#output-panel label { font-size: 0.75rem; color: #888; white-space: nowrap; padding-top: 6px; }
#output { flex: 1; background: #111; border: 1px solid #333; border-radius: 6px; padding: 10px 12px; font-family: monospace; font-size: 0.78rem; color: #7ec8e3; white-space: pre; overflow-x: auto; min-height: 54px; max-height: 140px; overflow-y: auto; cursor: text; }
#output.empty { color: #555; font-style: italic; }
#btnCopy { padding: 6px 14px; background: #1a4a7a; color: #7ec8e3; border-radius: 6px; border: none; cursor: pointer; font-size: 0.85rem; white-space: nowrap; align-self: flex-end; }
#btnCopy:hover { background: #2a6aaa; }
#btnCopy.copied { background: #1a6a3a; color: #7effa0; }
#counter { font-size: 0.8rem; color: #888; padding-top: 6px; white-space: nowrap; }
.bufferRow { display: flex; align-items: center; gap: 8px; }
.bufferRow label { font-size: 0.85rem; color: #aaa; }
.bufferRow input { width: 60px; padding: 5px 8px; background: #222; border: 1px solid #444; border-radius: 6px; color: #eee; font-size: 0.85rem; }
#help-bar { background: #0f0f23; padding: 6px 16px; font-size: 0.75rem; color: #666; border-top: 1px solid #222; }
#help-bar a { color: #4a9eff; text-decoration: none; }
#help-bar a:hover { text-decoration: underline; }
#back-link { font-size: 0.8rem; color: #4a9eff; text-decoration: none; white-space: nowrap; }
#back-link:hover { text-decoration: underline; }
</style>
</head>
<body>
<header>
<a href="/" id="back-link">← CoreScope</a>
<h1>GeoFilter Builder</h1>
<div class="controls">
<button id="btnUndo">↩ Undo</button>
<button id="btnClear">✕ Clear</button>
</div>
<div class="bufferRow">
<label for="bufferKm">Buffer km:</label>
<!-- Extra margin (km) outside the polygon edge that still passes the filter -->
<input type="number" id="bufferKm" value="20" min="0" max="500"/>
</div>
<span class="hint">Click on the map to add polygon points</span>
</header>
<div id="map"></div>
<!-- Output panel: shows the geo_filter JSON block ready to paste into config.json -->
<div id="output-panel">
<label>config.json</label>
<div id="output" class="empty">Add at least 3 points to generate config…</div>
<div style="display:flex;flex-direction:column;gap:8px;align-items:flex-end">
<span id="counter">0 points</span>
<button id="btnCopy">Copy</button>
</div>
</div>
<!-- Instructions: paste the output into config.json as a top-level "geo_filter" key, then restart the server -->
<div id="help-bar">
Copy the JSON above → paste as a top-level key in <code>config.json</code> → restart the server.
Nodes with no GPS fix always pass through. Remove the <code>geo_filter</code> block to disable filtering.
&nbsp;·&nbsp; <a href="https://github.com/Kpa-clawbot/CoreScope/blob/master/docs/user-guide/geofilter.md" target="_blank">Documentation ↗</a>
</div>
<script>
const map = L.map('map').setView([50.5, 4.4], 8);
L.tileLayer('https://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}{r}.png', {
attribution: '© OpenStreetMap © CartoDB',
maxZoom: 19
}).addTo(map);
let points = [];
let markers = [];
let polygon = null;
let closingLine = null;
function latLonPair(latlng) {
return [parseFloat(latlng.lat.toFixed(6)), parseFloat(latlng.lng.toFixed(6))];
}
function render() {
// Remove existing polygon and closing line
if (polygon) { map.removeLayer(polygon); polygon = null; }
if (closingLine) { map.removeLayer(closingLine); closingLine = null; }
if (points.length >= 3) {
polygon = L.polygon(points, {
color: '#4a9eff', weight: 2, fillColor: '#4a9eff', fillOpacity: 0.12
}).addTo(map);
} else if (points.length === 2) {
closingLine = L.polyline(points, { color: '#4a9eff', weight: 2, dashArray: '5,5' }).addTo(map);
}
updateOutput();
}
function updateOutput() {
const el = document.getElementById('output');
const counter = document.getElementById('counter');
counter.textContent = points.length + ' point' + (points.length !== 1 ? 's' : '');
if (points.length < 3) {
el.textContent = 'Add at least 3 points to generate config…';
el.classList.add('empty');
return;
}
el.classList.remove('empty');
const bufferKm = parseFloat(document.getElementById('bufferKm').value) || 0;
// Output format: { "geo_filter": { "bufferKm": N, "polygon": [[lat,lon], ...] } }
// Paste this as a top-level key in config.json
const config = { bufferKm, polygon: points };
el.textContent = JSON.stringify({ geo_filter: config }, null, 2);
}
map.on('click', function(e) {
const pt = latLonPair(e.latlng);
points.push(pt);
const idx = points.length;
const marker = L.circleMarker(e.latlng, {
radius: 6, color: '#4a9eff', weight: 2, fillColor: '#4a9eff', fillOpacity: 0.9
}).addTo(map).bindTooltip(String(idx), { permanent: true, direction: 'top', offset: [0, -8], className: 'pt-label' });
markers.push(marker);
render();
});
document.getElementById('btnUndo').addEventListener('click', function() {
if (!points.length) return;
points.pop();
const m = markers.pop();
if (m) map.removeLayer(m);
render();
});
document.getElementById('btnClear').addEventListener('click', function() {
points = [];
markers.forEach(m => map.removeLayer(m));
markers = [];
render();
});
document.getElementById('bufferKm').addEventListener('input', updateOutput);
document.getElementById('btnCopy').addEventListener('click', function() {
if (points.length < 3) return;
const text = document.getElementById('output').textContent;
navigator.clipboard.writeText(text).then(() => {
const btn = document.getElementById('btnCopy');
btn.textContent = 'Copied!';
btn.classList.add('copied');
setTimeout(() => { btn.textContent = 'Copy'; btn.classList.remove('copied'); }, 2000);
});
});
</script>
</body>
</html>
+1
View File
@@ -95,6 +95,7 @@
<script src="table-sort.js?v=__BUST__"></script>
<script src="packet-filter.js?v=__BUST__"></script>
<script src="packet-helpers.js?v=__BUST__"></script>
<script src="channel-decrypt.js?v=__BUST__"></script>
<script src="channel-colors.js?v=__BUST__"></script>
<script src="channel-color-picker.js?v=__BUST__"></script>
<script src="packets.js?v=__BUST__"></script>
+2 -1
View File
@@ -2730,6 +2730,7 @@
const preview = text ? ' ' + (text.length > 35 ? text.slice(0, 35) + '…' : text) : '';
const hopStr = hops.length ? `<span class="feed-hops">${hops.length}⇢</span>` : '';
const obsBadge = pkt.observation_count > 1 ? `<span class="badge badge-obs" style="font-size:10px;margin-left:4px">👁 ${pkt.observation_count}</span>` : '';
const anomalyIcon = (pkt.decoded && pkt.decoded.anomaly) ? '<span title="Anomaly detected" style="margin-left:4px">⚠️</span>' : '';
var _ccPayload2 = (pkt.decoded || {}).payload || {};
var _ccChan = (typeName === 'GRP_TXT' || typeName === 'CHAN') ? (_ccPayload2.channel || null) : null;
var dotHtml = _ccChan ? _feedColorDot(_ccChan) : '';
@@ -2744,7 +2745,7 @@
item.innerHTML = `
<span class="feed-icon" style="color:${color}">${icon}</span>
<span class="feed-type" style="color:${color}">${typeName}</span>
${dotHtml}${transportBadge(pkt.route_type)}${hopStr}${obsBadge}
${dotHtml}${transportBadge(pkt.route_type)}${hopStr}${obsBadge}${anomalyIcon}
<span class="feed-text">${escapeHtml(preview)}</span>
<span class="feed-time" data-ts="${pkt._ts || Date.now()}">${formatLiveTimestampHtml(pkt._ts || Date.now())}</span>
`;
+43 -9
View File
@@ -25,7 +25,7 @@
// Roles loaded from shared roles.js (ROLE_STYLE, ROLE_LABELS, ROLE_COLORS globals)
function makeMarkerIcon(role, isStale) {
function makeMarkerIcon(role, isStale, isAlsoObserver) {
const s = ROLE_STYLE[role] || ROLE_STYLE.companion;
const size = s.radius * 2 + 4;
const c = size / 2;
@@ -56,7 +56,22 @@
default: // circle
path = `<circle cx="${c}" cy="${c}" r="${c-2}" fill="${s.color}" stroke="#fff" stroke-width="2"/>`;
}
const svg = `<svg width="${size}" height="${size}" viewBox="0 0 ${size} ${size}" xmlns="http://www.w3.org/2000/svg">${path}</svg>`;
// If this node is also an observer, add a small star overlay
let obsOverlay = '';
if (isAlsoObserver) {
const starSize = 8;
const sx = size - starSize, sy = 0;
const scx = starSize / 2, scy = starSize / 2, so = starSize / 2 - 0.5, si = so * 0.4;
let starPts = '';
for (let i = 0; i < 5; i++) {
const aO = (i * 72 - 90) * Math.PI / 180;
const aI = ((i * 72) + 36 - 90) * Math.PI / 180;
starPts += `${scx + so * Math.cos(aO)},${scy + so * Math.sin(aO)} `;
starPts += `${scx + si * Math.cos(aI)},${scy + si * Math.sin(aI)} `;
}
obsOverlay = `<g transform="translate(${sx},${sy})"><polygon points="${starPts.trim()}" fill="${ROLE_COLORS.observer || '#f1c40f'}" stroke="#fff" stroke-width="0.8"/></g>`;
}
const svg = `<svg width="${size}" height="${size}" viewBox="0 0 ${size} ${size}" xmlns="http://www.w3.org/2000/svg">${path}${obsOverlay}</svg>`;
return L.divIcon({
html: svg,
className: 'meshcore-marker' + (isStale ? ' marker-stale' : ''),
@@ -66,14 +81,16 @@
});
}
function makeRepeaterLabelIcon(node, isStale) {
function makeRepeaterLabelIcon(node, isStale, isAlsoObserver) {
var s = ROLE_STYLE['repeater'] || ROLE_STYLE.companion;
var hs = node.hash_size || 1;
// Show the short mesh hash ID (first N bytes of pubkey, uppercased)
var shortHash = node.public_key ? node.public_key.slice(0, hs * 2).toUpperCase() : '??';
var bgColor = s.color;
// If this repeater is also an observer, show a star indicator inside the label
var obsIndicator = isAlsoObserver ? ' <span style="color:' + (ROLE_COLORS.observer || '#f1c40f') + ';font-size:13px;line-height:1;" title="Also an observer">★</span>' : '';
var html = '<div style="background:' + bgColor + ';color:#fff;font-weight:bold;font-size:11px;padding:2px 5px;border-radius:3px;border:2px solid #fff;box-shadow:0 1px 3px rgba(0,0,0,0.4);text-align:center;line-height:1.2;white-space:nowrap;">' +
shortHash + '</div>';
shortHash + obsIndicator + '</div>';
return L.divIcon({
html: html,
className: 'meshcore-marker meshcore-label-marker' + (isStale ? ' marker-stale' : ''),
@@ -547,7 +564,8 @@
const el = document.getElementById('mcRoleChecks');
if (!el) return;
el.innerHTML = '';
const obsCount = observers.filter(o => o.lat && o.lon).length;
const nodePubkeys = new Set(nodes.map(n => (n.public_key || '').toLowerCase()));
const obsCount = observers.filter(o => o.lat && o.lon && !(o.id && nodePubkeys.has(o.id.toLowerCase()))).length;
const roles = ['repeater', 'companion', 'room', 'sensor', 'observer'];
const shapeMap = { repeater: '◆', companion: '●', room: '■', sensor: '▲', observer: '★' };
@@ -638,6 +656,7 @@
var _renderingMarkers = false;
var _lastDeconflictZoom = null;
var _currentMarkerData = []; // stored marker data for zoom-only repositioning
var _observerByPubkey = new Map(); // observer id (pubkey) → observer object, rebuilt on each render
var _zoomResizeTimer = null;
function deconflictLabels(markers, mapRef) {
@@ -780,19 +799,31 @@
const allMarkers = [];
// Build a set of observer public keys for quick lookup
_observerByPubkey = new Map();
for (const obs of observers) {
if (obs.id) _observerByPubkey.set(obs.id.toLowerCase(), obs);
}
for (const node of filtered) {
const lastSeenTime = node.last_heard || node.last_seen;
const isStale = getNodeStatus(node.role || 'companion', lastSeenTime ? new Date(lastSeenTime).getTime() : 0) === 'stale';
const pk = (node.public_key || '').toLowerCase();
const isAlsoObserver = _observerByPubkey.has(pk);
const useLabel = node.role === 'repeater' && filters.hashLabels;
const icon = useLabel ? makeRepeaterLabelIcon(node, isStale) : makeMarkerIcon(node.role || 'companion', isStale);
const icon = useLabel ? makeRepeaterLabelIcon(node, isStale, isAlsoObserver) : makeMarkerIcon(node.role || 'companion', isStale, isAlsoObserver);
const latLng = L.latLng(node.lat, node.lon);
allMarkers.push({ latLng, node, icon, isLabel: useLabel, popupFn: function() { return buildPopup(node); }, alt: (node.name || 'Unknown') + ' (' + (node.role || 'node') + ')' });
allMarkers.push({ latLng, node, icon, isLabel: useLabel, popupFn: function() { return buildPopup(node); }, alt: (node.name || 'Unknown') + ' (' + (node.role || 'node') + (isAlsoObserver ? ' + observer' : '') + ')' });
}
// Add observer markers
// Add observer markers (skip observers already represented as a node marker)
// Build set of node pubkeys that are displayed on the map
const displayedNodePubkeys = new Set(filtered.map(n => (n.public_key || '').toLowerCase()));
if (filters.observer) {
for (const obs of observers) {
if (!obs.lat || !obs.lon) continue;
// Skip observers whose pubkey matches a displayed node — they're shown as combined markers
if (obs.id && displayedNodePubkeys.has(obs.id.toLowerCase())) continue;
const icon = makeMarkerIcon('observer');
const latLng = L.latLng(obs.lat, obs.lon);
allMarkers.push({ latLng, node: obs, icon, isLabel: false, popupFn: function() { return buildObserverPopup(obs); }, alt: (obs.name || obs.id || 'Unknown') + ' (observer)' });
@@ -909,6 +940,9 @@
const loc = (node.lat && node.lon) ? `${node.lat.toFixed(5)}, ${node.lon.toFixed(5)}` : '—';
const lastAdvert = node.last_seen ? timeAgo(node.last_seen) : '—';
const roleBadge = `<span style="display:inline-block;padding:2px 8px;border-radius:12px;font-size:11px;font-weight:600;background:${ROLE_COLORS[node.role] || '#4b5563'};color:#fff;">${(node.role || 'unknown').toUpperCase()}</span>`;
// Check if this node is also an observer (combined repeater+observer)
const matchingObs = node.public_key ? _observerByPubkey.get(node.public_key.toLowerCase()) : null;
const obsBadge = matchingObs ? ` <span style="display:inline-block;padding:2px 8px;border-radius:12px;font-size:11px;font-weight:600;background:${ROLE_COLORS.observer || '#f1c40f'};color:#fff;">OBSERVER</span>` : '';
const hs = node.hash_size || 1;
const hashPrefix = node.public_key ? node.public_key.slice(0, hs * 2).toUpperCase() : '—';
const hashPrefixRow = `<dt style="color:var(--text-muted);float:left;clear:left;width:80px;padding:2px 0;">Hash Prefix</dt>
@@ -917,7 +951,7 @@
return `
<div class="map-popup" style="font-family:var(--font);min-width:180px;">
<h3 style="font-weight:700;font-size:14px;margin:0 0 4px;">${safeEsc(node.name || 'Unknown')}</h3>
${roleBadge}
${roleBadge}${obsBadge}
<dl style="margin-top:8px;font-size:12px;">
${hashPrefixRow}
<dt style="color:var(--text-muted);float:left;clear:left;width:80px;padding:2px 0;">Key</dt>
+104 -23
View File
@@ -315,29 +315,34 @@
let regionChangeHandler = null;
// Show full-screen node detail view (works on any screen size)
function showFullScreenNode(pubkey) {
var app = document.getElementById('app');
app.innerHTML = '<div class="node-fullscreen">' +
'<div class="node-full-header">' +
'<button class="detail-back-btn node-back-btn" id="nodeBackBtn" aria-label="Back to nodes">←</button>' +
'<span class="node-full-title">Loading…</span>' +
'</div>' +
'<div class="node-full-body" id="nodeFullBody">' +
'<div class="text-center text-muted" style="padding:40px">Loading…</div>' +
'</div>' +
'</div>';
document.getElementById('nodeBackBtn').addEventListener('click', function() { location.hash = '#/nodes'; });
loadFullNode(pubkey);
document.addEventListener('keydown', function nodesEsc(e) {
if (e.key === 'Escape') {
document.removeEventListener('keydown', nodesEsc);
location.hash = '#/nodes';
}
});
}
function init(app, routeParam) {
directNode = routeParam || null;
if (directNode) {
// Full-screen single node view
app.innerHTML = `<div class="node-fullscreen">
<div class="node-full-header">
<button class="detail-back-btn node-back-btn" id="nodeBackBtn" aria-label="Back to nodes"></button>
<span class="node-full-title">Loading</span>
</div>
<div class="node-full-body" id="nodeFullBody">
<div class="text-center text-muted" style="padding:40px">Loading</div>
</div>
</div>`;
document.getElementById('nodeBackBtn').addEventListener('click', () => { location.hash = '#/nodes'; });
loadFullNode(directNode);
// Escape to go back to nodes list
document.addEventListener('keydown', function nodesEsc(e) {
if (e.key === 'Escape') {
document.removeEventListener('keydown', nodesEsc);
location.hash = '#/nodes';
}
});
if (directNode && window.innerWidth <= 640) {
// Full-screen single node view (mobile)
showFullScreenNode(directNode);
return;
}
@@ -363,7 +368,7 @@
</div>`;
RegionFilter.init(document.getElementById('nodesRegionFilter'));
regionChangeHandler = RegionFilter.onChange(function () { _allNodes = null; loadNodes(); });
regionChangeHandler = RegionFilter.onChange(function () { _allNodes = null; _fleetSkew = null; loadNodes(); });
if (search) {
var _si = document.getElementById('nodeSearch');
@@ -377,6 +382,7 @@
}, 250));
loadNodes();
if (directNode) selectNode(directNode);
// Auto-refresh when ADVERT packets arrive via WebSocket (fixes #131)
wsHandler = debouncedOnWS(function (msgs) {
const advertMsgs = msgs.filter(isAdvertMessage);
@@ -409,6 +415,7 @@
if (needReload) {
_allNodes = null;
_fleetSkew = null;
invalidateApiCache('/nodes');
}
loadNodes(true);
@@ -493,6 +500,8 @@
<tr><td>Hash Prefix</td><td>${n.hash_size ? '<code style="font-family:var(--mono);font-weight:700">' + n.public_key.slice(0, n.hash_size * 2).toUpperCase() + '</code> (' + n.hash_size + '-byte)' : 'Unknown'}${n.hash_size_inconsistent ? ' <span style="color:var(--status-yellow);cursor:help" title="Seen: ' + (Array.isArray(n.hash_sizes_seen) ? n.hash_sizes_seen : []).join(', ') + '-byte"> varies</span>' : ''}</td></tr>
</table>
<div class="node-full-card skew-detail-section" id="node-clock-skew" style="display:none"></div>
${observers.length ? `<div class="node-full-card" id="node-observers">
${(() => { const regions = [...new Set(observers.map(o => o.iata).filter(Boolean))]; return regions.length ? `<div style="margin-bottom:8px"><strong>Regions:</strong> ${regions.map(r => '<span class="badge" style="margin:0 2px">' + escapeHtml(r) + '</span>').join(' ')}</div>` : ''; })()}
<h4>Heard By (${observers.length} observer${observers.length > 1 ? 's' : ''})</h4>
@@ -624,6 +633,35 @@
headerSelector: '#fullNeighborsHeader'
});
// #690 — Clock Skew detail section
(async function loadClockSkew() {
var container = document.getElementById('node-clock-skew');
if (!container) return;
try {
var cs = await api('/nodes/' + encodeURIComponent(n.public_key) + '/clock-skew', { ttl: 30000 });
if (!cs || !cs.severity) return;
container.style.display = '';
var severityColor = SKEW_SEVERITY_COLORS[cs.severity] || 'var(--text-muted)';
var severityLabel = SKEW_SEVERITY_LABELS[cs.severity] || cs.severity;
var driftHtml = cs.driftPerDaySec ? '<div style="font-size:12px;color:var(--text-muted);margin-top:2px">Drift: ' + formatDrift(cs.driftPerDaySec) + '</div>' : '';
var sparkHtml = renderSkewSparkline(cs.samples, 200, 32);
var skewDisplay = cs.severity === 'no_clock'
? '<span style="font-size:18px;font-weight:700;color:var(--text-muted)">No Clock</span>'
: '<span style="font-size:18px;font-weight:700;font-family:var(--mono)">' + formatSkew(cs.medianSkewSec) + '</span>';
container.innerHTML =
'<h4 style="margin:0 0 6px">⏰ Clock Skew</h4>' +
'<div style="display:flex;align-items:center;gap:12px;flex-wrap:wrap">' +
skewDisplay +
renderSkewBadge(cs.severity, cs.medianSkewSec) +
(cs.calibrated ? ' <span style="font-size:10px;color:var(--text-muted)" title="Observer-calibrated">✓ calibrated</span>' : '') +
'</div>' +
driftHtml +
(sparkHtml ? '<div class="skew-sparkline-wrap" style="margin-top:8px">' + sparkHtml + '<div style="font-size:10px;color:var(--text-muted)">Skew over time (' + (cs.samples || []).length + ' samples)</div></div>' : '');
} catch (e) {
// Non-fatal — section stays hidden
}
})();
// Affinity debug panel — show if debugAffinity is enabled
(function loadAffinityDebug() {
var show = (window.CLIENT_CONFIG && window.CLIENT_CONFIG.debugAffinity) || localStorage.getItem('meshcore-affinity-debug') === 'true';
@@ -777,6 +815,22 @@
let _themeRefreshHandler = null;
let _allNodes = null; // cached full node list
let _fleetSkew = null; // cached clock skew map: pubkey → {severity, medianSkewSec, ...}
/** Fetch fleet clock skew once, return map keyed by pubkey */
async function getFleetSkew() {
if (_fleetSkew) return _fleetSkew;
try {
const data = await api('/nodes/clock-skew', { ttl: 30000 });
_fleetSkew = {};
(Array.isArray(data) ? data : []).forEach(function(cs) {
if (cs && cs.pubkey) _fleetSkew[cs.pubkey] = cs;
});
} catch (e) {
_fleetSkew = {};
}
return _fleetSkew;
}
// Build a map of lowercased name → count of distinct pubkeys sharing that name
function buildDupNameMap(allNodes) {
@@ -806,7 +860,10 @@
const params = new URLSearchParams({ limit: '5000' });
const rp = RegionFilter.getRegionParam();
if (rp) params.set('region', rp);
const data = await api('/nodes?' + params, { ttl: CLIENT_TTL.nodeList });
const [data] = await Promise.all([
api('/nodes?' + params, { ttl: CLIENT_TTL.nodeList }),
getFleetSkew() // pre-fetch clock skew in parallel
]);
_allNodes = data.nodes || [];
counts = data.counts || {};
}
@@ -979,6 +1036,7 @@
panel.classList.add('empty');
panel.innerHTML = '<span>Select a node to view details</span>';
selectedKey = null;
history.replaceState(null, '', '#/nodes');
renderRows();
}
}
@@ -986,11 +1044,31 @@
// #630: Close button for node detail panel (important for mobile full-screen overlay)
document.getElementById('nodesRight').addEventListener('click', function(e) {
// #778: Details/Analytics links don't navigate because replaceState
// already set the hash to #/nodes/PUBKEY, so clicking <a href="#/nodes/PUBKEY">
// is a same-hash no-op. Force navigation by temporarily clearing the hash.
var link = e.target.closest('a.btn-primary[href^="#/nodes/"]');
if (link) {
e.preventDefault();
var href = link.getAttribute('href');
var pubkey = decodeURIComponent(href.replace('#/nodes/', '').replace('/analytics', ''));
if (href.includes('/analytics')) {
// Navigate to analytics page
history.replaceState(null, '', '#/');
location.hash = '/nodes/' + encodeURIComponent(pubkey) + '/analytics';
} else {
// Show full-screen node detail view
showFullScreenNode(pubkey);
history.replaceState(null, '', '#/nodes/' + encodeURIComponent(pubkey));
}
return;
}
if (e.target.closest('.panel-close-btn')) {
const panel = document.getElementById('nodesRight');
panel.classList.add('empty');
panel.innerHTML = '<span>Select a node to view details</span>';
selectedKey = null;
history.replaceState(null, '', '#/nodes');
renderRows();
}
});
@@ -1029,8 +1107,10 @@
const lastSeenTime = n.last_heard || n.last_seen;
const status = getNodeStatus(n.role || 'companion', lastSeenTime ? new Date(lastSeenTime).getTime() : 0);
const lastSeenClass = status === 'active' ? 'last-seen-active' : 'last-seen-stale';
const cs = _fleetSkew && _fleetSkew[n.public_key];
const skewBadgeHtml = cs && cs.severity && cs.severity !== 'ok' ? renderSkewBadge(cs.severity, cs.medianSkewSec) : '';
return `<tr data-key="${n.public_key}" data-action="select" data-value="${n.public_key}" tabindex="0" role="row" class="${selectedKey === n.public_key ? 'selected' : ''}${isClaimed ? ' claimed-row' : ''}">
<td>${favStar(n.public_key, 'node-fav')}${isClaimed ? '<span class="claimed-badge" title="My Mesh">★</span> ' : ''}<strong>${n.name || '(unnamed)'}</strong>${dupNameBadge(n.name, n.public_key, dupMap)}</td>
<td>${favStar(n.public_key, 'node-fav')}${isClaimed ? '<span class="claimed-badge" title="My Mesh">★</span> ' : ''}<strong>${n.name || '(unnamed)'}</strong>${dupNameBadge(n.name, n.public_key, dupMap)}${skewBadgeHtml}</td>
<td class="mono col-pubkey">${truncate(n.public_key, 16)}</td>
<td><span class="badge" style="background:${roleColor}20;color:${roleColor}">${n.role}</span></td>
<td class="${lastSeenClass}">${renderNodeTimestampHtml(n.last_heard || n.last_seen)}</td>
@@ -1048,6 +1128,7 @@
return;
}
selectedKey = pubkey;
history.replaceState(null, '', '#/nodes/' + encodeURIComponent(pubkey));
renderRows();
const panel = document.getElementById('nodesRight');
panel.classList.remove('empty');
+54 -11
View File
@@ -45,6 +45,10 @@
var parts = [];
if (timeWindowMin && timeWindowMin !== DEFAULT_TIME_WINDOW) parts.push('timeWindow=' + timeWindowMin);
if (regionParam) parts.push('region=' + encodeURIComponent(regionParam));
if (filters.hash) parts.push('hash=' + encodeURIComponent(filters.hash));
if (filters.node) parts.push('node=' + encodeURIComponent(filters.node));
if (filters.observer) parts.push('observer=' + encodeURIComponent(filters.observer));
if (filters._filterExpr) parts.push('filter=' + encodeURIComponent(filters._filterExpr));
return parts.length ? '?' + parts.join('&') : '';
}
window.buildPacketsQuery = buildPacketsQuery;
@@ -342,6 +346,14 @@
}
var _urlRegion = _initUrlParams.get('region');
if (_urlRegion) _pendingUrlRegion = _urlRegion;
var _urlHash = _initUrlParams.get('hash');
if (_urlHash) filters.hash = _urlHash;
var _urlNode = _initUrlParams.get('node');
if (_urlNode) { filters.node = _urlNode; filters.nodeName = _urlNode.slice(0, 8); }
var _urlObserver = _initUrlParams.get('observer');
if (_urlObserver) filters.observer = _urlObserver;
var _urlFilterExpr = _initUrlParams.get('filter');
if (_urlFilterExpr) filters._filterExpr = _urlFilterExpr;
app.innerHTML = `<div class="split-layout detail-collapsed">
<div class="panel-left" id="pktLeft" aria-live="polite" aria-relevant="additions removals"></div>
@@ -797,6 +809,12 @@
var pfError = document.getElementById('packetFilterError');
var pfCount = document.getElementById('packetFilterCount');
if (!pfInput || !window.PacketFilter) return;
// Restore Wireshark filter expression from URL
if (filters._filterExpr) {
pfInput.value = filters._filterExpr;
var _restored = PacketFilter.compile(filters._filterExpr);
if (!_restored.error) { pfInput.classList.add('filter-active'); filters._packetFilter = _restored.filter; }
}
var pfTimer = null;
pfInput.addEventListener('input', function() {
clearTimeout(pfTimer);
@@ -807,6 +825,8 @@
pfError.style.display = 'none';
pfCount.style.display = 'none';
filters._packetFilter = null;
filters._filterExpr = undefined;
updatePacketsUrl();
renderTableRows();
return;
}
@@ -818,12 +838,16 @@
pfError.style.display = 'block';
pfCount.style.display = 'none';
filters._packetFilter = null;
filters._filterExpr = undefined;
updatePacketsUrl();
renderTableRows();
} else {
pfInput.classList.remove('filter-error');
pfInput.classList.add('filter-active');
pfError.style.display = 'none';
filters._packetFilter = compiled.filter;
filters._filterExpr = expr;
updatePacketsUrl();
renderTableRows();
}
}, 300);
@@ -868,6 +892,7 @@
if (filters.observer) localStorage.setItem('meshcore-observer-filter', filters.observer); else localStorage.removeItem('meshcore-observer-filter');
buildObserverMenu();
updateObsTrigger();
updatePacketsUrl();
renderTableRows();
});
@@ -930,7 +955,7 @@
// Filter event listeners
document.getElementById('fHash').value = filters.hash || '';
document.getElementById('fHash').addEventListener('input', debounce((e) => { filters.hash = e.target.value || undefined; loadPackets(); }, 300));
document.getElementById('fHash').addEventListener('input', debounce((e) => { filters.hash = e.target.value || undefined; updatePacketsUrl(); loadPackets(); }, 300));
// Time window dropdown — restore from localStorage and bind change
const fTimeWindow = document.getElementById('fTimeWindow');
@@ -1065,7 +1090,7 @@
if (!q) {
fNodeDrop.classList.add('hidden');
fNode.setAttribute('aria-expanded', 'false');
if (filters.node) { filters.node = undefined; filters.nodeName = undefined; loadPackets(); }
if (filters.node) { filters.node = undefined; filters.nodeName = undefined; updatePacketsUrl(); loadPackets(); }
return;
}
try {
@@ -1094,6 +1119,7 @@
fNode.setAttribute('aria-expanded', 'false');
fNode.setAttribute('aria-activedescendant', '');
nodeActiveIdx = -1;
updatePacketsUrl();
loadPackets();
}
@@ -1849,7 +1875,12 @@
}
}
const anomalyBanner = decoded.anomaly
? `<div class="anomaly-banner" style="background:var(--warning, #f0ad4e); color:#000; padding:8px 12px; border-radius:4px; margin-bottom:8px; font-weight:600;">⚠️ Anomaly: ${escapeHtml(decoded.anomaly)}</div>`
: '';
panel.innerHTML = `
${anomalyBanner}
<div class="detail-title">${hasRawHex ? `Packet Byte Breakdown (${size} bytes)` : typeName + ' Packet'}</div>
<div class="detail-hash">${pkt.hash || 'Packet #' + pkt.id}</div>
${messageHtml}
@@ -1981,13 +2012,9 @@
// Header section
rows += sectionRow('Header', 'section-header');
rows += fieldRow(0, 'Header Byte', '0x' + (buf.slice(0, 2) || '??'), `Route: ${routeTypeName(pkt.route_type)}, Payload: ${payloadTypeName(pkt.payload_type)}`);
const pathByte0 = parseInt(buf.slice(2, 4), 16);
const hashSizeVal = isNaN(pathByte0) ? '?' : ((pathByte0 >> 6) + 1);
const hashCountVal = isNaN(pathByte0) ? '?' : (pathByte0 & 0x3F);
rows += fieldRow(1, 'Path Length', '0x' + (buf.slice(2, 4) || '??'), hashCountVal === 0 ? `hash_count=0 (direct advert)` : `hash_size=${hashSizeVal} byte${hashSizeVal !== 1 ? 's' : ''}, hash_count=${hashCountVal}`);
// Transport codes
let off = 2;
// Transport codes come BEFORE path length for transport routes (bytes 1-4)
let off = 1;
if (pkt.route_type === 0 || pkt.route_type === 3) {
rows += sectionRow('Transport Codes', 'section-transport');
rows += fieldRow(off, 'Next Hop', buf.slice(off * 2, (off + 2) * 2), '');
@@ -1995,11 +2022,18 @@
off += 4;
}
// Path length byte is at current offset (byte 1 for non-transport, byte 5 for transport)
const pathLenOffset = off;
const pathByte0 = parseInt(buf.slice(off * 2, off * 2 + 2), 16);
const hashSizeVal = isNaN(pathByte0) ? '?' : ((pathByte0 >> 6) + 1);
const hashCountVal = isNaN(pathByte0) ? '?' : (pathByte0 & 0x3F);
rows += fieldRow(off, 'Path Length', '0x' + (buf.slice(off * 2, off * 2 + 2) || '??'), hashCountVal === 0 ? `hash_count=0 (direct advert)` : `hash_size=${hashSizeVal} byte${hashSizeVal !== 1 ? 's' : ''}, hash_count=${hashCountVal}`);
off += 1;
// Path
if (pathHops.length > 0) {
rows += sectionRow('Path (' + pathHops.length + ' hops)', 'section-path');
const pathByte = parseInt(buf.slice(2, 4), 16);
const hashSize = (pathByte >> 6) + 1;
const hashSize = isNaN(pathByte0) ? 1 : ((pathByte0 >> 6) + 1);
for (let i = 0; i < pathHops.length; i++) {
const hopHtml = HopDisplay.renderHop(pathHops[i], hopNameCache[pathHops[i]]);
const label = `Hop ${i}${hopHtml}`;
@@ -2012,7 +2046,7 @@
rows += sectionRow('Payload — ' + payloadTypeName(pkt.payload_type), 'section-payload');
if (decoded.type === 'ADVERT') {
if (hashCountVal !== 0) rows += fieldRow(1, 'Advertised Hash Size', hashSizeVal + ' byte' + (hashSizeVal !== 1 ? 's' : ''), 'From path byte 0x' + (buf.slice(2, 4) || '??') + ' — bits 7-6 = ' + (hashSizeVal - 1));
if (hashCountVal !== 0) rows += fieldRow(pathLenOffset, 'Advertised Hash Size', hashSizeVal + ' byte' + (hashSizeVal !== 1 ? 's' : ''), 'From path byte 0x' + (buf.slice(pathLenOffset * 2, pathLenOffset * 2 + 2) || '??') + ' — bits 7-6 = ' + (hashSizeVal - 1));
rows += fieldRow(off, 'Public Key (32B)', truncate(decoded.pubKey || '', 24), '');
rows += fieldRow(off + 32, 'Timestamp (4B)', decoded.timestampISO || '', 'Unix: ' + (decoded.timestamp || ''));
rows += fieldRow(off + 36, 'Signature (64B)', truncate(decoded.signature || '', 24), '');
@@ -2053,6 +2087,10 @@
rows += fieldRow(off, 'Raw', truncate(buf.slice(off * 2), 40), '');
}
if (decoded.anomaly) {
rows += `<tr class="anomaly-row" style="background:var(--warning, #f0ad4e); color:#000; font-weight:600;"><td colspan="2">⚠️ Anomaly</td><td colspan="2">${escapeHtml(decoded.anomaly)}</td></tr>`;
}
return `<table class="field-table">
<thead><tr><th scope="col">Offset</th><th scope="col">Field</th><th scope="col">Value</th><th scope="col">Description</th></tr></thead>
<tbody>${rows}</tbody>
@@ -2144,6 +2182,11 @@
let html = '<div class="byop-decoded">';
// Anomaly banner
if (d.anomaly) {
html += '<div class="anomaly-banner" style="background:var(--warning, #f0ad4e); color:#000; padding:8px 12px; border-radius:4px; margin-bottom:8px; font-weight:600;">⚠️ Anomaly: ' + escapeHtml(d.anomaly) + '</div>';
}
// Header section
html += '<div class="byop-section">'
+ '<div class="byop-section-title">Header</div>'
+64
View File
@@ -394,4 +394,68 @@
});
return html;
};
// #690 — Clock Skew shared helpers
var SKEW_SEVERITY_COLORS = {
ok: 'var(--status-green)',
warning: 'var(--status-yellow)',
critical: 'var(--status-orange)',
absurd: 'var(--status-purple)',
no_clock: 'var(--text-muted)'
};
var SKEW_SEVERITY_LABELS = {
ok: 'OK', warning: 'Warning', critical: 'Critical', absurd: 'Absurd', no_clock: 'No Clock'
};
var SKEW_SEVERITY_ORDER = { no_clock: 0, absurd: 1, critical: 2, warning: 3, ok: 4 };
window.SKEW_SEVERITY_COLORS = SKEW_SEVERITY_COLORS;
window.SKEW_SEVERITY_LABELS = SKEW_SEVERITY_LABELS;
window.SKEW_SEVERITY_ORDER = SKEW_SEVERITY_ORDER;
/** Format skew seconds into human-readable string like "+2m 34s" or "-15h 22m" */
window.formatSkew = function(sec) {
if (sec == null) return '—';
var abs = Math.abs(sec);
var sign = sec >= 0 ? '+' : '-';
if (abs < 60) return sign + Math.round(abs) + 's';
if (abs < 3600) return sign + Math.floor(abs / 60) + 'm ' + Math.round(abs % 60) + 's';
if (abs < 86400) return sign + Math.floor(abs / 3600) + 'h ' + Math.round((abs % 3600) / 60) + 'm';
return sign + Math.floor(abs / 86400) + 'd ' + Math.round((abs % 86400) / 3600) + 'h';
};
/** Format drift rate as "+X.Xs/day" or "—" if falsy */
window.formatDrift = function(secPerDay) {
if (!secPerDay) return '—';
return (secPerDay >= 0 ? '+' : '') + secPerDay.toFixed(1) + ' s/day';
};
/** Render a clock skew badge HTML */
window.renderSkewBadge = function(severity, skewSec) {
if (!severity) return '';
var cls = 'skew-badge skew-badge--' + severity;
if (severity === 'no_clock') {
return '<span class="' + cls + '" title="Uninitialized RTC — no valid clock">🚫 No Clock</span>';
}
var label = severity === 'ok' ? '⏰' : '⏰ ' + window.formatSkew(skewSec);
return '<span class="' + cls + '" title="Clock skew: ' + window.formatSkew(skewSec) + ' (' + (SKEW_SEVERITY_LABELS[severity] || severity) + ')">' + label + '</span>';
};
/** Render a skew sparkline SVG (inline, word-sized) */
window.renderSkewSparkline = function(samples, w, h) {
w = w || 120; h = h || 24;
if (!samples || samples.length < 2) return '';
var values = samples.map(function(s) { return s.skew; });
var max = Math.max.apply(null, values.map(function(v) { return Math.abs(v); }).concat([1]));
var pts = values.map(function(v, i) {
var x = i * (w / Math.max(values.length - 1, 1));
var y = h / 2 - (v / max) * (h / 2 - 2);
return x.toFixed(1) + ',' + y.toFixed(1);
}).join(' ');
// Zero line
var zeroY = h / 2;
return '<svg viewBox="0 0 ' + w + ' ' + h + '" style="width:' + w + 'px;height:' + h + 'px" role="img" aria-label="Clock skew sparkline">' +
'<title>Clock skew over time</title>' +
'<line x1="0" y1="' + zeroY + '" x2="' + w + '" y2="' + zeroY + '" stroke="var(--border)" stroke-width="0.5" stroke-dasharray="2"/>' +
'<polyline points="' + pts + '" fill="none" stroke="var(--accent)" stroke-width="1.5"/></svg>';
};
})();
+82
View File
@@ -11,6 +11,9 @@
--status-green: #22c55e;
--status-yellow: #eab308;
--status-red: #ef4444;
--status-orange: #f97316;
--status-purple: #a855f7;
--role-observer: #8b5cf6;
--accent-hover: #6db3ff;
--text: #1a1a2e;
--text-muted: #5b6370;
@@ -41,6 +44,8 @@
--status-green: #22c55e;
--status-yellow: #eab308;
--status-red: #ef4444;
--status-orange: #f97316;
--status-purple: #a855f7;
--surface-0: #0f0f23;
--surface-1: #1a1a2e;
--surface-2: #232340;
@@ -65,6 +70,8 @@
--status-green: #22c55e;
--status-yellow: #eab308;
--status-red: #ef4444;
--status-orange: #f97316;
--status-purple: #a855f7;
--surface-0: #0f0f23;
--surface-1: #1a1a2e;
--surface-2: #232340;
@@ -463,6 +470,14 @@ fieldset.mc-section legend.mc-label { padding: 0; }
.ch-sidebar-title {
display: flex; align-items: center; gap: 8px; font-size: 16px; font-weight: 700; margin-bottom: 8px;
}
.ch-encrypted-toggle {
display: flex; align-items: center; gap: 4px; font-size: 11px; color: var(--text-muted);
cursor: pointer; user-select: none; margin-bottom: 4px;
}
.ch-encrypted-toggle input { margin: 0; cursor: pointer; }
.ch-toggle-label { white-space: nowrap; }
.ch-item.ch-encrypted { opacity: 0.55; }
.ch-item.ch-encrypted .ch-item-name { font-style: italic; }
.ch-icon { font-size: 20px; }
.ch-sidebar-controls { display: flex; align-items: center; gap: 6px; }
.ch-region-select {
@@ -495,6 +510,9 @@ button.ch-item.selected { background: var(--selected-bg); }
.ch-item-top { display: flex; justify-content: space-between; align-items: baseline; margin-bottom: 2px; }
.ch-item-name { font-weight: 600; font-size: 14px; }
.ch-item-time { font-size: 11px; color: var(--text-muted); white-space: nowrap; }
.ch-remove-btn { background: none; border: none; color: var(--text-muted); cursor: pointer; font-size: 13px; padding: 0 2px; margin-left: 4px; opacity: 0; transition: opacity 0.15s; line-height: 1; }
button.ch-item:hover .ch-remove-btn { opacity: 0.6; }
.ch-remove-btn:hover { opacity: 1 !important; color: var(--danger, #dc2626); }
.ch-item-preview { font-size: 12px; color: var(--text-muted); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.ch-main { flex: 1; display: flex; flex-direction: column; overflow: hidden; position: relative; }
@@ -1097,6 +1115,49 @@ button.ch-item.ch-item-encrypted:hover { opacity: 0.7; }
button.ch-item.ch-item-encrypted.selected { opacity: 0.8; }
button.ch-item.ch-item-encrypted .ch-badge { filter: grayscale(0.6); }
/* Channel key input (#725 M2) */
.ch-key-input {
flex: 1;
min-width: 0;
box-sizing: border-box;
padding: 6px 8px;
border: 1px solid var(--border);
border-radius: 6px 0 0 6px;
background: var(--card-bg);
color: var(--text);
font-size: 12px;
font-family: inherit;
}
.ch-key-input:focus {
outline: 2px solid var(--accent, #3b82f6);
outline-offset: -1px;
border-color: var(--accent, #3b82f6);
}
.ch-key-input::placeholder { color: var(--text-muted); }
.ch-key-input-wrap { margin-bottom: 4px; }
.ch-wrong-key { color: var(--danger, #ef4444); font-weight: 500; }
/* Add channel form (#759) */
.ch-add-form { margin: 0; }
.ch-add-label { display: block; font-weight: 600; font-size: 13px; color: var(--text); margin-bottom: 4px; }
.ch-key-input, .ch-add-btn { height: 32px; box-sizing: border-box; }
.ch-add-row { display: flex; align-items: stretch; }
.ch-add-btn {
width: 32px; height: 32px; flex-shrink: 0;
border: 1px solid var(--accent, #3b82f6); border-left: none;
border-radius: 0 6px 6px 0;
background: var(--accent, #3b82f6); color: #fff;
font-size: 18px; font-weight: 700; line-height: 1;
cursor: pointer; display: flex; align-items: center; justify-content: center;
}
.ch-add-btn:hover { opacity: 0.85; }
.ch-add-hint { font-size: 11px; color: var(--text-muted); margin-top: 4px; line-height: 1.3; }
.ch-add-status { font-size: 12px; margin-top: 4px; padding: 4px 6px; border-radius: 4px; }
.ch-add-status--loading { color: var(--text-muted); }
.ch-add-status--success { color: var(--success, #22c55e); }
.ch-add-status--warn { color: var(--warning, #eab308); }
.ch-add-status--error { color: var(--danger, #ef4444); }
/* Touch-friendly tappable elements */
.ch-tappable {
cursor: pointer;
@@ -2211,3 +2272,24 @@ th[data-sort-key] { cursor: pointer; user-select: none; }
th[data-sort-key]:hover { background: var(--hover-bg, rgba(255,255,255,0.05)); }
th.sort-active { color: var(--accent, #60a5fa); }
.sort-arrow { font-size: 0.75em; opacity: 0.8; }
/* #690 — Clock Skew badges & fleet table */
.skew-badge { display: inline-block; font-size: 10px; padding: 1px 5px; border-radius: 3px; margin-left: 4px; font-weight: 600; white-space: nowrap; }
.skew-badge--ok { background: var(--status-green); color: #fff; }
.skew-badge--warning { background: var(--status-yellow); color: #000; }
.skew-badge--critical { background: var(--status-orange); color: #fff; }
.skew-badge--absurd { background: var(--status-purple); color: #fff; }
.skew-badge--no_clock { background: var(--text-muted); color: #fff; }
.skew-detail-section { padding: 10px 16px; margin-bottom: 8px; }
.skew-sparkline-wrap { margin-top: 6px; }
.skew-sparkline-wrap svg { display: block; }
.clock-fleet-row--warning { background: color-mix(in srgb, var(--status-yellow) 10%, transparent); }
.clock-fleet-row--critical { background: color-mix(in srgb, var(--status-orange) 10%, transparent); }
.clock-fleet-row--absurd { background: color-mix(in srgb, var(--status-purple) 10%, transparent); }
.clock-fleet-row--no_clock { background: color-mix(in srgb, var(--text-muted) 10%, transparent); }
.clock-filter-btn { font-size: 12px; padding: 3px 8px; border: 1px solid var(--border); border-radius: 4px; background: var(--card-bg, #fff); color: var(--text); cursor: pointer; margin-right: 4px; }
.clock-filter-btn.active { background: var(--accent); color: #fff; border-color: var(--accent); }
+64
View File
@@ -0,0 +1,64 @@
/**
* Tests for #759 Add channel UX: button, hint, status feedback.
* Validates the HTML structure rendered by channels.js init.
*/
'use strict';
const fs = require('fs');
let passed = 0;
let failed = 0;
function assert(cond, msg) {
if (cond) { passed++; console.log(' ✓ ' + msg); }
else { failed++; console.error(' ✗ ' + msg); }
}
function assertIncludes(html, substr, msg) {
assert(html.includes(substr), msg);
}
// Read the channels.js source to extract the HTML template
const src = fs.readFileSync(__dirname + '/public/channels.js', 'utf8');
// Extract the sidebar HTML from the template literal
const htmlMatch = src.match(/app\.innerHTML\s*=\s*`([\s\S]*?)`;/);
const html = htmlMatch ? htmlMatch[1] : '';
console.log('Test: Add channel UX (#759)');
// 1. Button renders in the form
assertIncludes(html, 'class="ch-add-btn"', 'Add button has ch-add-btn class');
assertIncludes(html, 'type="submit"', 'Button is type=submit');
assertIncludes(html, '>+</button>', 'Button shows + text');
// 2. Form has proper structure
assertIncludes(html, 'class="ch-add-form"', 'Form has ch-add-form class');
assertIncludes(html, 'class="ch-add-row"', 'Row wrapper present');
assert(!html.includes('class="ch-add-label"'), 'Label removed (redundant with hint)');
// 3. Hint text present
assertIncludes(html, 'class="ch-add-hint"', 'Hint div present');
assertIncludes(html, 'e.g. #LongFast or 32-char hex key', 'Hint text correct');
// 4. Status div present
assertIncludes(html, 'id="chAddStatus"', 'Status div has correct id');
assertIncludes(html, 'class="ch-add-status"', 'Status div has correct class');
assertIncludes(html, 'style="display:none"', 'Status div hidden by default');
// 5. showAddStatus function exists in source
assert(src.includes('function showAddStatus('), 'showAddStatus function defined');
assert(src.includes("'success'"), 'Success status type referenced');
assert(src.includes("'error'"), 'Error status type referenced');
// 6. CSS classes exist
const css = fs.readFileSync(__dirname + '/public/style.css', 'utf8');
assert(css.includes('.ch-add-form'), 'CSS: .ch-add-form defined');
assert(css.includes('.ch-add-btn'), 'CSS: .ch-add-btn defined');
assert(css.includes('.ch-add-hint'), 'CSS: .ch-add-hint defined');
assert(css.includes('.ch-add-status'), 'CSS: .ch-add-status defined');
assert(css.includes('.ch-add-row'), 'CSS: .ch-add-row defined');
// .ch-add-label CSS kept for backward compat but label removed from HTML
console.log('\n' + passed + ' passed, ' + failed + ' failed');
process.exit(failed > 0 ? 1 : 0);
+157
View File
@@ -0,0 +1,157 @@
/**
* Tests for #725 M3 (PSK hex key), M4 (channel removal), M5 (message caching).
* Runs in Node.js via vm.createContext to simulate browser environment.
*/
'use strict';
const vm = require('vm');
const fs = require('fs');
const { subtle } = require('crypto').webcrypto;
let passed = 0;
let failed = 0;
function assert(cond, msg) {
if (cond) { passed++; console.log(' ✓ ' + msg); }
else { failed++; console.error(' ✗ ' + msg); }
}
// Build a minimal browser-like sandbox
function createSandbox() {
const storage = {};
const localStorage = {
getItem: (k) => storage[k] !== undefined ? storage[k] : null,
setItem: (k, v) => { storage[k] = String(v); },
removeItem: (k) => { delete storage[k]; },
_data: storage
};
const ctx = {
window: {},
crypto: { subtle },
TextEncoder: TextEncoder,
TextDecoder: TextDecoder,
Uint8Array,
localStorage,
console,
Date,
JSON,
parseInt,
Math,
String,
Number,
Object,
Array,
RegExp,
Error,
Promise,
setTimeout,
btoa: (s) => Buffer.from(s, 'binary').toString('base64'),
atob: (s) => Buffer.from(s, 'base64').toString('binary'),
};
ctx.window = ctx;
ctx.self = ctx;
return ctx;
}
async function runTests() {
console.log('\n=== M3: PSK hex key detection ===');
// Load channel-decrypt.js in sandbox
const cdSrc = fs.readFileSync(__dirname + '/public/channel-decrypt.js', 'utf8');
const sandbox = createSandbox();
const context = vm.createContext(sandbox);
vm.runInContext(cdSrc, context);
const CD = sandbox.window.ChannelDecrypt;
// Test: isHexKey detection (via channels.js logic)
// We test the pattern directly since isHexKey is inside channels.js IIFE
const isHexKey = (val) => /^[0-9a-fA-F]{32}$/.test(val);
assert(isHexKey('0123456789abcdef0123456789abcdef'), 'Valid 32-char hex detected');
assert(isHexKey('AABBCCDD11223344AABBCCDD11223344'), 'Valid uppercase hex detected');
assert(!isHexKey('#LongFast'), 'Hashtag name NOT detected as hex');
assert(!isHexKey('0123456789abcdef'), 'Short hex (16 chars) NOT detected');
assert(!isHexKey('0123456789abcdef0123456789abcdefXX'), 'Too long NOT detected');
assert(!isHexKey('zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz'), 'Non-hex chars NOT detected');
// Test: PSK decrypt with known key bytes
console.log('\n=== M3: PSK decrypt produces correct plaintext ===');
// Derive a key from #LongFast for testing
const keyBytes = await CD.deriveKey('#LongFast');
assert(keyBytes.length === 16, 'Derived key is 16 bytes');
const keyHex = CD.bytesToHex(keyBytes);
assert(keyHex.length === 32, 'Key hex is 32 chars');
// Round-trip: hex → bytes → hex
const roundTrip = CD.bytesToHex(CD.hexToBytes(keyHex));
assert(roundTrip === keyHex, 'Hex round-trip preserves key');
// Channel hash computation works
const hashByte = await CD.computeChannelHash(keyBytes);
assert(typeof hashByte === 'number' && hashByte >= 0 && hashByte <= 255, 'Channel hash byte is valid (0-255)');
// PSK key (raw hex) stored and retrieved correctly
const pskHex = 'aabbccdd11223344aabbccdd11223344';
CD.storeKey('psk:aabbccdd', pskHex);
const keys = CD.getStoredKeys();
assert(keys['psk:aabbccdd'] === pskHex, 'PSK key stored and retrieved correctly');
console.log('\n=== M4: Channel removal clears key + cache ===');
// Store a key and some cached messages
CD.storeKey('#TestChannel', 'deadbeefdeadbeefdeadbeefdeadbeef');
CD.setCache('#TestChannel', [{ sender: 'A', text: 'hello', timestamp: '2026-01-01T00:00:00Z', packetHash: 'h1' }], '2026-01-01T00:00:00Z', 1);
// Verify they exist
var storedKeys = CD.getStoredKeys();
assert(storedKeys['#TestChannel'] === 'deadbeefdeadbeefdeadbeefdeadbeef', 'Key exists before removal');
var cachedBefore = CD.getCache('#TestChannel');
assert(cachedBefore && cachedBefore.messages.length === 1, 'Cache exists before removal');
// Remove the key (also clears cache)
CD.removeKey('#TestChannel');
var storedAfter = CD.getStoredKeys();
assert(!storedAfter['#TestChannel'], 'Key cleared after removal');
var cachedAfter = CD.getCache('#TestChannel');
assert(!cachedAfter, 'Cache cleared after removal');
console.log('\n=== M5: Cache operations ===');
// Test: setCache with count and size limit
var bigMessages = [];
for (var i = 0; i < 1200; i++) {
bigMessages.push({ sender: 'S', text: 'msg' + i, timestamp: '2026-01-01T00:00:' + String(i).padStart(2, '0') + 'Z', packetHash: 'h' + i });
}
CD.setCache('bigchannel', bigMessages, '2026-01-01T00:20:00Z', 1200);
var bigCached = CD.getCache('bigchannel');
assert(bigCached.messages.length <= 1000, 'Cache enforces 1000 message limit (got ' + bigCached.messages.length + ')');
assert(bigCached.count === 1200, 'Cache stores total count');
assert(bigCached.lastTimestamp === '2026-01-01T00:20:00Z', 'Cache stores lastTimestamp');
// Should keep most recent 1000
assert(bigCached.messages[0].packetHash === 'h200', 'Cache keeps most recent 1000 (first is h200)');
// Test: cache hit (delta fetch scenario)
CD.setCache('deltatest', [
{ sender: 'A', text: 'old', timestamp: '2026-01-01T00:00:00Z', packetHash: 'p1' }
], '2026-01-01T00:00:00Z', 1);
var deltaCache = CD.getCache('deltatest');
assert(deltaCache.messages.length === 1, 'Delta cache has 1 message');
assert(deltaCache.lastTimestamp === '2026-01-01T00:00:00Z', 'Delta cache lastTimestamp correct');
assert(deltaCache.count === 1, 'Delta cache count correct');
// Test: clearChannelCache
CD.setCache('clearthis', [{ sender: 'X', text: 'y' }], 'ts', 1);
assert(CD.getCache('clearthis') !== null, 'Cache exists before clear');
CD.clearChannelCache('clearthis');
assert(CD.getCache('clearthis') === null, 'Cache cleared by clearChannelCache');
console.log('\n=== Results ===');
console.log('Passed: ' + passed + ', Failed: ' + failed);
process.exit(failed > 0 ? 1 : 0);
}
runTests().catch(e => { console.error(e); process.exit(1); });
+90 -4
View File
@@ -231,6 +231,26 @@ async function run() {
assert(hasStatus, 'No status indicator found in node detail');
});
// Test: Node side panel Details link opens full-screen detail view (#778)
await test('Node side panel Details link opens full detail', async () => {
await page.goto(`${BASE}/#/nodes`, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('table tbody tr');
// Click first row to open side panel
const firstRow = await page.$('table tbody tr');
assert(firstRow, 'No node rows found');
await firstRow.click();
await page.waitForSelector('.node-detail');
// Find the Details link in the side panel
const detailsLink = await page.$('#nodesRight a.btn-primary[href^="#/nodes/"]');
assert(detailsLink, 'Details link not found in side panel');
// Click the Details link — should open full-screen node detail view
await detailsLink.click();
// Wait for the full-screen node detail view to render
await page.waitForSelector('.node-fullscreen', { timeout: 5000 });
const hasFullBody = await page.$('.node-full-body');
assert(hasFullBody, 'Full-screen node detail body not found');
});
// Test: Nodes page has WebSocket auto-update listener (#131)
await test('Nodes page has WebSocket auto-update', async () => {
await page.goto(`${BASE}/#/nodes`, { waitUntil: 'domcontentloaded' });
@@ -398,8 +418,13 @@ async function run() {
}
}, { timeout: 10000 });
// Full reload on the packets page — scripts re-execute, IIFE reads localStorage
await page.reload({ waitUntil: 'load' });
// Force a full page reload to reset module-level state (savedTimeWindowMin is
// read from localStorage once at IIFE time). Navigating from /#/packets to /#/packets
// is a hash-only change — no reload, so the IIFE never re-reads localStorage.
// Going to / first forces a fresh page load, then the hash change to /#/packets
// calls init() with the freshly-read savedTimeWindowMin = 60.
await page.goto(`${BASE}/`, { waitUntil: 'load' });
await page.goto(`${BASE}/#/packets`, { waitUntil: 'load' });
await page.waitForSelector('#fTimeWindow', { timeout: 10000 });
const timeWindowValue = await page.$eval('#fTimeWindow', (el) => el.value);
assert(timeWindowValue === '60', `Expected time window dropdown to restore 60, got ${timeWindowValue}`);
@@ -1470,13 +1495,16 @@ async function run() {
// ─── Neighbor section tests ───────────────────────────────────────────────
await test('Node detail: neighbors section exists with correct columns', async () => {
// Full-screen node view (with #node-neighbors) is mobile-only since #676 fix.
await page.setViewportSize({ width: 390, height: 844 });
// Navigate to a node detail page (use the first node in the list)
await page.goto(BASE + '/#/nodes');
await page.waitForSelector('#nodesBody tr[data-key]', { timeout: 10000 });
// Get the first node's pubkey from the row's data-key attribute
const pubkey = await page.$eval('#nodesBody tr[data-key]', el => el.dataset.key);
await page.goto(BASE + '/#/nodes/' + pubkey);
await page.waitForSelector('#node-neighbors', { timeout: 10000 });
// Use evaluate to change hash (reliable same-document navigation)
await page.evaluate((pk) => { location.hash = '#/nodes/' + pk; }, pubkey);
await page.waitForSelector('#node-neighbors', { timeout: 15000 });
// Check the section exists
const header = await page.$eval('#fullNeighborsHeader', el => el.textContent);
assert(header.startsWith('Neighbors'), 'Header should start with "Neighbors", got: ' + header);
@@ -1500,6 +1528,7 @@ async function run() {
const text = await page.$eval('#fullNeighborsContent', el => el.textContent);
assert(text.includes('No neighbor data') || text.includes('Could not load'), 'Should show empty or error state');
}
await page.setViewportSize({ width: 1280, height: 800 });
});
@@ -1653,6 +1682,33 @@ async function run() {
assert(url.includes('tab=room'), `URL should contain tab=room after click, got: ${url}`);
});
// Test: clicking a node on desktop updates URL hash (#676)
await test('Desktop: clicking a node updates URL to #/nodes/{pubkey}', async () => {
await page.setViewportSize({ width: 1280, height: 800 });
await page.goto(BASE + '#/nodes', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#nodesBody tr[data-key]', { timeout: 10000 });
const pubkey = await page.$eval('#nodesBody tr[data-key]', el => el.dataset.key);
await page.click('#nodesBody tr[data-key]');
await page.waitForTimeout(300);
const url = page.url();
assert(url.includes(encodeURIComponent(pubkey)), `URL should contain pubkey after click, got: ${url}`);
assert(!url.includes('node-fullscreen') || await page.$('#nodesRight:not(.empty)'), 'Split panel should be visible on desktop');
});
// Test: loading #/nodes/{pubkey} on desktop shows split panel (#676)
await test('Desktop: deep link #/nodes/{pubkey} opens split panel, not full-screen', async () => {
await page.setViewportSize({ width: 1280, height: 800 });
await page.goto(BASE + '#/nodes', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#nodesBody tr[data-key]', { timeout: 10000 });
const pubkey = await page.$eval('#nodesBody tr[data-key]', el => el.dataset.key);
await page.goto(BASE + '#/nodes/' + encodeURIComponent(pubkey), { waitUntil: 'domcontentloaded' });
await page.waitForTimeout(500);
const hasSplitPanel = await page.$('#nodesRight:not(.empty)');
const hasFullScreen = await page.$('.node-fullscreen');
assert(hasSplitPanel, 'Split panel should be open on desktop deep link');
assert(!hasFullScreen, 'Full-screen view should NOT appear on desktop deep link');
});
// Test: packets timeWindow deep link
await test('Packets timeWindow deep link restores dropdown', async () => {
await page.goto(BASE + '#/packets?timeWindow=60', { waitUntil: 'domcontentloaded' });
@@ -1663,6 +1719,36 @@ async function run() {
assert(url.includes('timeWindow=60'), `URL should still contain timeWindow=60, got: ${url}`);
});
// Test: hash filter updates URL and is restored (#682)
await test('Packets hash filter updates URL and restores on reload', async () => {
await page.goto(BASE + '#/packets', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#fHash', { timeout: 8000 });
await page.fill('#fHash', 'abc123');
await page.waitForTimeout(500);
const url = page.url();
assert(url.includes('hash=abc123'), `URL should contain hash=abc123, got: ${url}`);
// Reload and check input restored
await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#fHash', { timeout: 8000 });
const val = await page.$eval('#fHash', el => el.value);
assert(val === 'abc123', `fHash should be restored to abc123, got: ${val}`);
});
// Test: Wireshark filter expression updates URL and is restored (#682)
await test('Packets filter expression updates URL and restores on reload', async () => {
await page.goto(BASE + '#/packets', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#packetFilterInput', { timeout: 8000 });
await page.fill('#packetFilterInput', 'type == ADVERT');
await page.waitForTimeout(500);
const url = page.url();
assert(url.includes('filter=') && url.includes('ADVERT'), `URL should contain filter=type%3D%3DADVERT, got: ${url}`);
// Reload and check expression restored
await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#packetFilterInput', { timeout: 8000 });
const val = await page.$eval('#packetFilterInput', el => el.value);
assert(val === 'type == ADVERT', `packetFilterInput should be restored, got: ${val}`);
});
// Test: timeWindow change updates URL
await test('Packets timeWindow change updates URL', async () => {
await page.goto(BASE + '#/packets', { waitUntil: 'domcontentloaded' });
Binary file not shown.
+510 -1
View File
@@ -2345,7 +2345,9 @@ console.log('\n=== channels.js: shouldProcessWSMessageForRegion ===');
ctx.history = { replaceState() {} };
ctx.btoa = (s) => Buffer.from(String(s), 'utf8').toString('base64');
ctx.atob = (s) => Buffer.from(String(s), 'base64').toString('utf8');
loadInCtx(ctx, 'public/channels.js');
ctx.crypto = { subtle: require('crypto').webcrypto.subtle }; ctx.TextEncoder = TextEncoder; ctx.TextDecoder = TextDecoder; ctx.Uint8Array = Uint8Array;
loadInCtx(ctx, 'public/channel-decrypt.js');
loadInCtx(ctx, 'public/channels.js');
const shouldProcess = ctx.window._channelsShouldProcessWSMessageForRegion;
test('helper is exported', () => assert.ok(typeof shouldProcess === 'function'));
@@ -2467,6 +2469,8 @@ console.log('\n=== channels.js: WS batch + region snapshot integration ===');
ctx.btoa = (s) => Buffer.from(String(s), 'utf8').toString('base64');
ctx.atob = (s) => Buffer.from(String(s), 'base64').toString('utf8');
ctx.crypto = { subtle: require('crypto').webcrypto.subtle }; ctx.TextEncoder = TextEncoder; ctx.TextDecoder = TextDecoder; ctx.Uint8Array = Uint8Array;
loadInCtx(ctx, 'public/channel-decrypt.js');
loadInCtx(ctx, 'public/channels.js');
ctx._pageHandlers.init(appEl);
return { ctx, dom };
@@ -2586,6 +2590,8 @@ console.log('\n=== channels.js: WS batch + region snapshot integration ===');
ctx.btoa = (s) => Buffer.from(String(s), 'utf8').toString('base64');
ctx.atob = (s) => Buffer.from(String(s), 'base64').toString('utf8');
ctx.crypto = { subtle: require('crypto').webcrypto.subtle }; ctx.TextEncoder = TextEncoder; ctx.TextDecoder = TextDecoder; ctx.Uint8Array = Uint8Array;
loadInCtx(ctx, 'public/channel-decrypt.js');
loadInCtx(ctx, 'public/channels.js');
ctx._pageHandlers.init(appEl);
await Promise.resolve();
@@ -2681,6 +2687,8 @@ console.log('\n=== channels.js: WS batch + region snapshot integration ===');
ctx.btoa = (s) => Buffer.from(String(s), 'utf8').toString('base64');
ctx.atob = (s) => Buffer.from(String(s), 'base64').toString('utf8');
ctx.crypto = { subtle: require('crypto').webcrypto.subtle }; ctx.TextEncoder = TextEncoder; ctx.TextDecoder = TextDecoder; ctx.Uint8Array = Uint8Array;
loadInCtx(ctx, 'public/channel-decrypt.js');
loadInCtx(ctx, 'public/channels.js');
ctx._pageHandlers.init(appEl);
await Promise.resolve();
@@ -5079,6 +5087,507 @@ console.log('\n=== analytics.js: renderMultiByteAdopters ===');
}
}
// ===== packets.js: anomaly banner rendering =====
console.log('\n=== packets.js: anomaly UI rendering ===');
{
const packetsSource = fs.readFileSync('public/packets.js', 'utf8');
test('renderDetail shows anomaly banner when decoded.anomaly is set', () => {
assert.ok(packetsSource.includes('anomaly-banner'),
'packets.js should contain anomaly-banner class');
assert.ok(packetsSource.includes("decoded.anomaly"),
'packets.js should reference decoded.anomaly');
});
test('buildFieldTable includes anomaly row when present', () => {
assert.ok(packetsSource.includes('anomaly-row'),
'buildFieldTable should have anomaly-row class for highlighted row');
});
test('renderDecodedPacket shows anomaly banner', () => {
assert.ok(packetsSource.includes("d.anomaly"),
'renderDecodedPacket should check d.anomaly');
});
}
// ===== packets.js: buildFieldTable transport offset tests (#765) =====
console.log('\n=== packets.js: buildFieldTable transport offsets (#765) ===');
{
const ftCtx = makeSandbox();
ftCtx.registerPage = () => {};
ftCtx.onWS = () => {};
ftCtx.offWS = () => {};
ftCtx.api = () => Promise.resolve({});
ftCtx.window.getParsedPath = () => [];
ftCtx.window.getParsedDecoded = () => ({});
// Provide globals from app.js that packets.js depends on
const ROUTE_TYPES = {0:'TRANSPORT_FLOOD',1:'FLOOD',2:'DIRECT',3:'TRANSPORT_DIRECT'};
const PAYLOAD_TYPES = {0:'ADVERT',1:'TXT_MSG',2:'GRP_TXT',3:'REQ',4:'ACK'};
ftCtx.routeTypeName = (n) => ROUTE_TYPES[n] || 'UNKNOWN';
ftCtx.payloadTypeName = (n) => PAYLOAD_TYPES[n] || 'UNKNOWN';
ftCtx.window.routeTypeName = ftCtx.routeTypeName;
ftCtx.window.payloadTypeName = ftCtx.payloadTypeName;
ftCtx.truncate = (str, len) => str && str.length > len ? str.slice(0, len) + '…' : (str || '');
ftCtx.window.truncate = ftCtx.truncate;
ftCtx.escapeHtml = (s) => String(s || '').replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;');
ftCtx.window.escapeHtml = ftCtx.escapeHtml;
loadInCtx(ftCtx, 'public/packets.js');
const { buildFieldTable, fieldRow } = ftCtx.window._packetsTestAPI;
// Helper: build a hex string with specific bytes
function makeHex(bytes) { return bytes.map(b => b.toString(16).padStart(2, '0')).join(''); }
test('FLOOD (route_type=1): path_length at byte 1, no transport codes', () => {
// header=0x05 (route_type=1, payload=1), path_length=0x41 (hash_size=2, count=1), hop=AABB
const raw = makeHex([0x05, 0x41, 0xAA, 0xBB]);
const pkt = { raw_hex: raw, route_type: 1, payload_type: 1 };
const html = buildFieldTable(pkt, {}, [], {});
// Path Length should be at offset 1
assert.ok(html.includes('>1<') || html.includes('data-offset="1"'),
'FLOOD: Path Length row should reference byte offset 1');
// Should NOT contain transport codes
assert.ok(!html.includes('Next Hop'), 'FLOOD: should not show Next Hop transport');
assert.ok(!html.includes('Last Hop'), 'FLOOD: should not show Last Hop transport');
});
test('TRANSPORT_FLOOD (route_type=0): transport codes at bytes 1-4, path_length at byte 5', () => {
// header=0x04 (route_type=0, payload=1), next_hop=1122, last_hop=3344, path_length=0x41
const raw = makeHex([0x04, 0x11, 0x22, 0x33, 0x44, 0x41, 0xAA, 0xBB]);
const pkt = { raw_hex: raw, route_type: 0, payload_type: 1 };
const html = buildFieldTable(pkt, {}, [], {});
// Transport codes should appear
assert.ok(html.includes('Next Hop'), 'TRANSPORT_FLOOD: should show Next Hop');
assert.ok(html.includes('Last Hop'), 'TRANSPORT_FLOOD: should show Last Hop');
// Path Length should be at offset 5, not 1
// Check that Path Length row does NOT show offset 1
const pathLenMatch = html.match(/Path Length/);
assert.ok(pathLenMatch, 'TRANSPORT_FLOOD: should have Path Length row');
// The field table renders offset in first <td>. Check transport codes come before path length
const nextHopIdx = html.indexOf('Next Hop');
const pathLenIdx = html.indexOf('Path Length');
assert.ok(nextHopIdx < pathLenIdx,
'TRANSPORT_FLOOD: transport codes should appear before Path Length in table order');
});
test('TRANSPORT_DIRECT (route_type=3): same offsets as TRANSPORT_FLOOD', () => {
const raw = makeHex([0x0F, 0x11, 0x22, 0x33, 0x44, 0x41]);
const pkt = { raw_hex: raw, route_type: 3, payload_type: 3 };
const html = buildFieldTable(pkt, {}, [], {});
assert.ok(html.includes('Next Hop'), 'TRANSPORT_DIRECT: should show Next Hop');
assert.ok(html.includes('Last Hop'), 'TRANSPORT_DIRECT: should show Last Hop');
const nextHopIdx = html.indexOf('Next Hop');
const pathLenIdx = html.indexOf('Path Length');
assert.ok(nextHopIdx < pathLenIdx,
'TRANSPORT_DIRECT: transport codes should appear before Path Length');
});
test('field table row order matches byte layout for transport routes', () => {
const raw = makeHex([0x04, 0x11, 0x22, 0x33, 0x44, 0x41, 0xAA, 0xBB]);
const pkt = { raw_hex: raw, route_type: 0, payload_type: 1 };
const html = buildFieldTable(pkt, {}, [], {});
// Order: Header (0) → Next Hop (1) → Last Hop (3) → Path Length (5)
const headerIdx = html.indexOf('Header Byte');
const nextHopIdx = html.indexOf('Next Hop');
const lastHopIdx = html.indexOf('Last Hop');
const pathLenIdx = html.indexOf('Path Length');
assert.ok(headerIdx < nextHopIdx, 'Header should come before Next Hop');
assert.ok(nextHopIdx < lastHopIdx, 'Next Hop should come before Last Hop');
assert.ok(lastHopIdx < pathLenIdx, 'Last Hop should come before Path Length');
});
}
// ===== live.js: anomaly icon in feed =====
console.log('\n=== live.js: anomaly icon in feed ===');
{
const liveSource = fs.readFileSync('public/live.js', 'utf8');
test('addFeedItemDOM shows anomaly icon when decoded has anomaly', () => {
assert.ok(liveSource.includes('anomalyIcon'),
'live.js should have anomalyIcon variable for feed items');
assert.ok(liveSource.includes('pkt.decoded && pkt.decoded.anomaly'),
'live.js should check pkt.decoded.anomaly');
});
}
// ===== channel-decrypt.js: client-side crypto =====
console.log('\n=== channel-decrypt.js: key derivation, MAC, parsing, storage ===');
{
const cryptoModule = require('crypto');
const ctx = makeSandbox();
// Provide Web Crypto API in sandbox
ctx.crypto = { subtle: cryptoModule.webcrypto.subtle };
ctx.TextEncoder = TextEncoder;
ctx.TextDecoder = TextDecoder;
ctx.Uint8Array = Uint8Array;
loadInCtx(ctx, 'public/channel-decrypt.js');
const CD = ctx.ChannelDecrypt;
test('deriveKey: SHA256("#test")[:16] matches known value', async () => {
const key = await CD.deriveKey('#test');
const hex = CD.bytesToHex(key);
// Verify against Node.js crypto
const expected = cryptoModule.createHash('sha256').update('#test').digest('hex').substring(0, 32);
assert.strictEqual(hex, expected, 'deriveKey should produce SHA256("#test")[:16]');
});
test('deriveKey: returns 16 bytes', async () => {
const key = await CD.deriveKey('#LongFast');
assert.strictEqual(key.length, 16);
});
test('computeChannelHash: SHA256(key)[0]', async () => {
const key = await CD.deriveKey('#test');
const hashByte = await CD.computeChannelHash(key);
const keyHex = CD.bytesToHex(key);
const expected = cryptoModule.createHash('sha256').update(Buffer.from(keyHex, 'hex')).digest()[0];
assert.strictEqual(hashByte, expected);
});
test('verifyMAC: valid MAC passes', async () => {
// Create a known ciphertext and compute MAC using Node.js
const key = await CD.deriveKey('#test');
const secret = Buffer.alloc(32);
Buffer.from(CD.bytesToHex(key), 'hex').copy(secret, 0);
const ciphertext = Buffer.from('00112233445566778899aabbccddeeff', 'hex');
const mac = cryptoModule.createHmac('sha256', secret).update(ciphertext).digest();
const macHex = mac.slice(0, 2).toString('hex');
const result = await CD.verifyMAC(key, new Uint8Array(ciphertext), macHex);
assert.strictEqual(result, true, 'valid MAC should pass');
});
test('verifyMAC: invalid MAC fails', async () => {
const key = await CD.deriveKey('#test');
const ciphertext = new Uint8Array(16);
const result = await CD.verifyMAC(key, ciphertext, 'ffff');
assert.strictEqual(result, false, 'invalid MAC should fail');
});
test('parsePlaintext: extracts sender and message', () => {
// Build plaintext: timestamp(4 LE) + flags(1) + "alice: hello\0"
const msg = 'alice: hello\0';
const buf = new Uint8Array(5 + msg.length);
// timestamp = 1000 (LE)
buf[0] = 0xe8; buf[1] = 0x03; buf[2] = 0; buf[3] = 0;
buf[4] = 0; // flags
const enc = new TextEncoder();
const msgBytes = enc.encode(msg);
buf.set(msgBytes, 5);
const parsed = CD.parsePlaintext(buf);
assert.ok(parsed, 'should parse successfully');
assert.strictEqual(parsed.sender, 'alice');
assert.strictEqual(parsed.message, 'hello');
assert.strictEqual(parsed.timestamp, 1000);
});
test('parsePlaintext: no sender prefix returns empty sender', () => {
const msg = 'just a message\0';
const buf = new Uint8Array(5 + msg.length);
buf[0] = 1; buf[1] = 0; buf[2] = 0; buf[3] = 0; buf[4] = 0;
buf.set(new TextEncoder().encode(msg), 5);
const parsed = CD.parsePlaintext(buf);
assert.ok(parsed);
assert.strictEqual(parsed.sender, '');
assert.strictEqual(parsed.message, 'just a message');
});
test('parsePlaintext: returns null for too-short input', () => {
assert.strictEqual(CD.parsePlaintext(new Uint8Array(3)), null);
});
test('localStorage persistence: save/get/remove keys', () => {
CD.saveKey('#test', 'abcd1234abcd1234abcd1234abcd1234');
const keys = CD.getKeys();
assert.strictEqual(keys['#test'], 'abcd1234abcd1234abcd1234abcd1234');
CD.removeKey('#test');
const keys2 = CD.getKeys();
assert.strictEqual(keys2['#test'], undefined);
});
test('bytesToHex and hexToBytes roundtrip', () => {
const hex = 'deadbeef01020304';
const bytes = CD.hexToBytes(hex);
assert.strictEqual(CD.bytesToHex(bytes), hex);
});
}
// ===== Encrypted Channels Toggle Tests (#728) =====
{
console.log('\n--- Encrypted Channels Toggle (#728) ---');
test('encrypted toggle reads from localStorage', () => {
const store = {};
const ls = {
getItem: k => store[k] || null,
setItem: (k, v) => { store[k] = String(v); },
};
// Default: not set → should be false
assert.strictEqual(ls.getItem('channels-show-encrypted'), null);
const showEncrypted = ls.getItem('channels-show-encrypted') === 'true';
assert.strictEqual(showEncrypted, false);
// Set to true
ls.setItem('channels-show-encrypted', 'true');
assert.strictEqual(ls.getItem('channels-show-encrypted') === 'true', true);
// Set to false
ls.setItem('channels-show-encrypted', 'false');
assert.strictEqual(ls.getItem('channels-show-encrypted') === 'true', false);
});
test('encrypted channels get ch-encrypted CSS class', () => {
// Simulate the rendering logic from channels.js
const ch = { hash: 'enc_A1B2', name: 'Encrypted (0xA1B2)', encrypted: true, messageCount: 5 };
const isEncrypted = ch.encrypted === true;
const encClass = isEncrypted ? ' ch-encrypted' : '';
const className = 'ch-item' + encClass;
assert.ok(className.includes('ch-encrypted'), 'encrypted channel should have ch-encrypted class');
// Non-encrypted channel should NOT have the class
const ch2 = { hash: 'AABB', name: '#general', encrypted: false };
const encClass2 = ch2.encrypted === true ? ' ch-encrypted' : '';
const className2 = 'ch-item' + encClass2;
assert.ok(!className2.includes('ch-encrypted'), 'non-encrypted channel should not have ch-encrypted class');
});
}
// ===== #690 — Clock Skew UI Tests =====
{
console.log('\n--- Clock Skew UI (roles.js helpers) ---');
const ctx = makeSandbox();
vm.runInContext(fs.readFileSync('public/roles.js', 'utf8'), ctx);
test('formatSkew handles seconds', () => {
assert.strictEqual(ctx.window.formatSkew(30), '+30s');
assert.strictEqual(ctx.window.formatSkew(-45), '-45s');
});
test('formatSkew handles minutes', () => {
assert.strictEqual(ctx.window.formatSkew(154), '+2m 34s');
assert.strictEqual(ctx.window.formatSkew(-900), '-15m 0s');
});
test('formatSkew handles hours', () => {
assert.strictEqual(ctx.window.formatSkew(3661), '+1h 1m');
assert.strictEqual(ctx.window.formatSkew(-55320), '-15h 22m');
});
test('formatSkew handles days', () => {
assert.strictEqual(ctx.window.formatSkew(90000), '+1d 1h');
});
test('formatSkew handles null', () => {
assert.strictEqual(ctx.window.formatSkew(null), '—');
});
test('renderSkewBadge renders correct severity class', () => {
var html = ctx.window.renderSkewBadge('warning', 400);
assert.ok(html.includes('skew-badge--warning'), 'should contain warning class');
assert.ok(html.includes('⏰'), 'should contain clock emoji');
});
test('renderSkewBadge renders ok badge (icon only)', () => {
var html = ctx.window.renderSkewBadge('ok', 10);
assert.ok(html.includes('skew-badge--ok'), 'should contain ok class');
});
test('renderSkewBadge returns empty for null severity', () => {
assert.strictEqual(ctx.window.renderSkewBadge(null, 0), '');
});
test('renderSkewSparkline returns SVG with data points', () => {
var samples = [
{ ts: 1000, skew: 10 },
{ ts: 2000, skew: 20 },
{ ts: 3000, skew: -5 }
];
var svg = ctx.window.renderSkewSparkline(samples, 120, 24);
assert.ok(svg.includes('<svg'), 'should return SVG element');
assert.ok(svg.includes('polyline'), 'should contain polyline');
assert.ok(svg.includes('points='), 'should have points attribute');
});
test('renderSkewSparkline returns empty for insufficient data', () => {
assert.strictEqual(ctx.window.renderSkewSparkline([], 120, 24), '');
assert.strictEqual(ctx.window.renderSkewSparkline([{ ts: 1, skew: 5 }], 120, 24), '');
assert.strictEqual(ctx.window.renderSkewSparkline(null, 120, 24), '');
});
test('SKEW_SEVERITY_ORDER sorts worst first', () => {
var order = ctx.window.SKEW_SEVERITY_ORDER;
assert.ok(order.absurd < order.critical, 'absurd should sort before critical');
assert.ok(order.critical < order.warning, 'critical should sort before warning');
assert.ok(order.warning < order.ok, 'warning should sort before ok');
});
}
// ===== analytics.js: hashStatCardsHtml collision clickability (#757) =====
console.log('\n=== analytics.js: hashStatCardsHtml collision details ===');
{
function makeAnalyticsSandbox757() {
const ctx = makeSandbox();
loadInCtx(ctx, 'public/roles.js');
loadInCtx(ctx, 'public/app.js');
try { loadInCtx(ctx, 'public/analytics.js'); } catch (e) {
for (const k of Object.keys(ctx.window)) ctx[k] = ctx.window[k];
}
return ctx;
}
const ctx = makeAnalyticsSandbox757();
const hashStatCardsHtml = ctx.window._analyticsHashStatCardsHtml;
test('hashStatCardsHtml is exposed', () => assert.ok(hashStatCardsHtml, '_analyticsHashStatCardsHtml must be exposed'));
test('collision count > 0 renders clickable card with onclick', () => {
const html = hashStatCardsHtml(100, 50, '3-byte', 16777216, 48, 3);
assert.ok(html.includes('onclick='), 'should have onclick when collisions > 0');
assert.ok(html.includes('collisionRiskSection'), 'should scroll to collisionRiskSection');
assert.ok(html.includes('cursor:pointer'), 'should show pointer cursor');
assert.ok(html.includes('▼'), 'should show expand indicator');
});
test('collision count 0 renders non-clickable card', () => {
const html = hashStatCardsHtml(100, 50, '1-byte', 256, 48, 0);
assert.ok(!html.includes('onclick='), 'should not have onclick when collisions = 0');
assert.ok(!html.includes('cursor:pointer'), 'should not show pointer cursor');
});
}
// ===== analytics.js: renderCollisionsFromServer node links (#757) =====
console.log('\n=== analytics.js: renderCollisionsFromServer collision table ===');
{
function makeAnalyticsSandbox757b() {
const ctx = makeSandbox();
const collisionListEl = { innerHTML: '', querySelectorAll: () => [] };
const origGetById = ctx.document.getElementById;
ctx.document.getElementById = (id) => {
if (id === 'collisionList') return collisionListEl;
return origGetById ? origGetById(id) : null;
};
ctx.window.document = ctx.document;
loadInCtx(ctx, 'public/roles.js');
loadInCtx(ctx, 'public/app.js');
try { loadInCtx(ctx, 'public/analytics.js'); } catch (e) {
for (const k of Object.keys(ctx.window)) ctx[k] = ctx.window[k];
}
ctx._collisionListEl = collisionListEl;
return ctx;
}
const ctx = makeAnalyticsSandbox757b();
const renderCollisions = ctx.window._analyticsRenderCollisionsFromServer;
test('renderCollisionsFromServer is exposed', () => assert.ok(renderCollisions, '_analyticsRenderCollisionsFromServer must be exposed'));
test('renders collision table with node links to correct pubkey', () => {
const sizeData = {
collisions: [
{
prefix: 'A3F2C1',
byte_size: 3,
appearances: 2,
nodes: [
{ public_key: 'abc123def456', name: 'Mountain Repeater', role: 'repeater', lat: 34.0, lon: -118.0 },
{ public_key: 'def456abc789', name: 'Valley Node', role: 'repeater', lat: 34.5, lon: -118.5 }
],
max_dist_km: 45.2,
classification: 'local',
with_coords: 2
}
]
};
renderCollisions(sizeData, 3);
const html = ctx._collisionListEl.innerHTML;
assert.ok(html.includes('A3F2C1'), 'should show prefix');
assert.ok(html.includes('#/nodes/abc123def456'), 'first node link should point to correct pubkey');
assert.ok(html.includes('#/nodes/def456abc789'), 'second node link should point to correct pubkey');
assert.ok(html.includes('Mountain Repeater'), 'should show first node name');
assert.ok(html.includes('Valley Node'), 'should show second node name');
});
test('renders no-collision message when collisions empty', () => {
const sizeData = { collisions: [] };
renderCollisions(sizeData, 3);
const html = ctx._collisionListEl.innerHTML;
assert.ok(html.includes('No 3-byte prefix collisions'), 'should show no-collision message');
});
}
// ===== Observer role support (#753 / PR #774) =====
{
console.log('\n--- Observer role support (PR #774) ---');
// Test 1: ROLE_COLORS.observer is defined and not empty
test('ROLE_COLORS.observer is defined and not empty', () => {
const rolesJs = fs.readFileSync(__dirname + '/public/roles.js', 'utf8');
const ctx = makeSandbox();
vm.runInNewContext(rolesJs, ctx);
assert.ok(ctx.window.ROLE_COLORS.observer, 'ROLE_COLORS.observer should be defined');
assert.ok(ctx.window.ROLE_COLORS.observer.length > 0, 'ROLE_COLORS.observer should not be empty');
});
// Test 2: Observer checkbox exists in neighbor graph filter (unchecked by default)
test('Observer checkbox exists in neighbor graph filter section', () => {
const analyticsJs = fs.readFileSync(__dirname + '/public/analytics.js', 'utf8');
// The observer checkbox is added with data-role="observer" and NO "checked" attribute
assert.ok(analyticsJs.includes('data-role="observer"'), 'analytics.js should contain observer checkbox');
// Verify it's NOT checked by default (no "checked" attribute on observer checkbox)
const observerCheckboxMatch = analyticsJs.match(/data-role="observer"[^>]*>/);
assert.ok(observerCheckboxMatch, 'observer checkbox markup must exist');
assert.ok(!observerCheckboxMatch[0].includes('checked'), 'observer checkbox should NOT be checked by default');
});
// Test 3: Other role checkboxes ARE checked by default
test('Non-observer role checkboxes are checked by default', () => {
const analyticsJs = fs.readFileSync(__dirname + '/public/analytics.js', 'utf8');
// The main role loop uses "checked" attribute
const mainRoleCheckbox = analyticsJs.match(/data-role="\$\{r\}"[^>]*checked/);
assert.ok(mainRoleCheckbox, 'Main role checkboxes should have checked attribute');
});
// Test 4: --role-observer CSS variable exists in style.css
test('--role-observer CSS variable exists in style.css', () => {
const css = fs.readFileSync(__dirname + '/public/style.css', 'utf8');
assert.ok(css.includes('--role-observer:'), 'style.css should define --role-observer CSS variable');
});
// Test 5: Filter logic does NOT auto-include observer role
test('Filter logic excludes observer nodes when checkbox unchecked', () => {
const analyticsJs = fs.readFileSync(__dirname + '/public/analytics.js', 'utf8');
// Old code had: return checkedRoles.has(role) || role === 'unknown' || role === 'observer';
// New code: return checkedRoles.has(role) || role === 'unknown';
// Verify observer is NOT given special pass-through treatment
const filterLine = analyticsJs.match(/return checkedRoles\.has\(role\)[^;]+;/);
assert.ok(filterLine, 'filter line must exist');
assert.ok(!filterLine[0].includes("'observer'"), 'filter should NOT auto-include observer role');
assert.ok(filterLine[0].includes("'unknown'"), 'filter should still auto-include unknown role');
});
}
// ===== Neighbor Graph Min Score Slider Persistence =====
{
console.log('\n--- Neighbor Graph Slider Persistence ---');
test('default slider value is 70 (0.70)', () => {
// Read the raw HTML from analytics.js to verify default
const src = fs.readFileSync('public/analytics.js', 'utf8');
assert.ok(src.includes('value="70"'), 'ngMinScore input should default to value="70"');
assert.ok(src.includes('>0.70</span>'), 'ngMinScoreVal should display 0.70');
});
test('localStorage read on load is present in code', () => {
const src = fs.readFileSync('public/analytics.js', 'utf8');
assert.ok(src.includes("localStorage.getItem('ng-min-score')"), 'should read ng-min-score from localStorage on load');
});
test('localStorage write on slider change is present in code', () => {
const src = fs.readFileSync('public/analytics.js', 'utf8');
assert.ok(src.includes("localStorage.setItem('ng-min-score'"), 'should write ng-min-score to localStorage on change');
});
}
// ===== SUMMARY =====
Promise.allSettled(pendingTests).then(() => {
console.log(`\n${'═'.repeat(40)}`);