mirror of
https://github.com/Kpa-clawbot/meshcore-analyzer.git
synced 2026-09-27 01:48:13 +00:00
e857e0b1ce41c716f693fd2b8b4eccf767b37588
1430
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Kpa-clawbot
|
e857e0b1ce | ci: update go-server-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
9da7c71cc5 | ci: update go-ingestor-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
03484ea38d | ci: update frontend-tests.json [skip ci] | ||
|
Kpa-clawbot
|
27af4098e6 | ci: update frontend-coverage.json [skip ci] | ||
|
Kpa-clawbot
|
474023b9b7 | ci: update e2e-tests.json [skip ci] | ||
|
Kpa-clawbotandyou
|
f4484adb52 |
ci: move to GitHub-hosted runners, disable staging deploy (#908)
## Why The Azure staging VM (`meshcore-vm`) is offline. Self-hosted runners are unavailable, blocking all CI. ## What changed (per job) | Job | Change | Revert | |-----|--------|--------| | `e2e-test` | `runs-on: [self-hosted, Linux]` → `ubuntu-latest`; removed self-hosted-specific "Free disk space" step | Change `runs-on` back to `[self-hosted, Linux]`, restore disk cleanup step | | `build-and-publish` | `runs-on: [self-hosted, meshcore-runner-2]` → `ubuntu-latest`; removed "Free disk space" prune step (noop on fresh GH-hosted runners) | Change `runs-on` back, restore prune step | | `deploy` | `if: false # disabled` (was `github.event_name == 'push'`); `runs-on` kept as-is | Change `if:` back to `github.event_name == 'push'` | | `publish` | `runs-on: [self-hosted, Linux]` → `ubuntu-latest`; `needs: [deploy]` → `needs: [build-and-publish]` | Change both back | ## Notes - `go-test` and `release-artifacts` were already on `ubuntu-latest` — untouched. - The `deploy` job is disabled via `if: false` for trivial one-line revert when the VM returns. - No new `setup-*` actions were needed — `setup-node`, `setup-go`, `docker/setup-buildx-action`, and `docker/login-action` were already present. Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
a47fe26085 |
fix(channels): allow removing user-added keys for server-known channels (#898)
## Problem Adding a channel key in the Channels UI for a channel the server already knows about (e.g. `#public` from rainbow / config) leaves the localStorage entry **unremovable**: - `mergeUserChannels` sees the name already exists in the channel list and skips the user entry. - The existing channel row is never marked `userAdded:true`. - The ✕ button (`[data-remove-channel]`) is only rendered for `userAdded` rows. - Result: stuck localStorage key, no UI to delete it. There was also a latent bug in the remove handler — for non-`user:` rows, it used the raw hash (e.g. `enc_11`) as the `ChannelDecrypt.removeKey()` argument, but the storage key is the channel **name**. ## Fix 1. **`mergeUserChannels`**: when a stored key matches an existing channel by name/hash, mark the existing channel `userAdded=true` so the ✕ renders on it. (No magical/auto deletion of stored keys — the user explicitly chooses to remove.) 2. **Remove handler**: - Look up the channel object to get the correct display name for the localStorage key. - Keep server-known channels in the list when their ✕ is clicked (only the user's localStorage entry + cache are cleared, `userAdded` is unset). The channel still exists upstream. - Pure `user:`-prefixed channels are removed from the list as before. ## Repro 1. Open Channels. 2. Add a key for `#public` (or any rainbow-known channel). 3. Reload. Before this PR: row has no ✕, key is stuck. After this PR: ✕ appears, click clears the local key and cache. ## Files - `public/channels.js` only. ## Notes - No backend changes. - No new APIs. - Behaviour for purely user-added channels (e.g. `user:#somechannel` not known to the server) is unchanged. --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
abd9c46aa7 |
fix: side-panel Details button opens full-screen on desktop (#892)
## Symptom 🔍 Details button in the nodes side panel does nothing on click. ## Root cause (4th regression of the same shape) - Row click → `selectNode()` → `history.replaceState(null, '', '#/nodes/' + pk)` - Details button click → `location.hash = '#/nodes/' + pk` - Hash is already that value → assignment is a no-op → no `hashchange` event → no router → panel stays open. ## Fix Mirror the analytics-link branch already inside the panel click handler: `destroy()` then `init(appEl, pubkey)` directly (which hits the `directNode` full-screen branch unconditionally). Also `replaceState` to keep the URL in sync. ## Test New Playwright E2E: open side panel via row click, click Details, assert `.node-fullscreen` appears. ## Why this keeps regressing Every time we tighten the row-click handler to use `replaceState` (correct — avoids hashchange flicker), the button-click handler that uses `location.hash` becomes a no-op for the same pubkey. Need to remember they're coupled. Worth a follow-up to extract a `navigateToNode(pk)` helper that always works regardless of current hash state — filing as #890-followup if not already there. Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
6ca5e86df6 |
fix: compute hex-dump byte ranges client-side from per-obs raw_hex (#891)
## Symptom The colored byte strip in the packet detail pane is offset from the labeled byte breakdown below it. Off by N bytes where N is the difference between the top-level packet's path length and the displayed observation's path length. ## Root cause Server computes `breakdown.ranges` once from the top-level packet's raw_hex (in `BuildBreakdown`) and ships it in the API response. After #882 we render each observation's own raw_hex, but we keep using the top-level breakdown — so a 7-hop top-level packet shipped "Path: bytes 2-8", and when we rendered an 8-hop observation we coloured 7 of the 8 path bytes and bled into the payload. The labeled rows below (which use `buildFieldTable`) parse the displayed raw_hex on the client, so they were correct — they just didn't match the strip above. ## Fix Port `BuildBreakdown()` to JS as `computeBreakdownRanges()` in `app.js`. Use it in `renderDetail()` from the actually-rendered (per-obs) raw_hex. ## Test Manually verified the JS function output matches the Go implementation for FLOOD/non-transport, transport, ADVERT, and direct-advert (zero hops) cases. Closes nothing (caught in post-tag bug bash). --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
56ec590bc4 |
fix(#886): derive path_json from raw_hex at ingest (#887)
## Problem Per-observation `path_json` disagrees with `raw_hex` path section for TRACE packets. **Reproducer:** packet `af081a2c41281b1e`, observer `lutin🏡` - `path_json`: `["67","33","D6","33","67"]` (5 hops — from TRACE payload) - `raw_hex` path section: `30 2D 0D 23` (4 bytes — SNR values in header) ## Root Cause `DecodePacket` correctly parses TRACE packets by replacing `path.Hops` with hop IDs from the payload's `pathData` field (the actual route). However, the header path bytes for TRACE packets contain **SNR values** (one per completed hop), not hop IDs. `BuildPacketData` used `decoded.Path.Hops` to build `path_json`, which for TRACE packets contained the payload-derived hops — not the header path bytes that `raw_hex` stores. This caused `path_json` and `raw_hex` to describe completely different paths. ## Fix - Added `DecodePathFromRawHex(rawHex)` — extracts header path hops directly from raw hex bytes, independent of any TRACE payload overwriting. - `BuildPacketData` now calls `DecodePathFromRawHex(msg.Raw)` instead of using `decoded.Path.Hops`, guaranteeing `path_json` always matches the `raw_hex` path section. ## Tests (8 new) **`DecodePathFromRawHex` unit tests:** - hash_size 1, 2, 3, 4 - zero-hop direct packets - transport route (4-byte transport codes before path) **`BuildPacketData` integration tests:** - TRACE packet: asserts path_json matches raw_hex header path (not payload hops) - Non-TRACE packet: asserts path_json matches raw_hex header path All existing tests continue to pass (`go test ./...` for both ingestor and server). Fixes #886 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
67aa47175f |
fix: path pill and byte breakdown agree on hop count (#885)
## Problem On the packet detail pane, the **path pill** (top) and the **byte breakdown** (bottom) showed different numbers of hops for the same packet. Example: `46cf35504a21ef0d` rendered as `1 hop` badge followed by 8 node names in the path pill, while the byte breakdown listed only 1 hop row. ## Root cause Mixed data sources: - Path-pill badge used `(raw_hex path_len) & 0x3F` (= firmware truth for one observer = 1) - Path-pill names used `path_json.length` (= server-aggregated longest path across observers = 8) - Byte breakdown section header used `(raw_hex path_len) & 0x3F` (= 1) - Byte breakdown rows were sliced from `raw_hex` (= 1 row) - `renderPath(pathHops, ...)` iterated all `path_json` entries For group-header view, `packet.path_json` is aggregated across observers and therefore longer than the raw_hex of any single observer's packet. ## Fix Both surfaces now render from `pathHops` (= effective observation's `path_json`). The raw_hex vs path_json mismatch is still logged as a console.warn for diagnostics, but does not drive the UI. With per-observation `raw_hex` (#882) shipped, clicking an observation row already swaps the effective packet so both surfaces stay consistent. ## Testing - Adds E2E regression `Packet detail path pill and byte breakdown agree on hop count` that asserts: 1. `pill badge count == byte breakdown section count` 2. `rendered hop names ≈ badge count` (within 1 for separators) 3. `byte breakdown rendered rows == section count` - Manually reproduced on staging with `46cf35504a21ef0d` (8-name path + `1 hop` badge before fix). Related: #881 #882 #866 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
2b9f305698 |
fix(#874): hop-resolver affinity picker — score candidates by neighbor-graph edges + geographic centroid (#876)
## Problem `pickByAffinity` in `hop-resolver.js` picks wrong regional candidates when 1-byte pubkey prefixes collide. The old implementation only considers one adjacent hop (forward OR backward pass), leading to suboptimal picks when both neighbors provide useful context. Measured on staging: **61.6% of hops have ≥2 same-prefix candidates**, making collision resolution critical. ## Fix Replaced the separate forward/backward pass disambiguation with a **combined iterative resolver** that scores candidates against BOTH prev and next resolved hops: 1. **Neighbor-graph edge weight** (priority 1): Sum edge scores to prev + next pubkeys. Pick max sum. 2. **Geographic centroid** (priority 2): Average lat/lon of prev + next positions. Pick closest candidate by haversine distance. 3. **Single-anchor geo** (priority 3): When only one neighbor is resolved, use it directly. 4. **Fallback** (priority 4): First candidate when no context exists. The iterative approach resolves cascading dependencies — resolving one ambiguous hop may unlock context for its neighbors. ### Dev-mode trace Multi-candidate picks now emit: `[hop-resolver] hash=46 candidates=N scored=[...] chose=<pubkey> method=graph|centroid|fallback` ## Before/After (staging, 1539 packets, 12928 hops) | Metric | Before | After | |--------|--------|-------| | Unreliable hops | 39 (0.3%) | 23 (0.2%) | | Packets with unreliable | 33 (2.14%) | 17 (1.10%) | ~41% reduction in unreliable hops, ~48% reduction in affected packets. ## Tests 5 new tests in `test-frontend-helpers.js`: - Graph edge scoring picks correct regional candidate - Next hop breaks tie when prev has no edges - Centroid fallback when no graph edges exist - Centroid uses average of prev+next positions - Fallback when no context at all All 595 tests pass. No regressions in `test-packet-filter.js` (62 pass) or `test-aging.js` (29 pass). Closes #874 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
a605518d6d |
fix(#881): per-observation raw_hex — each observer sees different bytes on air (#882)
## Problem Each MeshCore observer receives a physically distinct over-the-air byte sequence for the same transmission (different path bytes, flags/hops remaining). The `observations` table stored only `path_json` per observer — all observations pointed at one `transmissions.raw_hex`. This prevented the hex pane from updating when switching observations in the packet detail view. ## Changes | Layer | Change | |-------|--------| | **Schema** | `ALTER TABLE observations ADD COLUMN raw_hex TEXT` (nullable). Migration: `observations_raw_hex_v1` | | **Ingestor** | `stmtInsertObservation` now stores per-observer `raw_hex` from MQTT payload | | **View** | `packets_v` uses `COALESCE(o.raw_hex, t.raw_hex)` — backward compatible with NULL historical rows | | **Server** | `enrichObs` prefers `obs.RawHex` when non-empty, falls back to `tx.RawHex` | | **Frontend** | No changes — `effectivePkt.raw_hex` already flows through `renderDetail` | ## Tests - **Ingestor**: `TestPerObservationRawHex` — two MQTT packets for same hash from different observers → both stored with distinct raw_hex - **Server**: `TestPerObservationRawHexEnrich` — enrichObs returns per-obs raw_hex when present, tx fallback when NULL - **E2E**: Playwright assertion in `test-e2e-playwright.js` for hex pane update on observation switch E2E assertion added: `test-e2e-playwright.js:1794` ## Scope - Historical observations: raw_hex stays NULL, UI falls back to transmission raw_hex silently - No backfill, no path_json reconstruction, no frontend changes Closes #881 --------- Co-authored-by: you <you@example.com> |
||
|
|
0ca559e348 |
fix(#866): per-observation children in expanded packet groups (#880)
## Problem
When a packet group is expanded in the Packets table, clicking any child
row pointed the side pane at the same aggregate packet — not the clicked
observation. URL would flip between `?obs=<packet_id>` values instead of
real observation ids.
## Root cause
The expand fetch used `/api/packets?hash=X&limit=20`, which returns ONE
aggregate row keyed by packet.id. Every child therefore carried
`data-value=<packet.id>`.
## Fix
Switch the expand fetch to `/api/packets/<hash>`, which includes the
full `observations[]` array. Build `_children` as `{...pkt, ...obs}` so
each child row gets a unique observation id and observation-level fields
(observer, path, timestamp, snr/rssi) override the aggregate.
## Verified live on staging
Tested on multiple packets:
- Click group-header → side pane shows observation 1 of N (first
observer)
- Click child row → pane updates to show THAT observer's details:
observer name, path, timestamp, obs counter (K of N), URL
`?obs=<observation_id>`
## Tests
592 frontend tests pass (no new ones — this is a wiring fix, live
E2E-verified instead).
Closes #866
---------
Co-authored-by: Kpa-clawbot <agent@corescope.local>
Co-authored-by: you <you@example.com>
|
||
|
Kpa-clawbotandyou
|
1d449eabc7 |
fix(#872): replace strikethrough with warning badge on unreliable hops (#875)
## Problem The `hop-unreliable` CSS class applied `text-decoration: line-through` and `opacity: 0.5`, making hop names look "dead" to operators. This caused confusion — the repeater itself is fine, only the name→hash assignment is uncertain. ## Fix - **CSS**: Removed `line-through` and heavy opacity from `.hop-unreliable`. Kept subtle `opacity: 0.85` for scanability. Added `.hop-unreliable-btn` style for the new badge. - **JS**: Added a `⚠️` warning badge button next to unreliable hops (similar pattern to existing conflict badges). The badge is always visible, keyboard-focusable, and has both `title` and `aria-label` with an informative tooltip explaining geographic inconsistency. - **Tests**: Added 2 tests in `test-frontend-helpers.js` asserting the badge renders for unreliable hops and does NOT render for reliable ones, and that no `line-through` is present. ### Before → After | Before | After | |--------|-------| | ~~NodeName~~ (struck through, 50% opacity) | NodeName ⚠️ (normal text, small warning badge with tooltip) | ## Scope Resolver logic untouched — #873 covers threshold tuning, #874 covers picker correctness. No candidate-dropdown UX (follow-up per issue discussion). Closes #872 Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
42ff5a291b |
fix(#866): full-page obs-switch — update hex + path + direction per observation (#870)
## Problem On `/#/packets/<hash>?obs=<id>`, clicking a different observation updated summary fields (Observer, SNR/RSSI, Timestamp) but **not** hex payload or path details. Sister bug to #849 (fixed in #851 for the detail dialog). ## Root Causes | Cause | Impact | |-------|--------| | `selectPacket` called `renderDetail` without `selectedObservationId` | Initial render missed observation context on some code paths | | `ObservationResp` missing `direction`, `resolved_path`, `raw_hex` | Frontend obs-switch lost direction and resolved_path context | | `obsPacket` construction omitted `direction` field | Direction not preserved when switching observations | ## Fix - `selectPacket` explicitly passes `selectedObservationId` to `renderDetail` - `ObservationResp` gains `Direction`, `ResolvedPath`, `RawHex` fields - `mapSliceToObservations` copies the three new fields - `obsPacket` spreads include `direction` from the observation ## Tests 7 new tests in `test-frontend-helpers.js`: - Observation switch updates `effectivePkt` path - `raw_hex` preserved from packet when obs has none - `raw_hex` from obs overrides when API provides it - `direction` carried through observation spread - `resolved_path` carried through observation spread - `getPathLenOffset` cross-check for transport routes - URL hash `?obs=` round-trip encoding All 584 frontend + 62 filter + 29 aging tests pass. Go server tests pass. Fixes #866 Co-authored-by: you <you@example.com> |
||
|
|
99029e41aa |
ci(#768): publish multi-arch (amd64+arm64) Docker image (#869)
## Problem
`docker pull` on ARM devices fails because the published image is
amd64-only.
## Fix
Enable multi-arch Docker builds via `docker buildx`. **Builder stage
uses native Go cross-compilation; only the runtime-stage `RUN` steps use
QEMU emulation.**
### Changes
| File | Change |
|------|--------|
| `Dockerfile` | Pin builder stage to `--platform=$BUILDPLATFORM`
(always native), accept `ARG TARGETOS`/`ARG TARGETARCH` from buildx, set
`GOOS=$TARGETOS GOARCH=$TARGETARCH CGO_ENABLED=0` on every `go build` |
| `.github/workflows/deploy.yml` | Add `docker/setup-buildx-action@v3` +
`docker/setup-qemu-action@v3` (latter needed only for runtime-stage
RUNs), set `platforms: linux/amd64,linux/arm64` |
### Build architecture
- **Builder stage** (`FROM --platform=$BUILDPLATFORM
golang:1.22-alpine`) — runs natively on amd64. Go toolchain
cross-compiles the binaries to `$TARGETARCH` via `GOOS/GOARCH`. No
emulation, ~10× faster than emulated builds. Works because
`modernc.org/sqlite` is pure Go (no CGO).
- **Runtime stage** (`FROM alpine:3.20`) — buildx pulls the per-arch
base. RUN steps (`apk add`, `mkdir/chown`, `chmod`) execute inside the
target-arch image, so QEMU is required to interpret arm64 binaries on
the amd64 host. Only a handful of short shell commands run under
emulation, so the QEMU cost is small.
### Verify
After merge, on an ARM device:
```bash
docker pull ghcr.io/kpa-clawbot/corescope:edge
docker inspect ghcr.io/kpa-clawbot/corescope:edge --format '{{.Architecture}}'
# → arm64
```
> First arm64 image appears on the next push to master after this
merges.
Closes #768
---------
Co-authored-by: you <you@example.com>
Co-authored-by: Kpa-clawbot <agent@corescope.local>
|
||
|
Kpa-clawbotandyou
|
c99aa1dadf |
fix(#855, #856, #857) + feat(#862): /nodes detail panel + search improvements (#868)
## Summary Four related `/nodes` page fixes batched to avoid merge conflicts (all touch `public/nodes.js`). --- ### #855 — "Show all neighbors" link doesn't expand **Problem:** The "View all N neighbors →" link in the side panel navigated to the full detail page instead of expanding the truncated list inline. **Fix:** Replaced navigation link with an inline "Show all N neighbors ▼" button that re-renders the neighbor table without the limit. **Acceptance:** Click the button → all neighbors appear in the same panel without page navigation. Closes #855 --- ### #856 — "Details" button is a no-op **Problem:** The "🔍 Details" link in the side panel was an `<a>` tag whose `href` matched the current hash (set by `replaceState`), making clicks a same-hash no-op. **Fix:** Changed from `<a>` link to a `<button>` with a direct click handler that sets `location.hash`, ensuring the router always fires. **Acceptance:** Click "🔍 Details" → navigates to full-screen node detail view. Closes #856 --- ### #857 — Recent Packets shows bullets but no content **Problem:** The "Recent Packets (N)" section could render entries with missing `hash` or `timestamp`, producing colored dots with no meaningful content beside them. **Fix:** Added `.filter(a => a.hash && a.timestamp)` before rendering, and updated the count header to reflect filtered entries only. **Acceptance:** Recent Packets section only shows entries with valid data; count matches visible items. Closes #857 --- ### #862 — Pubkey prefix search on /#/nodes **Problem:** Search box only matched node names. Operators couldn't search by pubkey prefix. **Fix:** Extended search to detect hex-only queries (`/^[0-9a-f]+$/i`) and match them against pubkey prefix (`startsWith`). Non-hex queries continue matching name as before. Both are composable in the same input. **Acceptance:** - Typing `3f` filters to nodes whose pubkey starts with `3f` - Typing `foo` still filters by name - Search placeholder updated to indicate pubkey support 5 new unit tests added for the search matching logic. Closes #862 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
20843979a7 |
fix(#861): restore sticky table headers on mobile packets page (#867)
## What Remove a single line in `makeColumnsResizable()` that set `th.style.position = 'relative'` on every `<th>` except the last, overriding the CSS `position: sticky` rule from `.data-table th`. ## Why The column-resize feature added inline `position: relative` to each header (except the last) to serve as a containing block for the absolute-positioned resize handles. This inadvertently broke `position: sticky` on all headers except "Details" (the last column) — visible on mobile when scrolling the packets table. `position: sticky` is itself a positioned value and serves as a containing block for absolute children, so the resize handles work identically without the override. ## Test - Open `/#/packets` on mobile (or narrow viewport) - Scroll down — ALL column headers now remain sticky at the top - Column resize handles still function correctly on desktop Fixes #861 Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
ea78581eea |
fix(#858): packets/hour chart — bars rendering + x-axis label decimation (#865)
Two bugs in the Overview tab Packets/Hour chart: 1. **Bars not rendering**: `barW` went negative when `data.length` was large (e.g. 720 hours for 30-day range), producing zero-width invisible bars. Fix: `Math.max(1, ...)` floor on bar width. 2. **X-axis labels overlapping**: Every single hour label was emitted (`02h03h04h...`). Fix: decimate labels based on time range — every 6h for ≤24h, every 12h for ≤72h, every 24h beyond. Shows `MM-DD` on midnight boundaries for multi-day ranges. **Scope**: Only touches the Overview tab `Packets / Hour` section and the shared `barChart` floor (one-line change). No modifications to Topology, Channels, Distance, or other tabs. Fixes #858 Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
b5372d6f73 |
fix(#859): remove opacity gradient from Per-Observer Reachability rows (#863)
Fixes #859 ## What The "Per-Observer Reachability" and "Best Path to Each Node" sections in the Topology tab had inline `opacity` styles on each `.reach-ring` row that decreased with hop count (`1 - hops * 0.06`, floored at 0.3). This made text progressively darker/unreadable toward the bottom. ## Fix Removed the inline `opacity:${opacity}` style from both `renderPerObserverReach()` and `renderBestPath()`. The rows now render at full opacity with text colors governed by CSS variables as intended. ## Changed - `public/analytics.js`: removed opacity computation and inline style in two functions (4 lines removed, 2 added) ## Scope Only touches Per-Observer Reachability and Best Path rendering. No changes to Overview, Channels, or shared helpers. Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
5afed0951b |
fix(#860): cap channel timeline chart to top 8 by volume (#864)
## What & Why The "Messages / Hour by Channel" chart on `/#/analytics` Channels tab rendered all channels in both the SVG and legend, causing legend overflow when 20+ channels are present. ## Fix - Sort channels by total message volume (descending) - Render only the top 8 in the chart and legend - Show "+N more" in the legend when channels are truncated - `maxCount` for Y-axis scaling is computed from visible channels only, so the chart uses its full vertical range Single-file change: `public/analytics.js` — only `renderChannelTimeline()` modified. No shared helpers touched. Fixes #860 Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
3630a32310 |
fix(#852): transport-route path_len offset + var(--muted) → var(--text-muted) (#853)
## Problem Two pre-existing bugs found during expert review of #851: ### 1. `hashSize` derivation ignores transport route types `public/packets.js` hardcoded path-length byte at offset 1: ```js const rawPathByte = pkt.raw_hex ? parseInt(pkt.raw_hex.slice(2, 4), 16) : NaN; ``` For transport routes (`route_type` 0 DIRECT or 3 TRANSPORT_ROUTE_FLOOD), bytes 1–4 are `next_hop` + `last_hop` and path-length is at offset 5. Same bug #846 fixed inside the byte-breakdown function. ### 2. `var(--muted)` CSS variable is undefined Used in 6 places in `public/packets.js`. No `--muted` variable is defined anywhere in `public/*.css` — only `--text-muted` exists. Text styled with `var(--muted)` silently falls through to inherited color, making badges/hints invisible. ## Fix ### Fix 1: transport-route path_len offset ```js const plOff = (pkt.route_type === 0 || pkt.route_type === 3) ? 5 : 1; const rawPathByte = pkt.raw_hex ? parseInt(pkt.raw_hex.slice(plOff * 2, plOff * 2 + 2), 16) : NaN; ``` ### Fix 2: `var(--muted)` → `var(--text-muted)` All 6 occurrences replaced. ## Tests (5 new, 572 total) - `hashSize` extraction for flood route (route_type=1, offset 1) - `hashSize` extraction for direct transport route (route_type=0, offset 5) - `hashSize` extraction for transport route flood (route_type=3, offset 5) - `hashSize` returns null for missing raw_hex - Regression guard: no `var(--muted)` in any `public/` JS/CSS file ## Changes - `public/packets.js`: 7 lines changed (1 offset fix + 6 CSS var fixes) - `test-frontend-helpers.js`: 46 lines added (5 tests) Closes #852 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandKpa-clawbot
|
ff05db7367 |
ci: fix staging smoke test port — read STAGING_GO_HTTP_PORT, not hardcoded 82 (#854)
## Problem The "Deploy Staging" job's Smoke Test always fails with `Staging /api/stats did not return engine field`. Root cause: the step hardcodes `http://localhost:82/api/stats`, but `docker-compose.staging.yml:21` publishes the container on `${STAGING_GO_HTTP_PORT:-80}:80`. Default is port 80, not 82. curl gets ECONNREFUSED, `-sf` swallows the error, `grep -q engine` sees empty input → failure. Verified on staging VM: `ss -lntp` shows only `:80` listening; `docker ps` confirms `0.0.0.0:80->80/tcp`. A `curl http://localhost:82` returns connection-refused. ## Fix Read `STAGING_GO_HTTP_PORT` (same default as compose) so the smoke test tracks the port the container was actually launched on. Failure message now includes the resolved port to make future port mismatches self-diagnosing. ## Tested Logic only — the curl + grep pattern is unchanged. If any CI env override sets `STAGING_GO_HTTP_PORT`, the smoke test now follows it. Co-authored-by: Kpa-clawbot <agent@corescope.local> |
||
|
Kpa-clawbotandyou
|
441409203e |
feat(#845): bimodal_clock severity — surface flaky-RTC nodes instead of hiding as 'No Clock' (#850)
## Problem Nodes with flaky RTC (firmware emitting interleaved good and nonsense timestamps) were classified as `no_clock` because the broken samples poisoned the recent median. Operators lost visibility into these nodes — they showed "No Clock" even though ~60% of their adverts had valid timestamps. Observed on staging: a node with 31K samples where recent adverts interleave good skew (-6.8s, -13.6s) with firmware nonsense (-56M, -60M seconds). Under the old logic, median of the mixed window → `no_clock`. ## Solution New `bimodal_clock` severity tier that surfaces flaky-RTC nodes with their real (good-sample) skew value. ### Classification order (first match wins) | Severity | Good Fraction | Description | |----------|--------------|-------------| | `no_clock` | < 10% | Essentially no real clock | | `bimodal_clock` | 10–80% (and bad > 0) | Mixed good/bad — flaky RTC | | `ok`/`warn`/`critical`/`absurd` | ≥ 80% | Normal classification | "Good" = `|skew| <= 1 hour`; "bad" = likely uninitialized RTC nonsense. When `bimodal_clock`, `recentMedianSkewSec` is computed from **good samples only**, so the dashboard shows the real working-clock value (e.g. -7s) instead of the broken median. ### Backend changes - New constant `BimodalSkewThresholdSec = 3600` - New severity `bimodal_clock` in classification logic - New API fields: `goodFraction`, `recentBadSampleCount`, `recentSampleCount` ### Frontend changes - Amber `Bimodal` badge with tooltip showing bad-sample percentage - Bimodal nodes render skew value like ok/warn/severe (not the "No Clock" path) - Warning line below sparkline: "⚠️ X of last Y adverts had nonsense timestamps (likely RTC reset)" ### Tests - 3 new Go unit tests: bimodal (60% good → bimodal_clock), all-bad (→ no_clock), 90%-good (→ ok) - 1 new frontend test: bimodal badge rendering with tooltip - Existing `TestReporterScenario_789` passes unchanged Builds on #789 (recent-window severity). Closes #845 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
a371d35bfd |
feat(#847): dedupe Top Longest Hops by pair + add obs count and SNR cues (#848)
## Problem The "Top 20 Longest Hops" RF analytics card shows the same repeater pair filling most slots because the query sorts raw hop records by distance with no pair deduplication. A single long link observed 12+ times dominates the leaderboard. ## Fix Dedupe by unordered `(pk1, pk2)` pair. Per pair, keep the max-distance record and compute reliability metrics: | Column | Description | |--------|-------------| | **Obs** | Total observations of this link | | **Best SNR** | Maximum SNR seen (dB) | | **Median SNR** | Median SNR across all observations (dB) | Tooltip on each row shows the timestamp of the best observation. ### Before | # | From | To | Distance | Type | SNR | Packet | |---|------|----|----------|------|-----|--------| | 1 | NodeX | NodeY | 200 mi | R↔R | 5 dB | abc… | | 2 | NodeX | NodeY | 199 mi | R↔R | 6 dB | def… | | 3 | NodeX | NodeY | 198 mi | R↔R | 4 dB | ghi… | ### After | # | From | To | Distance | Type | Obs | Best SNR | Median SNR | Packet | |---|------|----|----------|------|-----|----------|------------|--------| | 1 | NodeX | NodeY | 200 mi | R↔R | 12 | 8.0 dB | 5.2 dB | abc… | | 2 | NodeA | NodeB | 150 mi | C↔R | 3 | 6.5 dB | 6.5 dB | jkl… | ## Changes - **`cmd/server/store.go`**: Group `filteredHops` by unordered pair key, accumulate obs count / best SNR / median SNR per group, sort by max distance, take top 20 - **`cmd/server/types.go`**: Update `DistanceHop` struct — replace `SNR` with `BestSnr`, `MedianSnr`, add `ObsCount` - **`public/analytics.js`**: Replace single SNR column with Obs, Best SNR, Median SNR; add row tooltip with best observation timestamp - **`cmd/server/store_tophops_test.go`**: 3 unit tests — basic dedupe, reverse-pair merge, nil SNR edge case ## Test Coverage - `TestDedupeTopHopsByPair`: 5 records on pair (A,B) + 1 on (C,D) → 2 results, correct obsCount/dist/bestSnr/medianSnr - `TestDedupeTopHopsReversePairMerges`: (B,A) and (A,B) merge into one entry - `TestDedupeTopHopsNilSNR`: all-nil SNR records → bestSnr and medianSnr both nil - Existing `TestAnalyticsRFEndpoint` and `TestAnalyticsRFWithRegion` still pass Closes #847 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
7c01a97178 |
fix(#849): Packet Detail dialog — show exact clicked observation, not cross-observer aggregate (#851)
## Problem The Packet Detail dialog summary (Observer, Path, Hops, SNR/RSSI, Timestamp) used the **aggregated cross-observer view** (`_parsedPath` / `getParsedPath(pkt)`), which contradicted the byte breakdown after #844. A packet observed with 2 hops by one observer would show "Path: 7 hops" in the summary because it merged all observers' paths. ## Fix The dialog is now **per-observation**: - `renderDetail` resolves a `currentObservation` from `selectedObservationId` (set when clicking an observation child row) or defaults to `observations[0]` - All summary fields read from the current observation: Observer, SNR/RSSI, Timestamp, Path, Direction - Hop count badge comes from `path_len & 0x3F` of the observation's `raw_hex` (firmware truth, same source as byte breakdown). Cross-checked against `path_json` length — logs a console warning on mismatch - **Observations table** rendered inside the detail panel when multiple observations exist. Clicking a row updates `currentObservation` and re-renders the summary in-place (no dialog close/reopen) - `.observation-current` CSS class highlights the selected observation row ### Cross-observer aggregate (Option B) A read-only "Cross-observer aggregate" section below the observations table shows the longest observed path across all observers. This is **not** the default view — it's always visible as secondary context. ## Tests 8 new tests in `test-frontend-helpers.js`: - Hop count extraction from raw_hex (normal, direct, transport route types) - Inconsistency detection between path_json and raw_hex - Per-observation field override of aggregated packet fields - First observation used when no specific observation selected - Observation row click selects that observation - Null/missing raw_hex handling All 572 tests pass (564 frontend + 62 filter + 29 aging). ## Acceptance - Summary shows per-observation path/hops/SNR/RSSI/timestamp - Switching observations in the detail updates everything - Cross-observer aggregate available as secondary section - Byte breakdown untouched (owned by #846) ## Related - Closes #849 - Related: #844 (#846) — byte breakdown fix (separate PR, different code region) --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
f1eea9ee3c |
fix(#844): Packet Byte Breakdown — derive hop count from path_len, not aggregated _parsedPath (#846)
## Problem The Packet Detail dialog's "Packet Byte Breakdown" section was using the aggregated `_parsedPath` (longest path observed across all observers) to render hop entries, instead of deriving the hop count from the `path_len` byte in `raw_hex`. This caused: - Wrong hop count (e.g., "Path (7 hops)" when `raw_hex` only contains 2) - Hop values from the aggregated path displayed at incorrect byte offsets - Subsequent fields (pubkey, timestamp, signature) rendered at wrong offsets because `off` was advanced by the wrong amount ## Fix In `buildFieldTable()` (packets.js), the Path section now: 1. Derives `hashCountVal` from `path_len & 0x3F` (firmware truth per `Packet.h:79-83`) 2. Derives `hashSize` from `(path_len >> 6) + 1` 3. Reads each hop's hex value directly from `raw_hex` at the correct byte offset 4. Advances `off` by `hashSize * hashCountVal` 5. Skips the Path section entirely when `hashCountVal === 0` (direct advert) The "Path" summary section above the breakdown (which uses the aggregated path for route visualization) is unchanged — only the byte breakdown is fixed. ## Tests 3 new tests in `test-frontend-helpers.js`: - Verifies 2 hops rendered (not 7) when `path_len=0x42` despite 7-hop aggregated path - Verifies pubkey offset is 6 (not 16) after a 2-hop path - Verifies direct advert (`hashCount=0`) skips Path section Also fixed pre-existing `HopDisplay is not defined` failures in the `#765` transport offset test sandbox (added mock). All 559 tests pass. Closes #844 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandKpa-clawbot
|
f30e6bef28 |
qa(plan): reconcile §8.2/§5.3/§6.2 + add §8.7 (Recent Packets readability) (#838)
Doc-only reconciliation of v3.6.0-rc plan with what actually shipped. ## Changes - **§8.2** — desktop deep link now opens full-screen view (post-#823/#824), not split panel as the plan still asserted. - **§5.3** — pin that severity now derives from `recentMedianSkewSec` (#789), not the all-time `medianSkewSec` — a re-tester needs to know which field drives the badge. - **§6.2** — pin the existing observer-graph element location (`public/analytics.js:2048-2051`). - **New §8.7** — side-panel "Recent Packets" entries must navigate to a valid packet detail (DB-fallback per #827) AND text must be readable in the current theme (explicit color per #829). Both bugs surfaced this session. No CI gates. Co-authored-by: Kpa-clawbot <agent@corescope.local> |
||
|
Kpa-clawbotandKpa-clawbot
|
20f456da58 |
fix(#840): map popup 'Show Neighbors' link does nothing on iOS Safari (#841)
Closes #840 ## What Switch the map-popup "Show Neighbors" link from `<a href="#">` to `<a href="javascript:void(0)" role="button">` so iOS Safari doesn't navigate when the document-level click delegation fails to fire. ## Why On iOS Safari, when a user taps the link inside a Leaflet popup: - The document-level click delegation at `public/map.js:927` calls `e.preventDefault()` and triggers `selectReferenceNode`. - BUT inside a Leaflet popup, `L.DomEvent.disableClickPropagation()` is internally applied to popup content — on iOS Safari the click sometimes doesn't bubble to `document`. - When that happens, the browser's default `<a href="#">` action runs: - hash becomes empty (`#`) - `navigate()` in `app.js:458` sees empty hash → defaults to `'packets'` - map page is destroyed mid-tap → user perceives "nothing happened" (or a brief flash if they back-button) `href="javascript:void(0)"` removes the navigation fall-through entirely. The `role="button"` keeps a11y semantics, `cursor:pointer` keeps the visual cue. ## Tested - Headless Chromium desktop + iPhone 13 emulation: tap fires `/api/nodes/{pk}/neighbors?min_count=3`, marker count drops from 441 → 44, `#mcNeighbors` checkbox toggles on, URL stays on `/#/map`. Same as before. - Frontend helpers: 556/0 - Real iOS Safari fix verification needs a physical-device test post-deploy ## Out of scope (follow-up) - Same `<a href="#">` pattern exists for the topright "Close route" control at `public/map.js:389` — uses `L.DomEvent.preventDefault` so should work, but worth auditing if the symptom recurs. Co-authored-by: Kpa-clawbot <agent@corescope.local> |
||
|
Kpa-clawbotandKpa-clawbot
|
e31e14cae9 |
qa(plan): apply v3.6.0-rc QA findings (#832/#833/#836) (#837)
Apply v3.6.0-rc QA learnings to the plan. ## Changes - **§1.1** — 1 GB cap is unrealistic on real DBs without `GOMEMLIMIT` + bounded cold-load. Raised target to 3 GB and pointed to follow-up **#836**. (Investigation showed cold-load transient blows past any sub-2GB cap regardless of `maxMemoryMB` setting because `runtime.MemStats.NextGC` ignores cgroup ceilings.) - **§1.4** — `trackedBytes`/`trackedMB` is in-store packet bytes only and under-reports RSS by ~3–5× (no indexes, caches, runtime overhead, cgo). Switched the assertion to use `processRSSMB` exposed by **#832** (PR **#835**). - **§11.1** — noted the Playwright deep-link E2E assertion was updated by **#833** (PR **#834**) to match the post-#823 full-screen behavior. ## Why Three real findings from the QA ops sweep ([§1.4 fail comment](https://github.com/Kpa-clawbot/CoreScope/issues/809#issuecomment-4286113141)). Updating the plan so the next run doesn't replay the same false-fail/false-pass conditions. Co-authored-by: Kpa-clawbot <agent@corescope.local> |
||
|
Kpa-clawbotandyou
|
bb0f816a6b |
fix(channels): only show lock for confirmed-encrypted #channel deep links (#825) (#826)
Closes #825 ## Root cause PR #815 added a `#`-prefix branch in `selectChannel` that unconditionally rendered the lock affordance whenever the channel object wasn't in the loaded `channels` list. With the encrypted toggle off, unencrypted channels like `#test` are also absent from the list, so the new branch wrongly locked them instead of falling through to the REST fetch. ## Fix When no stored key matches, refetch `/channels?includeEncrypted=true` and check `ch.encrypted` before locking. Only render the lock when we positively know the channel is encrypted; otherwise fall through to the existing REST messages fetch. This regresses #815's behavior **only for the unencrypted case** (which is the bug). The encrypted-no-key (#811) and encrypted-with-stored-key (#815) paths are preserved. ## Tests 3 new regression tests in `test-frontend-helpers.js`: - `#test` (unencrypted) deep link → REST fetched, no lock - `#private` (encrypted, no key) deep link → lock, no REST (#811 preserved) - `#private` (encrypted, with stored key) deep link → decrypt path (#815 preserved) `node test-frontend-helpers.js` → 556 passed, 0 failed. ## Perf One extra REST call per cold deep link to a `#`-named channel that's not in the toggle-off list — same endpoint already cached via `CLIENT_TTL.channels`, so subsequent navigations are free. --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandMeshCore Agent
|
3f26dc7190 |
obs: surface real RSS alongside tracked store bytes in /api/stats (#832) (#835)
Closes #832. ## Root cause confirmed \`trackedMB\` (\`s.trackedBytes\` in \`store.go\`) only sums per-packet struct + payload sizes recorded at insertion. It excludes the index maps (\`byHash\`, \`byTxID\`, \`byNode\`, \`byObserver\`, \`byPathHop\`, \`byPayloadType\`, hash-prefix maps, name lookups), the analytics LRUs (rfCache/topoCache/hashCache/distCache/subpathCache/chanCache/collisionCache), WS broadcast queues, and Go runtime overhead. It's \"useful packet bytes,\" not RSS — typically 3–5× off on staging. ## Fix (Option C from the issue) Expose four memory fields on \`/api/stats\` from a single cached snapshot: | Field | Source | Semantics | |---|---|---| | \`storeDataMB\` | \`s.trackedBytes\` | in-store packet bytes; eviction watermark input | | \`goHeapInuseMB\` | \`runtime.MemStats.HeapInuse\` | live Go heap | | \`goSysMB\` | \`runtime.MemStats.Sys\` | total Go-managed memory | | \`processRSSMB\` | \`/proc/self/status VmRSS\` (Linux), falls back to \`goSysMB\` | what the kernel sees | \`trackedMB\` is retained as a deprecated alias for \`storeDataMB\` so existing dashboards/QA scripts keep working. Field invariants are documented on \`MemorySnapshot\`: \`processRSSMB ≥ goSysMB ≥ goHeapInuseMB ≥ storeDataMB\` (typical). ## Performance Single \`getMemorySnapshot\` call cached for 1s — \`runtime.ReadMemStats\` (stop-the-world) and the \`/proc/self/status\` read are amortized across burst polling. \`/proc\` read is bounded to 8 KiB, parsed with \`strconv\` only — no shell-out, no untrusted input. \`cgoBytesMB\` is omitted: the build uses pure-Go \`modernc.org/sqlite\`, so there is no cgo allocator to measure. Documented in code comment. ## Tests \`cmd/server/stats_memory_test.go\` asserts presence, types, sign, and ordering invariants. Avoids the flaky \"matches RSS to ±X%\" pattern. \`\`\` $ go test ./... -count=1 -timeout 180s ok github.com/corescope/server 19.410s \`\`\` ## QA plan §1.4 now compares \`processRSSMB\` against procfs RSS (the right invariant); threshold stays at 0.20. --------- Co-authored-by: MeshCore Agent <meshcore-agent@openclaw.local> |
||
|
Kpa-clawbotandyou
|
886aabf0ae |
fix(#827): /api/packets/{hash} falls back to DB when in-memory store misses (#831)
Closes #827. ## Problem `/api/packets/{hash}` only consulted the in-memory `PacketStore`. When a packet aged out of memory, the handler 404'd — even though SQLite still had it and `/api/nodes/{pubkey}` `recentAdverts` (which reads from the DB) was actively surfacing the hash. Net effect: the **Analyze →** link on older adverts in the node detail page led to a dead "Not found". Two-store inconsistency: DB has the packet, in-memory doesn't, node detail surfaces it from DB → packet detail can't serve it. ## Fix In `handlePacketDetail`: - After in-memory miss, fall back to `db.GetPacketByHash` (already existed) for hash lookups, and `db.GetTransmissionByID` for numeric IDs. - Track when the result came from the DB; if so and the store has no observations, populate from DB via a new `db.GetObservationsForHash` so the response shows real observations instead of the misleading `observation_count = 1` fallback. ## Tests - `TestPacketDetailFallsBackToDBWhenStoreMisses` — insert a packet directly into the DB after `store.Load()`, confirm store doesn't have it, assert 200 + populated observations. - `TestPacketDetail404WhenAbsentFromBoth` — neither store nor DB → 404 (no false positives). - `TestPacketDetailPrefersStoreOverDB` — both have it; store result wins (no double-fetch). - `TestHandlePacketDetailNoStore` updated: it previously asserted the old buggy 404 behavior; now asserts the correct DB-fallback 200. All `go test ./... -run "PacketDetail|Packet|GetPacket"` and the full `cmd/server` suite pass. ## Out of scope The `/api/packets?hash=` filter is the live in-memory list endpoint and intentionally store-only for performance. Not touched here — happy to file a follow-up if you'd rather harmonise. ## Repro context Verified against prod with a recently-adverting repeater whose recent advert hash lives in `recentAdverts` (DB) but had been evicted from the in-memory store; pre-fix 404, post-fix 200 with full observations. Co-authored-by: you <you@example.com> |
||
|
|
a0fddb50aa |
fix(#789): severity from recent samples; Theil-Sen drift with outlier rejection (#828)
Closes #789. ## The two bugs 1. **Severity from stale median.** `classifySkew(absMedian)` used the all-time `MedianSkewSec` over every advert ever recorded for the node. A repeater that was off for hours and then GPS-corrected stayed pinned to `absurd` because hundreds of historical bad samples poisoned the median. Reporter's case: `medianSkewSec: -59,063,561.8` while `lastSkewSec: -0.8` — current health was perfect, dashboard said catastrophic. 2. **Drift from a single correction jump.** Drift used OLS over every `(ts, skew)` pair, with no outlier rejection. A single GPS-correction event (skew jumps millions of seconds in ~30s) dominated the regression and produced `+1,793,549.9 s/day` — physically nonsense; the existing `maxReasonableDriftPerDay` cap then zeroed it (better than absurd, but still useless). ## The two fixes 1. **Recent-window severity.** New field `recentMedianSkewSec` = median over the last `N=5` samples or last `1h`, whichever is narrower (more current view). Severity now derives from `abs(recentMedianSkewSec)`. `MeanSkewSec`, `MedianSkewSec`, `LastSkewSec` are preserved unchanged so the frontend, fleet view, and any external consumers continue to work. 2. **Theil-Sen drift with outlier filter.** Drift now uses the Theil-Sen estimator (median of all pairwise slopes — textbook robust regression, ~29% breakdown point) on a series pre-filtered to drop samples whose skew jumps more than `maxPlausibleSkewJumpSec = 60s` from the previous accepted point. Real µC drift is fractions of a second per advert; clock corrections fall well outside. Capped at `theilSenMaxPoints = 200` (most-recent) so O(n²) stays bounded for chatty nodes. ## What stays the same - Epoch-0 / out-of-range advert filter (PR #769). - `minDriftSamples = 5` floor. - `maxReasonableDriftPerDay = 86400` hard backstop. - API shape: only additions (`recentMedianSkewSec`); no fields removed or renamed. ## Tests All in `cmd/server/clock_skew_test.go`: - `TestSeverityUsesRecentNotMedian` — 100 bad samples (-60s) + 5 good (-1s) → severity = `ok`, historical median still huge. - `TestDriftRejectsCorrectionJump` — 30 min of clean linear drift + one 1000s jump → drift small (~12 s/day). - `TestTheilSenMatchesOLSWhenClean` — clean linear data, Theil-Sen within ~1% of OLS. - `TestReporterScenario_789` — exact reproducer: 1662 samples, 1657 @ -683 days then 5 @ -1s → severity `ok`, `recentMedianSkewSec ≈ 0`, drift bounded; legacy `medianSkewSec` preserved as historical context. `go test ./... -count=1` (cmd/server) and `node test-frontend-helpers.js` both pass. --------- Co-authored-by: clawbot <bot@corescope.local> Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandKpa-clawbot
|
bb09123f34 |
test(#833): update deep-link Playwright assertion for full-screen desktop view (#834)
Closes #833 ## What Update Playwright E2E assertion for desktop deep link to `/#/nodes/{pubkey}`. Now expects `.node-fullscreen` to be present (matches the spec set by PR #824 / issue #823). ## Why The previous assertion encoded the old pre-#823 behavior — "split panel on desktop deep link." PR #824 intentionally removed the `window.innerWidth <= 640` gate so desktop deep links open the full-screen view (matching the Details link path that #779/#785/#824 ultimately made work). The test failed on every PR that rebased onto master, blocking `Deploy Staging`. ## Verified - 1-test diff, no other behavior change - Mobile-viewport assertions elsewhere already exercise the same `.node-fullscreen` selector Co-authored-by: Kpa-clawbot <agent@corescope.local> |
||
|
Kpa-clawbotandKpa-clawbot
|
31a0a944f9 |
fix(#829): node-detail side panel Recent Packets text invisible (#830)
Closes #829 ## What Add explicit `color: var(--text)` to `.advert-info` (and `var(--accent)` to its links) so the side-panel "Recent Packets" entries stay readable in all themes. ## Why `.advert-info` had only `font-size` + `line-height` rules — text inherited from ancestors. In default light/dark themes the inherited color happens to differ enough from `--card-bg`. Under custom themes where they collide, text becomes invisible — only the colored `.advert-dot` shows. Operator screenshot confirmed the symptom. Same class of bug as the existing fix at `style.css:660` ("Bug 7 fix: neighbor table text inherits accent color — force readable text") which forced `color: var(--text)` on `.node-detail-section .data-table td`. The advert timeline doesn't use a data-table, so it fell through. ## Verified - DOM contains correct text — only the rendered color was wrong - `getComputedStyle(.advert-info).color` previously matched `--card-bg` under affected themes - After fix: `.advert-info` resolves to `var(--text)` regardless of inherited chain - Frontend helpers: 553/0 - Full-screen `node-full-card` view (separate `.node-activity-item` markup) unaffected Co-authored-by: Kpa-clawbot <agent@corescope.local> |
||
|
efitenandClaude Sonnet 4.6
|
cad1f11073 |
fix: bypass IATA filter for status messages, fill SNR on duplicate obs (#694) (#802)
## Problems Two independent ingestor bugs identified in #694: ### 1. IATA filter drops status messages from out-of-region observers The IATA filter ran at the top of `handleMessage()` before any message-type discrimination. Status messages carrying observer metadata (`noise_floor`, battery, airtime) from observers outside the configured IATA regions were silently discarded before `UpsertObserver()` and `InsertMetrics()` ran. **Impact:** Observers running `meshcoretomqtt/1.0.8.0` in BFL and LAX — the only client versions that include `noise_floor` in status messages — had their health data dropped entirely on prod instances filtering to SJC. **Fix:** Moved the IATA filter to the packet path only (after the `parts[3] == "status"` branch). Status messages now always populate observer health data regardless of configured region filter. ### 2. `INSERT OR IGNORE` discards SNR/RSSI on late arrival When the same `(transmission_id, observer_idx, path_json)` observation arrived twice — first without RF fields, then with — `INSERT OR IGNORE` silently discarded the SNR/RSSI from the second arrival. **Fix:** Changed to `ON CONFLICT(...) DO UPDATE SET snr = COALESCE(excluded.snr, snr), rssi = ..., score = ...`. A later arrival with SNR fills in a `NULL`; a later arrival without SNR does not overwrite an existing value. ## Tests - `TestIATAFilterDoesNotDropStatusMessages` — verifies BFL status message is processed when IATA filter includes only SJC, and that BFL packet is still filtered - `TestInsertObservationSNRFillIn` — verifies SNR fills in on second arrival, and is not overwritten by a subsequent null arrival ## Related Partially addresses #694 (upstream client issue of missing SNR in packet messages is out of scope) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
efitenandClaude Sonnet 4.6
|
7f024b7aa7 |
fix(#673): replace raw JSON text search with byNode index for node packet queries (#803)
## Summary Fixes #673 - GRP_TXT packets whose message text contains a node's pubkey were incorrectly counted as packets for that node, inflating packet counts and type breakdowns - Two code paths in `store.go` used `strings.Contains` on the full `DecodedJSON` blob — this matched pubkeys appearing anywhere in the JSON, including inside chat message text - `filterPackets` slow path (combined node + other filters): replaced substring search with a hash-set membership check against `byNode[nodePK]` - `GetNodeAnalytics`: removed the full-packet-scan + text search branch entirely; always uses the `byNode` index (which already covers `pubKey`/`destPubKey`/`srcPubKey` via structured field indexing) ## Test Plan - [x] `TestGetNodeAnalytics_ExcludesGRPTXTWithPubkeyInText` — verifies a GRP_TXT packet with the node's pubkey in its text field is not counted in that node's analytics - [x] `TestFilterPackets_NodeQueryDoesNotMatchChatText` — verifies the combined-filter slow path of `filterPackets` returns only the indexed ADVERT, not the chat packet Both tests were written as failing tests against the buggy code and pass after the fix. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Kpa-clawbotandyou
|
ddd18cb12f |
fix(nodes): Details link opens full-screen on desktop (#823) (#824)
Closes #823 ## What Remove the `window.innerWidth <= 640` gate on the `directNode` full-screen branch in `init()` so the 🔍 Details link works on desktop. ## Why - #739 (`e6ace95`) gated full-screen to mobile so desktop **deep links** would land on the split panel. - But the same gate broke the **Details link** flow (#779/#785): the click handler calls `init(app, pubkey)` directly. On desktop the gated branch was skipped, the list re-rendered with `selectedKey = pubkey`, and the side panel was already open → no visible change. - Dropping the gate makes the directNode branch the single, unambiguous path to full-screen for both the Details link and any deep link. ## Why the desktop split-panel UX is still preserved Row clicks call `selectNode()`, which uses `history.replaceState` — no `hashchange` event, no router re-init, no `directNode` set. Only the Details link handler (which calls `init()` explicitly) and a fresh deep-link load reach this branch. ## Repro / verify 1. Desktop, viewport > 640px, open `/#/nodes`. 2. Click a node row → split panel opens (unchanged). 3. Click 🔍 Details inside the panel → full-screen single-node view (was broken; now works). 4. Back button / Escape → back to list view. 5. Paste `/#/nodes/{pubkey}` directly → full-screen on both desktop and mobile. ## Tests `node test-frontend-helpers.js` → 553 passed, 0 failed. Co-authored-by: you <you@example.com> |
||
|
efitenandClaude Sonnet 4.6
|
997bf190ce |
fix(mobile): close button accessible + toolbar scrollable (#797) (#805)
## Summary - **Node detail `top: 60px` → `64px`**: aligns with other overlay panels, gives proper clearance from the 52px fixed nav bar - **Mobile bottom sheet `z-index: 1050`**: node detail now renders above the VCR bar (`z-index: 1000`), close button never obscured - **Mobile `max-height: 60vh` → `60dvh`**: respects iOS Safari browser chrome correctly - **`.live-toggles` horizontal scroll**: `overflow-x: auto; flex-wrap: nowrap` — all 8 checkboxes reachable via horizontal swipe Fixes #797 ## Test plan - [x] Mobile portrait (<640px): tap a map node → bottom sheet slides up, close button (✕) visible and tappable above VCR bar - [x] Mobile portrait: scroll the live-header toggles horizontally → all checkboxes reachable - [x] Desktop/tablet (>640px): node detail panel top-right corner fully below the nav bar - [x] Desktop: close button functional, panel hides correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
5ff4b75a07 |
qa: automate §10.1/§10.2 nodeBlacklist test (#822)
Automates QA plan §10.1 (nodeBlacklist hide) and §10.2 (DB retain), flipping both rows from `human` to `auto`. Stacks on top of #808. **What** - New `qa/scripts/blacklist-test.sh` — env-driven harness: - Args: `BASELINE_URL TARGET_URL TEST_PUBKEY` - Env: `TARGET_SSH_HOST`, `TARGET_SSH_KEY` (default `/root/.ssh/id_ed25519`), `TARGET_CONFIG_PATH`, `TARGET_CONTAINER`, optional `TARGET_DB_PATH` / `ADMIN_API_TOKEN`. - Edits `nodeBlacklist` on target via remote `jq` (python3 fallback), atomic move with preserved perms. - Restarts container, waits up to 120 s for `/api/stats == 200`. - §10.1 asserts `/api/nodes/{pk}` is 404 **or** absent from `/api/nodes` listing, and `/api/topology` does not reference the pubkey. - §10.2 prefers `/api/admin/transmissions` if `ADMIN_API_TOKEN` set, else falls back to `sqlite3` inside the container (and host as last resort). - **Teardown is mandatory** (`trap … EXIT INT TERM`): removes pubkey, restarts, verifies the node is visible again. Teardown failures count toward exit code. - Exit code = number of failures; per-step ✅/❌ with classified failure modes (`ssh-failed`, `restart-stuck`, `hide-failed`, `retain-failed`, `teardown-failed`). - `qa/plans/v3.6.0-rc.md` §10.1 / §10.2 mode → `auto (qa/scripts/blacklist-test.sh)`. **Why** Manual blacklist verification was the slowest item in the §10 block and the easiest to get wrong (forgetting teardown leaks state into the next QA pass). Now it's a single command, public-repo-safe (zero PII / hardcoded hosts), and the trap guarantees the target is restored. `bash -n` passes locally. Live run requires staging credentials. --------- Co-authored-by: meshcore-agent <agent@meshcore> Co-authored-by: meshcore-agent <meshcore@openclaw.local> |
||
|
Kpa-clawbotandyou
|
2460e33f94 |
fix(#810): /health.recentPackets resolved_path falls back to longest sibling obs (#821)
## What + why
`fetchResolvedPathForTxBest` (used by every API path that fills the
top-level `resolved_path`, including
`/api/nodes/{pk}/health.recentPackets`) picked the observation with the
longest `path_json` and queried SQL for that single obs ID. When the
longest-path obs had `resolved_path` NULL but a shorter sibling had one,
the helper returned nil and the top-level field was dropped — even
though the data exists. QA #809 §2.1 caught it on the health endpoint
because that page surfaces it per-tx.
Fix: keep the LRU-friendly fast path (try the longest-path obs), then
fall back to scanning all observations of the tx and picking the longest
`path_json` that actually has a stored `resolved_path`.
## Changes
- `cmd/server/resolved_index.go`: extend `fetchResolvedPathForTxBest`
with a fallback through `fetchResolvedPathsForTx`.
- `cmd/server/issue810_repro_test.go`: regression test — seeds a tx
whose longest-path obs lacks `resolved_path` and a shorter sibling has
it, then asserts `/api/packets` and
`/api/nodes/{pk}/health.recentPackets` agree.
## Tests
`go test ./... -count=1` from `cmd/server` — PASS (full suite, ~19s).
## Perf
Fast path unchanged (single LRU/SQL lookup, dominant case). Fallback
only runs when the longest-path obs has NULL `resolved_path` — one
indexed query per affected tx, bounded by observations-per-tx (small).
Closes #810
---------
Co-authored-by: you <you@example.com>
|
||
|
Kpa-clawbotandyou
|
f701121672 |
Add qa/ — project-specific QA artifacts for the qa-suite skill (#808)
Adds the CoreScope-side artifacts that pair with the generic [`qa-suite` skill](https://github.com/Kpa-clawbot/ai-sdlc/pull/1). ## Layout ``` qa/ ├── README.md ├── plans/ │ └── v3.6.0-rc.md # 34-commit test plan since v3.5.1 └── scripts/ └── api-contract-diff.sh # CoreScope-tuned API contract diff ``` The skill ships the reusable engine + qa-engineer persona + an example plan. This PR adds the CoreScope-tuned plan and the CoreScope-tuned script (correct seed lookups for our `{packets, total}` response shape, our endpoint list, our `resolved_path` requirement). Read by the parent agent at runtime. ## How to use From chat: - `qa staging` — runs the latest `qa/plans/v*-rc.md` against staging, files a fresh GH issue with the report - `qa pr <N>` — uses `qa/plans/pr-<N>.md` if present, else latest RC plan; comments on the PR - `qa v3.6.0-rc` — runs that specific plan The qa-engineer subagent walks every step, classifying each as `auto` (script) / `browser` (UI assertion) / `human` (manual) / `browser+auto`. Quantified pass criteria are mandatory — banned phrases: 'visually aligned' / 'fast' / 'no regression'. ## Plan v3.6.0-rc contents Covers the 34 commits since v3.5.1: - §1 Memory & Load (#806, #790, #807) — heap thresholds, sawtooth pattern - §2 API contract (#806) — every endpoint that should carry `resolved_path`, auto-checked by `api-contract-diff.sh` - §3 Decoder & hashing (#787, #732, #747, #766, #794, #761) - §4 Channels (#725 series M1–M5) - §5 Clock skew (#690 series M1–M3) - §6 Observers (#764, #774) - §7 Multi-byte hash adopters (#758, #767) - §8 Frontend nav & deep linking (#739, #740, #779, #785, #776, #745) - §9 Geofilter (#735, #734) - §10 Node blacklist (#742) - §11 Deploy/ops Release blockers: §1.2, §2, §3. §4 is the headline-feature gate. ## Adding new plans Per release: copy `plans/v<last>-rc.md` to `plans/v<new>-rc.md` and update commit-range header, new sections, GO criteria. Per PR: create `plans/pr-<N>.md` with the bare minimum for that PR's surface area. Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
d7fe24e2db |
Fix channel filter on Packets page (UI + API) — #812 (#816)
Closes #812 ## Root causes **Server (`/api/packets?channel=…` returned identical totals):** The handler in `cmd/server/routes.go` never read the `channel` query parameter into `PacketQuery`, so it was silently ignored by both the SQLite path (`db.go::buildTransmissionWhere`) and the in-memory path (`store.go::filterPackets`). The codebase already had everything else in place — the `channel_hash` column with an index from #762, decoded `channel` / `channelHashHex` fields on each packet — it just wasn't wired up. **UI (`/#/packets` had no channel filter):** `public/packets.js` rendered observer / type / time-window / region filters but no channel control, and didn't read `?channel=` from the URL. ## Fix ### Server - New `Channel` field on `PacketQuery`; `handlePackets` reads `r.URL.Query().Get("channel")`. - DB path filters by the indexed `channel_hash` column (exact match). - In-memory path: helper `packetMatchesChannel` matches `decoded.channel` (plaintext, e.g. `#test`, `public`) or `enc_<HEX>` against `channelHashHex` for undecryptable GRP_TXT. Uses cached `ParsedDecoded()` so it's O(1) after first parse. Fast-path index guards and the grouped-cache key updated to include channel. - Regression test (`channel_filter_test.go`): `channel=#test` returns ≥1 GRP_TXT packet and fewer than baseline; `channel=nonexistentchannel` returns `total=0`. ### UI - New `<select id="fChannel">` populated from `/api/channels`. - Round-trips via `?channel=…` on the URL hash (read on init, written on change). - Pre-seeds the current value as an option so encrypted hashes not in `/api/channels` still display as selected on reload. - On change, calls `loadPackets()` so the server-side filter applies before pagination. ## Perf Filter adds at most one cached map lookup per packet (DB path uses indexed column, store path uses `ParsedDecoded()` cache). Staging baseline 149–190 ms for `?channel=#test&limit=50`; the new comparison is negligible. Target ≤ 500 ms preserved. ## Tests `cd cmd/server && go test ./... -count=1 -timeout 120s` → PASS. --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
a9732e64ae |
fix(nodes): render clock-skew section in side panel (#813) (#814)
Closes #813 ## Root cause The Node detail **side panel** (`renderDetail()`, `public/nodes.js:1145`) was missing both the `#node-clock-skew` placeholder div and the `loadClockSkew()` IIFE loader. Those exist only in the **full-screen** detail page (`loadFullNode`, lines 498 / 632), so any node opened via deep link or click in the listing — which uses the side panel — showed no clock-skew UI even when `/api/nodes/{pk}/clock-skew` returned rich data. ## Fix Mirror the full-screen template branch and IIFE in `renderDetail`: - Add `<div class="node-detail-section skew-detail-section" id="node-clock-skew" style="display:none">` to the side-panel template (right above Observers). - Add an async `loadClockSkewPanel()` IIFE after the panel `innerHTML` is set, using the same severity/badge/drift/sparkline rendering and the `severity === 'no_clock'` branch the full-screen view uses. No new helpers — reuses existing window globals (`formatSkew`, `formatDrift`, `renderSkewBadge`, `renderSkewSparkline`). ## Verification - Syntax check: `node -c public/nodes.js` ✓ - `node test-frontend-helpers.js` → 553/553 ✓ - Browser: staging runs master so I couldn't validate the deployed UI yet. Manual repro after deploy: 1. Open `https://analyzer.00id.net/#/nodes`, click any node with a known skew (e.g. Puppy Solar `a8dde6d7…` shows `⏰ -23d 8h` in listing). 2. Side panel should show a **⏰ Clock Skew** section with median skew, severity badge, drift line, and sparkline. 3. For `severity === 'no_clock'` (e.g. SKCE_RS `14531bd2…`), section shows "No Clock" instead of skew value. --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandmeshcore-agent
|
60be48dc5e |
fix(channels): lock affordance on deep link to encrypted channel without key (#815)
Closes #811 ## What Deep linking to `/#/channels/%23private` (encrypted channel, no key configured) now shows the existing 🔒 lock affordance instead of an empty "No messages in this channel yet" pane. ## Why `selectChannel` only rendered the lock message inside the `if (ch && ch.encrypted)` branch. On a cold deep link: - `loadChannels` omits encrypted channels unless the toggle is on, so `ch` is `undefined`. - The hash isn't `user:`-prefixed, so that branch is skipped too. - Code falls through to the REST fetch, returns 0 messages, and `renderMessages` shows the generic empty state. ## Fix Add a `#`-prefixed-hash branch immediately before the REST fetch: - If a stored key matches the channel name → decrypt and render. - Otherwise → reuse the existing 🔒 "encrypted and no decryption key is configured" message. ## Trace (URL → render) 1. `#/channels/%23private` → `init(routeParam='#private')` 2. `loadChannels()` → `channels` has no `#private` entry (toggle off) 3. `selectChannel('#private')` → `ch` undefined → skips encrypted branches → **new check fires** → lock message 4. With key stored: same check → `decryptAndRender` ## Validation - `node test-frontend-helpers.js` → 553 passed, 0 failed - Manual trace above; change is a 15-line localized guard before the REST fetch, no hot-path or perf impact. Co-authored-by: meshcore-agent <agent@corescope.local> |
||
|
Kpa-clawbotandyou
|
9e90548637 |
perf(#800): remove per-StoreTx ResolvedPath, replace with membership index + on-demand decode (#806)
## Summary Remove `ResolvedPath []*string` field from `StoreTx` and `StoreObs` structs, replacing it with a compact membership index + on-demand SQL decode. This eliminates the dominant heap cost identified in profiling (#791, #799). **Spec:** #800 (consolidated from two rounds of expert + implementer review on #799) Closes #800 Closes #791 ## Design ### Removed - `StoreTx.ResolvedPath []*string` - `StoreObs.ResolvedPath []*string` - `TransmissionResp.ResolvedPath`, `ObservationResp.ResolvedPath` struct fields ### Added | Structure | Purpose | Est. cost at 1M obs | |---|---|---:| | `resolvedPubkeyIndex map[uint64][]int` | FNV-1a(pubkey) → []txID forward index | 50–120 MB | | `resolvedPubkeyReverse map[int][]uint64` | txID → []hashes for clean removal | ~40 MB | | `apiResolvedPathLRU` (10K entries) | FIFO cache for on-demand API decode | ~2 MB | ### Decode-window discipline `resolved_path` JSON decoded once per packet. Consumers fed in order, temp slice dropped — never stored on struct: 1. `addToByNode` — relay node indexing 2. `touchRelayLastSeen` — relay liveness DB updates 3. `byPathHop` resolved-key entries 4. `resolvedPubkeyIndex` + reverse insert 5. WebSocket broadcast map (raw JSON bytes) 6. Persist batch (raw JSON bytes for SQL UPDATE) ### Collision safety When the forward index returns candidates, a batched SQL query confirms exact pubkey presence using `LIKE '%"pubkey"%'` on the `resolved_path` column. ### Feature flag `useResolvedPathIndex` (default `true`). Off-path is conservative: all candidates kept, index not consulted. For one-release rollback safety. ## Files changed | File | Changes | |---|---| | `resolved_index.go` | **New** — index structures, LRU cache, on-demand SQL helpers, collision safety | | `store.go` | Remove RP fields, decode-window discipline in Load/Ingest, on-demand txToMap/obsToMap/enrichObs, eviction cleanup via SQL, memory accounting update | | `types.go` | Remove RP fields from TransmissionResp/ObservationResp | | `routes.go` | Replace `nodeInResolvedPath` with `nodeInResolvedPathViaIndex`, remove RP from mapSlice helpers | | `neighbor_persist.go` | Refactor backfill: reverse-map removal → forward+reverse insert → LRU invalidation | ## Tests added (27 new) **Unit:** - `TestStoreTx_ResolvedPathFieldAbsent` — reflection guard - `TestResolvedPubkeyIndex_BuildFromLoad` — forward+reverse consistency - `TestResolvedPubkeyIndex_HashCollision` — SQL collision safety - `TestResolvedPubkeyIndex_IngestUpdate` — maps reflect new ingests - `TestResolvedPubkeyIndex_RemoveOnEvict` — clean removal via reverse map - `TestResolvedPubkeyIndex_PerObsCoverage` — non-best obs pubkeys indexed - `TestAddToByNode_WithoutResolvedPathField` - `TestTouchRelayLastSeen_WithoutResolvedPathField` - `TestWebSocketBroadcast_IncludesResolvedPath` - `TestBackfill_InvalidatesLRU` - `TestEviction_ByNodeCleanup_OnDemandSQL` - `TestExtractResolvedPubkeys`, `TestMergeResolvedPubkeys` - `TestResolvedPubkeyHash_Deterministic` - `TestLRU_EvictionOnFull` **Endpoint:** - `TestPathsThroughNode_NilResolvedPathFallback` - `TestPacketsAPI_OnDemandResolvedPath` - `TestPacketsAPI_OnDemandResolvedPath_LRUHit` - `TestPacketsAPI_OnDemandResolvedPath_Empty` **Feature flag:** - `TestFeatureFlag_OffPath_PreservesOldBehavior` - `TestFeatureFlag_Toggle_NoStateLeak` **Concurrency:** - `TestReverseMap_NoLeakOnPartialFailure` - `TestDecodeWindow_LockHoldTimeBounded` - `TestLivePolling_LRUUnderConcurrentIngest` **Regression:** - `TestRepeaterLiveness_StillAccurate` **Benchmarks:** - `BenchmarkLoad_BeforeAfter` - `BenchmarkResolvedPubkeyIndex_Memory` - `BenchmarkPathsThroughNode_Latency` - `BenchmarkLivePolling_UnderIngest` ## Benchmark results ``` BenchmarkResolvedPubkeyIndex_Memory/pubkeys=50K 429ms 103MB 777K allocs BenchmarkResolvedPubkeyIndex_Memory/pubkeys=500K 4205ms 896MB 7.67M allocs BenchmarkLoad_BeforeAfter 65ms 20MB 202K allocs BenchmarkPathsThroughNode_Latency 3.9µs 0B 0 allocs BenchmarkLivePolling_UnderIngest 5.4µs 545B 7 allocs ``` Key: per-obs `[]*string` overhead completely eliminated. At 1M obs with 3 hops average, this saves ~72 bytes/obs × 1M = ~68 MB just from the slice headers + pointers, plus the JSON-decoded string data (~900 MB at scale per profiling). ## Design choices - **FNV-1a instead of xxhash**: stdlib availability, no external dependency. Performance is equivalent for this use case (pubkey strings are short). - **FIFO LRU instead of true LRU**: simpler implementation, adequate for the access pattern (mostly sequential obs IDs from live polling). - **Grouped packets view omits resolved_path**: cold path, not worth SQL round-trip per page render. - **Backfill pending check uses reverse-map presence** instead of per-obs field: if a tx has any indexed pubkeys, its observations are considered resolved. Closes #807 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
a8e1cea683 |
fix: use payload type bits only in content hash (not full header byte) (#787)
## Problem The firmware computes packet content hash as: ``` SHA256(payload_type_byte + [path_len for TRACE] + payload) ``` Where `payload_type_byte = (header >> 2) & 0x0F` — just the payload type bits (2-5). CoreScope was using the **full header byte** in its hash computation, which includes route type bits (0-1) and version bits (6-7). This meant the same logical packet produced different content hashes depending on route type — breaking dedup and packet lookup. **Firmware reference:** `Packet.cpp::calculatePacketHash()` uses `getPayloadType()` which returns `(header >> PH_TYPE_SHIFT) & PH_TYPE_MASK`. ## Fix - Extract only payload type bits: `payloadType := (headerByte >> 2) & 0x0F` - Include `path_len` byte in hash for TRACE packets (matching firmware behavior) - Applied to both `cmd/server/decoder.go` and `cmd/ingestor/decoder.go` ## Tests Added - **Route type independence:** Same payload with FLOOD vs DIRECT route types produces identical hash - **TRACE path_len inclusion:** TRACE packets with different `path_len` produce different hashes - **Firmware compatibility:** Hash output matches manual computation of firmware algorithm ## Migration Impact Existing packets in the DB have content hashes computed with the old (incorrect) formula. Options: 1. **Recompute hashes** via migration (recommended for clean state) 2. **Dual lookup** — check both old and new hash on queries (backward compat) 3. **Accept the break** — old hashes become stale, new packets get correct hashes Recommend option 1 (migration) as a follow-up. The volume of affected packets depends on how many distinct route types were seen for the same logical packet. Fixes #786 --------- Co-authored-by: you <you@example.com> |
||
|
Kpa-clawbotandyou
|
bf674ebfa2 |
feat: validate advert signatures on ingest, reject corrupt packets (#794)
## Summary
Validates ed25519 signatures on ADVERT packets during MQTT ingest.
Packets with invalid signatures are rejected before storage, preventing
corrupt/truncated adverts from polluting the database.
## Changes
### Ingestor (`cmd/ingestor/`)
- **Signature validation on ingest**: After decoding an ADVERT, checks
`SignatureValid` from the decoder. Invalid signatures → packet dropped,
never stored.
- **Config flag**: `validateSignatures` (default `true`). Set to `false`
to disable validation for backward compatibility with existing installs.
- **`dropped_packets` table**: New SQLite table recording every rejected
packet with full attribution:
- `hash`, `raw_hex`, `reason`, `observer_id`, `observer_name`,
`node_pubkey`, `node_name`, `dropped_at`
- Indexed on `observer_id` and `node_pubkey` for investigation queries
- **`SignatureDrops` counter**: New atomic counter in `DBStats`, logged
in periodic stats output as `sig_drops=N`
- **Retention**: `dropped_packets` pruned alongside metrics on the same
`retention.metricsDays` schedule
### Server (`cmd/server/`)
- **`GET /api/dropped-packets`** (API key required): Returns recent
drops with optional `?observer=` and `?pubkey=` filters, `?limit=`
(default 100, max 500)
- **`signatureDrops`** field added to `/api/stats` response (count from
`dropped_packets` table)
### Tests (8 new)
| Test | What it verifies |
|------|-----------------|
| `TestSigValidation_ValidAdvertStored` | Valid advert passes validation
and is stored |
| `TestSigValidation_TamperedSignatureDropped` | Tampered signature →
dropped, recorded in `dropped_packets` with correct fields |
| `TestSigValidation_TruncatedAppdataDropped` | Truncated appdata
invalidates signature → dropped |
| `TestSigValidation_DisabledByConfig` | `validateSignatures: false`
skips validation, stores tampered packet |
| `TestSigValidation_DropCounterIncrements` | Counter increments
correctly across multiple drops |
| `TestSigValidation_LogContainsFields` | `dropped_packets` row contains
hash, reason, observer, pubkey, name |
| `TestPruneDroppedPackets` | Old entries pruned, recent entries
retained |
| `TestShouldValidateSignatures_Default` | Config helper returns correct
defaults |
### Config example
```json
{
"validateSignatures": true
}
```
Fixes #793
---------
Co-authored-by: you <you@example.com>
|