Commit Graph

1567 Commits

Author SHA1 Message Date
Kpa-clawbot 8dfcec2ff3 feat: include favorites and claimed nodes in export/import JSON (#1003)
## Summary

Extends the customizer v2 export/import to include favorite nodes and
claimed ("My Mesh") nodes, so users can transfer their full setup
between browsers/devices.

## Changes

### `public/customize-v2.js`
- `readOverrides()` now merges `favorites` (from `meshcore-favorites`)
and `myNodes` (from `meshcore-my-nodes`) into the exported JSON
- `writeOverrides()` extracts `favorites`/`myNodes` arrays and writes
them to their respective localStorage keys, keeping theme overrides
separate
- `validateShape()` validates both new keys as arrays, rejecting
non-array values
- `VALID_SECTIONS` updated to include `favorites` and `myNodes`

### `test-customizer-v2.js`
- 8 new tests covering read/write/validate for both favorites and
myNodes

## TDD
- Red commit: `0405fb7` (failing tests)
- Green commit: `bb9dc34` (implementation)

Fixes #895

---------

Co-authored-by: you <you@example.com>
2026-05-02 23:04:20 -07:00
Kpa-clawbot 84ffed96ed ci: update go-server-coverage.json [skip ci] 2026-05-03 05:28:20 +00:00
Kpa-clawbot b21db32d2e ci: update go-ingestor-coverage.json [skip ci] 2026-05-03 05:28:19 +00:00
Kpa-clawbot f34a233ba7 ci: update frontend-tests.json [skip ci] 2026-05-03 05:28:18 +00:00
Kpa-clawbot 9342ed2799 ci: update frontend-coverage.json [skip ci] 2026-05-03 05:28:17 +00:00
Kpa-clawbot e2d49a62ee ci: update e2e-tests.json [skip ci] 2026-05-03 05:28:16 +00:00
Kpa-clawbot 564d93d6aa fix: dedup topology analytics by resolved pubkey (#998)
## Fix topology analytics double-counting repeaters/pairs (#909)

### Problem

`computeAnalyticsTopology()` aggregates by raw hop hex string. When
firmware emits variable-length path hashes (1-3 bytes per hop), the same
physical node appears multiple times with different prefix lengths (e.g.
`"07"`, `"0735bc"`, `"0735bc6d"` all referring to the same node). This
inflates repeater counts and creates duplicate pair entries.

### Solution

Added a confidence-gated dedup pass after frequency counting:

1. **For each hop prefix**, check if it resolves unambiguously (exactly
1 candidate in the prefix map)
2. **Unambiguous prefixes** → group by resolved pubkey, sum counts, keep
longest prefix as display identifier
3. **Ambiguous prefixes** (multiple candidates for that prefix) → left
as separate entries (conservative)
4. **Same treatment for pairs**: canonicalize by sorted pubkey pair

### Addressing @efiten's collision concern

At scale (~2000+ repeaters), 1-byte prefixes (256 buckets) WILL collide.
This fix explicitly checks the prefix map candidate count. Ambiguous
prefixes (where `len(pm.m[hop]) > 1`) are never merged — they remain as
separate entries. Only prefixes with a single matching node are eligible
for dedup.

### TDD

- **Red commit**: `4dbf9c0` — added 3 failing tests
- **Green commit**: `d6cae9a` — implemented dedup, all tests pass

### Tests added

- `TestTopologyDedup_RepeatersMergeByPubkey` — verifies entries with
different prefix lengths for same node merge to single entry with summed
count
- `TestTopologyDedup_AmbiguousPrefixNotMerged` — verifies colliding
short prefix stays separate from unambiguous longer prefix
- `TestTopologyDedup_PairsMergeByPubkey` — verifies pair entries merge
by resolved pubkey pair

Fixes #909

---------

Co-authored-by: you <you@example.com>
2026-05-02 22:19:49 -07:00
Kpa-clawbot 0b7c4c41c6 ci: update go-server-coverage.json [skip ci] 2026-05-03 04:14:32 +00:00
Kpa-clawbot f87654e7d8 ci: update go-ingestor-coverage.json [skip ci] 2026-05-03 04:14:31 +00:00
Kpa-clawbot 0c9b305a99 ci: update frontend-tests.json [skip ci] 2026-05-03 04:14:30 +00:00
Kpa-clawbot 4aebc4d90b ci: update frontend-coverage.json [skip ci] 2026-05-03 04:14:29 +00:00
Kpa-clawbot 78d96d24db ci: update e2e-tests.json [skip ci] 2026-05-03 04:14:28 +00:00
Kpa-clawbot 440bda6244 fix(channels): channel color picker UX (closes #681) (#995)
## Summary

Fixes the channel color picker UX issues on both Live page and Channels
page.

Closes #681

## Repro Evidence (on master at HEAD)

- **Live feed dots**: 12px inline — too small to reliably click in a
fast-moving feed
- **Right-click hijack**: `contextmenu` listener on live feed conflicts
with browser context menu
- **Channels page**: No way to clear an assigned color without opening
the picker popover
- **Popover positioning**: 8px edge margin causes overlap with panel
borders

## Root Cause

| Issue | File:Line |
|-------|-----------|
| Tiny dots | `public/live.js:2847` — inline `width:12px;height:12px` |
| Context menu hijack | `public/channel-color-picker.js:231` —
`feed.addEventListener('contextmenu', ...)` |
| No clear affordance | `public/channels.js:1101` — dot rendered without
adjacent clear button |
| Popover overlap | `public/channel-color-picker.js:108-109` — `vw - pw
- 8` margin |

## Fix

1. Increased feed color dots to 18px (visible, clickable)
2. Removed contextmenu listener from live feed — dots are the
interaction point
3. Added inline `✕` clear button next to colored dots on channels page
4. Increased popover edge margin to 14px

## TDD Evidence

- **Red commit:** `2034071` — 6/8 tests fail (dot size, contextmenu,
clear affordance, margins)
- **Green commit:** `49636e5` — all 8 tests pass

## Verification

- `node test-color-picker-ux.js` — 8/8 pass
- `node test-channel-color-picker.js` — 17/17 pass (existing tests
unbroken)

---------

Co-authored-by: you <you@example.com>
2026-05-02 21:05:15 -07:00
Kpa-clawbot aea0a9caee fix(packets): preserve scroll position on filter change + group expand/collapse (closes #431) (#996)
## Summary

Closes #431. Preserves scroll position on the packets page when filters
change or groups are expanded/collapsed.

## Problem

When an operator scrolls down through packet history then changes a
filter (type, observer, packet-filter expression) or expands/collapses a
group, `renderTableRows()` rebuilds the DOM which resets `scrollTop` to
0. This forces the user back to the top — frustrating when digging
through hundreds of packets.

## Fix

Save `scrollContainer.scrollTop` at the start of `renderTableRows()`,
restore it after DOM rebuild completes. Two restore points:
1. **Empty-results path** (line ~1821): after `tbody.innerHTML = ...` 
2. **Normal virtual-scroll path** (line ~1840): after
`renderVisibleRows()`

### Key lines changed
- `public/packets.js` lines 1748–1749: save scrollTop
- `public/packets.js` line 1821: restore after empty-state DOM write  
- `public/packets.js` line 1840: restore after renderVisibleRows

## TDD evidence

- **Red commit:** a99ba21 — test asserts scrollTop preserved; fails
without fix
- **Green commit:** 35cc4bf — adds save/restore; test passes

## Anti-tautology

Removing the `scrollContainer.scrollTop = savedScrollTop` lines causes
the test to fail (scrollTop becomes 0 instead of 500). Verified locally.

## Verification

- `node test-packets.js` — 83 passed, 0 failed
- `node test-packet-filter.js` — 62 passed, 0 failed

---------

Co-authored-by: you <you@example.com>
2026-05-02 21:03:01 -07:00
Kpa-clawbot 01246f9412 ci: update go-server-coverage.json [skip ci] 2026-05-03 03:44:19 +00:00
Kpa-clawbot 4c309bad80 ci: update go-ingestor-coverage.json [skip ci] 2026-05-03 03:44:18 +00:00
Kpa-clawbot ce769950dd ci: update frontend-tests.json [skip ci] 2026-05-03 03:44:17 +00:00
Kpa-clawbot 73c04a9ba3 ci: update frontend-coverage.json [skip ci] 2026-05-03 03:44:17 +00:00
Kpa-clawbot e2eaf4c656 ci: update e2e-tests.json [skip ci] 2026-05-03 03:44:16 +00:00
Kpa-clawbot b7c280c20a fix: drop/filter packets with null hash or timestamp (closes #871) (#993)
## Summary

Closes #871

The `/api/packets` endpoint could return packets with `null` hash or
timestamp fields. This was caused by legacy data in SQLite (rows with
empty `hash` or `NULL`/empty `first_seen`) predating the ingestor's
existing validation guard (`if hash == "" { return false, nil }` at
`cmd/ingestor/db.go:610`).

## Root Cause

`cmd/server/store.go` `filterPackets()` had no data-integrity guard.
Legacy rows with empty `hash` or `first_seen` were loaded into the
in-memory store and returned verbatim. The `strOrNil("")` helper then
serialized these as JSON `null`.

## Fix

Added a data-integrity predicate at the top of `filterPackets`'s scan
callback (`cmd/server/store.go:2278`):

```go
if tx.Hash == "" || tx.FirstSeen == "" {
    return false
}
```

This filters bad legacy rows at query time. The write path (ingestor)
already rejects empty hashes, so no new bad data enters.

## TDD Evidence

- **Red commit:** `15774c3` — test `TestIssue871_NoNullHashOrTimestamp`
asserts no packet in API response has null/empty hash or timestamp
- **Green commit:** `281fd6f` — adds the filter guard, test passes

## Testing

- `go test ./...` in `cmd/server` passes (full suite)
- Client-side defensive filter from PR #868 remains as defense-in-depth

---------

Co-authored-by: you <you@example.com>
2026-05-02 20:35:15 -07:00
Kpa-clawbot d43c95a4bb fix(ingestor): warn when TRACE payload decode fails but observation stored (closes #889) (#992)
## Summary

Closes #889.

When a TRACE packet's payload is too short to decode (< 9 bytes),
`decodeTrace` returns an error in `Payload.Error` but the observation is
still stored with empty `Path.Hops`. Previously this was completely
silent — no log, no anomaly flag, no indication the row is degraded.

This fix populates `DecodedPacket.Anomaly` with the decode error message
(e.g., `"TRACE payload decode failed: too short"`) so operators and
downstream consumers can identify degraded observations.

## TDD Commit History

1. **Red commit** `04e0165` — failing test asserting `Anomaly` is set
when TRACE payload decode fails
2. **Green commit** `d3e72d1` — 3-line fix in `decoder.go` line 601-603:
check `payload.Error != ""` for TRACE packets and set anomaly

## What Changed

`cmd/ingestor/decoder.go` (lines 601-603): Added a check before the
existing TRACE path-parsing block. If `payload.Error` is non-empty for a
TRACE packet, `anomaly` is set to `"TRACE payload decode failed:
<error>"`.

`cmd/ingestor/decoder_test.go`: Added
`TestDecodeTracePayloadFailSetsAnomaly` — constructs a TRACE packet with
a 4-byte payload (too short), asserts the packet is still returned
(observation stored) and `Anomaly` is populated.

## Verification

- `go build ./...` ✓
- `go test ./...` ✓ (all pass including new test)
- Anti-tautology: reverting the fix causes the new test to fail (asserts
`pkt.Anomaly == ""` → error)

---------

Co-authored-by: you <you@example.com>
2026-05-02 20:34:27 -07:00
Kpa-clawbot bed5e0267f ci: update go-server-coverage.json [skip ci] 2026-05-03 03:24:29 +00:00
Kpa-clawbot 999ecfc84d ci: update go-ingestor-coverage.json [skip ci] 2026-05-03 03:24:28 +00:00
Kpa-clawbot f12428c460 ci: update frontend-tests.json [skip ci] 2026-05-03 03:24:26 +00:00
Kpa-clawbot 2199d404c9 ci: update frontend-coverage.json [skip ci] 2026-05-03 03:24:25 +00:00
Kpa-clawbot 016a6f2750 ci: update e2e-tests.json [skip ci] 2026-05-03 03:24:24 +00:00
Kpa-clawbot dd2f044f2b fix: cache RW SQLite connection + dedup DBConfig (closes #921) (#982)
Closes #921

## Summary

Follow-up to #920 (incremental auto-vacuum). Addresses both items from
the adversarial review:

### 1. RW connection caching

Previously, every call to `openRW(dbPath)` opened a new SQLite RW
connection and closed it after use. This happened in:
- `runIncrementalVacuum` (~4x/hour)
- `PruneOldPackets`, `PruneOldMetrics`, `RemoveStaleObservers`
- `buildAndPersistEdges`, `PruneNeighborEdges`
- All neighbor persist operations

Now a single `*sql.DB` handle (with `MaxOpenConns(1)`) is cached
process-wide via `cachedRW(dbPath)`. The underlying connection pool
manages serialization. The original `openRW()` function is retained for
one-shot test usage.

### 2. DBConfig dedup

`DBConfig` was defined identically in both `cmd/server/config.go` and
`cmd/ingestor/config.go`. Extracted to `internal/dbconfig/` as a shared
package; both binaries now use a type alias (`type DBConfig =
dbconfig.DBConfig`).

## Tests added

| Test | File |
|------|------|
| `TestCachedRW_ReturnsSameHandle` | `cmd/server/rw_cache_test.go` |
| `TestCachedRW_100Calls_SingleConnection` |
`cmd/server/rw_cache_test.go` |
| `TestGetIncrementalVacuumPages_Default` |
`internal/dbconfig/dbconfig_test.go` |
| `TestGetIncrementalVacuumPages_Configured` |
`internal/dbconfig/dbconfig_test.go` |

## Verification

```
ok  github.com/corescope/server    20.069s
ok  github.com/corescope/ingestor  47.117s
ok  github.com/meshcore-analyzer/dbconfig  0.003s
```

Both binaries build cleanly. 100 sequential `cachedRW()` calls return
the same handle with exactly 1 entry in the cache map.

---------

Co-authored-by: you <you@example.com>
2026-05-02 20:15:30 -07:00
Kpa-clawbot 736b09697d fix(analytics): apply customizer timestamp format to chart axes (closes #756) (#981)
## Summary

Fixes #756 — the customizer timestamp format setting (ISO/ISO+ms/locale)
and timezone (UTC/local) were not applied to chart X-axis labels,
tooltips, or certain inline timestamps in the analytics pages.

## Changes

### `public/app.js`
- Added `formatChartAxisLabel(date, shortForm)` — a shared helper that
reads the customizer's `timestampFormat` and `timestampTimezone`
preferences and formats dates for chart axes accordingly.
`shortForm=true` returns time-only (for intra-day charts),
`shortForm=false` returns date+time (for multi-day ranges).

### `public/analytics.js`
- `rfXAxisLabels()`: now calls `formatChartAxisLabel()` instead of
hardcoded `toLocaleTimeString()`
- `rfTooltipCircles()`: tooltip timestamps now use
`formatAbsoluteTimestamp()` instead of raw ISO
- Subpath detail first/last seen: now uses `formatAbsoluteTimestamp()`
- Neighbor graph last_seen: now uses `formatAbsoluteTimestamp()`

### `public/node-analytics.js`
- Packet timeline chart labels: now use `formatChartAxisLabel()`
(respects short vs long form based on time range)
- SNR over time chart labels: now use `formatChartAxisLabel()`

## Behavior by setting

| Setting | Chart axis (short) | Chart axis (long) |
|---------|-------------------|-------------------|
| ISO | `14:30` | `05-03 14:30` |
| ISO+ms | `14:30:05` | `05-03 14:30:05` |
| Locale | `2:30 PM` | `May 3, 2:30 PM` |

All respect the UTC/local timezone toggle.

## Testing

- Server builds cleanly (`go build`)
- Served `app.js` contains `formatChartAxisLabel` (verified via curl)
- Graceful fallback: all callsites check `typeof formatChartAxisLabel
=== 'function'` before calling, preserving backward compat if script
load order changes

---------

Co-authored-by: you <you@example.com>
2026-05-02 20:10:29 -07:00
Kpa-clawbot b3b96b3dda ci: update go-server-coverage.json [skip ci] 2026-05-03 03:02:27 +00:00
Kpa-clawbot 5c9860db46 ci: update go-ingestor-coverage.json [skip ci] 2026-05-03 03:02:26 +00:00
Kpa-clawbot de288e71da ci: update frontend-tests.json [skip ci] 2026-05-03 03:02:25 +00:00
Kpa-clawbot 3529b1334b ci: update frontend-coverage.json [skip ci] 2026-05-03 03:02:24 +00:00
Kpa-clawbot 7bd1f396df ci: update e2e-tests.json [skip ci] 2026-05-03 03:02:23 +00:00
Kpa-clawbot 58484ad924 feat(ingestor): backfill observations.path_json from raw_hex (closes #888) (#983)
## Summary

Adds an idempotent startup migration to the ingestor that backfills
`observations.path_json` from per-observation `raw_hex` (added in #882).

**Approach: Server-side migration (Option B)** — runs automatically at
startup, chunked in batches of 1000, tracked via `_migrations` table.
Chosen over a standalone script because:
1. Follows existing migration pattern (channel_hash, last_packet_at,
etc.)
2. Zero operator action required — just deploy
3. Idempotent — safe to restart mid-migration (uncommitted rows get
picked up next run)

## What it does

- Selects observations where `raw_hex` is populated but `path_json` is
NULL/empty/`[]`
- Excludes TRACE packets (`payload_type = 9`) at the SQL level — their
header bytes are SNR values, not hops
- Decodes hops via `packetpath.DecodePathFromRawHex` (reuses existing
helper)
- Updates `path_json` with the decoded JSON array
- Marks rows with undecoded/empty hops as `'[]'` to prevent infinite
re-scanning
- Records `backfill_path_json_from_raw_hex_v1` in `_migrations` when
complete

## Safety

- **Never overwrites** existing non-empty `path_json` — only fills where
missing
- **Batched** (1000 rows per iteration) — won't OOM on large DBs
- **TRACE-safe** — excluded at query level per
`packetpath.PathBytesAreHops` semantics

## Test

`TestBackfillPathJsonFromRawHex` — creates synthetic observations with:
- Empty path_json + valid raw_hex → verifies backfill populates
correctly
- NULL path_json → verifies backfill populates
- Existing path_json → verifies NO overwrite
- TRACE packet → verifies skip

Anti-tautology: test asserts specific decoded values (`["AABB","CCDD"]`)
from known raw_hex input, not just "something changed."

Closes #888

Co-authored-by: you <you@example.com>
2026-05-02 19:52:43 -07:00
Kpa-clawbot 1a2170bf92 ci: update go-server-coverage.json [skip ci] 2026-05-03 01:07:07 +00:00
Kpa-clawbot 8a3c87e5a2 ci: update go-ingestor-coverage.json [skip ci] 2026-05-03 01:07:06 +00:00
Kpa-clawbot 722cf480f8 ci: update frontend-tests.json [skip ci] 2026-05-03 01:07:05 +00:00
Kpa-clawbot 5cbfb4a8e7 ci: update frontend-coverage.json [skip ci] 2026-05-03 01:07:05 +00:00
Kpa-clawbot b7933553a6 ci: update e2e-tests.json [skip ci] 2026-05-03 01:07:04 +00:00
Kpa-clawbot fc57433f27 fix(analytics): merge channel buckets by hash byte; reject rainbow-table mismatches (closes #978) (#980)
## Summary

Closes #978 — analytics channels duplicated by encrypted/decrypted split
+ rainbow-table collisions.

## Root cause

Two distinct bugs in `computeAnalyticsChannels` (`cmd/server/store.go`):

1. **Encrypted/decrypted split**: The grouping key included the decoded
channel name (`hash + "_" + channel`), so packets from observers that
could decrypt a channel created a separate bucket from packets where
decryption failed. Same physical channel, two entries.

2. **Rainbow-table collisions**: Some observers' lookup tables map hash
bytes to wrong channel names. E.g., hash `72` incorrectly claimed to be
`#wardriving` (real hash is `129`). This created ghost 1-message
entries.

## Fix

1. **Always group by hash byte alone** (drop `_channel` suffix from
`chKey`). When any packet decrypts successfully, upgrade the bucket's
display name from placeholder (`chN`) to the real name
(first-decrypter-wins for stability).

2. **Validate channel names** against the firmware hash invariant:
`SHA256(SHA256("#name")[:16])[0] == channelHash`. Mismatches are treated
as encrypted (placeholder name, no trust in decoded channel). Guard is
in the analytics handler (not the ingestor) to avoid breaking other
surfaces that use the decoded field for display.

## Verification (e2e-fixture.db)

| Metric | BEFORE | AFTER |
|--------|--------|-------|
| Total channels | 22 | 19 |
| Duplicate hash bytes | 3 (hashes 217, 202, 17) | 0 |

## Tests added

- `TestComputeAnalyticsChannels_MergesEncryptedAndDecrypted` — same
hash, mixed encrypted/decrypted → ONE bucket
- `TestComputeAnalyticsChannels_RejectsRainbowTableMismatch` — hash 72
claimed as `#wardriving` (real=129) → rejected, stays `ch72`
- `TestChannelNameMatchesHash` — unit test for hash validation helper
- `TestIsPlaceholderName` — unit test for placeholder detection

Anti-tautology gate: both main tests fail when their respective fix
lines are reverted.

Co-authored-by: you <you@example.com>
2026-05-02 16:05:56 -07:00
Kpa-clawbot 53ab302dd6 fix(packets): clear-filters button (rebased + addresses greybeard) (closes #964) (#975)
Rebased version of #973 onto current master, with greybeard review
fixes.

## Changes from #973
- **Stowaway revert dropped**: The original PR branched from older
master and inadvertently reverted PR #926's MQTT connect-retry fix
(`cmd/ingestor/main.go` + `cmd/ingestor/main_test.go`). After rebasing
onto current master (which includes #926 + #970), these files no longer
appear in the diff.
- **Greybeard M1 fixed**: Time-window filter (`savedTimeWindowMin`,
`fTimeWindow` dropdown, `localStorage 'meshcore-time-window'`) is now
reset by the clear-filters button. The clear-button visibility predicate
also accounts for non-default time window.
- **Greybeard m1 fixed**: Replaced 7 tautological source-grep tests with
8 behavioral vm-sandbox tests that extract and execute the actual clear
handler + `updatePacketsUrl`, asserting real state transitions.

## Original feature (from #973)
Clear-filters button for the packets page — resets all filter state
(hash, node, observer, channel, type, expression, myNodes, time window,
region) and refreshes. Button visibility auto-toggles based on active
filter state.

Closes #964
Supersedes #973

## Tests
- `node test-clear-filters.js` — 8 behavioral tests 
- `node test-packets.js` — 82 tests 
- `cd cmd/ingestor && go test ./...` — 

---------

Co-authored-by: you <you@example.com>
2026-05-02 12:12:51 -07:00
Kpa-clawbot 5aa8f795cd feat(ingestor): per-source MQTT connect timeout (#931) (#977)
## Summary

Per-source MQTT connect timeout, correctly targeting the `WaitTimeout`
startup gate (#931).

## What changed

- Added `connectTimeoutSec` field to `MQTTSource` struct (per-source,
not global) — `config.go:24`
- Added `ConnectTimeoutOrDefault()` helper returning configured value or
30 (default from #926) — `config.go:29`
- Replaced hardcoded `WaitTimeout(30 * time.Second)` with
`WaitTimeout(time.Duration(connectTimeout) * time.Second)` —
`main.go:173`
- Updated `config.example.json` with field at source level
- Unit tests for default (30) and custom values

## Why this supersedes #976

PR #976 made paho's `SetConnectTimeout` (per-TCP-dial, was 10s)
configurable via a **global** `mqttConnectTimeoutSeconds` field. Issue
#931 explicitly references the **30s timeout** — which is
`WaitTimeout(30s)`, the startup gate from #926. It also requests
**per-source** config, not global.

This PR targets the correct timeout at the correct granularity.

## Live verification (Rule 18)

Two sources pointed at unreachable brokers:
- `fast` (`connectTimeoutSec: 5`): timed out in 5s 
- `default` (unset): timed out in 30s 

```
19:00:35 MQTT [fast] connect timeout: 5s
19:00:40 MQTT [fast] initial connection timed out — retrying in background
19:00:40 MQTT [default] connect timeout: 30s
19:01:10 MQTT [default] initial connection timed out — retrying in background
```

Closes #931
Supersedes #976

Co-authored-by: you <you@example.com>
2026-05-02 12:08:25 -07:00
Kpa-clawbot 1e7c187521 fix(ingestor): address review BLOCKERs from PR #926 (goroutine leak + guard semantics) [v2] (#974)
## fix(ingestor): address review BLOCKERs from PR #926 (goroutine leak +
guard semantics)

Supersedes #970. Rebased onto current master to resolve merge conflicts.

### Changes (same as #970)
- **BL1 (goroutine leak):** Call `client.Disconnect(0)` on the error
path after `Connect()` fails with `ConnectRetry=true`, preventing Paho's
internal retry goroutines from leaking.
- **BL2 (guard semantics):** Use `connectedCount == 0` instead of
`len(clients) == 0` to detect zero-connected state, since timed-out
clients are appended to the slice.
- **Tests:** `TestBL1_GoroutineLeakOnHardFailure` and
`TestBL2_ZeroConnectedFatals` covering both blockers.

### Context
- Fixes blockers raised in review of #926
- Related: #910 (original hang bug)

Co-authored-by: you <you@example.com>
2026-05-02 12:05:02 -07:00
Kpa-clawbot 4b8d8143f4 feat(server): explicit CORS policy with configurable origin allowlist (#883) (#971)
## Summary

Adds explicit CORS policy support to the CoreScope API server, closing
#883.

### Problem

The API relied on browser same-origin defaults with no way for operators
to configure cross-origin access. Operators running dashboards or
third-party frontends on different origins had no supported way to make
API calls.

### Solution

**New config option:** `corsAllowedOrigins` (string array, default `[]`)

**Middleware behavior:**
| Config | Behavior |
|--------|----------|
| `[]` (default) | No `Access-Control-*` headers added — browsers
enforce same-origin. **Preserves current behavior.** |
| `["https://dashboard.example.com"]` | Echoes matching `Origin`, sets
`Allow-Methods`/`Allow-Headers` |
| `["*"]` | Sets `Access-Control-Allow-Origin: *` (explicit opt-in only)
|

**Headers set when origin matches:**
- `Access-Control-Allow-Origin: <origin>` (or `*`)
- `Access-Control-Allow-Methods: GET, POST, OPTIONS`
- `Access-Control-Allow-Headers: Content-Type, X-API-Key`
- `Vary: Origin` (non-wildcard only)

**Preflight handling:** `OPTIONS` → `204 No Content` with CORS headers
(or `403` if origin not in allowlist).

### Config example

```json
{
  "corsAllowedOrigins": ["https://dashboard.example.com", "https://monitor.internal"]
}
```

### Files changed

| File | Change |
|------|--------|
| `cmd/server/cors.go` | New CORS middleware |
| `cmd/server/cors_test.go` | 7 unit tests covering all branches |
| `cmd/server/config.go` | `CORSAllowedOrigins` field |
| `cmd/server/routes.go` | Wire middleware before all routes |

### Testing

**Unit tests (7):**
- Default config → no CORS headers
- Allowlist match → headers present with `Vary: Origin`
- Allowlist miss → no CORS headers
- Preflight allowed → 204 with headers
- Preflight rejected → 403
- Wildcard → `*` without `Vary`
- No `Origin` header → pass-through

**Live verification (Rule 18):**

```
# Default (empty corsAllowedOrigins):
$ curl -I -H "Origin: https://evil.example" localhost:19883/api/health
HTTP/1.1 200 OK
# No Access-Control-* headers ✓

# With corsAllowedOrigins: ["https://good.example"]:
$ curl -I -H "Origin: https://good.example" localhost:19884/api/health
Access-Control-Allow-Origin: https://good.example
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Content-Type, X-API-Key
Vary: Origin ✓

$ curl -I -H "Origin: https://evil.example" localhost:19884/api/health
# No Access-Control-* headers ✓

$ curl -I -X OPTIONS -H "Origin: https://good.example" localhost:19884/api/health
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: https://good.example ✓
```

Closes #883

Co-authored-by: you <you@example.com>
2026-05-02 12:04:37 -07:00
Kpa-clawbot 3364eed303 feat: separate "Last Status Update" from "Last Packet Observation" for observers (v3 rebase) (#969)
Rebased version of #968 (which was itself a rebase of #905) — resolves
merge conflict with #906 (clock-skew UI) that landed on master.

## Conflict resolution

**`public/observers.js`** — master (#906) added "Clock Offset" column to
observer table; #968 split "Last Seen" into "Last Status" + "Last
Packet" columns. Combined both: the table now has Status | Name | Region
| Last Status | Last Packet | Packets | Packets/Hour | Clock Offset |
Uptime.

## What this PR adds (unchanged from #968/#905)

- `last_packet_at` column in observers DB table
- Separate "Last Status Update" and "Last Packet Observation" display in
observers list and detail page
- Server-side migration to add the column automatically
- Backfill heuristic for existing data
- Tests for ingestor and server

## Verification

- All Go tests pass (`cmd/server`, `cmd/ingestor`)
- Frontend tests pass (`test-packets.js`, `test-hash-color.js`)
- Built server, hit `/api/observers` — `last_packet_at` field present in
JSON
- Observer table header has all 9 columns including both Last Packet and
Clock Offset

## Prior PRs

- #905 — original (conflicts with master)
- #968 — first rebase (conflicts after #906 landed)
- This PR — second rebase, resolves #906 conflict

Supersedes #968. Closes #905.

---------

Co-authored-by: you <you@example.com>
2026-05-02 12:03:42 -07:00
efiten d65122491e fix(ingestor): unblock startup when one of multiple MQTT sources is unreachable (#926)
## Summary

- With `ConnectRetry=true`, paho's `token.Wait()` only returns on
success — it blocks forever for unreachable brokers, stalling the entire
startup loop before any other source connects
- Switches to `token.WaitTimeout(30s)`: on timeout the client is still
tracked so `ConnectRetry` keeps retrying in background; `OnConnect`
fires and subscribes when it eventually connects
- Adds `TestMQTTConnectRetryTimeoutDoesNotBlock` to confirm
`WaitTimeout` returns within deadline for unreachable brokers
(regression guard for this exact failure mode)

Fixes #910

## Test plan

- [x] Two MQTT sources configured, one unreachable: ingestor reaches
`Running` status and ingests from the reachable source immediately on
startup
- [x] Unreachable source logs `initial connection timed out — retrying
in background` and reconnects automatically when the broker comes back
- [x] Single source, reachable: behaviour unchanged (`Running — 1 MQTT
source(s) connected`)
- [x] Single source, unreachable: `Running — 0 MQTT source(s) connected,
1 retrying in background`; ingestion starts once broker is available
- [x] `go test ./...` passes (excluding pre-existing
`TestOpenStoreInvalidPath` failure on master)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 11:31:51 -07:00
Kpa-clawbot 4f0f7bc6dd fix(ui): fill remaining gaps in payload-type lookup tables (10/11/15) (#967)
## Summary

Fill the remaining gaps in payload-type lookup tables noted out-of-scope
on #965. Every firmware-defined payload type (0–11, 15) now has entries
in all four frontend tables.

## Changes

Three types were missing from one or more tables:

| Type | Name | `PAYLOAD_COLORS` (app.js) | `TYPE_NAMES` (packets.js) |
`TYPE_COLORS` (roles.js) | `TYPE_BADGE_MAP` (roles.js) |

|------|------|--------------------------|--------------------------|-------------------------|---------------------------|
| 10 | Multipart | added | added | added `#0d9488` | added |
| 11 | Control | added |  (already) | added `#b45309` | added |
| 15 | Raw Custom | added | added | added `#c026d3` | added |

## Color choices

- **MULTIPART** `#0d9488` (teal) — multi-fragment stitching, distinct
from PATH's `#14b8a6`
- **CONTROL** `#b45309` (amber) — warm brown, distinct hue from ACK's
grey `#6b7280`
- **RAW_CUSTOM** `#c026d3` (fuchsia) — magenta, distinct from TRACE's
pink `#ec4899`

All pass WCAG 3:1 contrast against both white and dark (#1e1e1e)
backgrounds.

## Tests

- `test-packets.js`: 82/82 
- `test-hash-color.js`: 32/32 

Badge CSS auto-generation: `syncBadgeColors()` in `roles.js` iterates
`TYPE_BADGE_MAP` keyed against `TYPE_COLORS`, so the three new entries
automatically get `.type-badge.multipart`, `.type-badge.control`, and
`.type-badge.raw-custom` CSS rules injected at page load.

Firmware source: `firmware/src/Packet.h:19-32` — types 0x00–0x0B and
0x0F. Types 0x0C–0x0E are not defined.

Follows up on #965.

---------

Co-authored-by: you <you@example.com>
2026-05-02 11:17:34 -07:00
efiten 40c3aa13f9 fix(paths): exclude false-positive paths from short-prefix collisions (#930)
Fixes #929

## Summary

- `handleNodePaths` pulls candidates from `byPathHop` using 2-char and
4-char prefix keys (e.g. `"7a"` for a node using 1-byte adverts)
- When two nodes share the same short prefix, paths through the *other*
node are included as candidates
- The `resolved_path` post-filter covers decoded packets but falls
through conservatively (`inIndex = true`) when `resolved_path` is NULL,
letting false positives reach the response

**Fix:** during the aggregation phase (which already calls `resolveHop`
per hop), add a `containsTarget` check. If every hop resolves to a
different node's pubkey, skip the path. Packets confirmed via the
full-pubkey index key or via SQL bypass the check. Unresolvable hops are
kept conservatively.

## Test plan
- [x] `TestHandleNodePaths_PrefixCollisionExclusion`: two nodes sharing
`"7a"` prefix; verifies the path with no `resolved_path` (false
positive) is excluded and the SQL-confirmed path (true positive) is
included
- [x] Full test suite: `go test github.com/corescope/server` — all pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 11:15:25 -07:00
Kpa-clawbot b47587f031 feat(#690): expose observer skew + per-hash evidence in clock UI (#906)
## Summary

UI completion of #690 — surfaces observer clock skew and per-hash
evidence that the backend already computes but wasn't exposed in the
frontend.

**Not related to #845/PR #894** (bimodal detection) — this is the UI
surface for the original #690 scope.

## Changes

### Backend: per-hash evidence in node clock-skew API (commit 1)
- Extended `GET /api/nodes/{pubkey}/clock-skew` to return
`recentHashEvidence` (most recent 10 hashes with per-observer
raw/corrected skew and observer offset) and `calibrationSummary`
(total/calibrated/uncalibrated counts).
- Evidence is cached during `ClockSkewEngine.Recompute()` — route
handler is cheap.
- Fleet endpoint omits evidence to keep payload small.

### Frontend: observer list page — clock offset column (commit 2)
- Added "Clock Offset" column to observers table.
- Fetches `/api/observers/clock-skew` once on page load, joins by
ObserverID.
- Color-coded severity badge + sample count tooltip.
- Singleton observers show "—" not "0".

### Frontend: observer-detail clock card (commit 3)
- Added clock offset card mirroring node clock card style.
- Shows: offset value, sample count, severity badge.
- Inline explainer describing how offset is computed from multi-observer
packets.

### Frontend: node clock card evidence panel (commit 4)
- Collapsible "Evidence" section in existing node clock skew card.
- Per-hash breakdown: observer count, median corrected skew,
per-observer raw/corrected/offset.
- Calibration summary line and plain-English severity reason at top.

## Test Results

```
go test ./... (cmd/server) — PASS (19.3s)
go test ./... (cmd/ingestor) — PASS (31.6s)
Frontend helpers: 610 passed, 0 failed
```

New test: `TestNodeClockSkew_EvidencePayload` — 3-observer scenario
verifying per-hash array shape, corrected = raw + offset math, and
median.

No frontend JS smoke test added — no existing test harness for
clock/observer rendering. Noted for future.

## Screenshots

Screenshots TBD

## Perf justification

Evidence is computed inside the existing `Recompute()` cycle (already
O(n) on samples). The `hashEvidence` map adds ~32 bytes per sample of
memory. Evidence is stripped from fleet responses. Per-node endpoint
returns at most 10 evidence entries — bounded payload.

---------

Co-authored-by: you <you@example.com>
2026-05-02 10:30:54 -07:00
Kpa-clawbot c67f3347ce fix(ui): add GRP_DATA (type 6) to filter dropdown + color tables (#965)
## Bug

Packet type 6 (`PAYLOAD_TYPE_GRP_DATA` per `firmware/src/Packet.h:25`)
was missing from three frontend lookup tables:
- `public/app.js:7` — `PAYLOAD_COLORS` had no entry for 6 → badge color
fell back to `unknown` (grey)
- `public/packets.js:29` — `TYPE_NAMES` (used by the Packets page
type-filter dropdown) had no entry for 6 → "Group Data" missing from the
menu
- `public/roles.js:17,24` — `TYPE_COLORS` and `TYPE_BADGE_MAP` had no
`GRP_DATA` entry → no dedicated CSS class

The packet detail page already handled it (via `PAYLOAD_TYPES` in
`app.js:6` which had `6: 'Group Data'`) so individual GRP_DATA packets
render correctly. The gap was only in the filter UI + badge styling.

## Fix

Add the missing entry in each table. 4 lines across 3 files.

- `app.js`: add `6: 'grp-data'` to `PAYLOAD_COLORS`
- `packets.js`: add `6:'Group Data'` to `TYPE_NAMES`
- `roles.js`: add `GRP_DATA: '#8b5cf6'` to `TYPE_COLORS` and `GRP_DATA:
'grp-data'` to `TYPE_BADGE_MAP`

Color choice `#8b5cf6` (violet) — distinct from GRP_TXT's blue but
visually adjacent so operators read them as related types.

## Verification (rule 18 + 19)

Built server locally, served the JS files, grepped the rendered output:

```
$ curl -s http://localhost:13900/packets.js | grep TYPE_NAMES
const TYPE_NAMES = { ... 5:'Channel Msg', 6:'Group Data', 7:'Anon Req' ... };

$ curl -s http://localhost:13900/app.js | grep PAYLOAD_TYPES
const PAYLOAD_TYPES = { ... 5: 'Channel Msg', 6: 'Group Data', 7: 'Anon Req' ... };

$ curl -s http://localhost:13900/roles.js | grep GRP_DATA
ADVERT: '#22c55e', GRP_TXT: '#3b82f6', GRP_DATA: '#8b5cf6', ...
ADVERT: 'advert', GRP_TXT: 'grp-txt', GRP_DATA: 'grp-data', ...
```

Frontend tests pass: `test-packets.js` 82/82, `test-hash-color.js`
32/32.

## Out of scope

Consolidating the duplicated PAYLOAD_TYPES / TYPE_NAMES tables into a
single source of truth is a separate cleanup. Two parallel name maps
continues to be a footgun (this is the second time a new type's been
added to one but not the other).

Co-authored-by: Kpa-clawbot <bot@example.invalid>
2026-05-02 09:55:09 -07:00