## Summary
Adds a global `observerIATAWhitelist` config field that restricts which
observer IATA regions are processed by the ingestor.
## Problem
Operators running regional instances (e.g., Sweden) want to ensure only
observers physically in their region contribute data. The existing
per-source `iataFilter` only filters packet messages but still allows
status messages through, meaning observers from other regions appear in
the database.
## Solution
New top-level config field `observerIATAWhitelist`:
- When non-empty, **all** messages (status + packets) from observers
outside the whitelist are silently dropped
- Case-insensitive matching
- Empty list = all regions allowed (fully backwards compatible)
- Lazy O(1) lookup via cached uppercase set (same pattern as
`observerBlacklist`)
### Config example
```json
{
"observerIATAWhitelist": ["ARN", "GOT"]
}
```
## TDD
- **Red commit:** `f19c2b2` — tests for `ObserverIATAWhitelist` field
and `IsObserverIATAAllowed` method (build fails)
- **Green commit:** `782f516` — implementation + integration test
## Files changed
- `cmd/ingestor/config.go` — new field, new method
`IsObserverIATAAllowed`
- `cmd/ingestor/main.go` — whitelist check in `handleMessage` before
status processing
- `cmd/ingestor/config_test.go` — unit tests for config parsing and
matching
- `cmd/ingestor/main_test.go` — integration test for handleMessage
filtering
Fixes#914
---------
Co-authored-by: you <you@example.com>
## Summary
Adds a **Flood / Direct packet filter** dropdown to the observer
comparison page. This addresses the issue that direct packets (heard by
only one observer) skew the comparison percentages.
## Changes
- **`public/compare.js`**: Added `filterPacketsByRoute(packets, mode)`
function and a "Packet Type" dropdown (All / Flood only / Direct only)
to the comparison controls. Changing the filter re-runs the comparison
with filtered packets.
- **`test-compare-flood-filter.js`**: Unit tests for the filter
function.
## Route type mapping (from firmware)
| Route Type | Value | Filter |
|---|---|---|
| TransportFlood | 0 | Flood |
| Flood | 1 | Flood |
| Direct | 2 | Direct |
| TransportDirect | 3 | Direct |
## TDD
- Red commit: `484fa72` (test only, fails)
- Green commit: `5661f71` (implementation, tests pass)
Fixes#928
---------
Co-authored-by: you <you@example.com>
## Summary
Documents the removal of 15 prefix helper tests
(`buildOneBytePrefixMap`, `buildTwoBytePrefixInfo`,
`buildCollisionHops`) from `test-frontend-helpers.js`.
These functions were moved server-side in PR #415. The equivalent logic
is now covered by Go tests:
- `cmd/server/collision_details_test.go` — collision prefix + node-pair
assertions
- `cmd/server/store_test.go` — hash-collision endpoint integration
Adds a documentation comment in the test file where the tests previously
lived, explaining the rationale and pointing to the Go test equivalents.
Fixes#437
---------
Co-authored-by: you <you@example.com>
## Summary
Show 2-byte hash support indicator on map markers. Fixes#903.
## What changed
### Backend (`cmd/server/store.go`, `cmd/server/routes.go`)
- **`EnrichNodeWithMultiByte()`** — new enrichment function that adds
`multi_byte_status` (confirmed/suspected/unknown), `multi_byte_evidence`
(advert/path), and `multi_byte_max_hash_size` fields to node API
responses
- **`GetMultiByteCapMap()`** — cached (15s TTL) map of pubkey →
`MultiByteCapEntry`, reusing the existing `computeMultiByteCapability()`
logic that combines advert-based and path-hop-based evidence
- Wired into both `/api/nodes` (list) and `/api/nodes/{pubkey}` (detail)
endpoints
### Frontend (`public/map.js`)
- Added **"Multi-byte support"** checkbox in the map Display controls
section
- When toggled on, repeater markers change color:
- 🟢 Green (`#27ae60`) — **confirmed** (advertised with hash_size ≥ 2)
- 🟡 Yellow (`#f39c12`) — **suspected** (seen as hop in multi-byte path)
- 🔴 Red (`#e74c3c`) — **unknown** (no multi-byte evidence)
- Popup tooltip shows multi-byte status and evidence for repeaters
- State persisted in localStorage (`meshcore-map-multibyte-overlay`)
## TDD
- Red commit: `2f49cbc` — failing test for `EnrichNodeWithMultiByte`
- Green commit: `4957782` — implementation + passing tests
## Performance
- `GetMultiByteCapMap()` uses a 15s TTL cache (same pattern as
`GetNodeHashSizeInfo`)
- Enrichment is O(n) over nodes, no per-item API calls
- Frontend color override is computed inline during existing marker
render loop — no additional DOM rebuilds
---------
Co-authored-by: you <you@example.com>
## Summary
One-time startup migration that deletes legacy packets (transmissions +
observations) with empty hash or empty `first_seen` timestamp. This is
the write-side cleanup following #993's read-side filter.
### Migration: `cleanup_legacy_null_hash_ts`
- Checks `_migrations` table for marker
- If not present: deletes observations referencing bad transmissions,
then deletes the transmissions themselves
- Logs count of deleted rows
- Records marker for idempotency
### TDD
- **Red commit:** `b1a24a1` — test asserts migration deletes bad rows
(fails without implementation)
- **Green commit:** `2b94522` — implements the migration, all tests pass
Fixes#994
---------
Co-authored-by: you <you@example.com>
## Summary
Extends the customizer v2 export/import to include favorite nodes and
claimed ("My Mesh") nodes, so users can transfer their full setup
between browsers/devices.
## Changes
### `public/customize-v2.js`
- `readOverrides()` now merges `favorites` (from `meshcore-favorites`)
and `myNodes` (from `meshcore-my-nodes`) into the exported JSON
- `writeOverrides()` extracts `favorites`/`myNodes` arrays and writes
them to their respective localStorage keys, keeping theme overrides
separate
- `validateShape()` validates both new keys as arrays, rejecting
non-array values
- `VALID_SECTIONS` updated to include `favorites` and `myNodes`
### `test-customizer-v2.js`
- 8 new tests covering read/write/validate for both favorites and
myNodes
## TDD
- Red commit: `0405fb7` (failing tests)
- Green commit: `bb9dc34` (implementation)
Fixes#895
---------
Co-authored-by: you <you@example.com>
## Fix topology analytics double-counting repeaters/pairs (#909)
### Problem
`computeAnalyticsTopology()` aggregates by raw hop hex string. When
firmware emits variable-length path hashes (1-3 bytes per hop), the same
physical node appears multiple times with different prefix lengths (e.g.
`"07"`, `"0735bc"`, `"0735bc6d"` all referring to the same node). This
inflates repeater counts and creates duplicate pair entries.
### Solution
Added a confidence-gated dedup pass after frequency counting:
1. **For each hop prefix**, check if it resolves unambiguously (exactly
1 candidate in the prefix map)
2. **Unambiguous prefixes** → group by resolved pubkey, sum counts, keep
longest prefix as display identifier
3. **Ambiguous prefixes** (multiple candidates for that prefix) → left
as separate entries (conservative)
4. **Same treatment for pairs**: canonicalize by sorted pubkey pair
### Addressing @efiten's collision concern
At scale (~2000+ repeaters), 1-byte prefixes (256 buckets) WILL collide.
This fix explicitly checks the prefix map candidate count. Ambiguous
prefixes (where `len(pm.m[hop]) > 1`) are never merged — they remain as
separate entries. Only prefixes with a single matching node are eligible
for dedup.
### TDD
- **Red commit**: `4dbf9c0` — added 3 failing tests
- **Green commit**: `d6cae9a` — implemented dedup, all tests pass
### Tests added
- `TestTopologyDedup_RepeatersMergeByPubkey` — verifies entries with
different prefix lengths for same node merge to single entry with summed
count
- `TestTopologyDedup_AmbiguousPrefixNotMerged` — verifies colliding
short prefix stays separate from unambiguous longer prefix
- `TestTopologyDedup_PairsMergeByPubkey` — verifies pair entries merge
by resolved pubkey pair
Fixes#909
---------
Co-authored-by: you <you@example.com>
## Summary
Closes#431. Preserves scroll position on the packets page when filters
change or groups are expanded/collapsed.
## Problem
When an operator scrolls down through packet history then changes a
filter (type, observer, packet-filter expression) or expands/collapses a
group, `renderTableRows()` rebuilds the DOM which resets `scrollTop` to
0. This forces the user back to the top — frustrating when digging
through hundreds of packets.
## Fix
Save `scrollContainer.scrollTop` at the start of `renderTableRows()`,
restore it after DOM rebuild completes. Two restore points:
1. **Empty-results path** (line ~1821): after `tbody.innerHTML = ...`
2. **Normal virtual-scroll path** (line ~1840): after
`renderVisibleRows()`
### Key lines changed
- `public/packets.js` lines 1748–1749: save scrollTop
- `public/packets.js` line 1821: restore after empty-state DOM write
- `public/packets.js` line 1840: restore after renderVisibleRows
## TDD evidence
- **Red commit:** a99ba21 — test asserts scrollTop preserved; fails
without fix
- **Green commit:** 35cc4bf — adds save/restore; test passes
## Anti-tautology
Removing the `scrollContainer.scrollTop = savedScrollTop` lines causes
the test to fail (scrollTop becomes 0 instead of 500). Verified locally.
## Verification
- `node test-packets.js` — 83 passed, 0 failed
- `node test-packet-filter.js` — 62 passed, 0 failed
---------
Co-authored-by: you <you@example.com>
## Summary
Closes#871
The `/api/packets` endpoint could return packets with `null` hash or
timestamp fields. This was caused by legacy data in SQLite (rows with
empty `hash` or `NULL`/empty `first_seen`) predating the ingestor's
existing validation guard (`if hash == "" { return false, nil }` at
`cmd/ingestor/db.go:610`).
## Root Cause
`cmd/server/store.go` `filterPackets()` had no data-integrity guard.
Legacy rows with empty `hash` or `first_seen` were loaded into the
in-memory store and returned verbatim. The `strOrNil("")` helper then
serialized these as JSON `null`.
## Fix
Added a data-integrity predicate at the top of `filterPackets`'s scan
callback (`cmd/server/store.go:2278`):
```go
if tx.Hash == "" || tx.FirstSeen == "" {
return false
}
```
This filters bad legacy rows at query time. The write path (ingestor)
already rejects empty hashes, so no new bad data enters.
## TDD Evidence
- **Red commit:** `15774c3` — test `TestIssue871_NoNullHashOrTimestamp`
asserts no packet in API response has null/empty hash or timestamp
- **Green commit:** `281fd6f` — adds the filter guard, test passes
## Testing
- `go test ./...` in `cmd/server` passes (full suite)
- Client-side defensive filter from PR #868 remains as defense-in-depth
---------
Co-authored-by: you <you@example.com>
## Summary
Closes#889.
When a TRACE packet's payload is too short to decode (< 9 bytes),
`decodeTrace` returns an error in `Payload.Error` but the observation is
still stored with empty `Path.Hops`. Previously this was completely
silent — no log, no anomaly flag, no indication the row is degraded.
This fix populates `DecodedPacket.Anomaly` with the decode error message
(e.g., `"TRACE payload decode failed: too short"`) so operators and
downstream consumers can identify degraded observations.
## TDD Commit History
1. **Red commit** `04e0165` — failing test asserting `Anomaly` is set
when TRACE payload decode fails
2. **Green commit** `d3e72d1` — 3-line fix in `decoder.go` line 601-603:
check `payload.Error != ""` for TRACE packets and set anomaly
## What Changed
`cmd/ingestor/decoder.go` (lines 601-603): Added a check before the
existing TRACE path-parsing block. If `payload.Error` is non-empty for a
TRACE packet, `anomaly` is set to `"TRACE payload decode failed:
<error>"`.
`cmd/ingestor/decoder_test.go`: Added
`TestDecodeTracePayloadFailSetsAnomaly` — constructs a TRACE packet with
a 4-byte payload (too short), asserts the packet is still returned
(observation stored) and `Anomaly` is populated.
## Verification
- `go build ./...` ✓
- `go test ./...` ✓ (all pass including new test)
- Anti-tautology: reverting the fix causes the new test to fail (asserts
`pkt.Anomaly == ""` → error)
---------
Co-authored-by: you <you@example.com>
Closes#921
## Summary
Follow-up to #920 (incremental auto-vacuum). Addresses both items from
the adversarial review:
### 1. RW connection caching
Previously, every call to `openRW(dbPath)` opened a new SQLite RW
connection and closed it after use. This happened in:
- `runIncrementalVacuum` (~4x/hour)
- `PruneOldPackets`, `PruneOldMetrics`, `RemoveStaleObservers`
- `buildAndPersistEdges`, `PruneNeighborEdges`
- All neighbor persist operations
Now a single `*sql.DB` handle (with `MaxOpenConns(1)`) is cached
process-wide via `cachedRW(dbPath)`. The underlying connection pool
manages serialization. The original `openRW()` function is retained for
one-shot test usage.
### 2. DBConfig dedup
`DBConfig` was defined identically in both `cmd/server/config.go` and
`cmd/ingestor/config.go`. Extracted to `internal/dbconfig/` as a shared
package; both binaries now use a type alias (`type DBConfig =
dbconfig.DBConfig`).
## Tests added
| Test | File |
|------|------|
| `TestCachedRW_ReturnsSameHandle` | `cmd/server/rw_cache_test.go` |
| `TestCachedRW_100Calls_SingleConnection` |
`cmd/server/rw_cache_test.go` |
| `TestGetIncrementalVacuumPages_Default` |
`internal/dbconfig/dbconfig_test.go` |
| `TestGetIncrementalVacuumPages_Configured` |
`internal/dbconfig/dbconfig_test.go` |
## Verification
```
ok github.com/corescope/server 20.069s
ok github.com/corescope/ingestor 47.117s
ok github.com/meshcore-analyzer/dbconfig 0.003s
```
Both binaries build cleanly. 100 sequential `cachedRW()` calls return
the same handle with exactly 1 entry in the cache map.
---------
Co-authored-by: you <you@example.com>
## Summary
Fixes#756 — the customizer timestamp format setting (ISO/ISO+ms/locale)
and timezone (UTC/local) were not applied to chart X-axis labels,
tooltips, or certain inline timestamps in the analytics pages.
## Changes
### `public/app.js`
- Added `formatChartAxisLabel(date, shortForm)` — a shared helper that
reads the customizer's `timestampFormat` and `timestampTimezone`
preferences and formats dates for chart axes accordingly.
`shortForm=true` returns time-only (for intra-day charts),
`shortForm=false` returns date+time (for multi-day ranges).
### `public/analytics.js`
- `rfXAxisLabels()`: now calls `formatChartAxisLabel()` instead of
hardcoded `toLocaleTimeString()`
- `rfTooltipCircles()`: tooltip timestamps now use
`formatAbsoluteTimestamp()` instead of raw ISO
- Subpath detail first/last seen: now uses `formatAbsoluteTimestamp()`
- Neighbor graph last_seen: now uses `formatAbsoluteTimestamp()`
### `public/node-analytics.js`
- Packet timeline chart labels: now use `formatChartAxisLabel()`
(respects short vs long form based on time range)
- SNR over time chart labels: now use `formatChartAxisLabel()`
## Behavior by setting
| Setting | Chart axis (short) | Chart axis (long) |
|---------|-------------------|-------------------|
| ISO | `14:30` | `05-03 14:30` |
| ISO+ms | `14:30:05` | `05-03 14:30:05` |
| Locale | `2:30 PM` | `May 3, 2:30 PM` |
All respect the UTC/local timezone toggle.
## Testing
- Server builds cleanly (`go build`)
- Served `app.js` contains `formatChartAxisLabel` (verified via curl)
- Graceful fallback: all callsites check `typeof formatChartAxisLabel
=== 'function'` before calling, preserving backward compat if script
load order changes
---------
Co-authored-by: you <you@example.com>
Kpa-clawbot